From python-dev at zesty.ca  Sat Jul  1 00:00:26 2006
From: python-dev at zesty.ca (Ka-Ping Yee)
Date: Fri, 30 Jun 2006 17:00:26 -0500 (CDT)
Subject: [Python-Dev] 2.5 and beyond
In-Reply-To: <001201c69c65$b3869750$6402a8c0@arkdesktop>
References: <001201c69c65$b3869750$6402a8c0@arkdesktop>
Message-ID: <Pine.LNX.4.58.0606301658240.17937@server1.LFW.org>

On Fri, 30 Jun 2006, Andrew Koenig wrote:
> I saw messages out of sequence and did not realize that this would be a
> change in behavior from 2.4.  Sigh.

Yes, this is not a good time to change it.

> I hope Py3000 has lexical scoping a la Scheme...

Me too -- that would be really nice.


-- ?!ng

From python-dev at zesty.ca  Sat Jul  1 00:05:25 2006
From: python-dev at zesty.ca (Ka-Ping Yee)
Date: Fri, 30 Jun 2006 17:05:25 -0500 (CDT)
Subject: [Python-Dev] For sandboxing: alternative to crippling file()
In-Reply-To: <bbaeab100606301109v558005bdrab4d5b39c18c7654@mail.gmail.com>
References: <bbaeab100606291148o1e03d44ardbb0d3358aae7513@mail.gmail.com>
	<20060630175205.GA17748@code0.codespeak.net>
	<bbaeab100606301109v558005bdrab4d5b39c18c7654@mail.gmail.com>
Message-ID: <Pine.LNX.4.58.0606301701290.17937@server1.LFW.org>

On Fri, 30 Jun 2006, Brett Cannon wrote:
> On 6/30/06, Armin Rigo <arigo at tunes.org> wrote:
> >     >>> object.__subclasses__()
> >     [..., <type 'file'>]
> >
> > Maybe this one won't work if __subclasses__ is forbidden, but in general
> > I think there *will* be a way to find this object.
>
> Yeah, that's been my (what I thought was paranoid) feeling.  Glad I am not
> the only one who thinks that hiding file() is near impossible.

If you want to do this right, it should be about *making* hiding
possible.  If you can't hide things, it will be hard to get very far.

I realize that may be difficult for Python 2.x, but hiding is pretty
essential for security.  It would be really good to keep this in mind
for the design of Python 3k.  (It doesn't mean we can't have introspection,
just that we need to agree on some discipline for how to do it.)


-- ?!ng

From jimjjewett at gmail.com  Sat Jul  1 00:13:37 2006
From: jimjjewett at gmail.com (Jim Jewett)
Date: Fri, 30 Jun 2006 18:13:37 -0400
Subject: [Python-Dev] traceback regression
Message-ID: <fb6fbf560606301513r21c07edbg5f1c0333ebdf45c7@mail.gmail.com>

python.org/sf/1515343 fixes python.org/sf/1515163, which is a
new-in-2.5 regression.

On the one hand, the regression only affects

>>> raise "string1", "string2"

which is both obscure and deprecated.  On the other hand, it is a
regression, and it is something I bumped into while working with
unittest.

[Note that I probably won't be checking email again for a week, so I'm
afraid I won't be very responsive to comments.]

-jJ

From guido at python.org  Sat Jul  1 00:24:04 2006
From: guido at python.org (Guido van Rossum)
Date: Fri, 30 Jun 2006 15:24:04 -0700
Subject: [Python-Dev] 2.5 and beyond
In-Reply-To: <Pine.LNX.4.58.0606301658240.17937@server1.LFW.org>
References: <001201c69c65$b3869750$6402a8c0@arkdesktop>
	<Pine.LNX.4.58.0606301658240.17937@server1.LFW.org>
Message-ID: <ca471dc20606301524n52ba3c2dj870b650fc600282@mail.gmail.com>

On 6/30/06, Ka-Ping Yee <python-dev at zesty.ca> wrote:
> On Fri, 30 Jun 2006, Andrew Koenig wrote:
> > I saw messages out of sequence and did not realize that this would be a
> > change in behavior from 2.4.  Sigh.
>
> Yes, this is not a good time to change it.
>
> > I hope Py3000 has lexical scoping a la Scheme...
>
> Me too -- that would be really nice.

That's not a very constructive proposal (especially since I don't know
Scheme). Perhaps you could elaborate on what needs to change?

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From brett at python.org  Sat Jul  1 00:33:20 2006
From: brett at python.org (Brett Cannon)
Date: Fri, 30 Jun 2006 15:33:20 -0700
Subject: [Python-Dev] For sandboxing: alternative to crippling file()
In-Reply-To: <Pine.LNX.4.58.0606301701290.17937@server1.LFW.org>
References: <bbaeab100606291148o1e03d44ardbb0d3358aae7513@mail.gmail.com>
	<20060630175205.GA17748@code0.codespeak.net>
	<bbaeab100606301109v558005bdrab4d5b39c18c7654@mail.gmail.com>
	<Pine.LNX.4.58.0606301701290.17937@server1.LFW.org>
Message-ID: <bbaeab100606301533x5a6041dcw28ce954350be4d18@mail.gmail.com>

On 6/30/06, Ka-Ping Yee <python-dev at zesty.ca> wrote:
>
> On Fri, 30 Jun 2006, Brett Cannon wrote:
> > On 6/30/06, Armin Rigo <arigo at tunes.org> wrote:
> > >     >>> object.__subclasses__()
> > >     [..., <type 'file'>]
> > >
> > > Maybe this one won't work if __subclasses__ is forbidden, but in
> general
> > > I think there *will* be a way to find this object.
> >
> > Yeah, that's been my (what I thought was paranoid) feeling.  Glad I am
> not
> > the only one who thinks that hiding file() is near impossible.
>
> If you want to do this right, it should be about *making* hiding
> possible.  If you can't hide things, it will be hard to get very far.


Well, this is only file() we are worrying about leaking out.  Stuff from
import are the worry here.

I realize that may be difficult for Python 2.x, but hiding is pretty
> essential for security.  It would be really good to keep this in mind
> for the design of Python 3k.  (It doesn't mean we can't have
> introspection,
> just that we need to agree on some discipline for how to do it.)


That's fine; I personally have no issue with tweaking the security model for
Py3K.  But I am worrying about 2.x here.

-Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20060630/172e6d14/attachment-0001.htm 

From python-dev at zesty.ca  Sat Jul  1 01:03:14 2006
From: python-dev at zesty.ca (Ka-Ping Yee)
Date: Fri, 30 Jun 2006 18:03:14 -0500 (CDT)
Subject: [Python-Dev] Lexical scoping in Python 3k
In-Reply-To: <ca471dc20606301524n52ba3c2dj870b650fc600282@mail.gmail.com>
References: <001201c69c65$b3869750$6402a8c0@arkdesktop>
	<Pine.LNX.4.58.0606301658240.17937@server1.LFW.org>
	<ca471dc20606301524n52ba3c2dj870b650fc600282@mail.gmail.com>
Message-ID: <Pine.LNX.4.58.0606301726170.17937@server1.LFW.org>

On Fri, 30 Jun 2006, Guido van Rossum wrote:
> On 6/30/06, Ka-Ping Yee <python-dev at zesty.ca> wrote:
> > On Fri, 30 Jun 2006, Andrew Koenig wrote:
> > > I hope Py3000 has lexical scoping a la Scheme...
> >
> > Me too -- that would be really nice.
>
> That's not a very constructive proposal (especially since I don't know
> Scheme). Perhaps you could elaborate on what needs to change?

Sorry!  I should have been more clear.

Right now, Python supports the existence of nested scopes:

    a = 3
    def f():
        a = 4
        def g():
            a = 5
            print a     # prints 5
        g()
        print a         # prints 4
    f()
    print a             # prints 3

The above example shows that there are three distinct scopes, and that
each one has a distinct binding named 'a' -- assigning to one doesn't
affect the others.

    a = 3
    def f():
        b = 4
        def g():
            c = 5
            print a, b, c       # i can see all three
        g()
    f()

The above example shows that all of the scopes can be *read*.  But
in today's Python, not all of the scopes can be *written*.

    a = 3
    def f():
        b = 4
        def g():
            c = 5
            a, b, c = 0, 1, 2   # changes local c, not outer a and b
        g()
    f()

The code in g() can affect its own local, 'c', and it can affect
the global variable 'a' if it declares 'global a', but no matter
what you write in g(), it cannot assign to 'b' (or to any other
intermediate scope).

This is a strange limitation and it would be nice to remove it.
The asymmetry comes from Python having one foot in the new paradigm
of nested lexical scopes and one foot still in the older paradigm
of only two scopes, local and global.

Most other languages that support lexical scoping (including Scheme,
JavaScript, Ruby, Perl, E, Java, Smalltalk) provide a uniform way
to read and write to scopes at all levels.  This is done by letting
programmers specify the scope in which they want a variable bound
(usually with a keyword like "var" in JavaScript, "my" in Perl, or
"define" in E).

So here are some thoughts on how Python might be adjusted to support
this.  I'm not saying these would be the only ways, but at least
they're some ideas to start with.

In JavaScript, the "var" keyword is required whenever you want to
declare a local variable.  Anything without "var" is assumed to be
a global name.  The cleanest and most consistent solution that
comes to mind would be to adopt exactly this for Python.

Without "var":

    a = 3                       # global
    def f():
        b = 4                   # global
        def g():
            c = 5               # global
            a, b, c = 0, 1, 2   # changes all three globals
        g()
    f()
    print a, b, c               # prints 0, 1, 2

With "var":

    var a = 3                   # global
    def f():
        var b = 4               # local to f
        def g():
            var c = 5           # local to g
            a, b, c = 0, 1, 2   # changes outer a, outer b, and c
            print c             # prints 2
        g()
        print b                 # prints 1
    f()
    print a                     # prints 0
    print b                     # no such name
    print c                     # no such name

But that is a big change.  Perhaps it would be too unpythonic to
suddenly require declarations for all local variables.  So an
alternative would be to retain the default assumption that
undeclared variables are local.  Here's what we'd get:

Without "var":

    a = 3
    def f():
        b = 4
        def g():
            c = 5
            a, b, c = 0, 1, 2   # changes local c, not outer a and b
        g()
    f()

With "var":

    var a = 3
    def f():
        var b = 4
        def g():
            var c = 5
            a, b, c = 0, 1, 2   # changes outer a, outer b, and c
        g()
    f()

Now i think this is a little bit weird, because the statement
"var b = 4" in an outer scope changes the meaning of "b" in an
inner scope.  But it does have the virtue of retaining behaviour
compatible with today's Python, while offering a way to get proper
lexical scopes for those who want to use them.

Thoughts?  Other ideas?


-- ?!ng


From ark at acm.org  Sat Jul  1 01:04:53 2006
From: ark at acm.org (Andrew Koenig)
Date: Fri, 30 Jun 2006 19:04:53 -0400
Subject: [Python-Dev] 2.5 and beyond
In-Reply-To: <ca471dc20606301524n52ba3c2dj870b650fc600282@mail.gmail.com>
Message-ID: <000d01c69c99$9c023ec0$6402a8c0@arkdesktop>

> That's not a very constructive proposal (especially since I don't know
> Scheme). Perhaps you could elaborate on what needs to change?

The fundamental principle is that the binding of every name is determined
during compilation, not during execution.  This property does not quite
apply to Python at present.  For example:

	x = 42
	def f():
		y = x
		x = 123
		return y
	f()

This example fails with "local variable 'x' referenced before assignment"
because the compiler sees that f contains an assignment to x, so it makes x
a local variable, and then when you try to assign x to y during execution,
it fails.  This behavior is consistent with the notion of lexical scoping.

However, if I write

	def g():
		return x
	x = 42
	g()

the result is 42.  With lexical scoping, I believe it should be undefined.

The reason is that when the compiler encounters the definition of g,
variable x is not yet bound, and there is nothing in the body of g that
would bind it.  Therefore, g requires a binding to exist at the time it is
compiled; because no such binding exists, this example would be an error (at
compile time) under lexical scoping.







From scott+python-dev at scottdial.com  Sat Jul  1 01:13:58 2006
From: scott+python-dev at scottdial.com (Scott Dial)
Date: Fri, 30 Jun 2006 19:13:58 -0400
Subject: [Python-Dev] 2.5 and beyond
In-Reply-To: <ca471dc20606301524n52ba3c2dj870b650fc600282@mail.gmail.com>
References: <001201c69c65$b3869750$6402a8c0@arkdesktop>	<Pine.LNX.4.58.0606301658240.17937@server1.LFW.org>
	<ca471dc20606301524n52ba3c2dj870b650fc600282@mail.gmail.com>
Message-ID: <44A5B036.7030905@scottdial.com>

Guido van Rossum wrote:
> On 6/30/06, Ka-Ping Yee <python-dev at zesty.ca> wrote:
>> On Fri, 30 Jun 2006, Andrew Koenig wrote:
>>> I hope Py3000 has lexical scoping a la Scheme...
>> Me too -- that would be really nice.
> 
> That's not a very constructive proposal (especially since I don't know
> Scheme). Perhaps you could elaborate on what needs to change?

I believe the essence of their request for lexical scope boils down to 
allowing rebinding. Such code like the following is legal in Scheme:

def f(x):
     def incr():
         x = x + 1
         return x
     def decr():
         x = x - 1
         return x
     return (incr, decr)

(incr, decr) = f(1)
print incr() # 2
print incr() # 3
print decr() # 2
print decr() # 1

--
FWIW, the Scheme equivalent would be something like:

(define f (lambda (x)
   (list
     (lambda () (set! x (+ x 1)) x)
     (lambda () (set! x (- x 1)) x))))

(let ([fs (f 1)])
   (let ([incr (car fs)] [decr (cadr fs)])
     (display (incr)) (newline)    ; 2
     (display (incr)) (newline)    ; 3
     (display (decr)) (newline)    ; 2
     (display (decr)) (newline)))  ; 1


As a more personal aside, I can't imagine where I would use this in any 
python program I have ever wrote. I actually never noticed that 
rebinding wasn't allowed until recently.

-- 
Scott Dial
scott at scottdial.com
scodial at indiana.edu

From tim.peters at gmail.com  Sat Jul  1 01:23:56 2006
From: tim.peters at gmail.com (Tim Peters)
Date: Fri, 30 Jun 2006 19:23:56 -0400
Subject: [Python-Dev] 2.5 and beyond
In-Reply-To: <ca471dc20606301524n52ba3c2dj870b650fc600282@mail.gmail.com>
References: <001201c69c65$b3869750$6402a8c0@arkdesktop>
	<Pine.LNX.4.58.0606301658240.17937@server1.LFW.org>
	<ca471dc20606301524n52ba3c2dj870b650fc600282@mail.gmail.com>
Message-ID: <1f7befae0606301623q4072f138vb4b7c24ca7bbff08@mail.gmail.com>

[Andrew Koenig]
>>> I saw messages out of sequence and did not realize that this would be a
>>> change in behavior from 2.4.  Sigh.

[Ka-Ping Yee]
>> Yes, this is not a good time to change it.

>>> I hope Py3000 has lexical scoping a la Scheme...

>> Me too -- that would be really nice.

[Guido]
> That's not a very constructive proposal (especially since I don't know
> Scheme). Perhaps you could elaborate on what needs to change?

It's effectively the opposite of Python <0.1 wink>:  a name is local
to a scope in Scheme if and only if a declaration says it is.  For
example, the "let" family of forms is often used for this purpose, and

(let ((x 2) (y 3))    # declares x and y as local to this `let`, and
gives initial values
  (let ((x 7)
        (z (+ x y)))      # x comes from outer `let` so is 2, and z is 2+3=5
    (* z x)))        # x comes from inner `let`, so this is 5*7=35

If you use `let*` instead of `let` in the inner one, z picks up the
inner x=7, so that z is 7+3=10, and the result is 7*10 = 70 instead.
The bindings in a `let` "happen" in an undefined order.  In `let*`, a
binding is visible "to its right" within the `let*`.  Then there's
`letrec`, which allows establishing mutually recursive bindings.

While the `let` family is entirely about declaration, there are lots
of other forms that mix in some declaration as part of their purpose
(for example, the names in a lambda's argument list are local to the
lambda's body), but they're all explicit in Scheme.

I read "a la Scheme" here as "actually nothing like Scheme, except I
want a non-tricky way to rebind a name from an enclosing scope within
an enclosed scope".  In Scheme, the scope a name x belongs to is found
by searching enclosing scopes until you hit the first with an explicit
"x belongs to me" declaration (OK, there's a hokey fallback to "top
level" definitions too).  Searching "textually up" always suffices
(there's no form of delayed declaration -- a name must be declared
before use).  Scheme's assignment of assignment:

     (set! variable expression)

has nothing to do with establishing the scope of `variable`.

From ark at acm.org  Sat Jul  1 01:27:06 2006
From: ark at acm.org (Andrew Koenig)
Date: Fri, 30 Jun 2006 19:27:06 -0400
Subject: [Python-Dev] 2.5 and beyond
In-Reply-To: <5.1.1.6.0.20060630190709.01ef5858@sparrow.telecommunity.com>
Message-ID: <001201c69c9c$b6531fd0$6402a8c0@arkdesktop>


> That sounds like a bug, not a feature.  It's frequently useful to have
> forward references in function bodies to names that are not yet globally
> bound, e.g. for classes, or mutually-recursive functions.

The trouble is that you don't necessarily know in what scope they will be
defined, so I think that forcing you to be explicit about it is useful.  Can
you show me an example of where you think it isn't?

Incidentally, I think that lexical scoping would also deal with the problem
that people often encounter in which they have to write things like "lambda
x=x:" where one would think "lambda x:" would suffice.




From ark at acm.org  Sat Jul  1 01:29:45 2006
From: ark at acm.org (Andrew Koenig)
Date: Fri, 30 Jun 2006 19:29:45 -0400
Subject: [Python-Dev] 2.5 and beyond
In-Reply-To: <1f7befae0606301623q4072f138vb4b7c24ca7bbff08@mail.gmail.com>
Message-ID: <001401c69c9d$12d53900$6402a8c0@arkdesktop>

> I read "a la Scheme" here as "actually nothing like Scheme, except I
> want a non-tricky way to rebind a name from an enclosing scope within
> an enclosed scope".

Almost.  What I really want is for it to be possible to determine the
binding of every name by inspecting the source text of the program.  Right
now, it is often possible to do so, but sometimes it isn't.




From pje at telecommunity.com  Sat Jul  1 01:09:41 2006
From: pje at telecommunity.com (Phillip J. Eby)
Date: Fri, 30 Jun 2006 19:09:41 -0400
Subject: [Python-Dev] 2.5 and beyond
In-Reply-To: <000d01c69c99$9c023ec0$6402a8c0@arkdesktop>
References: <ca471dc20606301524n52ba3c2dj870b650fc600282@mail.gmail.com>
Message-ID: <5.1.1.6.0.20060630190709.01ef5858@sparrow.telecommunity.com>

At 07:04 PM 6/30/2006 -0400, Andrew Koenig wrote:
>However, if I write
>
>         def g():
>                 return x
>         x = 42
>         g()
>
>the result is 42.  With lexical scoping, I believe it should be undefined.
>
>The reason is that when the compiler encounters the definition of g,
>variable x is not yet bound, and there is nothing in the body of g that
>would bind it.  Therefore, g requires a binding to exist at the time it is
>compiled; because no such binding exists, this example would be an error (at
>compile time) under lexical scoping.

That sounds like a bug, not a feature.  It's frequently useful to have 
forward references in function bodies to names that are not yet globally 
bound, e.g. for classes, or mutually-recursive functions.


From tim.peters at gmail.com  Sat Jul  1 01:39:03 2006
From: tim.peters at gmail.com (Tim Peters)
Date: Fri, 30 Jun 2006 19:39:03 -0400
Subject: [Python-Dev] 2.5 and beyond
In-Reply-To: <001201c69c9c$b6531fd0$6402a8c0@arkdesktop>
References: <5.1.1.6.0.20060630190709.01ef5858@sparrow.telecommunity.com>
	<001201c69c9c$b6531fd0$6402a8c0@arkdesktop>
Message-ID: <1f7befae0606301639u6f8c968cr29afce0ee7fbb43b@mail.gmail.com>

[Andrew Koenig]
> ...
> Incidentally, I think that lexical scoping would also deal with the problem
> that people often encounter in which they have to write things like "lambda
> x=x:" where one would think "lambda x:" would suffice.

They _shouldn't_ encounter that at all anymore.  For example,

>>> def f(x):
...     return lambda: x+1
>>> f(3)()
4

works fine in modern Pythons.  Earlier Python's had a no-exceptions
3-scope implementation (local, global, builtin), and in those the "x"
in the lambda body was "not local" (was either global or builtin,
although the compiler couldn't tell which of those two it was).  In
_those_ Pythons people had to write "lambda x=x: x+1" instead, to suck
the binding of the outer x into the lambda body, but if people are
still doing that they're confused.  Modern Pythons do have lexical
scoping + global + builtin, although there's no way to spell "rebind a
name local to an outer scope from within an inner scope".

From guido at python.org  Sat Jul  1 01:44:02 2006
From: guido at python.org (Guido van Rossum)
Date: Fri, 30 Jun 2006 16:44:02 -0700
Subject: [Python-Dev] 2.5 and beyond
In-Reply-To: <001401c69c9d$12d53900$6402a8c0@arkdesktop>
References: <1f7befae0606301623q4072f138vb4b7c24ca7bbff08@mail.gmail.com>
	<001401c69c9d$12d53900$6402a8c0@arkdesktop>
Message-ID: <ca471dc20606301644o76336253tff7670ab087e181f@mail.gmail.com>

On 6/30/06, Andrew Koenig <ark at acm.org> wrote:
> > I read "a la Scheme" here as "actually nothing like Scheme, except I
> > want a non-tricky way to rebind a name from an enclosing scope within
> > an enclosed scope".
>
> Almost.  What I really want is for it to be possible to determine the
> binding of every name by inspecting the source text of the program.  Right
> now, it is often possible to do so, but sometimes it isn't.

Then your example

def f(): return x
x = 42
print f()

is entirely well-defined -- x is a global and the compiler in fact
generates code that benefits from knowing that it's not a local.

Python knows which locals there are; also which locals there are in
surrounding function scopes.

It *could* also know which globals and builtins there are, except the
language currently allows dynamic rebinding of module-level variables
so that they replace builtins.

E.g.

def f(): return len([])
print f()  # prints 0
def len(x): return "booh"
print f()  # prints "booh"
del len
print f()  # prints 0 again

Worse, instead if explicitly overriding len in the module, it could
have been an assignment to __main__.len in some other module.

We've been thinking on how to deal with this for years, since nobody
really likes it in all its freedom.

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From jcarlson at uci.edu  Sat Jul  1 01:56:58 2006
From: jcarlson at uci.edu (Josiah Carlson)
Date: Fri, 30 Jun 2006 16:56:58 -0700
Subject: [Python-Dev] Lexical scoping in Python 3k
In-Reply-To: <Pine.LNX.4.58.0606301726170.17937@server1.LFW.org>
References: <ca471dc20606301524n52ba3c2dj870b650fc600282@mail.gmail.com>
	<Pine.LNX.4.58.0606301726170.17937@server1.LFW.org>
Message-ID: <20060630163625.10D1.JCARLSON@uci.edu>


Ka-Ping Yee <python-dev at zesty.ca> wrote:
[snip lexical scoping option]
> Now i think this is a little bit weird, because the statement
> "var b = 4" in an outer scope changes the meaning of "b" in an
> inner scope.  But it does have the virtue of retaining behaviour
> compatible with today's Python, while offering a way to get proper
> lexical scopes for those who want to use them.
> 
> Thoughts?  Other ideas?

Using a keyword in an outer scope to state that a variable could be used
in a nested scope is counter to the current method for accessing a
parent scope with 'global'.  Using 'var' as the equivalent of 'global',
only for nested scopes, would be a more reasonable approach.

However, I'm -1 on the feature now, for the same reasons I've been -1 on
the feature for the last 2 times it has come up.  In many of the cases
where lexically nested scopes have been used to solve problems in Python,
and programmers have run into a 'limitation' where not being able to
modify a value in a parent scope has hindered them, the problem could
have been better solved with another method that was more readable, more
extensible, etc.

What I asked before, and what I'd like to ask again, is if there are any
_nontrivial uses_ of lexically nested scopes which are made cumbersome
by our inability to write to parent scopes.  If there aren't, then I'm
going to again have to argue against new syntax, keywords, and their use.
If there are, then we'll see how compelling such uses are.

 - Josiah


From python-dev at zesty.ca  Sat Jul  1 01:55:12 2006
From: python-dev at zesty.ca (Ka-Ping Yee)
Date: Fri, 30 Jun 2006 18:55:12 -0500 (CDT)
Subject: [Python-Dev] 2.5 and beyond
In-Reply-To: <000d01c69c99$9c023ec0$6402a8c0@arkdesktop>
References: <000d01c69c99$9c023ec0$6402a8c0@arkdesktop>
Message-ID: <Pine.LNX.4.58.0606301851540.17937@server1.LFW.org>


On Fri, 30 Jun 2006, Andrew Koenig wrote:
> The fundamental principle is that the binding of every name is determined
> during compilation, not during execution.  This property does not quite
> apply to Python at present.

I think this property does apply.  In your example:

> 	def g():
> 		return x
> 	x = 42
> 	g()
>
> the result is 42.

It is already known at compile time that the "return x" in g()
refers to an 'x' in the outer scope.  'x' cannot be a local variable
to g() because there are no statements in g() that bind 'x'.

Regardless of whether the binding itself exists yet, you (the reader)
and the compiler can know which scope to look in for the binding at
runtime.  Have i understood your desired property correctly?


-- ?!ng

From tim.peters at gmail.com  Sat Jul  1 01:55:38 2006
From: tim.peters at gmail.com (Tim Peters)
Date: Fri, 30 Jun 2006 19:55:38 -0400
Subject: [Python-Dev] 2.5 and beyond
In-Reply-To: <001401c69c9d$12d53900$6402a8c0@arkdesktop>
References: <1f7befae0606301623q4072f138vb4b7c24ca7bbff08@mail.gmail.com>
	<001401c69c9d$12d53900$6402a8c0@arkdesktop>
Message-ID: <1f7befae0606301655i2425c0a1lf8d956cad0f028b7@mail.gmail.com>

[Andrew Koenig]
> Almost.  What I really want is for it to be possible to determine the
> binding of every name by inspecting the source text of the program.  Right
> now, it is often possible to do so, but sometimes it isn't.

Local names are always determined at compile-time in Python.  What you
can't always determine is whether a non-local (to any enclosing scope)
name will end up getting resolved from the module globals or from
__builtin__.  The runtime error in:

def f():
    y = x
    x = 1

f()

doesn't occur because Python doesn't know "x" is local to "f" at
compile-time (it does know that), it's because Python's compiler
doesn't do any flow analysis to detect potential
use-before-definition.  Instead the runtime initalizes locals to a
special "not bound yet" value that the LOAD_FAST (really "load local")
opcode special-cases.

Note that this is quite unlike Scheme, in which declaration must
appear before use (ignoring fancy letrec cases), and declaration must
also supply an initial binding (Scheme has no "unbound local" problem
because there's no way to create an uninitialized local).

From rasky at develer.com  Sat Jul  1 02:01:00 2006
From: rasky at develer.com (Giovanni Bajo)
Date: Sat, 1 Jul 2006 02:01:00 +0200
Subject: [Python-Dev] 2.5 and beyond
References: <5.1.1.6.0.20060630190709.01ef5858@sparrow.telecommunity.com><001201c69c9c$b6531fd0$6402a8c0@arkdesktop>
	<1f7befae0606301639u6f8c968cr29afce0ee7fbb43b@mail.gmail.com>
Message-ID: <020c01c69ca1$754c7310$d1b12997@bagio>

Tim Peters <tim.peters at gmail.com> wrote:

>> ...
>> Incidentally, I think that lexical scoping would also deal with the
>> problem
>> that people often encounter in which they have to write things like
>> "lambda
>> x=x:" where one would think "lambda x:" would suffice.
>
> They _shouldn't_ encounter that at all anymore.  For example,
>
>>>> def f(x):
> ...     return lambda: x+1
>>>> f(3)()
> 4
>
> works fine in modern Pythons.

Yes but:

>>> a = []
>>> for i in range(10):
...     a.append(lambda: i)
...
>>> print [x() for x in a]
[9, 9, 9, 9, 9, 9, 9, 9, 9, 9]

This subtle semantic of lambda is quite confusing, and still forces people to
use the "i=i" trick.

Giovanni Bajo


From tim.peters at gmail.com  Sat Jul  1 02:15:51 2006
From: tim.peters at gmail.com (Tim Peters)
Date: Fri, 30 Jun 2006 20:15:51 -0400
Subject: [Python-Dev] 2.5 and beyond
In-Reply-To: <020c01c69ca1$754c7310$d1b12997@bagio>
References: <5.1.1.6.0.20060630190709.01ef5858@sparrow.telecommunity.com>
	<001201c69c9c$b6531fd0$6402a8c0@arkdesktop>
	<1f7befae0606301639u6f8c968cr29afce0ee7fbb43b@mail.gmail.com>
	<020c01c69ca1$754c7310$d1b12997@bagio>
Message-ID: <1f7befae0606301715m6fe7044eue4e5c8d74a7c8a22@mail.gmail.com>

[Giovanni Bajo]
> Yes but:
>
> >>> a = []
> >>> for i in range(10):
> ...     a.append(lambda: i)
> ...
> >>> print [x() for x in a]
> [9, 9, 9, 9, 9, 9, 9, 9, 9, 9]
>
> This subtle semantic of lambda is quite confusing, and still forces people to
> use the "i=i" trick.

So stay away from excruciating abuses of lexical scoping you don't
understand ;-)  What do you _expect_ `i` to refer to?  "Oh, it should
guess that I didn't really mean to defer evaluation of the lambda body
at all, but instead evaluate the lambda body at the time I define the
lambda and then synthesize some other function that captures the
specific outer bindings in effect at lambda-definition time" doesn't
really cut it.

Try spelling what you think you want here in Scheme.  Before it works,
you'll probably end up with some equally "atrocious" (let ((i i)) ...)
gimmick to force capturing each binding for `i` as it flies by.  Else
Scheme will also use the outer binding for `i` in effect at the time
the lambdas are _executed_.

This isn't typical use for lambda, and I don't think it's what Andrew
had in mind.

From bjourne at gmail.com  Sat Jul  1 03:11:59 2006
From: bjourne at gmail.com (=?ISO-8859-1?Q?BJ=F6rn_Lindqvist?=)
Date: Sat, 1 Jul 2006 03:11:59 +0200
Subject: [Python-Dev] Lexical scoping in Python 3k
In-Reply-To: <Pine.LNX.4.58.0606301726170.17937@server1.LFW.org>
References: <001201c69c65$b3869750$6402a8c0@arkdesktop>
	<Pine.LNX.4.58.0606301658240.17937@server1.LFW.org>
	<ca471dc20606301524n52ba3c2dj870b650fc600282@mail.gmail.com>
	<Pine.LNX.4.58.0606301726170.17937@server1.LFW.org>
Message-ID: <740c3aec0606301811i6b323bei3abcfb05cde5b747@mail.gmail.com>

> With "var":
>
>     var a = 3
>     def f():
>         var b = 4
>         def g():
>             var c = 5
>             a, b, c = 0, 1, 2   # changes outer a, outer b, and c
>         g()
>     f()
>
> Now i think this is a little bit weird, because the statement
> "var b = 4" in an outer scope changes the meaning of "b" in an
> inner scope.  But it does have the virtue of retaining behaviour
> compatible with today's Python, while offering a way to get proper
> lexical scopes for those who want to use them.
>
> Thoughts?  Other ideas?


Maybe an object, like self, for referring to enclosing scopes?

     a = 3
     def f():
         b = 4
         def g():
             c = 5
             outer.outer.a, outer.b, c = 0, 1, 2   # changes outer a,
outer b, and c
         g()
     f()

Chaining the keyword looks a little weird, but it is not often that
you have to refer to variables in the enclosing scope of the enclosing
scope. I have often wanted something similar to that for global
variables, instead of the global declaration:

cache = None
def init():
    if not global.cache:
        global.cache = init_cache()

-- 
mvh Bj?rn

From janssen at parc.com  Sat Jul  1 03:31:12 2006
From: janssen at parc.com (Bill Janssen)
Date: Fri, 30 Jun 2006 18:31:12 PDT
Subject: [Python-Dev] 2.5 and beyond
In-Reply-To: Your message of "Fri, 30 Jun 2006 17:01:00 PDT."
	<020c01c69ca1$754c7310$d1b12997@bagio> 
Message-ID: <06Jun30.183119pdt."58641"@synergy1.parc.xerox.com>

> >>> a = []
> >>> for i in range(10):
> ...     a.append(lambda: i)
> ...
> >>> print [x() for x in a]
> [9, 9, 9, 9, 9, 9, 9, 9, 9, 9]

Isn't this exactly what you'd expect?  Maybe I've been writing Python
for too long... :-).

Bill

From rasky at develer.com  Sat Jul  1 03:43:01 2006
From: rasky at develer.com (Giovanni Bajo)
Date: Sat, 1 Jul 2006 03:43:01 +0200
Subject: [Python-Dev] 2.5 and beyond
References: <5.1.1.6.0.20060630190709.01ef5858@sparrow.telecommunity.com>
	<001201c69c9c$b6531fd0$6402a8c0@arkdesktop>
	<1f7befae0606301639u6f8c968cr29afce0ee7fbb43b@mail.gmail.com>
	<020c01c69ca1$754c7310$d1b12997@bagio>
	<1f7befae0606301715m6fe7044eue4e5c8d74a7c8a22@mail.gmail.com>
Message-ID: <027f01c69caf$b1c20450$d1b12997@bagio>

[Giovanni Bajo]
> Yes but:
>
>>>> a = []
>>>> for i in range(10):
> ...     a.append(lambda: i)
> ...
>>>> print [x() for x in a]
> [9, 9, 9, 9, 9, 9, 9, 9, 9, 9]
>
> This subtle semantic of lambda is quite confusing, and still forces people to
> use the "i=i" trick.

[Tim Peters]
> So stay away from excruciating abuses of lexical scoping you don't
> understand   What do you expect `i` to refer to?  "Oh, it should
> guess that I didn't really mean to defer evaluation of the lambda body
> at all, but instead evaluate the lambda body at the time I define the
> lambda and then synthesize some other function that captures the
> specific outer bindings in effect at lambda-definition time" doesn't
> really cut it.

I think I understand what happens, I just don't know whether this can be
"fixed" or not. Unless you are saying that the above behaviour is not only a
complex side-effect the way things are, but the way things should be. Do you
agree that it would be ideal if the above code generated range(10) instead of
[9]*10, or you believe that the current behaviour is more sound (and if so,
why)?

As for actual implementing this change of semantic, the fact that `i` is a
local variable in the outer scope (assuming it's all within a function),
doesn't make it possible for Python to early-bound it, by realizing that, since
`i` is not an argument of the lambda, and it's a local of the outer scope? At
worse, couldn't Python do the "i=i" trick by itself when it sees that `i` is a
local in the outer scope? Right now I can't think off-hand of a case in which
this would break things.

[Tim Peters]
> This isn't typical use for lambda,


Yes, maybe it's not the most used idiom and Andrew wasn't referring to this,
but it happens quite often to me (where 'often' means 'many times' among my
rare usages of lambda).

For instance, in GUI code, it's common to do things like:

for b in self.buttons:
     self.setEventCallback(b, "clicked", lambda: self.label.setText("I pressed
button %r" % b))

... which of course won't work, as written above.

Giovanni Bajo
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/gif
Size: 135 bytes
Desc: not available
Url : http://mail.python.org/pipermail/python-dev/attachments/20060701/ebb3a2b4/attachment.gif 

From guido at python.org  Sat Jul  1 04:55:03 2006
From: guido at python.org (Guido van Rossum)
Date: Fri, 30 Jun 2006 19:55:03 -0700
Subject: [Python-Dev] ImportWarning flood
In-Reply-To: <44A5D5DA.2060601@hathawaymix.org>
References: <ca471dc20606251751q36f2accbr3ff3fe8fbd24b20c@mail.gmail.com>
	<20060630194331.29014.836424634.divmod.quotient.16780@ohm>
	<ca471dc20606301251m6e05f50ayd81e6ba3afd33095@mail.gmail.com>
	<44A5D5DA.2060601@hathawaymix.org>
Message-ID: <ca471dc20606301955i270d466ka1f3076d8390df28@mail.gmail.com>

It's up to the release manager now to decide whether the pitchforks at
Google or the pitchforks in the larger Python community are sharper.
;-)

--Guido (ducks)

On 6/30/06, Shane Hathaway <shane at hathawaymix.org> wrote:
> Guido van Rossum wrote:
> > On 6/30/06, Jean-Paul Calderone <exarkun at divmod.com> wrote:
> >> How about if someone grovels through import.c and figures out how to make
> >> the warning information only show up if the import actually fails?
> >
> > That would work I think. But it's not easy.
>
> I just posted a patch intended to solve this.  With my patch,
> find_module() collects the warnings in a list and only generates
> ImportWarning if no package is found.  It works, but I have not done
> extensive testing.
>
> I also discovered and tried to resolve what appear to be memory leaks
> involving the "copy" variable.  If the ImportWarning patch doesn't fly,
> I'll be happy to post a different patch that only fixes the leaks.
>
> The patch is #1515361:
>
> https://sourceforge.net/tracker/index.php?func=detail&aid=1515361&group_id=5470&atid=305470
>
> Shane
>
>


-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From skip at pobox.com  Sat Jul  1 05:32:01 2006
From: skip at pobox.com (skip at pobox.com)
Date: Fri, 30 Jun 2006 22:32:01 -0500
Subject: [Python-Dev] zlib module build failure on Mac OSX 10.4.7
Message-ID: <17573.60593.759293.549583@montanaro.dyndns.org>

Just upgraded my Mac to OSX 10.4.7 yesterday.  svn up'd Python trunk, then
"make clean ; configure ; make" and I see that building the zlib module
fails:

    gcc -fno-strict-aliasing -Wno-long-double -no-cpp-precomp -mno-fused-madd -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -I. -I/Users/skip/src/python-svn/trunk/./Include -I/Users/skip/src/python-svn/trunk/./Mac/Include -I/Users/skip/local/include -I../Include -I. -I/usr/local/include -I/Users/skip/src/python-svn/trunk/Include -I/Users/skip/src/python-svn/trunk/build -c /Users/skip/src/python-svn/trunk/Modules/zlibmodule.c -o build/temp.macosx-10.3-ppc-2.5/Users/skip/src/python-svn/trunk/Modules/zlibmodule.o
    /Users/skip/src/python-svn/trunk/Modules/zlibmodule.c: In function 'PyZlib_uncopy':
    /Users/skip/src/python-svn/trunk/Modules/zlibmodule.c:724: warning: implicit declaration of function 'inflateCopy'
    gcc -bundle -undefined dynamic_lookup build/temp.macosx-10.3-ppc-2.5/Users/skip/src/python-svn/trunk/Modules/zlibmodule.o -L/Users/skip/local/lib -L/usr/local/lib -lz -o build/lib.macosx-10.3-ppc-2.5/zlib.so -Wl,-search_paths_first
    *** WARNING: renaming "zlib" since importing it failed: dlopen(build/lib.macosx-10.3-ppc-2.5/zlib.so, 2): Symbol not found: _inflateCopy
      Referenced from: build/lib.macosx-10.3-ppc-2.5/zlib.so
      Expected in: dynamic lookup

Anybody else seen this?  I checked the buildbot trunk osx 10.4.  It seemed
to have no trouble.  And what's with the "10.3" bit in the directory names?

Skip


From greg.ewing at canterbury.ac.nz  Sat Jul  1 05:45:02 2006
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sat, 01 Jul 2006 15:45:02 +1200
Subject: [Python-Dev] doc for new restricted execution design for Python
In-Reply-To: <043001c69b45$35742290$100a0a0a@enfoldsystems.local>
References: <043001c69b45$35742290$100a0a0a@enfoldsystems.local>
Message-ID: <44A5EFBE.9050300@canterbury.ac.nz>

Mark Hammond wrote:

> that helps "mozilla the platform" more than it helps "firebox the browser"
                                                         ^^^^^^^

Firebox - the sandfoxed web browser!

--
Greg

From tjreedy at udel.edu  Sat Jul  1 06:27:16 2006
From: tjreedy at udel.edu (Terry Reedy)
Date: Sat, 1 Jul 2006 00:27:16 -0400
Subject: [Python-Dev] 2.5 and beyond
References: <5.1.1.6.0.20060630190709.01ef5858@sparrow.telecommunity.com><001201c69c9c$b6531fd0$6402a8c0@arkdesktop><1f7befae0606301639u6f8c968cr29afce0ee7fbb43b@mail.gmail.com>
	<020c01c69ca1$754c7310$d1b12997@bagio>
Message-ID: <e84tj3$sj8$1@sea.gmane.org>


"Giovanni Bajo" <rasky at develer.com> wrote in message 
news:020c01c69ca1$754c7310$d1b12997 at bagio...
> Yes but:
>
>>>> a = []
>>>> for i in range(10):
> ...     a.append(lambda: i)
> ...
>>>> print [x() for x in a]
> [9, 9, 9, 9, 9, 9, 9, 9, 9, 9]
>
> This subtle semantic of lambda is quite confusing, and still forces 
> people to
> use the "i=i" trick.

The 'subtle sematic' had nothing to do with lambda but with Python 
functions.

The above is exactly equivalent (except the different .funcname) to

a = []
for i in range(10):
    def f(): return i
    a.append(f)
    del f

That should be equally confusing (or not), and equally requires the 'i=i' 
trick (or not).

As is, either function definitiion is a constant and the loop makes useless 
duplicates.  Either form would have the same effect is hoisted out of the 
loop.

Terry Jan Reedy




From greg.ewing at canterbury.ac.nz  Sat Jul  1 06:59:19 2006
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sat, 01 Jul 2006 16:59:19 +1200
Subject: [Python-Dev] For sandboxing: alternative to crippling file()
In-Reply-To: <bbaeab100606291148o1e03d44ardbb0d3358aae7513@mail.gmail.com>
References: <bbaeab100606291148o1e03d44ardbb0d3358aae7513@mail.gmail.com>
Message-ID: <44A60127.3030706@canterbury.ac.nz>

Brett Cannon wrote:

> 1) Is removing 'file' from the builtins dict in PyInterpreterState (and 
> maybe some other things) going to be safe enough to sufficiently hide 
> 'file' confidently (short of someone being stupid in their C extension 
> module and exposing 'file' directly)?
> 
> 2) Changing open() to return C-implemented delegate objects for files 
> (and thus won't type check, but this is Python so I am not worried about 
> that too much) and delegate socket objects for IP and URL addresses.

My suggestion is to change things so that the constructor
of the file type doesn't open files (at least in restricted
mode). Then it wouldn't matter if untrusted code had real
file objects, as they couldn't use them to get access to
any other files.

--
Greg

From tjreedy at udel.edu  Sat Jul  1 07:00:40 2006
From: tjreedy at udel.edu (Terry Reedy)
Date: Sat, 1 Jul 2006 01:00:40 -0400
Subject: [Python-Dev] 2.5 and beyond
References: <5.1.1.6.0.20060630190709.01ef5858@sparrow.telecommunity.com><001201c69c9c$b6531fd0$6402a8c0@arkdesktop><1f7befae0606301639u6f8c968cr29afce0ee7fbb43b@mail.gmail.com><020c01c69ca1$754c7310$d1b12997@bagio><1f7befae0606301715m6fe7044eue4e5c8d74a7c8a22@mail.gmail.com>
	<027f01c69caf$b1c20450$d1b12997@bagio>
Message-ID: <e84vhn$c8$1@sea.gmane.org>


"Giovanni Bajo" <rasky at develer.com> wrote in message 
news:027f01c69caf$b1c20450$d1b12997 at bagio...
> [Giovanni Bajo]
>> Yes but:
>>
>>>>> a = []
>>>>> for i in range(10):
>> ...     a.append(lambda: i)
>> ...
>>>>> print [x() for x in a]
>> [9, 9, 9, 9, 9, 9, 9, 9, 9, 9]

>. Do you  agree that it would be ideal if the above code
> generated range(10) instead of  [9]*10,

No.  You are trying to reify an optical illusion resulting from putting a 
constant function definition inside a loop.  Making the meaning of  'def 
f(): return i' depend on the definition-time context by partially and 
variably evaluating the body would make code much harder to read and 
understand.  Consider:

if a: i=666
<intervening code>
def f(): return i

> At
> worse, couldn't Python do the "i=i" trick by itself when it sees that `i` 
> is a
> local in the outer scope? Right now I can't think off-hand of a case in 
> which
> this would break things.

It would make code more fragile.

for i in range(666): print name[i]
...
<intervening code>
...
def total(num): return cost[item]*num

Now someone decides first loop should have more expressive loop var name 
and changes the first line to

for item in range(666): print name[item]

and the meaning of total is completely changed.  Adding such long-range 
coupling between language statements strikes me as a poor idea.

Terry Jan Reedy




From greg.ewing at canterbury.ac.nz  Sat Jul  1 07:39:05 2006
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sat, 01 Jul 2006 17:39:05 +1200
Subject: [Python-Dev] Lexical scoping in Python 3k
In-Reply-To: <Pine.LNX.4.58.0606301726170.17937@server1.LFW.org>
References: <001201c69c65$b3869750$6402a8c0@arkdesktop>
	<Pine.LNX.4.58.0606301658240.17937@server1.LFW.org>
	<ca471dc20606301524n52ba3c2dj870b650fc600282@mail.gmail.com>
	<Pine.LNX.4.58.0606301726170.17937@server1.LFW.org>
Message-ID: <44A60A79.8090802@canterbury.ac.nz>

Ka-Ping Yee wrote:

 > while offering a way to get proper
> lexical scopes for those who want to use them.

I don't disagree with anything you said, but I think it
would be a good idea to avoid using phrases like "proper
lexical scopes", which is likely to set people off on
a tangent. The issue isn't lexicality, it's writeability.

--
Greg

From nnorwitz at gmail.com  Sat Jul  1 07:53:26 2006
From: nnorwitz at gmail.com (Neal Norwitz)
Date: Fri, 30 Jun 2006 22:53:26 -0700
Subject: [Python-Dev] zlib module build failure on Mac OSX 10.4.7
In-Reply-To: <17573.60593.759293.549583@montanaro.dyndns.org>
References: <17573.60593.759293.549583@montanaro.dyndns.org>
Message-ID: <ee2a432c0606302253o69d2bdc8i904a7a29fcc2e1b1@mail.gmail.com>

Maybe do a make distclean.  There was a problem where old versions of
zlib (those without inflateCopy) weren't supported.  They are now, but
it's a configure check.  That coupled with the upgrade and the 10.3 in
the pathname, seems like it's just something didn't get cleaned up
properly.  You could always rm -rf build/

n
--

On 6/30/06, skip at pobox.com <skip at pobox.com> wrote:
> Just upgraded my Mac to OSX 10.4.7 yesterday.  svn up'd Python trunk, then
> "make clean ; configure ; make" and I see that building the zlib module
> fails:
>
>     gcc -fno-strict-aliasing -Wno-long-double -no-cpp-precomp -mno-fused-madd -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -I. -I/Users/skip/src/python-svn/trunk/./Include -I/Users/skip/src/python-svn/trunk/./Mac/Include -I/Users/skip/local/include -I../Include -I. -I/usr/local/include -I/Users/skip/src/python-svn/trunk/Include -I/Users/skip/src/python-svn/trunk/build -c /Users/skip/src/python-svn/trunk/Modules/zlibmodule.c -o build/temp.macosx-10.3-ppc-2.5/Users/skip/src/python-svn/trunk/Modules/zlibmodule.o
>     /Users/skip/src/python-svn/trunk/Modules/zlibmodule.c: In function 'PyZlib_uncopy':
>     /Users/skip/src/python-svn/trunk/Modules/zlibmodule.c:724: warning: implicit declaration of function 'inflateCopy'
>     gcc -bundle -undefined dynamic_lookup build/temp.macosx-10.3-ppc-2.5/Users/skip/src/python-svn/trunk/Modules/zlibmodule.o -L/Users/skip/local/lib -L/usr/local/lib -lz -o build/lib.macosx-10.3-ppc-2.5/zlib.so -Wl,-search_paths_first
>     *** WARNING: renaming "zlib" since importing it failed: dlopen(build/lib.macosx-10.3-ppc-2.5/zlib.so, 2): Symbol not found: _inflateCopy
>       Referenced from: build/lib.macosx-10.3-ppc-2.5/zlib.so
>       Expected in: dynamic lookup
>
> Anybody else seen this?  I checked the buildbot trunk osx 10.4.  It seemed
> to have no trouble.  And what's with the "10.3" bit in the directory names?
>
> Skip
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/nnorwitz%40gmail.com
>

From greg.ewing at canterbury.ac.nz  Sat Jul  1 08:14:28 2006
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sat, 01 Jul 2006 18:14:28 +1200
Subject: [Python-Dev] 2.5 and beyond
In-Reply-To: <001201c69c9c$b6531fd0$6402a8c0@arkdesktop>
References: <001201c69c9c$b6531fd0$6402a8c0@arkdesktop>
Message-ID: <44A612C4.6020908@canterbury.ac.nz>

Andrew Koenig wrote:

> Incidentally, I think that lexical scoping would also deal with the problem
> that people often encounter in which they have to write things like "lambda
> x=x:" where one would think "lambda x:" would suffice.

This is another red herring. Python's problem here
is not because its scoping isn't lexical (it is).
It's because Scheme implicitly introduces new scopes
in various places where the equivalent Python
constructs don't.

--
Greg

From greg.ewing at canterbury.ac.nz  Sat Jul  1 08:14:54 2006
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sat, 01 Jul 2006 18:14:54 +1200
Subject: [Python-Dev] 2.5 and beyond
In-Reply-To: <1f7befae0606301655i2425c0a1lf8d956cad0f028b7@mail.gmail.com>
References: <1f7befae0606301623q4072f138vb4b7c24ca7bbff08@mail.gmail.com>
	<001401c69c9d$12d53900$6402a8c0@arkdesktop>
	<1f7befae0606301655i2425c0a1lf8d956cad0f028b7@mail.gmail.com>
Message-ID: <44A612DE.1080503@canterbury.ac.nz>

Tim Peters wrote:

> Note that this is quite unlike Scheme, in which declaration must
> appear before use (ignoring fancy letrec cases),

I think that's overstating things a bit -- mutually
recursive functions are quite easy to write in Scheme
and don't look at all "fancy" (unless you object for
some reason to using (define ...)).

> and declaration must
> also supply an initial binding (Scheme has no "unbound local" problem
> because there's no way to create an uninitialized local).

That much is true.

--
Greg

From greg.ewing at canterbury.ac.nz  Sat Jul  1 08:15:04 2006
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sat, 01 Jul 2006 18:15:04 +1200
Subject: [Python-Dev] Lexical scoping in Python 3k
In-Reply-To: <20060630163625.10D1.JCARLSON@uci.edu>
References: <ca471dc20606301524n52ba3c2dj870b650fc600282@mail.gmail.com>
	<Pine.LNX.4.58.0606301726170.17937@server1.LFW.org>
	<20060630163625.10D1.JCARLSON@uci.edu>
Message-ID: <44A612E8.8050205@canterbury.ac.nz>

Josiah Carlson wrote:

> What I asked before, and what I'd like to ask again, is if there are any
> _nontrivial uses_ of lexically nested scopes which are made cumbersome
> by our inability to write to parent scopes.

The trouble with taking that position is that the very
cases which would benefit are very *simple* ones, where
it would be cumbersome to refactor it to use a class,
or mutable object in the outer scope, etc. So you've
effectively set up your acceptance criteria to be
unmeetable.

> If there aren't, then I'm
> going to again have to argue against new syntax, keywords, and their use.

There's one very simple way we could do this in Py3k
without requiring any new syntax or keywords: just
redefine the meaning of "global" to mean "not local".

--
Greg

From greg.ewing at canterbury.ac.nz  Sat Jul  1 08:15:29 2006
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sat, 01 Jul 2006 18:15:29 +1200
Subject: [Python-Dev] 2.5 and beyond
In-Reply-To: <020c01c69ca1$754c7310$d1b12997@bagio>
References: <5.1.1.6.0.20060630190709.01ef5858@sparrow.telecommunity.com>
	<001201c69c9c$b6531fd0$6402a8c0@arkdesktop>
	<1f7befae0606301639u6f8c968cr29afce0ee7fbb43b@mail.gmail.com>
	<020c01c69ca1$754c7310$d1b12997@bagio>
Message-ID: <44A61301.2070805@canterbury.ac.nz>

Giovanni Bajo wrote:

>>>>a = []
>>>>for i in range(10):
> 
> ...     a.append(lambda: i)
> ...
> 
>>>>print [x() for x in a]
> 
> [9, 9, 9, 9, 9, 9, 9, 9, 9, 9]
> 
> This subtle semantic of lambda is quite confusing, and still forces people to
> use the "i=i" trick.

This has *nothing* to do with the semantics of lambda!
It's because Python's for-loop doesn't put its control
variable in a new scope, the way Scheme's equivalent
construct does.

*That's* what needs to be addressed to fix this problem.
I've made a suggestion about that before, but Guido
rejected it, so I won't repeat it here.

--
Greg


From tim.peters at gmail.com  Sat Jul  1 09:28:35 2006
From: tim.peters at gmail.com (Tim Peters)
Date: Sat, 1 Jul 2006 03:28:35 -0400
Subject: [Python-Dev] 2.5 and beyond
In-Reply-To: <44A612DE.1080503@canterbury.ac.nz>
References: <1f7befae0606301623q4072f138vb4b7c24ca7bbff08@mail.gmail.com>
	<001401c69c9d$12d53900$6402a8c0@arkdesktop>
	<1f7befae0606301655i2425c0a1lf8d956cad0f028b7@mail.gmail.com>
	<44A612DE.1080503@canterbury.ac.nz>
Message-ID: <1f7befae0607010028q5da09c6ahb2f8c6358dbcd510@mail.gmail.com>

[Tim Peters]
>> Note that this is quite unlike Scheme, in which declaration must
>> appear before use (ignoring fancy letrec cases),

[Greg Ewing]
> I think that's overstating things a bit --

So do I :-), but I don't really care about Scheme here.

> mutually recursive functions are quite easy to write in
> Scheme and don't look at all "fancy" (unless you object for
> some reason to using (define ...)).

In this context, yes, I object to using "define", because the
semantics of internal definitions are defined in terms of an
equivalent (letrec ...) form.  The "fancy" gimmick is that letrec
views all its bindings as occurring simultaneously, so strains a
natural, linear understanding of "no use before declaration".

But none of this appears to have any relevance to Python, so I'm
happiest _here_ just calling that "fancy" and ignoring the details.
Ditto "top level" definitions, which have unique rules of their own.

From anthony at interlink.com.au  Sat Jul  1 09:57:59 2006
From: anthony at interlink.com.au (Anthony Baxter)
Date: Sat, 1 Jul 2006 17:57:59 +1000
Subject: [Python-Dev] ImportWarning flood
In-Reply-To: <ca471dc20606301955i270d466ka1f3076d8390df28@mail.gmail.com>
References: <ca471dc20606251751q36f2accbr3ff3fe8fbd24b20c@mail.gmail.com>
	<44A5D5DA.2060601@hathawaymix.org>
	<ca471dc20606301955i270d466ka1f3076d8390df28@mail.gmail.com>
Message-ID: <200607011758.03756.anthony@interlink.com.au>

On Saturday 01 July 2006 12:55, Guido van Rossum wrote:
> It's up to the release manager now to decide whether the pitchforks
> at Google or the pitchforks in the larger Python community are
> sharper. ;-)

At this point, I think removing the warning code is the prudent 
course. If someone wanted to find an easy and safe way to make it 
only be triggered when the import fails, it could stay in. I'm not 
convinced that _anything_ in import.c is easy and safe. <wink>

Anthony

-- 
Anthony Baxter     <anthony at interlink.com.au>
It's never too late to have a happy childhood.

From ncoghlan at gmail.com  Sat Jul  1 10:01:29 2006
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 01 Jul 2006 18:01:29 +1000
Subject: [Python-Dev] ImportWarning flood
In-Reply-To: <ca471dc20606301955i270d466ka1f3076d8390df28@mail.gmail.com>
References: <ca471dc20606251751q36f2accbr3ff3fe8fbd24b20c@mail.gmail.com>	<20060630194331.29014.836424634.divmod.quotient.16780@ohm>	<ca471dc20606301251m6e05f50ayd81e6ba3afd33095@mail.gmail.com>	<44A5D5DA.2060601@hathawaymix.org>
	<ca471dc20606301955i270d466ka1f3076d8390df28@mail.gmail.com>
Message-ID: <44A62BD9.7000906@gmail.com>

Guido van Rossum wrote:
> It's up to the release manager now to decide whether the pitchforks at
> Google or the pitchforks in the larger Python community are sharper.
> ;-)
> 
> --Guido (ducks)

I vaguely recall one of the reasons we went with the warning approach was to 
find out whether or not dropping __init__.py would cause serious problems - I 
think we have our answer to that question now :)

How does this sound for a way forward?:

2.5b2:
  - ignore ImportWarning by default (like PendingDeprecationWarning)
  - include in What's New instructions to enable it via the command line or 
Python code

2.6:
  - only show ImportWarning if the import ultimately fails
  - enable ImporWarning by default
  - consider allowing a .py extension on a directory name as an alternative to 
an __init__.py file.

Google could then change their sitecustomize.py to enable the warning by 
default when they roll out 2.5 :)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
             http://www.boredomandlaziness.org

From anthony at interlink.com.au  Sat Jul  1 10:02:21 2006
From: anthony at interlink.com.au (Anthony Baxter)
Date: Sat, 1 Jul 2006 18:02:21 +1000
Subject: [Python-Dev] 2.5 and beyond
In-Reply-To: <44A57954.7030905@v.loewis.de>
References: <ee2a432c0606300005g256b3391na6e684123d3e9e93@mail.gmail.com>
	<86FF939A-5D9F-42C0-B633-95837FD7C991@fuhm.net>
	<44A57954.7030905@v.loewis.de>
Message-ID: <200607011802.23996.anthony@interlink.com.au>

On Saturday 01 July 2006 05:19, Martin v. L?wis wrote:
> James Y Knight wrote:
> > I just submitted http://python.org/sf/1515169 for the
> > ImportWarning issue previously discussed here. IMO it's
> > important.
>
> At the moment (i.e. without an acceptable alternative
> implementation) it's primarily a policy issue. There really isn't
> any bug here; (to speak with Microsoft's words): This behavior is
> by design.
>
> Only the release manager or the BDFL could revert the feature, and
> Guido already stated that the warning stays until Python 3, and
> probably even after that. I personally believe the only chance to
> get this changed now is a well-designed alternative implementation
> (although this is no promise that such an alternative would
> actually be accepted).

given the number of people and ways that this can emit a spurious 
warning, I think it should be reverted for 2.5. At _best_ we could 
maybe have a new -W switch to make it be generated, but this should 
be off by default.

Anthony
-- 
Anthony Baxter     <anthony at interlink.com.au>
It's never too late to have a happy childhood.

From tim.peters at gmail.com  Sat Jul  1 10:08:39 2006
From: tim.peters at gmail.com (Tim Peters)
Date: Sat, 1 Jul 2006 04:08:39 -0400
Subject: [Python-Dev] 2.5 and beyond
In-Reply-To: <44A61301.2070805@canterbury.ac.nz>
References: <5.1.1.6.0.20060630190709.01ef5858@sparrow.telecommunity.com>
	<001201c69c9c$b6531fd0$6402a8c0@arkdesktop>
	<1f7befae0606301639u6f8c968cr29afce0ee7fbb43b@mail.gmail.com>
	<020c01c69ca1$754c7310$d1b12997@bagio>
	<44A61301.2070805@canterbury.ac.nz>
Message-ID: <1f7befae0607010108hb99c574l9e0d7e3cbf378355@mail.gmail.com>

[Giovanni Bajo]
>> >>> a = []
>> >>> for i in range(10):
>>
>> ...     a.append(lambda: i)
>> ...
>>
>> >>> print [x() for x in a]
>>
>> [9, 9, 9, 9, 9, 9, 9, 9, 9, 9]
>>
>> This subtle semantic of lambda is quite confusing, and still forces people to
>> use the "i=i" trick.

[Greg Ewing]
> This has *nothing* to do with the semantics of lambda!
> It's because Python's for-loop doesn't put its control
> variable in a new scope, the way Scheme's equivalent
> construct does.

I don't think I follow that.  Scheme has no loops in Python's sense --
things like "do" are shorthand for expressing stylized recursion,
where each conceptual iteration gets a fresh set of "loop variables".
When people talk about giving a Python for-loop vrbl its own scope,
they generally don't mean a new scope on _each iteration_, they just
mean that, e.g.,

    i = 5
    for i in range(10):
         # do stuff
    print i

prints 5 intead of 9, about the same as creating a nested block with
its own autos in C.  The Scheme way is more like:

    i = 5
    def step(i):
         # do stuff
         if i < 9:
             step(i+1)
    step(0)
    print i

except with tail-recursion elimination.  That also prints 5, but does
a hell of a lot more than _just_ arrange for that.

> *That's* what needs to be addressed to fix this problem.
> I've made a suggestion about that before, but Guido
> rejected it, so I won't repeat it here.

Don't recall what that was, but creating a new scope on each iteration
sounds hard to explain in Python.  If Giovanni wants the Scheme way
;-), it's available:

"""
a = []
def step(i):
    a.append(lambda: i)
    if i < 9:
        step(i+1)
step(0)
print [x() for x in a]
"""

prints [0, 1, 2, 3, 4, 5, 6, 7, 8, 9], although it's more sanely
written in Python with a loop:

"""
def make_lambda(i):
    return lambda: i
a = []
for i in range(10):
    a.append(make_lambda(i))
print [x() for x in a]
"""

Abusing the default-argument machinery to capture current bindings is
never necessary, and _is_ abuse.  Of course I do it too -- but rarely
:-)

From ncoghlan at gmail.com  Sat Jul  1 10:35:02 2006
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 01 Jul 2006 18:35:02 +1000
Subject: [Python-Dev] 2.5 and beyond
In-Reply-To: <020c01c69ca1$754c7310$d1b12997@bagio>
References: <5.1.1.6.0.20060630190709.01ef5858@sparrow.telecommunity.com><001201c69c9c$b6531fd0$6402a8c0@arkdesktop>	<1f7befae0606301639u6f8c968cr29afce0ee7fbb43b@mail.gmail.com>
	<020c01c69ca1$754c7310$d1b12997@bagio>
Message-ID: <44A633B6.3020403@gmail.com>

Giovanni Bajo wrote:
> Yes but:
> 
>>>> a = []
>>>> for i in range(10):
> ...     a.append(lambda: i)
> ...
>>>> print [x() for x in a]
> [9, 9, 9, 9, 9, 9, 9, 9, 9, 9]
> 
> This subtle semantic of lambda is quite confusing, and still forces people to
> use the "i=i" trick.

If you'd like each function instance to have a separate closure scope, then 
*give* each function a separate closure scope, instead of making them all 
share the same one the way you have above:

 >>> def make_f(i):
...     def f():
...         return i
...     return f
...
 >>> a = []
 >>> for i in range(10):
...     a.append(make_f(i))
...
 >>> print [x() for x in a]
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
             http://www.boredomandlaziness.org

From ncoghlan at gmail.com  Sat Jul  1 10:37:37 2006
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 01 Jul 2006 18:37:37 +1000
Subject: [Python-Dev] 2.5 and beyond
In-Reply-To: <200607011802.23996.anthony@interlink.com.au>
References: <ee2a432c0606300005g256b3391na6e684123d3e9e93@mail.gmail.com>	<86FF939A-5D9F-42C0-B633-95837FD7C991@fuhm.net>	<44A57954.7030905@v.loewis.de>
	<200607011802.23996.anthony@interlink.com.au>
Message-ID: <44A63451.6010909@gmail.com>

Anthony Baxter wrote:
> given the number of people and ways that this can emit a spurious 
> warning, I think it should be reverted for 2.5. At _best_ we could 
> maybe have a new -W switch to make it be generated, but this should 
> be off by default.

Last line of warnings.py

Copy, paste, s/PendingDeprecationWarning/ImportWarning

-Wd to enable it again :)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
             http://www.boredomandlaziness.org

From arigo at tunes.org  Sat Jul  1 10:49:00 2006
From: arigo at tunes.org (Armin Rigo)
Date: Sat, 1 Jul 2006 10:49:00 +0200
Subject: [Python-Dev] Cleanup of test harness for Python
In-Reply-To: <4dab5f760606300705l41c208c8tfb83f09f74badf2e@mail.gmail.com>
References: <4dab5f760606300705l41c208c8tfb83f09f74badf2e@mail.gmail.com>
Message-ID: <20060701084900.GB17748@code0.codespeak.net>

Hi all,

On Fri, Jun 30, 2006 at 10:05:14AM -0400, Frank Wierzbicki wrote:
> some checks for CPython internal tests that should be excluded from
> Jython

I know Frank already knows about this, but I take the occasion to remind
us that
http://codespeak.net/svn/pypy/dist/lib-python/modified-2.4.1/test
already shows which tests we had to modify for PyPy to make them less
implementation-detail-dependent, and which changes were made.

A possible first step here would be to find a consistent way to check,
in the test, which implementation we are running on top of, so that we
can (re-)write the tests accordingly.


A bientot,

Armin

From arigo at tunes.org  Sat Jul  1 10:52:21 2006
From: arigo at tunes.org (Armin Rigo)
Date: Sat, 1 Jul 2006 10:52:21 +0200
Subject: [Python-Dev] sys.settrace() in Python 2.3 vs. 2.4
In-Reply-To: <20060630131620.10CE.JCARLSON@uci.edu>
References: <20060630094140.10C8.JCARLSON@uci.edu>
	<44A582AE.7010902@v.loewis.de>
	<20060630131620.10CE.JCARLSON@uci.edu>
Message-ID: <20060701085221.GC17748@code0.codespeak.net>

Hi Josiah,

On Fri, Jun 30, 2006 at 01:27:24PM -0700, Josiah Carlson wrote:
> I'll just have to gracefully degrade functionality for older Pythons. 

More precisely, the bug shows up because in

  while 1:
      pass

the current line remains on the 'pass' forever.  It works for a loop
like that:

  while 1:
      sys
      sys

but it's admittedly quite obscure.


Armin

From jcarlson at uci.edu  Sat Jul  1 11:01:12 2006
From: jcarlson at uci.edu (Josiah Carlson)
Date: Sat, 01 Jul 2006 02:01:12 -0700
Subject: [Python-Dev] Lexical scoping in Python 3k
In-Reply-To: <44A612E8.8050205@canterbury.ac.nz>
References: <20060630163625.10D1.JCARLSON@uci.edu>
	<44A612E8.8050205@canterbury.ac.nz>
Message-ID: <20060701013710.10D7.JCARLSON@uci.edu>


Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
> 
> Josiah Carlson wrote:
> 
> > What I asked before, and what I'd like to ask again, is if there are any
> > _nontrivial uses_ of lexically nested scopes which are made cumbersome
> > by our inability to write to parent scopes.
> 
> The trouble with taking that position is that the very
> cases which would benefit are very *simple* ones, where
> it would be cumbersome to refactor it to use a class,
> or mutable object in the outer scope, etc. So you've
> effectively set up your acceptance criteria to be
> unmeetable.

If the only code that benefits from such changes are "very *simple*",
then I think that says something about its necessity.  That is, if
anything more complicated than those that are "very *simple*" generally
don't benefit, then I don't believe that such a modification would be
beneficial to the language overall. Further, a simple namespace factory
can handle much of the current issues, without needing to create or
change keywords.

def namespace(**kwds):
    class namespace(object):
        __slots__ = kwds.keys()
        def __init__(self):
            for i,j in kwds.iteritems():
                setattr(self, i,j)
    return namespace()

def trivial_counter(start):
    ns = namespace(current=start-1)
    def next():
        ns.current += 1
        return ns.current
    return next

Maybe a variant of the above namespace factory should make it into the
collections module.


> > If there aren't, then I'm
> > going to again have to argue against new syntax, keywords, and their use.
> 
> There's one very simple way we could do this in Py3k
> without requiring any new syntax or keywords: just
> redefine the meaning of "global" to mean "not local".

I would probably be a solid -0 on such a proposal; I still don't think
it's really necessary, but I've never used (or really seen) global more
than one level deep, so would guess its impact would be low.


 - Josiah


From g.brandl at gmx.net  Sat Jul  1 11:00:13 2006
From: g.brandl at gmx.net (Georg Brandl)
Date: Sat, 01 Jul 2006 11:00:13 +0200
Subject: [Python-Dev] Bug in stringobject?
In-Reply-To: <e6u6at$sbu$1@sea.gmane.org>
References: <e6u6at$sbu$1@sea.gmane.org>
Message-ID: <e85diu$rqc$1@sea.gmane.org>

Georg Brandl wrote:
> In string_replace, there is
> 
> 	if (PyString_Check(from)) {
> 	  /* Can this be made a '!check' after the Unicode check? */
> 	}
> #ifdef Py_USING_UNICODE
> 	if (PyUnicode_Check(from))
> 		return PyUnicode_Replace((PyObject *)self,
> 					 from, to, count);
> #endif
> 	else if (PyObject_AsCharBuffer(from, &tmp_s, &tmp_len))
> 		return NULL;
> 
> [the same check with "to"]
> 
> 	return (PyObject *)replace((PyStringObject *) self,
> 				   (PyStringObject *) from,
> 				   (PyStringObject *) to, count);
> 
> 
> Can this be correct if from or to isn't a string object, but a
> char buffer compatible object?

May I note that this is still unresolved? I can submit a bug report
and add it to PEP 356, too...

Georg


From fredrik at pythonware.com  Sat Jul  1 11:29:56 2006
From: fredrik at pythonware.com (Fredrik Lundh)
Date: Sat, 01 Jul 2006 11:29:56 +0200
Subject: [Python-Dev] Bug in stringobject?
In-Reply-To: <e85diu$rqc$1@sea.gmane.org>
References: <e6u6at$sbu$1@sea.gmane.org> <e85diu$rqc$1@sea.gmane.org>
Message-ID: <e85faf$mo$2@sea.gmane.org>

Georg Brandl wrote:

>> Can this be correct if from or to isn't a string object, but a
>> char buffer compatible object?
> 
> May I note that this is still unresolved? I can submit a bug report
> and add it to PEP 356, too...

it's already on my todo list, but that list is full of stuff, so having 
it on the official todo list is probably a good idea.  if you do add it, 
assign it to me.

</F>


From g.brandl at gmx.net  Sat Jul  1 11:41:53 2006
From: g.brandl at gmx.net (Georg Brandl)
Date: Sat, 01 Jul 2006 11:41:53 +0200
Subject: [Python-Dev] Bug in stringobject?
In-Reply-To: <e85faf$mo$2@sea.gmane.org>
References: <e6u6at$sbu$1@sea.gmane.org> <e85diu$rqc$1@sea.gmane.org>
	<e85faf$mo$2@sea.gmane.org>
Message-ID: <e85g11$2go$1@sea.gmane.org>

Fredrik Lundh wrote:
> Georg Brandl wrote:
> 
>>> Can this be correct if from or to isn't a string object, but a
>>> char buffer compatible object?
>> 
>> May I note that this is still unresolved? I can submit a bug report
>> and add it to PEP 356, too...
> 
> it's already on my todo list, but that list is full of stuff, so having 
> it on the official todo list is probably a good idea.  if you do add it, 
> assign it to me.

Done. #1515471.

Georg


From andrewdalke at gmail.com  Sat Jul  1 11:59:00 2006
From: andrewdalke at gmail.com (Andrew Dalke)
Date: Sat, 1 Jul 2006 11:59:00 +0200
Subject: [Python-Dev] PEP 328 and PEP 338, redux
In-Reply-To: <018e01c69b77$a9172e40$d503030a@trilan>
References: <44A11EA1.1000605@iinet.net.au>
	<5.1.1.6.0.20060627120926.02021fe0@sparrow.telecommunity.com>
	<033001c69a07$44140890$d503030a@trilan>
	<44A1CE4A.2000900@canterbury.ac.nz>
	<ca471dc20606271846l2a0de4fevf667c60dec004039@mail.gmail.com>
	<039201c69ad7$4dc91df0$d503030a@trilan> <44A3B648.2060104@gmail.com>
	<018e01c69b77$a9172e40$d503030a@trilan>
Message-ID: <d78db4cd0607010259h6677c54es475e8a6aa7cd36d5@mail.gmail.com>

Giovanni Bajo <rasky at develer.com> wrote:
> Real-world usage case for import __main__? Otherwise, I say screw it :)

I have used it as a workaround for timeit.py's requirement that I pass
it strings instead of functions.

>>> def compute():
...     1+1
...
>>> import timeit
>>> t = timeit.Timer("__main__.compute()", "import __main__")
>>> t.timeit()
1.9755008220672607
>>>

You can argue (as many have) that timeit.py needs a better API for
this.  That's a different world than the existing real one.

                                Andrew
                                dalke at dalkescientific.com

From tomerfiliba at gmail.com  Sat Jul  1 15:49:46 2006
From: tomerfiliba at gmail.com (tomer filiba)
Date: Sat, 1 Jul 2006 15:49:46 +0200
Subject: [Python-Dev] weakattr
Message-ID: <1d85506f0607010649y65c9fb24mbc5c636360970e52@mail.gmail.com>

weakattr (weak attributes) are attributes that are weakly referenced
by their containing object. they are very useful for cyclic references --
an object that holds a reference to itself.

when a cyclic reference is found by the GC, the memory may be
freed, but __del__ is not called, because it's impossible to tell which
__del__ to call first. this is an awkward asymmetry with no clean
solution: most such objects provide a "close" or "dispose" method
that must be called explicitly.

weakattrs to solve this problem, by providing a "magical" attribute
that "disappears" when the attribute is no longer strongly-referenced.

you can find the code, as well as some examples, on this link
http://sebulba.wikispaces.com/recipe+weakattr

since the stdlib already comes with weakref.py, which provides
higher level concepts over the builtin _weakref module, i'd like to
make weakattr a part of it.

it's only ~20 lines of code, and imho saves the trouble of explicitly
releasing the resource of un__del__able objects.

i think it's useful. here's a snippet:

>>> from weakref import weakattr
>>>
>>> class blah(object):
...     yada = weakref()
...
>>> o1 = blah()
>>> o2 = blah()
>>> o1.yada = o2
>>> o2.yada = o1

o1.yada is a *weakref* to o2, so that when o2 is no longer
strongly-referenced...
>>> del o2
o1.yada "magically" disappears as well.
>>> o1.yada
... AttributeError(...)

since the programmer explicitly defined "yada" as a weakatt, he/she
knows it might "disappear". it might look awkward at first, but that's
exactly the *desired* behavior (otherwise we'd just use the regular
strong attributes).

another thing to note is that weakattrs are likely to be gone only
when the object's __del__ is already invoked, so the only code that
needs to take such precautions is __del__ (which already has some
constraints)

for example:

>>> class blah(object):
...     me = weakattr()
...
...     def __init__(self):
...         self.me  = self
...
...     def something(self):
...         # we can rest assure me exists at this stage
...         print self.me
...
...     def __del__(self):
...         # by the time __del__ is called, "me" is removed
...         print "me exists?", hasattr(self, "me")
...
>>> b = blah()
>>> b.me
<__main__.blah object at 0x00C0EC10>
>>> b.something()
<__main__.blah object at 0x00C0EC10>
>>> del b
>>> import gc
>>> gc.collect()
me exists? False
0
>>>



-tomer
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20060701/b174e77e/attachment.htm 

From nas at arctrix.com  Sat Jul  1 17:41:29 2006
From: nas at arctrix.com (Neil Schemenauer)
Date: Sat, 1 Jul 2006 15:41:29 +0000 (UTC)
Subject: [Python-Dev] Lexical scoping in Python 3k
References: <001201c69c65$b3869750$6402a8c0@arkdesktop>
	<Pine.LNX.4.58.0606301658240.17937@server1.LFW.org>
	<ca471dc20606301524n52ba3c2dj870b650fc600282@mail.gmail.com>
	<Pine.LNX.4.58.0606301726170.17937@server1.LFW.org>
Message-ID: <e86539$o2e$1@sea.gmane.org>

Ka-Ping Yee <python-dev at zesty.ca> wrote:
> Most other languages that support lexical scoping (including Scheme,
> JavaScript, Ruby, Perl, E, Java, Smalltalk) provide a uniform way
> to read and write to scopes at all levels.  This is done by letting
> programmers specify the scope in which they want a variable bound
> (usually with a keyword like "var" in JavaScript, "my" in Perl, or
> "define" in E).

That's not the Python way, IMO.  I think the right way (assuming we
actually want to allow it) is to introduce a pure assignment
statement in addition to the assignment/declaration statement that
we already have.  For example:


    a = 1 
    def f():
        b = 2
        a := 2
        def g():
            b := 3
            print a, b, c
        g()
    f()
    
would print "2 3 4".  The := would assign but not declare a variable
in the current scope.

  Neil


From ronaldoussoren at mac.com  Sat Jul  1 18:24:15 2006
From: ronaldoussoren at mac.com (Ronald Oussoren)
Date: Sat, 1 Jul 2006 18:24:15 +0200
Subject: [Python-Dev] zlib module build failure on Mac OSX 10.4.7
In-Reply-To: <17573.60593.759293.549583@montanaro.dyndns.org>
References: <17573.60593.759293.549583@montanaro.dyndns.org>
Message-ID: <130298B1-7336-480A-A90F-97B68327BE4A@mac.com>


On Jul 1, 2006, at 5:32 AM, skip at pobox.com wrote:

> Just upgraded my Mac to OSX 10.4.7 yesterday.  svn up'd Python  
> trunk, then
> "make clean ; configure ; make" and I see that building the zlib  
> module
> fails:
>
>     gcc -fno-strict-aliasing -Wno-long-double -no-cpp-precomp -mno- 
> fused-madd -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -I. -I/Users/ 
> skip/src/python-svn/trunk/./Include -I/Users/skip/src/python-svn/ 
> trunk/./Mac/Include -I/Users/skip/local/include -I../Include -I. -I/ 
> usr/local/include -I/Users/skip/src/python-svn/trunk/Include -I/ 
> Users/skip/src/python-svn/trunk/build -c /Users/skip/src/python-svn/ 
> trunk/Modules/zlibmodule.c -o build/temp.macosx-10.3-ppc-2.5/Users/ 
> skip/src/python-svn/trunk/Modules/zlibmodule.o
>     /Users/skip/src/python-svn/trunk/Modules/zlibmodule.c: In  
> function 'PyZlib_uncopy':
>     /Users/skip/src/python-svn/trunk/Modules/zlibmodule.c:724:  
> warning: implicit declaration of function 'inflateCopy'
>     gcc -bundle -undefined dynamic_lookup build/temp.macosx-10.3- 
> ppc-2.5/Users/skip/src/python-svn/trunk/Modules/zlibmodule.o -L/ 
> Users/skip/local/lib -L/usr/local/lib -lz -o build/lib.macosx-10.3- 
> ppc-2.5/zlib.so -Wl,-search_paths_first
>     *** WARNING: renaming "zlib" since importing it failed: dlopen 
> (build/lib.macosx-10.3-ppc-2.5/zlib.so, 2): Symbol not found:  
> _inflateCopy
>       Referenced from: build/lib.macosx-10.3-ppc-2.5/zlib.so
>       Expected in: dynamic lookup
>
> Anybody else seen this?  I checked the buildbot trunk osx 10.4.  It  
> seemed
> to have no trouble.  And what's with the "10.3" bit in the  
> directory names?

Are you sure you're building on a 10.4 box? Both the macosx-10.3  
thingy and lack of inflateCopy seem to indicate that you're running  
on 10.3.

Ronald

>
> Skip
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/ 
> ronaldoussoren%40mac.com

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 2157 bytes
Desc: not available
Url : http://mail.python.org/pipermail/python-dev/attachments/20060701/f51f75e9/attachment.bin 

From almann.goo at gmail.com  Sat Jul  1 18:42:27 2006
From: almann.goo at gmail.com (Almann T. Goo)
Date: Sat, 1 Jul 2006 12:42:27 -0400
Subject: [Python-Dev] Lexical scoping in Python 3k
In-Reply-To: <20060701013710.10D7.JCARLSON@uci.edu>
References: <20060630163625.10D1.JCARLSON@uci.edu>
	<44A612E8.8050205@canterbury.ac.nz>
	<20060701013710.10D7.JCARLSON@uci.edu>
Message-ID: <7e9b97090607010942r599bfed9u4bc02a564becaf2b@mail.gmail.com>

On 7/1/06, Josiah Carlson <jcarlson at uci.edu> wrote:

> > There's one very simple way we could do this in Py3k
> > without requiring any new syntax or keywords: just
> > redefine the meaning of "global" to mean "not local".
>
> I would probably be a solid -0 on such a proposal; I still don't think
> it's really necessary, but I've never used (or really seen) global more
> than one level deep, so would guess its impact would be low.
>

This has been discussed at length in the following thread that I started in
February and at least one time before that.

http://mail.python.org/pipermail/python-dev/2006-February/061568.html

I think using the "global" keyword is probably the lowest impact form and
has the least amount of backwards incompatibility.  Below is the part of the
last thread that I talked about changing the meaning of "global."

http://mail.python.org/pipermail/python-dev/2006-February/061852.html

Having the "global" keyword semantics changed to be "lexically global"
would break in the cases that "global" is used on a name within a
nested scope that has an enclosing scope with the same name.  I would
suppose that actual instances in real code of this would be rare.

Consider:
>>>* x = 1
*>>>* def f() :
*...   x = 2
...   def inner() :
...     global x
...     print x
...   inner()
...
>>>* f()
*1

Under the proposed rules:
>>>* f()
*2

PEP 227 also had backwards incompatibilities that were similar and I
suggest handling them the same way by issuing a warning in these cases
when the new semantics are not being used (i.e. no "from __future__").


Most people probably think that this is a low impact "wart" on the Python
language that not really worth fixing as there are workarounds (i.e. mutable
objects) or other ways to express (i.e. use classes) such things, but it
does trip people up from time to time as warts typically do--I guess that's
why this gets brought up now and again.

Best regards,
Almann

-- 
Almann T. Goo
almann.goo at gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20060701/7f3f113c/attachment.html 

From skip at pobox.com  Sat Jul  1 18:57:34 2006
From: skip at pobox.com (skip at pobox.com)
Date: Sat, 1 Jul 2006 11:57:34 -0500
Subject: [Python-Dev] zlib module build failure on Mac OSX 10.4.7
In-Reply-To: <130298B1-7336-480A-A90F-97B68327BE4A@mac.com>
References: <17573.60593.759293.549583@montanaro.dyndns.org>
	<130298B1-7336-480A-A90F-97B68327BE4A@mac.com>
Message-ID: <17574.43390.560231.425494@montanaro.dyndns.org>


    Ronald> Are you sure you're building on a 10.4 box?  Both the
    Ronald> macosx-10.3 thingy and lack of inflateCopy seem to indicate that
    Ronald> you're running on 10.3.

Well, yeah, pretty sure.  Let's see.  The box with the disk says "Mac OS X
Tiger - Version 10.4" on the spine. The "About This Mac" popup says
"10.4.7".  It used to run 10.3 though.  Is there some possibility the update
from 10.3 to 10.4 had problems?

Note that the compile log on the buildbot 10.4 box also has "10.3" in its
directory names.  If I remember correctly, it came from Apple with 10.4
installed.

Skip

From sergey at optimaltec.com  Sat Jul  1 18:52:19 2006
From: sergey at optimaltec.com (Sergey A. Lipnevich)
Date: Sat, 01 Jul 2006 12:52:19 -0400
Subject: [Python-Dev] ImportWarning flood
In-Reply-To: <44A282E3.1000009@v.loewis.de>
References: <20060628122558.19551.qmail@web31505.mail.mud.yahoo.com>
	<44A282E3.1000009@v.loewis.de>
Message-ID: <e869cj$5rd$1@sea.gmane.org>

All,

I tried to implement Jean-Paul Calderone's idea for the following patch,
plagiarizing Ralf W. Grosse-Kunstleve's error text. It delays import
warning until end of search for modules, but remembers how many
potential modules (candidates without __init__.py) it didn't import. I
didn't really try to analyze any conditions, instead I simply assumed
that wherever ImportWarning would be issued, we have a suitable
candidate, and saved it on the stack. If nothing is found, Python emits
ImportWarning right before ImportError, and explains what happened.
Please let me know if this would work and if anything needs to be done
for this patch to be accepted.
Thank you!

Sergey.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: import_warning.diff
Type: text/x-patch
Size: 1867 bytes
Desc: not available
Url : http://mail.python.org/pipermail/python-dev/attachments/20060701/94ab9c43/attachment-0001.bin 

From aahz at pythoncraft.com  Sat Jul  1 19:31:00 2006
From: aahz at pythoncraft.com (Aahz)
Date: Sat, 1 Jul 2006 10:31:00 -0700
Subject: [Python-Dev] ImportWarning flood
In-Reply-To: <e869cj$5rd$1@sea.gmane.org>
References: <20060628122558.19551.qmail@web31505.mail.mud.yahoo.com>
	<44A282E3.1000009@v.loewis.de> <e869cj$5rd$1@sea.gmane.org>
Message-ID: <20060701173100.GA22185@panix.com>

On Sat, Jul 01, 2006, Sergey A. Lipnevich wrote:
>
> Please let me know if this would work and if anything needs to be done
> for this patch to be accepted.

The first thing you need to do for ANY patch to be considered is to post
it so SourceForge (or at least post to python-dev explaining that you'll
post to SF as soon as it comes back).
-- 
Aahz (aahz at pythoncraft.com)           <*>         http://www.pythoncraft.com/

"I saw `cout' being shifted "Hello world" times to the left and stopped
right there."  --Steve Gonedes

From martin at v.loewis.de  Sat Jul  1 19:32:29 2006
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sat, 01 Jul 2006 19:32:29 +0200
Subject: [Python-Dev] ImportWarning flood
In-Reply-To: <e869cj$5rd$1@sea.gmane.org>
References: <20060628122558.19551.qmail@web31505.mail.mud.yahoo.com>	<44A282E3.1000009@v.loewis.de>
	<e869cj$5rd$1@sea.gmane.org>
Message-ID: <44A6B1AD.6020204@v.loewis.de>

Sergey A. Lipnevich wrote:
> I tried to implement Jean-Paul Calderone's idea for the following patch,
> plagiarizing Ralf W. Grosse-Kunstleve's error text. It delays import
> warning until end of search for modules, but remembers how many
> potential modules (candidates without __init__.py) it didn't import. I
> didn't really try to analyze any conditions, instead I simply assumed
> that wherever ImportWarning would be issued, we have a suitable
> candidate, and saved it on the stack. If nothing is found, Python emits
> ImportWarning right before ImportError, and explains what happened.
> Please let me know if this would work and if anything needs to be done
> for this patch to be accepted.

Please notice that there is also python.org/sf/1515361

I had no time to compare this with your patch, yet.

Regards,
Martin

From ronaldoussoren at mac.com  Sat Jul  1 19:45:17 2006
From: ronaldoussoren at mac.com (Ronald Oussoren)
Date: Sat, 1 Jul 2006 19:45:17 +0200
Subject: [Python-Dev] zlib module build failure on Mac OSX 10.4.7
In-Reply-To: <17574.43390.560231.425494@montanaro.dyndns.org>
References: <17573.60593.759293.549583@montanaro.dyndns.org>
	<130298B1-7336-480A-A90F-97B68327BE4A@mac.com>
	<17574.43390.560231.425494@montanaro.dyndns.org>
Message-ID: <315DECAC-D582-4692-A0CA-13D3C7BF992C@mac.com>


On Jul 1, 2006, at 6:57 PM, skip at pobox.com wrote:

>
>     Ronald> Are you sure you're building on a 10.4 box?  Both the
>     Ronald> macosx-10.3 thingy and lack of inflateCopy seem to  
> indicate that
>     Ronald> you're running on 10.3.
>
> Well, yeah, pretty sure.  Let's see.  The box with the disk says  
> "Mac OS X
> Tiger - Version 10.4" on the spine. The "About This Mac" popup says
> "10.4.7".

That gets the easy solution out of the way ;-)

>   It used to run 10.3 though.  Is there some possibility the update
> from 10.3 to 10.4 had problems?
>
> Note that the compile log on the buildbot 10.4 box also has "10.3"  
> in its
> directory names.  If I remember correctly, it came from Apple with  
> 10.4
> installed.

/me slaps head.

Having 10.3 in the directory names is intentional, the version in the  
directory name is the value of MACOSX_DEPLOYMENT_TARGET, with is  
defaulted to 10.3 in the configure script.

What  I don't understand yet is why your copy of libz doesn't have  
inflateCopy. What does /usr/lib/libz.dylib point to on your system?  
On my 10.4 box it is a symlink that points to libz.1.2.3.dylib and  
there is an older version of libz (libz.1.1.3.dylib) in /usr/lib as  
well.

Ronald
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 2157 bytes
Desc: not available
Url : http://mail.python.org/pipermail/python-dev/attachments/20060701/d2da4e4b/attachment.bin 

From sergey at optimaltec.com  Sat Jul  1 20:10:42 2006
From: sergey at optimaltec.com (Sergey A. Lipnevich)
Date: Sat, 01 Jul 2006 14:10:42 -0400
Subject: [Python-Dev] ImportWarning flood
In-Reply-To: <44A6B1AD.6020204@v.loewis.de>
References: <20060628122558.19551.qmail@web31505.mail.mud.yahoo.com>	<44A282E3.1000009@v.loewis.de>	<e869cj$5rd$1@sea.gmane.org>
	<44A6B1AD.6020204@v.loewis.de>
Message-ID: <e86dvk$h21$1@sea.gmane.org>

Martin v. L?wis wrote:
> Sergey A. Lipnevich wrote:
>> I tried to implement Jean-Paul Calderone's idea for the following patch,
>> plagiarizing Ralf W. Grosse-Kunstleve's error text. It delays import
...
> Please notice that there is also python.org/sf/1515361
> 
> I had no time to compare this with your patch, yet.

Thanks! I made python.org/sf/1515609. The difference (also documented in
the description) is that my patch saves memory and some CPU cycles by
not trying to collect all directories Python did not import because of
missing __init__.py. It only reports how many such directories there are
and what is the first one.

Sergey.


From s.percivall at chello.se  Sat Jul  1 20:27:47 2006
From: s.percivall at chello.se (Simon Percivall)
Date: Sat, 1 Jul 2006 20:27:47 +0200
Subject: [Python-Dev] Lexical scoping in Python 3k
In-Reply-To: <7e9b97090607010942r599bfed9u4bc02a564becaf2b@mail.gmail.com>
References: <20060630163625.10D1.JCARLSON@uci.edu>
	<44A612E8.8050205@canterbury.ac.nz>
	<20060701013710.10D7.JCARLSON@uci.edu>
	<7e9b97090607010942r599bfed9u4bc02a564becaf2b@mail.gmail.com>
Message-ID: <8B4078DF-F730-4E46-9F2C-04807B178471@chello.se>

What about doing something similar to how import was changed?

.a = 5 # this scope (self might be too magical
..a = 3 # up one scope
...a # up three

Of course, this looks ... perhaps a bit strange. Also, counting is a  
bother.

//Simon

From martin at v.loewis.de  Sat Jul  1 20:46:44 2006
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sat, 01 Jul 2006 20:46:44 +0200
Subject: [Python-Dev] zlib module build failure on Mac OSX 10.4.7
In-Reply-To: <315DECAC-D582-4692-A0CA-13D3C7BF992C@mac.com>
References: <17573.60593.759293.549583@montanaro.dyndns.org>	<130298B1-7336-480A-A90F-97B68327BE4A@mac.com>	<17574.43390.560231.425494@montanaro.dyndns.org>
	<315DECAC-D582-4692-A0CA-13D3C7BF992C@mac.com>
Message-ID: <44A6C314.2000709@v.loewis.de>

Ronald Oussoren wrote:
> What  I don't understand yet is why your copy of libz doesn't have
> inflateCopy. 

What I don't understand is that configure does not detect that.

Regards,
Martin

From ark at acm.org  Sat Jul  1 21:03:19 2006
From: ark at acm.org (Andrew Koenig)
Date: Sat, 1 Jul 2006 15:03:19 -0400
Subject: [Python-Dev] 2.5 and beyond
In-Reply-To: <44A61301.2070805@canterbury.ac.nz>
Message-ID: <000601c69d41$073dbf00$6402a8c0@arkdesktop>


> a = []
> for i in range(10):
>     a.append(lambda: i)
> print [x() for x in a]
>
> [9, 9, 9, 9, 9, 9, 9, 9, 9, 9]

Aha! -- Thank you for jogging my memory.

You seem to be right -- the problem is not that Python is lexically scoped,
but that when you define a variable with =, it leaks out into the
surrounding function scope.

Here's an example:

	If True:
		y = 123
	print y

It may be obvious that this should print 123, but that's only because =
combines properties of assignment and definition.  In particular, if we were
to write

	y = 42
	if True:
		y = 123
	print y

it would be very surprising if this example were to print anything but 123.

Here is a corresponding fragment in C++:

	int y = 42;
	if (true) {
		y = 123;
	}
	std::cout << y << "\n";

The "int" in the first line means that the variable y is being defined.  Its
lack in the third line means that y refers to a variable defined in an outer
scope.  So both instances of y here refer to the same variable, as they do
in Python.

But because definition and assignment are separated in C++, we can also
write

	int y = 42;
	if (true) {
		int y = 123;
	}
	std::cout << y << "\n";

and the fragment will print 42.  In this example, there are two distinct
variables, both named y.

So the problem, as I see it, is indeed that in Python there are suites that
look to me as if they should define scopes, but don't.  Indeed, if I write

	if (foo):
		y = 123

I can't even determine by inspecting the program whether y is defined at
all.  I might argue that y is always defined, by virtue of appearing before
= somewhere in this scope, but the compiler tells me "name 'y' is not
defined" if I try it, so I guess that's the right way to treat it.

So here's how I understand what Greg was saying.

Suppose I write

	x = []
	for i in range(10):
		x.append(lambda:i)
	print [f() for f in x]

This example will print [9, 9, 9, 9, 9, 9, 9, 9, 9, 9], which I think is
wildly unintuitive.

My intuition in this matter is partly formed by C++, but it is also formed
by other languages going as far back as Algol 68.  That intuition says that
because the suite controlled by a "for" statement is executed any number of
times, potentially including zero, it should be considered as its own scope,
and any variables defined in that scope should stay there.

In particular, the variable "i" should be defined in the scope of the "for",
which implies that each time through the loop, the name "i" should be
(re)bound to a different object.

What surprises me even more is that if I try to define such a variable
explicitly, it still doesn't work:

	x = []
	for i in range(10):
		j = i
		x.append(lambda:j)
	print [f() for f in x]

This example still prints [9, 9, 9, 9, 9, 9, 9, 9, 9, 9].  If I understand
the reason correctly, it is because even though j is defined only in the
body of the loop, loop bodies are not scopes, so the variable's definition
is hoisted out into the surrounding function scope.

To convince myself of this behavior, I defined an extra function scope, the
purpose of which is to localize j:

	x = []
	for i in range(10):
		def foo():
			j = i
			return lambda:j
		x.append(foo())
	print [f() for f in x]

Indeed, this example prints [0, 1, 2, 3, 4, 5, 6, 7, 8, 9].  The example
also points up the fact that

	x.append(lambda:i)

and

	def foo():
		j = i
		return lambda:j
	x.append(foo())

behave differently, where my intuition (and, I suspect, many other people's
as well) would be that they would be equivalent.

Finally, I observe that this second example above is also equivalent to

	x.append(lambda i=i: i)

which is what explains the fairly common idiom

	x = []
	for i in range(10):
		x.append(lambda i=i:i)
	print [f() for f in x]

So maybe what I meant when I asked for lexical scopes was two things:

	1) Every indentation level should be a scope;
	2) In general, variable definitions should not leak into
	   surrounding scopes.

I realize that (2) is too simplistic.  Someone who writes

	if x < 0:
		y = -x
	else:
		y = x

will expect y to be defined in the scope surrounding the "if" even if it was
not already defined there.  On the other hand, I think that the subtle
pitfalls that come from allowing "for" variables to leak into the
surrounding scopes are much harder to deal with and understand than would be
the consequences of restricting their scopes as outlined above.




From ark at acm.org  Sat Jul  1 21:06:38 2006
From: ark at acm.org (Andrew Koenig)
Date: Sat, 1 Jul 2006 15:06:38 -0400
Subject: [Python-Dev] Lexical scoping in Python 3k
In-Reply-To: <8B4078DF-F730-4E46-9F2C-04807B178471@chello.se>
Message-ID: <000701c69d41$7dc7a9b0$6402a8c0@arkdesktop>

> What about doing something similar to how import was changed?
> 
> .a = 5 # this scope (self might be too magical
> ..a = 3 # up one scope
> ...a # up three
> 
> Of course, this looks ... perhaps a bit strange. Also, counting is a
> bother.

I'd rather see a simpler rule: = never defines a variable in a surrounding
scope.  If you want to affect the binding of such a variable, you have to
define it explicitly in the scope in which you want it.

Example:

	x = 42
	def f():
		x = 123		# rebinds x as defined above
		y = 123		# defines local variable
	f()
	print x			# prints 123
	print y			# error -- y not defined

Yes, I know that rule is too simplistic.  But I think I'd still prefer it to
the way things are now.




From ronaldoussoren at mac.com  Sat Jul  1 21:12:24 2006
From: ronaldoussoren at mac.com (Ronald Oussoren)
Date: Sat, 1 Jul 2006 21:12:24 +0200
Subject: [Python-Dev] zlib module build failure on Mac OSX 10.4.7
In-Reply-To: <44A6C314.2000709@v.loewis.de>
References: <17573.60593.759293.549583@montanaro.dyndns.org>
	<130298B1-7336-480A-A90F-97B68327BE4A@mac.com>
	<17574.43390.560231.425494@montanaro.dyndns.org>
	<315DECAC-D582-4692-A0CA-13D3C7BF992C@mac.com>
	<44A6C314.2000709@v.loewis.de>
Message-ID: <0006F35E-FD5F-4BAD-A608-642039BBE7E8@mac.com>


On Jul 1, 2006, at 8:46 PM, Martin v. L?wis wrote:

> Ronald Oussoren wrote:
>> What  I don't understand yet is why your copy of libz doesn't have
>> inflateCopy.
>
> What I don't understand is that configure does not detect that.

You may be onto something there. Skip, do you have another copy of  
libz somewhere? Given the link line in your first message either in / 
usr/local/lib or /Users/skip/local/lib. And if you have, is that a  
static library (libz.a) instead of a dylib?

As background to my question: the linker on OSX behaves slightly  
different than the one on most other unix-y systems. It first  
searches the entire linker path for shared libraries (dylibs) before  
looking for static libraries. I added a flag to the link flags for  
the zlib extension a while back that changes the search order into a  
more traditional one: look in every directory on the linker path for  
either a dylib or static library.  The new flag is  -Wl,- 
search_paths_first.

If skip does indeed have libz somewhere else we'll either have to  
make a matching update to configure, or roll back my change. If the  
latter I'll have to tweak the build script for the binary installer  
for OSX because I want to link that using a static copy of libz for  
binary compatibility with OSX 10.3.9.

Ronald
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 2157 bytes
Desc: not available
Url : http://mail.python.org/pipermail/python-dev/attachments/20060701/e0a2a627/attachment-0001.bin 

From bob at redivi.com  Sat Jul  1 21:19:58 2006
From: bob at redivi.com (Bob Ippolito)
Date: Sat, 1 Jul 2006 12:19:58 -0700
Subject: [Python-Dev] zlib module build failure on Mac OSX 10.4.7
In-Reply-To: <315DECAC-D582-4692-A0CA-13D3C7BF992C@mac.com>
References: <17573.60593.759293.549583@montanaro.dyndns.org>
	<130298B1-7336-480A-A90F-97B68327BE4A@mac.com>
	<17574.43390.560231.425494@montanaro.dyndns.org>
	<315DECAC-D582-4692-A0CA-13D3C7BF992C@mac.com>
Message-ID: <A4A2C829-3032-4BF7-BD03-974ADB6C3BAC@redivi.com>


On Jul 1, 2006, at 10:45 AM, Ronald Oussoren wrote:

>
> On Jul 1, 2006, at 6:57 PM, skip at pobox.com wrote:
>
>>
>>     Ronald> Are you sure you're building on a 10.4 box?  Both the
>>     Ronald> macosx-10.3 thingy and lack of inflateCopy seem to  
>> indicate that
>>     Ronald> you're running on 10.3.
>>
>> Well, yeah, pretty sure.  Let's see.  The box with the disk says  
>> "Mac OS X
>> Tiger - Version 10.4" on the spine. The "About This Mac" popup says
>> "10.4.7".
>
> That gets the easy solution out of the way ;-)
>
>>   It used to run 10.3 though.  Is there some possibility the update
>> from 10.3 to 10.4 had problems?
>>
>> Note that the compile log on the buildbot 10.4 box also has "10.3"  
>> in its
>> directory names.  If I remember correctly, it came from Apple with  
>> 10.4
>> installed.
>
> /me slaps head.
>
> Having 10.3 in the directory names is intentional, the version in  
> the directory name is the value of MACOSX_DEPLOYMENT_TARGET, with  
> is defaulted to 10.3 in the configure script.
>
> What  I don't understand yet is why your copy of libz doesn't have  
> inflateCopy. What does /usr/lib/libz.dylib point to on your system?  
> On my 10.4 box it is a symlink that points to libz.1.2.3.dylib and  
> there is an older version of libz (libz.1.1.3.dylib) in /usr/lib as  
> well.

Maybe Skip didn't upgrade to the latest version of Xcode? Perhaps  
he's still got an old SDK?

-bob


From rasky at develer.com  Sat Jul  1 21:24:04 2006
From: rasky at develer.com (Giovanni Bajo)
Date: Sat, 1 Jul 2006 21:24:04 +0200
Subject: [Python-Dev] 2.5 and beyond
References: <000601c69d41$073dbf00$6402a8c0@arkdesktop>
Message-ID: <024901c69d43$ea5a6480$d503030a@trilan>

Andrew Koenig wrote:

> Suppose I write
>
> x = []
> for i in range(10):
> x.append(lambda:i)
> print [f() for f in x]
>
> This example will print [9, 9, 9, 9, 9, 9, 9, 9, 9, 9], which I think
> is wildly unintuitive.

That is my point: to me, it's counter-intuitive just like the infamous
"except NameError, TypeError". I believe that names in
lambdas/nested-functions referring to local names in the outer scope should
really be bound at function definition time (much like default arguments
are).

> What surprises me even more is that if I try to define such a variable
> explicitly, it still doesn't work:
>
> x = []
> for i in range(10):
> j = i
> x.append(lambda:j)
> print [f() for f in x]
>
> This example still prints [9, 9, 9, 9, 9, 9, 9, 9, 9, 9].  If I
> understand the reason correctly, it is because even though j is
> defined only in the body of the loop, loop bodies are not scopes, so
> the variable's definition is hoisted out into the surrounding
> function scope.

Yes. And by itself, I like this fact because it's very handy in many cases.
And it's also handy that the iteration variable of the for loop is
accessible after the for loop is terminated (in fact, this specific
behaviour is already listed among the wont-change for Py3k).

> On the other hand, I think that
> the subtle pitfalls that come from allowing "for" variables to leak
> into the surrounding scopes are much harder to deal with and
> understand than would be the consequences of restricting their scopes
> as outlined above.

As I said, to me there's nothing wrong with the way Python variables leak
out of the suites; or, in other words, with the fact that Python has only
two namespaces, the function-local and the global namespace. What I don't
like is that the lookup of lambda's names are fully deferred at execution
time. This behaviour is already not fully followed for local variables in
functions, since:

>>> y = 0
>>> def foo():
...     print y
...     y = 2
...
>>> foo()
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
  File "<stdin>", line 2, in foo
UnboundLocalError: local variable 'y' referenced before assignment

which means that Python users *already* know that a variable is not really
looked up only at run-time, but there's "something" going on even at
function definition time. I don't see anything wrong if lambdas (or nested
scopes) did the same for names provably coming from the outer scope.
-- 
Giovanni Bajo


From noamraph at gmail.com  Sat Jul  1 22:06:28 2006
From: noamraph at gmail.com (Noam Raphael)
Date: Sat, 1 Jul 2006 23:06:28 +0300
Subject: [Python-Dev] Empty Subscript PEP on Wiki - keep or toss?
In-Reply-To: <e840po$k1i$1@sea.gmane.org>
References: <17573.33939.473091.920283@montanaro.dyndns.org>
	<e840po$k1i$1@sea.gmane.org>
Message-ID: <b348a0850607011306i6ca3d001w132c81379b6a1889@mail.gmail.com>

Hello,

I posted it as a "pre-PEP", in the hope that it may become a PEP and
be accepted. As it happened, Guido said "no" at the end, so I stopped
pushing the subject.

I think that the main reason for the "no" was that my use case wasn't
convincing enough - the objections were that this wasn't useful
enough, not that it does anything harmful*. As the one who does think
it's useful, I have the tiniest hope that if, in the future, people
will become familiar with the package and see the usefulness of
allowing empty subscript list, the decision will change. If the only
result of me posting it as a PEP is a final "rejected" status that
will prevent any chance of that happening, I don't think I'll bother
to make it a PEP. If it's not the case, then I'll make it a PEP and
post it.

Have a good week,
Noam

* Yes, I know that adding unneeded feature to the language can be
considered "harmful". You may not agree with my distinction in this
case. As it is, I barely consider this as an "added feature" - I would
say it's mostly "a small generalization".

2006/6/30, Georg Brandl <g.brandl at gmx.net>:
> skip at pobox.com wrote:
> > Noam Raphael posted an empty subscript PEP on the Python Wiki:
> >
> >     http://wiki.python.org/moin/EmptySubscriptListPEP
> >
> > It's not linked to by any other pages on the wiki.  Is there a reason it
> > wasn't added to the peps repository?
>
> Perhaps the author forgot to submit it to the PEP editor, or he decided
> to abandon it after the mostly negative discussion here.
>
> Georg
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/noamraph%40gmail.com
>

From pedronis at strakt.com  Sat Jul  1 22:10:40 2006
From: pedronis at strakt.com (Samuele Pedroni)
Date: Sat, 01 Jul 2006 22:10:40 +0200
Subject: [Python-Dev] doc for new restricted execution design for Python
In-Reply-To: <bbaeab100606281054l49bc7442sd4d14ccb005f35e1@mail.gmail.com>
References: <bbaeab100606211733x3dd58226g15e2919e8e2cdd7e@mail.gmail.com>	<Pine.LNX.4.58.0606261937050.17937@server1.LFW.org>	<bbaeab100606261800x5949cb89h97424fc052e33534@mail.gmail.com>	<44A2BF2E.4020706@activestate.com>
	<bbaeab100606281054l49bc7442sd4d14ccb005f35e1@mail.gmail.com>
Message-ID: <44A6D6C0.70408@strakt.com>

Brett Cannon wrote:
> 
> I don't know how JavaScript is doing it yet.  The critical thing for me 
> for this month was trying to come up with a security model.
> 
> And if you don't think it is going to scale, how do you think it should 
> be done?

if I remember correctly, the boundary/granularity of mutual isolation
is practically web domains, pages from the same domain can liberally 
access each other data, javascript state.

From brett at python.org  Sat Jul  1 22:27:51 2006
From: brett at python.org (Brett Cannon)
Date: Sat, 1 Jul 2006 13:27:51 -0700
Subject: [Python-Dev] For sandboxing: alternative to crippling file()
In-Reply-To: <44A60127.3030706@canterbury.ac.nz>
References: <bbaeab100606291148o1e03d44ardbb0d3358aae7513@mail.gmail.com>
	<44A60127.3030706@canterbury.ac.nz>
Message-ID: <bbaeab100607011327y56adf85fid0ec8b10efd8849c@mail.gmail.com>

On 6/30/06, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
>
> Brett Cannon wrote:
>
> > 1) Is removing 'file' from the builtins dict in PyInterpreterState (and
> > maybe some other things) going to be safe enough to sufficiently hide
> > 'file' confidently (short of someone being stupid in their C extension
> > module and exposing 'file' directly)?
> >
> > 2) Changing open() to return C-implemented delegate objects for files
> > (and thus won't type check, but this is Python so I am not worried about
> > that too much) and delegate socket objects for IP and URL addresses.
>
> My suggestion is to change things so that the constructor
> of the file type doesn't open files (at least in restricted
> mode). Then it wouldn't matter if untrusted code had real
> file objects, as they couldn't use them to get access to
> any other files.


So require use of open() to open a file and then put the access restrictions
in open() while turning off the constructor for file?

Seems reasonable.  It basically shifts the access restrictions to open()
instead of 'file'.  For some reason this proposal makes me want to remove
the checks in read/write methods as well.  That way there is only open()
that needs to do the checks and 'file' can have the constructor crippled and
that be it.  Really minimize the impact of code on 'file' itself.

Do people think that having the restriction checks for every read/write
method is necessary?  I originally thought of doing that so that if an open
file object leaked into a restricted interpreter by accident there would
still be proper protections, but perhaps that should not be the
resopnsibility of 'file' and instead should be more up to modules not
passing back exposed 'file' objects into code.

-Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20060701/1087426f/attachment.html 

From talin at acm.org  Sat Jul  1 22:32:02 2006
From: talin at acm.org (Talin)
Date: Sat, 01 Jul 2006 13:32:02 -0700
Subject: [Python-Dev] More Switch: Explicit freezing
Message-ID: <44A6DBC2.5080302@acm.org>

Here's another stab at the "explicit freezing" school of thought on the 
switch semantics. The idea is to borrow the freeze protocol and apply it 
to functions.

In this scheme, the default behavior of switch is to rebuild the 
dictionary each time the switch is executed. However, by calling 
freeze(), you can get a 'frozen' version of the function in which all 
switch dictionaries contained within the function are precalculated:

    # Emulating 'freeze at function definition time'

    def myfunc( x ):
       switch y:
       case a:
          ...
       case b:
          ...

    myfunc = freeze( myfunc )

This of course lends itself well to decorator syntax:

    @freeze
    def myfunc( x ):
       switch y:
       case a:
          ...
       case b:
          ...

You can also get 'freeze on first use' via the appropriate decorator 
function, although that's a litte harder to white (essentially, you need 
a way to test if the function is already frozen.)

Each time you call freeze(), you get a new copy of the function object 
with the switch dictionaries bound to the values that were in the scope 
of the call to 'freeze'. This means that you can call freeze several 
times and get several different versions of the function:

    def myfunc( x ):
       switch y:
       case a:
          ...
       case b:
          ...

    a = 1
    b = 2

    f1 = freeze( myfunc )

    a = 3
    b = 4

    f2 = freeze( myfunc )

Now we have two versions of the function, each having a different switch 
dictionary.

Note that 'switch' is still usable without 'freeze', it just won't run 
as fast. This means that the folks who are interested in a switch 
statement purely for its improved expressiveness can simply not bother 
with freezing the function.

-- Talin

From skip at pobox.com  Sun Jul  2 00:26:17 2006
From: skip at pobox.com (skip at pobox.com)
Date: Sat, 1 Jul 2006 17:26:17 -0500
Subject: [Python-Dev] zlib module build failure on Mac OSX 10.4.7
In-Reply-To: <315DECAC-D582-4692-A0CA-13D3C7BF992C@mac.com>
References: <17573.60593.759293.549583@montanaro.dyndns.org>
	<130298B1-7336-480A-A90F-97B68327BE4A@mac.com>
	<17574.43390.560231.425494@montanaro.dyndns.org>
	<315DECAC-D582-4692-A0CA-13D3C7BF992C@mac.com>
Message-ID: <17574.63113.771422.208090@montanaro.dyndns.org>


    Ronald> What does /usr/lib/libz.dylib point to on your system?  

    % cd /usr/lib
    % ls -l libz.*
    lrwxr-xr-x   1 root  wheel     12 Feb 12 00:32 libz.1.1.3.dylib -> libz.1.dylib
    -rwxr-xr-x   1 root  wheel  72588 Jun 29 18:36 libz.1.2.3.dylib
    lrwxr-xr-x   1 root  wheel     16 Feb 12 00:32 libz.1.dylib -> libz.1.2.3.dylib
    lrwxr-xr-x   1 root  wheel     16 Feb 12 00:32 libz.dylib -> libz.1.2.3.dylib

Looks like everything on my system winds up at 1.2.3.

    Ronald> What I don't understand yet is why your copy of libz doesn't
    Ronald> have inflateCopy. 

It appears to:

    % nm libz.1.2.3.dylib | egrep -i inflate
             U _inflate
             U _inflateEnd
             U _inflateInit2_
             U _inflateReset
             U _inflate
             U _inflateEnd
             U _inflateInit_
    libz.1.2.3.dylib(inflate.o):
    9110ea18 T _inflate
    911168cc T _inflateCopy
    9110e5d8 T _inflateEnd
    91116694 t _inflateGetHeader
    9110dc84 T _inflateInit2_
    9110e680 T _inflateInit_
    91116524 t _inflatePrime
    9110dddc T _inflateReset
    91116584 T _inflateSetDictionary
    91116744 T _inflateSync
    91116888 T _inflateSyncPoint
             u _inflate_fast
             u _inflate_table
    91116b90 T _inflateBack
    91117a6c T _inflateBackEnd
    91116a38 T _inflateBackInit_
             u _inflate_fast
             u _inflate_table
    91118334 s _inflate_copyright
    9111047c t _inflate_table
    91110930 t _inflate_fast

However, even though the zlib.so is linked with -lz, there's no evidence of
it in the otool -L output:

    % make
    case $MAKEFLAGS in \
    *-s*)  CC='gcc' LDSHARED='gcc  -bundle -undefined dynamic_lookup' OPT='-DNDEBUG -g -O3 -Wall -Wstrict-prototypes' ./python.exe -E ../setup.py -q build;; \
    *)  CC='gcc' LDSHARED='gcc  -bundle -undefined dynamic_lookup' OPT='-DNDEBUG -g -O3 -Wall -Wstrict-prototypes' ./python.exe -E ../setup.py build;; \
    esac
    running build
    running build_ext
    db.h: found (4, 2) in /sw/include/db4
    db lib: using (4, 2) db-4.2
    sqlite: found /usr/include/sqlite3.h
    /usr/include/sqlite3.h: version 3.1.3
    building 'zlib' extension
    gcc -fno-strict-aliasing -Wno-long-double -no-cpp-precomp -mno-fused-madd -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -I. -I/Users/skip/src/python-svn/trunk/./Include -I/Users/skip/src/python-svn/trunk/./Mac/Include -I/Users/skip/local/include -I../Include -I. -I/usr/local/include -I/Users/skip/src/python-svn/trunk/Include -I/Users/skip/src/python-svn/trunk/build -c /Users/skip/src/python-svn/trunk/Modules/zlibmodule.c -o build/temp.macosx-10.3-ppc-2.5/Users/skip/src/python-svn/trunk/Modules/zlibmodule.o
    /Users/skip/src/python-svn/trunk/Modules/zlibmodule.c: In function 'PyZlib_uncopy':
    /Users/skip/src/python-svn/trunk/Modules/zlibmodule.c:724: warning: implicit declaration of function 'inflateCopy'
    gcc -bundle -undefined dynamic_lookup build/temp.macosx-10.3-ppc-2.5/Users/skip/src/python-svn/trunk/Modules/zlibmodule.o -L/Users/skip/local/lib -L/usr/local/lib -lz -o build/lib.macosx-10.3-ppc-2.5/zlib.so -Wl,-search_paths_first
    *** WARNING: renaming "zlib" since importing it failed: dlopen(build/lib.macosx-10.3-ppc-2.5/zlib.so, 2): Symbol not found: _inflateCopy
      Referenced from: build/lib.macosx-10.3-ppc-2.5/zlib.so
      Expected in: dynamic lookup

    running build_scripts
    montanaro:build% otool -L build/lib.macosx-10.3-ppc-2.5/zlib_failed.so 
    build/lib.macosx-10.3-ppc-2.5/zlib_failed.so:
            /usr/lib/libmx.A.dylib (compatibility version 1.0.0, current version 93.0.0)
            /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 88.1.6)

Ah, found it!  There was an antique libz.a in /usr/local/lib dating from
2003.  It's all better now.

Skip

From skip at pobox.com  Sun Jul  2 00:44:33 2006
From: skip at pobox.com (skip at pobox.com)
Date: Sat, 1 Jul 2006 17:44:33 -0500
Subject: [Python-Dev] Empty Subscript PEP on Wiki - keep or toss?
In-Reply-To: <b348a0850607011306i6ca3d001w132c81379b6a1889@mail.gmail.com>
References: <17573.33939.473091.920283@montanaro.dyndns.org>
	<e840po$k1i$1@sea.gmane.org>
	<b348a0850607011306i6ca3d001w132c81379b6a1889@mail.gmail.com>
Message-ID: <17574.64209.730168.967635@montanaro.dyndns.org>


    Noam> If the only result of me posting it as a PEP is a final "rejected"
    Noam> status that will prevent any chance of that happening, I don't
    Noam> think I'll bother to make it a PEP. If it's not the case, then
    Noam> I'll make it a PEP and post it.

Even if it's ultimately rejected, it still serves as useful documentation of
the process.  I'd go ahead and update it to reflect the latest discussions,
submit it, then let the chips fall where they may.  That would also get it
out of the wiki, orphaned page that it is.

Skip

From python-dev at zesty.ca  Sun Jul  2 00:50:22 2006
From: python-dev at zesty.ca (Ka-Ping Yee)
Date: Sat, 1 Jul 2006 17:50:22 -0500 (CDT)
Subject: [Python-Dev] Lexical scoping in Python 3k
In-Reply-To: <44A60A79.8090802@canterbury.ac.nz>
References: <001201c69c65$b3869750$6402a8c0@arkdesktop>
	<Pine.LNX.4.58.0606301658240.17937@server1.LFW.org>
	<ca471dc20606301524n52ba3c2dj870b650fc600282@mail.gmail.com>
	<Pine.LNX.4.58.0606301726170.17937@server1.LFW.org>
	<44A60A79.8090802@canterbury.ac.nz>
Message-ID: <Pine.LNX.4.58.0607011748250.17937@server1.LFW.org>

On Sat, 1 Jul 2006, Greg Ewing wrote:
> I don't disagree with anything you said, but I think it
> would be a good idea to avoid using phrases like "proper
> lexical scopes", which is likely to set people off on
> a tangent. The issue isn't lexicality, it's writeability.

"Fully functional" lexical scopes, then?  Python's scopes are
lexical (except for builtins) but currently somewhat hamstrung.


-- ?!ng

From ark at acm.org  Sun Jul  2 01:42:28 2006
From: ark at acm.org (Andrew Koenig)
Date: Sat, 1 Jul 2006 19:42:28 -0400
Subject: [Python-Dev] Lexical scoping in Python 3k
In-Reply-To: <Pine.LNX.4.58.0607011748250.17937@server1.LFW.org>
Message-ID: <001301c69d68$070e5fe0$6402a8c0@arkdesktop>

> "Fully functional" lexical scopes, then? 

Fine-grained scopes?




From ark at acm.org  Sun Jul  2 02:22:46 2006
From: ark at acm.org (Andrew Koenig)
Date: Sat, 1 Jul 2006 20:22:46 -0400
Subject: [Python-Dev] 2.5 and beyond
In-Reply-To: <1f7befae0607010108hb99c574l9e0d7e3cbf378355@mail.gmail.com>
Message-ID: <001401c69d6d$a7cd8d20$6402a8c0@arkdesktop>

> Don't recall what that was, but creating a new scope on each iteration
> sounds hard to explain in Python.

I don't think it's particularly hard to explain.  For example, one way to
explain it is to say that

	for i in <<stuff>>:
		body

is equivalent to

	for <<hiddenvar>> in <<stuff>>:
		local i = <<hiddenvar>>
		body

This explanation doesn't need to rest on recursion.




From tim.peters at gmail.com  Sun Jul  2 03:08:25 2006
From: tim.peters at gmail.com (Tim Peters)
Date: Sat, 1 Jul 2006 21:08:25 -0400
Subject: [Python-Dev] 2.5 and beyond
In-Reply-To: <001401c69d6d$a7cd8d20$6402a8c0@arkdesktop>
References: <1f7befae0607010108hb99c574l9e0d7e3cbf378355@mail.gmail.com>
	<001401c69d6d$a7cd8d20$6402a8c0@arkdesktop>
Message-ID: <1f7befae0607011808x7f7b2661h7f96a57f4bedc672@mail.gmail.com>

[Tim]
>> Don't recall what that was, but creating a new scope on each iteration
>> sounds hard to explain in Python.

[Andrew Koenig]
> I don't think it's particularly hard to explain.  For example, one way to explain it is
> to say that
>
>         for i in <<stuff>>:
>                 body
>
> is equivalent to
>
>         for <<hiddenvar>> in <<stuff>>:
>                 local i = <<hiddenvar>>
>                 body
>
> This explanation doesn't need to rest on recursion.

Sorry, but as a Python programmer that explanation makes little sense
to me.  In effect, it pushes the mystery into what "local" is supposed
to mean, but there's nothing _already_ in Python that acts the way you
need "local" to act.

Scope in Python is defined wrt "blocks", so you need to phrase this in
terms of blocks, and there are very few kinds of blocks in Python's
execution model:

    A block is a piece of Python program text that is executed as a unit.
    The following are blocks: a module, a function body, and a class definition.
    Each command typed interactively is a block. A script file (a file given as
    standard input to the interpreter or specified on the interpreter command
    line the first argument) is a code block. A script command (a command
    specified on the interpreter command line with the `-c' option) is a code
    block. The file read by the built-in function execfile() is a code block.
    The string argument passed to the built-in function eval() and to the exec
    statement is a code block. The expression read and evaluated by the
    built-in function input() is a code block.

That's from section "Naming and binding" of the Python Reference
Manual.  I expect most Python programmers have "module, function,
class ... plus some weird stuff I don't much care about" in mind.
Python's execution model also has a one-to-one correspondence between
active blocks and execution frames (see the rest of that section),
which would need to be snapped to consider a finer-grained notion of
block that didn't have its own execution frame.

In short, it's only easy to define this in Python, without invoking
nested functions, if you don't have Python's execution model in mind
to begin with.

From fdrake at acm.org  Sun Jul  2 04:04:47 2006
From: fdrake at acm.org (Fred L. Drake, Jr.)
Date: Sat, 1 Jul 2006 22:04:47 -0400
Subject: [Python-Dev]
	=?iso-8859-1?q?how_long_to_wait_for_expat_to_incorpo?=
	=?iso-8859-1?q?rate_a_fix_to=09prevent_a_crasher=3F?=
In-Reply-To: <44A583A6.6010602@v.loewis.de>
References: <bbaeab100606301119jebac400vc10b96d8ac1d3029@mail.gmail.com>
	<44A583A6.6010602@v.loewis.de>
Message-ID: <200607012204.47631.fdrake@acm.org>

On Friday 30 June 2006 16:03, Martin v. L?wis wrote:
 > If you have a patch, you should commit it to our copy. Make sure you
 > activate the test case, so that somebody incorporating the next Expat
 > release doesn't mistakenly roll back your change.

A modified version of Brett's patch has been committed to Expat, along with 
regression tests for two specific cases that it handles (only one of which is 
relevant to Python).  The patch to xmlparse.c has also been committed to 
Python's copy, and the crasher test has been moved to the regular 
xml.parsers.expat tests.

 > Of course, you might wait a few days to see whether Fred creates another
 > release that we could incorporate without introducing new features.

I'm not ready to push for an Expat release, since I've not had much time to 
pay attention to that project over the past year.  I'm trying to catch up on 
that project's email, but don't expect it to be quick.  Once I've had time to 
discuss this with the current principal maintainer, it shouldn't be difficult 
to get a 2.0.1 release out the door.  Once that's done, it'll be time to sync 
with the Expat release again.


  -Fred

-- 
Fred L. Drake, Jr.   <fdrake at acm.org>

From ncoghlan at gmail.com  Sun Jul  2 05:26:48 2006
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 02 Jul 2006 13:26:48 +1000
Subject: [Python-Dev] Lexical scoping in Python 3k
In-Reply-To: <e86539$o2e$1@sea.gmane.org>
References: <001201c69c65$b3869750$6402a8c0@arkdesktop>	<Pine.LNX.4.58.0606301658240.17937@server1.LFW.org>	<ca471dc20606301524n52ba3c2dj870b650fc600282@mail.gmail.com>	<Pine.LNX.4.58.0606301726170.17937@server1.LFW.org>
	<e86539$o2e$1@sea.gmane.org>
Message-ID: <44A73CF8.2080300@gmail.com>

Neil Schemenauer wrote:
>   The := would assign but not declare a variable
> in the current scope.

There are other benefits to such a statement, too, since we can make it 
similar to other augmented assignments by letting the object being assigned to 
interfere with the process.

   a := 2

could translate to something like:

   a = a.__assign__(2)

with the default behaviour of __assign__ simply being:

   def __assign__(rhs)
       return rhs

This gives you:
   - runtime checking for typos (you can't accidentally declare a new variable 
with := when you really meant to assign to an existing one)
   - if/when control flow analysis is added to the AST compiler, it will be 
picked up as an error at compile time along with the other augmented assignments
   - the object being assigned to can validate/modify its replacement (e.g. 
automatically wrapping it in a weakref proxy, or checking that it has the 
correct type)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
             http://www.boredomandlaziness.org

From jcarlson at uci.edu  Sun Jul  2 05:56:07 2006
From: jcarlson at uci.edu (Josiah Carlson)
Date: Sat, 01 Jul 2006 20:56:07 -0700
Subject: [Python-Dev] sys.settrace() in Python 2.3 vs. 2.4
In-Reply-To: <20060701085221.GC17748@code0.codespeak.net>
References: <20060630131620.10CE.JCARLSON@uci.edu>
	<20060701085221.GC17748@code0.codespeak.net>
Message-ID: <20060701205520.10DD.JCARLSON@uci.edu>


Armin Rigo <arigo at tunes.org> wrote:
> 
> Hi Josiah,
> 
> On Fri, Jun 30, 2006 at 01:27:24PM -0700, Josiah Carlson wrote:
> > I'll just have to gracefully degrade functionality for older Pythons. 
> 
> More precisely, the bug shows up because in
> 
>   while 1:
>       pass
> 
> the current line remains on the 'pass' forever.  It works for a loop
> like that:
> 
>   while 1:
>       sys
>       sys
> 
> but it's admittedly quite obscure.

That is good to know, thank you Armin.

 - Josiah


From jcarlson at uci.edu  Sun Jul  2 06:50:11 2006
From: jcarlson at uci.edu (Josiah Carlson)
Date: Sat, 01 Jul 2006 21:50:11 -0700
Subject: [Python-Dev] weakattr
In-Reply-To: <1d85506f0607010649y65c9fb24mbc5c636360970e52@mail.gmail.com>
References: <1d85506f0607010649y65c9fb24mbc5c636360970e52@mail.gmail.com>
Message-ID: <20060701213745.10E0.JCARLSON@uci.edu>


"tomer filiba" <tomerfiliba at gmail.com> wrote:
> weakattr (weak attributes) are attributes that are weakly referenced
> by their containing object. they are very useful for cyclic references --
> an object that holds a reference to itself.

I like the added functionality offered with weakattrs as defined.  I'm
not terribly in love with the syntax of their creation, and I'm curious
as to how it plays with __slots__ (not quite having the time to look at
its implementation right now), but it is quite explicit, so I can get
past that. It would allow us to say, "stop using __del__, use weakattrs",
but I'm not sure how well that would work, generally.

Toss it out in python-list, I think some people over there would be able
to offer more feedback.

 - Josiah


From jcarlson at uci.edu  Sun Jul  2 07:46:57 2006
From: jcarlson at uci.edu (Josiah Carlson)
Date: Sat, 01 Jul 2006 22:46:57 -0700
Subject: [Python-Dev] More Switch: Explicit freezing
In-Reply-To: <44A6DBC2.5080302@acm.org>
References: <44A6DBC2.5080302@acm.org>
Message-ID: <20060701215239.10E3.JCARLSON@uci.edu>


Talin <talin at acm.org> wrote:
> Here's another stab at the "explicit freezing" school of thought on the 
> switch semantics. The idea is to borrow the freeze protocol and apply it 
> to functions.

-1 .  Freezing was previously questionably useful in the realm of
general data structures.  Using switch/case as a use-case for this
functionality, I think, is a non-starter.

 - Josiah


From arigo at tunes.org  Sun Jul  2 13:23:44 2006
From: arigo at tunes.org (Armin Rigo)
Date: Sun, 2 Jul 2006 13:23:44 +0200
Subject: [Python-Dev] LOAD_CONST POP_TOP
In-Reply-To: <e83r4i$t31$1@sea.gmane.org>
References: <e83r4i$t31$1@sea.gmane.org>
Message-ID: <20060702112344.GA30481@code0.codespeak.net>

Hi Georg,

On Fri, Jun 30, 2006 at 08:39:13PM +0200, Georg Brandl wrote:
> +                       case LOAD_CONST:
> +                               cumlc = lastlc + 1;
> +                               /* Skip over LOAD_CONST POP_TOP */
> +                               if (codestr[i+3] == POP_TOP) {

This is missing a ISBASICBLOCK() check.  It makes the following example
segfault:

    a = 5
    for i in range(1000):
        a or 4

Attached an updated patch to the SF tracker.


A bientot,

Armin.

From jan-python at maka.demon.nl  Sun Jul  2 13:58:55 2006
From: jan-python at maka.demon.nl (jan-python at maka.demon.nl)
Date: Sun, 02 Jul 2006 13:58:55 +0200
Subject: [Python-Dev] 2.5 and beyond
In-Reply-To: <mailman.0.1151840193.19336.python-dev@python.org>
References: <mailman.0.1151840193.19336.python-dev@python.org>
Message-ID: <20060702135855.bdk6r8kugw0c0w0o@webmail.nl.demon.net>

Hi everyone, Even though I'm new on this list I think I've got something
sensible to say on this one. (I've been following this list a bit through the
archives)

Andrew Koenig wrote:
> You seem to be right -- the problem is not that Python is lexically   scoped,
> but that when you define a variable with =, it leaks out into the
> surrounding function scope.

> So maybe what I meant when I asked for lexical scopes was two things:
>
> 1) Every indentation level should be a scope;
> 2) In general, variable definitions should not leak into
>    surrounding scopes.
>
> I realize that (2) is too simplistic.  Someone who writes

I believe the problem has nothing to do with how many scopes a block/function
definition has, but with what the lambda does with the scope it's given.
Currently it remembers the block and looks up the nescessary variables in it
when it's invoked. I think it shoud should have just taken the values of the
needed variables and rememberd those as it's own local variables. So 
the closed
over variables become just local variables initialised to the value 
they have in
the outer scope.

Without having any blocks to be confused about, I think this is
counterintuitive
as well:

>>> x = 1
>>> f = lambda: x
>>> x = 2
>>> g = lambda: x
>>> f()
2
>>> g()
2

I think it should have been:
....
>>> f()
1
>>> g()
2

Using the lambda x=x: x trick gives exactly this behaviour because it
apparently does copy the value of x. As far as I can see it also solves the
>>> for i in range(10):
  ...     a.append(lambda: i)
case, and other similar examples.
(However, this would probably be a to big change for 2.5)


From pje at telecommunity.com  Sun Jul  2 16:07:57 2006
From: pje at telecommunity.com (Phillip J. Eby)
Date: Sun, 02 Jul 2006 10:07:57 -0400
Subject: [Python-Dev] 2.5 and beyond
In-Reply-To: <20060702135855.bdk6r8kugw0c0w0o@webmail.nl.demon.net>
References: <mailman.0.1151840193.19336.python-dev@python.org>
	<mailman.0.1151840193.19336.python-dev@python.org>
Message-ID: <5.1.1.6.0.20060702100646.01f1b878@sparrow.telecommunity.com>

At 01:58 PM 7/2/2006 +0200, jan-python at maka.demon.nl wrote:
>I believe the problem has nothing to do with how many scopes a block/function
>definition has, but with what the lambda does with the scope it's given.
>Currently it remembers the block and looks up the nescessary variables in it
>when it's invoked. I think it shoud should have just taken the values of the
>needed variables and rememberd those as it's own local variables. So
>the closed
>over variables become just local variables initialised to the value
>they have in
>the outer scope.

That won't work.  Consider this code, that's perfectly valid Python today:

    def foo():
        def bar():
            print x
        for x in range(10):
            bar()


From amk at amk.ca  Sun Jul  2 18:00:44 2006
From: amk at amk.ca (A.M. Kuchling)
Date: Sun, 2 Jul 2006 12:00:44 -0400
Subject: [Python-Dev] Another 2.5 bug candidate?
Message-ID: <20060702160044.GA21217@rogue.amk.ca>

http://www.python.org/sf/1488934 argues that Python's use of fwrite()
has incorrect error checking; this most affects file.write(), but
there are other uses of fwrite() in the core.  It seems fwrite() can
return N bytes written even if an error occurred, and the code needs
to also check ferror(f->fp).

At the last sprint I tried to assemble a small test case to exhibit
the problem but failed.  The reporter's test case uses SSH, and I did
verify that Python does loop infinitely if executed under SSH, but a
test case would need to work without SSH.

Should this be fixed in 2.5?  I'm nervous about such a change to error
handling without a test case to add; maybe it'll cause problems on one
of our platforms.

--amk

From nmm1 at cus.cam.ac.uk  Sun Jul  2 20:08:19 2006
From: nmm1 at cus.cam.ac.uk (Nick Maclaren)
Date: Sun, 02 Jul 2006 19:08:19 +0100
Subject: [Python-Dev]  Another 2.5 bug candidate?
Message-ID: <E1Fx6Ml-0002fU-8r@virgo.cus.cam.ac.uk>

"A.M. Kuchling" <amk at amk.ca> wrote:
> 
> http://www.python.org/sf/1488934 argues that Python's use of fwrite()
> has incorrect error checking; this most affects file.write(), but
> there are other uses of fwrite() in the core.  It seems fwrite() can
> return N bytes written even if an error occurred, and the code needs
> to also check ferror(f->fp).
> 
> At the last sprint I tried to assemble a small test case to exhibit
> the problem but failed.  The reporter's test case uses SSH, and I did
> verify that Python does loop infinitely if executed under SSH, but a
> test case would need to work without SSH.
>         
> Should this be fixed in 2.5?  I'm nervous about such a change to error
> handling without a test case to add; maybe it'll cause problems on one
> of our platforms.

So would assembling a test case.  NOTHING will cause ferror to return
True that isn't classed as undefined behaviour, and therefore may fail
on some platforms.


Regards,
Nick Maclaren,
University of Cambridge Computing Service,
New Museums Site, Pembroke Street, Cambridge CB2 3QH, England.
Email:  nmm1 at cam.ac.uk
Tel.:  +44 1223 334761    Fax:  +44 1223 334679

From python-dev at zesty.ca  Mon Jul  3 00:07:43 2006
From: python-dev at zesty.ca (Ka-Ping Yee)
Date: Sun, 2 Jul 2006 17:07:43 -0500 (CDT)
Subject: [Python-Dev] Lexical scoping in Python 3k
In-Reply-To: <000701c69d41$7dc7a9b0$6402a8c0@arkdesktop>
References: <000701c69d41$7dc7a9b0$6402a8c0@arkdesktop>
Message-ID: <Pine.LNX.4.58.0607021645090.17937@server1.LFW.org>

On Sat, 1 Jul 2006, Andrew Koenig wrote:
> I'd rather see a simpler rule: = never defines a variable in a surrounding
> scope.  If you want to affect the binding of such a variable, you have to
> define it explicitly in the scope in which you want it.
>
> Example:
>
> 	x = 42
> 	def f():
> 		x = 123		# rebinds x as defined above
> 		y = 123		# defines local variable
> 	f()
> 	print x			# prints 123
> 	print y			# error -- y not defined
>
> Yes, I know that rule is too simplistic.  But I think I'd still prefer it to
> the way things are now.

I agree with you that this is a nicer and more consistent rule.
What do you think of the proposal for a keyword to say "don't rebind"?
It would achieve the same distinction you're aiming for above, but
without the drastic incompatibility with today's Python.

This has been previously discussed as "change the meaning of 'global'
to mean 'not local'":

    http://mail.python.org/pipermail/python-dev/2006-February/061568.html

    http://mail.python.org/pipermail/python-dev/2006-July/066908.html

I support this proposal, though i would prefer a clearer keyword
such as "outer x" or "nonlocal x".  If we can't agree on another
keyword (or can't afford to spend one more keyword), i'm willing
to support "global" for this purpose.


-- ?!ng

From greg.ewing at canterbury.ac.nz  Mon Jul  3 03:11:43 2006
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Mon, 03 Jul 2006 13:11:43 +1200
Subject: [Python-Dev] Lexical scoping in Python 3k
In-Reply-To: <20060701013710.10D7.JCARLSON@uci.edu>
References: <20060630163625.10D1.JCARLSON@uci.edu>
	<44A612E8.8050205@canterbury.ac.nz>
	<20060701013710.10D7.JCARLSON@uci.edu>
Message-ID: <44A86ECF.4060705@canterbury.ac.nz>

Josiah Carlson wrote:

> If the only code that benefits from such changes are "very *simple*",
> then I think that says something about its necessity.

The point is that they're only "very simple" if you
can write them using access to an outer scope. Without
that ability, they become less simple, less efficient,
more convoluted, harder to follow, etc.

Also I don't buy the argument that something has to
be useful for big, complicated things in order to be
worth having in the language.

--
Greg

From greg.ewing at canterbury.ac.nz  Mon Jul  3 03:36:31 2006
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Mon, 03 Jul 2006 13:36:31 +1200
Subject: [Python-Dev] 2.5 and beyond
In-Reply-To: <024901c69d43$ea5a6480$d503030a@trilan>
References: <000601c69d41$073dbf00$6402a8c0@arkdesktop>
	<024901c69d43$ea5a6480$d503030a@trilan>
Message-ID: <44A8749F.2000801@canterbury.ac.nz>

Giovanni Bajo wrote:
> I believe that names in
> lambdas/nested-functions referring to local names in the outer scope should
> really be bound at function definition time

No, you don't want that, because it would make functions that
call each other very awkward to arrange.

> And it's also handy that the iteration variable of the for loop is
> accessible after the for loop is terminated (in fact, this specific
> behaviour is already listed among the wont-change for Py3k).

I'd just like to point out that the create-a-new-cell
behaviour that I have proposed for loop variables
*preserves* this ability!

   for new i in range(10):
     ...
   print i

will still print 9.

--
Greg

From greg.ewing at canterbury.ac.nz  Mon Jul  3 03:50:54 2006
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Mon, 03 Jul 2006 13:50:54 +1200
Subject: [Python-Dev] 2.5 and beyond
In-Reply-To: <1f7befae0607010108hb99c574l9e0d7e3cbf378355@mail.gmail.com>
References: <5.1.1.6.0.20060630190709.01ef5858@sparrow.telecommunity.com>
	<001201c69c9c$b6531fd0$6402a8c0@arkdesktop>
	<1f7befae0606301639u6f8c968cr29afce0ee7fbb43b@mail.gmail.com>
	<020c01c69ca1$754c7310$d1b12997@bagio>
	<44A61301.2070805@canterbury.ac.nz>
	<1f7befae0607010108hb99c574l9e0d7e3cbf378355@mail.gmail.com>
Message-ID: <44A877FE.6040004@canterbury.ac.nz>

Tim Peters wrote:

> Scheme has no loops in Python's sense --
> things like "do" are shorthand for expressing stylized recursion

But it does have foreach and map, which are the
moral equivalent of Python's for-loops and list
comprehensions. The body is a lambda which takes
the loop variable as a parameter, thus providing
the extra level of scope. Recursion isn't needed.

> When people talk about giving a Python for-loop vrbl its own scope,
> they generally don't mean a new scope on _each iteration_,

But that's exactly what you *do* need in order for
a for-loop with a lambda in it to behave intuitively.
If that's not what people mean, it's because they
don't fully understand what they really mean to
mean. :-)

BTW, I'm not suggesting that a new stack frame
gets allocated for every iteration -- all you need
is a cell.

> about the same as creating a nested block with
> its own autos in C.

Analogies with C aren't very helpful here, because
it doesn't have closures, so it's only a matter of
visibility, not lifetime.

 > creating a new scope on each iteration
 > sounds hard to explain in Python.

But is it harder to explain than the reason someone's
loop-with-a-lambda doesn't do what they expect?

BTW, I wouldn't explain it by saying it creates a new
scope, I'd say it creates a new binding on each
iteration, or something like that.

In my earlier proposal, you would actually say that
explicitly, with something like

   for new i in range(10):
     ...

> Abusing the default-argument machinery to capture current bindings is
> never necessary, and _is_ abuse.

But it's abuse that still happens, because although
scoping has been fixed, other parts of the story are
still missing.

--
Greg

From steven.bethard at gmail.com  Mon Jul  3 01:28:08 2006
From: steven.bethard at gmail.com (Steven Bethard)
Date: Sun, 2 Jul 2006 17:28:08 -0600
Subject: [Python-Dev] DRAFT: python-dev summary for 2006-06-01 to 2006-06-15
Message-ID: <d11dcfba0607021628x34824e4o45ccafca11a719f3@mail.gmail.com>

Here's the summary for the first half of June.  Thanks in advance for
your comments and corrections!


=============
Announcements
=============

-------------------
Python 2.5 schedule
-------------------

Python 2.5 is moving steadily towards its next release.  See `PEP
356`_ for more details and the full schedule.

.. _PEP 356: http://www.python.org/dev/peps/pep-0356/

Contributing threads:

- `beta1 coming real soon
<http://mail.python.org/pipermail/python-dev/2006-June/065729.html>`__
- `2.5 issues need resolving in a few days
<http://mail.python.org/pipermail/python-dev/2006-June/065730.html>`__

-----------------------------------------------
Request for Bug Trackers to replace SourceForge
-----------------------------------------------

The Python Software Foundation's Infrastructure committee asked for
suggestions for tracker systems that could replace SourceForge. The
minimum requirements are:

* Can import SourceForge data
* Can export data
* Has an email interface

and if you'd like to suggest a particular tracker system all you need to do is:

* Install a test tracker
* Import the `SourceForge data dump`_
* Make the `Infrastructure committee members`_ administrators of the tracker
* Add your tracker to the `wiki page`_
* Email `the Infrastructure committee`_

Be sure to check the `wiki page`_ for additional information.

.. _SourceForge data dump: http://effbot.org/zone/sandbox-sourceforge.htm
.. _Infrastructure committee members:
http://wiki.python.org/moin/PythonSoftwareFoundationCommittees#infrastructure-committee-ic
.. _wiki page: http://wiki.python.org/moin/CallForTrackers
.. _the Infrastructure committee: infrastructure at python.org

Contributing thread:

- `Request for trackers to evaluate as SF replacement for Python
development <http://mail.python.org/pipermail/python-dev/2006-June/065597.html>`__


=========
Summaries
=========

--------------------------------------------
Getting more comparable results from pybench
--------------------------------------------

Skip Montanaro mentioned that the NeedForSpeed_ folks had some trouble
with the pybench_ string and unicode tests. In some discussions both
on the checkins list and off-list, Fredrik Lundh had concluded that
stringbench more reliably reported performance than pybench. There was
then a long discussion about how to improve pybench including:

* Using time.clock() on Windows and time.time() on Linux. This was
accompanied by a long debate about whether to use wall-time or process
time.  Both wall time and process time can see interference from other
programs running at the same time; wall time because the time consumed
by other programs running at the same time is also counted, and
process time because it is sampled so that other processes can charge
their time to the running process by using less than a full time
slice. In general, the answer was to use the timer with the best
resolution.

* Using the minimum time rather than the average. Andrew Dalke
explained that timing results do not have a Gaussian distribution
(they have more of a gamma distribution) and provided some graphs
generated on his machine to demonstrate this. Since the slower runs
are typically caused by other things running at the same time (which
is pretty much unpredictable), it's much better to report the fastest
run, which should more consistently approximate the best possible
time.

* Making sure to use an appropriate warp factor. Marc-Andre Lemburg
explained that each testing round of pybench is expected to take
around 20-50 seconds. If rounds are much shorter than this, pybench's
warp factor should be adjusted until they are long enough.

At the end of the thread, Marc-Andre checked in pybench_ 2.0, which
included the improvements suggested above.

.. _NeedForSpeed: http://wiki.python.org/moin/NeedForSpeed
.. _pybench: http://svn.python.org/view/python/trunk/Tools/pybench/
.. _stringbench: http://svn.python.org/view/sandbox/trunk/stringbench/

Contributing threads:

- `Python Benchmarks
<http://mail.python.org/pipermail/python-dev/2006-May/065480.html>`__
- `Python Benchmarks
<http://mail.python.org/pipermail/python-dev/2006-June/065525.html>`__

---------------------------------------
PEP 360: Externally Maintained Packages
---------------------------------------

After checking wsgiref into the Python repository, Phillip J. Eby
requested in `PEP 360`_ that patches to wsgiref be passed to him
before being committed on the trunk. After a number of changes were
committed to the trunk and he had to go through a complicated two-way
merge, he complained that people were not following the posted
procedures. Guido suggested that `PEP 360`_ was a mistake, and that
whenever possible, development for any module in the stdlib should be
done in the Python trunk, not externally. He also requested that the
PEP indicate that even for externally maintained modules, bugfixes and
ordinary maintenance should be allowed on the trunk so that bugs in
external modules don't hold up Python core development.

A number of solutions were discussed for authors of code that is also
distributed standalone. Using svn:externals is mostly undesirable
because svn is much slower at checking whether or not an svn:externals
directory is updated, and because upgrading to a newer version would
require making sure that no changes made by Python developers were
lost in the new version. Phillip suggested adding an "Externals"
directory and modifying Python's setup to invoke all the
``Externals/*/setup.py`` scripts, though this would mean having some
Python code that lives outside of the Lib/ subtree. Barry Warsaw
explained that for the email package, he maintains a directory in the
sandbox with all the distutils and documentation stuff needed for the
standalone releases as well as the email package from the Python
repository through svn:externals. This means having to create some
extra directories (since svn:externals doesn't work with individual
files) and having one checkout per version of Python supported, but
seemed to work pretty well for Barry. People seemed to like Phillip's
Externals idea (possibly renamed to Packages), but work on that was
postponed for Python 2.6.

One of the side benefits of these discussions was that Thomas Heller
generously offered to move ctypes development fully into the Python
repository.

.. _PEP 360: http://www.python.org/dev/peps/pep-0360/

Contributing threads:

- `wsgiref documentation
<http://mail.python.org/pipermail/python-dev/2006-June/065576.html>`__
- `wsgiref doc draft; reviews/patches wanted
<http://mail.python.org/pipermail/python-dev/2006-June/065647.html>`__
- `[Web-SIG] wsgiref doc draft; reviews/patches wanted
<http://mail.python.org/pipermail/python-dev/2006-June/065701.html>`__
- `FYI: wsgiref is now checked in
<http://mail.python.org/pipermail/python-dev/2006-June/065767.html>`__
- `Please stop changing wsgiref on the trunk
<http://mail.python.org/pipermail/python-dev/2006-June/065908.html>`__
- `Dropping externally maintained packages (Was: Please stop changing
wsgiref on the trunk)
<http://mail.python.org/pipermail/python-dev/2006-June/065919.html>`__
- `External Package Maintenance (was Re: Please stop changing wsgiref
on the trunk) <http://mail.python.org/pipermail/python-dev/2006-June/065920.html>`__
- `External Package Maintenance
<http://mail.python.org/pipermail/python-dev/2006-June/065937.html>`__
- `rewording PEP 360
<http://mail.python.org/pipermail/python-dev/2006-June/066001.html>`__
- `Updating packages from external ?
<http://mail.python.org/pipermail/python-dev/2006-June/066035.html>`__

--------------------------------------
Universally unique identifiers (UUIDs)
--------------------------------------

Ka-Ping Yee was looking to put his `uuid module`_ into Python 2.5. He
addressed a number of requests from the last round of discussions,
including making UUIDs immutable, removing curly braces from the UUID
string and adding the necessary tests to the test suite. Then he asked
about how best to address the fact that ``uuid1()`` required looking
up a MAC address, a potentially slow procedure. At the suggestion of
Fredrik Lundh, he changed the API to allow a MAC address to be passed
in if it was already known. If a MAC address is not passed in to
``uuid1()``, the ``getnode()`` utility function is called, which
searches for the MAC address through a variety of routes, including
some quicker paths through ctypes that Thomas Heller and others helped
Ka-Ping with. The code was checked into the Python trunk.

.. _uuid module: http://zesty.ca/python/uuid.py

Contributing thread:

- `UUID module <http://mail.python.org/pipermail/python-dev/2006-June/065733.html>`__

-------------------------------------
PEP 275: Switching on Multiple Values
-------------------------------------

Thomas Lee offered up a `patch implementing the switch statement`_
from `PEP 275`_. People brought up a number of concerns with the
implementation (and the switch statement in general). The
implementation didn't allow for any way of allowing multiple values to
be mapped to the same case (without repeating the code in the case).
The implementation also made the switch statement essentially
syntactic sugar for a series of if/elif/else statements, and people
were concerned that just adding another way to write if/elif/else was
not much of a gain for Python. The discussion continued on into the
next fortnight.

.. _patch implementing the switch statement: http://bugs.python.org/1504199
.. _PEP 275: http://www.python.org/dev/peps/pep-0275/

Contributing thread:

- `Switch statement
<http://mail.python.org/pipermail/python-dev/2006-June/065827.html>`__

---------------------------------------------------------
The period of the random module's random number generator
---------------------------------------------------------

Alex Martelli noticed a note in random.shuffle.__doc__ which said that
most permutations of a long sequence would never be generated due to
the short period of the random number generator. This turned out to be
an artifact from back when Python used the Whichman-Hill generator
instead of the Mersenne Twister generator it uses currently. There was
some discussion as to whether the comment should be removed or
updated, and Robert Kern pointed out that at sequence lengths of 2081
or greater, the comment was still true. Tim Peters decided it was best
to just remove the comment, explaining that "anyone sophisticated
enough to *understand* an accurate warning correctly would have no
need to be warned".

Contributing thread:

- `a note in random.shuffle.__doc__ ...
<http://mail.python.org/pipermail/python-dev/2006-June/065815.html>`__

-------------------------------------------------------
Pre-PEP: Allow Empty Subscript List Without Parentheses
-------------------------------------------------------

Noam Raphael presented a `pre-PEP for empty subscript lists`_ in
getitem-style access to objects. This would allow zero-dimensional
arrays to work in a similar manner to all other N dimensional arrays,
and make all of the following equivalences hold::

   x[i, j]     <-->  x[(i, j)]
   x[i,]       <-->  x[(i,)]
   x[i]        <-->  x[(i)]
   x[]         <-->  x[()]

Most people felt that zero-dimensional arrays were uncommon enough
that either they could be replaced with simple names, e.g. ``x``, or
could use the currently available syntax, i.e. ``x[()]``.
Zero-dimensional arrays are even uncommon in numpy_ where, after
`rehashing the issue`_ innumerable times, zero-dimensional arrays have
been almost entirely replaced with scalars.

.. _pre-PEP for empty subscript lists:
http://wiki.python.org/moin/EmptySubscriptListPEP
.. _numpy: http://numeric.scipy.org/
.. _rehashing the issue:
http://projects.scipy.org/scipy/numpy/wiki/ZeroRankArray

Contributing thread:

- `Pre-PEP: Allow Empty Subscript List Without Parentheses
<http://mail.python.org/pipermail/python-dev/2006-June/065762.html>`__

----------------------------------------------
PEP 337: Logging Usage in the Standard Library
----------------------------------------------

For the `Google Summer of Code`_, Jackilyn Hoxworth has been working
on implementing parts of `PEP 337`_ to use the logging module in parts
of the stdlib. When Jim Jewett, who is mentoring her, brought up a few
issues, people got concerned that this work was being done at all,
being that `PEP 337`_ has not been approved. Jim and A.M. Kuchling
clarified that the goal of Jackilyn's work is to both clarify the PEP
(e.g. determine exactly which modules would benefit from logging) and
to provide an implementation that can be tweaked as necessary if the
PEP is accepted. For the first draft at least, it looked like Jackilyn
would keep things simple -- using "py." + __name__ for the logger
name, not adding any new logging messages, not changing any message
formats, and generally aiming only to give stderr and stdout messages
across different modules a common choke point.

.. _Google Summer of Code: http://code.google.com/soc/
.. _PEP 337: http://www.python.org/dev/peps/pep-0337/

Contributing thread:

- `Stdlib Logging questions (PEP 337 SoC)
<http://mail.python.org/pipermail/python-dev/2006-June/065601.html>`__

-------------------
inspect.isgenerator
-------------------

Michele Simionato asked for a new function in the inspect module that
would identify a function as being a generator function. Phillip J.
Eby pointed out that any function can return a generator-iterator
(though generator functions are of course guaranteed to do so) and
suggested that the perceived need for this inspect function was
misguided. Michele agreed and withdrew the proposal.

Contributing threads:

- `feature request: inspect.isgenerator
<http://mail.python.org/pipermail/python-dev/2006-May/065334.html>`__
- `feature request: inspect.isgenerator
<http://mail.python.org/pipermail/python-dev/2006-June/065508.html>`__

--------------------------------
Unescaping entities with sgmllib
--------------------------------

Sam Ruby asked why sgmllib unescapes entities selectively, not all or
nothing (which would be easier to work around), and Fred L. Drake, Jr.
explained that sgmllib is really only intended as support for htmllib.
Sam suggested isolating the code that attempts to resolve character
references into a single method so that subclasses could override this
behavior as needed. Martin v. L?wis agreed that this seemed
reasonable, though he suggested two functions, one for character
references and one for entity references. Sam implemented the
suggested behavior and provided a `patch to sgmllib`_.

.. _patch to sgmllib: http://bugs.python.org/1504676

Contributing thread:

- `sgmllib Comments
<http://mail.python.org/pipermail/python-dev/2006-June/065861.html>`__

-------------------------------------------------------------------------
Scoping vs augmented assignment vs sets (Re: 'fast locals' in Python 2.5)
-------------------------------------------------------------------------

A bug in Python 2.5 that did not detect augmented assignment as
creating a local name allowed code like the following to work::

    >>> g = 1
    >>> def f1():
    ...     g += 1
    ...
    >>> f1()
    >>> g
    2

This of course started the usual discussion about giving Python a way
to rebind names in enclosing scopes. Boris Borcic in particular was
hoping that the bug could be considered a feature, but Terry Reedy
explained that Python was not willing to give up the near equivalence
between ``x = x + 1`` and ``x += 1``. Since the former creates a local
name, the latter ought to do the same thing. The thread seemed like it
might drift on further until Guido cut it off, pronouncing that the
behavior of augmented assignments creating local names was not going
to change.

Contributing threads:

- `'fast locals' in Python 2.5
<http://mail.python.org/pipermail/python-dev/2006-June/065648.html>`__
- `Scoping vs augmented assignment vs sets (Re: 'fast locals' in
Python 2.5) <http://mail.python.org/pipermail/python-dev/2006-June/065902.html>`__
- `Comparing closures and arguments (was Re: Scoping vs augmented
assignment vs sets (Re: 'fast locals' in Python 2.5)
<http://mail.python.org/pipermail/python-dev/2006-June/066064.html>`__
- `The baby and the bathwater (Re: Scoping, augmented assignment,
'fast locals' - conclusion)
<http://mail.python.org/pipermail/python-dev/2006-June/066080.html>`__

---------------------------------------
Checking out an older version of Python
---------------------------------------

Skip Montanaro asked about checking out a particular version of
Python. Oleg Broytmann and Tim Peters explained that tags are just
directories in Subversion, and you can vew all the existing ones and
their corresponding revision numbers at
http://svn.python.org/projects/python/tags/. Oleg also explained that
the difference between::

    svn switch svn+ssh://pythondev at svn.python.org/python/tags/r242

and noting that the r242 tag corresponds to revision 39619 and doing::

    svn up -r 39619

is that with the latter, commits will go to the trunk (assuming the
update was performed on a trunk checkout), while with the former,
updates will go to the appropriate tag or branch. Giovanni Bajo
provided a nice explanation of this, describing Subversion's 2D
coordinate system of [url, revision] and Skip added the explanation to
the `Development FAQ`_.

.. _Development FAQ: http://www.python.org/dev/faq/

Contributing thread:

- `Subversion repository question - back up to older versions
<http://mail.python.org/pipermail/python-dev/2006-June/065709.html>`__

--------------------
Source control tools
--------------------

In the externally maintained packages discussion, Guido suggested
offhand that some other version control project might make it easier
to resolve some of the issues. Thomas Wouters put forward a number of
considerations.  On the negative side of changing to one of the newer
version control systems:

* Workflow would have to change somewhat to use most of the new
branch-oriented systems.
* Everyone would have to download the whole repository (at least once)
since with the newer systems everyone usually has their own
repository.

But on the positive side:

* History can be preserved for merges of brances (unlike Subversion),
which is a big gain for when the trunk is switched to 3.0.

Thomas tried importing the Python repository into a number of
different systems, and after playing around with them, concluded that
in the short term, none of the other version control systems were
quite ready yet, though he seemed optimistic for them in the next few
years. He also promised to publish imports of the Python repository
into Git, Darcs, Mercurial, Bazaar-NG and Monotone somewhere once he
was able to successfully import them all.

Contributing thread:

- `Source control tools
<http://mail.python.org/pipermail/python-dev/2006-June/065971.html>`__

----------------------------------------
Is implicit underscore assignment buggy?
----------------------------------------

Raymond Hettinger noted that in the interactive interpreter, an
expression that returns None not only suppresses the printing of that
None, but also suppresses the assignment to ``_``. Raymond asked if
this was intentional as it makes code like the following break::

    >>> import re, string
    >>> re.search('lmnop', string.letters)
    <_sre.SRE_Match object at 0xb6f2c480>
    >>> re.search('pycon', string.letters)
    >>> if _ is not None:
    ...     print _.group()
    lmnop

Fredrik Lundh pointed out that users just need to recognize that the
``_`` holds the most recently *printed* result. Guido pronounced that
this would not change. Terry Reedy suggested adding some documentation
for this behavior to either Language Reference 2.3.2 Reserved Classes
of Identifiers and/or to Tutorial 2.1.2 Interactive Mode, but it was
unclear if any doc changes were committed.

Contributing thread:

- `Is implicit underscore assignment buggy?
<http://mail.python.org/pipermail/python-dev/2006-June/065679.html>`__

-----------------------
Removing MAC OS 9 cruft
-----------------------

A number of old MAC OS 9 bits and pieces that are no longer used were removed:

* IDE scripts
* MPW
* Tools/IDE
* Tools/macfreeze
* Unsupported/mactcp/dnrglue.c
* Wastemods

This should solve some problems for Windows checkouts where files with
trailing dots are not supported.

Contributing threads:

- `Removing Mac OS 9 cruft
<http://mail.python.org/pipermail/python-dev/2006-June/065526.html>`__
- `Mac/wastemodule build failing
<http://mail.python.org/pipermail/python-dev/2006-June/065588.html>`__

------------------------------------------
Fixing buffer object's char buffer support
------------------------------------------

Brett Cannon found that ``import array;
int(buffer(array.array('c')))`` caused the interpreter to segfault
because buffer objects were redirecting tp_as_buffer->bf_getcharbuffer
to the wrong tp_as_buffer slot. Brett fixed the bug and updated the
docs a bit to clarify what was intended for the implementation, but
kept changes pretty minimal as Python 3.0 will ditch buffer for the
bytes type anyway.

Contributing threads:

- `How to fix the buffer object's broken char buffer support
<http://mail.python.org/pipermail/python-dev/2006-June/065646.html>`__
- `Is "t#" argument format meant to be char buffer, or just read-only?
<http://mail.python.org/pipermail/python-dev/2006-June/065678.html>`__

-------------------------------
Importing subpackages in Jython
-------------------------------

In Jython 2.1, importing a module makes all subpackages beneath it
available, unlike in regular Python, where subpackages must be
imported separately. Samuele Pedroni explained that this was
intentional so that imports in Jython would work like imports in Java
do. Guido suggested that having imports work this way in Jython was
fine as long as a Java package was being imported, but when a Python
package was being imported, Jython should use the Python semantics.

Contributing thread:

- `Import semantics
<http://mail.python.org/pipermail/python-dev/2006-June/065864.html>`__

---------------------------------------------
RFC 3986: Uniform Resource Identifiers (URIs)
---------------------------------------------

There was some continued discussion of Paul Jimenez's proposed
`uriparse module`_ which more faithfully implements `RFC 3986`_ than
the current urlparse module. Nick Coghlan submitted an `alternate
implementation`_ that kept all parsed URIs as (scheme, authority,
path, query, fragment) tuples by allowing some of these elements to be
non-strings, e.g. authority could be a (user, password, host, port)
tuple, and path could be a (user, host) tuple. People seemed to like
Nick's implementation, but no final decision on the module was made.

.. _uriparse module: http://bugs.python.org/1462525
.. _RFC 3986: http://www.ietf.org/rfc/rfc3986.txt
.. _alternate implementation: http://bugs.python.org/1500504

Contributing thread:

- `Some more comments re new uriparse module, patch 1462525
<http://mail.python.org/pipermail/python-dev/2006-June/065552.html>`__

-----------------------------------------------------
False instead of TypeError for frozenset.__contains__
-----------------------------------------------------

Collin Winter suggested that code like ``{} in frozenset([1, 2, 3])``
should return False instead of raising a TypeError. Guido didn't like
the idea because he thought it would mask bugs where, say, a
user-defined __hash__() method accidentally raised a TypeError.

Contributing thread:

- `Unhashable objects and __contains__()
<http://mail.python.org/pipermail/python-dev/2006-June/065577.html>`__

--------------------------------------------
IOError or ValueError for invalid file modes
--------------------------------------------

Kristj?n V. J?nsson asked why open()/file() throws an IOError for an
invalid mode string instead of a ValueError. Georg Brandl explained
that either an IOError or a ValueError can be raised depending on
whether the invalid mode was detected in Python's code or in the OS's
fopen call. Guido suggested that this couldn't really be fixed until
Python gets rid of its stdio-based implementation in Python 3.0.

Contributing thread:

- `file() <http://mail.python.org/pipermail/python-dev/2006-June/065903.html>`__

---------------------------------------------------------------
[Python-checkins] r46603 - python/trunk/Lib/test/test_struct.py
---------------------------------------------------------------

Martin Blais checked in un-unittestification of test_struct, and a
number of people questioned whether that was a wise thing to do.
Thomas Wouters suggested that unittest should merge as many features
from py.test_ as possible. This would reduce some of the class-based
boilerplate currently required, and also allow some nice additional
features like test cases generated on the fly. He didn't get much of a
response though, so it was unclear what the plans for Python 2.6 were.

.. _py.test: http://codespeak.net/py/current/doc/test.html

Contributing thread:

- `[Python-checkins] r46603 - python/trunk/Lib/test/test_struct.py
<http://mail.python.org/pipermail/python-dev/2006-June/065584.html>`__

-----------------------------------------------
Should hex() yield 'L' suffix for long numbers?
-----------------------------------------------

Ka-Ping Yee asked why hex() and oct() still produced an 'L' suffix for
long numbers even now that ints and longs have basically been unified.
`PEP 237`_ had mentioned the removal of this suffix, but not given it
a specific release for removal, so people decided it was best to wait
until Python 3.0 when the 'L' suffix will also be removed from repr().

.. _PEP 237: http://www.python.org/dev/peps/pep-0237/

Contributing thread:

- `Should hex() yield 'L' suffix for long numbers?
<http://mail.python.org/pipermail/python-dev/2006-June/065870.html>`__

---------------------------------
Adding an index of Python symbols
---------------------------------

Terry Reedy suggested adding a page to the Python Language Reference
index that would list each symbol in Python (e.g. ``()``, ``[]`` and
``@``) along with the places in the documentation where it was
discussed. Terry promised to submit a plain-text version in time for
the Python 2.5 release, so that someone could convert it to LaTeX and
merge it into the docs.

Contributing thread:

- `Symbol page for Language Reference Manual Index
<http://mail.python.org/pipermail/python-dev/2006-June/065681.html>`__

------------------------------------------
Behavior of searching for empty substrings
------------------------------------------

Fredrik Lundh resolved the issues discussed previously with searching
for an empty substring at a position past the end of the string. The
current behavior looks like::

    >>> "ab".find("")
    0
    >>> "ab".find("", 1)
    1
    >>> "ab".find("", 2)
    2
    >>> "ab".find("", 3)
    -1

Both Tim Peters and Guido applauded the final resolution.

Contributing thread:

- `Search for empty substrings (was Re: Let's stop eating exceptions
in dict lookup)
<http://mail.python.org/pipermail/python-dev/2006-June/065500.html>`__

-----------------
subprocess.IGNORE
-----------------

Martin Blais asked about adding subprocess.IGNORE along the lines of
subprocess.PIPE which would ignore the child's output without being
susceptible to buffer deadlock problems. Under Unix, IGNORE could be
implemented as ``open('/dev/null', 'w')``, and on Windows,
``open('nul:', 'w')``. People seemed to think this was a useful
feature, but at the time of this summary, no patch had yet been
provided.

Contributing thread:

- `subprocess.Popen(.... stdout=IGNORE, ...)
<http://mail.python.org/pipermail/python-dev/2006-June/065868.html>`__


================
Deferred Threads
================
- `Improve error msgs?
<http://mail.python.org/pipermail/python-dev/2006-June/066048.html>`__
- `Keeping interned strings in a set
<http://mail.python.org/pipermail/python-dev/2006-June/066084.html>`__
- `Documentation enhancement: "MS free compiler"?
<http://mail.python.org/pipermail/python-dev/2006-June/066182.html>`__
- `Code coverage reporting.
<http://mail.python.org/pipermail/python-dev/2006-June/066184.html>`__
- `Numerical robustness, IEEE etc.
<http://mail.python.org/pipermail/python-dev/2006-June/066186.html>`__


==================
Previous Summaries
==================
- `Let's stop eating exceptions in dict lookup
<http://mail.python.org/pipermail/python-dev/2006-June/065499.html>`__
- `ssize_t question: longs in header files
<http://mail.python.org/pipermail/python-dev/2006-June/065574.html>`__
- `ssize_t: ints in header files
<http://mail.python.org/pipermail/python-dev/2006-June/065624.html>`__
- `zlib module doesn't build - inflateCopy() not found
<http://mail.python.org/pipermail/python-dev/2006-June/065706.html>`__


===============
Skipped Threads
===============
- `Segmentation fault of Python if build on Solaris 9 or10 with Sun
Studio 11 <http://mail.python.org/pipermail/python-dev/2006-June/065495.html>`__
- `Possible bug in complexobject.c (still in Python 2.5)
<http://mail.python.org/pipermail/python-dev/2006-June/065496.html>`__
- `[Python-checkins] r46300 - in python/trunk: Lib/socket.py
Lib/test/test_socket.py Lib/test/test_struct.py Modules/_struct.c
Modules/arraymodule.c Modules/socketmodule.c
<http://mail.python.org/pipermail/python-dev/2006-June/065501.html>`__
- `test_struct failure on 64 bit platforms
<http://mail.python.org/pipermail/python-dev/2006-June/065506.html>`__
- `string inconsistency
<http://mail.python.org/pipermail/python-dev/2006-June/065507.html>`__
- `S/390 buildbot URLs problematic
<http://mail.python.org/pipermail/python-dev/2006-June/065517.html>`__
- `SF patch #1473257: "Add a gi_code attr to generators"
<http://mail.python.org/pipermail/python-dev/2006-June/065524.html>`__
- `test_unicode failure on MIPS
<http://mail.python.org/pipermail/python-dev/2006-June/065529.html>`__
- `valgrind report
<http://mail.python.org/pipermail/python-dev/2006-June/065532.html>`__
- `test_ctypes failures on ppc64 debian
<http://mail.python.org/pipermail/python-dev/2006-June/065544.html>`__
- `Request for patch review
<http://mail.python.org/pipermail/python-dev/2006-June/065581.html>`__
- `patch #1454481 vs buildbot
<http://mail.python.org/pipermail/python-dev/2006-June/065592.html>`__
- `Seeking Core Developers for Vancouver Python Workshop
<http://mail.python.org/pipermail/python-dev/2006-June/065600.html>`__
- `[Python-checkins] Python Regression Test Failures refleak (1)
<http://mail.python.org/pipermail/python-dev/2006-June/065613.html>`__
- `Include/structmember.h, Py_ssize_t
<http://mail.python.org/pipermail/python-dev/2006-June/065619.html>`__
- `DC Python sprint on July 29th
<http://mail.python.org/pipermail/python-dev/2006-June/065625.html>`__
- `tarfile and unicode filenames in windows
<http://mail.python.org/pipermail/python-dev/2006-June/065710.html>`__
- `[Python-checkins] buildbot warnings in hppa Ubuntu dapper trunk
<http://mail.python.org/pipermail/python-dev/2006-June/065719.html>`__
- `-Wi working for anyone else?
<http://mail.python.org/pipermail/python-dev/2006-June/065779.html>`__
- `Inject some tracing ...
<http://mail.python.org/pipermail/python-dev/2006-June/065781.html>`__
- `Segmentation fault in collections.defaultdict
<http://mail.python.org/pipermail/python-dev/2006-June/065828.html>`__
- `Add pure python PNG writer module to stdlib?
<http://mail.python.org/pipermail/python-dev/2006-June/065834.html>`__
- `crash in dict on gc collect
<http://mail.python.org/pipermail/python-dev/2006-June/065855.html>`__
- `"can't unpack IEEE 754 special value on non-IEEE platform"
<http://mail.python.org/pipermail/python-dev/2006-June/065957.html>`__
- `socket._socketobject.close() doesn't really close sockets
<http://mail.python.org/pipermail/python-dev/2006-June/065991.html>`__
- `DRAFT: python-dev summary for 2006-04-16 to 2006-04-30
<http://mail.python.org/pipermail/python-dev/2006-June/066011.html>`__
- `[Python-checkins] r46795 - in python/trunk: Doc/lib/libstdtypes.tex
Lib/test/string_tests.py Misc/NEWS Objects/stringobject.c
Objects/unicodeobject.c
<http://mail.python.org/pipermail/python-dev/2006-June/066013.html>`__
- `xrange vs. int.__getslice__
<http://mail.python.org/pipermail/python-dev/2006-June/066014.html>`__
- `request for review: patch 1446489 (zip64 extensions in zipfile)
<http://mail.python.org/pipermail/python-dev/2006-June/066018.html>`__
- `DRAFT: python-dev summary for 2006-05-01 to 2006-05-15
<http://mail.python.org/pipermail/python-dev/2006-June/066020.html>`__
- `pychecker warnings in Lib/encodings
<http://mail.python.org/pipermail/python-dev/2006-June/066021.html>`__
- `Moving PEP 343 to Final
<http://mail.python.org/pipermail/python-dev/2006-June/066027.html>`__
- `Python sprint at Google Aug. 21-24
<http://mail.python.org/pipermail/python-dev/2006-June/066043.html>`__
- `Long options support
<http://mail.python.org/pipermail/python-dev/2006-June/066049.html>`__
- `High Level Virtual Machine
<http://mail.python.org/pipermail/python-dev/2006-June/066053.html>`__
- `sqlite3 test errors - was : Re: [Python-checkins] r46936 - in
python/trunk: Lib/sqlite3/test/regression.py Lib/sqlite3/test/types.py
Lib/sqlite3/test/userfunctions.py Modules/_sqlite/connection.c
Modules/_sqlite/cursor.c Modules/_sqlite/module.c
Modules/_sqlite/module.h
<http://mail.python.org/pipermail/python-dev/2006-June/066055.html>`__
- `[Python-checkins] sqlite3 test errors - was : Re: r46936 - in
python/trunk: Lib/sqlite3/test/regression.py Lib/sqlite3/test/types.py
Lib/sqlite3/test/userfunctions.py Modules/_sqlite/connection.c
Modules/_sqlite/cursor.c Modules/_sqlite/module.c
Modules/_sqlite/module.h
<http://mail.python.org/pipermail/python-dev/2006-June/066056.html>`__
- `[Python-checkins] sqlite3 test errors - was : Re: r46936 - in
python/trunk: Lib/sqlite3/test/regression.py Lib/sqlite3/test/types.py
Lib/sqlite3/test/userfunctions.py Modules/_sqlite/connection.c
Modules/_sqlite/cursor.c Modules/_sqlite/module.c
<http://mail.python.org/pipermail/python-dev/2006-June/066060.html>`__
- `[Python-checkins] sqlite3 test errors - was : Re: r46936 - in
python/trunk: Lib/sqlite3/test/regression.py Lib/sqlite3/test/types.py
Lib/sqlite3/test/userfunctions.py Modules/_sqlite/connection.c
Modules/_sqlite/cursor.c Modules/_sql
<http://mail.python.org/pipermail/python-dev/2006-June/066068.html>`__
- `Last-minute curses patch
<http://mail.python.org/pipermail/python-dev/2006-June/066076.html>`__
- `DRAFT: python-dev summary for 2006-05-16 to 2006-05-31
<http://mail.python.org/pipermail/python-dev/2006-June/066078.html>`__
- `Bug: xml.dom.pulldom never gives you END_DOCUMENT events with an
Expat parser <http://mail.python.org/pipermail/python-dev/2006-June/066179.html>`__
- `Misleading error message from PyObject_GenericSetAttr
<http://mail.python.org/pipermail/python-dev/2006-June/066183.html>`__
- `About dynamic module loading
<http://mail.python.org/pipermail/python-dev/2006-June/066185.html>`__

From jcarlson at uci.edu  Mon Jul  3 07:15:39 2006
From: jcarlson at uci.edu (Josiah Carlson)
Date: Sun, 02 Jul 2006 22:15:39 -0700
Subject: [Python-Dev] Lexical scoping in Python 3k
In-Reply-To: <44A86ECF.4060705@canterbury.ac.nz>
References: <20060701013710.10D7.JCARLSON@uci.edu>
	<44A86ECF.4060705@canterbury.ac.nz>
Message-ID: <20060702211647.10F1.JCARLSON@uci.edu>


Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
> Josiah Carlson wrote:
> > If the only code that benefits from such changes are "very *simple*",
> > then I think that says something about its necessity.
> 
> The point is that they're only "very simple" if you
> can write them using access to an outer scope. Without
> that ability, they become less simple, less efficient,
> more convoluted, harder to follow, etc.

As is known and has been stated, assigning to a parent scope can be
emulated in various ways, either through an explicit namespace object, or
through a namespace list.


> Also I don't buy the argument that something has to
> be useful for big, complicated things in order to be
> worth having in the language.

I never claimed that something needed to be useful for "big, complicated
things" in order to be worth having in the language.  To be explicit, if
nontrivial code isn't improved, that doesn't necessarily mean that the
feature is useless.  However, if the feature is really only useful for
generally trivial cases *without* the feature, then making them even
more trivial, I think, is a bit of over optimization.

 - Josiah


From guido at python.org  Mon Jul  3 07:25:21 2006
From: guido at python.org (Guido van Rossum)
Date: Mon, 3 Jul 2006 07:25:21 +0200
Subject: [Python-Dev] Lexical scoping in Python 3k
In-Reply-To: <20060702211647.10F1.JCARLSON@uci.edu>
References: <20060701013710.10D7.JCARLSON@uci.edu>
	<44A86ECF.4060705@canterbury.ac.nz>
	<20060702211647.10F1.JCARLSON@uci.edu>
Message-ID: <ca471dc20607022225u2a5c8e7v963194bcf61d9abc@mail.gmail.com>

On 7/3/06, Josiah Carlson <jcarlson at uci.edu> wrote:
>
> Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
> > Josiah Carlson wrote:
> > > If the only code that benefits from such changes are "very *simple*",
> > > then I think that says something about its necessity.
> >
> > The point is that they're only "very simple" if you
> > can write them using access to an outer scope. Without
> > that ability, they become less simple, less efficient,
> > more convoluted, harder to follow, etc.
>
> As is known and has been stated, assigning to a parent scope can be
> emulated in various ways, either through an explicit namespace object, or
> through a namespace list.

And the fact that this desire and need remains, even amongst people
who should know better, suggests that it may be worth supporting it
more directly, as the current work-arounds ain't pretty.

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From jcarlson at uci.edu  Mon Jul  3 08:08:33 2006
From: jcarlson at uci.edu (Josiah Carlson)
Date: Sun, 02 Jul 2006 23:08:33 -0700
Subject: [Python-Dev] Lexical scoping in Python 3k
In-Reply-To: <ca471dc20607022225u2a5c8e7v963194bcf61d9abc@mail.gmail.com>
References: <20060702211647.10F1.JCARLSON@uci.edu>
	<ca471dc20607022225u2a5c8e7v963194bcf61d9abc@mail.gmail.com>
Message-ID: <20060702225058.10F4.JCARLSON@uci.edu>


"Guido van Rossum" <guido at python.org> wrote:
> 
> On 7/3/06, Josiah Carlson <jcarlson at uci.edu> wrote:
> >
> > Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
> > > Josiah Carlson wrote:
> > > > If the only code that benefits from such changes are "very *simple*",
> > > > then I think that says something about its necessity.
> > >
> > > The point is that they're only "very simple" if you
> > > can write them using access to an outer scope. Without
> > > that ability, they become less simple, less efficient,
> > > more convoluted, harder to follow, etc.
> >
> > As is known and has been stated, assigning to a parent scope can be
> > emulated in various ways, either through an explicit namespace object, or
> > through a namespace list.
> 
> And the fact that this desire and need remains, even amongst people
> who should know better, suggests that it may be worth supporting it
> more directly, as the current work-arounds ain't pretty.


Perhaps not pretty, but not wholly ugly either.  Or expressed another
way, it's a wart, but the wart isn't 1" across on a forehead, it's
fairly small and tucked away on an elbow.

I had hoped that there would be a response to my second (and I believe
more applicable statement); "if the feature is really only useful for
generally trivial cases *without* the feature, then making them even
more trivial, I think, is a bit of over optimization."

As for a solution, I find the "global means 'not local'" proposition is
the least undesireable of the possibilities.  It suffers from a change
in semantics and potential name masking issues, but I don't believe
these are any more serious than normal global masking for the former,
and the latter is solvable with a __future__ (at least for 2.6). I'm a
solid -0 on this particular proposition, which is far better than the -1
I am on all of the other recent lexical scoping propositions.

 - Josiah


From martin at v.loewis.de  Mon Jul  3 08:19:14 2006
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Mon, 03 Jul 2006 08:19:14 +0200
Subject: [Python-Dev] Proposal to eliminate PySet_Fini
In-Reply-To: <20060627184404.GL10485@performancedrivers.com>
References: <d38f5330606271109s39f64022w53261832cd17c6b@mail.gmail.com>
	<20060627184404.GL10485@performancedrivers.com>
Message-ID: <44A8B6E2.2040003@v.loewis.de>

Jack Diederich wrote:
> PyObject_MALLOC does a good job of reusing small allocations but it
> can't quite manage the same speed as a free list, especially for things that
> have some extra setup involved (tuples have a free list for each length).

I would question that statement, for any practical purposed. The cost of
tuple comes from setting the elements to NULL, and that has to be done
regardless of whether they were allocated new or came from the list.
Likewise, the GC management has to be done regardless. So I expect that
the speedup is rather minor, and not worth it.

Regards,
Martin

From talin at acm.org  Mon Jul  3 09:27:04 2006
From: talin at acm.org (Talin)
Date: Mon, 03 Jul 2006 00:27:04 -0700
Subject: [Python-Dev] Lexical scoping in Python 3k
In-Reply-To: <20060702225058.10F4.JCARLSON@uci.edu>
References: <20060702211647.10F1.JCARLSON@uci.edu>	<ca471dc20607022225u2a5c8e7v963194bcf61d9abc@mail.gmail.com>
	<20060702225058.10F4.JCARLSON@uci.edu>
Message-ID: <44A8C6C8.3030401@acm.org>

Josiah Carlson wrote:
> I had hoped that there would be a response to my second (and I believe
> more applicable statement); "if the feature is really only useful for
> generally trivial cases *without* the feature, then making them even
> more trivial, I think, is a bit of over optimization."

It really depends on how common the trivial case is. In other words, 
multiply the savings for each occurance times the number of occurances. 
(Unfortunately, I don't know what units to measure said savings in - is 
there a unit of 'mental disconnect' or unintuitiveness? :)

In an idealy world, the language would allow everything to be said in 
the most comprehensible way possible. Longer and more verbose ways of 
stating something are at an inherent disadvantage in this, simply 
because of the time it takes to scan and absorb the information by the 
human brain. However, losing excess syntax has to be done in a way that 
doesn't also lose information. Highly compressed representations of a 
concept may require such a level of abstraction that it is as much work 
to puzzle out their meaning as it would be to read the longer version 
and more.

To put it another way - I am an advocate of applying Claude Shannon's 
theory of information to language design. The highest level of 
compression should be used for expressions that occur the most frequently.

> As for a solution, I find the "global means 'not local'" proposition is
> the least undesireable of the possibilities.  It suffers from a change
> in semantics and potential name masking issues, but I don't believe
> these are any more serious than normal global masking for the former,
> and the latter is solvable with a __future__ (at least for 2.6). I'm a
> solid -0 on this particular proposition, which is far better than the -1
> I am on all of the other recent lexical scoping propositions.

I'd say that the more common case is where you want global to really 
mean global - that is, you want to be able to write to some module-level 
variable, regardless of how deeply nested your function scope is. While 
being able to access the 'next outer scope' is occasionally useful, it's 
not all that common. So changing the behavior of 'global' in this case 
would be both confusing (since it no longer means 'global'), and less 
useful (because it doesn't match the most common case.)

(This assumes that I haven't completely understood the meaning of the 
phrase 'not local' - I assumed that it means 'not defined in this scope')

Of course, the reason why it's not all that common may be because of the 
fact that it's not as easy to do, and so people tend to (consciously or 
otherwise) avoid that pattern in their designs. That being said, I don't 
think that's necessarily such a bad thing. Python isn't Scheme, and the 
scoping rules of Python are IMHO more oriented towards practicality and 
common sense than theoretical purity.

This is why I'm not bothered by the fact that Python doesn't create a 
new scope for loop statements and such. Most of the time, this is what 
you want. It does mean that you need to name all of your variables 
uniquely, but that's good programming style in any case. The same is 
true for local variables not needing to be specially declared as 'my' or 
'var' - most of the time, a local variable is what you want.

On the other hand, the thing about theoretical purity is that it can be 
so mouth-wateringly powerful at times. For example, a language that 
supports closures is, IMHO, at least twice as powerful as a language 
that doesn't -- because you can use them in so many different and 
interesting ways.

OK, so about the lexical scoping issue - let me brainstorm a moment:

One idea would be to introduce the keyword 'local' which would have the 
effect of capturing any 'global' statements in any enclosing scope. So 
for example:

    f = 1
    def a():
       local f
       f = 2
       def b():
          global f
          f = 3

So in this case, the 'global' statement, which would normally associate 
'f' with the outermost (module-level) scope, would instead associate 'f' 
with the innermost 'local' declaration of that variable. So in the above 
example, assigning 3 to f assigns it to the middle scope, but does not 
affect the module-level definition.

Admittedly, that's a bit confusing and also verbose, considering that 
you are not only adding an extra keyword, but also using two statements 
to specify the home of one variable.

Another alternative would be a way to declare an explicitly scoped 
variable. Lets use the keyword 'my' to indicate this:

    f = 1
    def a():
       my f = 2
       def b():
          f = 3

In this case, what the 'my' statement is doing is indicating that this 
scope 'owns' the definition of 'f' -- in other words, the definition is 
hoisted out of any enclosed scopes. So again, in the above example, the 
innermost assignment will be to the definition of 'f' in the middle scope.

What's interesting about this is that you can use the same method with 
globals:

    my f = 1
    def a():
       f = 2

    a()
    print f   # prints '2'

So again, you are indicating that the global scope 'owns' the definition 
of 'f', and any enclosed scopes should use that definition, and not 
create their own.

Of course, if you really *do* need to have your own version, you can 
always override the 'my' statement with another 'my' statement:

    my f = 1
    def a():
       my f = 2

    a()
    print f   # prints '1'

The 'my' statement essentially changes the scoping rules for all 
variables of that name, within the defining scope and all enclosed scopes.

Of course, you can also override this behavior using the 'global' 
statement, which does exactly what it does now - makes the reference 
global (i.e. module-level):

    my f = 1
    def a():
       global f
       f = 2

    a()
    print f   # prints '2'

All right, I'm pretty happy with that. Brainstorming done. :)

-- Talin

From tim.peters at gmail.com  Mon Jul  3 10:14:42 2006
From: tim.peters at gmail.com (Tim Peters)
Date: Mon, 3 Jul 2006 04:14:42 -0400
Subject: [Python-Dev] Proposal to eliminate PySet_Fini
In-Reply-To: <44A8B6E2.2040003@v.loewis.de>
References: <d38f5330606271109s39f64022w53261832cd17c6b@mail.gmail.com>
	<20060627184404.GL10485@performancedrivers.com>
	<44A8B6E2.2040003@v.loewis.de>
Message-ID: <1f7befae0607030114h49fb33edwa61bc5e55ec3b569@mail.gmail.com>

[Jack Diederich]
>> PyObject_MALLOC does a good job of reusing small allocations but it
>> can't quite manage the same speed as a free list, especially for things that
>> have some extra setup involved (tuples have a free list for each length).

[Martin v. L?wis]
> I would question that statement, for any practical purposed. The cost of
> tuple comes from setting the elements to NULL, and that has to be done
> regardless of whether they were allocated new or came from the list.

Except _that_ overhead is trivial for small tuples, and small tuples
are the only kind the free lists cache.  There are many other
overheads.  If a tuple is taken off a free list, we get to skip
integer multiplication and division checking for overflow before
calling PyObject_GC_NewVar.  We also get to skip the call to
PyObject_GC_NewVar.  That in turns skips another integer
multiplication in the _PyObject_VAR_SIZE macro, and a call to
_PyObject_GC_Malloc.  That it turn skips a call to PyObject_MALLOC,
and conditionals checking whether it's time to trigger a gc
collection.  All of that is highly significant compared to the cost of
setting at most a handful of slots to NULL inline.

> Likewise, the GC management has to be done regardless.

_PyObject_GC_TRACK expands to 5 inlined simple stores, and a
predictable branch, so it is often more expensive than setting the
tuple slots to NULL.  But, as above, we get to skip three layers of
function call and "will it overflow?" arithmetic in the service of
_setting up_ an object for gc initially.  Only the gc track/untrack
pair remains for tuples in a free list.

> So I expect that the speedup is rather minor, and not worth it.

Depends on the app :-)  Here's a test case that gets supernatural
benefit from small-tuple caching:

"""
def doit():
    N1000 = [None] * 1000
    basetup = (5,)
    for i in N1000:
        tups = []
        push = tups.append
        for j in xrange(10):
            for k in N1000:
                push(basetup * j)

from time import clock as now
times = []
for i in range(3):
    start = now()
    doit()
    finish = now()
    times.append(finish - start)
print sorted(times)
"""

With current trunk that printed

[2.9363677646013846, 2.9489729031005703, 2.9689538729183949]

After changing

#define MAXSAVEDTUPLES  2000

to

#define MAXSAVEDTUPLES  0

the times zoomed to

[4.5894824930441587, 4.6023111649343242, 4.629560027293957]

That's pretty dramatic.  OTOH, I don't have any apps that do that <0.5
wink>, and there's another downside:  on SF recently, someone
complained that the 2.5 obmalloc work to release unused arenas wasn't
doing much in his (perhaps equally artificial -- don't know) test
case.  It surprised me too, so I dug into it.  The problem turned out
to be that piles of arenas were being kept "artificially" alive
because obmalloc was the original source of thousands of tuple objects
being kept alive (from obmalloc's POV) in tupleobject.c's free lists.
If you're unlucky, it only takes one tiny tuple in one free list to
keep an entire 256KB arena alive -- and if you're very unlucky, you
manage to allocate objects in such a way that this happens repeatedly.
 His test also created lots of dicts along the way, and each arena got
mostly filled up with dict objects and a relative handful of small
tuples.  By the time all of this became trash, the tuples were spread
over a few hundred areans, which effectively became immortal.

While it doesn't make any real sense, I've seen repeatedly that users
_try_ calling gc.collect() in such cases (doesn't make sense because
it has nothing to do with cyclic gc).  But that suggests it could be a
_pragmatic_ win to add an internal "clear all known free lists"
function, which gc could call at times, or even just be forced by the
user via a `gc` entry point or gc.collect() option.

The immortal & unbounded int and float free lists don't cause obmalloc
arenas to stay alive unduly, because they get their memory in chunks
straight from the system malloc().  But they have to be the cause of
the most frequent source of complaints from newbies and Zope users
;-), who do things like

    range(10000000)

and then marvel that some 120MB of VM is still being used after nuking
the list and doing a gc.collect().  I get weary of explaining that one
:-(.

From greg.ewing at canterbury.ac.nz  Mon Jul  3 11:19:14 2006
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Mon, 03 Jul 2006 21:19:14 +1200
Subject: [Python-Dev] Lexical scoping in Python 3k
In-Reply-To: <20060702225058.10F4.JCARLSON@uci.edu>
References: <20060702211647.10F1.JCARLSON@uci.edu>
	<ca471dc20607022225u2a5c8e7v963194bcf61d9abc@mail.gmail.com>
	<20060702225058.10F4.JCARLSON@uci.edu>
Message-ID: <44A8E112.1050309@canterbury.ac.nz>

Josiah Carlson wrote:

> I had hoped that there would be a response to my second (and I believe
> more applicable statement); "if the feature is really only useful for
> generally trivial cases *without* the feature, then making them even
> more trivial, I think, is a bit of over optimization."

I don't think "trivial" is the right word to use here,
since it implies something that's of so little importance
that it can be ignored. But the simple cases are precisely
the ones where this wart hurts the most, so we can't
ignore them.

Arguments that a feature is undesirable because this
or that workaround exists seem like post-hoc justifications
to me. Think about it the other way around -- if writing
to outer scopes had been straightforward from the
beginning, would you be arguing for the *removal* of
that ability? Would it even have occurred to anyone to
do such a thing?

--
Greg

From greg.ewing at canterbury.ac.nz  Mon Jul  3 11:50:06 2006
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Mon, 03 Jul 2006 21:50:06 +1200
Subject: [Python-Dev] Lexical scoping in Python 3k
In-Reply-To: <44A8C6C8.3030401@acm.org>
References: <20060702211647.10F1.JCARLSON@uci.edu>
	<ca471dc20607022225u2a5c8e7v963194bcf61d9abc@mail.gmail.com>
	<20060702225058.10F4.JCARLSON@uci.edu> <44A8C6C8.3030401@acm.org>
Message-ID: <44A8E84E.2040806@canterbury.ac.nz>

Talin wrote:

> To put it another way - I am an advocate of applying Claude Shannon's 
> theory of information to language design. The highest level of 
> compression should be used for expressions that occur the most frequently.

I believe the proposal in question would cause no
net worsening in this information content, and may
actually improve it slightly, due to allowing a few
things to be written in a shorter and clearer way,
while allowing the vast majority of existing things
to be written in exactly the same way.

> I'd say that the more common case is where you want global to really 
> mean global - that is, you want to be able to write to some module-level 
> variable, regardless of how deeply nested your function scope is.

It would still mean that, except in the (expected to
be *extremely* rare) case where you happened to have
a variable with the same name assigned in some
intermediate scope. Such cases would be easily fixed
by renaming the intermediate variable -- using a
name of shorter or equal length, if you like, to
keep the information content up. :-)

> So changing the behavior of 'global' in this case 
> would be both confusing (since it no longer means 'global'),

An alternative would be to change the keyword as
well, to something like 'outer', which would better
match its semantics. But if that were done, I would
argue for the *removal* of the existing 'global'
keyword, which would then be almost completely
redundant. This would break large amounts of existing
code, however, and it's highly dubious whether that
would be worth the small increase in pendantic
accuracy, even in Py3k. We're not supposed to be
*gratuitously* breaking things in Py3k, after all.

> (This assumes that I haven't completely understood the meaning of the 
> phrase 'not local' - I assumed that it means 'not defined in this scope')

Yes, the new meaning would be "in the next outermost
scope where there is an assignment to this name, or
the module scope if you get that far".

> Python isn't Scheme, and the scoping rules of Python are
 > IMHO more oriented towards practicality and common sense
 > than theoretical purity.

Yes, but I find it hard to regard being *forbidden* from
assigning to intermediate scopes as something driven
by practical need rather than just being a historical
accident.

Back when there were strictly two scopes, many people
argued themselves blue in the face that this was
actually a *good* thing, even if you didn't realise
it, and that Python was doing you a favour by
enforcing it.

Eventually a measure of sanity prevailed, and we got
something a lot more like traditional lexical scoping.
But one remnant of the old system remained, like a
vestigial organ -- the 'global' statement that reaches
all the way out to the module scope, regardless of
what exists in between. To someone used to lexical
scoping in almost any other language that has it,
this is a *very* strange and unintuitive thing.

Looking back, I think the meaning of 'global' should
have been redefined right then at the same time.
That would have been the logical and consistent thing
to do, and in my opinion would have resulted in a
scoping model that was simpler, more useful and
no less practical.

The most theoretically pure thing would have been
to change it to 'outer' at the same time, but that
would have broken too much code, and would therefore
not have been practical. See, I'm not immune to
practicality arguments. :-)

> One idea would be to introduce the keyword 'local' which would have the 
> effect of capturing any 'global' statements in any enclosing scope.

That seems unnecessary to me. Or at least not
necessary enough to be worth the extra complexity
in the scoping model.

--
Greg


From skip at pobox.com  Mon Jul  3 15:01:45 2006
From: skip at pobox.com (skip at pobox.com)
Date: Mon, 3 Jul 2006 08:01:45 -0500
Subject: [Python-Dev] Lexical scoping in Python 3k
In-Reply-To: <20060702225058.10F4.JCARLSON@uci.edu>
References: <20060702211647.10F1.JCARLSON@uci.edu>
	<ca471dc20607022225u2a5c8e7v963194bcf61d9abc@mail.gmail.com>
	<20060702225058.10F4.JCARLSON@uci.edu>
Message-ID: <17577.5433.301972.602166@montanaro.dyndns.org>

    Josiah> As for a solution, I find the "global means 'not local'"
    Josiah> proposition is the least undesireable of the possibilities.  It
    Josiah> suffers from a change in semantics and potential name masking
    Josiah> issues...

Pychecker and PyLint both already identify cases where builtins are masked
by locals or module globals (and may identify cases where locals mask module
globals - I don't recall).  I suspect both could be generalized in this
regard without a huge effort.  That's probably the best place for this sort
of warning.

Skip


From ark at acm.org  Mon Jul  3 17:08:18 2006
From: ark at acm.org (Andrew Koenig)
Date: Mon, 3 Jul 2006 11:08:18 -0400
Subject: [Python-Dev] Lexical scoping in Python 3k
In-Reply-To: <44A8E112.1050309@canterbury.ac.nz>
Message-ID: <003801c69eb2$872dc990$6402a8c0@arkdesktop>

> I don't think "trivial" is the right word to use here,
> since it implies something that's of so little importance
> that it can be ignored. But the simple cases are precisely
> the ones where this wart hurts the most, so we can't
> ignore them.

I'd like to inject an example that might help make this discussion more
concrete.

Consider the following function:

	def for_each(seq, f):
		for i in seq:
			f(i)

I'm sure I've seen more than one instance of someone on comp.lang.python
trying to do the equivalent of using a function such as this one to compute
the sum of the elements of a sequence as follows:

	def sum(seq):
		result = 0
		def accum(i):
			result += i
		for_each(seq, accum)
		return result

and wonder why it doesn't work.  Still odder, why it doesn't work and the
following does:

	def sum(seq):
		result = [0]
		def accum(i):
			result[0] += i
		for_each(seq, accum)
		return result[0]

Transforming the first definition of sum above into the second may be
trivial, but only if you've encountered the technique before.  Moreover, the
first version of sum uses a technique that is more than 45 years old (!), as
it was available to Algol 60 programmers.



From martin at v.loewis.de  Mon Jul  3 17:10:16 2006
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Mon, 03 Jul 2006 17:10:16 +0200
Subject: [Python-Dev] Proposal to eliminate PySet_Fini
In-Reply-To: <1f7befae0607030114h49fb33edwa61bc5e55ec3b569@mail.gmail.com>
References: <d38f5330606271109s39f64022w53261832cd17c6b@mail.gmail.com>	<20060627184404.GL10485@performancedrivers.com>	<44A8B6E2.2040003@v.loewis.de>
	<1f7befae0607030114h49fb33edwa61bc5e55ec3b569@mail.gmail.com>
Message-ID: <44A93358.3040000@v.loewis.de>

Tim Peters wrote:
> With current trunk that printed
> 
> [2.9363677646013846, 2.9489729031005703, 2.9689538729183949]
> 
> After changing
> 
> #define MAXSAVEDTUPLES  2000
> 
> to
> 
> #define MAXSAVEDTUPLES  0
> 
> the times zoomed to
> 
> [4.5894824930441587, 4.6023111649343242, 4.629560027293957]
> 
> That's pretty dramatic.

Interesting. I ran this through gprof, and found the following
changes to the number of function calls

                   with-cache   without-cache
PyObject_Malloc         59058        24055245
tupletraverse           33574        67863194
visit_decref           131333       197199417
visit_reachable        131333       197199417
collect                    17           33006
(for reference:)
tuplerepeat          30000000        30000000

According to gprof, these functions (excluding tuplerepeat)
together account for 40% of the execution time in the without-cache
(i.e. MAXSAVEDTUPLES 0) case.

So it appears that much of the slowdown in disabling the fast
tuple allocator is due to the higher frequency of garbage collection
in your example.

Can you please re-run the example with gc disabled?

Of course, it's really no surprise that GC is called more often:
if the tuples are allocated from the cache, that doesn't count
as an allocation wrt. GC. It so happens that your example just
triggers gc a few times in its inner loop; I wouldn't attribute
that overhead to obmalloc per se.

Regards,
Martin

From guido at python.org  Mon Jul  3 17:45:49 2006
From: guido at python.org (Guido van Rossum)
Date: Mon, 3 Jul 2006 17:45:49 +0200
Subject: [Python-Dev] Lexical scoping in Python 3k
In-Reply-To: <003801c69eb2$872dc990$6402a8c0@arkdesktop>
References: <44A8E112.1050309@canterbury.ac.nz>
	<003801c69eb2$872dc990$6402a8c0@arkdesktop>
Message-ID: <ca471dc20607030845p288f6f46xd6bdc45beb138f27@mail.gmail.com>

On 7/3/06, Andrew Koenig <ark at acm.org> wrote:
> > I don't think "trivial" is the right word to use here,
> > since it implies something that's of so little importance
> > that it can be ignored. But the simple cases are precisely
> > the ones where this wart hurts the most, so we can't
> > ignore them.
>
> I'd like to inject an example that might help make this discussion more
> concrete.
>
> Consider the following function:
>
>         def for_each(seq, f):
>                 for i in seq:
>                         f(i)
>
> I'm sure I've seen more than one instance of someone on comp.lang.python
> trying to do the equivalent of using a function such as this one to compute
> the sum of the elements of a sequence as follows:
>
>         def sum(seq):
>                 result = 0
>                 def accum(i):
>                         result += i
>                 for_each(seq, accum)
>                 return result
>
> and wonder why it doesn't work.  Still odder, why it doesn't work and the
> following does:
>
>         def sum(seq):
>                 result = [0]
>                 def accum(i):
>                         result[0] += i
>                 for_each(seq, accum)
>                 return result[0]
>
> Transforming the first definition of sum above into the second may be
> trivial, but only if you've encountered the technique before.  Moreover, the
> first version of sum uses a technique that is more than 45 years old (!), as
> it was available to Algol 60 programmers.

Much though the Algol 60 tickles my nostalgia (it was my first
programming language!) I don't think that it's a particularly strong
argument. I like to think that we have better ways these days.

I think you need to come up with a better motivating example; the
above is particular un-idiomatic Python. It starts by defining a
higher-order function for_each that has little to offer over writing
an explicit for loop, and then uses this to motivate writing a simple
operation (result += i) as a function instead so that it fits in the
inconvenient for_each() API.

I understand that both for_each() and accum() are just examples of
more complicated functions, but I can't help thinking that the problem
here only occurs for very *simple* functions in the place of accum();
a more complicated form of accum would likely be a bound method of a
class instance which carries the state.

A better way to decompose these kinds of problems is probably by using
generators. The equivalent of for_each() would not take a function
parameter but *yield* the successive values instead of calling f()
with successive values; e.g.:

  def for_each(seq):
    for i in seq:
      yield i

Then the sum() function could be written like this:

  def sum(seq):
    result = 0
    for i in for_each(seq):
      result += i
    return result

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From ark at acm.org  Mon Jul  3 17:56:02 2006
From: ark at acm.org (Andrew Koenig)
Date: Mon, 3 Jul 2006 11:56:02 -0400
Subject: [Python-Dev] Lexical scoping in Python 3k
In-Reply-To: <ca471dc20607030845p288f6f46xd6bdc45beb138f27@mail.gmail.com>
Message-ID: <006c01c69eb9$323f37a0$6402a8c0@arkdesktop>

> Much though the Algol 60 tickles my nostalgia (it was my first
> programming language!) I don't think that it's a particularly strong
> argument. I like to think that we have better ways these days.

Even if so, that's not the point I was trying to make.  The point is that
there is a programming technique that is widely used, works in many
languages, and has been around for 45 years; and when you try to use it in
Python, it fails.

I believe that such failures, even if there are alternative ways of solving
the problems that engender them, are barriers to learning that should be
removed if it is possible to do so without substantial cost.




From theller at python.net  Mon Jul  3 18:19:02 2006
From: theller at python.net (Thomas Heller)
Date: Mon, 03 Jul 2006 18:19:02 +0200
Subject: [Python-Dev] Moving the ctypes repository to python.org
In-Reply-To: <44A57D6D.70701@v.loewis.de>
References: <e7ge79$kg8$1@sea.gmane.org>	<20060623145602.GB10250@niemeyer.net>	<e7h24t$t7c$1@sea.gmane.org>	<449C5157.5050004@v.loewis.de>	<e82pea$22d$1@sea.gmane.org>
	<44A57D6D.70701@v.loewis.de>
Message-ID: <44A94376.2030004@python.net>

Martin v. L?wis schrieb:
> Thomas Heller wrote:
>> - Do I need special rights to call 'svnadmin load' to import this dumpfile
>>   into Python SVN, or are the normal commit rights sufficient?
> 
> It's called "svnadmin" for a reason :-)
> 
> Neal Norwitz or myself will have to do that; we need to do it on the
> repository machine locally. I would likely take subversion write
> access offline for the time of the import, so that I can rollback
> the entire repository in case of an operator mistake.

Please tell me when you or Neal have time for the import.

>>   What exactly is the URL/PATH where it should be imported (some sandbox,
>>   I assume)?
> 
> My view is that this is the "projects" repository; with ctypes being a
> project, it should go into the root directory (i.e. as a sibling to
> python, peps, distutils, stackless, ...). If you prefer to see it in
> sandbox, this could work as well.

Having it in "projects" is fine, this matches the directory structure that
cvs2svn creates (ctypes/trunk, ctypes/tags, ctypes/branches).


>> - What about the Python trunk?  Should changes from the sandbox be merged
>>   into Modules/_ctypes and Lib/ctypes, or would it be better (or possible at all)
>>   to use the external mechanism?
> 
> I would prefer to see two-way merges going on, at least until 2.5 is
> released (i.e. no changes to Modules/ctypes except for bug fixes).

Ok.

> Using svn:external is a risky thing wrt. to branching/tagging:
> 
> When we tag the Python tree, we want to tag the entire source tree.
> With svn:external, only the external link would be in the tag,
> i.e. later changes to the external link would modify old tags.
> This is undesirable.
> 
> This problem could be solved with a versioned external link;
> this would mean that ctypes could not be edited directly, but
> that one would have to go through the original repository
> URL to perform modifications, and then update the external
> link.
> 
> So I think I still would prefer two-way merges. There are
> tools to make the merges pretty mechanic.

I have no experience at all with svn:external, so that's fine with me too.

Thanks,
Thomas


From theller at python.net  Mon Jul  3 18:20:09 2006
From: theller at python.net (Thomas Heller)
Date: Mon, 03 Jul 2006 18:20:09 +0200
Subject: [Python-Dev] Moving the ctypes repository to python.org
In-Reply-To: <44A57D6D.70701@v.loewis.de>
References: <e7ge79$kg8$1@sea.gmane.org>	<20060623145602.GB10250@niemeyer.net>	<e7h24t$t7c$1@sea.gmane.org>	<449C5157.5050004@v.loewis.de>	<e82pea$22d$1@sea.gmane.org>
	<44A57D6D.70701@v.loewis.de>
Message-ID: <44A943B9.9080708@python.net>

Martin v. L?wis schrieb:
> Thomas Heller wrote:
>> - Do I need special rights to call 'svnadmin load' to import this dumpfile
>>   into Python SVN, or are the normal commit rights sufficient?
> 
> It's called "svnadmin" for a reason :-)
> 
> Neal Norwitz or myself will have to do that; we need to do it on the
> repository machine locally. I would likely take subversion write
> access offline for the time of the import, so that I can rollback
> the entire repository in case of an operator mistake.

Please tell me when you or Neal have time for the import.

>>   What exactly is the URL/PATH where it should be imported (some sandbox,
>>   I assume)?
> 
> My view is that this is the "projects" repository; with ctypes being a
> project, it should go into the root directory (i.e. as a sibling to
> python, peps, distutils, stackless, ...). If you prefer to see it in
> sandbox, this could work as well.

Having it in "projects" is fine, this matches the directory structure that
cvs2svn creates (ctypes/trunk, ctypes/tags, ctypes/branches).


>> - What about the Python trunk?  Should changes from the sandbox be merged
>>   into Modules/_ctypes and Lib/ctypes, or would it be better (or possible at all)
>>   to use the external mechanism?
> 
> I would prefer to see two-way merges going on, at least until 2.5 is
> released (i.e. no changes to Modules/ctypes except for bug fixes).

Ok.

> Using svn:external is a risky thing wrt. to branching/tagging:
> 
> When we tag the Python tree, we want to tag the entire source tree.
> With svn:external, only the external link would be in the tag,
> i.e. later changes to the external link would modify old tags.
> This is undesirable.
> 
> This problem could be solved with a versioned external link;
> this would mean that ctypes could not be edited directly, but
> that one would have to go through the original repository
> URL to perform modifications, and then update the external
> link.
> 
> So I think I still would prefer two-way merges. There are
> tools to make the merges pretty mechanic.

I have no experience at all with svn:external, so that's fine with me too.

Thanks,
Thomas


From tomerfiliba at gmail.com  Mon Jul  3 19:00:09 2006
From: tomerfiliba at gmail.com (tomer filiba)
Date: Mon, 3 Jul 2006 19:00:09 +0200
Subject: [Python-Dev] weakattr
In-Reply-To: <20060701213745.10E0.JCARLSON@uci.edu>
References: <1d85506f0607010649y65c9fb24mbc5c636360970e52@mail.gmail.com>
	<20060701213745.10E0.JCARLSON@uci.edu>
Message-ID: <1d85506f0607031000q578c8dccka762b13ba3413de7@mail.gmail.com>

> I like the added functionality offered with weakattrs as defined.  I'm
> not terribly in love with the syntax of their creation, and I'm curious
> as to how it plays with __slots__
weakattrs are data descriptors, just like properties etc. they are part
of the class, not the instance, so there shouldn't be any trouble with
mixing those with __slots__

moreover, adding those to stdlib is very staight-forward. we don't even
need to introduce a new module. if people show interest, i'll write a
bit of a longer doc string and add some unit tests (although there's
not much to test :) )

> Toss it out in python-list, I think some people over there would be able
> to offer more feedback.
will do... although i doubt they will offer any



-tomer

On 7/2/06, Josiah Carlson <jcarlson at uci.edu> wrote:
>
> "tomer filiba" <tomerfiliba at gmail.com> wrote:
> > weakattr (weak attributes) are attributes that are weakly referenced
> > by their containing object. they are very useful for cyclic references --
> > an object that holds a reference to itself.
>
> I like the added functionality offered with weakattrs as defined.  I'm
> not terribly in love with the syntax of their creation, and I'm curious
> as to how it plays with __slots__ (not quite having the time to look at
> its implementation right now), but it is quite explicit, so I can get
> past that. It would allow us to say, "stop using __del__, use weakattrs",
> but I'm not sure how well that would work, generally.
>
> Toss it out in python-list, I think some people over there would be able
> to offer more feedback.
>
>  - Josiah
>
>

From facundobatista at gmail.com  Mon Jul  3 20:07:20 2006
From: facundobatista at gmail.com (Facundo Batista)
Date: Mon, 3 Jul 2006 15:07:20 -0300
Subject: [Python-Dev] Time-out in URL Open
Message-ID: <e04bdf310607031107ld304d14i7db52e555f8b5ddf@mail.gmail.com>

I need a timeout in urlopen, just to be able to make:

>>> urllib2.urlopen("http://no.host.org", timeout=2)

This is actually not possible, but I'll make it work.

I want to know, please, if this is useful in general, for me to post a
patch in SF.

Regards,

-- 
.    Facundo

Blog: http://www.taniquetil.com.ar/plog/
PyAr: http://www.python.org/ar/

From fdrake at acm.org  Mon Jul  3 20:12:22 2006
From: fdrake at acm.org (Fred L. Drake, Jr.)
Date: Mon, 3 Jul 2006 14:12:22 -0400
Subject: [Python-Dev] Time-out in URL Open
In-Reply-To: <e04bdf310607031107ld304d14i7db52e555f8b5ddf@mail.gmail.com>
References: <e04bdf310607031107ld304d14i7db52e555f8b5ddf@mail.gmail.com>
Message-ID: <200607031412.22874.fdrake@acm.org>

On Monday 03 July 2006 14:07, Facundo Batista wrote:
 > I want to know, please, if this is useful in general, for me to post a
 > patch in SF.

It seems like something that should be easy, and lots of people need to 
consider this for applications.


  -Fred

-- 
Fred L. Drake, Jr.   <fdrake at acm.org>

From martin at v.loewis.de  Mon Jul  3 22:27:56 2006
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Mon, 03 Jul 2006 22:27:56 +0200
Subject: [Python-Dev] Time-out in URL Open
In-Reply-To: <e04bdf310607031107ld304d14i7db52e555f8b5ddf@mail.gmail.com>
References: <e04bdf310607031107ld304d14i7db52e555f8b5ddf@mail.gmail.com>
Message-ID: <44A97DCC.2040202@v.loewis.de>

Facundo Batista wrote:
>>>> urllib2.urlopen("http://no.host.org", timeout=2)
> 
> This is actually not possible, but I'll make it work.
> 
> I want to know, please, if this is useful in general, for me to post a
> patch in SF.

While it might be useful, it can only be added to Python 2.6 now.
So take your time with that patch.

Regards,
Martin

From guido at python.org  Mon Jul  3 23:00:38 2006
From: guido at python.org (Guido van Rossum)
Date: Mon, 3 Jul 2006 23:00:38 +0200
Subject: [Python-Dev] Lexical scoping in Python 3k
In-Reply-To: <006c01c69eb9$323f37a0$6402a8c0@arkdesktop>
References: <ca471dc20607030845p288f6f46xd6bdc45beb138f27@mail.gmail.com>
	<006c01c69eb9$323f37a0$6402a8c0@arkdesktop>
Message-ID: <ca471dc20607031400s80e08b3wdcc5eb195f1247eb@mail.gmail.com>

On 7/3/06, Andrew Koenig <ark at acm.org> wrote:
> > Much though the Algol 60 tickles my nostalgia (it was my first
> > programming language!) I don't think that it's a particularly strong
> > argument. I like to think that we have better ways these days.
>
> Even if so, that's not the point I was trying to make.  The point is that
> there is a programming technique that is widely used, works in many
> languages, and has been around for 45 years; and when you try to use it in
> Python, it fails.

That's true for lots of things that have been around for a long time.

Can you provide a better example? (The use of += is not particularly
relevant to the example; it could just as well have said "result =
result + i".)

> I believe that such failures, even if there are alternative ways of solving
> the problems that engender them, are barriers to learning that should be
> removed if it is possible to do so without substantial cost.

And that is of course the crucial question.

Probably the only proposal that has any chance of succeeding is to
extend the 'global' statement so that it also applies to variables in
intermediate outer scopes; or perhaps a new keyword (since "global" is
not a very good name for the extended semantics). We would have to
decide what this example would do:

  def outer():
    def inner1(x):
      global a
      a = x
    def inner2():
      return a
    return inner1, inner2
  f1, f2 = outer()
  g1, g2 = outer()
  f1(42)
  g1(0)
  print f2()    # Does it print 0 or 42 ???

In current Python this prints 0: there's only one (global) variable a,
and the call to g1(0) overrides the value that was stored by f1(42).
If global were changed to mean "nonlocal" what should it do? The
question the example poses is that a is not initialized except in
inner1() -- we somehow have to decide whether this is an error, or
whether it chooses some well-defined outer scope, with the choices
being the nearest enclosing scope or the outermost (truly global)
scope. We have one guideline: if there is exactly one outer scope that
defines a variable named a, we would like it to be referenced (by the
'global a') statement and the variable references governed by it
automatically. Also, of there's more than one such scope, we'd like it
to reference the innermost one. But this doesn't have a natural
extension to what should happen if there's no such scope!

Perhaps the best solution would be to make it an error if there wasn't
a visible variable named a in an outer scope. That would suit me fine
because I'd like to migrate towards more static analysis of variables
anyway. If that means equipping modues with traps for attempts to
store arbitrary variables into their namespaces, that's fine with me
(as long as there's some escape -- and of course instancs and classes
remain fully dynamic).

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From guido at python.org  Mon Jul  3 23:06:02 2006
From: guido at python.org (Guido van Rossum)
Date: Mon, 3 Jul 2006 23:06:02 +0200
Subject: [Python-Dev] Time-out in URL Open
In-Reply-To: <e04bdf310607031107ld304d14i7db52e555f8b5ddf@mail.gmail.com>
References: <e04bdf310607031107ld304d14i7db52e555f8b5ddf@mail.gmail.com>
Message-ID: <ca471dc20607031406q8ddaa9fhd7b0972b8d7a1412@mail.gmail.com>

To fake things like this, socket.setdefaulttimeout() was added, though
I don't know if it actually works. Have you tried that?

--Guido

On 7/3/06, Facundo Batista <facundobatista at gmail.com> wrote:
> I need a timeout in urlopen, just to be able to make:
>
> >>> urllib2.urlopen("http://no.host.org", timeout=2)
>
> This is actually not possible, but I'll make it work.
>
> I want to know, please, if this is useful in general, for me to post a
> patch in SF.
>
> Regards,
>
> --
> .    Facundo
>
> Blog: http://www.taniquetil.com.ar/plog/
> PyAr: http://www.python.org/ar/
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org
>


-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From skip at pobox.com  Mon Jul  3 23:30:11 2006
From: skip at pobox.com (skip at pobox.com)
Date: Mon, 3 Jul 2006 16:30:11 -0500
Subject: [Python-Dev] Time-out in URL Open
In-Reply-To: <e04bdf310607031107ld304d14i7db52e555f8b5ddf@mail.gmail.com>
References: <e04bdf310607031107ld304d14i7db52e555f8b5ddf@mail.gmail.com>
Message-ID: <17577.35939.965788.902096@montanaro.dyndns.org>


    Facundo> I need a timeout in urlopen, just to be able to make:

    >>>> urllib2.urlopen("http://no.host.org", timeout=2)

    Facundo> This is actually not possible, but I'll make it work.

    Facundo> I want to know, please, if this is useful in general, for me to
    Facundo> post a patch in SF.

As others have posted, yes, it would be useful for 2.6.  However, you should
consider how that might be applied to the other Internet service modules
(ftplib, telnetlib, urllib, etc).

Skip

From skip at pobox.com  Mon Jul  3 23:31:12 2006
From: skip at pobox.com (skip at pobox.com)
Date: Mon, 3 Jul 2006 16:31:12 -0500
Subject: [Python-Dev] Time-out in URL Open
In-Reply-To: <ca471dc20607031406q8ddaa9fhd7b0972b8d7a1412@mail.gmail.com>
References: <e04bdf310607031107ld304d14i7db52e555f8b5ddf@mail.gmail.com>
	<ca471dc20607031406q8ddaa9fhd7b0972b8d7a1412@mail.gmail.com>
Message-ID: <17577.36000.58814.231162@montanaro.dyndns.org>


    Guido> To fake things like this, socket.setdefaulttimeout() was added,
    Guido> though I don't know if it actually works. Have you tried that?

I'm pretty sure it does, but is a rather blunt instrument for the task, as
it affects all socket connections the app might make.

Skip

From skip at pobox.com  Mon Jul  3 23:31:12 2006
From: skip at pobox.com (skip at pobox.com)
Date: Mon, 3 Jul 2006 16:31:12 -0500
Subject: [Python-Dev] Time-out in URL Open
In-Reply-To: <ca471dc20607031406q8ddaa9fhd7b0972b8d7a1412@mail.gmail.com>
References: <e04bdf310607031107ld304d14i7db52e555f8b5ddf@mail.gmail.com>
	<ca471dc20607031406q8ddaa9fhd7b0972b8d7a1412@mail.gmail.com>
Message-ID: <17577.36000.58814.231162@montanaro.dyndns.org>


    Guido> To fake things like this, socket.setdefaulttimeout() was added,
    Guido> though I don't know if it actually works. Have you tried that?

I'm pretty sure it does, but is a rather blunt instrument for the task, as
it affects all socket connections the app might make.

Skip

From jjl at pobox.com  Mon Jul  3 23:57:36 2006
From: jjl at pobox.com (John J Lee)
Date: Mon, 3 Jul 2006 21:57:36 +0000 (UTC)
Subject: [Python-Dev] Time-out in URL Open
In-Reply-To: <ca471dc20607031406q8ddaa9fhd7b0972b8d7a1412@mail.gmail.com>
References: <e04bdf310607031107ld304d14i7db52e555f8b5ddf@mail.gmail.com>
	<ca471dc20607031406q8ddaa9fhd7b0972b8d7a1412@mail.gmail.com>
Message-ID: <Pine.LNX.4.64.0607032154100.8487@localhost>

On Mon, 3 Jul 2006, Guido van Rossum wrote:

> To fake things like this, socket.setdefaulttimeout() was added, though
> I don't know if it actually works. Have you tried that?
[...]

It works.  I think there's some issue with SSL, though (can't seem to find 
the issue now).

Of course, feeding through the timeout to the individual protocol modules 
would be a good thing.


John


From billchi at microsoft.com  Tue Jul  4 01:17:36 2006
From: billchi at microsoft.com (Bill Chiles)
Date: Mon, 3 Jul 2006 16:17:36 -0700
Subject: [Python-Dev] 2.5 and beyond
In-Reply-To: <06Jun30.183119pdt."58641"@synergy1.parc.xerox.com>
Message-ID: <9B706ACDA57B23438AAE4822369672AF04062AD1@RED-MSG-10.redmond.corp.microsoft.com>

For Common Lispers and probably Schemers, Python has some surprising
semantics around scope and lifetime extent of variables.  Three that
leap out at me are:
 * function parameters with default values are NOT new bindings for each
invocation, so a
   default value of [] changes if you destructively modify this list
object in the function
 * loop variables are NOT distinct lexical variables.  The binding gloms
on to a variable in the
   function's scope, both changing that lexical binding and not creating
a new one for the
   loop (that goes away when the loop's scope ends)
 * loop variables are NOT distinct bindings per iteration, leading to
the surprising results
   below

Bill

-----Original Message-----
From: python-dev-bounces+billchi=microsoft.com at python.org
[mailto:python-dev-bounces+billchi=microsoft.com at python.org] On Behalf
Of Bill Janssen
Sent: Friday, June 30, 2006 6:31 PM
To: Giovanni Bajo
Cc: Phillip J. Eby; Ka-Ping Yee; Guido van Rossum; Tim Peters;
python-dev at python.org
Subject: Re: [Python-Dev] 2.5 and beyond

> >>> a = []
> >>> for i in range(10):
> ...     a.append(lambda: i)
> ...
> >>> print [x() for x in a]
> [9, 9, 9, 9, 9, 9, 9, 9, 9, 9]

Isn't this exactly what you'd expect?  Maybe I've been writing Python
for too long... :-).

Bill
_______________________________________________
Python-Dev mailing list
Python-Dev at python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/billchi%40microsoft.co
m

From jcarlson at uci.edu  Tue Jul  4 02:20:28 2006
From: jcarlson at uci.edu (Josiah Carlson)
Date: Mon, 03 Jul 2006 17:20:28 -0700
Subject: [Python-Dev] 2.5 and beyond
In-Reply-To: <9B706ACDA57B23438AAE4822369672AF04062AD1@RED-MSG-10.redmond.corp.microsoft.com>
References: <06Jun30.183119pdt."58641"@synergy1.parc.xerox.com>
	<9B706ACDA57B23438AAE4822369672AF04062AD1@RED-MSG-10.redmond.corp.microsoft.com>
Message-ID: <20060703171546.1108.JCARLSON@uci.edu>


"Bill Chiles" <billchi at microsoft.com> wrote:
> 
> For Common Lispers and probably Schemers, Python has some surprising
> semantics around scope and lifetime extent of variables.  Three that
> leap out at me are:

One thing to remember is that Python is not Scheme/Lisp.  It borrows
some ideas from Scheme/Lisp, but that borrowing does not necessitate a
it also use a completely equivalent scoping mechanism.  From what I have
been hearing about Python 2.6, and 3.0, the three "surprises" you
describe are not going to be "fixed" (with respect to expected
Scheme/Lisp semantics).

Feel free to argue as to why they should be "fixed" in Py3k (unless
Guido says, "you're dreaming"), but please do so in the py3k list.

 - Josiah


>  * function parameters with default values are NOT new bindings for each
> invocation, so a
>    default value of [] changes if you destructively modify this list
> object in the function
>  * loop variables are NOT distinct lexical variables.  The binding gloms
> on to a variable in the
>    function's scope, both changing that lexical binding and not creating
> a new one for the
>    loop (that goes away when the loop's scope ends)
>  * loop variables are NOT distinct bindings per iteration, leading to
> the surprising results
>    below


From guido at python.org  Tue Jul  4 06:56:41 2006
From: guido at python.org (Guido van Rossum)
Date: Tue, 4 Jul 2006 06:56:41 +0200
Subject: [Python-Dev] 2.5 and beyond
In-Reply-To: <20060703171546.1108.JCARLSON@uci.edu>
References: <9B706ACDA57B23438AAE4822369672AF04062AD1@RED-MSG-10.redmond.corp.microsoft.com>
	<20060703171546.1108.JCARLSON@uci.edu>
Message-ID: <ca471dc20607032156t4d281194u3cd8681e59442426@mail.gmail.com>

On 7/4/06, Josiah Carlson <jcarlson at uci.edu> wrote:
> One thing to remember is that Python is not Scheme/Lisp.  It borrows
> some ideas from Scheme/Lisp,

I can say it stronger. Any resemblance between Python and Scheme or
Lisp is purely a coincidence. Neither language is in Python's
ancestry, at least not explicitly; I'd never used or tried to learn
Scheme when I started Python (still haven't) and my Lisp experience
was limited to copying Emacs startup code from friends (still is).

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From g.brandl at gmx.net  Tue Jul  4 09:03:44 2006
From: g.brandl at gmx.net (Georg Brandl)
Date: Tue, 04 Jul 2006 09:03:44 +0200
Subject: [Python-Dev] Time-out in URL Open
In-Reply-To: <17577.35939.965788.902096@montanaro.dyndns.org>
References: <e04bdf310607031107ld304d14i7db52e555f8b5ddf@mail.gmail.com>
	<17577.35939.965788.902096@montanaro.dyndns.org>
Message-ID: <e8d3sg$pkn$1@sea.gmane.org>

skip at pobox.com wrote:
>     Facundo> I need a timeout in urlopen, just to be able to make:
> 
>     >>>> urllib2.urlopen("http://no.host.org", timeout=2)
> 
>     Facundo> This is actually not possible, but I'll make it work.
> 
>     Facundo> I want to know, please, if this is useful in general, for me to
>     Facundo> post a patch in SF.
> 
> As others have posted, yes, it would be useful for 2.6.  However, you should
> consider how that might be applied to the other Internet service modules
> (ftplib, telnetlib, urllib, etc).

There was one patch that did this: http://python.org/sf/723312.

Georg


From talin at acm.org  Tue Jul  4 09:59:50 2006
From: talin at acm.org (Talin)
Date: Tue, 04 Jul 2006 00:59:50 -0700
Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
Message-ID: <44AA1FF6.1050501@acm.org>

This is sort of a re-do of an earlier proposal which seems to have 
gotten lost in the shuffle of the larger debate.

I propose to create a new type of scoping rule, which I will call 
"explicit" lexical scoping, that will co-exist with the current 
"implicit" scoping rule that exists in Python today.


Definitions:

Implicit scoping is what we have now - a variable is defined within a 
scope implicitly by assignment. More specifically, when a name is 
assigned, the name is defined at the innermost function-level scope from 
where the assignment took place.

Explicit scoping is where the programmer explicitly specifies which 
scope the variable should be defined in. Unlike implicit scoping, 
assignments to a named variable do not automatically redefine that 
variable within the current scope.


Syntax:

Borrowing from Perl, the keyword 'my' is used to declare an explicitly 
scoped variable:

    def f1():
       my x = 1
       def f2():
          x = 2   # Does not create a new x

In the above example, the statement 'my x = 1' declares that the scope 
of the variable 'x' is the outer function f1. Any assignment to x will 
modify the existing x, rather than creating a new definition.

Note that the 'my' prefix can be combined with an assignment operation. 
It is anticipated that the 'my' prefix will be used quite frequently 
(and encouraged), so it makes sense to cut down on the number of 
statements by combining declaration and assignment.

Explicitly scoped variables can also be declared at the module level:

    my x = 1
    def f1():
       x = 2   # Modifies the global X

Declaring a module-level variable with an explicit scope eliminates the 
need for a 'global' statement for that variable.


Nested Scopes:

Each occurance of the keyword 'my' creates a new scope which hides any 
outer definitions of the name. So for example:

    my x = 1
    def f1():
       my x = 2  # This is a different 'x' than the global
       def f2():
          x = 3  # This is the 'x' defined within f1()


Interaction between explicit scoping and globals:

The 'global' statement, when used with explicitly scoped variables, 
means exactly the same as it does with implicitly scoped variables: It 
allows access to the outermost scope, overriding any intermediate 
definitions in surrounding scopes:

    x = 1
    def f1():
       my x = 2
       def f2():
          global x
          x = 3     # This is the module-level 'x'


Explicit scoping and code block structure:

Implicitly scoped variables are always defined at the nearest enclosing 
function scope, even if they are created within a code block.

It might be worth considering allowing explicitly scoped variables to be 
defined within other scopes. For example, we might choose to allow 
explicit scope declarations to be limited to the current suite:

    def f1():
       for x in range(0,10):
          my y = x*x          # A new definition of y for each iteration

Note that this is a speculation only, and not a core part of the 
proposal (so please don't reject the proposal on this one point.)


Formal definition:

When a value is assigned to a local variable name, the rules for 
determining which scope the variable will be defined in are as follows:

    1) Starting with the current (innermost) scope, examine all of the 
currently active scopes:
       1a) If the current scope contains a 'global' statement for the 
given name, then set the result scope to the outermost (module-level) scope.
       1b) If the current scope contains a 'my' statement for the given 
name, then set the result scope to the scope in which the 'my' statement 
occurred.
    2) Otherwise, continue until we run out of scopes. If neither a 
'global' or 'my' declaration was discovered, then use the innermost 
scope as the result scope.


How is this different from 'outer'?

The explicit scope proposal requires that the scope be specified at the 
place where the variable is *defined* as opposed to where it is *used*. 
This definition is inherited by all inner scopes.

This allows a finer degree of control, for less typing, than the 'outer' 
proposal. With explicit scoping, there is no confusion as to which scope 
is being considered; And explicit scoping allows a single declaration of 
a variable to be shared by many different inner scopes, which would 
otherwise require a separate 'outer' statement for each one.


Explicit scoping and static analysis:

It should be easier to do static analysis of code with explicit scoping, 
since you always know what scope a variable is defined in (as opposed to 
implicit scoping, where a variable may switch from global to local as a 
result of an assignment.)

Note that this implies that the creation of the scope does not occur at 
the time of the assignment, but rather at the time the function is 
entered. Thus:

    x = 1
    def f1():
       print x   # Error, unassigned value
       my x = 2

In the above example, even though the 'my' statement occurs after the 
print, the scope created by the 'my' statement is in effect for the 
entire function, although the actual *assignment* takes place after the 
print. The reason for this is that the scope creation is actually done 
by the compiler.

-- Talin

From python-dev at zesty.ca  Tue Jul  4 12:32:00 2006
From: python-dev at zesty.ca (Ka-Ping Yee)
Date: Tue, 4 Jul 2006 05:32:00 -0500 (CDT)
Subject: [Python-Dev] doc for new restricted execution design for Python
In-Reply-To: <bbaeab100606211733x3dd58226g15e2919e8e2cdd7e@mail.gmail.com>
References: <bbaeab100606211733x3dd58226g15e2919e8e2cdd7e@mail.gmail.com>
Message-ID: <Pine.LNX.4.58.0607040428210.17937@server1.LFW.org>

Hi Brett,

Here are some comments on the description of the restricted execution
model that you posted.

> When referring to the state of an interpreter, it is either "trusted" or
> "untrusted".  A trusted interpreter has no restrictions imposed upon any
> resource.  An untrusted interpreter has at least one, possibly more, resource
> with a restriction placed upon it.

In response to Guido's comment about confusing the words "trusted" and
"untrusted", how about "empowered" and "restricted"?

> When the Interpreter Is Embedded
> ================================
>
> Single Untrusted Interpreter
> ----------------------------
>
> This use case is when an application embeds the interpreter and never has more
> than one interpreter running.
>
> The main security issue to watch out for is not having default abilities be
> provided to the interpreter by accident.

I'd rather rephrase this in the opposite direction.  The onus shouldn't
be on the application to hunt down each possible dangerous authority and
deactivate them all one by one.  The main security issue is to let the
application choose which abilities it wants the restricted interpreter
to have, and then ensure that the restricted interpreter gets only those
abilities.

> Multiple Untrusted Interpreters
> -------------------------------
>
> When multiple interpreters, all untrusted at varying levels, need to be
> running within a single application.  This is the key use case that this
> proposed design is targetted for.
>
> On top of the security issues from a single untrusted interpreter,
> there is one additional worry.  Resources cannot end up being leaked
> into other interpreters where they are given escalated rights.

What is your model here for communication between interpreters?  If two
interpreters can communicate, any attempt to "prevent leakage" of
resources is meaningless.  When you say "leaked into other interpreters"
are you talking about a Python object leaking or something else at a
lower level?

Suppose for example that the application wants to embed two interpreters,
P and Q, and that the application wants P to be able to write files but
Q to be restricted against writing files.  When you say "leaked" above,
that suggests to me that you want to prevent something like

    # code running in P
    import spam
    f = open('/home/doofus/.ssh/authorized_keys', 'a')
    spam.f = f

    # code running in Q
    import spam
    spam.f.write('blargh')

The above example supposes that P and Q can communicate through a
shared module, spam, where they can pass Python objects.

But notice that even if you prevent them from passing Python objects
like open files, any form of communication is sufficient to leak
resources:

    # code running in P
    def add_key(key):
        f = open('/home/doofus/.ssh/authorized_keys', 'a')
        f.write(key + '\n')
        f.close()

    import socket
    s = socket.socket()
    s.bind(('', 6666))
    s.listen(1)
    ns, addr = s.accept()
    add_key(ns.recv(100))


    # code running in Q
    import webbrowser
    webbrowser.open('http://localhost:6666/zebra')

As long as P can listen for instructions from Q, it can give Q
the power to write to the filesystem.

> Filesystem
> ===================
>
> The most obvious facet of a filesystem to protect is reading from it.
> One does not want what is stored in ``/etc/passwd`` to get out.  And
> one also does not want writing to the disk unless explicitly allowed
> for basically the same reason; if someone can write ``/etc/passwd``
> then they can set the password for the root account.

There's a big difference between modifying (or erasing) an existing file
and writing a new file (e.g. for temporary storage).  If i give you a
little filesystem of your own to play in, and it starts out empty, you
can put whatever you want in it without violating my secrecy or the
integrity of my files.

I think you should be talking about this in terms of specifically
what abilities you want to be able to allow, based on examples of
real-life applications.

> Physical Resources
> ===================
>
> Memory should be protected.  It is a limited resource on the system
> that can have an impact on other running programs if it is exhausted.
> Being able to restrict the use of memory would help alleviate issues
> from denial-of-service (DoS) attacks.

> Networking
> ===================
>
> Networking is somewhat like the filesystem in terms of wanting similar
> protections.  You do not want to let untrusted code make tons of socket
> connections or accept them to do possibly nefarious things (e.g., acting
> as a zombie).
>
> You also want to prevent finding out information about the network you are
> connected to.  This includes doing DNS resolution since that allows one
> to find out what addresses your intranet has or what subnets you use.

Again, it's risky to describe only individual cases of things to
prevent.  What networking abilities are safe or necessary for the
kinds of applications you have in mind?  Start from nothing and
work up from there.

> Interpreter
> ===================
>
> One must make sure that the interpreter is not harmed in any way.
> There are several ways to possibly do this.  One is generating
> hostile bytecode.  Another is some buffer overflow.  In general any
> ability to crash the interpreter is unacceptable.

This is hard for me to understand.  What exactly do you trust and
not trust?  It seems to me that crashing an interpreter is only a
problem if a single interpreter is running both trusted and untrusted
code -- then if the untrusted code crashes the interpreter, the
trusted code suffers.

But there doesn't seem to be any such thing in your model.  Each
interpreter is either trusted or untrusted.  If the interpreter is
trusted, and the code running in it causes it to crash, i assume
you would consider that to be the code's "own fault", right?
And if the interpreter is untrusted, and the code running in it
causes it to crash, then the code has only harmed itself.

It seems to me that we need only be concerned about crashing when
the crash of an embedded interpreter will bring down its host
application, or there are multiple interpreters embedded at once
and one interpreter causes another interpreter to crash.

> Resource Hiding
> =============================
[...]
> This can be viewed as a passive system for security.
[...]
> Resource Crippling
> =============================
> Another approach to security is to provide constant, proactive security
> checking of rights to use a resource.

I think you have this backwards.  Resource hiding is proactive:
before untrusted code has a chance to abuse anything, you decide
what you want to allow it to do.  It defaults to no access, and
only gets access to resources you have proactively decided to provide.

Resource crippling is the opposite: it begins by giving carte blanche
to the untrusted code, then you run around trying to plug holes
by stopping everything you don't want.  This is a lot more work,
and it is also much more dangerous.  If you forget to plug even
one hole, you're hosed.

Back to what you wrote about resource hiding:

> This can be viewed as a passive system for security.  Once a resource
> has been given to code there are no more checks to make sure the
> security model is being violated.

This last sentence doesn't make any sense.  If you decided to give
the resource, how is using the resource a violation?  Either you
want to enable the resource or you don't.  If you want to enable
it, give it; if you don't, don't give it.  As a criticism of the
resource hiding approach, it's a red herring -- there's no way
to interpret this sentence that doesn't make it also an
unfalsifiable criticism of any possible security model.

> The most common implementation of resource hiding is capabilities.
> In this type of system a resource's reference acts as a ticket that
> represents the right to use the resource.  Once code has a reference
> it is considered to have full use of that resource it represents and
> no further security checks are performed.

Same thing.  What "further security checks" are we worried about?
Woult it check to see whether we've authorized the interpreter to
have access to the resource ... which we already know to be true?

> To allow customizable restrictions one can pass references to wrappers of
> resources.  This allows one to provide custom security to resources instead of
> requiring an all-or-nothing approach.

The ability to customize security restrictions is an important
advantage of the resource hiding approach, since resource crippling
requires that the architect of the security model anticipate every
possible security restriction that future programmers might need.

Using resource crippling is analogous to removing "def" from the
language and requiring Python programmers to only use functions
that are provided in the built-in modules instead of writing their
own functions.

> To use an analogy, imagine you are providing security for your home.
> With capabilities, security came from not having any way to know
> where your house is without being told where it was; a reference
> to its location.  You might be able to ask a guard (e.g., Java's
> ClassLoader) for a map, but if they refuse there is no way for you
> to guess its location without being told.  But once you knew where
> it was, you had complete use of the house.

This analogy is only fair if you compare it to the same analogy for
the resource crippling approach.  Resource crippling doesn't get you
any finer-grained control either!  The comparison story is:

    With resource crippling, security comes from having a guard
    at the door to your house.  When a Python interpreter comes
    up to the door, the guard checks to see if the interpreter
    has permission to enter the house, and if it does, then it
    gets complete use of the house.

Why is the granularity of control described as the whole house
in the resource-hiding story, but as each door in the house in
the resource-crippling story?

> And that complete access is an issue with a capability system.
> If someone played a little loose with a reference for a resource
> then you run the risk of it getting out.

Could you be more specific about what you mean by "it getting out"?

If you mean getting from a trusted interpreter to an untrusted
interpreter -- then how is a resource going to travel between
interpreters?

Or if not, then are you thinking of a situation in which one
piece of code is trusted with the resource, but another piece of
code is not, and both are running in the same interpreter?



-- ?!ng

From fuzzyman at voidspace.org.uk  Tue Jul  4 13:08:21 2006
From: fuzzyman at voidspace.org.uk (Fuzzyman)
Date: Tue, 04 Jul 2006 12:08:21 +0100
Subject: [Python-Dev] doc for new restricted execution design for Python
In-Reply-To: <Pine.LNX.4.58.0607040428210.17937@server1.LFW.org>
References: <bbaeab100606211733x3dd58226g15e2919e8e2cdd7e@mail.gmail.com>
	<Pine.LNX.4.58.0607040428210.17937@server1.LFW.org>
Message-ID: <44AA4C25.3020706@voidspace.org.uk>

Ka-Ping Yee wrote:

>Hi Brett,
>
>Here are some comments on the description of the restricted execution
>model that you posted.
>
> [snip...]
>
>>Filesystem
>>===================
>>
>>The most obvious facet of a filesystem to protect is reading from it.
>>One does not want what is stored in ``/etc/passwd`` to get out.  And
>>one also does not want writing to the disk unless explicitly allowed
>>for basically the same reason; if someone can write ``/etc/passwd``
>>then they can set the password for the root account.
>>    
>>
>
>There's a big difference between modifying (or erasing) an existing file
>and writing a new file (e.g. for temporary storage).  If i give you a
>little filesystem of your own to play in, and it starts out empty, you
>can put whatever you want in it without violating my secrecy or the
>integrity of my files.
>
>I think you should be talking about this in terms of specifically
>what abilities you want to be able to allow, based on examples of
>real-life applications.
>  
>
>  
>
As an adjunct to this, one of the barriers to Javascript applications is
the lack of client-side data persistence.

This makes (amongst other things) offline, or entirely clientside,
applications very difficult. All sorts of novel ways round this have
been found [
http://codinginparadise.org/weblog/2006/04/now-in-browser-near-you-offline-access.html
].

If a 'standard' interpreter running in the browser had sandboxed access
to the filesystem, this would be great.

Of course Mozilla would probably disable it by default, and only provide
horrific means for users to re-enable it [
http://www.mozilla.org/editor/midasdemo/securityprefs.html ].

All the best,

Michael Foord
http://www.voidspace.org.uk/python/index.shtml

>
>
>-- ?!ng
>_______________________________________________
>Python-Dev mailing list
>Python-Dev at python.org
>http://mail.python.org/mailman/listinfo/python-dev
>Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk
>
>  
>


From theller at python.net  Tue Jul  4 13:35:13 2006
From: theller at python.net (Thomas Heller)
Date: Tue, 04 Jul 2006 13:35:13 +0200
Subject: [Python-Dev] 2.5 and beyond
In-Reply-To: <ee2a432c0606300005g256b3391na6e684123d3e9e93@mail.gmail.com>
References: <ee2a432c0606300005g256b3391na6e684123d3e9e93@mail.gmail.com>
Message-ID: <e8djpk$cr6$1@sea.gmane.org>

Neal Norwitz schrieb:
> I'm glad to see Anthony ratcheting down.  At this point, we need to be
> fixing bugs and improving doc.  Maybe Anthony and I should have a
> contest to see who can revert the most changes. :-)
> 
Neal (and/or Anthony),

I would like to ask about the possibility to add some improvements to ctypes
in Python 2.5, although the feature freeze is now in effect.  Hopefully former
third-party libraries can have the freeze relaxed somewhat;-).

I intend to do these changes, the first is a small and trivial one, but allows
a lot of flexibility:

- Remove the restriction that the argtypes attribute of foreign functions must
  be ctypes types.  Instead they are only required to implement a .from_param
  class method.  The advantage is that custom objects can be used as function
  parameters.  One usecase is to allow numpy arrays as function parameters
  without any conversion - this change at least allows to code this in Python.
  The patch is attached as from_param.patch.

The second one is more involved, and not yet complete.  I can post the patch
or a link to it for review when it is implemented completely:

- Implement the __array_struct__ attribute as describes by the numpy pep at
  http://numpy.scipy.org/array_interface.html.  The properties needed to implement
  the __array_struct__ attribute could be calculated from a given ctypes array type,
  however, it would be more efficient to calculate them at type creation time.
  This requires the StgDSictObject that holds information about the ctypes type
  to grow a few fields: 'int nd' - contains the number of dimensions,
  'char typekind' - a struct-like character for the item type, and
  'Py_intptr_t *shape' - an array of size 'nd' containing shape information.

Thanks for investigating this,
Thomas
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: from_param.patch
Url: http://mail.python.org/pipermail/python-dev/attachments/20060704/3c0500e4/attachment.pot 

From ncoghlan at iinet.net.au  Tue Jul  4 14:32:38 2006
From: ncoghlan at iinet.net.au (Nick Coghlan)
Date: Tue, 04 Jul 2006 22:32:38 +1000
Subject: [Python-Dev] Proposed beta 2 changes (Q for Anthony/Neal)
Message-ID: <44AA5FE6.2030801@iinet.net.au>

I've got a couple of changes ready to go for beta 2, but need a go ahead from 
one of the release managers before committing either of them:

1. Finishing the __module_name__ workaround to allow relative imports from the 
main module when using -m.

   I'd really like to finish this, because having PEP 328 and 338 not playing 
well together is a wart that's quite visible to end users. I'd rather not have 
people's first encounter with the features provided by either PEP involve 
discovering that they're "broken".

   The patch to fix this also takes care of adding a couple of paragraphs to 
the tutorial about explicit relative imports (which aren't currently written 
up in the main documentation). The patch is attached directly to the beta 1 
bug report about the problem [1].

   (Guido gave a +1 to the concept, but explicitly deferred to Anthony and 
Neal as to whether or not the fix should go in for beta 2)

2. Adding an 'ignore' filter for ImportWarning at the end of warnings.py

   This is a safe and easy fix to silence import warning spam for people that 
don't want it. I don't believe there are any Pending Deprecation Warnings at 
the moment, so -Wd at the command line would be sufficient to enable 
ImportWarning for people that want to see it.
   Adding the line "warnings.simplefilter('default', ImportWarning)" to 
sitecustomize.py would be sufficient for organisations to turn the warning on 
across the board if they so chose.
   The reason I haven't checked this in directly is that there's no point if 
Anthony and/or Neal intend to accept one of the patches that tries to make the 
import machinery more intelligent about missing __init__.py files.

Cheers,
Nick.

[1] http://www.python.org/sf/1510172

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
             http://www.boredomandlaziness.org

From rasky at develer.com  Tue Jul  4 14:43:40 2006
From: rasky at develer.com (Giovanni Bajo)
Date: Tue, 4 Jul 2006 14:43:40 +0200
Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
References: <44AA1FF6.1050501@acm.org>
Message-ID: <010601c69f67$7a535f90$d503030a@trilan>

Talin wrote:

> This is sort of a re-do of an earlier proposal which seems to have
> gotten lost in the shuffle of the larger debate.
>
> I propose to create a new type of scoping rule, which I will call
> "explicit" lexical scoping, that will co-exist with the current
> "implicit" scoping rule that exists in Python today.

Interesting. What if for-loops implicitally used "my" on the iteration
variable? That would solve the binding problem we were discussing and make
lambdas "Do The Right Thing"(TM) when used in loops.
-- 
Giovanni Bajo


From ark at acm.org  Tue Jul  4 16:16:59 2006
From: ark at acm.org (Andrew Koenig)
Date: Tue, 4 Jul 2006 10:16:59 -0400
Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
In-Reply-To: <44AA1FF6.1050501@acm.org>
Message-ID: <001901c69f74$86a28b60$6402a8c0@arkdesktop>

> Borrowing from Perl, the keyword 'my' is used to declare an explicitly
> scoped variable:
> 
>     def f1():
>        my x = 1
>        def f2():
>           x = 2   # Does not create a new x
> 
> In the above example, the statement 'my x = 1' declares that the scope
> of the variable 'x' is the outer function f1. Any assignment to x will
> modify the existing x, rather than creating a new definition.

-1, for this reason:

	def f()
		x = 2		# Does this create a local variable?

Today, the answer is yes.  Under this proposal, you can't answer the
question without inspecting the entire context in which f is defined.

For that reason, I would much rather have the first assignment in a block
say explicitly whether it is intended to create a local variable:

	def f1():
	   x = 1
	   def f2():
	      global x
	      x = 2    # Does not create a new x

This might even be abbreviated:

	def f1():
	   x = 1
	   def f2():
	      global x = 2   # Equivalent to the last example above




From guido at python.org  Tue Jul  4 16:28:30 2006
From: guido at python.org (Guido van Rossum)
Date: Tue, 4 Jul 2006 16:28:30 +0200
Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
In-Reply-To: <44AA1FF6.1050501@acm.org>
References: <44AA1FF6.1050501@acm.org>
Message-ID: <ca471dc20607040728y1035e27ej3d6d589eaed00080@mail.gmail.com>

Please move this to the python-3000 list.

Also please explain what problem you are solving before proposing a solution.

I note that we are seeing quite a flurry of language change proposals.
I have to recommend restraint; I *don't* want to turn the entire
language upside down. That's not a comment on this particular
proposal, but on the issue of too many proposals. From actual users of
the language I get more complaints about the breakneck speed of
Python's evolution than about the brokenness of the current language.

--Guido

On 7/4/06, Talin <talin at acm.org> wrote:
> This is sort of a re-do of an earlier proposal which seems to have
> gotten lost in the shuffle of the larger debate.
>
> I propose to create a new type of scoping rule, which I will call
> "explicit" lexical scoping, that will co-exist with the current
> "implicit" scoping rule that exists in Python today.
>
>
> Definitions:
>
> Implicit scoping is what we have now - a variable is defined within a
> scope implicitly by assignment. More specifically, when a name is
> assigned, the name is defined at the innermost function-level scope from
> where the assignment took place.
>
> Explicit scoping is where the programmer explicitly specifies which
> scope the variable should be defined in. Unlike implicit scoping,
> assignments to a named variable do not automatically redefine that
> variable within the current scope.
>
>
> Syntax:
>
> Borrowing from Perl, the keyword 'my' is used to declare an explicitly
> scoped variable:
>
>     def f1():
>        my x = 1
>        def f2():
>           x = 2   # Does not create a new x
>
> In the above example, the statement 'my x = 1' declares that the scope
> of the variable 'x' is the outer function f1. Any assignment to x will
> modify the existing x, rather than creating a new definition.
>
> Note that the 'my' prefix can be combined with an assignment operation.
> It is anticipated that the 'my' prefix will be used quite frequently
> (and encouraged), so it makes sense to cut down on the number of
> statements by combining declaration and assignment.
>
> Explicitly scoped variables can also be declared at the module level:
>
>     my x = 1
>     def f1():
>        x = 2   # Modifies the global X
>
> Declaring a module-level variable with an explicit scope eliminates the
> need for a 'global' statement for that variable.
>
>
> Nested Scopes:
>
> Each occurance of the keyword 'my' creates a new scope which hides any
> outer definitions of the name. So for example:
>
>     my x = 1
>     def f1():
>        my x = 2  # This is a different 'x' than the global
>        def f2():
>           x = 3  # This is the 'x' defined within f1()
>
>
> Interaction between explicit scoping and globals:
>
> The 'global' statement, when used with explicitly scoped variables,
> means exactly the same as it does with implicitly scoped variables: It
> allows access to the outermost scope, overriding any intermediate
> definitions in surrounding scopes:
>
>     x = 1
>     def f1():
>        my x = 2
>        def f2():
>           global x
>           x = 3     # This is the module-level 'x'
>
>
> Explicit scoping and code block structure:
>
> Implicitly scoped variables are always defined at the nearest enclosing
> function scope, even if they are created within a code block.
>
> It might be worth considering allowing explicitly scoped variables to be
> defined within other scopes. For example, we might choose to allow
> explicit scope declarations to be limited to the current suite:
>
>     def f1():
>        for x in range(0,10):
>           my y = x*x          # A new definition of y for each iteration
>
> Note that this is a speculation only, and not a core part of the
> proposal (so please don't reject the proposal on this one point.)
>
>
> Formal definition:
>
> When a value is assigned to a local variable name, the rules for
> determining which scope the variable will be defined in are as follows:
>
>     1) Starting with the current (innermost) scope, examine all of the
> currently active scopes:
>        1a) If the current scope contains a 'global' statement for the
> given name, then set the result scope to the outermost (module-level) scope.
>        1b) If the current scope contains a 'my' statement for the given
> name, then set the result scope to the scope in which the 'my' statement
> occurred.
>     2) Otherwise, continue until we run out of scopes. If neither a
> 'global' or 'my' declaration was discovered, then use the innermost
> scope as the result scope.
>
>
> How is this different from 'outer'?
>
> The explicit scope proposal requires that the scope be specified at the
> place where the variable is *defined* as opposed to where it is *used*.
> This definition is inherited by all inner scopes.
>
> This allows a finer degree of control, for less typing, than the 'outer'
> proposal. With explicit scoping, there is no confusion as to which scope
> is being considered; And explicit scoping allows a single declaration of
> a variable to be shared by many different inner scopes, which would
> otherwise require a separate 'outer' statement for each one.
>
>
> Explicit scoping and static analysis:
>
> It should be easier to do static analysis of code with explicit scoping,
> since you always know what scope a variable is defined in (as opposed to
> implicit scoping, where a variable may switch from global to local as a
> result of an assignment.)
>
> Note that this implies that the creation of the scope does not occur at
> the time of the assignment, but rather at the time the function is
> entered. Thus:
>
>     x = 1
>     def f1():
>        print x   # Error, unassigned value
>        my x = 2
>
> In the above example, even though the 'my' statement occurs after the
> print, the scope created by the 'my' statement is in effect for the
> entire function, although the actual *assignment* takes place after the
> print. The reason for this is that the scope creation is actually done
> by the compiler.
>
> -- Talin
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org
>


-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From facundobatista at gmail.com  Tue Jul  4 17:43:35 2006
From: facundobatista at gmail.com (Facundo Batista)
Date: Tue, 4 Jul 2006 12:43:35 -0300
Subject: [Python-Dev] Time-out in URL Open
In-Reply-To: <ca471dc20607031406q8ddaa9fhd7b0972b8d7a1412@mail.gmail.com>
References: <e04bdf310607031107ld304d14i7db52e555f8b5ddf@mail.gmail.com>
	<ca471dc20607031406q8ddaa9fhd7b0972b8d7a1412@mail.gmail.com>
Message-ID: <e04bdf310607040843m33c36a34ka938685598a70f01@mail.gmail.com>

2006/7/3, Guido van Rossum <guido at python.org>:

> To fake things like this, socket.setdefaulttimeout() was added, though
> I don't know if it actually works. Have you tried that?

This affect all the sockets. And I hit the problem when servicing
information with a web service (TCPServer), and I need to timeout the
connection of the URLOpen *only*.

Regards,

-- 
.    Facundo

Blog: http://www.taniquetil.com.ar/plog/
PyAr: http://www.python.org/ar/

From fperez.net at gmail.com  Tue Jul  4 22:14:43 2006
From: fperez.net at gmail.com (Fernando Perez)
Date: Tue, 04 Jul 2006 14:14:43 -0600
Subject: [Python-Dev] 2.5 and beyond
References: <ee2a432c0606300005g256b3391na6e684123d3e9e93@mail.gmail.com>
	<e8djpk$cr6$1@sea.gmane.org>
Message-ID: <e8ei61$qn8$1@sea.gmane.org>

Thomas Heller wrote:

> I would like to ask about the possibility to add some improvements to
> ctypes
> in Python 2.5, although the feature freeze is now in effect.  Hopefully
> former third-party libraries can have the freeze relaxed somewhat;-).
> 
> I intend to do these changes, the first is a small and trivial one, but
> allows a lot of flexibility:

[...]

I'd just like to provide a bit of context for Thomas' request (disclaimer:
he did NOT ask me to write this, nor did anyone else).  I understand the
release managers' need to be strict with the freeze, but perhaps knowing
what's behind this particular change will help them make a more informed
decision.

Numpy (http://numpy.scipy.org/) is the new python array package for
numerical computing, which has been developed at enormous effort by Travis
Oliphant (with community help) over the last year, as a way to unify the
old Numeric package (written by Jim Hugunin, of Jython and IronPython fame)
and Numarray (written by the Hubble telescope team).  The effect of numpy
in the community, even in its current pre-1.0 form, has been tremendous. 
There is a real, pressing need in the scientific world for open source and
technically superior replacements to Matlab and IDL, the propietary 800-lb
gorillas of the field.  Many major research institutions across the world
are seriously looking at python as fulfilling this role, but the previous
situation of a divided library (Numeric/numarray) was keeping a lot of
people on the fence.

With Travis' effort and numpy maturing towards a 1.0 release right around
the time of python 2.5, a LOT of people have come out of the woodwork to
contribute code, ideas, documentation, etc.  There is a real sense that the
combination of python2.5 (with better 64-bit and __index__ support) and
numpy will provide a significant advancement for scientific computing with
modern, high-level tools.

In this particular community, the need to interface with low-level existing
libraries is probably much more common than in other fields.  There are
literally millions of lines of C/C++ code for scientific work which we have
to use efficiently, and this is an everyday need for us.  While there are a
number of tools for this (SWIG, Boost::Python, pyrex, scipy.weave,...),
very recently people have discovered how useful ctypes can be for this
task.  One of the Google SoC projects (http://2006.planet-soc.com/blog/140)
started trying to wrap libsvm with SWIG and a week of frustrated efforts
led nowhere.  Albert then discovered ctypes and in a few hours was up and
running.  This has generated a lot of interest in the numpy crowd for
ctypes, and people would really, really like to see python2.5 come 'out of
the box' with as solid a support as possible from ctypes for numpy array
interfacing.

Ultimately the decision is up to the release team, I know that.  But at
least with this info, I hope you can understand:

1. why this is important to this community

2. why the timing isn't ideal: it is only /very/ recently that the numpy
team 'discovered' how much ctypes could truly help with a necessary (and
often very unpleasant) task in the numerical/python world.


Thanks for reading,


f


From nnorwitz at gmail.com  Tue Jul  4 23:21:55 2006
From: nnorwitz at gmail.com (Neal Norwitz)
Date: Tue, 4 Jul 2006 14:21:55 -0700
Subject: [Python-Dev] zlib module build failure on Mac OSX 10.4.7
In-Reply-To: <0006F35E-FD5F-4BAD-A608-642039BBE7E8@mac.com>
References: <17573.60593.759293.549583@montanaro.dyndns.org>
	<130298B1-7336-480A-A90F-97B68327BE4A@mac.com>
	<17574.43390.560231.425494@montanaro.dyndns.org>
	<315DECAC-D582-4692-A0CA-13D3C7BF992C@mac.com>
	<44A6C314.2000709@v.loewis.de>
	<0006F35E-FD5F-4BAD-A608-642039BBE7E8@mac.com>
Message-ID: <ee2a432c0607041421y37438adboc658313e85114e52@mail.gmail.com>

Ronald, Bob,

I know Skip found and fixed his problem, however, is this problem
likely to affect other users?  Is there anything we can do to help
alleviate/diagnose this problem?

n
--

On 7/1/06, Ronald Oussoren <ronaldoussoren at mac.com> wrote:
>
> On Jul 1, 2006, at 8:46 PM, Martin v. L?wis wrote:
>
> > Ronald Oussoren wrote:
> >> What  I don't understand yet is why your copy of libz doesn't have
> >> inflateCopy.
> >
> > What I don't understand is that configure does not detect that.
>
> You may be onto something there. Skip, do you have another copy of
> libz somewhere? Given the link line in your first message either in /
> usr/local/lib or /Users/skip/local/lib. And if you have, is that a
> static library (libz.a) instead of a dylib?
>
> As background to my question: the linker on OSX behaves slightly
> different than the one on most other unix-y systems. It first
> searches the entire linker path for shared libraries (dylibs) before
> looking for static libraries. I added a flag to the link flags for
> the zlib extension a while back that changes the search order into a
> more traditional one: look in every directory on the linker path for
> either a dylib or static library.  The new flag is  -Wl,-
> search_paths_first.
>
> If skip does indeed have libz somewhere else we'll either have to
> make a matching update to configure, or roll back my change. If the
> latter I'll have to tweak the build script for the binary installer
> for OSX because I want to link that using a static copy of libz for
> binary compatibility with OSX 10.3.9.
>
> Ronald
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/nnorwitz%40gmail.com
>
>
>
>

From talin at acm.org  Tue Jul  4 23:39:44 2006
From: talin at acm.org (Talin)
Date: Tue, 04 Jul 2006 14:39:44 -0700
Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
In-Reply-To: <ca471dc20607040728y1035e27ej3d6d589eaed00080@mail.gmail.com>
References: <44AA1FF6.1050501@acm.org>
	<ca471dc20607040728y1035e27ej3d6d589eaed00080@mail.gmail.com>
Message-ID: <44AAE020.5060204@acm.org>

Guido van Rossum wrote:
> Please move this to the python-3000 list.
> 
> Also please explain what problem you are solving before proposing a 
> solution.
> 
> I note that we are seeing quite a flurry of language change proposals.
> I have to recommend restraint; I *don't* want to turn the entire
> language upside down. That's not a comment on this particular
> proposal, but on the issue of too many proposals. From actual users of
> the language I get more complaints about the breakneck speed of
> Python's evolution than about the brokenness of the current language.
> 
> --Guido

Actually, the "problem" I am trying to solve is the debate on the 
mailing list. That is, I listen to what people are asking for, and what 
disagreements they have, and then I try to provide a solution that 
resolves the debate.

In this case, there was a lot of discussion about lexical scoping, and 
various people proposing solutions (such as redefining the behavior of 
'global') that I thought were (a) problematic, and (b) not a lot of bang 
for the buck (i.e. the disruption vs. utility tradeoff was poor IMHO.)

To be honest, I really have no stake in this proposal, and I don't 
intend to spend any time defending it other than to correct 
misperceptions - however, I offer it as a potential starting point for 
people who are interested in the whole lexical scoping issue. If someone 
feels that this proposal gives them what they want, then great - 
otherwise I'll drop it.

-- Talin

From guido at python.org  Tue Jul  4 23:58:04 2006
From: guido at python.org (Guido van Rossum)
Date: Tue, 4 Jul 2006 23:58:04 +0200
Subject: [Python-Dev] Time-out in URL Open
In-Reply-To: <e04bdf310607040843m33c36a34ka938685598a70f01@mail.gmail.com>
References: <e04bdf310607031107ld304d14i7db52e555f8b5ddf@mail.gmail.com>
	<ca471dc20607031406q8ddaa9fhd7b0972b8d7a1412@mail.gmail.com>
	<e04bdf310607040843m33c36a34ka938685598a70f01@mail.gmail.com>
Message-ID: <ca471dc20607041458w22ee86cey99eb9bbe38e44f7f@mail.gmail.com>

On 7/4/06, Facundo Batista <facundobatista at gmail.com> wrote:
> 2006/7/3, Guido van Rossum <guido at python.org>:
>
> > To fake things like this, socket.setdefaulttimeout() was added, though
> > I don't know if it actually works. Have you tried that?
>
> This affect all the sockets.

So, assuming your app is single-threaded, set the timeout, call
urlopen(), and reset the timeout to None.

> And I hit the problem when servicing
> information with a web service (TCPServer), and I need to timeout the
> connection of the URLOpen *only*.

That's not so easy even if you were to have a timeout parameter to
urlopen(). You'd have to implement that peculiarity in all the layers
(in this case, urllib and httplib; and possibly ftplib, gopherlib etc.
:-)

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From guido at python.org  Wed Jul  5 00:18:13 2006
From: guido at python.org (Guido van Rossum)
Date: Wed, 5 Jul 2006 00:18:13 +0200
Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
In-Reply-To: <44AAE020.5060204@acm.org>
References: <44AA1FF6.1050501@acm.org>
	<ca471dc20607040728y1035e27ej3d6d589eaed00080@mail.gmail.com>
	<44AAE020.5060204@acm.org>
Message-ID: <ca471dc20607041518j3f3bfa6csef3d0888b2ad301@mail.gmail.com>

On 7/4/06, Talin <talin at acm.org> wrote:
> Guido van Rossum wrote:
> > Also please explain what problem you are solving before proposing a
> > solution.

> Actually, the "problem" I am trying to solve is the debate on the
> mailing list. That is, I listen to what people are asking for, and what
> disagreements they have, and then I try to provide a solution that
> resolves the debate.
>
> In this case, there was a lot of discussion about lexical scoping, and
> various people proposing solutions (such as redefining the behavior of
> 'global') that I thought were (a) problematic, and (b) not a lot of bang
> for the buck (i.e. the disruption vs. utility tradeoff was poor IMHO.)
>
> To be honest, I really have no stake in this proposal, and I don't
> intend to spend any time defending it other than to correct
> misperceptions - however, I offer it as a potential starting point for
> people who are interested in the whole lexical scoping issue. If someone
> feels that this proposal gives them what they want, then great -
> otherwise I'll drop it.

Thanks; I appreciate the attempt. I just think that we're not quite
ready for more proposals (and certainly not for radical ones).
Instead, I'd like to go back to review the needs and desires first.

I think the needs are actually pretty simple. Python currently doesn't
allow assignment to variables in an outer non-global scope, and people
have shown by their behavior that they cannot get used to this
(otherwise the debate would have fizzled by now).

There are two fundamentally different mechanisms seen in programming
languages to control the binding of such variables. The most common
approach is to require declaration of variables in the scope to which
they belong. But Python doesn't do this, and I think it would be a
shame if we had to start doing this now -- the objections against your
proposal clearly show the problems if we try to mix this with Python's
traditional "assignment is declaration" philosophy. The other approach
is an extension of what Python already does for variables in the
global scope. ABC did this too (the SHARE command, see
http://homepages.cwi.nl/~steven/abc/qr.html#HOWTOs).

I think we have to continue to search for a solution that extends the
idea of global declarations.

I've proposed extending its meaning to refer to the nearest outer
scope where the variable is set; if there is no such scope it's an
error. This will break a small number of program but probably not very
many; still, it'll require a future statement or waiting until Python
3.0. The downside is that "global" is not a very intuitive word for
this new meaning. (Maybe ABC's SHARE would have been better.) We could
use a different keyword instead, e.g. 'outer'. I believe I've also
seen proposals in the past that used a number to indicate how many
scopes to go out; I don't like that at all.

I don't see anything else that's attractive. The realistic options are:

1. do nothing
2. extend global's meaning
3. add outer keyword

Personally I think I'd vote for (2) since it doesn't require a new
keyword. But that's only a slight preference over the other two.

Personally it's not a burning need; by the time you start feeling the
need to modify variables in an outer scope you should probably
consider refactoring using an explicit class to hold the state. But I
used the same argument to keep the current form of nested scopes (can
we say "closures"? But what exactly is the closure?) out of the door
and I lost that argument.

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From pje at telecommunity.com  Wed Jul  5 00:25:38 2006
From: pje at telecommunity.com (Phillip J. Eby)
Date: Tue, 04 Jul 2006 18:25:38 -0400
Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
In-Reply-To: <ca471dc20607041518j3f3bfa6csef3d0888b2ad301@mail.gmail.com
 >
References: <44AAE020.5060204@acm.org> <44AA1FF6.1050501@acm.org>
	<ca471dc20607040728y1035e27ej3d6d589eaed00080@mail.gmail.com>
	<44AAE020.5060204@acm.org>
Message-ID: <5.1.1.6.0.20060704182302.0207e890@sparrow.telecommunity.com>

At 12:18 AM 7/5/2006 +0200, Guido van Rossum wrote:
>I don't see anything else that's attractive. The realistic options are:
>
>1. do nothing
>2. extend global's meaning
>3. add outer keyword

Did you also consider and reject:

* Alternate binding operators (e.g. ":=", ".=", etc.)

* Alternate spelling of outer names when binding (e.g. ".x = whatever" to 
bind an outer x)

If so, then these should probably be added to the "rejected alternatives" 
for Py3K so they don't get rehashed.


From nnorwitz at gmail.com  Wed Jul  5 01:49:13 2006
From: nnorwitz at gmail.com (Neal Norwitz)
Date: Tue, 4 Jul 2006 16:49:13 -0700
Subject: [Python-Dev] User's complaints
Message-ID: <ee2a432c0607041649i7ecb6cc1w8ebdf47f7d02097a@mail.gmail.com>

On 7/4/06, Guido van Rossum <guido at python.org> wrote:
>
>  From actual users of
> the language I get more complaints about the breakneck speed of
> Python's evolution than about the brokenness of the current language.

Guido,

I'm really interested in your perspective here.  I assume you hear far
more "average" complaints from Joe Random User.  Can you help give the
rest of us an idea about the top 10 complaints/problems people have?
I realize this will be subjective, that's ok.  Perhaps we should try
to focus our energies on some of these issues.

For example, we heard grumblings about the releases coming too often.
Once we went to an 18 month release schedule, there was minimal
complaining.  It should be fairly safe to assume this silence means
people think we are doing a good job.  What are the things that could
be fixed that would silence the most number of user's complaints?

n
--
PS.  One thing I tend to talk to users about is stability of the
interpreter.  When I talk about crashing the interpreter, the most
common first reaction I get is "you can crash the interpreter? How do
you do that?"  I take that answer as a good sign. :-)

From aahz at pythoncraft.com  Wed Jul  5 05:04:29 2006
From: aahz at pythoncraft.com (Aahz)
Date: Tue, 4 Jul 2006 20:04:29 -0700
Subject: [Python-Dev] 2.5b1 Windows install
In-Reply-To: <449F7CA9.9010105@v.loewis.de>
References: <20060626004726.GA24988@panix.com> <449F7CA9.9010105@v.loewis.de>
Message-ID: <20060705030429.GA18105@panix.com>

On Mon, Jun 26, 2006, "Martin v. L?wis" wrote:
> Aahz wrote:
>>
>> Has anyone else tried doing an admin install with "compile .py files"
>> checked?  It's causing my install to blow up, but I'd prefer to assume
>> it's some weird Windows config/bug unless other people also have it, in
>> which case I'll file an SF report.
> 
> It works fine for me. One way for it to fail is if you uncompilable
> modules in the target directory. Currently, it invokes
> 
>   [TARGETDIR]python.exe -Wi [TARGETDIR]Lib\compileall.py -f -x
> bad_coding|badsyntax|site-packages [TARGETDIR]Lib
> 
> where TARGETDIR is, well, the target directory of the installation.
> You could try to run this after you installed Python without pyc
> compilation, to see whether it succeeds.

Ah-ha!  I haven't actually tested this directly, but I bet I know what's
going on: this isn't properly quoted and fails with TARGETDIR of
"C:\Program Files\Python25" because of the space.  I did test to see that
it works fine with "C:\Python25"

Shall I file a bug?  Or do you want to just document this as a
limitation?
-- 
Aahz (aahz at pythoncraft.com)           <*>         http://www.pythoncraft.com/

"I saw `cout' being shifted "Hello world" times to the left and stopped
right there."  --Steve Gonedes

From guido at python.org  Wed Jul  5 05:49:10 2006
From: guido at python.org (Guido van Rossum)
Date: Wed, 5 Jul 2006 05:49:10 +0200
Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
In-Reply-To: <5.1.1.6.0.20060704182302.0207e890@sparrow.telecommunity.com>
References: <44AA1FF6.1050501@acm.org>
	<ca471dc20607040728y1035e27ej3d6d589eaed00080@mail.gmail.com>
	<44AAE020.5060204@acm.org>
	<5.1.1.6.0.20060704182302.0207e890@sparrow.telecommunity.com>
Message-ID: <ca471dc20607042049r6a801e6dp67bcd9dd2781df07@mail.gmail.com>

On 7/5/06, Phillip J. Eby <pje at telecommunity.com> wrote:
> At 12:18 AM 7/5/2006 +0200, Guido van Rossum wrote:
> >I don't see anything else that's attractive. The realistic options are:
> >
> >1. do nothing
> >2. extend global's meaning
> >3. add outer keyword
>
> Did you also consider and reject:
>
> * Alternate binding operators (e.g. ":=", ".=", etc.)

Brr.

> * Alternate spelling of outer names when binding (e.g. ".x = whatever" to
> bind an outer x)

We looked at and rejected "globals.x = whatever". I think the same
reasoning applies here.

> If so, then these should probably be added to the "rejected alternatives"
> for Py3K so they don't get rehashed.

Georgbot?

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From martin at v.loewis.de  Wed Jul  5 07:23:54 2006
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Wed, 05 Jul 2006 07:23:54 +0200
Subject: [Python-Dev] 2.5b1 Windows install
In-Reply-To: <20060705030429.GA18105@panix.com>
References: <20060626004726.GA24988@panix.com> <449F7CA9.9010105@v.loewis.de>
	<20060705030429.GA18105@panix.com>
Message-ID: <44AB4CEA.9080705@v.loewis.de>

Aahz wrote:
> Ah-ha!  I haven't actually tested this directly, but I bet I know what's
> going on: this isn't properly quoted and fails with TARGETDIR of
> "C:\Program Files\Python25" because of the space.  I did test to see that
> it works fine with "C:\Python25"
> 
> Shall I file a bug?  Or do you want to just document this as a
> limitation?

If this is indeed the problem, it should be fixed. Before filing the bug
report, please confirm that this actually is a problem.

Regards,
Martin

From ronaldoussoren at mac.com  Wed Jul  5 07:35:25 2006
From: ronaldoussoren at mac.com (Ronald Oussoren)
Date: Wed, 5 Jul 2006 07:35:25 +0200
Subject: [Python-Dev] zlib module build failure on Mac OSX 10.4.7
In-Reply-To: <ee2a432c0607041421y37438adboc658313e85114e52@mail.gmail.com>
References: <17573.60593.759293.549583@montanaro.dyndns.org>
	<130298B1-7336-480A-A90F-97B68327BE4A@mac.com>
	<17574.43390.560231.425494@montanaro.dyndns.org>
	<315DECAC-D582-4692-A0CA-13D3C7BF992C@mac.com>
	<44A6C314.2000709@v.loewis.de>
	<0006F35E-FD5F-4BAD-A608-642039BBE7E8@mac.com>
	<ee2a432c0607041421y37438adboc658313e85114e52@mail.gmail.com>
Message-ID: <738A121B-3529-411D-89A7-EBD8C6B42A29@mac.com>


On Jul 4, 2006, at 11:21 PM, Neal Norwitz wrote:

> Ronald, Bob,
>
> I know Skip found and fixed his problem, however, is this problem
> likely to affect other users?  Is there anything we can do to help
> alleviate/diagnose this problem?

I'll either enhance configure or roll back my change to setup.py. I'd  
prefer to do the former, but if beta2 gets too close I'll just change  
setup.py.

Ronald

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 2157 bytes
Desc: not available
Url : http://mail.python.org/pipermail/python-dev/attachments/20060705/256a01dc/attachment.bin 

From pje at telecommunity.com  Wed Jul  5 08:26:25 2006
From: pje at telecommunity.com (Phillip J. Eby)
Date: Wed, 05 Jul 2006 02:26:25 -0400
Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
In-Reply-To: <ca471dc20607042049r6a801e6dp67bcd9dd2781df07@mail.gmail.co
 m>
References: <5.1.1.6.0.20060704182302.0207e890@sparrow.telecommunity.com>
	<44AA1FF6.1050501@acm.org>
	<ca471dc20607040728y1035e27ej3d6d589eaed00080@mail.gmail.com>
	<44AAE020.5060204@acm.org>
	<5.1.1.6.0.20060704182302.0207e890@sparrow.telecommunity.com>
Message-ID: <5.1.1.6.0.20060705020306.0207f890@sparrow.telecommunity.com>

At 05:49 AM 7/5/2006 +0200, Guido van Rossum wrote:
>>* Alternate spelling of outer names when binding (e.g. ".x = whatever" to
>>bind an outer x)
>
>We looked at and rejected "globals.x = whatever". I think the same
>reasoning applies here.

I thought the 'globals.x' proposal required that 'x' always be accessed 
using 'globals', even if it wasn't being rebound.  I don't see a problem 
with requiring '.x' to be used for both reading and writing of outer-scope 
names; it just shouldn't be required for an outer-scope name that you don't 
rebind in the current scope.  That symmetry requirement can't be 
implemented with the 'globals.x' approach unless 'globals' is treated 
specially by the compiler.

Using the classic nonsense example:

     def counter(num):
         def inc():
             .num += 1
             return .num
         return inc

If inc() only needed to *read* num, it could just use 'num' without the 
'.', and be nicely backward compatible with today's Python.

(Note: It should be illegal to use both '.num' and 'num' in the same scope, 
whether writing or reading the value, to prevent readers from becoming 
confused about what variable you mean.  It should also be required that the 
compiler can see a definition of 'num' in an outer scope if you use the 
'.num' syntax, so that misspelling a name doesn't create a global variable.)

I personally think this approach could be the overall least-intrusive 
solution as far as syntax goes.  It also allows for dropping the 'global' 
keyword in 3.x, and it has a nice EIBTI feel to it, as it allows you to 
highlight closure variables in an inner function by using the '.'.  It's 
often not obvious when an inner function (such as a decorator returned by a 
decorator factory) is using variables that were defined in the outer scope; 
the leading '.' would make them stand out, and so could be considered the 
recommended code style when referring to outer variables.

In addition, there's a nice symmetry between nested functions and top-level 
functions, e.g. in this global version of the counter example:

     num = 0
     def inc():
         .num += 1
         return .num

The principle downside taht I see is that it uses semi-random punctuation 
in place of keywords.  OTOH, we are already using more-or-less this syntax 
for relative imports, so reusing it to mean "relative variables" seems to 
at least avoid creating any entirely new principles.  :)

Anyway, I won't argue this one further; I just wanted to make sure it had 
been considered, as I'm not sure that you were reading the thread where it 
was first brought up (possibly as long as a few months ago).


From guido at python.org  Wed Jul  5 10:12:53 2006
From: guido at python.org (Guido van Rossum)
Date: Wed, 5 Jul 2006 10:12:53 +0200
Subject: [Python-Dev] Import semantics
In-Reply-To: <4dab5f760606251334o718323adt409294e2952e954a@mail.gmail.com>
References: <cfb578b20606111531t6806d5c9kd35fd8ba29638174@mail.gmail.com>
	<448D1F0D.7000405@strakt.com>
	<ca471dc20606120948h119f1c1fw2725e7d434e287df@mail.gmail.com>
	<4dab5f760606251334o718323adt409294e2952e954a@mail.gmail.com>
Message-ID: <ca471dc20607050112n1bae4fd6i941999617bba5a27@mail.gmail.com>

On 6/25/06, Frank Wierzbicki <fwierzbicki at gmail.com> wrote:
> Sorry for the untrimmed conversation, but I've cc'ed jython-dev, my
> comments are at the bottom.
>
> On 6/12/06, Guido van Rossum <guido at python.org> wrote:
> > On 6/12/06, Samuele Pedroni <pedronis at strakt.com> wrote:
> > > Fabio Zadrozny wrote:
> > > > Python and Jython import semantics differ on how sub-packages should be
> > > > accessed after importing some module:
> > > >
> > > > Jython 2.1 on java1.5.0 (JIT: null)
> > > > Type "copyright", "credits" or "license" for more information.
> > > >  >>> import xml
> > > >  >>> xml.dom
> > > > <module xml.dom at 10340434>
> > > >
> > > > Python 2.4.2 (#67, Sep 28 2005, 12:41:11) [MSC v.1310 32 bit (Intel)] on
> > > > win32
> > > > Type "help", "copyright", "credits" or "license" for more information.
> > > >  >>> import xml
> > > >  >>> xml.dom
> > > > Traceback (most recent call last):
> > > >   File "<stdin>", line 1, in ?
> > > > AttributeError: 'module' object has no attribute 'dom'
> > > >  >>> from xml.dom import pulldom
> > > >  >>> xml.dom
> > > > <module 'xml.dom' from 'C:\bin\Python24\lib\xml\dom\__init__.pyc'>
> > > >
> > > > Note that in Jython importing a module makes all subpackages beneath it
> > > > available, whereas in python, only the tokens available in __init__.py
> > > > are accessible, but if you do load the module later even if not getting
> > > > it directly into the namespace, it gets accessible too -- this seems
> > > > more like something unexpected to me -- I would expect it to be
> > > > available only if I did some "import xml.dom" at some point.
> > > >
> > > > My problem is that in Pydev, in static analysis, I would only get the
> > > > tokens available for actually imported modules, but that's not true for
> > > > Jython, and I'm not sure if the current behaviour in Python was expected.
> > > >
> > > > So... which would be the right semantics for this?
> > >
> > > the difference in Jython is deliberate. I think the reason was to mimic
> > > more the Java style for this, in java fully qualified names always work.
> > > In jython importing the top level packages is enough to get a similar
> > > effect.
> > >
> > > This is unlikely to change for backward compatibility reasons, at least
> > > from my POV.
> >
> > IMO it should do this only if the imported module is really a Java
> > package. If it's a Python package it should stick to python semantics
> > if possible.
> >
> > --
> > --Guido van Rossum (home page: http://www.python.org/~guido/)
>
> This is a tough one since the BDFL and Samuele disagree here.  Perhaps
> we should document the Java import behavior as permanent, but document
> the Python imports in Jython as being deprecated but available until
> some future release?  I believe we would keep it at least through
> Jython 2.3.

Hi Frank,

Have you and/or the Jython community made up your mind about this? The
thread seems to have disappeared after you posted (or perhaps it
continued only on jython-dev, which I don't read?).

Also, I just realized that you're the new Jython maintainer. Is *that*
official? I'd like to offer you my congratulations, and, more
importantly, any support you might need. I find Jython an important
part for Python's long-term stategy. I'm asked occasionally what the
status of Jython is; people point out that the last release was 2.1
many years ago and the website has no news since early 2005; thy're
afraid that Jython is dying and that it's not a viable choice for new
projects. I'm very happy to be able to tell them that soon there will
be a 2.3 release and yes there *is* continued support... So if you
need anything from me or from the PSF, please let me know!

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From anthony at interlink.com.au  Wed Jul  5 10:17:59 2006
From: anthony at interlink.com.au (Anthony Baxter)
Date: Wed, 5 Jul 2006 18:17:59 +1000
Subject: [Python-Dev] Import semantics
In-Reply-To: <ca471dc20607050112n1bae4fd6i941999617bba5a27@mail.gmail.com>
References: <cfb578b20606111531t6806d5c9kd35fd8ba29638174@mail.gmail.com>
	<4dab5f760606251334o718323adt409294e2952e954a@mail.gmail.com>
	<ca471dc20607050112n1bae4fd6i941999617bba5a27@mail.gmail.com>
Message-ID: <200607051818.02742.anthony@interlink.com.au>

On Wednesday 05 July 2006 18:12, Guido van Rossum wrote:
> I'm asked
> occasionally what the status of Jython is; people point out that
> the last release was 2.1 many years ago and the website has no news
> since early 2005; thy're afraid that Jython is dying and that it's
> not a viable choice for new projects. I'm very happy to be able to
> tell them that soon there will be a 2.3 release and yes there *is*
> continued support... So if you need anything from me or from the
> PSF, please let me know!

In that case, why not post a news item saying this? The website is 
probably the first place people look...

Anthony

-- 
Anthony Baxter     <anthony at interlink.com.au>
It's never too late to have a happy childhood.

From guido at python.org  Wed Jul  5 10:21:00 2006
From: guido at python.org (Guido van Rossum)
Date: Wed, 5 Jul 2006 10:21:00 +0200
Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
In-Reply-To: <5.1.1.6.0.20060705020306.0207f890@sparrow.telecommunity.com>
References: <44AA1FF6.1050501@acm.org>
	<ca471dc20607040728y1035e27ej3d6d589eaed00080@mail.gmail.com>
	<44AAE020.5060204@acm.org>
	<5.1.1.6.0.20060704182302.0207e890@sparrow.telecommunity.com>
	<5.1.1.6.0.20060705020306.0207f890@sparrow.telecommunity.com>
Message-ID: <ca471dc20607050121o48dd51a7v5f12ee15b9f67ea3@mail.gmail.com>

On 7/5/06, Phillip J. Eby <pje at telecommunity.com> wrote:
> At 05:49 AM 7/5/2006 +0200, Guido van Rossum wrote:
> >>* Alternate spelling of outer names when binding (e.g. ".x = whatever" to
> >>bind an outer x)
> >
> >We looked at and rejected "globals.x = whatever". I think the same
> >reasoning applies here.
>
> I thought the 'globals.x' proposal required that 'x' always be accessed
> using 'globals', even if it wasn't being rebound.  I don't see a problem
> with requiring '.x' to be used for both reading and writing of outer-scope
> names; it just shouldn't be required for an outer-scope name that you don't
> rebind in the current scope.  That symmetry requirement can't be
> implemented with the 'globals.x' approach unless 'globals' is treated
> specially by the compiler.
>
> Using the classic nonsense example:
>
>      def counter(num):
>          def inc():
>              .num += 1
>              return .num
>          return inc
>
> If inc() only needed to *read* num, it could just use 'num' without the
> '.', and be nicely backward compatible with today's Python.
>
> (Note: It should be illegal to use both '.num' and 'num' in the same scope,
> whether writing or reading the value, to prevent readers from becoming
> confused about what variable you mean.  It should also be required that the
> compiler can see a definition of 'num' in an outer scope if you use the
> '.num' syntax, so that misspelling a name doesn't create a global variable.)
>
> I personally think this approach could be the overall least-intrusive
> solution as far as syntax goes.  It also allows for dropping the 'global'
> keyword in 3.x, and it has a nice EIBTI feel to it, as it allows you to
> highlight closure variables in an inner function by using the '.'.  It's
> often not obvious when an inner function (such as a decorator returned by a
> decorator factory) is using variables that were defined in the outer scope;
> the leading '.' would make them stand out, and so could be considered the
> recommended code style when referring to outer variables.
>
> In addition, there's a nice symmetry between nested functions and top-level
> functions, e.g. in this global version of the counter example:
>
>      num = 0
>      def inc():
>          .num += 1
>          return .num
>
> The principle downside taht I see is that it uses semi-random punctuation
> in place of keywords.  OTOH, we are already using more-or-less this syntax
> for relative imports, so reusing it to mean "relative variables" seems to
> at least avoid creating any entirely new principles.  :)
>
> Anyway, I won't argue this one further; I just wanted to make sure it had
> been considered, as I'm not sure that you were reading the thread where it
> was first brought up (possibly as long as a few months ago).

Thanks for bringing this up. I'm not sure what I think of it yet. One
problem I see is that there might end up being two ways to reference
variables in outer scopes: .num if you plan to assign to it, or just
num if you only reference it. I find that the most disurbing issue so
far; modified global declarations or outer declarations don't have
this problem.

Would this also use ..num to refer to num in an outer scope two levels removed?

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From guido at python.org  Wed Jul  5 10:22:16 2006
From: guido at python.org (Guido van Rossum)
Date: Wed, 5 Jul 2006 10:22:16 +0200
Subject: [Python-Dev] Import semantics
In-Reply-To: <200607051818.02742.anthony@interlink.com.au>
References: <cfb578b20606111531t6806d5c9kd35fd8ba29638174@mail.gmail.com>
	<4dab5f760606251334o718323adt409294e2952e954a@mail.gmail.com>
	<ca471dc20607050112n1bae4fd6i941999617bba5a27@mail.gmail.com>
	<200607051818.02742.anthony@interlink.com.au>
Message-ID: <ca471dc20607050122s5e1b6784wbbe69943ecf04b4b@mail.gmail.com>

On 7/5/06, Anthony Baxter <anthony at interlink.com.au> wrote:
> On Wednesday 05 July 2006 18:12, Guido van Rossum wrote:
> > I'm asked
> > occasionally what the status of Jython is; people point out that
> > the last release was 2.1 many years ago and the website has no news
> > since early 2005; thy're afraid that Jython is dying and that it's
> > not a viable choice for new projects. I'm very happy to be able to
> > tell them that soon there will be a 2.3 release and yes there *is*
> > continued support... So if you need anything from me or from the
> > PSF, please let me know!
>
> In that case, why not post a news item saying this? The website is
> probably the first place people look...

I'm all for that; but I don't have webmaster privileges (nor do I want
them :-). Frank?

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From anthony at interlink.com.au  Wed Jul  5 10:25:16 2006
From: anthony at interlink.com.au (Anthony Baxter)
Date: Wed, 5 Jul 2006 18:25:16 +1000
Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
In-Reply-To: <ca471dc20607050121o48dd51a7v5f12ee15b9f67ea3@mail.gmail.com>
References: <44AA1FF6.1050501@acm.org>
	<5.1.1.6.0.20060705020306.0207f890@sparrow.telecommunity.com>
	<ca471dc20607050121o48dd51a7v5f12ee15b9f67ea3@mail.gmail.com>
Message-ID: <200607051825.17720.anthony@interlink.com.au>

On Wednesday 05 July 2006 18:21, Guido van Rossum wrote:
> Would this also use ..num to refer to num in an outer scope two
> levels removed?

Ew! 

I don't want to even think about debugging

  ...x
vs
  ....x

Anthony
-- 
Anthony Baxter     <anthony at interlink.com.au>
It's never too late to have a happy childhood.

From guido at python.org  Wed Jul  5 10:28:45 2006
From: guido at python.org (Guido van Rossum)
Date: Wed, 5 Jul 2006 10:28:45 +0200
Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
In-Reply-To: <200607051825.17720.anthony@interlink.com.au>
References: <44AA1FF6.1050501@acm.org>
	<5.1.1.6.0.20060705020306.0207f890@sparrow.telecommunity.com>
	<ca471dc20607050121o48dd51a7v5f12ee15b9f67ea3@mail.gmail.com>
	<200607051825.17720.anthony@interlink.com.au>
Message-ID: <ca471dc20607050128m5507083by50215e630dc01486@mail.gmail.com>

Sorry, I should have added a <wink>... :-)

On 7/5/06, Anthony Baxter <anthony at interlink.com.au> wrote:
> On Wednesday 05 July 2006 18:21, Guido van Rossum wrote:
> > Would this also use ..num to refer to num in an outer scope two
> > levels removed?
>
> Ew!
>
> I don't want to even think about debugging
>
>   ...x
> vs
>   ....x
>
> Anthony
> --
> Anthony Baxter     <anthony at interlink.com.au>
> It's never too late to have a happy childhood.
>


-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From g.brandl at gmx.net  Wed Jul  5 10:29:27 2006
From: g.brandl at gmx.net (Georg Brandl)
Date: Wed, 05 Jul 2006 10:29:27 +0200
Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
In-Reply-To: <ca471dc20607042049r6a801e6dp67bcd9dd2781df07@mail.gmail.com>
References: <44AA1FF6.1050501@acm.org>	<ca471dc20607040728y1035e27ej3d6d589eaed00080@mail.gmail.com>	<44AAE020.5060204@acm.org>	<5.1.1.6.0.20060704182302.0207e890@sparrow.telecommunity.com>
	<ca471dc20607042049r6a801e6dp67bcd9dd2781df07@mail.gmail.com>
Message-ID: <e8ft97$h2q$1@sea.gmane.org>

Guido van Rossum wrote:
> On 7/5/06, Phillip J. Eby <pje at telecommunity.com> wrote:
>> At 12:18 AM 7/5/2006 +0200, Guido van Rossum wrote:
>> >I don't see anything else that's attractive. The realistic options are:
>> >
>> >1. do nothing
>> >2. extend global's meaning
>> >3. add outer keyword
>>
>> Did you also consider and reject:
>>
>> * Alternate binding operators (e.g. ":=", ".=", etc.)
> 
> Brr.
> 
>> * Alternate spelling of outer names when binding (e.g. ".x = whatever" to
>> bind an outer x)
> 
> We looked at and rejected "globals.x = whatever". I think the same
> reasoning applies here.
> 
>> If so, then these should probably be added to the "rejected alternatives"
>> for Py3K so they don't get rehashed.
> 
> Georgbot?

I added the alternative binding operators. The discussion about ".x" seems
to be still in progress.

Georg


From scott+python-dev at scottdial.com  Wed Jul  5 11:40:40 2006
From: scott+python-dev at scottdial.com (Scott Dial)
Date: Wed, 05 Jul 2006 05:40:40 -0400
Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
In-Reply-To: <ca471dc20607050121o48dd51a7v5f12ee15b9f67ea3@mail.gmail.com>
References: <44AA1FF6.1050501@acm.org>	<ca471dc20607040728y1035e27ej3d6d589eaed00080@mail.gmail.com>	<44AAE020.5060204@acm.org>	<5.1.1.6.0.20060704182302.0207e890@sparrow.telecommunity.com>	<5.1.1.6.0.20060705020306.0207f890@sparrow.telecommunity.com>
	<ca471dc20607050121o48dd51a7v5f12ee15b9f67ea3@mail.gmail.com>
Message-ID: <44AB8918.1040308@scottdial.com>

Guido van Rossum wrote:
> Would this also use ..num to refer to num in an outer scope two levels removed?

I realize this was a wink, but it is a valid problem with the 
"dot"-proposal.

def foo(n):
   def bar(n):
     def baz():
       return .n

So, which 'n' outer 'n' is being referenced? Seems like you need to 
either be able to do multiple dots (ugly, hard to read) or only do a 
single-step outwards reference. But then that has it's own problems, if 
I meant the 'n' passed into 'foo', then I have to resort to such 
nonsense as:

def foo(n):
   def bar(n):
     foon = .n
     def baz():
       return .foon

It would almost be cute if you could do something like ".foo.n" to get 
to the correct variable. If python maintains it's current scoping rules, 
then it seems like it works out, but I haven't thought this one all the 
way through.

def foo(n):
   def bar(n):
     def baz():
       return .foo.n + .bar.n

-- 
Scott Dial
scott at scottdial.com
scodial at indiana.edu

From and-dev at doxdesk.com  Wed Jul  5 11:18:32 2006
From: and-dev at doxdesk.com (Andrew Clover)
Date: Wed, 05 Jul 2006 18:18:32 +0900
Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
In-Reply-To: <ca471dc20607041518j3f3bfa6csef3d0888b2ad301@mail.gmail.com>
References: <44AA1FF6.1050501@acm.org>	<ca471dc20607040728y1035e27ej3d6d589eaed00080@mail.gmail.com>	<44AAE020.5060204@acm.org>
	<ca471dc20607041518j3f3bfa6csef3d0888b2ad301@mail.gmail.com>
Message-ID: <44AB83E8.2070303@doxdesk.com>

Guido van Rossum <guido at python.org> wrote:

> 1. do nothing
> 2. extend global's meaning
> 3. add outer keyword

2.5. extend global syntax to cover both [really global] and [innermost 
matching scope].

eg.

   global x, y outer   # trailing non-keyword
   global in x, y      # re-use keyword
   not global x        # ceci n'est pas un global
   ...                 # something less ugly?

> Personally it's not a burning need

Agreed. Inability to write as well as read nested scopes is more of an 
aesthetic wart than a practical one IMO.

-- 
And Clover
mailto:and at doxdesk.com
http://www.doxdesk.com/

-- 
And Clover
mailto:and at doxdesk.com
http://www.doxdesk.com/

From just at letterror.com  Wed Jul  5 11:46:50 2006
From: just at letterror.com (Just van Rossum)
Date: Wed,  5 Jul 2006 11:46:50 +0200
Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
In-Reply-To: <ca471dc20607042049r6a801e6dp67bcd9dd2781df07@mail.gmail.com>
Message-ID: <r01050400-1039-300385330C0B11DB9236001124365170@[10.0.0.24]>

Guido van Rossum wrote:

> On 7/5/06, Phillip J. Eby <pje at telecommunity.com> wrote:
> > Did you also consider and reject:
> >
> > * Alternate binding operators (e.g. ":=", ".=", etc.)
> 
> Brr.

That's too bad :(

I still find a rebinding operator (":=" being my favorite) much, *much*
more appealing than any of the alternative proposals. It's beautifully
symmetrical with "assignment means local". It also pretty much makes the
global statement redundant.

The only downside I see is that it may cause a fairly big shift in
style: I for one would use := for rebinding local names. While I think
that would be an improvement (eg. by catching typo's earlier), it's
*different*.

Just

-- 
Change is bad. We fear change. -- Garth Algar

From guido at python.org  Wed Jul  5 12:02:16 2006
From: guido at python.org (Guido van Rossum)
Date: Wed, 5 Jul 2006 12:02:16 +0200
Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
In-Reply-To: <r01050400-1039-300385330C0B11DB9236001124365170@10.0.0.24>
References: <ca471dc20607042049r6a801e6dp67bcd9dd2781df07@mail.gmail.com>
	<r01050400-1039-300385330C0B11DB9236001124365170@10.0.0.24>
Message-ID: <ca471dc20607050302j9499d00h5510f5e96bf28eae@mail.gmail.com>

On 7/5/06, Just van Rossum <just at letterror.com> wrote:
> Guido van Rossum wrote:
>
> > On 7/5/06, Phillip J. Eby <pje at telecommunity.com> wrote:
> > > Did you also consider and reject:
> > >
> > > * Alternate binding operators (e.g. ":=", ".=", etc.)
> >
> > Brr.
>
> That's too bad :(
>
> I still find a rebinding operator (":=" being my favorite) much, *much*
> more appealing than any of the alternative proposals. It's beautifully
> symmetrical with "assignment means local". It also pretty much makes the
> global statement redundant.
>
> The only downside I see is that it may cause a fairly big shift in
> style: I for one would use := for rebinding local names. While I think
> that would be an improvement (eg. by catching typo's earlier), it's
> *different*.

Hallo broer! :-)

I wonder what this should mean then:

def outer():
  def inner():
    x := 1

What is x's scope?

Also, a := operator allows all sorts of left-hand sides that don't
necessarily make sense, e.g.

x.foo := 1
x[0] := 1

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From imbaczek at gmail.com  Wed Jul  5 12:17:16 2006
From: imbaczek at gmail.com (=?ISO-8859-2?Q?Marek_"Baczek"_Baczy=F1ski?=)
Date: Wed, 5 Jul 2006 12:17:16 +0200
Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
In-Reply-To: <r01050400-1039-300385330C0B11DB9236001124365170@10.0.0.24>
References: <ca471dc20607042049r6a801e6dp67bcd9dd2781df07@mail.gmail.com>
	<r01050400-1039-300385330C0B11DB9236001124365170@10.0.0.24>
Message-ID: <5f3d2c310607050317t51d796d1g348fc2d036798128@mail.gmail.com>

2006/7/5, Just van Rossum <just at letterror.com>:
> Guido van Rossum wrote:
>
> > On 7/5/06, Phillip J. Eby <pje at telecommunity.com> wrote:
> > > Did you also consider and reject:
> > >
> > > * Alternate binding operators (e.g. ":=", ".=", etc.)
> >
> > Brr.
>
> That's too bad :(
>
> I still find a rebinding operator (":=" being my favorite) much, *much*
> more appealing than any of the alternative proposals. It's beautifully
> symmetrical with "assignment means local". It also pretty much makes the
> global statement redundant.
>
> The only downside I see is that it may cause a fairly big shift in
> style: I for one would use := for rebinding local names. While I think
> that would be an improvement (eg. by catching typo's earlier), it's
> *different*.

<delurk>

I suggest <- as an assignment operator instead of := - it's used in
OCaml and it looks *very* different, yet still makes sense.

  x = 0
  print x
  def f():
    x =  1 # bind locally
    print x
  def g():
    x <- 42 # assign "lexically"
    print x
  f()
  print x
  g()
  print x

prints

0
1
0
42
42

</delurk>

From barry at python.org  Wed Jul  5 12:48:25 2006
From: barry at python.org (Barry Warsaw)
Date: Wed, 5 Jul 2006 06:48:25 -0400
Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
In-Reply-To: <5f3d2c310607050317t51d796d1g348fc2d036798128@mail.gmail.com>
References: <ca471dc20607042049r6a801e6dp67bcd9dd2781df07@mail.gmail.com>
	<r01050400-1039-300385330C0B11DB9236001124365170@10.0.0.24>
	<5f3d2c310607050317t51d796d1g348fc2d036798128@mail.gmail.com>
Message-ID: <2FD28BA2-32D9-49C4-8D86-92B0A53346A6@python.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

PCTotD (pre-coffee thought of the day):

On Jul 5, 2006, at 6:17 AM, Marek "Baczek" Baczy?ski wrote:

> 2006/7/5, Just van Rossum <just at letterror.com>:
>> Guido van Rossum wrote:
>>
>>> On 7/5/06, Phillip J. Eby <pje at telecommunity.com> wrote:
>>>> Did you also consider and reject:
>>>>
>>>> * Alternate binding operators (e.g. ":=", ".=", etc.)
>>>
>>> Brr.

Clearly we need the "as if in" operator:

x = 0
def foo():
    x = 1
    def bar():
       x = 2
       def baz():
           x as if in foo = 3
       x as if in global += 1

(Personally, I've never really needed this much, but if you have to  
have it, be explicit! :)

- -Barry

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.3 (Darwin)

iQCVAwUBRKuY+XEjvBPtnXfVAQI7WAQAp9fp8SZU8lJdyllrrbFhR9lWxF+pbZ9d
+OBv0VEFiFD3aLuKc7y7rAyCWfRVMsoO3FsapeC0/CFJDpRoDDRB7qLzX7He9Fyf
JXh3g489RdokxwZClIl38RVgM7WNVK8lzFr8Kf8eTjEUItJoJjk5m2vhymQpfWYG
/OeU5xCABxw=
=OBRd
-----END PGP SIGNATURE-----

From just at letterror.com  Wed Jul  5 12:54:07 2006
From: just at letterror.com (Just van Rossum)
Date: Wed,  5 Jul 2006 12:54:07 +0200
Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
In-Reply-To: <ca471dc20607050302j9499d00h5510f5e96bf28eae@mail.gmail.com>
Message-ID: <r01050400-1039-963B95DA0C1411DB9236001124365170@[10.0.0.24]>

Guido van Rossum wrote:

> Hallo broer! :-)

Yo :)

> I wonder what this should mean then:
> 
> def outer():
>   def inner():
>     x := 1
>
> What is x's scope?

UnboundVariableError: variable 'x' referenced before assignment

Or a SyntaxError if the compiler can detect it.

> Also, a := operator allows all sorts of left-hand sides that don't
> necessarily make sense, e.g.
> 
> x.foo := 1
> x[0] := 1

True, although maybe they could be made to make sense by defining
special methods:

    __rebindattr__
    __rebinditem__

<0.5 wink>

Just

From s.percivall at chello.se  Wed Jul  5 14:18:00 2006
From: s.percivall at chello.se (Simon Percivall)
Date: Wed, 5 Jul 2006 14:18:00 +0200
Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
In-Reply-To: <ca471dc20607050302j9499d00h5510f5e96bf28eae@mail.gmail.com>
References: <ca471dc20607042049r6a801e6dp67bcd9dd2781df07@mail.gmail.com>
	<r01050400-1039-300385330C0B11DB9236001124365170@10.0.0.24>
	<ca471dc20607050302j9499d00h5510f5e96bf28eae@mail.gmail.com>
Message-ID: <D6FFF1A2-FEEB-4D06-862A-DC5FAA54DF5B@chello.se>

I know this is very similar to the "global.x =" syntax, which
was already shot down?, but wouldn't allowing access to a
functions locals from within, by prefixing the name, be a good
way to disambiguate what happens (instead of any operator to
indicate outer scope, like .x = 3 or the like)? I guess this
necessitates "global.x =" as well, though.

def foo():
     def bar():
         foo.x = 3
     print x # prints 3

I seem to recall that this syntax has been proposed before,
though not in this discussion. But my memory is murky.

//Simon





From just at letterror.com  Wed Jul  5 14:23:15 2006
From: just at letterror.com (Just van Rossum)
Date: Wed,  5 Jul 2006 14:23:15 +0200
Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
In-Reply-To: <5f3d2c310607050317t51d796d1g348fc2d036798128@mail.gmail.com>
Message-ID: <r01050400-1039-0A0E1D8C0C2111DB9236001124365170@[10.0.0.24]>

Marek "Baczek" Baczy?ski wrote:

> I suggest <- as an assignment operator instead of := - it's used in
> OCaml and it looks *very* different, yet still makes sense.

Except it's currently valid Python syntax:

    >>> x = 0
    >>> x <- 42
    False
    >>> 

Just

From skip at pobox.com  Wed Jul  5 15:39:05 2006
From: skip at pobox.com (skip at pobox.com)
Date: Wed, 5 Jul 2006 08:39:05 -0500
Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
In-Reply-To: <2FD28BA2-32D9-49C4-8D86-92B0A53346A6@python.org>
References: <ca471dc20607042049r6a801e6dp67bcd9dd2781df07@mail.gmail.com>
	<r01050400-1039-300385330C0B11DB9236001124365170@10.0.0.24>
	<5f3d2c310607050317t51d796d1g348fc2d036798128@mail.gmail.com>
	<2FD28BA2-32D9-49C4-8D86-92B0A53346A6@python.org>
Message-ID: <17579.49401.682772.29091@montanaro.dyndns.org>


    Barry> Clearly we need the "as if in" operator:

Why not be more direct?

    x = 0
    def foo():
        x = 1
        def bar():
           x = 2
           def baz():
               x in foo = 3
           x in global += 1

By naming the function in which the binding is to occur you avoid problems
of someone coming along and adding or deleting functions between the
assignment (in baz) and the target of the assignment (x in foo) but then
forgetting to increment or decrement the counters that refer to a fixed
number of levels above the current function.

    Barry> (Personally, I've never really needed this much, but if you have
    Barry> to have it, be explicit! :)

Nor I.  I can't think of any situations in my programming where I've used
nested functions, but I was never a LISPer...

Skip

From s.percivall at chello.se  Wed Jul  5 16:04:43 2006
From: s.percivall at chello.se (Simon Percivall)
Date: Wed, 5 Jul 2006 16:04:43 +0200
Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
In-Reply-To: <44AB8918.1040308@scottdial.com>
References: <44AA1FF6.1050501@acm.org>	<ca471dc20607040728y1035e27ej3d6d589eaed00080@mail.gmail.com>	<44AAE020.5060204@acm.org>	<5.1.1.6.0.20060704182302.0207e890@sparrow.telecommunity.com>	<5.1.1.6.0.20060705020306.0207f890@sparrow.telecommunity.com>
	<ca471dc20607050121o48dd51a7v5f12ee15b9f67ea3@mail.gmail.com>
	<44AB8918.1040308@scottdial.com>
Message-ID: <6A5A1492-EDD0-4FAF-9A2C-B4A59AF19053@chello.se>

On 5 jul 2006, at 11.40, Scott Dial wrote:
> Guido van Rossum wrote:
>> Would this also use ..num to refer to num in an outer scope two  
>> levels removed?
>
> I realize this was a wink, but it is a valid problem with the
> "dot"-proposal.
>
> def foo(n):
>    def bar(n):
>      def baz():
>        return .n
>
> So, which 'n' outer 'n' is being referenced? Seems like you need to
> either be able to do multiple dots (ugly, hard to read) or only do a
> single-step outwards reference. But then that has it's own  
> problems, if
> I meant the 'n' passed into 'foo', then I have to resort to such
> nonsense as:
<snip>

No, it's actually not a problem. foo()'s "n" should just be
hidden. If you don't want it to be hidden, don't write your
function that way. If you find you need deeply nested functions
where local names shadow names in outer scopes that you need to
access you might want to think of another way to solve your
problem.

//Simon

From just at letterror.com  Wed Jul  5 16:16:15 2006
From: just at letterror.com (Just van Rossum)
Date: Wed,  5 Jul 2006 16:16:15 +0200
Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
In-Reply-To: <6A5A1492-EDD0-4FAF-9A2C-B4A59AF19053@chello.se>
Message-ID: <r01050400-1039-D31103DB0C3011DB9236001124365170@[10.0.0.24]>

Simon Percivall wrote:

> On 5 jul 2006, at 11.40, Scott Dial wrote:
> > Guido van Rossum wrote:
> >> Would this also use ..num to refer to num in an outer scope two  
> >> levels removed?
> >
> > I realize this was a wink, but it is a valid problem with the
> > "dot"-proposal.
> >
> > def foo(n):
> >    def bar(n):
> >      def baz():
> >        return .n
> >
> > So, which 'n' outer 'n' is being referenced? Seems like you need to
> > either be able to do multiple dots (ugly, hard to read) or only do a
> > single-step outwards reference. But then that has it's own  
> > problems, if
> > I meant the 'n' passed into 'foo', then I have to resort to such
> > nonsense as:
> <snip>
> 
> No, it's actually not a problem. foo()'s "n" should just be
> hidden. If you don't want it to be hidden, don't write your
> function that way. If you find you need deeply nested functions
> where local names shadow names in outer scopes that you need to
> access you might want to think of another way to solve your
> problem.

+1, YAGNI.

Just

From barry at python.org  Wed Jul  5 16:21:33 2006
From: barry at python.org (Barry Warsaw)
Date: Wed, 5 Jul 2006 10:21:33 -0400
Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
In-Reply-To: <17579.49401.682772.29091@montanaro.dyndns.org>
References: <ca471dc20607042049r6a801e6dp67bcd9dd2781df07@mail.gmail.com>
	<r01050400-1039-300385330C0B11DB9236001124365170@10.0.0.24>
	<5f3d2c310607050317t51d796d1g348fc2d036798128@mail.gmail.com>
	<2FD28BA2-32D9-49C4-8D86-92B0A53346A6@python.org>
	<17579.49401.682772.29091@montanaro.dyndns.org>
Message-ID: <7CAE772C-8050-4ED2-8799-1ACB15A0131C@python.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Jul 5, 2006, at 9:39 AM, skip at pobox.com wrote:

>
>     Barry> Clearly we need the "as if in" operator:
>
> Why not be more direct?

Sure, why not? :)

Then we can reserve the "as if" operator for those things that Guido  
has rejected, but that we sneak in while he's not looking.

like-omg-gag-me-with-a-spoon-ly y'rs,
- -Barry

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.3 (Darwin)

iQCVAwUBRKvK8nEjvBPtnXfVAQLbswQArfvIoCdCHmryk3qkOvG6BE0Q1iW7dk0O
eI178nG1tY+02JLyrPb1RcjdJG0W0wPwugvVNwVlz29cNkt048uEme6ZBfv3wCt/
bQSWTnDym/OWtQhUtsaw7V5K1o/bP5noqS2MQAcafk4lARv7TAWbBNkPqpk/yFmp
2yNhIfngjts=
=thYd
-----END PGP SIGNATURE-----

From brad.doctor at gmail.com  Wed Jul  5 17:21:10 2006
From: brad.doctor at gmail.com (Brad Doctor)
Date: Wed, 5 Jul 2006 09:21:10 -0600
Subject: [Python-Dev] Patch for commands.py to provide callback
Message-ID: <a07586b0607050821i10bc4acav8c2d3b4cfeb4ab99@mail.gmail.com>

Greetings all,

I have attached a patch for commands.py to provide a callback ability.
Example use:

-------------------------------------
import commands

cmd = 'top -b -n2'

def fancy(out):
    print 'GOT(%s)' % out.strip()

commands.cb = fancy

(s,o) = commands.getstatusoutput(cmd)
print 'OUTPUT (%s)' % o
-------------------------------------

I am not sure if this is the proper forum or means to submit something like
this, so please forgive me and advise accordingly if I am in error.  The
basic idea is obvious, to allow long-running commands to call back whenever
there is output.  This is against python 2.4.  Please let me know if you
have any questions or suggestions.

thanks!!
-brad
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20060705/62b26e37/attachment.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: python_commandsCallBack.diff
Type: text/x-patch
Size: 891 bytes
Desc: not available
Url : http://mail.python.org/pipermail/python-dev/attachments/20060705/62b26e37/attachment-0001.bin 

From fredrik at pythonware.com  Wed Jul  5 17:24:33 2006
From: fredrik at pythonware.com (Fredrik Lundh)
Date: Wed, 05 Jul 2006 17:24:33 +0200
Subject: [Python-Dev] Patch for commands.py to provide callback
In-Reply-To: <a07586b0607050821i10bc4acav8c2d3b4cfeb4ab99@mail.gmail.com>
References: <a07586b0607050821i10bc4acav8c2d3b4cfeb4ab99@mail.gmail.com>
Message-ID: <e8gljc$7t8$1@sea.gmane.org>

Brad Doctor wrote:

> I am not sure if this is the proper forum or means to submit something 
> like this, so please forgive me and advise accordingly if I am in error.

to make sure that they don't just disappear under a zillion other mails, 
patches should be submitted to the patch tracker:

     http://sourceforge.net/patch/?group_id=5470

</F>


From brad.doctor at gmail.com  Wed Jul  5 17:26:44 2006
From: brad.doctor at gmail.com (Brad Doctor)
Date: Wed, 5 Jul 2006 09:26:44 -0600
Subject: [Python-Dev] Patch for commands.py to provide callback
In-Reply-To: <e8gljc$7t8$1@sea.gmane.org>
References: <a07586b0607050821i10bc4acav8c2d3b4cfeb4ab99@mail.gmail.com>
	<e8gljc$7t8$1@sea.gmane.org>
Message-ID: <a07586b0607050826y68992981u9b1642369f584da@mail.gmail.com>

Cool, thank you Fredrik -- going there now.

-brad

On 7/5/06, Fredrik Lundh <fredrik at pythonware.com> wrote:
>
> Brad Doctor wrote:
>
> > I am not sure if this is the proper forum or means to submit something
> > like this, so please forgive me and advise accordingly if I am in error.
>
> to make sure that they don't just disappear under a zillion other mails,
> patches should be submitted to the patch tracker:
>
>      http://sourceforge.net/patch/?group_id=5470
>
> </F>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/brad.doctor%40gmail.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20060705/1c9a90e8/attachment.htm 

From guido at python.org  Wed Jul  5 17:31:57 2006
From: guido at python.org (Guido van Rossum)
Date: Wed, 5 Jul 2006 17:31:57 +0200
Subject: [Python-Dev] Patch for commands.py to provide callback
In-Reply-To: <a07586b0607050821i10bc4acav8c2d3b4cfeb4ab99@mail.gmail.com>
References: <a07586b0607050821i10bc4acav8c2d3b4cfeb4ab99@mail.gmail.com>
Message-ID: <ca471dc20607050831m4a6f216exdf4b047b179a7770@mail.gmail.com>

Since commands.getstatusoutput() is such a trivial wrapper around
os.popen(), why bother patching commands.py?

On 7/5/06, Brad Doctor <brad.doctor at gmail.com> wrote:
> Greetings all,
>
> I have attached a patch for commands.py to provide a callback ability.
> Example use:
>
> -------------------------------------
> import commands
>
> cmd = 'top -b -n2'
>
> def fancy(out):
>      print 'GOT(%s)' % out.strip()
>
> commands.cb = fancy
>
> (s,o) = commands.getstatusoutput(cmd)
> print 'OUTPUT (%s)' % o
> -------------------------------------
>
> I am not sure if this is the proper forum or means to submit something like
> this, so please forgive me and advise accordingly if I am in error.  The
> basic idea is obvious, to allow long-running commands to call back whenever
> there is output.  This is against python 2.4.  Please let me know if you
> have any questions or suggestions.
>
> thanks!!
> -brad
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/guido%40python.org
>
>
>
>


-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From brad.doctor at gmail.com  Wed Jul  5 17:35:50 2006
From: brad.doctor at gmail.com (Brad Doctor)
Date: Wed, 5 Jul 2006 09:35:50 -0600
Subject: [Python-Dev] Patch for commands.py to provide callback
In-Reply-To: <ca471dc20607050831m4a6f216exdf4b047b179a7770@mail.gmail.com>
References: <a07586b0607050821i10bc4acav8c2d3b4cfeb4ab99@mail.gmail.com>
	<ca471dc20607050831m4a6f216exdf4b047b179a7770@mail.gmail.com>
Message-ID: <a07586b0607050835l6da081a2x719e32b9c17d4c51@mail.gmail.com>

Because it is a great way to consistently use popen().  Rather than write
something specific each time, our site/company prefers to use commands to
keep it all consistent.

-brad

On 7/5/06, Guido van Rossum <guido at python.org> wrote:
>
> Since commands.getstatusoutput() is such a trivial wrapper around
> os.popen(), why bother patching commands.py?
>
> On 7/5/06, Brad Doctor <brad.doctor at gmail.com> wrote:
> > Greetings all,
> >
> > I have attached a patch for commands.py to provide a callback ability.
> > Example use:
> >
> > -------------------------------------
> > import commands
> >
> > cmd = 'top -b -n2'
> >
> > def fancy(out):
> >      print 'GOT(%s)' % out.strip()
> >
> > commands.cb = fancy
> >
> > (s,o) = commands.getstatusoutput(cmd)
> > print 'OUTPUT (%s)' % o
> > -------------------------------------
> >
> > I am not sure if this is the proper forum or means to submit something
> like
> > this, so please forgive me and advise accordingly if I am in error.  The
> > basic idea is obvious, to allow long-running commands to call back
> whenever
> > there is output.  This is against python 2.4.  Please let me know if you
> > have any questions or suggestions.
> >
> > thanks!!
> > -brad
> >
> > _______________________________________________
> > Python-Dev mailing list
> > Python-Dev at python.org
> > http://mail.python.org/mailman/listinfo/python-dev
> > Unsubscribe:
> > http://mail.python.org/mailman/options/python-dev/guido%40python.org
> >
> >
> >
> >
>
>
> --
> --Guido van Rossum (home page: http://www.python.org/~guido/)
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20060705/5eb2f554/attachment.html 

From fwierzbicki at gmail.com  Wed Jul  5 17:47:03 2006
From: fwierzbicki at gmail.com (Frank Wierzbicki)
Date: Wed, 5 Jul 2006 11:47:03 -0400
Subject: [Python-Dev] Import semantics
In-Reply-To: <200607051818.02742.anthony@interlink.com.au>
References: <cfb578b20606111531t6806d5c9kd35fd8ba29638174@mail.gmail.com>
	<4dab5f760606251334o718323adt409294e2952e954a@mail.gmail.com>
	<ca471dc20607050112n1bae4fd6i941999617bba5a27@mail.gmail.com>
	<200607051818.02742.anthony@interlink.com.au>
Message-ID: <4dab5f760607050847s6a128435v8ad31262e697ceef@mail.gmail.com>

> In that case, why not post a news item saying this? The website is
> probably the first place people look...
>
I think any news other than "here is the beta -- follow this link to
download it" would be kind of a waste at this point.  And that will
come when I finish __slots__, subclassable "type", and figure out how
to put a release together, which will be "real soon now" -- but
really.

Thanks,

-Frank

From guido at python.org  Wed Jul  5 17:50:10 2006
From: guido at python.org (Guido van Rossum)
Date: Wed, 5 Jul 2006 17:50:10 +0200
Subject: [Python-Dev] Patch for commands.py to provide callback
In-Reply-To: <a07586b0607050835l6da081a2x719e32b9c17d4c51@mail.gmail.com>
References: <a07586b0607050821i10bc4acav8c2d3b4cfeb4ab99@mail.gmail.com>
	<ca471dc20607050831m4a6f216exdf4b047b179a7770@mail.gmail.com>
	<a07586b0607050835l6da081a2x719e32b9c17d4c51@mail.gmail.com>
Message-ID: <ca471dc20607050850x7516d34ci187f5d848045341c@mail.gmail.com>

Hm. It sounds like your company would be better off developing a
library of handy tools that it uses over and over, instead of on the
one hand using only standard library, and then if the standard library
doesn't provide the feature you need, proposing a patch.

I wouldn't be so critical if I thought others would benefit from your
patch. But it seems a rather arbitrary place to insert a debugging
hook (if that's what you need it for), and I don't think I've seen
others with the same need.

Or perhaps you could start by explaining the problem you are trying to
solve before proposing a specific patch?

--Guido

On 7/5/06, Brad Doctor <brad.doctor at gmail.com> wrote:
> Because it is a great way to consistently use popen().  Rather than write
> something specific each time, our site/company prefers to use commands to
> keep it all consistent.
>
> -brad
>
>
>  On 7/5/06, Guido van Rossum <guido at python.org> wrote:
> > Since commands.getstatusoutput() is such a trivial wrapper around
> > os.popen(), why bother patching commands.py?
> >
> > On 7/5/06, Brad Doctor <brad.doctor at gmail.com> wrote:
> > > Greetings all,
> > >
> > > I have attached a patch for commands.py to provide a callback ability.
> > > Example use:
> > >
> > > -------------------------------------
> > > import commands
> > >
> > > cmd = 'top -b -n2'
> > >
> > > def fancy(out):
> > >      print 'GOT(%s)' % out.strip()
> > >
> > > commands.cb = fancy
> > >
> > > (s,o) = commands.getstatusoutput(cmd)
> > > print 'OUTPUT (%s)' % o
> > > -------------------------------------
> > >
> > > I am not sure if this is the proper forum or means to submit something
> like
> > > this, so please forgive me and advise accordingly if I am in error.  The
> > > basic idea is obvious, to allow long-running commands to call back
> whenever
> > > there is output.  This is against python 2.4.  Please let me know if you
> > > have any questions or suggestions.
> > >
> > > thanks!!
> > > -brad
> > >
> > > _______________________________________________
> > > Python-Dev mailing list
> > > Python-Dev at python.org
> > > http://mail.python.org/mailman/listinfo/python-dev
> > > Unsubscribe:
> > >
> http://mail.python.org/mailman/options/python-dev/guido%40python.org
> > >
> > >
> > >
> > >
> >
> >
> > --
> > --Guido van Rossum (home page: http://www.python.org/~guido/)
> >
>
>


-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From facundobatista at gmail.com  Wed Jul  5 17:51:47 2006
From: facundobatista at gmail.com (Facundo Batista)
Date: Wed, 5 Jul 2006 12:51:47 -0300
Subject: [Python-Dev] Time-out in URL Open
In-Reply-To: <ca471dc20607041458w22ee86cey99eb9bbe38e44f7f@mail.gmail.com>
References: <e04bdf310607031107ld304d14i7db52e555f8b5ddf@mail.gmail.com>
	<ca471dc20607031406q8ddaa9fhd7b0972b8d7a1412@mail.gmail.com>
	<e04bdf310607040843m33c36a34ka938685598a70f01@mail.gmail.com>
	<ca471dc20607041458w22ee86cey99eb9bbe38e44f7f@mail.gmail.com>
Message-ID: <e04bdf310607050851v7cf6a292wb3892cca4682c7a8@mail.gmail.com>

2006/7/4, Guido van Rossum <guido at python.org>:

> > This affect all the sockets.
>
> So, assuming your app is single-threaded, set the timeout, call
> urlopen(), and reset the timeout to None.

No, it's multithreaded, :D


> > And I hit the problem when servicing
> > information with a web service (TCPServer), and I need to timeout the
> > connection of the URLOpen *only*.
>
> That's not so easy even if you were to have a timeout parameter to
> urlopen(). You'd have to implement that peculiarity in all the layers
> (in this case, urllib and httplib; and possibly ftplib, gopherlib etc.
> :-)

Yes, it's not SO easy, because, as you said, you have to dig into the
layers until you hit the actual socket creation and modify the timeout
for that socket only.

That's why I think that this should be handled in the standard library
and not left to implement to whoever will need it, :)

-- 
.    Facundo

Blog: http://www.taniquetil.com.ar/plog/
PyAr: http://www.python.org/ar/

From pje at telecommunity.com  Wed Jul  5 17:53:50 2006
From: pje at telecommunity.com (Phillip J. Eby)
Date: Wed, 05 Jul 2006 11:53:50 -0400
Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
In-Reply-To: <ca471dc20607050121o48dd51a7v5f12ee15b9f67ea3@mail.gmail.co
 m>
References: <5.1.1.6.0.20060705020306.0207f890@sparrow.telecommunity.com>
	<44AA1FF6.1050501@acm.org>
	<ca471dc20607040728y1035e27ej3d6d589eaed00080@mail.gmail.com>
	<44AAE020.5060204@acm.org>
	<5.1.1.6.0.20060704182302.0207e890@sparrow.telecommunity.com>
	<5.1.1.6.0.20060705020306.0207f890@sparrow.telecommunity.com>
Message-ID: <5.1.1.6.0.20060705113910.04698e50@sparrow.telecommunity.com>

At 10:21 AM 7/5/2006 +0200, Guido van Rossum wrote:
>Thanks for bringing this up. I'm not sure what I think of it yet. One
>problem I see is that there might end up being two ways to reference
>variables in outer scopes: .num if you plan to assign to it, or just
>num if you only reference it. I find that the most disurbing issue so
>far; modified global declarations or outer declarations don't have
>this problem.

Well, you could make it mandatory in Py3K I suppose, though I'm not sure I 
like it being mandatory, due to the frequent need to reference top-level 
names that would cause an awful lot of dots to start popping up.

But for existing Python, the optional nature of the '.' allows existing 
code to run unchanged.  And for versions that support the new syntax, using 
a leading '.' for all non-global, non-locals should be considered "good 
style" since it highlights the dependency and practically shouts "tricky 
stuff here, pay attention."

Ironically, having *only* one way to refer to outer variables makes it 
impossible to communicate this distinction in the common read-only case.


>Would this also use ..num to refer to num in an outer scope two levels 
>removed?

I think that's unnecessary; it would be much better to use variables with 
distinct names.

By the way, an interesting thought for Py3K is that you could maybe use 
this syntax to do away with explicit 'self', if you consider the class' 
namespace to be part of a function's closure.

E.g.:

      class Foo:

          whee = 42

          def bar(baz):
              print .whee

Consider this: if Foo were a function rather than a class, each invocation 
of Foo would yield a new namespace in which 'whee' is defined.  However, 
each invocation of a *class* also yields a new namespace.  So there's a 
definite symmetry in using .whee to refer to an instance attribute of Foo.

The big problem that comes to mind with that idea is that it makes it 
impossible to have argument names that are the same as attribute names, 
unless the 'whee'/'.whee' prohibition were relaxed.  :(  But it's an 
intriguing thought, nonetheless.


From guido at python.org  Wed Jul  5 17:56:25 2006
From: guido at python.org (Guido van Rossum)
Date: Wed, 5 Jul 2006 17:56:25 +0200
Subject: [Python-Dev] Time-out in URL Open
In-Reply-To: <e04bdf310607050851v7cf6a292wb3892cca4682c7a8@mail.gmail.com>
References: <e04bdf310607031107ld304d14i7db52e555f8b5ddf@mail.gmail.com>
	<ca471dc20607031406q8ddaa9fhd7b0972b8d7a1412@mail.gmail.com>
	<e04bdf310607040843m33c36a34ka938685598a70f01@mail.gmail.com>
	<ca471dc20607041458w22ee86cey99eb9bbe38e44f7f@mail.gmail.com>
	<e04bdf310607050851v7cf6a292wb3892cca4682c7a8@mail.gmail.com>
Message-ID: <ca471dc20607050856m14cdb746i4a2e10c43569ab8@mail.gmail.com>

OK, you've convinced me. Now where's that SF patch you were promising? :-)

--Guido

On 7/5/06, Facundo Batista <facundobatista at gmail.com> wrote:
> 2006/7/4, Guido van Rossum <guido at python.org>:
>
> > > This affect all the sockets.
> >
> > So, assuming your app is single-threaded, set the timeout, call
> > urlopen(), and reset the timeout to None.
>
> No, it's multithreaded, :D
>
>
> > > And I hit the problem when servicing
> > > information with a web service (TCPServer), and I need to timeout the
> > > connection of the URLOpen *only*.
> >
> > That's not so easy even if you were to have a timeout parameter to
> > urlopen(). You'd have to implement that peculiarity in all the layers
> > (in this case, urllib and httplib; and possibly ftplib, gopherlib etc.
> > :-)
>
> Yes, it's not SO easy, because, as you said, you have to dig into the
> layers until you hit the actual socket creation and modify the timeout
> for that socket only.
>
> That's why I think that this should be handled in the standard library
> and not left to implement to whoever will need it, :)
>
> --
> .    Facundo
>
> Blog: http://www.taniquetil.com.ar/plog/
> PyAr: http://www.python.org/ar/
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org
>


-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From pje at telecommunity.com  Wed Jul  5 17:57:03 2006
From: pje at telecommunity.com (Phillip J. Eby)
Date: Wed, 05 Jul 2006 11:57:03 -0400
Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
In-Reply-To: <44AB8918.1040308@scottdial.com>
References: <ca471dc20607050121o48dd51a7v5f12ee15b9f67ea3@mail.gmail.com>
	<44AA1FF6.1050501@acm.org>
	<ca471dc20607040728y1035e27ej3d6d589eaed00080@mail.gmail.com>
	<44AAE020.5060204@acm.org>
	<5.1.1.6.0.20060704182302.0207e890@sparrow.telecommunity.com>
	<5.1.1.6.0.20060705020306.0207f890@sparrow.telecommunity.com>
	<ca471dc20607050121o48dd51a7v5f12ee15b9f67ea3@mail.gmail.com>
Message-ID: <5.1.1.6.0.20060705115525.045c4cb0@sparrow.telecommunity.com>

At 05:40 AM 7/5/2006 -0400, Scott Dial wrote:
>Guido van Rossum wrote:
> > Would this also use ..num to refer to num in an outer scope two levels 
> removed?
>
>I realize this was a wink, but it is a valid problem with the
>"dot"-proposal.

Actually, it isn't.  :)  See below.


>def foo(n):
>    def bar(n):
>      def baz():
>        return .n
>
>So, which 'n' outer 'n' is being referenced?

Notice that this is a made-up example.  Existing Python scoping rules don't 
allow this!  Thus your example is not a bug, it's a feature request.  And I 
say we say "no" to adding this feature.  ;-)


From guido at python.org  Wed Jul  5 17:58:15 2006
From: guido at python.org (Guido van Rossum)
Date: Wed, 5 Jul 2006 17:58:15 +0200
Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
In-Reply-To: <5.1.1.6.0.20060705113910.04698e50@sparrow.telecommunity.com>
References: <44AA1FF6.1050501@acm.org>
	<ca471dc20607040728y1035e27ej3d6d589eaed00080@mail.gmail.com>
	<44AAE020.5060204@acm.org>
	<5.1.1.6.0.20060704182302.0207e890@sparrow.telecommunity.com>
	<5.1.1.6.0.20060705020306.0207f890@sparrow.telecommunity.com>
	<5.1.1.6.0.20060705113910.04698e50@sparrow.telecommunity.com>
Message-ID: <ca471dc20607050858n41efb3a3le1c6e43868f9d0d4@mail.gmail.com>

On 7/5/06, Phillip J. Eby <pje at telecommunity.com> wrote:
> By the way, an interesting thought for Py3K is that you could maybe use
> this syntax to do away with explicit 'self', if you consider the class'
> namespace to be part of a function's closure.

Sorry, but now I am *definitely* -1.

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From fwierzbicki at gmail.com  Wed Jul  5 18:21:31 2006
From: fwierzbicki at gmail.com (Frank Wierzbicki)
Date: Wed, 5 Jul 2006 12:21:31 -0400
Subject: [Python-Dev] Import semantics
In-Reply-To: <ca471dc20607050112n1bae4fd6i941999617bba5a27@mail.gmail.com>
References: <cfb578b20606111531t6806d5c9kd35fd8ba29638174@mail.gmail.com>
	<448D1F0D.7000405@strakt.com>
	<ca471dc20606120948h119f1c1fw2725e7d434e287df@mail.gmail.com>
	<4dab5f760606251334o718323adt409294e2952e954a@mail.gmail.com>
	<ca471dc20607050112n1bae4fd6i941999617bba5a27@mail.gmail.com>
Message-ID: <4dab5f760607050921h548cf2fdx9f125599ccbe8bfc@mail.gmail.com>

On 7/5/06, Guido van Rossum <guido at python.org> wrote:
> Hi Frank,
>
> Have you and/or the Jython community made up your mind about this? The
> thread seems to have disappeared after you posted (or perhaps it
> continued only on jython-dev, which I don't read?).
The thread pretty much stopped there.  I think a modification of the
import semantics won't really come up again until 2.3 is out, since
releases and reviving Jython take priority over backwards incompatible
features.  For my part I would like to think that a goal for Jython
should be:

Pure Python code developed first on Jython should run without change on CPython.

Which would mean we will eventually need to change the import
semantics for importing Python code (though I doubt we will change the
semantics for importing Java packages any time soon).  Whether that
can be done in 2.x, or if this change is so incompatible that we need
to think about it in a "Jython 3000" way, I don't really know.

>
> Also, I just realized that you're the new Jython maintainer. Is *that*
> official?
It is official, at least in the unofficial way that Jython appears to
work: Brian handed the baton to me after (I presume) Samuele Pedroni
had handed the baton to Brian.

Brian's email is here:
http://sourceforge.net/mailarchive/message.php?msg_id=13859029

That said, I still regard Samuele Pedroni as the ultimate authority on
Jython and give him pretty much full veto power.  He fortunately
continues to watch the checkins and prods me when I go in the wrong
direction.

> I'd like to offer you my congratulations, and, more
> importantly, any support you might need.
Thanks!

> I find Jython an important
> part for Python's long-term stategy.
That's good to know.

> I'm asked occasionally what the
> status of Jython is; people point out that the last release was 2.1
> many years ago and the website has no news since early 2005; they're
> afraid that Jython is dying and that it's not a viable choice for new
> projects.
> I'm very happy to be able to tell them that soon there will
> be a 2.3 release and yes there *is* continued support...
Perhaps for large values of "soon" -- but seriously, I am working hard
to polish the coming release, make it easier for new developers to
read the code, and when I have a chance, update the website.  Jython
is my first serious plunge into open source contributions (it is
really too bad that I couldn't have been a journeyman for another year
or so first, but circumstances did not allow that).  I have suffered
from some over-optimism when asked for release dates, so I'm really
afraid to give too sharp of a definition for "soon".  That said, I
believe a 2.2 version will be out sometime this summer, and a 2.3
should follow relatively quickly (maybe 6 months or so)

The 2.2->2.3 will (hopefully) be relatively quick because 2.2 is an
unfortunate but at this point unavoidable mix of 2.2 and 2.3 features,
with heavy favoring of 2.3 features.

Jython has had a history of having a very small number of developers,
and a history of too much off-list discussions so that new developers
have a very hard time catching up.  I'm trying hard to change that
culture.  There are 4 or 5 developers contributing patches lately --
so I'm actually spending more time trying to help them along instead
of concentrating 100% on a release -- I really think they are the most
important predictor of the future of Jython.  I have high hopes for
the project's future health.

Anyhow, this post is getting long, so to sum up:

Jython is alive, and we aren't going to let it die.

Regards,

-Frank

From aleaxit at gmail.com  Wed Jul  5 18:42:54 2006
From: aleaxit at gmail.com (Alex Martelli)
Date: Wed, 5 Jul 2006 09:42:54 -0700
Subject: [Python-Dev] Time-out in URL Open
In-Reply-To: <ca471dc20607050856m14cdb746i4a2e10c43569ab8@mail.gmail.com>
References: <e04bdf310607031107ld304d14i7db52e555f8b5ddf@mail.gmail.com>
	<ca471dc20607031406q8ddaa9fhd7b0972b8d7a1412@mail.gmail.com>
	<e04bdf310607040843m33c36a34ka938685598a70f01@mail.gmail.com>
	<ca471dc20607041458w22ee86cey99eb9bbe38e44f7f@mail.gmail.com>
	<e04bdf310607050851v7cf6a292wb3892cca4682c7a8@mail.gmail.com>
	<ca471dc20607050856m14cdb746i4a2e10c43569ab8@mail.gmail.com>
Message-ID: <e8a0972d0607050942m2aed7c5dvb56f3956636029f7@mail.gmail.com>

What about doing it with a per-thread-timeout in TLS (overriding the
global one if a thread does have it set in its TLS)?  Not as clean,
but perhaps far easier to implement than patching dozens of
modules/functions/classes to provide timeout= options everywhere...


Alex


On 7/5/06, Guido van Rossum <guido at python.org> wrote:
> OK, you've convinced me. Now where's that SF patch you were promising? :-)
>
> --Guido
>
> On 7/5/06, Facundo Batista <facundobatista at gmail.com> wrote:
> > 2006/7/4, Guido van Rossum <guido at python.org>:
> >
> > > > This affect all the sockets.
> > >
> > > So, assuming your app is single-threaded, set the timeout, call
> > > urlopen(), and reset the timeout to None.
> >
> > No, it's multithreaded, :D
> >
> >
> > > > And I hit the problem when servicing
> > > > information with a web service (TCPServer), and I need to timeout the
> > > > connection of the URLOpen *only*.
> > >
> > > That's not so easy even if you were to have a timeout parameter to
> > > urlopen(). You'd have to implement that peculiarity in all the layers
> > > (in this case, urllib and httplib; and possibly ftplib, gopherlib etc.
> > > :-)
> >
> > Yes, it's not SO easy, because, as you said, you have to dig into the
> > layers until you hit the actual socket creation and modify the timeout
> > for that socket only.
> >
> > That's why I think that this should be handled in the standard library
> > and not left to implement to whoever will need it, :)
> >
> > --
> > .    Facundo
> >
> > Blog: http://www.taniquetil.com.ar/plog/
> > PyAr: http://www.python.org/ar/
> > _______________________________________________
> > Python-Dev mailing list
> > Python-Dev at python.org
> > http://mail.python.org/mailman/listinfo/python-dev
> > Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org
> >
>
>
> --
> --Guido van Rossum (home page: http://www.python.org/~guido/)
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/aleaxit%40gmail.com
>

From skip at pobox.com  Wed Jul  5 18:49:22 2006
From: skip at pobox.com (skip at pobox.com)
Date: Wed, 5 Jul 2006 11:49:22 -0500
Subject: [Python-Dev] Time-out in URL Open
In-Reply-To: <ca471dc20607050856m14cdb746i4a2e10c43569ab8@mail.gmail.com>
References: <e04bdf310607031107ld304d14i7db52e555f8b5ddf@mail.gmail.com>
	<ca471dc20607031406q8ddaa9fhd7b0972b8d7a1412@mail.gmail.com>
	<e04bdf310607040843m33c36a34ka938685598a70f01@mail.gmail.com>
	<ca471dc20607041458w22ee86cey99eb9bbe38e44f7f@mail.gmail.com>
	<e04bdf310607050851v7cf6a292wb3892cca4682c7a8@mail.gmail.com>
	<ca471dc20607050856m14cdb746i4a2e10c43569ab8@mail.gmail.com>
Message-ID: <17579.60818.203745.712478@montanaro.dyndns.org>


    Guido> OK, you've convinced me. Now where's that SF patch you were
    Guido> promising? :-)

A starting point is probably the patch Georg referred to a couple days ago:

    Georg> There was one patch that did this: http://python.org/sf/723312.

Alas, it's assigned to me and I let it get so stale that Martin asked the
author to update it for 2.5 a couple months ago.

Skip

From skip at pobox.com  Wed Jul  5 18:53:13 2006
From: skip at pobox.com (skip at pobox.com)
Date: Wed, 5 Jul 2006 11:53:13 -0500
Subject: [Python-Dev] Import semantics
In-Reply-To: <4dab5f760607050921h548cf2fdx9f125599ccbe8bfc@mail.gmail.com>
References: <cfb578b20606111531t6806d5c9kd35fd8ba29638174@mail.gmail.com>
	<448D1F0D.7000405@strakt.com>
	<ca471dc20606120948h119f1c1fw2725e7d434e287df@mail.gmail.com>
	<4dab5f760606251334o718323adt409294e2952e954a@mail.gmail.com>
	<ca471dc20607050112n1bae4fd6i941999617bba5a27@mail.gmail.com>
	<4dab5f760607050921h548cf2fdx9f125599ccbe8bfc@mail.gmail.com>
Message-ID: <17579.61049.476643.960849@montanaro.dyndns.org>


    Frank> That said, I still regard Samuele Pedroni as the ultimate
    Frank> authority on Jython and give him pretty much full veto power.  He
    Frank> fortunately continues to watch the checkins and prods me when I
    Frank> go in the wrong direction.

Does that make Samele the DBPV (Dictator benevolo per vita)? ;-)

Skip

From aleaxit at gmail.com  Wed Jul  5 18:54:43 2006
From: aleaxit at gmail.com (Alex Martelli)
Date: Wed, 5 Jul 2006 09:54:43 -0700
Subject: [Python-Dev] Import semantics
In-Reply-To: <17579.61049.476643.960849@montanaro.dyndns.org>
References: <cfb578b20606111531t6806d5c9kd35fd8ba29638174@mail.gmail.com>
	<448D1F0D.7000405@strakt.com>
	<ca471dc20606120948h119f1c1fw2725e7d434e287df@mail.gmail.com>
	<4dab5f760606251334o718323adt409294e2952e954a@mail.gmail.com>
	<ca471dc20607050112n1bae4fd6i941999617bba5a27@mail.gmail.com>
	<4dab5f760607050921h548cf2fdx9f125599ccbe8bfc@mail.gmail.com>
	<17579.61049.476643.960849@montanaro.dyndns.org>
Message-ID: <e8a0972d0607050954g2e4d2fecucd84acec7e9d753b@mail.gmail.com>

In Italian that would be DBAV (Dittatore benevolo a vita)...;-)


Alex


On 7/5/06, skip at pobox.com <skip at pobox.com> wrote:
>
>     Frank> That said, I still regard Samuele Pedroni as the ultimate
>     Frank> authority on Jython and give him pretty much full veto power.  He
>     Frank> fortunately continues to watch the checkins and prods me when I
>     Frank> go in the wrong direction.
>
> Does that make Samele the DBPV (Dictator benevolo per vita)? ;-)
>
> Skip
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/aleaxit%40gmail.com
>

From fwierzbicki at gmail.com  Wed Jul  5 19:13:39 2006
From: fwierzbicki at gmail.com (Frank Wierzbicki)
Date: Wed, 5 Jul 2006 13:13:39 -0400
Subject: [Python-Dev] Import semantics
In-Reply-To: <17579.61049.476643.960849@montanaro.dyndns.org>
References: <cfb578b20606111531t6806d5c9kd35fd8ba29638174@mail.gmail.com>
	<448D1F0D.7000405@strakt.com>
	<ca471dc20606120948h119f1c1fw2725e7d434e287df@mail.gmail.com>
	<4dab5f760606251334o718323adt409294e2952e954a@mail.gmail.com>
	<ca471dc20607050112n1bae4fd6i941999617bba5a27@mail.gmail.com>
	<4dab5f760607050921h548cf2fdx9f125599ccbe8bfc@mail.gmail.com>
	<17579.61049.476643.960849@montanaro.dyndns.org>
Message-ID: <4dab5f760607051013k3d29f39er2583d075225e7635@mail.gmail.com>

On 7/5/06, skip at pobox.com <skip at pobox.com> wrote:
>
>     Frank> That said, I still regard Samuele Pedroni as the ultimate
>     Frank> authority on Jython and give him pretty much full veto power.  He
>     Frank> fortunately continues to watch the checkins and prods me when I
>     Frank> go in the wrong direction.
>
> Does that make Samele the DBPV (Dictator benevolo per vita)? ;-)
>
> Skip
>
I wonder if Samuele even wants that role?

Anyway, I believe Samuele has more experience with Jython as it is
currently implemented than anyone else, so I must take anything he
says about Jython very seriously.  However, when it comes to pure
*Python* matters (no C and no J) which includes half of Jython's
import semantics, I think it is still Guido's opinion that matters
most.  Without that, there is too much chaos.

-Frank

From mcherm at mcherm.com  Wed Jul  5 19:18:21 2006
From: mcherm at mcherm.com (Michael Chermside)
Date: Wed, 05 Jul 2006 10:18:21 -0700
Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
Message-ID: <20060705101821.7sqnxo7lu5cgg4ks@login.werra.lunarpages.com>

Guido writes:
   [discussion of how to fix the can't-bind-outer-scope-vars wart]
> I think we have to continue to search for a solution that extends the
> idea of global declarations.
>
> I've proposed extending its meaning to refer to the nearest outer
> scope where the variable is set; if there is no such scope it's an
> error. This will break a small number of program but probably not very
> many; still, it'll require a future statement or waiting until Python
> 3.0. The downside is that "global" is not a very intuitive word for
> this new meaning.

I disagree with your last statement -- I think "global" _is_ a very
intuitive word for this. As I understand it, in programming "global"
has two meanings, closely intertwined. One is "universal, same
throughout the system". For instance, "The singleton pattern is used
to create a single, global instance of a type." The second meaning is
the term "global variable". This term developed (I believe) in
languages that had only two scopes: local-to-current-function and
global-to-entire-program. But the term "global variable" refers to
any variable whose assignment is a "side effect", regardless of
whether that variable is global-to-entire-program, global-to-module,
or even global-to-enclosing-function. I have even heard the term
"global variable" (mis)used to refer to any kind of side effect.

Anyhow, in Python only builtins is _really_ global -- even today's
global keyword only refers to module scope. So I believe that it
would be a very reasonable interpretation of "global" to mean
"not local", and implement as "search enclosing scopes in order
to find the binding".

-- Michael Chermside


From skip at pobox.com  Wed Jul  5 19:19:53 2006
From: skip at pobox.com (skip at pobox.com)
Date: Wed, 5 Jul 2006 12:19:53 -0500
Subject: [Python-Dev] Import semantics
In-Reply-To: <e8a0972d0607050954g2e4d2fecucd84acec7e9d753b@mail.gmail.com>
References: <cfb578b20606111531t6806d5c9kd35fd8ba29638174@mail.gmail.com>
	<448D1F0D.7000405@strakt.com>
	<ca471dc20606120948h119f1c1fw2725e7d434e287df@mail.gmail.com>
	<4dab5f760606251334o718323adt409294e2952e954a@mail.gmail.com>
	<ca471dc20607050112n1bae4fd6i941999617bba5a27@mail.gmail.com>
	<4dab5f760607050921h548cf2fdx9f125599ccbe8bfc@mail.gmail.com>
	<17579.61049.476643.960849@montanaro.dyndns.org>
	<e8a0972d0607050954g2e4d2fecucd84acec7e9d753b@mail.gmail.com>
Message-ID: <17579.62649.924278.973929@montanaro.dyndns.org>


    Skip> Does that make Samele the DBPV (Dictator benevolo per vita)? ;-)

    Alex> In Italian that would be DBAV (Dittatore benevolo a vita)...;-)

Damn Google Translator.  File a bug report for me please Alex (or Guido or
Jeremy or Neal or ...). ;-)

Skip

From guido at python.org  Wed Jul  5 19:27:41 2006
From: guido at python.org (Guido van Rossum)
Date: Wed, 5 Jul 2006 19:27:41 +0200
Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
In-Reply-To: <20060705101821.7sqnxo7lu5cgg4ks@login.werra.lunarpages.com>
References: <20060705101821.7sqnxo7lu5cgg4ks@login.werra.lunarpages.com>
Message-ID: <ca471dc20607051027y677c7127wfe098401c366dc74@mail.gmail.com>

On 7/5/06, Michael Chermside <mcherm at mcherm.com> wrote:
> Guido writes:
>    [discussion of how to fix the can't-bind-outer-scope-vars wart]
> > I think we have to continue to search for a solution that extends the
> > idea of global declarations.
> >
> > I've proposed extending its meaning to refer to the nearest outer
> > scope where the variable is set; if there is no such scope it's an
> > error. This will break a small number of program but probably not very
> > many; still, it'll require a future statement or waiting until Python
> > 3.0. The downside is that "global" is not a very intuitive word for
> > this new meaning.
>
> I disagree with your last statement -- I think "global" _is_ a very
> intuitive word for this. As I understand it, in programming "global"
> has two meanings, closely intertwined. One is "universal, same
> throughout the system". For instance, "The singleton pattern is used
> to create a single, global instance of a type." The second meaning is
> the term "global variable". This term developed (I believe) in
> languages that had only two scopes: local-to-current-function and
> global-to-entire-program. But the term "global variable" refers to
> any variable whose assignment is a "side effect", regardless of
> whether that variable is global-to-entire-program, global-to-module,
> or even global-to-enclosing-function. I have even heard the term
> "global variable" (mis)used to refer to any kind of side effect.
>
> Anyhow, in Python only builtins is _really_ global -- even today's
> global keyword only refers to module scope. So I believe that it
> would be a very reasonable interpretation of "global" to mean
> "not local", and implement as "search enclosing scopes in order
> to find the binding".

I really wish I could agree with you, because that would make the
choice so much easier.

However I still don't believe "global" has the stretchiness in its
meaning that you claim it has. Have you ever heard a Python programmer
talking about closures use the word "global variable"?

Are there any other native speakers who side with Michael?

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From guido at python.org  Wed Jul  5 19:29:59 2006
From: guido at python.org (Guido van Rossum)
Date: Wed, 5 Jul 2006 19:29:59 +0200
Subject: [Python-Dev] Import semantics
In-Reply-To: <4dab5f760607051013k3d29f39er2583d075225e7635@mail.gmail.com>
References: <cfb578b20606111531t6806d5c9kd35fd8ba29638174@mail.gmail.com>
	<448D1F0D.7000405@strakt.com>
	<ca471dc20606120948h119f1c1fw2725e7d434e287df@mail.gmail.com>
	<4dab5f760606251334o718323adt409294e2952e954a@mail.gmail.com>
	<ca471dc20607050112n1bae4fd6i941999617bba5a27@mail.gmail.com>
	<4dab5f760607050921h548cf2fdx9f125599ccbe8bfc@mail.gmail.com>
	<17579.61049.476643.960849@montanaro.dyndns.org>
	<4dab5f760607051013k3d29f39er2583d075225e7635@mail.gmail.com>
Message-ID: <ca471dc20607051029n44ca5c98l5414f388d4060aba@mail.gmail.com>

On 7/5/06, Frank Wierzbicki <fwierzbicki at gmail.com> wrote:
> On 7/5/06, skip at pobox.com <skip at pobox.com> wrote:
> >
> >     Frank> That said, I still regard Samuele Pedroni as the ultimate
> >     Frank> authority on Jython and give him pretty much full veto power.  He
> >     Frank> fortunately continues to watch the checkins and prods me when I
> >     Frank> go in the wrong direction.
> >
> > Does that make Samele the DBPV (Dictator benevolo per vita)? ;-)
> >
> > Skip
> >
> I wonder if Samuele even wants that role?
>
> Anyway, I believe Samuele has more experience with Jython as it is
> currently implemented than anyone else, so I must take anything he
> says about Jython very seriously.  However, when it comes to pure
> *Python* matters (no C and no J) which includes half of Jython's
> import semantics, I think it is still Guido's opinion that matters
> most.  Without that, there is too much chaos.

Actually, I started this discussion because Samuele told me that there
was a new Jython maintainer, and I jumped out of my chair of joy.
Samuele is happy to give advice and otherwise help out when asked, but
he also seems happy to delegate everything to you, which sounds like a
great endorsement.

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From mcherm at mcherm.com  Wed Jul  5 19:31:23 2006
From: mcherm at mcherm.com (Michael Chermside)
Date: Wed, 05 Jul 2006 10:31:23 -0700
Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
Message-ID: <20060705103123.fvv54v7e3s0ko8wo@login.werra.lunarpages.com>

Phillip Eby writes:
> I don't see a problem with requiring '.x' to be used for both  
> reading and writing of outer-scope names; it just shouldn't be  
> required for an outer-scope name that you don't rebind in the  
> current scope.
>
>      def counter(num):
>          def inc():
>              .num += 1
>              return .num
>          return inc

I am reminded of Tim Peter's declaration in response to a similar
proposal some time ago:

    Syntax should not look like grit on my monitor.

(Sorry, no reference... but I swear it's word-for-word accurate because
the quote burned itself into my memory.)

-- Michael Chermside


From mcherm at mcherm.com  Wed Jul  5 19:31:25 2006
From: mcherm at mcherm.com (Michael Chermside)
Date: Wed, 05 Jul 2006 10:31:25 -0700
Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
Message-ID: <20060705103125.4rc33rnnzg8w0w4c@login.werra.lunarpages.com>

Phillip Eby writes:
> The big problem that comes to mind with that idea is that it makes  
> it impossible to have argument names that are the same as attribute  
> names, unless the 'whee'/'.whee' prohibition were relaxed.  :(  But  
> it's an intriguing thought, nonetheless.

My three-year-old has been working on that 'whee'/'.whee' prohibition,
but he hasn't mastered it yet.

Gotta-go-wash-another-load-of-underpants -lly yours,

Michael Chermside


From pje at telecommunity.com  Wed Jul  5 19:43:55 2006
From: pje at telecommunity.com (Phillip J. Eby)
Date: Wed, 05 Jul 2006 13:43:55 -0400
Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
In-Reply-To: <ca471dc20607051027y677c7127wfe098401c366dc74@mail.gmail.co
 m>
References: <20060705101821.7sqnxo7lu5cgg4ks@login.werra.lunarpages.com>
	<20060705101821.7sqnxo7lu5cgg4ks@login.werra.lunarpages.com>
Message-ID: <5.1.1.6.0.20060705133753.026a7698@sparrow.telecommunity.com>

At 07:27 PM 7/5/2006 +0200, Guido van Rossum wrote:
>However I still don't believe "global" has the stretchiness in its
>meaning that you claim it has. Have you ever heard a Python programmer
>talking about closures use the word "global variable"?
>
>Are there any other native speakers who side with Michael?

I don't, and am -1 on using "global" to mean "outer".

Of course, I also don't really care for using "global" to refer to a module 
level variable, or it being the only non-executable statement in Python, 
but these are very minor things and relatively easily explained to the 
programmers I've worked with.  Using "global" to mean "outer", on the other 
hand, would surely lead to much wailing and grinding of teeth in response 
to any attempt at rational explanation.  :)


From nick at craig-wood.com  Wed Jul  5 19:57:33 2006
From: nick at craig-wood.com (Nick Craig-Wood)
Date: Wed, 5 Jul 2006 18:57:33 +0100
Subject: [Python-Dev] Time-out in URL Open
In-Reply-To: <e8a0972d0607050942m2aed7c5dvb56f3956636029f7@mail.gmail.com>
References: <e04bdf310607031107ld304d14i7db52e555f8b5ddf@mail.gmail.com>
	<ca471dc20607031406q8ddaa9fhd7b0972b8d7a1412@mail.gmail.com>
	<e04bdf310607040843m33c36a34ka938685598a70f01@mail.gmail.com>
	<ca471dc20607041458w22ee86cey99eb9bbe38e44f7f@mail.gmail.com>
	<e04bdf310607050851v7cf6a292wb3892cca4682c7a8@mail.gmail.com>
	<ca471dc20607050856m14cdb746i4a2e10c43569ab8@mail.gmail.com>
	<e8a0972d0607050942m2aed7c5dvb56f3956636029f7@mail.gmail.com>
Message-ID: <20060705175733.3A06114C1BD@irishsea.home.craig-wood.com>

Alex Martelli <aleaxit at gmail.com> wrote:
>  What about doing it with a per-thread-timeout in TLS (overriding the
>  global one if a thread does have it set in its TLS)?  Not as clean,
>  but perhaps far easier to implement than patching dozens of
>  modules/functions/classes to provide timeout= options everywhere...

Yes please!

I wrote a sketch of a module which did this on c.l.py recently

  http://groups.google.com/group/comp.lang.python/browse_thread/thread/d897c00b67cadca5/fd2ceb4e014de7ce?lnk=st&q=TimeoutError&rnum=2&hl=en#fd2ceb4e014de7ce

It would be much better if it had help from the core though.

-- 
Nick Craig-Wood <nick at craig-wood.com> -- http://www.craig-wood.com/nick

From brett at python.org  Wed Jul  5 20:56:09 2006
From: brett at python.org (Brett Cannon)
Date: Wed, 5 Jul 2006 11:56:09 -0700
Subject: [Python-Dev] doc for new restricted execution design for Python
In-Reply-To: <Pine.LNX.4.58.0607040428210.17937@server1.LFW.org>
References: <bbaeab100606211733x3dd58226g15e2919e8e2cdd7e@mail.gmail.com>
	<Pine.LNX.4.58.0607040428210.17937@server1.LFW.org>
Message-ID: <bbaeab100607051156h4c341055w9377fb73a0f0c6fc@mail.gmail.com>

On 7/4/06, Ka-Ping Yee <python-dev at zesty.ca> wrote:
>
> Hi Brett,
>
> Here are some comments on the description of the restricted execution
> model that you posted.
>
> > When referring to the state of an interpreter, it is either "trusted" or
> > "untrusted".  A trusted interpreter has no restrictions imposed upon any
> > resource.  An untrusted interpreter has at least one, possibly more,
> resource
> > with a restriction placed upon it.
>
> In response to Guido's comment about confusing the words "trusted" and
> "untrusted", how about "empowered" and "restricted"?


Maybe.  I am really starting to lean towards trusted and sandboxed.

> When the Interpreter Is Embedded
> > ================================
> >
> > Single Untrusted Interpreter
> > ----------------------------
> >
> > This use case is when an application embeds the interpreter and never
> has more
> > than one interpreter running.
> >
> > The main security issue to watch out for is not having default abilities
> be
> > provided to the interpreter by accident.
>
> I'd rather rephrase this in the opposite direction.  The onus shouldn't
> be on the application to hunt down each possible dangerous authority and
> deactivate them all one by one.  The main security issue is to let the
> application choose which abilities it wants the restricted interpreter
> to have, and then ensure that the restricted interpreter gets only those
> abilities.


Right.  I am thinking more of an implementation screw up that somehow
provides access to an object that has escalated rights.

> Multiple Untrusted Interpreters
> > -------------------------------
> >
> > When multiple interpreters, all untrusted at varying levels, need to be
> > running within a single application.  This is the key use case that this
> > proposed design is targetted for.
> >
> > On top of the security issues from a single untrusted interpreter,
> > there is one additional worry.  Resources cannot end up being leaked
> > into other interpreters where they are given escalated rights.
>
> What is your model here for communication between interpreters?  If two
> interpreters can communicate, any attempt to "prevent leakage" of
> resources is meaningless.  When you say "leaked into other interpreters"
> are you talking about a Python object leaking or something else at a
> lower level?


I am talking about Python objects.

As for communication, I was planning on something included directly in
globals or some custom object to handle that.  I have not been focusing on
that aspect so far.

Suppose for example that the application wants to embed two interpreters,
> P and Q, and that the application wants P to be able to write files but
> Q to be restricted against writing files.  When you say "leaked" above,
> that suggests to me that you want to prevent something like
>
>     # code running in P
>     import spam
>     f = open('/home/doofus/.ssh/authorized_keys', 'a')
>     spam.f = f
>
>     # code running in Q
>     import spam
>     spam.f.write('blargh')
>
> The above example supposes that P and Q can communicate through a
> shared module, spam, where they can pass Python objects.


Right.  But Python modules are separate per interpreter and only C extension
modules are in any way shared between interpreters.  But sharing an open
file like that is bad and why C extension modules must be whitelisted to be
used.

But notice that even if you prevent them from passing Python objects
> like open files, any form of communication is sufficient to leak
> resources:
>
>     # code running in P
>     def add_key(key):
>         f = open('/home/doofus/.ssh/authorized_keys', 'a')
>         f.write(key + '\n')
>         f.close()
>
>     import socket
>     s = socket.socket()
>     s.bind(('', 6666))
>     s.listen(1)
>     ns, addr = s.accept()
>     add_key(ns.recv(100))
>
>
>     # code running in Q
>     import webbrowser
>     webbrowser.open('http://localhost:6666/zebra')
>
> As long as P can listen for instructions from Q, it can give Q
> the power to write to the filesystem.


Right, which is why sockets and files are restricted and turned off by
default.  You have to give explicit permission to use either resource.

> Filesystem
> > ===================
> >
> > The most obvious facet of a filesystem to protect is reading from it.
> > One does not want what is stored in ``/etc/passwd`` to get out.  And
> > one also does not want writing to the disk unless explicitly allowed
> > for basically the same reason; if someone can write ``/etc/passwd``
> > then they can set the password for the root account.
>
> There's a big difference between modifying (or erasing) an existing file
> and writing a new file (e.g. for temporary storage).  If i give you a
> little filesystem of your own to play in, and it starts out empty, you
> can put whatever you want in it without violating my secrecy or the
> integrity of my files.
>
> I think you should be talking about this in terms of specifically
> what abilities you want to be able to allow, based on examples of
> real-life applications.


Fair enough.  But since you have the ability to only list files
specifically, you can give temporary file access by giving access to such a
non-existent file for writing.  If you don't like an existing file then you
don't get access to it.

> Physical Resources
> > ===================
> >
> > Memory should be protected.  It is a limited resource on the system
> > that can have an impact on other running programs if it is exhausted.
> > Being able to restrict the use of memory would help alleviate issues
> > from denial-of-service (DoS) attacks.
>
> > Networking
> > ===================
> >
> > Networking is somewhat like the filesystem in terms of wanting similar
> > protections.  You do not want to let untrusted code make tons of socket
> > connections or accept them to do possibly nefarious things (e.g., acting
> > as a zombie).
> >
> > You also want to prevent finding out information about the network you
> are
> > connected to.  This includes doing DNS resolution since that allows one
> > to find out what addresses your intranet has or what subnets you use.
>
> Again, it's risky to describe only individual cases of things to
> prevent.  What networking abilities are safe or necessary for the
> kinds of applications you have in mind?  Start from nothing and
> work up from there.


That's the plan.  I am planning to go through socket function by function
and explicitly allow access as warranted and block everything else.  It is
not going to be "let's block DNS and allow everything else".  Sorry if that
wasn't clear.  This is mostly just to say "I plan on restricting this kind
of stuff, here is an example".

> Interpreter
> > ===================
> >
> > One must make sure that the interpreter is not harmed in any way.
> > There are several ways to possibly do this.  One is generating
> > hostile bytecode.  Another is some buffer overflow.  In general any
> > ability to crash the interpreter is unacceptable.
>
> This is hard for me to understand.  What exactly do you trust and
> not trust?  It seems to me that crashing an interpreter is only a
> problem if a single interpreter is running both trusted and untrusted
> code -- then if the untrusted code crashes the interpreter, the
> trusted code suffers.
>
> But there doesn't seem to be any such thing in your model.  Each
> interpreter is either trusted or untrusted.  If the interpreter is
> trusted, and the code running in it causes it to crash, i assume
> you would consider that to be the code's "own fault", right?
> And if the interpreter is untrusted, and the code running in it
> causes it to crash, then the code has only harmed itself.
>
> It seems to me that we need only be concerned about crashing when
> the crash of an embedded interpreter will bring down its host
> application, or there are multiple interpreters embedded at once
> and one interpreter causes another interpreter to crash.


Right.  But being embedded, won't any segfault of an interpreter bring down
the embedded application?

But you are correct, I am only concerned with preventing a crash of a
sandboxed interperter.

> Resource Hiding
> > =============================
> [...]
> > This can be viewed as a passive system for security.
> [...]
> > Resource Crippling
> > =============================
> > Another approach to security is to provide constant, proactive security
> > checking of rights to use a resource.
>
> I think you have this backwards.  Resource hiding is proactive:
> before untrusted code has a chance to abuse anything, you decide
> what you want to allow it to do.  It defaults to no access, and
> only gets access to resources you have proactively decided to provide.


I am using "proactive" as in constantly checking the security model.

Resource crippling is the opposite: it begins by giving carte blanche
> to the untrusted code, then you run around trying to plug holes
> by stopping everything you don't want.  This is a lot more work,
> and it is also much more dangerous.  If you forget to plug even
> one hole, you're hosed.


Yeah, I know, which is why I am only bothering with 'file' and 'socket'.

Back to what you wrote about resource hiding:
>
> > This can be viewed as a passive system for security.  Once a resource
> > has been given to code there are no more checks to make sure the
> > security model is being violated.
>
> This last sentence doesn't make any sense.  If you decided to give
> the resource, how is using the resource a violation?  Either you
> want to enable the resource or you don't.  If you want to enable
> it, give it; if you don't, don't give it.  As a criticism of the
> resource hiding approach, it's a red herring -- there's no way
> to interpret this sentence that doesn't make it also an
> unfalsifiable criticism of any possible security model.


Yeah, I figured that out after I wrote this.

> The most common implementation of resource hiding is capabilities.

> > In this type of system a resource's reference acts as a ticket that
> > represents the right to use the resource.  Once code has a reference
> > it is considered to have full use of that resource it represents and
> > no further security checks are performed.
>
> Same thing.  What "further security checks" are we worried about?
> Woult it check to see whether we've authorized the interpreter to
> have access to the resource ... which we already know to be true?
>
> > To allow customizable restrictions one can pass references to wrappers
> of
> > resources.  This allows one to provide custom security to resources
> instead of
> > requiring an all-or-nothing approach.
>
> The ability to customize security restrictions is an important
> advantage of the resource hiding approach, since resource crippling
> requires that the architect of the security model anticipate every
> possible security restriction that future programmers might need.
>
> Using resource crippling is analogous to removing "def" from the
> language and requiring Python programmers to only use functions
> that are provided in the built-in modules instead of writing their
> own functions.
>
> > To use an analogy, imagine you are providing security for your home.
> > With capabilities, security came from not having any way to know
> > where your house is without being told where it was; a reference
> > to its location.  You might be able to ask a guard (e.g., Java's
> > ClassLoader) for a map, but if they refuse there is no way for you
> > to guess its location without being told.  But once you knew where
> > it was, you had complete use of the house.
>
> This analogy is only fair if you compare it to the same analogy for
> the resource crippling approach.  Resource crippling doesn't get you
> any finer-grained control either!  The comparison story is:
>
>     With resource crippling, security comes from having a guard
>     at the door to your house.  When a Python interpreter comes
>     up to the door, the guard checks to see if the interpreter
>     has permission to enter the house, and if it does, then it
>     gets complete use of the house.
>
> Why is the granularity of control described as the whole house
> in the resource-hiding story, but as each door in the house in
> the resource-crippling story?


Because, as you said above, if you want someone to have the resource (the
house, or in more concrete terms, a 'file') you just give it to them.  If
you cripple it, though, you might provide a 'file' object but restrict how
many bytes are written.

But I also realize that resource hiding handles this by providing a wrapper
that provides the protection.

> And that complete access is an issue with a capability system.
> > If someone played a little loose with a reference for a resource
> > then you run the risk of it getting out.
>
> Could you be more specific about what you mean by "it getting out"?


Out of a trusted interpreter and ending up in a sandboxed interpreter some
how.

If you mean getting from a trusted interpreter to an untrusted
> interpreter -- then how is a resource going to travel between
> interpreters?


Beats me, but I am always scared of Armin and Samuele.  =)

It seems that your criticisms are aimed at resource crippling being a "plug
holes as needed but if you foul up you are screwed" with resource hiding
being more "fix the fundamental issues and just don't present access to
resources you don't want to give access to (or wrap accordingly)".  And in
general I agree with this assessment.  But I also realize that Python was
not designed for security in mind and there seems to be new ways to get
access to 'file'.  If I felt confident that I could find and hide 'file' as
needed, I would go that route immediately.  But I don't think I can (and
Armin has said this as well).

If you think you can help figure out every place a reference to 'file' can
be found through the standard interpreter, then fine, let's go that way.  I
just don't have faith this can be done effectively.

-Brett


Or if not, then are you thinking of a situation in which one
> piece of code is trusted with the resource, but another piece of
> code is not, and both are running in the same interpreter?
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20060705/8fe64cfc/attachment.htm 

From brett at python.org  Thu Jul  6 00:12:33 2006
From: brett at python.org (Brett Cannon)
Date: Wed, 5 Jul 2006 15:12:33 -0700
Subject: [Python-Dev] branch for sandox work created: bcannon-sandboxing
Message-ID: <bbaeab100607051512m7ec7b26fld56d72541b5edd1e@mail.gmail.com>

I have created a branch in svn off of HEAD for holding the sandboxing work.
It's called bcannon-sandboxing and I have checked in the design doc in the
root as sandboxing_design_doc.txt .  You can keep an eye on the checkout
message for incremental changes, but I will email the list once I have gone
through at least one thorough revision.

-Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20060705/101ed16e/attachment.htm 

From mcherm at mcherm.com  Thu Jul  6 00:47:50 2006
From: mcherm at mcherm.com (Michael Chermside)
Date: Wed, 05 Jul 2006 15:47:50 -0700
Subject: [Python-Dev] doc for new restricted execution design for	Python
Message-ID: <20060705154750.xzn3ft0hq2gw0koc@login.werra.lunarpages.com>

Ka-Ping Yee writes:
> If you mean getting from a trusted interpreter to an untrusted
> interpreter -- then how is a resource going to travel between
> interpreters?

Brett Cannon responds:
> Beats me, but I am always scared of Armin and Samuele.  =)

Okay, those two scare me also, but I would still rather not
spread FUD. Your proposal contains lots of details about how to
address the danger that Python objects can cross from one
interpreter to another. Could we instead attack that straight-on
and try to find a convincing proof that objects cannot possibly
cross the interpreter barrier? If so, it would simplify a bit
of your proposal, and make me feel a little less worried.

-- Michael Chermside

From mcherm at mcherm.com  Thu Jul  6 00:51:49 2006
From: mcherm at mcherm.com (Michael Chermside)
Date: Wed, 05 Jul 2006 15:51:49 -0700
Subject: [Python-Dev] In defense of Capabilities [was: doc for new
	restricted execution	design for Python]
Message-ID: <20060705155149.eipxm6j6oycgkcs8@login.werra.lunarpages.com>

In response to Ka-Ping's comments on the subject of "Resource Hiding"
vs "Resource Crippling", Brett says:

> It seems that your criticisms are aimed at resource crippling
> being a "plug holes as needed but if you foul up you are screwed"
> with resource hiding being more "fix the fundamental issues and
> just don't present access to resources you don't want to give
> access to (or wrap accordingly)".  And in general I agree with
> this assessment.  But I also realize that Python was not designed
> for security in mind and there seems to be new ways to get access
> to 'file'.  If I felt confident that I could find and hide 'file'
> as needed, I would go that route immediately.  But I don't think I
> can (and Armin has said this as well).

I agree completely. "Resource Hiding" (specifically, Capabilities)
has the cleanest concept, yet there are valid reasons to worry
about implementing it in Python. However, I would like to point out
one other advantage that capabilities has over "Resource Crippling".

Resource Crippling implements the restrictions as changes in the
underlying C-level objects capable of performing dangerous operations.
That means that the restrictions are implemented by the interpreter.
The advantage is obvious: we can trust the interpreter. But the
disadvantages are (1) it's slow to fix (requires a bugfix release
followed by everyone in the world upgrading), and (2) it cannot be
extended by the user.

With resource crippling, you will need to decide just what kind of
restrictions the file type will implement. I believe you are
planning to restrict to a list of known filenames and known
directories for reading and for writing. (Actually, you said mode
string, but I presume that you won't maintain separate lists for
'r' and 'rb' modes.) Then there was discussion of whether the
directories ought to be recursive, whether the total number of
files opened ought to be restricted, whether the total size written
should be restricted, and even whether the size should be measured
in bytes or blocks. Such conversations could go on for a long time,
and in the end you must make some compromises.

If you were using capabilities, you would need to ensure that
restricted interpreters could only get the file object that they
were given. But then _all_ of these fancy versions of the
restrictions would be immediately supported: it would be up to the
users to create secure wrappers implementing the specific
restrictions desired.

I really like this feature of capabilities: that they can be
extended (well, restricted) by the user, not just by the language
implementer. That's a powerful feature, and I don't want to give
it up. But on the other hand, I don't quite see how to maintain
it -- here are my best ideas, perhaps they will help.

Python already has one essential ingredient for capabilities:
unforgable references. But it fails in two other ways: having
secure data, and having no auxiliary means of accessing
objects.

Python's powerful introspection and visible implementation
(eg: __dict__) make it impossible to encapsulate data in an
object in a way that prevents malicious users from accessing
it. But that is actually surprisingly easy to fix. Just create
a new type (maybe a new metaclass), implemented in C, which
contains private data and a means to control access to it. You
would provide a dict which would be stored privately without
access from Python, and then provide methods and attributes
along with a Python function for evaluating access to each. The
type would ensure that the access test was evaluated with
access to the private dict before any method or attribute was
accessed. Such a construct is simple enough that I believe we
could implement it and be reasonably confident that it was
reliably secure. (I have played with this in the past and been
reasonably pleased with the results.) Adding restrictions would
then incur some performance penalties, but that seems
unproblematic.

That leaves the other problem: auxiliary means of accessing
objects. There are things like gc.get_objects(). In the special
case of file, which is a type that's also dangerous, there are
tricks like "object().__class__.__subclasses__()". I would love
to believe that we could plug all of these holes, but experience
(rexec) proves otherwise. For something like sockets, I am
fairly sure that there's a clear bottleneck (the low-level
socket module), but still numerous existing libraries that use
this low-level module without being handed a capability.

So this is where my alternative plan starts to fall apart. Your
(Brett's) plan to use resource crippling for these kinds of
restrictions involves putting C code around all direct access
to files, sockets, or whatever resource is being accessed.
Perhaps instead of your C code doing the security checks
directly, it could make sure that the objects returned were
contained within the correct secure wrappers. That's OK so far
as it goes, but the C checks are per interpreter instance, so
how do we get them to apply the correct wrappers? Maybe the
interpreter maintains (in C) a stack of wrappers per-thread and
provides a means for stack frames (functions) to register
wrappers? We would wind up wrapping the wrappers, but that's
just the nature of capabilities. Perhaps it would look
something like this:

     def invoke_user_function():
         PyXXX.set_file_wrapper(ReadOnlyRestriction)
         PyXXX.set_socket_wrapper(
                 SingleDomainRestriction('example.com'))
         untrusted_object.do_stuff()

   ...

To sum up: I agree that you cannot rely on prevent all the
possible "python tricks", but I still think that capabilities
are a superior solution. I'd like to find a way to achieve
the user-customizability of capabilities without throwing
out the entire standard library -- maybe some hybrid of
"resource hiding" and "resource crippling". I can almost see
a way to achieve it, but there are still a few flaws.

-- Michael Chermside


From brett at python.org  Thu Jul  6 01:11:14 2006
From: brett at python.org (Brett Cannon)
Date: Wed, 5 Jul 2006 16:11:14 -0700
Subject: [Python-Dev] doc for new restricted execution design for Python
In-Reply-To: <20060705154750.xzn3ft0hq2gw0koc@login.werra.lunarpages.com>
References: <20060705154750.xzn3ft0hq2gw0koc@login.werra.lunarpages.com>
Message-ID: <bbaeab100607051611pf70f63drd1c4a52d30d8f544@mail.gmail.com>

On 7/5/06, Michael Chermside <mcherm at mcherm.com> wrote:
>
> Ka-Ping Yee writes:
> > If you mean getting from a trusted interpreter to an untrusted
> > interpreter -- then how is a resource going to travel between
> > interpreters?
>
> Brett Cannon responds:
> > Beats me, but I am always scared of Armin and Samuele.  =)
>
> Okay, those two scare me also, but I would still rather not
> spread FUD.


I don't consider it FUD.  Armin in an email said that he thought it was a
losing battle to try to hide 'file' from an interpreter.  That is what I am
worried about, period.  Everythign else can be protected through resource
hiding.

Your proposal contains lots of details about how to
> address the danger that Python objects can cross from one
> interpreter to another. Could we instead attack that straight-on
> and try to find a convincing proof that objects cannot possibly
> cross the interpreter barrier? If so, it would simplify a bit
> of your proposal, and make me feel a little less worried.


As I said to Ping, if people *really* think this is doable and are willing
to help out with this, then fine, I am willing to give this a shot.  But I
know I don't personally know enough about every random corner of the code
base like Armin and Samuele know in order to feel comfortable in claiming I
can pull this off by myself.

-Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20060705/db937623/attachment.htm 

From brett at python.org  Thu Jul  6 01:26:08 2006
From: brett at python.org (Brett Cannon)
Date: Wed, 5 Jul 2006 16:26:08 -0700
Subject: [Python-Dev] In defense of Capabilities [was: doc for new
	restricted execution design for Python]
In-Reply-To: <20060705155149.eipxm6j6oycgkcs8@login.werra.lunarpages.com>
References: <20060705155149.eipxm6j6oycgkcs8@login.werra.lunarpages.com>
Message-ID: <bbaeab100607051626g6aef5ebfle46e7fdd9b71d00f@mail.gmail.com>

On 7/5/06, Michael Chermside <mcherm at mcherm.com> wrote:
>
> In response to Ka-Ping's comments on the subject of "Resource Hiding"
> vs "Resource Crippling", Brett says:
>
> > It seems that your criticisms are aimed at resource crippling
> > being a "plug holes as needed but if you foul up you are screwed"
> > with resource hiding being more "fix the fundamental issues and
> > just don't present access to resources you don't want to give
> > access to (or wrap accordingly)".  And in general I agree with
> > this assessment.  But I also realize that Python was not designed
> > for security in mind and there seems to be new ways to get access
> > to 'file'.  If I felt confident that I could find and hide 'file'
> > as needed, I would go that route immediately.  But I don't think I
> > can (and Armin has said this as well).
>
> I agree completely. "Resource Hiding" (specifically, Capabilities)
> has the cleanest concept, yet there are valid reasons to worry
> about implementing it in Python. However, I would like to point out
> one other advantage that capabilities has over "Resource Crippling".
>
> Resource Crippling implements the restrictions as changes in the
> underlying C-level objects capable of performing dangerous operations.
> That means that the restrictions are implemented by the interpreter.
> The advantage is obvious: we can trust the interpreter. But the
> disadvantages are (1) it's slow to fix (requires a bugfix release
> followed by everyone in the world upgrading), and (2) it cannot be
> extended by the user.
>
> With resource crippling, you will need to decide just what kind of
> restrictions the file type will implement. I believe you are
> planning to restrict to a list of known filenames and known
> directories for reading and for writing. (Actually, you said mode
> string, but I presume that you won't maintain separate lists for
> 'r' and 'rb' modes.) Then there was discussion of whether the
> directories ought to be recursive, whether the total number of
> files opened ought to be restricted, whether the total size written
> should be restricted, and even whether the size should be measured
> in bytes or blocks. Such conversations could go on for a long time,
> and in the end you must make some compromises.
>
> If you were using capabilities, you would need to ensure that
> restricted interpreters could only get the file object that they
> were given. But then _all_ of these fancy versions of the
> restrictions would be immediately supported: it would be up to the
> users to create secure wrappers implementing the specific
> restrictions desired.


I agree.  I would prefer this way of doing it.  But as I have said, making
sure that 'file' does not get out into the wild is tough.

I really like this feature of capabilities: that they can be
> extended (well, restricted) by the user, not just by the language
> implementer. That's a powerful feature, and I don't want to give
> it up. But on the other hand, I don't quite see how to maintain
> it -- here are my best ideas, perhaps they will help.
>
> Python already has one essential ingredient for capabilities:
> unforgable references. But it fails in two other ways: having
> secure data, and having no auxiliary means of accessing
> objects.


Right.  Private attributes only exist at the C level.

Python's powerful introspection and visible implementation
> (eg: __dict__) make it impossible to encapsulate data in an
> object in a way that prevents malicious users from accessing
> it. But that is actually surprisingly easy to fix. Just create
> a new type (maybe a new metaclass), implemented in C, which
> contains private data and a means to control access to it. You
> would provide a dict which would be stored privately without
> access from Python, and then provide methods and attributes
> along with a Python function for evaluating access to each. The
> type would ensure that the access test was evaluated with
> access to the private dict before any method or attribute was
> accessed. Such a construct is simple enough that I believe we
> could implement it and be reasonably confident that it was
> reliably secure. (I have played with this in the past and been
> reasonably pleased with the results.) Adding restrictions would
> then incur some performance penalties, but that seems
> unproblematic.


Right, but as you mention below this still does not protect the C level
objects such as 'file'.  If you can prevent references to 'file' from
getting out, you can change open() to return wrapped instances of 'file'
with the desired security measures in place.

That leaves the other problem: auxiliary means of accessing
> objects. There are things like gc.get_objects(). In the special
> case of file, which is a type that's also dangerous, there are
> tricks like "object().__class__.__subclasses__()". I would love
> to believe that we could plug all of these holes, but experience
> (rexec) proves otherwise. For something like sockets, I am
> fairly sure that there's a clear bottleneck (the low-level
> socket module), but still numerous existing libraries that use
> this low-level module without being handed a capability.


Right, but for modules that cheat, you just don't add to the whiltelist.
This is why you should never blindly add modules to the whitelist of
extension modules that you can import.

So this is where my alternative plan starts to fall apart. Your
> (Brett's) plan to use resource crippling for these kinds of
> restrictions involves putting C code around all direct access
> to files, sockets, or whatever resource is being accessed.


Just 'file' and sockets, nothing else.  Everything else is protected by not
allowing importation of the module.

Perhaps instead of your C code doing the security checks
> directly, it could make sure that the objects returned were
> contained within the correct secure wrappers. That's OK so far
> as it goes, but the C checks are per interpreter instance, so
> how do we get them to apply the correct wrappers? Maybe the
> interpreter maintains (in C) a stack of wrappers per-thread and
> provides a means for stack frames (functions) to register
> wrappers? We would wind up wrapping the wrappers, but that's
> just the nature of capabilities. Perhaps it would look
> something like this:
>
>      def invoke_user_function():
>          PyXXX.set_file_wrapper(ReadOnlyRestriction)
>          PyXXX.set_socket_wrapper(
>                  SingleDomainRestriction('example.com'))
>          untrusted_object.do_stuff()
>
>    ...
>
> To sum up: I agree that you cannot rely on prevent all the
> possible "python tricks", but I still think that capabilities
> are a superior solution.


And I have never disagreed with this.  It is just a "practicality vs.
purity" thing.

I'd like to find a way to achieve
> the user-customizability of capabilities without throwing
> out the entire standard library -- maybe some hybrid of
> "resource hiding" and "resource crippling". I can almost see
> a way to achieve it, but there are still a few flaws.


It's already a hybrid solution with the import protections for capabilities
and the 'file' constructor lock-out for crippling.

-Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20060705/5fb3b1cb/attachment-0001.html 

From brett at python.org  Thu Jul  6 02:01:48 2006
From: brett at python.org (Brett Cannon)
Date: Wed, 5 Jul 2006 17:01:48 -0700
Subject: [Python-Dev] what can we do to hide the 'file' type?
Message-ID: <bbaeab100607051701h144c81cam82f5a9dce33fae59@mail.gmail.com>

To make sure I don't unfairly block out capabilities as a complete security
model instead of just crippling 'file's constructor (I do like capabilities
and think it is a good model, really!), let's discuss how one can get to the
'file' type without importing any extension modules (that can be protected
at the import level so I am ignoring the 'gc' module trick and such).

First, it's in __builtin__.  That reference can just be left out of the dict
at the PyInterpreterState stuct's dict for built-ins.  But we all know it
isn't that simple.

Second, there is __subclasses__().  That method could just not be allowed to
be in the 'type' class at the Python level (hiding it, crippling it,
whatever), but would that break much code?  I don't know, but I doubt it.

Third, for any wrappers returned by open(), it cannot be a subclass because
chaining __class__ attribute, mro() (or any of the other methods provided on
'object' or 'type'), or type() will get you to the original 'file' type.
The actual 'file' reference will need to be stored at the C struct level for
the wrapper and not accessed except by the wrapper directly which would be
implemented in C.

Can anyone think of any other way to gain access to 'file' without importing
a module?  At that point one would need to be *very* careful about what an
extension module exported to the world, but I can live with that (as that is
already part of the plan).

Please seriously try to think of ways to get to 'file' everybody.  If we
really cannot come up with anything beyond these three ways, then I am
totally willing to go with a much more complete capabilities system for
security in Python and really minimize any crippling.  I just need to be
convinced that we won't be plugging holes in how to hide 'file' rather than
plugging holes from crippling 'file' (which, at this point, I am not
convinced of).

And if Armin and/or Samuele sign off that what we find is most likely (with
"most likely" equalling 99% chance) all there is, then bonus points and I
will *really* be convinced.  =)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20060705/60204925/attachment.html 

From greg.ewing at canterbury.ac.nz  Thu Jul  6 02:07:37 2006
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Thu, 06 Jul 2006 12:07:37 +1200
Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
In-Reply-To: <5f3d2c310607050317t51d796d1g348fc2d036798128@mail.gmail.com>
References: <ca471dc20607042049r6a801e6dp67bcd9dd2781df07@mail.gmail.com>
	<r01050400-1039-300385330C0B11DB9236001124365170@10.0.0.24>
	<5f3d2c310607050317t51d796d1g348fc2d036798128@mail.gmail.com>
Message-ID: <44AC5449.5020104@canterbury.ac.nz>

Marek "Baczek" Baczy?ski wrote:

> I suggest <- as an assignment operator instead of := - it's used in
> OCaml and it looks *very* different, yet still makes sense.

But assigning to an outer scope isn't *very* different,
it's only slightly different.

--
"And now for something slightly different..."
Greg

From greg.ewing at canterbury.ac.nz  Thu Jul  6 02:30:54 2006
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Thu, 06 Jul 2006 12:30:54 +1200
Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
In-Reply-To: <D6FFF1A2-FEEB-4D06-862A-DC5FAA54DF5B@chello.se>
References: <ca471dc20607042049r6a801e6dp67bcd9dd2781df07@mail.gmail.com>
	<r01050400-1039-300385330C0B11DB9236001124365170@10.0.0.24>
	<ca471dc20607050302j9499d00h5510f5e96bf28eae@mail.gmail.com>
	<D6FFF1A2-FEEB-4D06-862A-DC5FAA54DF5B@chello.se>
Message-ID: <44AC59BE.2020503@canterbury.ac.nz>

Simon Percivall wrote:

> def foo():
>      def bar():
>          foo.x = 3

That already had a different meaning - it assigns
to an attribute of the function object created by
executing def foo().

--
Greg

From greg.ewing at canterbury.ac.nz  Thu Jul  6 02:33:39 2006
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Thu, 06 Jul 2006 12:33:39 +1200
Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
In-Reply-To: <17579.49401.682772.29091@montanaro.dyndns.org>
References: <ca471dc20607042049r6a801e6dp67bcd9dd2781df07@mail.gmail.com>
	<r01050400-1039-300385330C0B11DB9236001124365170@10.0.0.24>
	<5f3d2c310607050317t51d796d1g348fc2d036798128@mail.gmail.com>
	<2FD28BA2-32D9-49C4-8D86-92B0A53346A6@python.org>
	<17579.49401.682772.29091@montanaro.dyndns.org>
Message-ID: <44AC5A63.2020602@canterbury.ac.nz>

skip at pobox.com wrote:

> By naming the function in which the binding is to occur you avoid problems
> of someone coming along and adding or deleting functions between the
> assignment (in baz) and the target of the assignment (x in foo) but then
> forgetting to increment or decrement the counters that refer to a fixed
> number of levels above the current function.

But it doesn't do anything for the (I expect much more
common) case of factoring out something in a function body
and making it a nested function -- you'd still have to
change the form of all the references to the name in
that case.

Better not to have a scheme that uses counters or
scope names at all, I think.

--
Greg

From greg.ewing at canterbury.ac.nz  Thu Jul  6 02:59:21 2006
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Thu, 06 Jul 2006 12:59:21 +1200
Subject: [Python-Dev] In defense of Capabilities [was: doc for
	new	restricted execution	design for Python]
In-Reply-To: <20060705155149.eipxm6j6oycgkcs8@login.werra.lunarpages.com>
References: <20060705155149.eipxm6j6oycgkcs8@login.werra.lunarpages.com>
Message-ID: <44AC6069.3050905@canterbury.ac.nz>

Michael Chermside wrote:

> That leaves the other problem: auxiliary means of accessing
> objects. There are things like gc.get_objects(). In the special
> case of file, which is a type that's also dangerous, there are
> tricks like "object().__class__.__subclasses__()".

My approach to that would be to not provide access to
these kinds of things via attributes, but via builtin
functions. E.g there wouldn't be a __subclasses__
attribute, but a subclasses() function. Then that
capability can be denied by not providing that
function.

--
Greg

From greg.ewing at canterbury.ac.nz  Thu Jul  6 03:01:04 2006
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Thu, 06 Jul 2006 13:01:04 +1200
Subject: [Python-Dev] doc for new restricted execution design for Python
In-Reply-To: <bbaeab100607051611pf70f63drd1c4a52d30d8f544@mail.gmail.com>
References: <20060705154750.xzn3ft0hq2gw0koc@login.werra.lunarpages.com>
	<bbaeab100607051611pf70f63drd1c4a52d30d8f544@mail.gmail.com>
Message-ID: <44AC60D0.1040508@canterbury.ac.nz>

Brett Cannon wrote:

> Armin in an email said that he thought it was 
> a losing battle to try to hide 'file' from an interpreter.

And I would change file() so that it didn't open
files. Then it would be harmless for code to have
access to the file class.

--
Greg

From brett at python.org  Thu Jul  6 03:09:54 2006
From: brett at python.org (Brett Cannon)
Date: Wed, 5 Jul 2006 18:09:54 -0700
Subject: [Python-Dev] In defense of Capabilities [was: doc for new
	restricted execution design for Python]
In-Reply-To: <44AC6069.3050905@canterbury.ac.nz>
References: <20060705155149.eipxm6j6oycgkcs8@login.werra.lunarpages.com>
	<44AC6069.3050905@canterbury.ac.nz>
Message-ID: <bbaeab100607051809n120a7f30macbb084103b3139f@mail.gmail.com>

On 7/5/06, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
>
> Michael Chermside wrote:
>
> > That leaves the other problem: auxiliary means of accessing
> > objects. There are things like gc.get_objects(). In the special
> > case of file, which is a type that's also dangerous, there are
> > tricks like "object().__class__.__subclasses__()".
>
> My approach to that would be to not provide access to
> these kinds of things via attributes, but via builtin
> functions. E.g there wouldn't be a __subclasses__
> attribute, but a subclasses() function. Then that
> capability can be denied by not providing that
> function.



__subclasses__ is a function.  And yes, if we go this route, that is what
would happen most likely.  The trick is figuring out any and all ways one
can get to 'file' from a standard interpreter prompt.

-Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20060705/796de055/attachment.htm 

From anthony at interlink.com.au  Thu Jul  6 05:24:16 2006
From: anthony at interlink.com.au (Anthony Baxter)
Date: Thu, 6 Jul 2006 13:24:16 +1000
Subject: [Python-Dev] ImportWarning decision
Message-ID: <200607061324.19000.anthony@interlink.com.au>

So there's 3 choices here:

a) revert the importwarning entirely
b) make it suppressed by default
c) more complicated code in import.c to only emit the warning if the 
import fails.

After a bit of a chat with Neal, I think the best combination of 
prudence and functionality is (b). (a) also works for me. (c) is a 
bit too scary, post-beta. import.c is a bad place where programmers 
go to die - I'd rather not mess about with it at this stage of the 
release cycle.

Unless I hear screams in the near future, (b) is what's going to be 
done for beta2, and therefore 2.5 final. We can re-evaluate the 
situation for 2.6 - maybe a more complex solution like (c) can be 
done for that.

This means Google can just turn it on in sitecustomize.py and Guido 
can avoid the hordes of peasants with pitchforks and burning torches.

Anthony


-- 
Anthony Baxter     <anthony at interlink.com.au>
It's never too late to have a happy childhood.

From anthony at interlink.com.au  Thu Jul  6 05:35:19 2006
From: anthony at interlink.com.au (Anthony Baxter)
Date: Thu, 6 Jul 2006 13:35:19 +1000
Subject: [Python-Dev] Proposed beta 2 changes (Q for Anthony/Neal)
In-Reply-To: <44AA5FE6.2030801@iinet.net.au>
References: <44AA5FE6.2030801@iinet.net.au>
Message-ID: <200607061335.21179.anthony@interlink.com.au>

On Tuesday 04 July 2006 22:32, Nick Coghlan wrote:
> 1. Finishing the __module_name__ workaround to allow relative
> imports from the main module when using -m.
>
>    I'd really like to finish this, because having PEP 328 and 338
> not playing well together is a wart that's quite visible to end
> users. I'd rather not have people's first encounter with the
> features provided by either PEP involve discovering that they're
> "broken".
>
>    The patch to fix this also takes care of adding a couple of
> paragraphs to the tutorial about explicit relative imports (which
> aren't currently written up in the main documentation). The patch
> is attached directly to the beta 1 bug report about the problem
> [1].
>
>    (Guido gave a +1 to the concept, but explicitly deferred to
> Anthony and Neal as to whether or not the fix should go in for beta
> 2)

I have some nervousness about this. Are there cases in the stdlib 
where this is an issue, today? Are there any cases which can't be 
addressed by using absolute imports? For 2.5, wouldn't it be better 
to simply say "use absolute imports in this case"? 

My nervousness is that I don't want a late change to introduce a 
language misfeature that we'll all regret later.

> 2. Adding an 'ignore' filter for ImportWarning at the end of
> warnings.py

Already covered this one in another email... Yes, this seems the best 
approach for 2.5.

Anthony

-- 
Anthony Baxter     <anthony at interlink.com.au>
It's never too late to have a happy childhood.

From nnorwitz at gmail.com  Thu Jul  6 05:48:56 2006
From: nnorwitz at gmail.com (Neal Norwitz)
Date: Wed, 5 Jul 2006 20:48:56 -0700
Subject: [Python-Dev] 2.5 and beyond
In-Reply-To: <e8djpk$cr6$1@sea.gmane.org>
References: <ee2a432c0606300005g256b3391na6e684123d3e9e93@mail.gmail.com>
	<e8djpk$cr6$1@sea.gmane.org>
Message-ID: <ee2a432c0607052048m50346575kedfb471160fbd155@mail.gmail.com>

On 7/4/06, Thomas Heller <theller at python.net> wrote:
>
> I would like to ask about the possibility to add some improvements to ctypes
> in Python 2.5, although the feature freeze is now in effect.  Hopefully former
> third-party libraries can have the freeze relaxed somewhat;-).

Ok, former third-party libraries get a 1e-308 reprieve. :-)

I think the general answer is to plan some way that you could make an
external release to support the other communities like numpy.  I don't
know the details, but I believe Barry does this with email.  This
would allow Python to be more stable, but allow additional features
for the communities that are interested in installing an additional
ctypes release.

> I intend to do these changes, the first is a small and trivial one, but allows
> a lot of flexibility:
>
> - Remove the restriction that the argtypes attribute of foreign functions must
>   be ctypes types.  Instead they are only required to implement a .from_param
>   class method.

This patch is ok.  I will add comments to the patch on SF.

> The second one is more involved, and not yet complete.  I can post the patch
> or a link to it for review when it is implemented completely:
>
> - Implement the __array_struct__ attribute as describes by the numpy pep at
>   http://numpy.scipy.org/array_interface.html.

This is too much to add to a beta.  Perhaps you could work on a branch
to add these features and integrate the branch back in after 2.5 has
been released.  You could make an external release from the branch.

n

From pje at telecommunity.com  Thu Jul  6 06:32:11 2006
From: pje at telecommunity.com (Phillip J. Eby)
Date: Thu, 06 Jul 2006 00:32:11 -0400
Subject: [Python-Dev] ImportWarning decision
In-Reply-To: <200607061324.19000.anthony@interlink.com.au>
Message-ID: <5.1.1.6.0.20060706002836.01f1a4b0@sparrow.telecommunity.com>

At 01:24 PM 7/6/2006 +1000, Anthony Baxter wrote:
>This means Google can just turn it on in sitecustomize.py and Guido
>can avoid the hordes of peasants with pitchforks and burning torches.

Is that really true?  It seems to me that Guido indicated a 
sitecustomize-solution wasn't possible, in which case suppression is no 
better than elimination.

By the way, while I haven't formally reviewed any of the patches to delay 
the warning, the one I saw that went by on Python-Dev looked quite 
straightforward and minimal.  Until I saw it, I also believed that trying 
to implement something like that would be difficult and delicate.  But the 
patch I saw looked very simple and direct.


From nnorwitz at gmail.com  Thu Jul  6 06:39:08 2006
From: nnorwitz at gmail.com (Neal Norwitz)
Date: Wed, 5 Jul 2006 21:39:08 -0700
Subject: [Python-Dev] import screwiness
Message-ID: <ee2a432c0607052139hb7ec02ej7e358a58e5f6e85c@mail.gmail.com>

In import.c starting around line 1210 (I removed a bunch of code that
doesn't matter for the problem):

                if (PyUnicode_Check(v)) {
                        copy = PyUnicode_Encode(PyUnicode_AS_UNICODE(v),
                                PyUnicode_GET_SIZE(v),
Py_FileSystemDefaultEncoding, NULL);
                        v = copy;
                }
                len = PyString_GET_SIZE(v);
                if (len + 2 + namelen + MAXSUFFIXSIZE >= buflen) {
                        Py_XDECREF(copy);
                        continue; /* Too long */
                }
                strcpy(buf, PyString_AS_STRING(v));

***
So if v is originally unicode, then copy is unicode from the second
line, right?  Then we assign v to copy, so v is still unicode.  Then
later on we do PyString_GET_SIZE and PyString_AS_STRING.  That doesn't
work, does it?  What am I missing?

n

From tim.peters at gmail.com  Thu Jul  6 06:53:54 2006
From: tim.peters at gmail.com (Tim Peters)
Date: Thu, 6 Jul 2006 00:53:54 -0400
Subject: [Python-Dev] import screwiness
In-Reply-To: <ee2a432c0607052139hb7ec02ej7e358a58e5f6e85c@mail.gmail.com>
References: <ee2a432c0607052139hb7ec02ej7e358a58e5f6e85c@mail.gmail.com>
Message-ID: <1f7befae0607052153r70563202lc3f6a464b4e91a44@mail.gmail.com>

[Neal Norwitz]
> In import.c starting around line 1210 (I removed a bunch of code that
> doesn't matter for the problem):
>
>                 if (PyUnicode_Check(v)) {
>                         copy = PyUnicode_Encode(PyUnicode_AS_UNICODE(v),
>                                 PyUnicode_GET_SIZE(v),
> Py_FileSystemDefaultEncoding, NULL);
>                         v = copy;
>                 }
>                 len = PyString_GET_SIZE(v);
>                 if (len + 2 + namelen + MAXSUFFIXSIZE >= buflen) {
>                         Py_XDECREF(copy);
>                         continue; /* Too long */
>                 }
>                 strcpy(buf, PyString_AS_STRING(v));
>
> ***
> So if v is originally unicode, then copy is unicode from the second
> line, right?

No.  An encoded unicode string is of type str, and PyUnicode_Encode()
returns an encoded string.  Like so:

>>> u"\u1122".encode('utf-8')
'\xe1\x84\xa2'
>>> type(_)
<type 'str'>

>  Then we assign v to copy, so v is still unicode.

Almost ;-)

> Then later on we do PyString_GET_SIZE and PyString_AS_STRING.  That doesn't
> work, does it?  What am I missing?

The conceptual type of the object returned by PyUnicode_Encode().

From nnorwitz at gmail.com  Thu Jul  6 06:59:49 2006
From: nnorwitz at gmail.com (Neal Norwitz)
Date: Wed, 5 Jul 2006 21:59:49 -0700
Subject: [Python-Dev] import screwiness
In-Reply-To: <1f7befae0607052153r70563202lc3f6a464b4e91a44@mail.gmail.com>
References: <ee2a432c0607052139hb7ec02ej7e358a58e5f6e85c@mail.gmail.com>
	<1f7befae0607052153r70563202lc3f6a464b4e91a44@mail.gmail.com>
Message-ID: <ee2a432c0607052159l22c023ecmb9f4e5615edc35ec@mail.gmail.com>

On 7/5/06, Tim Peters <tim.peters at gmail.com> wrote:
>
> > Then later on we do PyString_GET_SIZE and PyString_AS_STRING.  That doesn't
> > work, does it?  What am I missing?
>
> The conceptual type of the object returned by PyUnicode_Encode().

Phew, I sure am glad I was missing that.  :-)

I saw as the first line in PyUnicode_Encode that it was creating a
unicode string.  I then went to look at PyString_Encode and saw the
same.  There I realized that the string created initially wasn't the
string returned (it's the object from _AsEncodedString).  Which
naturally tells me what you just did.  Thanks.  Now can you fix
test_subprocess hanging? :-)  Please, I'll even buy you lunch and a
coke at the sprints at Google.  You are coming, right?

n

From python-dev at zesty.ca  Thu Jul  6 07:02:19 2006
From: python-dev at zesty.ca (Ka-Ping Yee)
Date: Thu, 6 Jul 2006 00:02:19 -0500 (CDT)
Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
In-Reply-To: <ca471dc20607050121o48dd51a7v5f12ee15b9f67ea3@mail.gmail.com>
References: <44AA1FF6.1050501@acm.org>
	<ca471dc20607040728y1035e27ej3d6d589eaed00080@mail.gmail.com>
	<44AAE020.5060204@acm.org>
	<5.1.1.6.0.20060704182302.0207e890@sparrow.telecommunity.com>
	<5.1.1.6.0.20060705020306.0207f890@sparrow.telecommunity.com>
	<ca471dc20607050121o48dd51a7v5f12ee15b9f67ea3@mail.gmail.com>
Message-ID: <Pine.LNX.4.58.0607052351050.22834@server1.LFW.org>

On Wed, 5 Jul 2006, Guido van Rossum wrote:
> On 7/5/06, Phillip J. Eby <pje at telecommunity.com> wrote:
> > Using the classic nonsense example:
> >
> >      def counter(num):
> >          def inc():
> >              .num += 1
> >              return .num
> >          return inc
> >
> Would this also use ..num to refer to num in an outer scope two
> levels removed?

I don't think there's any need for that.  I see '.num' as just another
way of saying "num, but don't make a new binding".

I agree with Guido that the best proposals so far are converging on
the idea that it's more Pythonic to say "don't make a new binding"
when a variable is used, than to declare "this is the scope for this
binding" ahead of time.

Of those there are two kinds:

    (a) State once (anywhere in a scope, but preferably at the
        beginning) that a variable is non-local.  This is like
        the "global" keyword works now, and this category includes:

          - Change the meaning of 'global'.
          - Add a new keyword 'outer' or 'nonlocal', etc.

    (b) Indicate, when mentioning a variable, that the variable
        is non-local.  This category includes:

          - Say 'global.x' or 'outer.x' instead of 'x'.
          - Say '.x' instead of 'x'.

My favourite so far is to use a new keyword -- i think changing the
meaning of 'global' would be misleading.  '.x' is probably my next
favourite, though i share Guido's concern about allowing both 'x'
and '.x' to refer to the same thing.

I see that 'outer' is used as an identifier in hmac.py in the
standard library and also as a variable in test_set*.py.  On the
other hand 'nonlocal' does not appear anywhere in the standard
library.  Thus, 'nonlocal' is the best option that i've seen so far;
it's less likely to break anything and it says exactly what it means.
I can't think of a more accurate keyword.


-- ?!ng

From talin at acm.org  Thu Jul  6 07:13:07 2006
From: talin at acm.org (Talin)
Date: Wed, 05 Jul 2006 22:13:07 -0700
Subject: [Python-Dev] In defense of Capabilities [was: doc for
 new	restricted execution design for Python]
In-Reply-To: <bbaeab100607051626g6aef5ebfle46e7fdd9b71d00f@mail.gmail.com>
References: <20060705155149.eipxm6j6oycgkcs8@login.werra.lunarpages.com>
	<bbaeab100607051626g6aef5ebfle46e7fdd9b71d00f@mail.gmail.com>
Message-ID: <44AC9BE3.4000301@acm.org>

Brett Cannon wrote:
> On 7/5/06, Michael Chermside <mcherm at mcherm.com> wrote:
>> If you were using capabilities, you would need to ensure that
>> restricted interpreters could only get the file object that they
>> were given. But then _all_ of these fancy versions of the
>> restrictions would be immediately supported: it would be up to the
>> users to create secure wrappers implementing the specific
>> restrictions desired.
> 
> I agree.  I would prefer this way of doing it.  But as I have said, making
> sure that 'file' does not get out into the wild is tough.

I seem to recall someone mentioned earlier in this discussion the notion 
of somehow throwing an exception when sandboxed code attempts to push a 
file reference onto the interpreter stack.

I'm not an expert in these matters, so perhaps what I am going to say 
will make no sense, but here goes:

What if there were two copies of the evaluator function. One copy would 
be a slightly slower 'checked' function, that would test all objects for 
a 'check' bit. Any attempt to evaluate a reference to an object with a 
check bit set would throw an exception.

The other eval function would be the 'unchecked' version that would run 
at full speed, just like it does today.

Transitioning from the checked to the unchecked state could only be done 
via C code. So the 'file' wrapper, for example, would switch over to the 
unchecked interpreter before calling the actual methods of 'file'. That 
C wrapper might also check the current permission state to see what 
operations were legal.

Note that whenever a C function sets the interpreter state to 
'unchecked', that fact is saved on the stack, so that when the function 
returns, the previous state is restored. The function for setting the 
interpreter state is something like PyCall_Unchecked( ... ), which 
restores the interpreter state back to where it was.

Transitioning from unchecked to checked is trickier. Specifically, you 
don't want to ever run sandboxed code in the unchecked state - this is a 
problem for generators, callbacks, and so on. I can think of two 
approaches to handling this:

First approach is to mark all sandboxed code with a bit indicating the 
code is untrusted. Any attempt to call or otherwise invoke a function 
that has this bit set would throw the interpreter into the 'checked' 
state. (Note that transitioning the other way is *not* automatic - i.e. 
calling trusted code does not automatically transition to unchecked state.)

The approach is good because it means that if you have intermediary code 
between the wrapper and the sandboxed code, the interpreter still does 
the right thing - it sets the interpreter into checked state.

One problem is how to restore the 'unchecked' state when a function call 
returns. Probably you would have to build this into the code that does 
the state transition.

If marking the sandboxed code isn't feasible, then you'd have to have 
the wrapper objects wrap all of the callbacks with code that goes to 
checked state before calling the callbacks. This means finding all the 
possible holes - however, I suspect that there are far fewer such holes 
than trying to hide all possible 'file' methods. However, one advantage 
of doing this is that restoring the 'unchecked' state after the call 
returns is fairly straightforward.

The advantage of the this whole approach is that once you set the 
'check' bit on 'file', it doesn't matter whether 'file' gets loose or 
not - you can't do anything with it without throwing an exception. 
Moreover, it also doesn't matter what code path you go through to access 
it. Only C code that flips the interpreter into unchecked state can call 
methods on 'file' without blowing up.

So essentially, what I propose is to define a simple security primitive 
- which essentially comes down to checking a single bit - and use that 
as a basis to create more complex and subtle security mechanisms.

-- Talin

From guido at python.org  Thu Jul  6 07:22:21 2006
From: guido at python.org (Guido van Rossum)
Date: Thu, 6 Jul 2006 07:22:21 +0200
Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
In-Reply-To: <Pine.LNX.4.58.0607052351050.22834@server1.LFW.org>
References: <44AA1FF6.1050501@acm.org>
	<ca471dc20607040728y1035e27ej3d6d589eaed00080@mail.gmail.com>
	<44AAE020.5060204@acm.org>
	<5.1.1.6.0.20060704182302.0207e890@sparrow.telecommunity.com>
	<5.1.1.6.0.20060705020306.0207f890@sparrow.telecommunity.com>
	<ca471dc20607050121o48dd51a7v5f12ee15b9f67ea3@mail.gmail.com>
	<Pine.LNX.4.58.0607052351050.22834@server1.LFW.org>
Message-ID: <ca471dc20607052222k15762cd4p90b896160b79ebd5@mail.gmail.com>

+1 on nonlocal.

I think that the := operator is also in case (b), but as I don't like
it I'm find with not mentioning it. :-)

Could someone write a PEP for this? Doesn't have to be very long but
I'd like it to summarize the main options proposed and discuss them,
like I did for the switch PEP. It's a p3yk PEP. (We really need to
move this to the py3k list...)

--Guido

On 7/6/06, Ka-Ping Yee <python-dev at zesty.ca> wrote:
> On Wed, 5 Jul 2006, Guido van Rossum wrote:
> > On 7/5/06, Phillip J. Eby <pje at telecommunity.com> wrote:
> > > Using the classic nonsense example:
> > >
> > >      def counter(num):
> > >          def inc():
> > >              .num += 1
> > >              return .num
> > >          return inc
> > >
> > Would this also use ..num to refer to num in an outer scope two
> > levels removed?
>
> I don't think there's any need for that.  I see '.num' as just another
> way of saying "num, but don't make a new binding".
>
> I agree with Guido that the best proposals so far are converging on
> the idea that it's more Pythonic to say "don't make a new binding"
> when a variable is used, than to declare "this is the scope for this
> binding" ahead of time.
>
> Of those there are two kinds:
>
>     (a) State once (anywhere in a scope, but preferably at the
>         beginning) that a variable is non-local.  This is like
>         the "global" keyword works now, and this category includes:
>
>           - Change the meaning of 'global'.
>           - Add a new keyword 'outer' or 'nonlocal', etc.
>
>     (b) Indicate, when mentioning a variable, that the variable
>         is non-local.  This category includes:
>
>           - Say 'global.x' or 'outer.x' instead of 'x'.
>           - Say '.x' instead of 'x'.
>
> My favourite so far is to use a new keyword -- i think changing the
> meaning of 'global' would be misleading.  '.x' is probably my next
> favourite, though i share Guido's concern about allowing both 'x'
> and '.x' to refer to the same thing.
>
> I see that 'outer' is used as an identifier in hmac.py in the
> standard library and also as a variable in test_set*.py.  On the
> other hand 'nonlocal' does not appear anywhere in the standard
> library.  Thus, 'nonlocal' is the best option that i've seen so far;
> it's less likely to break anything and it says exactly what it means.
> I can't think of a more accurate keyword.
>
>
> -- ?!ng
>


-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From martin at v.loewis.de  Thu Jul  6 07:24:03 2006
From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=)
Date: Thu, 06 Jul 2006 07:24:03 +0200
Subject: [Python-Dev] what can we do to hide the 'file' type?
In-Reply-To: <bbaeab100607051701h144c81cam82f5a9dce33fae59@mail.gmail.com>
References: <bbaeab100607051701h144c81cam82f5a9dce33fae59@mail.gmail.com>
Message-ID: <44AC9E73.7030304@v.loewis.de>

Brett Cannon wrote:
> Can anyone think of any other way to gain access to 'file' without
> importing a module?

In principle, it might be possible to find file in the func_defaults
or func_globals of some function, which might be defined as

orig_file = file
def file(...):
    ...

I couldn't find any such function in the standard library, though.

Regards,
Martin

From guido at python.org  Thu Jul  6 07:28:27 2006
From: guido at python.org (Guido van Rossum)
Date: Thu, 6 Jul 2006 07:28:27 +0200
Subject: [Python-Dev] ImportWarning decision
In-Reply-To: <5.1.1.6.0.20060706002836.01f1a4b0@sparrow.telecommunity.com>
References: <200607061324.19000.anthony@interlink.com.au>
	<5.1.1.6.0.20060706002836.01f1a4b0@sparrow.telecommunity.com>
Message-ID: <ca471dc20607052228k48fe1896j5df2a990631616d5@mail.gmail.com>

On 7/6/06, Phillip J. Eby <pje at telecommunity.com> wrote:
> At 01:24 PM 7/6/2006 +1000, Anthony Baxter wrote:
> >This means Google can just turn it on in sitecustomize.py and Guido
> >can avoid the hordes of peasants with pitchforks and burning torches.
>
> Is that really true?  It seems to me that Guido indicated a
> sitecustomize-solution wasn't possible, in which case suppression is no
> better than elimination.

Don't worry about it. It's irrelevant anyway since we're not upgrading
to 2.5 any time soon.

BTW The pitchforks pitch was widely misunderstood -- it was a
dramatization to emphasize what I still consider a general problem; if
I believed the problem was limited to Google I would have proposed an
internal solution.

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From python-dev at zesty.ca  Thu Jul  6 07:39:53 2006
From: python-dev at zesty.ca (Ka-Ping Yee)
Date: Thu, 6 Jul 2006 00:39:53 -0500 (CDT)
Subject: [Python-Dev] doc for new restricted execution design for Python
In-Reply-To: <bbaeab100607051156h4c341055w9377fb73a0f0c6fc@mail.gmail.com>
References: <bbaeab100606211733x3dd58226g15e2919e8e2cdd7e@mail.gmail.com>
	<Pine.LNX.4.58.0607040428210.17937@server1.LFW.org>
	<bbaeab100607051156h4c341055w9377fb73a0f0c6fc@mail.gmail.com>
Message-ID: <Pine.LNX.4.58.0607060022430.22834@server1.LFW.org>

On Wed, 5 Jul 2006, Brett Cannon wrote:
> On 7/4/06, Ka-Ping Yee <python-dev at zesty.ca> wrote:
> > In response to Guido's comment about confusing the words "trusted" and
> > "untrusted", how about "empowered" and "restricted"?
>
> Maybe.  I am really starting to lean towards trusted and sandboxed.

It can be risky to use words of the form '*trust*' when talking
about security because 'trust' can have different meanings in
different contexts.  (Examples: 'X is trusted' might mean 'X is
something i feel safe running without restrictions' or 'X *is*
in fact allowed to run without restrictions' or 'i need to run X
without restrictions in order to accomplish my task' or 'X is
something i am relying on for the security of my system'.)

The most common context for 'trusted' that i seem to come across
is in the phrase 'trusted computing base' (TCB), which refers to
'the thing that is supposed to be enforcing security restrictions'
as opposed to 'something i'm willing to let run unrestricted'.
So when i read 'trusted code' what comes to mind is the C implementation
of the Python interpreter, and it may be a good idea to reserve that
phrase for that purpose, if it's to be used at all.

In any case, let's try to nail down clear names for the different
pieces we're talking about here, and i gently advise avoiding
'*trust*' words or using them with very limited meaning.


-- ?!ng

From tim.peters at gmail.com  Thu Jul  6 07:55:01 2006
From: tim.peters at gmail.com (Tim Peters)
Date: Thu, 6 Jul 2006 01:55:01 -0400
Subject: [Python-Dev] import screwiness
In-Reply-To: <ee2a432c0607052159l22c023ecmb9f4e5615edc35ec@mail.gmail.com>
References: <ee2a432c0607052139hb7ec02ej7e358a58e5f6e85c@mail.gmail.com>
	<1f7befae0607052153r70563202lc3f6a464b4e91a44@mail.gmail.com>
	<ee2a432c0607052159l22c023ecmb9f4e5615edc35ec@mail.gmail.com>
Message-ID: <1f7befae0607052255p16254c7fr95c2de8e39a4d8f6@mail.gmail.com>

[Neal]
>>> Then later on we do PyString_GET_SIZE and PyString_AS_STRING.  That doesn't
>>> work, does it?  What am I missing?

[Tim]
>> The conceptual type of the object returned by PyUnicode_Encode().

[Neal]
> Phew, I sure am glad I was missing that.  :-)
>
> I saw as the first line in PyUnicode_Encode that it was creating a
> unicode string.  I then went to look at PyString_Encode and saw the
> same.  There I realized that the string created initially wasn't the
> string returned (it's the object from _AsEncodedString).  Which
> naturally tells me what you just did.  Thanks.

Good!  You're welcome.

>  Now can you fix test_subprocess hanging? :-)

Sure:  break it on Windows and I'll fix it tonight :-)

> Please, I'll even buy you lunch and a coke at the sprints at Google.  You
> are coming, right?

Sorry, doesn't look like it.  I'm living on investment income this
year, and the markets haven't been kind so far -- I don't want to go
more in the hole than is truly necessary.  I even switched from
Campbell's soup to the store generic brand last week ;-)

From python-dev at zesty.ca  Thu Jul  6 08:11:14 2006
From: python-dev at zesty.ca (Ka-Ping Yee)
Date: Thu, 6 Jul 2006 01:11:14 -0500 (CDT)
Subject: [Python-Dev] Restricted execution: what's the threat model?
In-Reply-To: <bbaeab100607051156h4c341055w9377fb73a0f0c6fc@mail.gmail.com>
References: <bbaeab100606211733x3dd58226g15e2919e8e2cdd7e@mail.gmail.com>
	<Pine.LNX.4.58.0607040428210.17937@server1.LFW.org>
	<bbaeab100607051156h4c341055w9377fb73a0f0c6fc@mail.gmail.com>
Message-ID: <Pine.LNX.4.58.0607060041190.22834@server1.LFW.org>

After reading the messages on this thread i'm starting to think
that it would be good to clarify what kinds of threats we are
trying to defend against, and specify what invariants we are
intending to preserve.

For example, here are a few things Brett mentioned:

> Right.  I am thinking more of an implementation screw up that somehow
> provides access to an object that has escalated rights.

> But you are correct, I am only concerned with preventing a crash of a
> sandboxed interperter.

[on what is meant by "it getting out"]
> Out of a trusted interpreter and ending up in a sandboxed interpreter
> some how.

So here are a couple of questions for clarification (some with my
guesses as to their answers):

1.  When we say "restricted/untrusted/<whatever> interpreter" we
    don't really mean that the *interpreter* is untrusted, right?
    We mean that the Python code that runs in that interpreter is
    untrusted (i.e. to be prevented from doing harm), right?

2.  I'm assuming that the implementation of the Python interpreter
    is always trusted.  As a starting point it seems to me we have
    to draw the line somewhere -- around at least the C code that
    implements the Python interpreter, and possibly more.  What do
    we take the Trusted Computing Base to include?  The Python VM
    implementation -- plus all the builtin objects and C modules?
    Plus the whole standard library?

    ("trusted" = "behaves safely because it's our job to write it
                  correctly, not due to something else imposing
                  restrictions upon it";
   "untrusted" = "we wish to be able to impose restrictions on it")

3.  Is it part of the plan that we want to protect Python code from
    other Python code?  For example, should a Python program/function
    X be able to say "i want to launch/call program/function Y with
    *these* parameters and have it run under *these* limitations?"
    This has a big impact on the model.

And here are some possible goals or invariants to consider.  It
will be helpful to decide on some of these so that, when someone
points to what they think is a flaw in the security implementation,
we can say "yes, that is our responsibility" or "no, it isn't".

We want to be able to guarantee that...

    A.  The interpreter will not crash no matter what Python code
        it is given to execute.

    B.  Python programs running in different interpreters embedded
        in the same process cannot communicate with each other.

    C.  Python programs running in different interpreters embedded
        in the same process cannot access each other's Python objects.

    D.  A given piece of Python code cannot access or communicate
        with certain Python objects in the same interpreter.

    E.  A given piece of Python code can access only a limited set
        of Python objects in the same interpreter.

I think in order to get truly useful restricted interpreters we
will end up wanting to make guarantees of all of these kinds.
There may be others i haven't thought of -- feel free to edit or
add others.


-- ?!ng

From pedronis at strakt.com  Thu Jul  6 09:24:45 2006
From: pedronis at strakt.com (Samuele Pedroni)
Date: Thu, 06 Jul 2006 09:24:45 +0200
Subject: [Python-Dev] Import semantics
In-Reply-To: <4dab5f760607050921h548cf2fdx9f125599ccbe8bfc@mail.gmail.com>
References: <cfb578b20606111531t6806d5c9kd35fd8ba29638174@mail.gmail.com>	<448D1F0D.7000405@strakt.com>	<ca471dc20606120948h119f1c1fw2725e7d434e287df@mail.gmail.com>	<4dab5f760606251334o718323adt409294e2952e954a@mail.gmail.com>	<ca471dc20607050112n1bae4fd6i941999617bba5a27@mail.gmail.com>
	<4dab5f760607050921h548cf2fdx9f125599ccbe8bfc@mail.gmail.com>
Message-ID: <44ACBABD.6020300@strakt.com>

Frank Wierzbicki wrote:
> On 7/5/06, Guido van Rossum <guido at python.org> wrote:
> 
>>Hi Frank,
>>
>>Have you and/or the Jython community made up your mind about this? The
>>thread seems to have disappeared after you posted (or perhaps it
>>continued only on jython-dev, which I don't read?).
> 
> The thread pretty much stopped there.  I think a modification of the
> import semantics won't really come up again until 2.3 is out, since
> releases and reviving Jython take priority over backwards incompatible
> features.  For my part I would like to think that a goal for Jython
> should be:
> 
> Pure Python code developed first on Jython should run without change on CPython.
> 
> Which would mean we will eventually need to change the import
> semantics for importing Python code (though I doubt we will change the
> semantics for importing Java packages any time soon).  Whether that
> can be done in 2.x, or if this change is so incompatible that we need
> to think about it in a "Jython 3000" way, I don't really know.

Changing the behavior of getattr for python modules and leave it as it 
is for java packages seems a reasonable compromise.


> 
> 
>>Also, I just realized that you're the new Jython maintainer. Is *that*
>>official?
> 
> It is official, at least in the unofficial way that Jython appears to
> work: Brian handed the baton to me after (I presume) Samuele Pedroni
> had handed the baton to Brian.
> 
> Brian's email is here:
> http://sourceforge.net/mailarchive/message.php?msg_id=13859029
> 
> That said, I still regard Samuele Pedroni as the ultimate authority on
> Jython and give him pretty much full veto power.  He fortunately
> continues to watch the checkins and prods me when I go in the wrong
> direction.
> 

as things stand this a reasonable way to go about things, I'm the only 
person with a bit historical prospective, a bit of time to help
(less than I would like still), knowing both python semantics well
and Java and knowing the Jython code base, also I designed how
the new-style classes implemantion work in Jython atm, so I should know
how things are supposed to hang together in that respect, also for 
things that involve the Java glueing aspects of Jython one needs
to grow some language design experience and sense, and I'm around since
enough time for Jython and on python-dev to have some of that :)

regards.


From theller at python.net  Thu Jul  6 11:15:24 2006
From: theller at python.net (Thomas Heller)
Date: Thu, 06 Jul 2006 11:15:24 +0200
Subject: [Python-Dev] 2.5 and beyond
In-Reply-To: <ee2a432c0607052048m50346575kedfb471160fbd155@mail.gmail.com>
References: <ee2a432c0606300005g256b3391na6e684123d3e9e93@mail.gmail.com>	<e8djpk$cr6$1@sea.gmane.org>
	<ee2a432c0607052048m50346575kedfb471160fbd155@mail.gmail.com>
Message-ID: <44ACD4AC.4070704@python.net>

Neal Norwitz schrieb:
> On 7/4/06, Thomas Heller <theller at python.net> wrote:
>>
>> I would like to ask about the possibility to add some improvements to ctypes
>> in Python 2.5, although the feature freeze is now in effect.  Hopefully former
>> third-party libraries can have the freeze relaxed somewhat;-).
> 
> Ok, former third-party libraries get a 1e-308 reprieve. :-)
> 
> I think the general answer is to plan some way that you could make an
> external release to support the other communities like numpy.  I don't
> know the details, but I believe Barry does this with email.  This
> would allow Python to be more stable, but allow additional features
> for the communities that are interested in installing an additional
> ctypes release.
> 
>> I intend to do these changes, the first is a small and trivial one, but allows
>> a lot of flexibility:
>>
>> - Remove the restriction that the argtypes attribute of foreign functions must
>>   be ctypes types.  Instead they are only required to implement a .from_param
>>   class method.
> 
> This patch is ok.  I will add comments to the patch on SF.

Patch 1517790 is now committed, thanks.

>> The second one is more involved, and not yet complete.  I can post the patch
>> or a link to it for review when it is implemented completely:
>>
>> - Implement the __array_struct__ attribute as describes by the numpy pep at
>>   http://numpy.scipy.org/array_interface.html.
> 
> This is too much to add to a beta.  Perhaps you could work on a branch
> to add these features and integrate the branch back in after 2.5 has
> been released.  You could make an external release from the branch.

We will see how this works out.  Not what I expected to hear, but I'm sure
it is a wise decision.

Thanks,
Thomas


From ronaldoussoren at mac.com  Thu Jul  6 12:14:34 2006
From: ronaldoussoren at mac.com (Ronald Oussoren)
Date: Thu, 6 Jul 2006 12:14:34 +0200
Subject: [Python-Dev] zlib module build failure on Mac OSX 10.4.7
In-Reply-To: <738A121B-3529-411D-89A7-EBD8C6B42A29@mac.com>
References: <17573.60593.759293.549583@montanaro.dyndns.org>
	<130298B1-7336-480A-A90F-97B68327BE4A@mac.com>
	<17574.43390.560231.425494@montanaro.dyndns.org>
	<315DECAC-D582-4692-A0CA-13D3C7BF992C@mac.com>
	<44A6C314.2000709@v.loewis.de>
	<0006F35E-FD5F-4BAD-A608-642039BBE7E8@mac.com>
	<ee2a432c0607041421y37438adboc658313e85114e52@mail.gmail.com>
	<738A121B-3529-411D-89A7-EBD8C6B42A29@mac.com>
Message-ID: <31F39BDC-A396-4EFD-B6C6-6E93458F3F7D@mac.com>


On Jul 5, 2006, at 7:35 AM, Ronald Oussoren wrote:

>
> On Jul 4, 2006, at 11:21 PM, Neal Norwitz wrote:
>
>> Ronald, Bob,
>>
>> I know Skip found and fixed his problem, however, is this problem
>> likely to affect other users?  Is there anything we can do to help
>> alleviate/diagnose this problem?
>
> I'll either enhance configure or roll back my change to setup.py.  
> I'd prefer to do the former, but if beta2 gets too close I'll just  
> change setup.py.

I have checked in a patch that fixes this issue (that is, configure  
will correctly set HAVE_ZLIB_COPY even if you have an old static  
library of zlib in /usr/local/lib) in revision 47267.

I don't really like that patch, it doesn't smell right. I wonder if  
we should just add '-Wl,-search_path_first' to the default LDFLAGS on  
OSX, that way the linker would behave more like the one on other unix- 
y platforms and special-casing for several libraries could be dropped.

To recap the problem I'm trying to solve: python uses several 3th- 
party libraries like libz and libbz2 to build extensions. Some of  
these libraries are shipped with OSX as shared libraries. I want to  
make it possible to link with local copies of those libraries, mostly  
because that makes it possible to run the resulting binary on older  
versions of the OS but also because some of the libraries on OSX are  
pretty old. I also want to link using static libraries instead of  
dynamic libraries. To do that you must use -Wl,-search_paths_first  
because otherwise the OSX linker will look for a dylib anywhere on  
the path before looking for a static library.

Ronald

>
> Ronald
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/ 
> ronaldoussoren%40mac.com

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 2157 bytes
Desc: not available
Url : http://mail.python.org/pipermail/python-dev/attachments/20060706/d2dfc651/attachment.bin 

From arigo at tunes.org  Thu Jul  6 13:03:12 2006
From: arigo at tunes.org (Armin Rigo)
Date: Thu, 6 Jul 2006 13:03:12 +0200
Subject: [Python-Dev] what can we do to hide the 'file' type?
In-Reply-To: <bbaeab100607051701h144c81cam82f5a9dce33fae59@mail.gmail.com>
References: <bbaeab100607051701h144c81cam82f5a9dce33fae59@mail.gmail.com>
Message-ID: <20060706110312.GB15690@code0.codespeak.net>

Hi Brett,

On Wed, Jul 05, 2006 at 05:01:48PM -0700, Brett Cannon wrote:
> And if Armin and/or Samuele sign off that what we find is most likely (with
> "most likely" equalling 99% chance) all there is, then bonus points and I
> will *really* be convinced.  =)

I don't think I can "sign off" that.  Really hiding Python objects is
quite hard IMHO.


Armin

From ncoghlan at gmail.com  Thu Jul  6 13:43:56 2006
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 06 Jul 2006 21:43:56 +1000
Subject: [Python-Dev] Proposed beta 2 changes (Q for Anthony/Neal)
In-Reply-To: <200607061335.21179.anthony@interlink.com.au>
References: <44AA5FE6.2030801@iinet.net.au>
	<200607061335.21179.anthony@interlink.com.au>
Message-ID: <44ACF77C.5030404@gmail.com>

Anthony Baxter wrote:
> On Tuesday 04 July 2006 22:32, Nick Coghlan wrote:
>> 1. Finishing the __module_name__ workaround to allow relative
>> imports from the main module when using -m.
> 
> I have some nervousness about this. Are there cases in the stdlib 
> where this is an issue, today? Are there any cases which can't be 
> addressed by using absolute imports? For 2.5, wouldn't it be better 
> to simply say "use absolute imports in this case"?

> My nervousness is that I don't want a late change to introduce a 
> language misfeature that we'll all regret later.

I actually come to agree with this POV - particularly after spending some time 
thinking about how I'd like to it to look in Py3k (with a simple boolean check 
replacing the __name__ == "__main__" idiom).

I'll revert the change that added __module_name__, too. I'd love to fix this 
properly for 2.5 but the timing of the discovery of the limitation is 
unfortunately lousy :(

I'll add something to PEP 338, and possibly elsewhere, noting that the -m 
switch adds the current directory to sys.path, so absolute imports will work 
fine so long as you invoke Python from the directory containing the relevant 
top level package directory, even if it isn't anywhere else on sys.path.

Some time between now and late 2007 I'll write something up about the 
interaction between __name__, sys.path[0] and implicit relative imports that 
actually makes implicit relative imports appear to work when a module inside a 
package is executed directly instead of via -m (probably when I get around to 
writing a Python 2.6 PEP on the topic).

Heh. I was going to be clever with the PEP number relating to this for 2.6, 
and I think I've discovered that trying to mix PEP 328 with 338 is going to be 
a doomed effort - add the PEP numbers together to find out why ;)

>> 2. Adding an 'ignore' filter for ImportWarning at the end of
>> warnings.py
> 
> Already covered this one in another email... Yes, this seems the best 
> approach for 2.5.

OK, I'll make the change to warnings.py, and update the warnings module 
documentation. AMK will still need to update What's New.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
             http://www.boredomandlaziness.org

From ncoghlan at gmail.com  Thu Jul  6 13:50:32 2006
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 06 Jul 2006 21:50:32 +1000
Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
In-Reply-To: <5.1.1.6.0.20060705133753.026a7698@sparrow.telecommunity.com>
References: <20060705101821.7sqnxo7lu5cgg4ks@login.werra.lunarpages.com>	<20060705101821.7sqnxo7lu5cgg4ks@login.werra.lunarpages.com>
	<5.1.1.6.0.20060705133753.026a7698@sparrow.telecommunity.com>
Message-ID: <44ACF908.1020403@gmail.com>

Phillip J. Eby wrote:
> At 07:27 PM 7/5/2006 +0200, Guido van Rossum wrote:
>> However I still don't believe "global" has the stretchiness in its
>> meaning that you claim it has. Have you ever heard a Python programmer
>> talking about closures use the word "global variable"?
>>
>> Are there any other native speakers who side with Michael?
> 
> I don't, and am -1 on using "global" to mean "outer".

Another -1 on stretching the meaning of global here. I quite like 'nonlocal', 
though.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
             http://www.boredomandlaziness.org

From tom at vector-seven.com  Thu Jul  6 10:51:32 2006
From: tom at vector-seven.com (Thomas Lee)
Date: Thu, 06 Jul 2006 18:51:32 +1000
Subject: [Python-Dev] zipfile.ZipFile('foo.zip',
	'a'): file not found -> create?
Message-ID: <44ACCF14.1020901@vector-seven.com>

Hi all,

In reference to:

http://sourceforge.net/tracker/index.php?func=detail&aid=1514451&group_id=5470&atid=105470


I wrote a patch for this "bug", but a valid point was raised by Ronald 
Oussorren: this borders on being more of a "feature" than a bug fix, 
although - IMHO - this fix improves consistency with the rest of the 
Python standard library.

Can I get some opinions on this? My patch for this issue currently lives 
here:

http://sourceforge.net/tracker/index.php?func=detail&aid=1517891&group_id=5470&atid=305470


Cheers,
Tom

From mcherm at mcherm.com  Thu Jul  6 14:15:43 2006
From: mcherm at mcherm.com (Michael Chermside)
Date: Thu, 06 Jul 2006 05:15:43 -0700
Subject: [Python-Dev] doc for new restricted execution design for	Python
Message-ID: <20060706051543.wxi1nukehvfkk4c4@login.werra.lunarpages.com>

I wrote:
> I would still rather not spread FUD.

Brett Cannon responds:
> I don't consider it FUD.  Armin in an email said that he thought it
> was a losing battle to try to hide 'file' from an interpreter.  That
> is what I am worried about, period.  Everythign else can be
> protected through resource hiding.

I never intended to say that "we can't hide 'file' from the user"
was FUD. Only objects crossing between interpreters. And it *might*
not be FUD, but so far I haven't heard anyone say that they thought
objects *could* cross between interpreters. I think your proposal
would read better with a categorical statement that objects cannot
cross between interpreters (right along with your categorical
statement that Python code cannot directly access dangerous resources
without help from the C level), so I present the following challenge:
can anyone think of a way that an object could "cross" interpreters?
For instance, are certain basic objects shared (str and int for
instance) and objects could be passed between interpreters by
attaching them as attributes of these core objects? I just don't
know enough about multiple interpreters to be sure -- but somehow I
thought they had separate object pools.

-- Michael Chermside


From mcherm at mcherm.com  Thu Jul  6 14:37:56 2006
From: mcherm at mcherm.com (Michael Chermside)
Date: Thu, 06 Jul 2006 05:37:56 -0700
Subject: [Python-Dev] Restricted execution: what's the threat model?
Message-ID: <20060706053756.ovcm9a05kko0o008@login.werra.lunarpages.com>

Ka-Ping Yee writes:
> i'm starting to think
> that it would be good to clarify what kinds of threats we are
> trying to defend against, and specify what invariants we are
> intending to preserve.

Yes!

> So here are a couple of questions for clarification (some with my
> guesses as to their answers):

Okay, I'll throw in my thoughts also.

> 1.  When we say "restricted/untrusted/<whatever> interpreter" we
>     don't really mean that the *interpreter* is untrusted, right?
>     We mean that the Python code that runs in that interpreter is
>     untrusted (i.e. to be prevented from doing harm), right?

Agreed. My interpretation of the proposal was that interpreters
were either "sandboxed" or "trusted". "Sandboxed" means that there
are security restrictions imposed at some level (perhaps even NO
restrictions). "Trusted" means that the interpreter implements
no security restrictions (beyond what CPython already implements,
which isn't much) and thus runs faster.

> 2.  I'm assuming that the implementation of the Python interpreter
>     is always trusted

Sure... it's got to be.

> What do
> we take the Trusted Computing Base to include?  The Python VM
> implementation -- plus all the builtin objects and C modules?
> Plus the whole standard library?

My interpretation of Brett's proposal is that the CPython developers
would try to ensure that Python VM had no "security holes" when
running in sandboxed mode. Of course, we also "try" to ensure no
crashes are possible also, and while we're quite good, we're not
perfect.

Beyond that, all pure-python modules with source available (whether
in the stdlib or not) can be "trusted" because they run in a
sandboxed VM. All C modules are *up to the user*. Brett proposes
to provide a default list of useful-but-believed-to-be-safe modules
in the stdlib, but the user can configure the C-module whitelist
to whatever she desires.

> 3.  Is it part of the plan that we want to protect Python code from
>     other Python code?  For example, should a Python program/function
>     X be able to say "i want to launch/call program/function Y with
>     *these* parameters and have it run under *these* limitations?"
>     This has a big impact on the model.

Now *that* is a good question. I would say the answer is a partial
"no", because there are pieces of Brett's security model that are
tied to the interpreter instance. Python code cannot launch another
interpreter (but perhaps it *should* be able to?), so it cannot
modify those restrictions for new Python code it launches.

However, I would rather like to allow Python code to execute other
code with greater restrictions, although I would accept all kinds
of limitations and performance penalties to do so. I would be
satisfied if the caller could restrict certain things (like web
and file access) but not others (like memory limits or use of
stdout). I would satisfied if the caller paid huge overhead costs
of launching a separate interpreter -- heck, even a separate
process. And if it is willing to launch a separate process, then
Brett's model works just fine: allow the calling code to start
a new (restricted) Python VM.

> We want to be able to guarantee that...
>
>   A.  The interpreter will not crash no matter what Python code
>       it is given to execute.

Agreed. We already want to guarantee that, with the caveat that the
guarantee doesn't apply to a few special modules (like ctypes).

>  B.  Python programs running in different interpreters embedded
>       in the same process cannot communicate with each other.

I don't want to guarantee this, does someone else? It's
astonishingly hard... there are all kinds of clever "knock on the
walls" tricks. For instance, communicate by varying your CPU
utilization up and down in regular patterns.

I'd be satisfied if they could pass information (perhaps even
someday provide a library making it *easy* to do so), but could
not pass unforgable items like Python object references, open file
descriptors, and so forth.

>   C.  Python programs running in different interpreters embedded
>       in the same process cannot access each other's Python objects.

I strengthen that slightly to all "unforgable" items, not just
object references.

>   D.  A given piece of Python code cannot access or communicate
>       with certain Python objects in the same interpreter.
>
>   E.  A given piece of Python code can access only a limited set
>       of Python objects in the same interpreter.

Hmmm. I'm not sure.


-- Michael Chermside

From mwh at python.net  Thu Jul  6 14:39:32 2006
From: mwh at python.net (Michael Hudson)
Date: Thu, 06 Jul 2006 13:39:32 +0100
Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
In-Reply-To: <20060705103123.fvv54v7e3s0ko8wo@login.werra.lunarpages.com>
	(Michael Chermside's message of "Wed, 05 Jul 2006 10:31:23 -0700")
References: <20060705103123.fvv54v7e3s0ko8wo@login.werra.lunarpages.com>
Message-ID: <2m64ia6h6j.fsf@starship.python.net>

Michael Chermside <mcherm at mcherm.com> writes:

> Phillip Eby writes:
>> I don't see a problem with requiring '.x' to be used for both  
>> reading and writing of outer-scope names; it just shouldn't be  
>> required for an outer-scope name that you don't rebind in the  
>> current scope.
>>
>>      def counter(num):
>>          def inc():
>>              .num += 1
>>              return .num
>>          return inc
>
> I am reminded of Tim Peter's declaration in response to a similar
> proposal some time ago:
>
>     Syntax should not look like grit on my monitor.
>
> (Sorry, no reference... but I swear it's word-for-word accurate because
> the quote burned itself into my memory.)

I think it was Anthony:

http://mail.python.org/pipermail/python-dev/2005-July/054581.html

Cheers,
mwh

-- 
  SCSI is not magic. There are fundamental technical reasons why it
  is necessary to sacrifice a young goat to your SCSI chain now and
  then.                                                  -- John Woods

From tim.hochberg at ieee.org  Thu Jul  6 15:38:54 2006
From: tim.hochberg at ieee.org (Tim Hochberg)
Date: Thu, 06 Jul 2006 06:38:54 -0700
Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
In-Reply-To: <Pine.LNX.4.58.0607052351050.22834@server1.LFW.org>
References: <44AA1FF6.1050501@acm.org>	<ca471dc20607040728y1035e27ej3d6d589eaed00080@mail.gmail.com>	<44AAE020.5060204@acm.org>	<5.1.1.6.0.20060704182302.0207e890@sparrow.telecommunity.com>	<5.1.1.6.0.20060705020306.0207f890@sparrow.telecommunity.com>	<ca471dc20607050121o48dd51a7v5f12ee15b9f67ea3@mail.gmail.com>
	<Pine.LNX.4.58.0607052351050.22834@server1.LFW.org>
Message-ID: <e8j3pp$tp3$1@sea.gmane.org>

Ka-Ping Yee wrote:
> On Wed, 5 Jul 2006, Guido van Rossum wrote:
>> On 7/5/06, Phillip J. Eby <pje at telecommunity.com> wrote:
>>> Using the classic nonsense example:
>>>
>>>      def counter(num):
>>>          def inc():
>>>              .num += 1
>>>              return .num
>>>          return inc
>>>
>> Would this also use ..num to refer to num in an outer scope two
>> levels removed?
> 
> I don't think there's any need for that.  I see '.num' as just another
> way of saying "num, but don't make a new binding".
> 
> I agree with Guido that the best proposals so far are converging on
> the idea that it's more Pythonic to say "don't make a new binding"
> when a variable is used, than to declare "this is the scope for this
> binding" ahead of time.
> 
> Of those there are two kinds:
> 
>     (a) State once (anywhere in a scope, but preferably at the
>         beginning) that a variable is non-local.  This is like
>         the "global" keyword works now, and this category includes:
> 
>           - Change the meaning of 'global'.
>           - Add a new keyword 'outer' or 'nonlocal', etc.
> 
>     (b) Indicate, when mentioning a variable, that the variable
>         is non-local.  This category includes:
> 
>           - Say 'global.x' or 'outer.x' instead of 'x'.
>           - Say '.x' instead of 'x'.
> 
> My favourite so far is to use a new keyword -- i think changing the
> meaning of 'global' would be misleading.  '.x' is probably my next
> favourite, though i share Guido's concern about allowing both 'x'
> and '.x' to refer to the same thing.
> 
> I see that 'outer' is used as an identifier in hmac.py in the
> standard library and also as a variable in test_set*.py.  On the
> other hand 'nonlocal' does not appear anywhere in the standard
> library.  Thus, 'nonlocal' is the best option that i've seen so far;
> it's less likely to break anything and it says exactly what it means.
> I can't think of a more accurate keyword.

'extant' (= already existing) is probably pretty rare in existing code, 
and has pretty close to exactly the correct meaning, but may be too obscure.

-tim


> 
> 
> -- ?!ng
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/python-python-dev%40m.gmane.org
> 


From jan-python at maka.demon.nl  Thu Jul  6 16:55:39 2006
From: jan-python at maka.demon.nl (jan-python at maka.demon.nl)
Date: Thu, 06 Jul 2006 16:55:39 +0200
Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
In-Reply-To: <mailman.29417.1152163446.27774.python-dev@python.org>
References: <mailman.29417.1152163446.27774.python-dev@python.org>
Message-ID: <20060706165539.sl1zeof09w0sks0o@webmail.nl.demon.net>

So.. are we only thinking about implementing this outer scope 
assignment because
there's lots of talk about it on the list, or are there actually use 
cases that
would become clearer if assigning to an outer scope variable was allowed? I
tend to think that almost _any_ piece of code that could be written 
using outer
scope assignment would become less clear by doing so as opposed to some other
way.

So, I'm not keen on the whole idea.
If we must, however, I think a declaration like 'nonlocal' would be 
best (well,
least-bad, that is).


From skip at pobox.com  Thu Jul  6 17:05:42 2006
From: skip at pobox.com (skip at pobox.com)
Date: Thu, 6 Jul 2006 10:05:42 -0500
Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
In-Reply-To: <20060706165539.sl1zeof09w0sks0o@webmail.nl.demon.net>
References: <mailman.29417.1152163446.27774.python-dev@python.org>
	<20060706165539.sl1zeof09w0sks0o@webmail.nl.demon.net>
Message-ID: <17581.9926.915438.200819@montanaro.dyndns.org>


    jan-python> So.. are we only thinking about implementing this outer
    jan-python> scope assignment because there's lots of talk about it on
    jan-python> the list, ...

:-)

    jan-python> ... or are there actually use cases that would become
    jan-python> clearer if assigning to an outer scope variable was allowed?

I think full lexical scoping will only be of use to people who use nested
scopes heavily.  The more typical user will be happy to just refer to values
in outser scopes without modifying them and rely on classes to save changed
state across calls.  I think it's almost a YAGNI, but I'm sure others will
disagree.

Skip

Repeat after me: Python is not Lisp...

From mcherm at mcherm.com  Thu Jul  6 17:31:27 2006
From: mcherm at mcherm.com (Michael Chermside)
Date: Thu, 06 Jul 2006 08:31:27 -0700
Subject: [Python-Dev] what can we do to hide the 'file' type?
Message-ID: <20060706083127.csxctx0sahr4kwk8@login.werra.lunarpages.com>

Armin Rigo writes:
> I don't think I can "sign off" [on hiding the file type].  Really hiding
> Python objects is quite hard IMHO.

I agree. But we don't have to give up yet. How about instead of hiding
file, we cripple it. Completely. Modify the file type so that when
executing on a sandboxed interpreter, all of the dangerous methods
and attributes of file throw exceptions.

Then we create a separate thing (in C) called a "SecureFileWrapper".
It has methods that are passed a reference to a file object and
can invoke the methods without error. We provide a means for obtaining
a SecureFileWrapper bound to a given file (perhaps open()).

Essentially, we give up on hiding file, which is a frequently-used
type, and very hard to hide, and instead we rely on our ability to
write a reliably secure "SecureFileWrapper" class (in C).

-- Michael Chermside


From ark at acm.org  Thu Jul  6 17:40:23 2006
From: ark at acm.org (Andrew Koenig)
Date: Thu, 6 Jul 2006 11:40:23 -0400
Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
In-Reply-To: <ca471dc20607051027y677c7127wfe098401c366dc74@mail.gmail.com>
Message-ID: <001301c6a112$81daa0f0$6402a8c0@arkdesktop>

> However I still don't believe "global" has the stretchiness in its
> meaning that you claim it has. Have you ever heard a Python programmer
> talking about closures use the word "global variable"?

I guess the term I've heard most often is "free variable," but I wouldn't be
surprised if I saw the term "global" used to describe a free variable.

However, I should point out that Dijkstra used "global" in a similar way in
his 1971 book "A Discipline of Programming."  The program examples in that
book are in a language he devised for the purpose; one of the language's
features is that every variable used in every block must be explicitly
declared, even if it is taken from a surrounding block.  If I remember
correctly, the terms he used for variables taken from a surrounding block
and variables defined and used in the same block were "global" and
"private," respectively.




From almann.goo at gmail.com  Thu Jul  6 17:49:16 2006
From: almann.goo at gmail.com (Almann T. Goo)
Date: Thu, 6 Jul 2006 11:49:16 -0400
Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
In-Reply-To: <ca471dc20607052222k15762cd4p90b896160b79ebd5@mail.gmail.com>
References: <44AA1FF6.1050501@acm.org>
	<ca471dc20607040728y1035e27ej3d6d589eaed00080@mail.gmail.com>
	<44AAE020.5060204@acm.org>
	<5.1.1.6.0.20060704182302.0207e890@sparrow.telecommunity.com>
	<5.1.1.6.0.20060705020306.0207f890@sparrow.telecommunity.com>
	<ca471dc20607050121o48dd51a7v5f12ee15b9f67ea3@mail.gmail.com>
	<Pine.LNX.4.58.0607052351050.22834@server1.LFW.org>
	<ca471dc20607052222k15762cd4p90b896160b79ebd5@mail.gmail.com>
Message-ID: <7e9b97090607060849j42d62d8aqd6e6a15475df590c@mail.gmail.com>

On 7/6/06, Guido van Rossum <guido at python.org> wrote:
>
> +1 on nonlocal.
>
> I think that the := operator is also in case (b), but as I don't like
> it I'm find with not mentioning it. :-)
>
> Could someone write a PEP for this? Doesn't have to be very long but
> I'd like it to summarize the main options proposed and discuss them,
> like I did for the switch PEP. It's a p3yk PEP. (We really need to
> move this to the py3k list...)
>

Drat, too bad this wasn't back in February when I was all for writing the
PEP--sadly, I don't have time to do this, maybe later if no one steps up to
the plate...

For reference, here is a link to the other, rather large, thread on this
back then:
http://thread.gmane.org/gmane.comp.python.devel/76532/focus=76532

My option one, is essentially "nonlocal", though I spelled it as "use".
Really, I am personally agreeable to almost any spelling of such a keyword.
I am +1 on "nonlocal", though I know the biggest dissent against it is
adding another "global"-like keyword that is not "pythonic".

Also in regard to the "prefix-dot" notation (i.e. ".x"), I am -1 for the
reason that it introduces a subtle alternate way of spelling local
variables.  The rules for the usage would have to be strict enough to
prevent subtle code obscurity (like what is the semantics for using ".x" and
"x" in a scope--are both spellings allowed, and if so, what is the meaning
if assignment is involved or is not involved).  Of course, if the semantics
were well defined such that it would be difficult for users to trap
themselves with different spellings of local variables, then I would be +0
for it.

Best Regards,
Almann

-- 
Almann T. Goo
almann.goo at gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20060706/db27bf8e/attachment.htm 

From pje at telecommunity.com  Thu Jul  6 18:28:12 2006
From: pje at telecommunity.com (Phillip J. Eby)
Date: Thu, 06 Jul 2006 12:28:12 -0400
Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
In-Reply-To: <17581.9926.915438.200819@montanaro.dyndns.org>
References: <20060706165539.sl1zeof09w0sks0o@webmail.nl.demon.net>
	<mailman.29417.1152163446.27774.python-dev@python.org>
	<20060706165539.sl1zeof09w0sks0o@webmail.nl.demon.net>
Message-ID: <5.1.1.6.0.20060706120457.01f1bab8@sparrow.telecommunity.com>

At 10:05 AM 7/6/2006 -0500, skip at pobox.com wrote:

>     jan-python> So.. are we only thinking about implementing this outer
>     jan-python> scope assignment because there's lots of talk about it on
>     jan-python> the list, ...
>
>:-)
>
>     jan-python> ... or are there actually use cases that would become
>     jan-python> clearer if assigning to an outer scope variable was allowed?
>
>I think full lexical scoping will only be of use to people who use nested
>scopes heavily.  The more typical user will be happy to just refer to values
>in outser scopes without modifying them and rely on classes to save changed
>state across calls.  I think it's almost a YAGNI, but I'm sure others will
>disagree.

Here's the reason I think this keeps coming up, and why Guido's "just use a 
class" argument doesn't really address the actual problem that's taking place.

When you are writing some function, and you find yourself using a nested 
function because it's the closest match for something you're doing, there 
occasionally comes a point at which you realize that, gosh, you need to 
mutate something in an outer scope.  At that point, your choices are to 
either kludge it with a mutable, or to reorganize the whole thing.  Both of 
these *feel* like warts because they're making you do less-than-optimal 
things.  Your mental "flow" is disrupted because the language is forcing 
you to work around it, rather than having a way of working with it.

This is a user experience issue, not a technical one.  The fact that you 
can say, "you should've done it differently in the first place" doesn't do 
anything for your flow.  In theory, you could design cars without any 
brakes, because people could just coast to a stop if they planned well 
enough in advance.  ;-)  In practice, you need the brakes because people 
often don't discover their need to stop until much later in the process.

This is a flow issue that's specific to *incremental* development.  What 
happens is that first you refactor the code in a function to include nested 
functions.  The variable references don't change, you're just indenting 
some code.  *Then*, it later comes up that you need to rebind a variable, 
and now you have to globally change the variable to make it work as a 
mutable instead, or else you have to refactor all the variables to be 
'self.' references and put a class in somewhere.

This destroys the flow of incrementally developing whatever it is you're 
developing, and makes you stop to do excise work.  That's why being able to 
rebind a variable without redefining it is important.

In short: in *theory*, a rebinding operator or "nonlocal" declaration is 
unnecessary.  In *practice*, having one seems quite useful every time you 
wander down the path that leads to having to rewrite your code just because 
the language won't let you do that one tiny thing -- or so it feels like to 
the person who's experiencing it.


From brett at python.org  Thu Jul  6 18:59:37 2006
From: brett at python.org (Brett Cannon)
Date: Thu, 6 Jul 2006 09:59:37 -0700
Subject: [Python-Dev] In defense of Capabilities [was: doc for new
	restricted execution design for Python]
In-Reply-To: <44AC9BE3.4000301@acm.org>
References: <20060705155149.eipxm6j6oycgkcs8@login.werra.lunarpages.com>
	<bbaeab100607051626g6aef5ebfle46e7fdd9b71d00f@mail.gmail.com>
	<44AC9BE3.4000301@acm.org>
Message-ID: <bbaeab100607060959w6f018b3dwe893d7c44f1db2f5@mail.gmail.com>

On 7/5/06, Talin <talin at acm.org> wrote:
>
> Brett Cannon wrote:
> > On 7/5/06, Michael Chermside <mcherm at mcherm.com> wrote:
> >> If you were using capabilities, you would need to ensure that
> >> restricted interpreters could only get the file object that they
> >> were given. But then _all_ of these fancy versions of the
> >> restrictions would be immediately supported: it would be up to the
> >> users to create secure wrappers implementing the specific
> >> restrictions desired.
> >
> > I agree.  I would prefer this way of doing it.  But as I have said,
> making
> > sure that 'file' does not get out into the wild is tough.
>
> I seem to recall someone mentioned earlier in this discussion the notion
> of somehow throwing an exception when sandboxed code attempts to push a
> file reference onto the interpreter stack.


That was AMK.

I'm not an expert in these matters, so perhaps what I am going to say
> will make no sense, but here goes:
>
> What if there were two copies of the evaluator function. One copy would
> be a slightly slower 'checked' function, that would test all objects for
> a 'check' bit. Any attempt to evaluate a reference to an object with a
> check bit set would throw an exception.


The other eval function would be the 'unchecked' version that would run
> at full speed, just like it does today.
>
> Transitioning from the checked to the unchecked state could only be done
> via C code. So the 'file' wrapper, for example, would switch over to the
> unchecked interpreter before calling the actual methods of 'file'. That
> C wrapper might also check the current permission state to see what
> operations were legal.


So add the proper checks in Python/ceval.c:call_function() to check for this
flag on every object passed in that is called?

Note that whenever a C function sets the interpreter state to
> 'unchecked', that fact is saved on the stack, so that when the function
> returns, the previous state is restored. The function for setting the
> interpreter state is something like PyCall_Unchecked( ... ), which
> restores the interpreter state back to where it was.
>
> Transitioning from unchecked to checked is trickier. Specifically, you
> don't want to ever run sandboxed code in the unchecked state - this is a
> problem for generators, callbacks, and so on. I can think of two
> approaches to handling this:
>
> First approach is to mark all sandboxed code with a bit indicating the
> code is untrusted. Any attempt to call or otherwise invoke a function
> that has this bit set would throw the interpreter into the 'checked'
> state. (Note that transitioning the other way is *not* automatic - i.e.
> calling trusted code does not automatically transition to unchecked
> state.)
>
> The approach is good because it means that if you have intermediary code
> between the wrapper and the sandboxed code, the interpreter still does
> the right thing - it sets the interpreter into checked state.
>
> One problem is how to restore the 'unchecked' state when a function call
> returns. Probably you would have to build this into the code that does
> the state transition.
>
> If marking the sandboxed code isn't feasible, then you'd have to have
> the wrapper objects wrap all of the callbacks with code that goes to
> checked state before calling the callbacks. This means finding all the
> possible holes - however, I suspect that there are far fewer such holes
> than trying to hide all possible 'file' methods. However, one advantage
> of doing this is that restoring the 'unchecked' state after the call
> returns is fairly straightforward.
>
> The advantage of the this whole approach is that once you set the
> 'check' bit on 'file', it doesn't matter whether 'file' gets loose or
> not - you can't do anything with it without throwing an exception.
> Moreover, it also doesn't matter what code path you go through to access
> it. Only C code that flips the interpreter into unchecked state can call
> methods on 'file' without blowing up.
>
> So essentially, what I propose is to define a simple security primitive
> - which essentially comes down to checking a single bit - and use that
> as a basis to create more complex and subtle security mechanisms.


Right, but it does require that the proper verification function be turned
on so that the permission bit on 'file' is checked.  It kind of seems like
'rexec' and its f_restricted flag it set on execution frames, except you are
adding an object-level flag as well.

Either way, the trick is not fouling up switching between the two checking
functions.

-Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20060706/72e33923/attachment-0001.html 

From brett at python.org  Thu Jul  6 19:15:32 2006
From: brett at python.org (Brett Cannon)
Date: Thu, 6 Jul 2006 10:15:32 -0700
Subject: [Python-Dev] doc for new restricted execution design for Python
In-Reply-To: <20060706051543.wxi1nukehvfkk4c4@login.werra.lunarpages.com>
References: <20060706051543.wxi1nukehvfkk4c4@login.werra.lunarpages.com>
Message-ID: <bbaeab100607061015n343e1af2qbcb1a5da0a46c4c0@mail.gmail.com>

On 7/6/06, Michael Chermside <mcherm at mcherm.com> wrote:
>
> I wrote:
> > I would still rather not spread FUD.
>
> Brett Cannon responds:
> > I don't consider it FUD.  Armin in an email said that he thought it
> > was a losing battle to try to hide 'file' from an interpreter.  That
> > is what I am worried about, period.  Everythign else can be
> > protected through resource hiding.
>
> I never intended to say that "we can't hide 'file' from the user"
> was FUD. Only objects crossing between interpreters. And it *might*
> not be FUD, but so far I haven't heard anyone say that they thought
> objects *could* cross between interpreters. I think your proposal
> would read better with a categorical statement that objects cannot
> cross between interpreters (right along with your categorical
> statement that Python code cannot directly access dangerous resources
> without help from the C level), so I present the following challenge:
> can anyone think of a way that an object could "cross" interpreters?
> For instance, are certain basic objects shared (str and int for
> instance) and objects could be passed between interpreters by
> attaching them as attributes of these core objects? I just don't
> know enough about multiple interpreters to be sure -- but somehow I
> thought they had separate object pools.


C extension module attributes will share those objects across interpreters.
Because the initializing function is not called again on extension modules
when imported into another interpreter anything that is exposed by the
module is the same across all interpreters.

-Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20060706/c1f33aaf/attachment.htm 

From brett at python.org  Thu Jul  6 19:16:37 2006
From: brett at python.org (Brett Cannon)
Date: Thu, 6 Jul 2006 10:16:37 -0700
Subject: [Python-Dev] doc for new restricted execution design for Python
In-Reply-To: <Pine.LNX.4.58.0607060022430.22834@server1.LFW.org>
References: <bbaeab100606211733x3dd58226g15e2919e8e2cdd7e@mail.gmail.com>
	<Pine.LNX.4.58.0607040428210.17937@server1.LFW.org>
	<bbaeab100607051156h4c341055w9377fb73a0f0c6fc@mail.gmail.com>
	<Pine.LNX.4.58.0607060022430.22834@server1.LFW.org>
Message-ID: <bbaeab100607061016v363714d9h6a72b997bb3e1c92@mail.gmail.com>

On 7/5/06, Ka-Ping Yee <python-dev at zesty.ca> wrote:
>
> On Wed, 5 Jul 2006, Brett Cannon wrote:
> > On 7/4/06, Ka-Ping Yee <python-dev at zesty.ca> wrote:
> > > In response to Guido's comment about confusing the words "trusted" and
> > > "untrusted", how about "empowered" and "restricted"?
> >
> > Maybe.  I am really starting to lean towards trusted and sandboxed.
>
> It can be risky to use words of the form '*trust*' when talking
> about security because 'trust' can have different meanings in
> different contexts.  (Examples: 'X is trusted' might mean 'X is
> something i feel safe running without restrictions' or 'X *is*
> in fact allowed to run without restrictions' or 'i need to run X
> without restrictions in order to accomplish my task' or 'X is
> something i am relying on for the security of my system'.)
>
> The most common context for 'trusted' that i seem to come across
> is in the phrase 'trusted computing base' (TCB), which refers to
> 'the thing that is supposed to be enforcing security restrictions'
> as opposed to 'something i'm willing to let run unrestricted'.
> So when i read 'trusted code' what comes to mind is the C implementation
> of the Python interpreter, and it may be a good idea to reserve that
> phrase for that purpose, if it's to be used at all.
>
> In any case, let's try to nail down clear names for the different
> pieces we're talking about here, and i gently advise avoiding
> '*trust*' words or using them with very limited meaning.


For the next draft I am going with "trusted" and "sandboxed" just because I
have already revised a decent amount and it is what my brain is defaulting
to right now, but I can change the wording later.

-Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20060706/fa853fe3/attachment.html 

From mcherm at mcherm.com  Thu Jul  6 19:24:12 2006
From: mcherm at mcherm.com (Michael Chermside)
Date: Thu, 06 Jul 2006 10:24:12 -0700
Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
Message-ID: <20060706102412.7wdlblbikhs4cos8@login.werra.lunarpages.com>

I quoted this unwritten bit of Python Zen, attributing it to Tim:
>     Syntax should not look like grit on my monitor.

mwh writes:
> I think it was Anthony:
>
> http://mail.python.org/pipermail/python-dev/2005-July/054581.html

But that's not the original. Turns out, it WAS Anthony, and I had
misquoted. The correct source appears to be:

http://mail.python.org/pipermail/python-dev/2004-August/047179.html

And the correct quote (for the record) is:

       Syntax Shall Not Look Like Grit On Tim's Monitor.


From brett at python.org  Thu Jul  6 19:27:29 2006
From: brett at python.org (Brett Cannon)
Date: Thu, 6 Jul 2006 10:27:29 -0700
Subject: [Python-Dev] doc for new restricted execution design for Python
In-Reply-To: <44AC60D0.1040508@canterbury.ac.nz>
References: <20060705154750.xzn3ft0hq2gw0koc@login.werra.lunarpages.com>
	<bbaeab100607051611pf70f63drd1c4a52d30d8f544@mail.gmail.com>
	<44AC60D0.1040508@canterbury.ac.nz>
Message-ID: <bbaeab100607061027p7d742287n441b153370fef2e3@mail.gmail.com>

On 7/5/06, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
>
> Brett Cannon wrote:
>
> > Armin in an email said that he thought it was
> > a losing battle to try to hide 'file' from an interpreter.
>
> And I would change file() so that it didn't open
> files. Then it would be harmless for code to have
> access to the file class.



Right, that is essentially what I proposed initially with the whole
crippling idea.

What the capabilities supporters are saying is that if we go that route we
will be constantly finding objects that require similar crippling.  It will
degenerate into this constant chasing of our tail to plug security holes
where an object that we did not account for leaked out and wreaked havoc.
What they are saying is that if we harden Python so that references don't
get out without us knowing about it we won't have this run-around.

But then my question has been what makes us trying to cripple objects any
less of a run-around then finding new ways to get at references of 'file' or
any other object?  I have been suggesting the former requires less running
around than the latter.  That is why I have asked that people see how many
ways they can come up with to get to 'file' from a standard interpreter
prompt so we can gauge how bad hiding references might be.

-Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20060706/9a96853e/attachment.htm 

From tomerfiliba at gmail.com  Thu Jul  6 19:22:51 2006
From: tomerfiliba at gmail.com (tomer filiba)
Date: Thu, 6 Jul 2006 19:22:51 +0200
Subject: [Python-Dev] introducing __dir__?
Message-ID: <1d85506f0607061022v76806e5dt6b59f5771ef3f1e3@mail.gmail.com>

i'd guess this has been brought up before, but i'm not aware of it.

anyway --
by what i've gathered, today's dir() "collects" the attributes of the
object from __dict__, __methods__, __members__, and ultimately
__class__. also, if i understood correctly, __methods__ and
__members__ are deprecated. moreover, they are sequence objects,
rather than callbale functions. that means my code can't

my suggestion is simple -- replace this mechanism with a __dir__ -
a special method that returns the list of attributes of the object.

rationale:
* remove deprecated __methods__, etc.
* symmetry -- just like hex() calls __hex__, etc.
* __methods__ and __members__ are lists rather than callable
objects, which means they cannot be updated on-demand

and the reason for bringing this up in the first place --
allowing proxy objects to show the attributes of the internal object.
for example, in rpyc, i had to override __builtin__.dir to achieve
this functionallity, which is of course not the optimal sollution.

implementation:
very straight forward and similar to the rest of the special methods:
* dir() - returns the current scope
* dir(obj) - retuns obj.__dir__(); if the object does not provide __dir__,
uses the current mechanism as a fallback, or something similar

the ultimate object type would grow a default __dir__ method that
uses the same modus operandi as today's dir, which deriving
classes could override to provide custom listings.
therefore, i don't expect any backward-compatibility issues.

aye or nay?


-tomer

From brett at python.org  Thu Jul  6 19:40:38 2006
From: brett at python.org (Brett Cannon)
Date: Thu, 6 Jul 2006 10:40:38 -0700
Subject: [Python-Dev] Restricted execution: what's the threat model?
In-Reply-To: <20060706053756.ovcm9a05kko0o008@login.werra.lunarpages.com>
References: <20060706053756.ovcm9a05kko0o008@login.werra.lunarpages.com>
Message-ID: <bbaeab100607061040w4716bceje0e146794a86419a@mail.gmail.com>

[replying to both Ping and Michael in the same email]

On 7/6/06, Michael Chermside <mcherm at mcherm.com> wrote:
>
> Ka-Ping Yee writes:
> > i'm starting to think
> > that it would be good to clarify what kinds of threats we are
> > trying to defend against, and specify what invariants we are
> > intending to preserve.
>
> Yes!
>
> > So here are a couple of questions for clarification (some with my
> > guesses as to their answers):
>
> Okay, I'll throw in my thoughts also.
>
> > 1.  When we say "restricted/untrusted/<whatever> interpreter" we
> >     don't really mean that the *interpreter* is untrusted, right?
> >     We mean that the Python code that runs in that interpreter is
> >     untrusted (i.e. to be prevented from doing harm), right?
>
> Agreed. My interpretation of the proposal was that interpreters
> were either "sandboxed" or "trusted". "Sandboxed" means that there
> are security restrictions imposed at some level (perhaps even NO
> restrictions). "Trusted" means that the interpreter implements
> no security restrictions (beyond what CPython already implements,
> which isn't much) and thus runs faster.


Yep.

> 2.  I'm assuming that the implementation of the Python interpreter
> >     is always trusted
>
> Sure... it's got to be.


Yep.

> What do
> > we take the Trusted Computing Base to include?  The Python VM
> > implementation -- plus all the builtin objects and C modules?
> > Plus the whole standard library?
>
> My interpretation of Brett's proposal is that the CPython developers
> would try to ensure that Python VM had no "security holes" when
> running in sandboxed mode. Of course, we also "try" to ensure no
> crashes are possible also, and while we're quite good, we're not
> perfect.
>
> Beyond that, all pure-python modules with source available (whether
> in the stdlib or not) can be "trusted" because they run in a
> sandboxed VM. All C modules are *up to the user*. Brett proposes
> to provide a default list of useful-but-believed-to-be-safe modules
> in the stdlib, but the user can configure the C-module whitelist
> to whatever she desires.


Michael has it on the money.

> 3.  Is it part of the plan that we want to protect Python code from
> >     other Python code?  For example, should a Python program/function
> >     X be able to say "i want to launch/call program/function Y with
> >     *these* parameters and have it run under *these* limitations?"
> >     This has a big impact on the model.
>
> Now *that* is a good question. I would say the answer is a partial
> "no", because there are pieces of Brett's security model that are
> tied to the interpreter instance. Python code cannot launch another
> interpreter (but perhaps it *should* be able to?), so it cannot
> modify those restrictions for new Python code it launches.
>
> However, I would rather like to allow Python code to execute other
> code with greater restrictions, although I would accept all kinds
> of limitations and performance penalties to do so. I would be
> satisfied if the caller could restrict certain things (like web
> and file access) but not others (like memory limits or use of
> stdout). I would satisfied if the caller paid huge overhead costs
> of launching a separate interpreter -- heck, even a separate
> process. And if it is willing to launch a separate process, then
> Brett's model works just fine: allow the calling code to start
> a new (restricted) Python VM.


The plan is that there is no sandboxed eval() that runs unsafe code from a
trusted interpreter within its namespace.  I hope to provide Python code
access to running a sandboxed interpreter where you can pass in a string to
be executed, but the namespace for that sandboxed interpreter will be fresh
and will not carry over in any way from the trusted interpreter.

> We want to be able to guarantee that...
> >
> >   A.  The interpreter will not crash no matter what Python code
> >       it is given to execute.
>
> Agreed. We already want to guarantee that, with the caveat that the
> guarantee doesn't apply to a few special modules (like ctypes).


Right, which is why I have been trying to plug the various known crashers
that do not rely upon a specific extension module from being imported.

>  B.  Python programs running in different interpreters embedded
> >       in the same process cannot communicate with each other.
>
> I don't want to guarantee this, does someone else? It's
> astonishingly hard... there are all kinds of clever "knock on the
> walls" tricks. For instance, communicate by varying your CPU
> utilization up and down in regular patterns.
>
> I'd be satisfied if they could pass information (perhaps even
> someday provide a library making it *easy* to do so), but could
> not pass unforgable items like Python object references, open file
> descriptors, and so forth.


Or at least cannot communicate without explicit allowances to do so.

As for knocking on the walls, if you protect access to that kind of
information well, it shouldn't be a problem.

>   C.  Python programs running in different interpreters embedded
> >       in the same process cannot access each other's Python objects.
>
> I strengthen that slightly to all "unforgable" items, not just
> object references.


I would change that to add the caveat that what is exposed by a C extension
module attribute will be shared.  That is an implementation detail of
multiple interpreters.

>   D.  A given piece of Python code cannot access or communicate
> >       with certain Python objects in the same interpreter.
> >
> >   E.  A given piece of Python code can access only a limited set
> >       of Python objects in the same interpreter.
>
> Hmmm. I'm not sure.


Not quite sure what you are getting at here, Ping.  Are you saying to run
code within an interpreter (sandboxed and not) and restricted even more
beyond what the interpreter has been given by the security settings?

These emails have convinced me to add a "Threat Model" section for the next
draft of the design doc.

-Brett

-- Michael Chermside
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20060706/97850f3c/attachment.htm 

From fdrake at acm.org  Thu Jul  6 19:41:56 2006
From: fdrake at acm.org (Fred L. Drake, Jr.)
Date: Thu, 6 Jul 2006 13:41:56 -0400
Subject: [Python-Dev] introducing __dir__?
In-Reply-To: <1d85506f0607061022v76806e5dt6b59f5771ef3f1e3@mail.gmail.com>
References: <1d85506f0607061022v76806e5dt6b59f5771ef3f1e3@mail.gmail.com>
Message-ID: <200607061341.56945.fdrake@acm.org>

On Thursday 06 July 2006 13:22, tomer filiba wrote:
 > my suggestion is simple -- replace this mechanism with a __dir__ -
 > a special method that returns the list of attributes of the object.
 >
 > rationale:
 > * remove deprecated __methods__, etc.
 > * symmetry -- just like hex() calls __hex__, etc.
 > * __methods__ and __members__ are lists rather than callable
 > objects, which means they cannot be updated on-demand

+1


  -Fred

-- 
Fred L. Drake, Jr.   <fdrake at acm.org>

From brett at python.org  Thu Jul  6 19:48:39 2006
From: brett at python.org (Brett Cannon)
Date: Thu, 6 Jul 2006 10:48:39 -0700
Subject: [Python-Dev] what can we do to hide the 'file' type?
In-Reply-To: <20060706083127.csxctx0sahr4kwk8@login.werra.lunarpages.com>
References: <20060706083127.csxctx0sahr4kwk8@login.werra.lunarpages.com>
Message-ID: <bbaeab100607061048x1f8f8239m653ea93735b61a16@mail.gmail.com>

On 7/6/06, Michael Chermside <mcherm at mcherm.com> wrote:
>
> Armin Rigo writes:
> > I don't think I can "sign off" [on hiding the file type].  Really hiding
> > Python objects is quite hard IMHO.


I agree. But we don't have to give up yet. How about instead of hiding
> file, we cripple it. Completely. Modify the file type so that when
> executing on a sandboxed interpreter, all of the dangerous methods
> and attributes of file throw exceptions.


This is basically what I proposed in the first place! <runs around in
circles, pulling at his hair like a crazy man>

Then we create a separate thing (in C) called a "SecureFileWrapper".
> It has methods that are passed a reference to a file object and
> can invoke the methods without error. We provide a means for obtaining
> a SecureFileWrapper bound to a given file (perhaps open()).


Yeah, it would be through open() if we returned wrappers instead of
performing the checks directly in file itself.

Essentially, we give up on hiding file, which is a frequently-used
> type, and very hard to hide, and instead we rely on our ability to
> write a reliably secure "SecureFileWrapper" class (in C).


 That is another possibility.  Should simplify the code as well by having
less checks and just have pure PySandbox_IsTrusted() checks in 'file' itself
in unsafe places instead of a ton checks that the file being accessed is
allowed.

-Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20060706/68c72935/attachment.htm 

From brett at python.org  Thu Jul  6 19:51:40 2006
From: brett at python.org (Brett Cannon)
Date: Thu, 6 Jul 2006 10:51:40 -0700
Subject: [Python-Dev] what can we do to hide the 'file' type?
In-Reply-To: <20060706110312.GB15690@code0.codespeak.net>
References: <bbaeab100607051701h144c81cam82f5a9dce33fae59@mail.gmail.com>
	<20060706110312.GB15690@code0.codespeak.net>
Message-ID: <bbaeab100607061051j7a7b5d1cjef8b2399cdbf9edc@mail.gmail.com>

On 7/6/06, Armin Rigo <arigo at tunes.org> wrote:
>
> Hi Brett,
>
> On Wed, Jul 05, 2006 at 05:01:48PM -0700, Brett Cannon wrote:
> > And if Armin and/or Samuele sign off that what we find is most likely
> (with
> > "most likely" equalling 99% chance) all there is, then bonus points and
> I
> > will *really* be convinced.  =)
>
> I don't think I can "sign off" that.  Really hiding Python objects is
> quite hard IMHO.


=)  That's fine.  I didn't expect you to, especially without people either
finding more instances of ways to get to 'file' or stating that they really
tried and couldn't find any way to get to it.


-Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20060706/acbb3b0b/attachment.html 

From mcherm at mcherm.com  Thu Jul  6 21:07:51 2006
From: mcherm at mcherm.com (Michael Chermside)
Date: Thu, 06 Jul 2006 12:07:51 -0700
Subject: [Python-Dev] what can we do to hide the 'file' type?
Message-ID: <20060706120751.kd6pvgex6jtsgg4k@login.werra.lunarpages.com>

Me:
> I agree. But we don't have to give up yet. How about instead of hiding
> file, we cripple it. Completely. Modify the file type so that when
> executing on a sandboxed interpreter, all of the dangerous methods
> and attributes of file throw exceptions.

Brett:
> This is basically what I proposed in the first place! <runs around  
> in circles, pulling at his hair like a crazy man>

Not quite. Your original proposal had the file type throwing
exceptions when the user did something they weren't allowed to
(access a file not on the list, etc). This version proposes that
it *always* fails, and creates a separate beast (the
SecureFileWrapper) for applying the restrictions. Why? Because
if the C code in file enforces the rules, then all possible
rules need to be written in advance, and you have to hold long
arguments about whether to allow subdirectories, restrict file
sizes, etc. Whereas SecureFileWrapper could delegate its
restrictions to Python functions provided by the USER and then
USERS could design whatever level of restriction they wanted.

Imagine we want some code to be able to open files only if they
contain the letter 'w':

     # get my own wrapper from __builtins__
     myFileWrapper = file

     # define a function to constrain my callers
     def filenameContainsLetterW(path, mode):
         filename = os.path.basename(path)
         if 'w' not in filename and 'W' not in filename:
             raise PyXXX_SecurityException

     # create more restrictive wrapper
     class MustHaveW_FileWrapper:
         __metaclass__ = SecureFileWrapperMeta
         wrapped_file =
         init_condition = filenameContainsLetterW

     # register the wrapper so it applies to any code
     # in this stack frame or lower. The restriction is
     # automatically lifted when the current stack
     # frame exits.
     PyXXX_RegisterFileWrapper( MustHaveW_FileWrapper )

     # Invoke less-trusted code with restrictions in place
     less_trusted_code()

If the code fragment shown above ALREADY received
a wrapped form of file which was restricted to read-only
access, then less_trusted_code() would be restricted
to read-only access to files containing 'w'.

Okay, the syntax needs work, but the idea is that I can
defin restrictions *in python* and apply them to other
code. Using the stack to enforce wrappers is something
Python code cannot get around (although it does prevent
passing more-powerful callbacks to later stackframes).

It all depends on whether we think that a simple set
of restrictions implementable in C (such as 4 lists:
read-only files, read-write files, read-only dirs,
write-only dirs) will be sufficient, or if it is
valuable to allow end users to fine-tune the
restrictions.

-- Michael Chermside


From martin at v.loewis.de  Thu Jul  6 21:52:27 2006
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Thu, 06 Jul 2006 21:52:27 +0200
Subject: [Python-Dev] Subversion outage Friday 15:00 GMT
Message-ID: <44AD69FB.7040002@v.loewis.de>

I plan to do some subversion administration
tomorrow; in order to be able to roll back changes,
I have to disable write access during these
changes.

The outage shouldn't last longer than one hour;
most likely, it will be much faster.

Regards,
Martin

From brett at python.org  Thu Jul  6 22:32:16 2006
From: brett at python.org (Brett Cannon)
Date: Thu, 6 Jul 2006 13:32:16 -0700
Subject: [Python-Dev] what can we do to hide the 'file' type?
In-Reply-To: <20060706120751.kd6pvgex6jtsgg4k@login.werra.lunarpages.com>
References: <20060706120751.kd6pvgex6jtsgg4k@login.werra.lunarpages.com>
Message-ID: <bbaeab100607061332i630d99fl84be952d4018c910@mail.gmail.com>

On 7/6/06, Michael Chermside <mcherm at mcherm.com> wrote:
>
> Me:
> > I agree. But we don't have to give up yet. How about instead of hiding
> > file, we cripple it. Completely. Modify the file type so that when
> > executing on a sandboxed interpreter, all of the dangerous methods
> > and attributes of file throw exceptions.
>
> Brett:
> > This is basically what I proposed in the first place! <runs around
> > in circles, pulling at his hair like a crazy man>
>
> Not quite. Your original proposal had the file type throwing
> exceptions when the user did something they weren't allowed to
> (access a file not on the list, etc). This version proposes that
> it *always* fails, and creates a separate beast (the
> SecureFileWrapper) for applying the restrictions. Why? Because
> if the C code in file enforces the rules, then all possible
> rules need to be written in advance, and you have to hold long
> arguments about whether to allow subdirectories, restrict file
> sizes, etc. Whereas SecureFileWrapper could delegate its
> restrictions to Python functions provided by the USER and then
> USERS could design whatever level of restriction they wanted.


Ah, OK, that makes more sense.  I was not thinking about allowing for
specifying a factory function to return the specific object to use when
using open().  That could be rather handy and cool.  I will definitely see
if I can work it into the API in a reasonable way.

-Brett

Imagine we want some code to be able to open files only if they
> contain the letter 'w':
>
>      # get my own wrapper from __builtins__
>      myFileWrapper = file
>
>      # define a function to constrain my callers
>      def filenameContainsLetterW(path, mode):
>          filename = os.path.basename(path)
>          if 'w' not in filename and 'W' not in filename:
>              raise PyXXX_SecurityException
>
>      # create more restrictive wrapper
>      class MustHaveW_FileWrapper:
>          __metaclass__ = SecureFileWrapperMeta
>          wrapped_file =
>          init_condition = filenameContainsLetterW
>
>      # register the wrapper so it applies to any code
>      # in this stack frame or lower. The restriction is
>      # automatically lifted when the current stack
>      # frame exits.
>      PyXXX_RegisterFileWrapper( MustHaveW_FileWrapper )
>
>      # Invoke less-trusted code with restrictions in place
>      less_trusted_code()
>
> If the code fragment shown above ALREADY received
> a wrapped form of file which was restricted to read-only
> access, then less_trusted_code() would be restricted
> to read-only access to files containing 'w'.
>
> Okay, the syntax needs work, but the idea is that I can
> defin restrictions *in python* and apply them to other
> code. Using the stack to enforce wrappers is something
> Python code cannot get around (although it does prevent
> passing more-powerful callbacks to later stackframes).
>
> It all depends on whether we think that a simple set
> of restrictions implementable in C (such as 4 lists:
> read-only files, read-write files, read-only dirs,
> write-only dirs) will be sufficient, or if it is
> valuable to allow end users to fine-tune the
> restrictions.
>
> -- Michael Chermside
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20060706/52bc6eb9/attachment.htm 

From python-dev at zesty.ca  Fri Jul  7 00:17:44 2006
From: python-dev at zesty.ca (Ka-Ping Yee)
Date: Thu, 6 Jul 2006 17:17:44 -0500 (CDT)
Subject: [Python-Dev] Musings on concurrency and scoping ("replacing"
	Javascript)
In-Reply-To: <bbaeab100606211733x3dd58226g15e2919e8e2cdd7e@mail.gmail.com>
References: <bbaeab100606211733x3dd58226g15e2919e8e2cdd7e@mail.gmail.com>
Message-ID: <Pine.LNX.4.58.0607061642420.22834@server1.LFW.org>

On Wed, 21 Jun 2006, Brett Cannon wrote:
> I have been working on a design doc for restricted execution of Python
> as part of my dissertation for getting Python into Firefox to replace
> JavaScript on the web.

I've been doing a bunch of Firefox extension programming in Javascript
and suddenly a few of the recent topics here came together in my head
in a silent kapow of thoughts.  This is kind of a side note to the
security discussion, but they're all interconnected: network
programming, concurrency, lexical scoping, security.

Client-side web scripting tends to have a callback/continuation-ish
concurrency style because it has to deal with network transactions
(which can stall for long periods of time) in a user interface that
is expected to stay always responsive.  The Firefox API is full of
listeners/observers, events, and continuation-like things.  So one
thing to consider is that, when Python is used for these purposes,
it may be written in a specialized style.

As i write JavaScript in this style i find i use nested functions
a lot.  When i want to set up a callback that uses variables in the
current context, the natural thing to do is to define a new function
in the local namespace.  And if that function has to also provide a
callback, then it has another function nested within it and so on.

    function spam() {
        var local_A = do_work();
        do_network_transaction(
            new function(result_1) {
                var local_B = do_work(result_1);
                do_network_transaction(
                    new function(result_2) {
                        do_work(local_A, local_B, result_1, result_2);
                        ...
                    }
                );
            }
        );
    }

So it is a possible consequence of embedding Python in Firefox that
people will be using nested functions and lexical scoping in Python
more often, which makes the recent discussion about access to
enclosing scopes more significant.

This is even related to security as well.  Namespaces and lexical
scoping are a natural and visually apparent way to limit access.
If, for example, result_1 and result_2 in the above example are
security-sensitive objects like secret keys or networking functions,
you can see just by inspection that they cannot leak outside of
spam() except by directly being passed in an outgoing function call.

The standard Pythonic response to nested functions is to translate
them into classes.  But the nested function style has two advantages:

    1.  Variables are more appropriately scoped; they exist
        only where they are meaningful.  (In a class, all the
        variables would be mixed into one namespace, where
        some of them would be invalid some of the time.)

    2.  Local variables are private.  (Class instances don't
        get their own private namespaces to play in.)

The first becomes more significant when in a more continuation-y
style because it helps keep the various continuations from
interfering with each other.  The second becomes more significant
if you care about restricting untrusted Python code.


-- ?!ng

From brett at python.org  Thu Jul  6 00:07:23 2006
From: brett at python.org (Brett Cannon)
Date: Wed, 5 Jul 2006 15:07:23 -0700
Subject: [Python-Dev] About a month until PSF call for test trackers closes!
Message-ID: <bbaeab100607051507t44987bffnf0372039966c2b3f@mail.gmail.com>

Back at the beginning of June, the Python Software Foundation's
Infrastructure committee sent out an email requesting people to help us find
a replacement tracker for SourceForge (the original announcement can be
found at http://wiki.python.org/moin/CallForTrackers ).  We asked that
people put test trackers loaded with a data dump of Python's SourceForge
tracker data online for us to play with.  We said that we would close
acceptance of test trackers as of August 7.

That close date is a little over a month away!  If you want to help get your
favorite tracker up and going for the Infrastructure committee to evaluate,
please visit http://wiki.python.org/moin/CallForTrackers and see if you can
help out!  We already have a tracker for JIRA up and loaded with the SF data
to poke around with.  There are also Trac and Roundup trackers in the
planning stages that could use some help in getting the SF data imported and
getting the servers up.  In order for a tracker to be considered it *must*
have the SF data loaded up.  This will be a necessary step if the tracker is
accepted, plus it lets us see how well the tracker lets us manage a large
number of bugs.

If you have questions or concerns, please email infrastructure at
python.org(it is subscription-protected, but your email will be
accepted; the
subscription page is at
http://mail.python.org/mailman/listinfo/infrastructure ).  Thank you to
those that have helped so far and those that do in the future.

-Brett Cannon
Chairman, PSF Infrastructure Committee
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20060705/2cc322e8/attachment.html 

From pje at telecommunity.com  Fri Jul  7 01:04:04 2006
From: pje at telecommunity.com (Phillip J. Eby)
Date: Thu, 06 Jul 2006 19:04:04 -0400
Subject: [Python-Dev] Musings on concurrency and scoping ("replacing"
 Javascript)
In-Reply-To: <Pine.LNX.4.58.0607061642420.22834@server1.LFW.org>
References: <bbaeab100606211733x3dd58226g15e2919e8e2cdd7e@mail.gmail.com>
	<bbaeab100606211733x3dd58226g15e2919e8e2cdd7e@mail.gmail.com>
Message-ID: <5.1.1.6.0.20060706185558.01f1fe40@sparrow.telecommunity.com>

At 05:17 PM 7/6/2006 -0500, Ka-Ping Yee wrote:
>On Wed, 21 Jun 2006, Brett Cannon wrote:
> > I have been working on a design doc for restricted execution of Python
> > as part of my dissertation for getting Python into Firefox to replace
> > JavaScript on the web.
>
>I've been doing a bunch of Firefox extension programming in Javascript
>and suddenly a few of the recent topics here came together in my head
>in a silent kapow of thoughts.  This is kind of a side note to the
>security discussion, but they're all interconnected: network
>programming, concurrency, lexical scoping, security.
>
>Client-side web scripting tends to have a callback/continuation-ish
>concurrency style because it has to deal with network transactions
>(which can stall for long periods of time) in a user interface that
>is expected to stay always responsive.  The Firefox API is full of
>listeners/observers, events, and continuation-like things.  So one
>thing to consider is that, when Python is used for these purposes,
>it may be written in a specialized style.
>
>As i write JavaScript in this style i find i use nested functions
>a lot.  When i want to set up a callback that uses variables in the
>current context, the natural thing to do is to define a new function
>in the local namespace.  And if that function has to also provide a
>callback, then it has another function nested within it and so on.
>
>     function spam() {
>         var local_A = do_work();
>         do_network_transaction(
>             new function(result_1) {
>                 var local_B = do_work(result_1);
>                 do_network_transaction(
>                     new function(result_2) {
>                         do_work(local_A, local_B, result_1, result_2);
>                         ...
>                     }
>                 );
>             }
>         );
>     }
>
>So it is a possible consequence of embedding Python in Firefox that
>people will be using nested functions and lexical scoping in Python
>more often, which makes the recent discussion about access to
>enclosing scopes more significant.

As much as I'd love to have the nested scope feature, I think it's only 
right to point out that the above can be rewritten as something like this 
in Python 2.5:

     def spam():
         local_A = do_work()
         result_1 = yield do_network_transaction()
         local_B = do_work(result_1)
         result_2 = yield do_network_transaction()
         do_work(local_A, local_B, result_1, result_2)
         ...

All you need is an appropriate trampoline (possibly just a decorator) that 
takes the objects yielded by the function, and uses them up to set up 
callbacks that resume the generator with the returned result.


>This is even related to security as well.  Namespaces and lexical
>scoping are a natural and visually apparent way to limit access.
>If, for example, result_1 and result_2 in the above example are
>security-sensitive objects like secret keys or networking functions,
>you can see just by inspection that they cannot leak outside of
>spam() except by directly being passed in an outgoing function call.
>
>The standard Pythonic response to nested functions is to translate
>them into classes.  But the nested function style has two advantages:
>
>     1.  Variables are more appropriately scoped; they exist
>         only where they are meaningful.  (In a class, all the
>         variables would be mixed into one namespace, where
>         some of them would be invalid some of the time.)
>
>     2.  Local variables are private.  (Class instances don't
>         get their own private namespaces to play in.)
>
>The first becomes more significant when in a more continuation-y
>style because it helps keep the various continuations from
>interfering with each other.  The second becomes more significant
>if you care about restricting untrusted Python code.

These are all legitimate benefits of a functional style; it's just not 
necessarily the case that they also argue in favor of write access to outer 
function scopes, since this isn't necessary for writing callbacks, as in 
your original example.  Co-routines are clearly superior to nested-function 
callbacks when the intended control flow is relatively linear.  This is, if 
I recall correctly, why we added them.  :)


From evan at 4-am.com  Fri Jul  7 01:08:22 2006
From: evan at 4-am.com (Evan Simpson)
Date: Thu, 06 Jul 2006 18:08:22 -0500
Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
In-Reply-To: <44AA1FF6.1050501@acm.org>
References: <44AA1FF6.1050501@acm.org>
Message-ID: <e8k58c$q7u$1@sea.gmane.org>

Talin wrote:
> I propose to create a new type of scoping rule, which I will call 
> "explicit" lexical scoping, that will co-exist with the current 
> "implicit" scoping rule that exists in Python today.

I'd like to toss one more variant into the mix.  If we really need to
address variables in an intermediate scope, the most explicit way that I
can think of doing so is to write (using Philip's example):

def counter(num):
    scope as outer # "outer" is an arbitrary identifier
    def inc():
        outer.num += 1
        return outer.num
    return inc

This is somewhat similar to the unworkable "use the function name"
method that's been suggested.  The "scope as X" statement associates the
name "X" with the namespace of local variables in the scope in which it
is executed.  Such names are "lexically scoped", and only allow access
to or rebinding of existing names from the originating scope (i.e. no
"del outer.num" allowed).

Cheers,

Evan @ 4-am


From jan-python at maka.demon.nl  Fri Jul  7 01:00:03 2006
From: jan-python at maka.demon.nl (Jan Kanis)
Date: Fri, 07 Jul 2006 01:00:03 +0200
Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
In-Reply-To: <5.1.1.6.0.20060706120457.01f1bab8@sparrow.telecommunity.com>
References: <20060706165539.sl1zeof09w0sks0o@webmail.nl.demon.net>
	<mailman.29417.1152163446.27774.python-dev@python.org>
	<5.1.1.6.0.20060706120457.01f1bab8@sparrow.telecommunity.com>
Message-ID: <op.tcah6dc7aed6q0@e500>

On Thu, 06 Jul 2006 18:28:12 +0200, Phillip J. Eby <pje at telecommunity.com>  
wrote:

> Here's the reason I think this keeps coming up, and why Guido's "just  
> use a class" argument doesn't really address the actual problem that's  
> taking place.

I agree this argument is not generally applicable in every case, but why  
not in this specific situation?

> In short: in *theory*, a rebinding operator or "nonlocal" declaration is  
> unnecessary.  In *practice*, having one seems quite useful every time  
> you wander down the path that leads to having to rewrite your code just  
> because the language won't let you do that one tiny thing -

I think this argument is a too general one. To me it is too close to  
"let's add every possible feature we can find, because it might be usefull  
to someone" :)
One of the things I like about python is that it doesn't do this, and  
therefore the manual stays relatively small and I don't have to remember  
all kinds of rarely used features to make best use of the language. (I  
assume this is not a point of debate. repeating: Python is not Lisp ;-) )

Most of the arguments I've seen on the list are about 'how can we  
implement this', I'd like to see more arguments on whether this should be  
implemented at all.
I was hoping someone would come up with a good example, so does anyone  
have one??

> - or so it feels like to the person who's experiencing it.

Have you ever been that person, or come across such a situation?

O, and I don't think the inc() example is a good one. In this incrementer  
the function call is all about the side effects, it's even in the name  
'increment'. Incrementing is useless unless you increment /something/, so  
this should be better implemented as a class.

From pje at telecommunity.com  Fri Jul  7 01:25:19 2006
From: pje at telecommunity.com (Phillip J. Eby)
Date: Thu, 06 Jul 2006 19:25:19 -0400
Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
In-Reply-To: <op.tcah6dc7aed6q0@e500>
References: <5.1.1.6.0.20060706120457.01f1bab8@sparrow.telecommunity.com>
	<20060706165539.sl1zeof09w0sks0o@webmail.nl.demon.net>
	<mailman.29417.1152163446.27774.python-dev@python.org>
	<5.1.1.6.0.20060706120457.01f1bab8@sparrow.telecommunity.com>
Message-ID: <5.1.1.6.0.20060706190445.0200a960@sparrow.telecommunity.com>

At 01:00 AM 7/7/2006 +0200, Jan Kanis wrote:
>On Thu, 06 Jul 2006 18:28:12 +0200, Phillip J. Eby <pje at telecommunity.com>
>wrote:
>
>>Here's the reason I think this keeps coming up, and why Guido's "just
>>use a class" argument doesn't really address the actual problem that's
>>taking place.
>
>I agree this argument is not generally applicable in every case, but why
>not in this specific situation?

I'm saying that "just use a class" doesn't help in the incremental 
case.  If you happen to know *ahead of time* that you will need a callback 
to modify state, you can perhaps do this, just like you can always coast 
your car to a stop if you have enough advance notice.  However, sometimes 
you really want to have brakes.  :)


>>In short: in *theory*, a rebinding operator or "nonlocal" declaration is
>>unnecessary.  In *practice*, having one seems quite useful every time
>>you wander down the path that leads to having to rewrite your code just
>>because the language won't let you do that one tiny thing -
>
>I think this argument is a too general one. To me it is too close to
>"let's add every possible feature we can find, because it might be usefull
>to someone" :)

Not at all.  It's an argument regarding the incremental evolution of 
code.  Python tries to keep the growth of complexity in a code base 
relatively flat -- incremental effort should be rewarded with incremental 
benefit.  Discontinuities of complexity are to be avoided.

The classic example is "Hello world", where in Java you must learn about 
six or eight different things, but in Python it is just 'print "Hello 
world"'.  If you want to then make it a function, you indent it and add a 
'def'.  It is all very minimal and incremental, and for the most part, the 
things that people want to add to Python tend to be wherever 
discontinuities can occur.

It is also true that, because Python is already so well-designed, that 
these remaining feature areas tend to lean toward the minor and/or obscure: 
coroutines, conditional expressions, etc. etc.


>One of the things I like about python is that it doesn't do this, and
>therefore the manual stays relatively small and I don't have to remember
>all kinds of rarely used features to make best use of the language. (I
>assume this is not a point of debate. repeating: Python is not Lisp ;-) )

And one of the things that makes it not Lisp is that not everyone is free 
to go around and actually add the new syntax to support their use 
cases.  Guido must first be convinced.  Discussions like this one are how 
we convince him.  ;)


>>- or so it feels like to the person who's experiencing it.
>
>Have you ever been that person, or come across such a situation?

Many times.  The hard thing about trying to provide use cases for this is 
that of course you can always find another way to write it.  It's just that 
sometimes the nested function is a perfect solution at point in time A, and 
then at point in time B, a change in the program requires that the nested 
function mutate a bit of state, resulting in either a rewrite to do it the 
"right" way, or hacking mutable objects.

Both are user-experience discontinuities: a sudden sharp rise in effort 
compared to reward.  This then produces the perception of a "wart".  Yes, 
it's purely a superficial problem.  That's why it's called a "wart".  :)


>O, and I don't think the inc() example is a good one. In this incrementer
>the function call is all about the side effects, it's even in the name
>'increment'. Incrementing is useless unless you increment /something/, so
>this should be better implemented as a class.

Any example that's intended to motivate lexical scope modification is going 
to suffer from either being bogus due to oversimplification, or confusing 
due to application-specificity.  That doesn't make it something that 
doesn't happen, it's just that the circumstances of it happening are 
somewhat more organic.

By the way, I'm leaving out the people who have backgrounds in Lisp or 
Scheme and are trying to write it in Python.  I don't really care for Lisp, 
myself, although my Python has become increasingly functional over 
time.  Not because of Lisp (which I've never used apart from a few Emacs 
hacks 12 years ago, when I didn't even know what functional programming 
*was*), but because it's often the easiest way to do something.  At least, 
until you bump into the rebinding problem.

Personally, the idea to use a simple namespace object to store mutable 
variables as attributes seems rather appealing as a workaround solution in 
the absence of a rebinding operator.  The other solution I thought of was 
providing a function called 'rebind()' that could be used like this:

      rebind(x=23)

to change 'x' in the nearest enclosing scope.  This solves the problem 
without adding any syntax, and it's incremental at the point of use.


From jan-python at maka.demon.nl  Fri Jul  7 01:47:50 2006
From: jan-python at maka.demon.nl (Jan Kanis)
Date: Fri, 07 Jul 2006 01:47:50 +0200
Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
In-Reply-To: <5.1.1.6.0.20060706190445.0200a960@sparrow.telecommunity.com>
References: <5.1.1.6.0.20060706120457.01f1bab8@sparrow.telecommunity.com>
	<20060706165539.sl1zeof09w0sks0o@webmail.nl.demon.net>
	<mailman.29417.1152163446.27774.python-dev@python.org>
	<5.1.1.6.0.20060706190445.0200a960@sparrow.telecommunity.com>
Message-ID: <op.tcakd0ldaed6q0@e500>

On Fri, 07 Jul 2006 01:25:19 +0200, Phillip J. Eby <pje at telecommunity.com>  
wrote:

>
>>> - or so it feels like to the person who's experiencing it.
>>
>> Have you ever been that person, or come across such a situation?
>
> Many times.  The hard thing about trying to provide use cases for this  
> is that of course you can always find another way to write it.  It's  
> just that sometimes the nested function is a perfect solution at point  
> in time A, and then at point in time B, a change in the program requires  
> that the nested function mutate a bit of state, resulting in either a  
> rewrite to do it the "right" way, or hacking mutable objects.

Well, not an actual example, but close enough.  Leaving the rest to Guido  
then.

From python-dev at zesty.ca  Fri Jul  7 02:04:18 2006
From: python-dev at zesty.ca (Ka-Ping Yee)
Date: Thu, 6 Jul 2006 19:04:18 -0500 (CDT)
Subject: [Python-Dev] Musings on concurrency and scoping ("replacing"
 Javascript)
In-Reply-To: <5.1.1.6.0.20060706185558.01f1fe40@sparrow.telecommunity.com>
References: <bbaeab100606211733x3dd58226g15e2919e8e2cdd7e@mail.gmail.com>
	<bbaeab100606211733x3dd58226g15e2919e8e2cdd7e@mail.gmail.com>
	<5.1.1.6.0.20060706185558.01f1fe40@sparrow.telecommunity.com>
Message-ID: <Pine.LNX.4.58.0607061900020.22834@server1.LFW.org>

On Thu, 6 Jul 2006, Phillip J. Eby wrote:
> As much as I'd love to have the nested scope feature, I think it's only
> right to point out that the above can be rewritten as something like this
> in Python 2.5:
>
>      def spam():
>          local_A = do_work()
>          result_1 = yield do_network_transaction()
>          local_B = do_work(result_1)
>          result_2 = yield do_network_transaction()
>          do_work(local_A, local_B, result_1, result_2)
>          ...
>
> All you need is an appropriate trampoline (possibly just a decorator) that
> takes the objects yielded by the function, and uses them up to set up
> callbacks that resume the generator with the returned result.

Clever!  Could you help me understand what goes on in
do_network_transaction() when you write it this way?  In the
Firefox/JavaScript world, the network transaction is fired off
in another thread, and when it's done it posts an event back
to the JavaScript thread, which triggers the callback.

And what happens if you want to supply more than one continuation?
In my JavaScript code i'm setting up two continuations per step --
one for success and one for failure, since with a network you never
know what might happen.


-- ?!ng

From guido at python.org  Fri Jul  7 02:10:59 2006
From: guido at python.org (Guido van Rossum)
Date: Fri, 7 Jul 2006 02:10:59 +0200
Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
In-Reply-To: <5.1.1.6.0.20060706120457.01f1bab8@sparrow.telecommunity.com>
References: <mailman.29417.1152163446.27774.python-dev@python.org>
	<20060706165539.sl1zeof09w0sks0o@webmail.nl.demon.net>
	<17581.9926.915438.200819@montanaro.dyndns.org>
	<5.1.1.6.0.20060706120457.01f1bab8@sparrow.telecommunity.com>
Message-ID: <ca471dc20607061710w52123d4cv83fcb5a8400f5f60@mail.gmail.com>

On 7/6/06, Phillip J. Eby <pje at telecommunity.com> wrote:
> Here's the reason I think this keeps coming up, and why Guido's "just use a
> class" argument doesn't really address the actual problem that's taking place.

(And note that I've recently gone on record as doubting that argument myself.)

> When you are writing some function, and you find yourself using a nested
> function because it's the closest match for something you're doing, there
> occasionally comes a point at which you realize that, gosh, you need to
> mutate something in an outer scope.  At that point, your choices are to
> either kludge it with a mutable, or to reorganize the whole thing.  Both of
> these *feel* like warts because they're making you do less-than-optimal
> things.  Your mental "flow" is disrupted because the language is forcing
> you to work around it, rather than having a way of working with it.

Right. I'm actually not a big fan of the word "flow" -- it often
doesn't apply to the way I work myself, and it tends to emphasize the
needs of the writer at the expense of the reader. (Perl programmers
have excellent flow due to the many ways of doing things in that
language. 'nuff said.)

But I agree that the need for a refactoring like this to be a simple,
local thing is important. (Compare my argument for making print a
function -- which by the way received an applause at EuroPython once I
explained it, after getting boohs when the slide first went up.)

> This is a user experience issue, not a technical one.  The fact that you
> can say, "you should've done it differently in the first place" doesn't do
> anything for your flow.  In theory, you could design cars without any
> brakes, because people could just coast to a stop if they planned well
> enough in advance.  ;-)  In practice, you need the brakes because people
> often don't discover their need to stop until much later in the process.

I don't like the brakeless car example either. It is too extreme and
opens your argument up to easy objections (the similarity between the
cases isn't that great anyway).

> This is a flow issue that's specific to *incremental* development.  What
> happens is that first you refactor the code in a function to include nested
> functions.  The variable references don't change, you're just indenting
> some code.  *Then*, it later comes up that you need to rebind a variable,
> and now you have to globally change the variable to make it work as a
> mutable instead, or else you have to refactor all the variables to be
> 'self.' references and put a class in somewhere.
>
> This destroys the flow of incrementally developing whatever it is you're
> developing, and makes you stop to do excise work.  That's why being able to
> rebind a variable without redefining it is important.

Quite apart from the coder's flow, if the code is being developed
incrementally, there are probably readers reviewing it before and
after the refactoring. It's a lot easier to verify that a refactoring
is correct when it doesn't choose a completely different approach.

> In short: in *theory*, a rebinding operator or "nonlocal" declaration is
> unnecessary.  In *practice*, having one seems quite useful every time you
> wander down the path that leads to having to rewrite your code just because
> the language won't let you do that one tiny thing -- or so it feels like to
> the person who's experiencing it.

There really is hardly any need to argue the point any more; I've said
that I will accept the feature if we can find a mutually acceptable
syntax. I still only see these two alternatives as viable: a new
keyword (nonlocal being my current favorite) or stretching the meaning
of global. After Andrew Koenig's Dijkstra reference I'm beginning to
think that the latter may actually be fine.

(Oh, and somebody can add "get rid of global" to PEP 3099 -- that
ain't gonna happen. :-)

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From bob at redivi.com  Fri Jul  7 02:22:57 2006
From: bob at redivi.com (Bob Ippolito)
Date: Thu, 6 Jul 2006 17:22:57 -0700
Subject: [Python-Dev] Musings on concurrency and scoping ("replacing"
	Javascript)
In-Reply-To: <Pine.LNX.4.58.0607061900020.22834@server1.LFW.org>
References: <bbaeab100606211733x3dd58226g15e2919e8e2cdd7e@mail.gmail.com>
	<bbaeab100606211733x3dd58226g15e2919e8e2cdd7e@mail.gmail.com>
	<5.1.1.6.0.20060706185558.01f1fe40@sparrow.telecommunity.com>
	<Pine.LNX.4.58.0607061900020.22834@server1.LFW.org>
Message-ID: <8C44EB52-F933-4169-A7F4-45544DE68C83@redivi.com>


On Jul 6, 2006, at 5:04 PM, Ka-Ping Yee wrote:

> On Thu, 6 Jul 2006, Phillip J. Eby wrote:
>> As much as I'd love to have the nested scope feature, I think it's  
>> only
>> right to point out that the above can be rewritten as something  
>> like this
>> in Python 2.5:
>>
>>      def spam():
>>          local_A = do_work()
>>          result_1 = yield do_network_transaction()
>>          local_B = do_work(result_1)
>>          result_2 = yield do_network_transaction()
>>          do_work(local_A, local_B, result_1, result_2)
>>          ...
>>
>> All you need is an appropriate trampoline (possibly just a  
>> decorator) that
>> takes the objects yielded by the function, and uses them up to set up
>> callbacks that resume the generator with the returned result.
>
> Clever!  Could you help me understand what goes on in
> do_network_transaction() when you write it this way?  In the
> Firefox/JavaScript world, the network transaction is fired off
> in another thread, and when it's done it posts an event back
> to the JavaScript thread, which triggers the callback.
>
> And what happens if you want to supply more than one continuation?
> In my JavaScript code i'm setting up two continuations per step --
> one for success and one for failure, since with a network you never
> know what might happen.

When you have a failure the yield expression raises an exception  
instead of returning a result.

-bob


From bioinformed at gmail.com  Fri Jul  7 03:56:06 2006
From: bioinformed at gmail.com (Kevin Jacobs <jacobs@bioinformed.com>)
Date: Thu, 6 Jul 2006 21:56:06 -0400
Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
In-Reply-To: <e8k58c$q7u$1@sea.gmane.org>
References: <44AA1FF6.1050501@acm.org> <e8k58c$q7u$1@sea.gmane.org>
Message-ID: <2e1434c10607061856t6ef10e92lb8bec4f879cbfcc8@mail.gmail.com>

On 7/6/06, Evan Simpson <evan at 4-am.com> wrote:
>
> Talin wrote:
> > I propose to create a new type of scoping rule, which I will call
> > "explicit" lexical scoping, that will co-exist with the current
> > "implicit" scoping rule that exists in Python today.
>
> I'd like to toss one more variant into the mix.  If we really need to
> address variables in an intermediate scope, the most explicit way that I
> can think of doing so is to write (using Philip's example):
>
> def counter(num):
>     scope as outer # "outer" is an arbitrary identifier
>     def inc():
>         outer.num += 1
>         return outer.num
>     return inc
>



Why not extend the interface to the locals builtin and add a __getitem__
that returns a proxy to access locals defined in other lexical scopes via
__{get/set/del}attr_:

def counter(num):
    num = 1
    def inc():
        locals[1].num += 1
        return outer.num
    return inc


Where, for CPython, locals[n] gives access to
NamespaceProxy(sys._getframe(n).f_locals).  In addition to having a
relatively pleasing and explicit syntax, this may be a feasible method for
allowing portable introspection into outer scopes without having to export
the whole frame object a la sys._getframe(n).  I strongly suspect that
Jython, IronPython, and PyPy would have little difficulty supporting (and
optimizing) this construct.

Lacking core language support, it is easy to roll an object that does just
what I suggest.  Actual implementation is left to a more motivated reader,
of course.

Just another crazy idea to throw into the pot.

-Kevin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20060706/3e422aae/attachment.html 

From pje at telecommunity.com  Fri Jul  7 03:59:09 2006
From: pje at telecommunity.com (Phillip J. Eby)
Date: Thu, 06 Jul 2006 21:59:09 -0400
Subject: [Python-Dev] Musings on concurrency and scoping ("replacing"
 Javascript)
In-Reply-To: <Pine.LNX.4.58.0607061900020.22834@server1.LFW.org>
References: <5.1.1.6.0.20060706185558.01f1fe40@sparrow.telecommunity.com>
	<bbaeab100606211733x3dd58226g15e2919e8e2cdd7e@mail.gmail.com>
	<bbaeab100606211733x3dd58226g15e2919e8e2cdd7e@mail.gmail.com>
	<5.1.1.6.0.20060706185558.01f1fe40@sparrow.telecommunity.com>
Message-ID: <5.1.1.6.0.20060706215215.01f1f938@sparrow.telecommunity.com>

At 07:04 PM 7/6/2006 -0500, Ka-Ping Yee wrote:
>On Thu, 6 Jul 2006, Phillip J. Eby wrote:
> > As much as I'd love to have the nested scope feature, I think it's only
> > right to point out that the above can be rewritten as something like this
> > in Python 2.5:
> >
> >      def spam():
> >          local_A = do_work()
> >          result_1 = yield do_network_transaction()
> >          local_B = do_work(result_1)
> >          result_2 = yield do_network_transaction()
> >          do_work(local_A, local_B, result_1, result_2)
> >          ...
> >
> > All you need is an appropriate trampoline (possibly just a decorator) that
> > takes the objects yielded by the function, and uses them up to set up
> > callbacks that resume the generator with the returned result.
>
>Clever!  Could you help me understand what goes on in
>do_network_transaction() when you write it this way?  In the
>Firefox/JavaScript world, the network transaction is fired off
>in another thread, and when it's done it posts an event back
>to the JavaScript thread, which triggers the callback.

The only difference here is that the callback is to the generator 
instance's send() method.  How the actual callback machinery works, and 
whether threads are involved are orthogonal to the code itself, making it 
an ideal implementation-independent way to express asynchronous algorithms.


>And what happens if you want to supply more than one continuation?
>In my JavaScript code i'm setting up two continuations per step --
>one for success and one for failure, since with a network you never
>know what might happen.

Errors can be passed back via the generator instance's throw() method.

See also peak.events (which could be considered a prototype implementation 
of PEP 342 and associated trampoline features) and Twisted's Deferred 
machinery (which allows sending either a result or a "failure" (where 
failures can be converted back into thrown exceptions).


From talin at acm.org  Fri Jul  7 08:12:31 2006
From: talin at acm.org (Talin)
Date: Thu, 06 Jul 2006 23:12:31 -0700
Subject: [Python-Dev] In defense of Capabilities [was: doc for new
 restricted execution design for Python]
In-Reply-To: <bbaeab100607060959w6f018b3dwe893d7c44f1db2f5@mail.gmail.com>
References: <20060705155149.eipxm6j6oycgkcs8@login.werra.lunarpages.com>	
	<bbaeab100607051626g6aef5ebfle46e7fdd9b71d00f@mail.gmail.com>	
	<44AC9BE3.4000301@acm.org>
	<bbaeab100607060959w6f018b3dwe893d7c44f1db2f5@mail.gmail.com>
Message-ID: <44ADFB4F.5070207@acm.org>

Brett Cannon wrote:
> On 7/5/06, Talin <talin at acm.org> wrote:
>> Transitioning from the checked to the unchecked state could only be done
>> via C code. So the 'file' wrapper, for example, would switch over to the
>> unchecked interpreter before calling the actual methods of 'file'. That
>> C wrapper might also check the current permission state to see what
>> operations were legal.
> 
> So add the proper checks in Python/ceval.c:call_function() to check for 
> this
> flag on every object passed in that is called?

Right. I also realized that you would need to add code that propagates 
the checked bit from the class to any instances of the class. So 
whenever you call a class to create an object, if the class has the 
checked bit, the instance will have it set as well.

>> So essentially, what I propose is to define a simple security primitive
>> - which essentially comes down to checking a single bit - and use that
>> as a basis to create more complex and subtle security mechanisms.
> 
> Right, but it does require that the proper verification function be turned
> on so that the permission bit on 'file' is checked.  It kind of seems like
> 'rexec' and its f_restricted flag it set on execution frames, except you 
> are
> adding an object-level flag as well.
> 
> Either way, the trick is not fouling up switching between the two checking
> functions.
> 
> -Brett

I wasn't aware of how rexec worked, but that seems correct to me.

Given a 'restricted' flag on a stack frame, and one on the object as 
well, then the code for checking for permission violations is nothing 
more than:

    if (object.restricted && exec_frame.restricted)
        raise SecurityException

In particular, there's no need to call a function to check a "permission 
level" or "access rights" or anything of the sort - all that stuff is 
implemented at a higher level.

By making the check very simple, it can also be made very fast. And by 
making it fast, we can afford to call it a lot - for every operation in 
fact.

And if we can call it for every operation, then we don't have to spend 
time hunting down all of the possible loopholes and ways in which 'file' 
or other restricted objects might be accessed.

Originally I had thought to simply add a check like the above into the 
interpreter. However, that would mean that *all* code, whether 
restricted or not, would have to pay the (slight) performance penalty of 
checking that flag. So instead, I thought it might be more efficient to 
have two different code paths, one with the check and one without. But 
all this is based on profound ignorance of the interpreter - there might 
be a hundred other, better ways to do this without having to create two 
versions of ceval.

Another interesting think about the check bit idea is that you can set 
it on any kind of object. For example, you could set it on individual 
methods of a class rather than the class as a whole. However, that's 
probably needlessly elaborate, since fine-grained access control will be 
much more elegantly achieved via trusted wrappers.

-- Talin

From nnorwitz at gmail.com  Fri Jul  7 08:12:47 2006
From: nnorwitz at gmail.com (Neal Norwitz)
Date: Thu, 6 Jul 2006 23:12:47 -0700
Subject: [Python-Dev] 2.5 and beyond
In-Reply-To: <20060630124911.29014.360583188.divmod.quotient.16329@ohm>
References: <ee2a432c0606300005g256b3391na6e684123d3e9e93@mail.gmail.com>
	<20060630124911.29014.360583188.divmod.quotient.16329@ohm>
Message-ID: <ee2a432c0607062312w5d95e2bdt528391f86fecb7b0@mail.gmail.com>

On 6/30/06, Jean-Paul Calderone <exarkun at divmod.com> wrote:
> >
> >There are at least 6 bugs that really, really need to be fixed before
> >release.  Several of these are AST bugs.  Jeremy knows about them and
> >plans to fix them once he's back from vacation.  Anyone else wanna
> >help out?  One is for a socket problem and another is for doc.  The
> >current list of serious bugs are in the PEP:
> >
> >  http://www.python.org/dev/peps/pep-0356/
> >
>
> Please add #1494314 to the list.
>
> http://sourceforge.net/tracker/index.php?func=detail&aid=1494314&group_id=5470&atid=105470

I updated the patch attached to the bug report.  Can people test (or
review) it and provide comments to the report?

Thanks,
n

From rasky at develer.com  Fri Jul  7 09:35:42 2006
From: rasky at develer.com (Giovanni Bajo)
Date: Fri, 7 Jul 2006 09:35:42 +0200
Subject: [Python-Dev] Switch and static, redux
References: <ca471dc20607050354x4042c7d9o9fc41b78e46ba4af@mail.gmail.com>
Message-ID: <018501c6a197$f3bdbe70$d3402597@bagio>

Guido van Rossum <guido at python.org> wrote:

> So, my proposal is to give up on static, accept PEP 3103 with the
> following options:
>   - Syntax alternative 2+B (unindented cases, 'case in ...' for
> multiple cases).
>   - Semantics option 3 (def-time freezing)


I know it's only a bikeshed issue here, but wouldn't it be the first case where
a statement ending with ":" does not introduce an indented suite? Is it really
worth to create this non-orthogonality in the language?

IMHO, if we went for indentend cases, we could teach editors to indent cases
*only* 1 or 2 spaces. That would preserve orthogonality of the language, and
allow not to consume too much horizontal space.

Or, what about optionally indented cases? That is, allow both forms as correct
syntax. It would make it just a matter of style at that point (and Python will
finally have its first religious wars over indentation.... AT LAST! :)

Giovanni Bajo


From greg.ewing at canterbury.ac.nz  Fri Jul  7 09:48:52 2006
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Fri, 07 Jul 2006 19:48:52 +1200
Subject: [Python-Dev] doc for new restricted execution design for Python
In-Reply-To: <bbaeab100607061027p7d742287n441b153370fef2e3@mail.gmail.com>
References: <20060705154750.xzn3ft0hq2gw0koc@login.werra.lunarpages.com>
	<bbaeab100607051611pf70f63drd1c4a52d30d8f544@mail.gmail.com>
	<44AC60D0.1040508@canterbury.ac.nz>
	<bbaeab100607061027p7d742287n441b153370fef2e3@mail.gmail.com>
Message-ID: <44AE11E4.9040809@canterbury.ac.nz>

Brett Cannon wrote:
> On 7/5/06, *Greg Ewing* <greg.ewing at canterbury.ac.nz 

>     And I would change file() so that it didn't open
>     files. Then it would be harmless for code to have
>     access to the file class.

> Right, that is essentially what I proposed initially with the whole 
> crippling idea.
> 
> What the capabilities supporters are saying is that if we go that route 
> we will be constantly finding objects that require similar crippling.

We've got our wires crossed somewhere. I *am* a capabilities
supporter. But for a capability model to work, the capabilities
need to be isolated and encapsulated in functions or objects
that can be independently provided or not provided. In the
case of file(), that means separating the capability of
being able to open a file from the capability of accessing
an already-opened file. These two things are currently
conflated in the file class.

BTW, I object to the term "crippling" in this particular
case. If you have access to open(), there's no need for
file() to be able to do the same thing. And if you don't
have access to open(), it's because someone doesn't want
you to be able to open files.

--
Greg

From greg.ewing at canterbury.ac.nz  Fri Jul  7 09:50:46 2006
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Fri, 07 Jul 2006 19:50:46 +1200
Subject: [Python-Dev] doc for new restricted execution design for Python
In-Reply-To: <bbaeab100607061027p7d742287n441b153370fef2e3@mail.gmail.com>
References: <20060705154750.xzn3ft0hq2gw0koc@login.werra.lunarpages.com>
	<bbaeab100607051611pf70f63drd1c4a52d30d8f544@mail.gmail.com>
	<44AC60D0.1040508@canterbury.ac.nz>
	<bbaeab100607061027p7d742287n441b153370fef2e3@mail.gmail.com>
Message-ID: <44AE1256.1000509@canterbury.ac.nz>

Another thing I perhaps should point out is that
I'm proposing the separation of open() and file()
for *all* code, not just restricted code. So it's
not a matter of "crippling" file() specially for
restricted code.

--
Greg

From just at letterror.com  Fri Jul  7 12:09:26 2006
From: just at letterror.com (Just van Rossum)
Date: Fri,  7 Jul 2006 12:09:26 +0200
Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
In-Reply-To: <e8k58c$q7u$1@sea.gmane.org>
Message-ID: <r01050400-1039-AEFC2D140DA011DBB8DF001124365170@[10.0.0.24]>

Evan Simpson wrote:

> I'd like to toss one more variant into the mix.  If we really need to
> address variables in an intermediate scope, the most explicit way
> that I can think of doing so is to write (using Philip's example):
> 
> def counter(num):
>     scope as outer # "outer" is an arbitrary identifier
>     def inc():
>         outer.num += 1
>         return outer.num
>     return inc
> 
> This is somewhat similar to the unworkable "use the function name"
> method that's been suggested.  The "scope as X" statement associates
> the name "X" with the namespace of local variables in the scope in
> which it is executed.  Such names are "lexically scoped", and only
> allow access to or rebinding of existing names from the originating
> scope (i.e. no "del outer.num" allowed).

Why couldn't at least augmented assignment be implicitly rebinding? It
has been suggested before (in the context of a rebinding operator), but
I'm wondering, is this also off the table?

    def counter(num):
        def inc():
            num += 1
            return num
        return inc

Reads very natural to me. It's likely the most frequent example of what
people try before they learn that rebinding to outer scopes isn't
allowed. It could Just Work.

Just

From ncoghlan at gmail.com  Fri Jul  7 12:29:21 2006
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 07 Jul 2006 20:29:21 +1000
Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
In-Reply-To: <17581.9926.915438.200819@montanaro.dyndns.org>
References: <mailman.29417.1152163446.27774.python-dev@python.org>	<20060706165539.sl1zeof09w0sks0o@webmail.nl.demon.net>
	<17581.9926.915438.200819@montanaro.dyndns.org>
Message-ID: <44AE3781.7080906@gmail.com>

skip at pobox.com wrote:
>     jan-python> So.. are we only thinking about implementing this outer
>     jan-python> scope assignment because there's lots of talk about it on
>     jan-python> the list, ...
> 
> :-)
> 
>     jan-python> ... or are there actually use cases that would become
>     jan-python> clearer if assigning to an outer scope variable was allowed?
> 
> I think full lexical scoping will only be of use to people who use nested
> scopes heavily.  The more typical user will be happy to just refer to values
> in outser scopes without modifying them and rely on classes to save changed
> state across calls.  I think it's almost a YAGNI, but I'm sure others will
> disagree.

I think it falls into the same category as Guido's ultimate acceptance of PEP 
308. There are assorted ways to live *without* conditional expressions, but 
each of the workarounds for its absence had issues. Switching to a statement 
worked properly, but meant you didn't have a single expression any more. Use 
the and-or trick kept the single expression characteristic, but was easy to 
get wrong. Hence, PEP 308: One Obvious Way to do it, assuming you want to do 
it in the first place.

I think writing to outer scopes is similar. You can box the variable, or make 
it an attribute of an object, but either approach requires you to refactor 
*all* uses of the variable, rather than just the one you currently care about. 
Hence, 'nonlocal': One Obvious Way to do it, assuming you want to do it in the 
first place.

That way, when you're *reading* the code of someone who likes to use such 
tricks, you only need to know how to read the one obvious way, rather than 
having to decipher whichever method they've chosen to work around the limitation.

The perennial accumulator example still takes 6 lines, though:

def accumulator(n):
     def increment(i):
         nonlocal n
         n += i
         return n
     return increment

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
             http://www.boredomandlaziness.org

From fredrik at pythonware.com  Fri Jul  7 12:38:43 2006
From: fredrik at pythonware.com (Fredrik Lundh)
Date: Fri, 07 Jul 2006 12:38:43 +0200
Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
In-Reply-To: <r01050400-1039-AEFC2D140DA011DBB8DF001124365170@[10.0.0.24]>
References: <e8k58c$q7u$1@sea.gmane.org>
	<r01050400-1039-AEFC2D140DA011DBB8DF001124365170@[10.0.0.24]>
Message-ID: <e8ldjg$eq7$1@sea.gmane.org>

Just van Rossum wrote:

> Why couldn't at least augmented assignment be implicitly rebinding? It
> has been suggested before (in the context of a rebinding operator), but
> I'm wondering, is this also off the table?
> 
>     def counter(num):
>         def inc():
>             num += 1
>             return num
>         return inc
> 
> Reads very natural to me. It's likely the most frequent example of what
> people try before they learn that rebinding to outer scopes isn't
> allowed. It could Just Work.

note that most examples of this type already work, if the target type is 
mutable, and implement the right operations:

      def counter(num):
          num = mutable_int(num)
          def inc():
              num += 1
              return num
          return inc

maybe we should consider adding mutable strings and mutable numbers to 
Python 3.0 ?  and a "mutable" built-in, that does the opposite of the 
"freeze" stuff:

      def counter(num):
          num = mutable(num)
          def inc():
              num += 1
              return num
          return inc

(what is this thread doing on python-dev, btw?  shouldn't it be over at 
the 3000 list, so I can enjoy my vacation without being drawn into yet 
another endless discussion thread ;-)

</F>


From ncoghlan at gmail.com  Fri Jul  7 13:45:04 2006
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 07 Jul 2006 21:45:04 +1000
Subject: [Python-Dev] doc for new restricted execution design for Python
In-Reply-To: <44AE1256.1000509@canterbury.ac.nz>
References: <20060705154750.xzn3ft0hq2gw0koc@login.werra.lunarpages.com>	<bbaeab100607051611pf70f63drd1c4a52d30d8f544@mail.gmail.com>	<44AC60D0.1040508@canterbury.ac.nz>	<bbaeab100607061027p7d742287n441b153370fef2e3@mail.gmail.com>
	<44AE1256.1000509@canterbury.ac.nz>
Message-ID: <44AE4940.5020603@gmail.com>

Greg Ewing wrote:
> Another thing I perhaps should point out is that
> I'm proposing the separation of open() and file()
> for *all* code, not just restricted code. So it's
> not a matter of "crippling" file() specially for
> restricted code.

What would the signature of the file constructor be in that case? Would it 
accept a single CObject instance, with open() bypassing the normal 
constructor, and handing the file pointer directly to the file object instance?

It seems like a reasonable approach for making 'dangerous' objects like file 
and socket much easier to secure - have a separate factory function to create 
*new* instances at the C level, so that you can hand the objects over without 
worrying about providing access to the constructor (because the constructor 
*accepts* the OS-level object as an argument, rather than creating it anew).

Alternatively, using a "no-Python-level-introspection" metaclass might be 
another way to achieve the same effect in a more universal fashion.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
             http://www.boredomandlaziness.org

From ncoghlan at gmail.com  Fri Jul  7 13:55:45 2006
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 07 Jul 2006 21:55:45 +1000
Subject: [Python-Dev] introducing __dir__?
In-Reply-To: <200607061341.56945.fdrake@acm.org>
References: <1d85506f0607061022v76806e5dt6b59f5771ef3f1e3@mail.gmail.com>
	<200607061341.56945.fdrake@acm.org>
Message-ID: <44AE4BC1.1000203@gmail.com>

Fred L. Drake, Jr. wrote:
> On Thursday 06 July 2006 13:22, tomer filiba wrote:
>  > my suggestion is simple -- replace this mechanism with a __dir__ -
>  > a special method that returns the list of attributes of the object.
>  >
>  > rationale:
>  > * remove deprecated __methods__, etc.
>  > * symmetry -- just like hex() calls __hex__, etc.
>  > * __methods__ and __members__ are lists rather than callable
>  > objects, which means they cannot be updated on-demand
> 
> +1

+1 here, too.

It would also allow objects which override __getattribute__ and/or __getattr__ 
to make dir() provide a sane answer (or raise an exception to indicate that a 
sane answer isn't possible). (This was something that actually came up when 
trying to implement a namespace object that *didn't* automatically fall back 
to its class namespace for Python level attribute access)

For backwards compatibility, dir() could still fall back to the current 
mechanism if __dir__ isn't found.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
             http://www.boredomandlaziness.org

From martin at v.loewis.de  Fri Jul  7 17:28:23 2006
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Fri, 07 Jul 2006 17:28:23 +0200
Subject: [Python-Dev] SVN write access is back
Message-ID: <44AE7D97.6070201@v.loewis.de>

I just turned the subversion write access back on.

Unfortunately, I did not manage to perform the changes
I wanted (import ctypes), so I'll have to retry later
when the open issues have been clarified.

Regards,
Martin

From theller at python.net  Fri Jul  7 18:19:58 2006
From: theller at python.net (Thomas Heller)
Date: Fri, 07 Jul 2006 18:19:58 +0200
Subject: [Python-Dev] test_ctypes failure on Mac OS X/PowerPC 10.3.9
	(Panther)
In-Reply-To: <3B96F3FF-B847-4E57-ACC2-F7D979DCA5BA@mac.com>
References: <44982ADE.5070404@activestate.com>
	<4498393E.1020101@python.net>	<71C6D0BF-B569-48BD-9F9D-05D3660FF2AA@mac.com>
	<3B96F3FF-B847-4E57-ACC2-F7D979DCA5BA@mac.com>
Message-ID: <44AE89AE.1040105@python.net>

Ronald Oussoren schrieb:
> On 20-jun-2006, at 20:50, Ronald Oussoren wrote:
> 
>>
>> On 20-jun-2006, at 20:06, Thomas Heller wrote:
>>
>>> Trent Mick schrieb:
>>>> Thomas and others,
>>>>
>>>> Has anyone else seen failures in test_ctypes on older Mac OS X/
>>>> PowerPC?
>>>> Results are below. This is running a build of the trunk from last
>>>> night:
>>>>
>>>> 	./configure && make && ./python.exe Lib/test/test_ctypes.py
>>>>
>>>> Note that the test does NOT fail on the Mac OS X/x86 10.4.6 box
>>>> that I have.
>>>
>>> It also works on 10.4.?? Power PC.  I guess the fix has to wait until
>>> I'm able to install 10.3 on my mac, I have the DVDs already but
>>> have not
>>> yet had the time.  If anyone is willing to give me ssh access to a
>>> 10.3
>>> box I can try to fix this earlier.
>>
>> I had some problems with my 10.3-capable box, but happily enough it
>> decided to come alive again. I'm currently booted into 10.3.9 and
>> will have a look.
> 
> It is a platform bug, RTLD_LOCAL doesn't work on 10.3. The following  
> C snippet fails with the same error as ctypes: FAIL: dlcompat: unable  
> to open this file with RTLD_LOCAL. This seems to be confirmed by this  
> sourcet test file from darwin: http://darwinsource.opendarwin.org/ 
> 10.4.1/dyld-43/unit-tests/test-cases/dlopen-RTLD_LOCAL/main.c.

I think that this problem still exists.  From what I have found out
so far (still I don't have access to osx 10.3 myself), it seems that
there are three possibilities:

- ignore this issue ;-)

- make RTLD_GLOBAL the default mode on osx: either on 10.3 only,
  or on all versions (how can I determine 10.3 from 10.4?)

- change the *test* only so that RTLD_GLOBAL is used on osx.


I do not know how other ctypes users on osx are affected by this problem,
however

  http://codespeak.net/pypy/dist/pypy/doc/rctypes.html#ctypes-version-and-platform-notes

recommends to patch ctypes/__init__.py on osx 10.3.

Thanks,
Thomas


From martin at v.loewis.de  Fri Jul  7 18:30:08 2006
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Fri, 07 Jul 2006 18:30:08 +0200
Subject: [Python-Dev] Extended Subversion outage: Friday 16:40 GMT
In-Reply-To: <44AD69FB.7040002@v.loewis.de>
References: <44AD69FB.7040002@v.loewis.de>
Message-ID: <44AE8C10.2040008@v.loewis.de>

Martin v. L?wis wrote:
> I plan to do some subversion administration
> tomorrow; in order to be able to roll back changes,
> I have to disable write access during these
> changes.

I'm going to make a second attempt ten minutes from
now.

Regards,
Martin

From martin at v.loewis.de  Fri Jul  7 18:54:12 2006
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Fri, 07 Jul 2006 18:54:12 +0200
Subject: [Python-Dev] Extended Subversion outage: Friday 16:40 GMT
In-Reply-To: <44AE8C10.2040008@v.loewis.de>
References: <44AD69FB.7040002@v.loewis.de> <44AE8C10.2040008@v.loewis.de>
Message-ID: <44AE91B4.9070903@v.loewis.de>

Martin v. L?wis wrote:
> Martin v. L?wis wrote:
>> I plan to do some subversion administration
>> tomorrow; in order to be able to roll back changes,
>> I have to disable write access during these
>> changes.
> 
> I'm going to make a second attempt ten minutes from
> now.

I completed importing the ctypes history, so the
repository is now back online.

Regards,
Martin

From brett at python.org  Fri Jul  7 18:56:33 2006
From: brett at python.org (Brett Cannon)
Date: Fri, 7 Jul 2006 09:56:33 -0700
Subject: [Python-Dev] In defense of Capabilities [was: doc for new
	restricted execution design for Python]
In-Reply-To: <44ADFB4F.5070207@acm.org>
References: <20060705155149.eipxm6j6oycgkcs8@login.werra.lunarpages.com>
	<bbaeab100607051626g6aef5ebfle46e7fdd9b71d00f@mail.gmail.com>
	<44AC9BE3.4000301@acm.org>
	<bbaeab100607060959w6f018b3dwe893d7c44f1db2f5@mail.gmail.com>
	<44ADFB4F.5070207@acm.org>
Message-ID: <bbaeab100607070956p54260951k74e4ce0871f1606b@mail.gmail.com>

On 7/6/06, Talin <talin at acm.org> wrote:
>
> Brett Cannon wrote:
> > On 7/5/06, Talin <talin at acm.org> wrote:
> >> Transitioning from the checked to the unchecked state could only be
> done
> >> via C code. So the 'file' wrapper, for example, would switch over to
> the
> >> unchecked interpreter before calling the actual methods of 'file'. That
> >> C wrapper might also check the current permission state to see what
> >> operations were legal.
> >
> > So add the proper checks in Python/ceval.c:call_function() to check for
> > this
> > flag on every object passed in that is called?
>
> Right. I also realized that you would need to add code that propagates
> the checked bit from the class to any instances of the class. So
> whenever you call a class to create an object, if the class has the
> checked bit, the instance will have it set as well.
>
> >> So essentially, what I propose is to define a simple security primitive
> >> - which essentially comes down to checking a single bit - and use that
> >> as a basis to create more complex and subtle security mechanisms.
> >
> > Right, but it does require that the proper verification function be
> turned
> > on so that the permission bit on 'file' is checked.  It kind of seems
> like
> > 'rexec' and its f_restricted flag it set on execution frames, except you
> > are
> > adding an object-level flag as well.
> >
> > Either way, the trick is not fouling up switching between the two
> checking
> > functions.
> >
> > -Brett
>
> I wasn't aware of how rexec worked, but that seems correct to me.
>
> Given a 'restricted' flag on a stack frame, and one on the object as
> well, then the code for checking for permission violations is nothing
> more than:
>
>     if (object.restricted && exec_frame.restricted)
>         raise SecurityException
>
> In particular, there's no need to call a function to check a "permission
> level" or "access rights" or anything of the sort - all that stuff is
> implemented at a higher level.
>
> By making the check very simple, it can also be made very fast. And by
> making it fast, we can afford to call it a lot - for every operation in
> fact.
>
> And if we can call it for every operation, then we don't have to spend
> time hunting down all of the possible loopholes and ways in which 'file'
> or other restricted objects might be accessed.


Not true.  You have to set this object restriction flag, right?  What
happens if you don't set it on all of the proper classes/types?  You end up
in the exact same situation you are with crippling; making sure you cover
your ass with what you flag as  unsafe else you risk having something get
passed you.

Originally I had thought to simply add a check like the above into the
> interpreter. However, that would mean that *all* code, whether
> restricted or not, would have to pay the (slight) performance penalty of
> checking that flag. So instead, I thought it might be more efficient to
> have two different code paths, one with the check and one without. But
> all this is based on profound ignorance of the interpreter - there might
> be a hundred other, better ways to do this without having to create two
> versions of ceval.


Yeah, keep it simple, especially when it comes to ceval.c .

Another interesting think about the check bit idea is that you can set
> it on any kind of object. For example, you could set it on individual
> methods of a class rather than the class as a whole. However, that's
> probably needlessly elaborate, since fine-grained access control will be
> much more elegantly achieved via trusted wrappers.


Yeah, that seems a bit extreme.

-Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20060707/caeaa100/attachment.html 

From evan at 4-am.com  Fri Jul  7 18:56:01 2006
From: evan at 4-am.com (Evan Simpson)
Date: Fri, 07 Jul 2006 11:56:01 -0500
Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
In-Reply-To: <2e1434c10607061856t6ef10e92lb8bec4f879cbfcc8@mail.gmail.com>
References: <44AA1FF6.1050501@acm.org> <e8k58c$q7u$1@sea.gmane.org>
	<2e1434c10607061856t6ef10e92lb8bec4f879cbfcc8@mail.gmail.com>
Message-ID: <44AE9221.9020704@4-am.com>

Kevin Jacobs <jacobs at bioinformed.com> wrote:
> Why not extend the interface to the locals builtin and add a __getitem__
> that returns a proxy to access locals defined in other lexical scopes
> via __{get/set/del}attr_:
> 
> def counter(num):
>     num = 1
>     def inc():
>         locals[1].num += 1
>         return outer.num
>     return inc
> 
> Where, for CPython, locals[n] gives access to
> NamespaceProxy(sys._getframe(n).f_locals).

Two nits:  First, I suspect that you meant to write "return
locals[1].num".  Second, sys._getframe doesn't do what you want, here.
It reaches back up the call chain, not out into lexically containing scopes.

That said, I like the idea of giving "locals[]" the meaning you
intended.  It has the advantage of not adding any new keywords or
syntactic constructs, but the disadvantage of not explicitly signaling
that locals in a given scope will be twiddled elsewhere.  Also, for
efficiency's sake, it might be desirable to only allow a literal integer
as the index, and to deal with use of "locals[]" at compile time rather
than dynamically.  I'm not sure how much overhead would be involved in
enabling dynamic lookup of arbitrary lexically containing scopes, but I
would *not* want to enable stuff like "locals[i * 2 - 1]".

Cheers,

Evan @ 4-am


From theller at python.net  Fri Jul  7 18:57:40 2006
From: theller at python.net (Thomas Heller)
Date: Fri, 07 Jul 2006 18:57:40 +0200
Subject: [Python-Dev] Extended Subversion outage: Friday 16:40 GMT
In-Reply-To: <44AE91B4.9070903@v.loewis.de>
References: <44AD69FB.7040002@v.loewis.de> <44AE8C10.2040008@v.loewis.de>
	<44AE91B4.9070903@v.loewis.de>
Message-ID: <e8m3q4$4cu$1@sea.gmane.org>

Martin v. L?wis schrieb:
> Martin v. L?wis wrote:
>> Martin v. L?wis wrote:
>>> I plan to do some subversion administration
>>> tomorrow; in order to be able to roll back changes,
>>> I have to disable write access during these
>>> changes.
>> 
>> I'm going to make a second attempt ten minutes from
>> now.
> 
> I completed importing the ctypes history, so the
> repository is now back online.

Thanks, Martin.

Thomas


From bborcic at gmail.com  Fri Jul  7 19:02:52 2006
From: bborcic at gmail.com (Boris Borcic)
Date: Fri, 07 Jul 2006 19:02:52 +0200
Subject: [Python-Dev] [slighly OT] Native speakers and hurting brains
In-Reply-To: <ca471dc20607051027y677c7127wfe098401c366dc74@mail.gmail.com>
References: <20060705101821.7sqnxo7lu5cgg4ks@login.werra.lunarpages.com>
	<ca471dc20607051027y677c7127wfe098401c366dc74@mail.gmail.com>
Message-ID: <e8m44k$5qb$1@sea.gmane.org>

Guido van Rossum wrote:
> On 7/5/06, Michael Chermside <mcherm at mcherm.com> wrote:
>> Guido writes:
>>    [discussion of how to fix the can't-bind-outer-scope-vars wart]
...
> 
> Are there any other native speakers who side with Michael?
> 

A bit OT, but why should native speakers (eg of English) have special authority 
? I mean this neither as an attack on native speakers nor as a rethorical question.

I recently re-read the "sum()" discussion to figure out how it came about that 
sum() half bails out on sequences_of_sequences (by requiring me to specify the 
start value) except when the nested sequences are strings, in which cases it 90% 
bails out : by first asking for an adequate start value, and *then* - once I've 
provided it - telling me it understands my intent but that I should use ''.join 
instead. Is this a programming language shell or a text-based adventure game ? 
<0.8 wink>

AFAICT, the cause of this funny strangeness is that a very respected member of 
pydev and native speaker of English with an truly unique sense of humor, found 
early in the discussion that sum(sequence_of_strings) "made his brain hurt"; and 
while this person is the very opposite of a newbie, his judgment was combined 
with concerns relating to newbies to result in the current behavior.

I believe that in this case native linguistic intuition made the decision... and 
that a non-native intuition should have reversed it by noting in due time that 
the cognitive dissonance that native speakers could find to sum(seq_of_str), 
isn't in fact distinct from the cognitive dissonance that any newbie should find 
to the use of the + operator to concatenate strings or sequences rather than 
summing numbers.

To restate, as a non-native speaker, and inasmuch "it hurts the brain" is the 
real reason against sum(seq_of_str) calling ''.join(seq_of_str) directly, I 
believe this is a case of excess authority being given to native linguistic 
intuition (of english speaking python experts) with the result of /adding/ to 
the cognitive dissonance of using + to concatenate strings (for all programming 
newbies).

Best regards to all,

Boris Borcic
--
"On na?t tous les m?tres du m?me monde"








From brett at python.org  Fri Jul  7 19:03:40 2006
From: brett at python.org (Brett Cannon)
Date: Fri, 7 Jul 2006 10:03:40 -0700
Subject: [Python-Dev] doc for new restricted execution design for Python
In-Reply-To: <44AE11E4.9040809@canterbury.ac.nz>
References: <20060705154750.xzn3ft0hq2gw0koc@login.werra.lunarpages.com>
	<bbaeab100607051611pf70f63drd1c4a52d30d8f544@mail.gmail.com>
	<44AC60D0.1040508@canterbury.ac.nz>
	<bbaeab100607061027p7d742287n441b153370fef2e3@mail.gmail.com>
	<44AE11E4.9040809@canterbury.ac.nz>
Message-ID: <bbaeab100607071003l666c8e46j177a1b50f0b7bfcc@mail.gmail.com>

On 7/7/06, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
>
> Brett Cannon wrote:
> > On 7/5/06, *Greg Ewing* <greg.ewing at canterbury.ac.nz
>
> >     And I would change file() so that it didn't open
> >     files. Then it would be harmless for code to have
> >     access to the file class.
>
> > Right, that is essentially what I proposed initially with the whole
> > crippling idea.
> >
> > What the capabilities supporters are saying is that if we go that route
> > we will be constantly finding objects that require similar crippling.
>
> We've got our wires crossed somewhere. I *am* a capabilities
> supporter. But for a capability model to work, the capabilities
> need to be isolated and encapsulated in functions or objects
> that can be independently provided or not provided. In the
> case of file(), that means separating the capability of
> being able to open a file from the capability of accessing
> an already-opened file. These two things are currently
> conflated in the file class.


OK, putting that way makes sense for labelling it as capabilities support.
It just seems like everyone else who has been calling for more capabilities
have not been wanting to change file() in any way.

BTW, I object to the term "crippling" in this particular
> case. If you have access to open(), there's no need for
> file() to be able to do the same thing. And if you don't
> have access to open(), it's because someone doesn't want
> you to be able to open files.


 But, from the perspective of what file() can do now, it is losing an
ability that it once had and not quite as powerful as it was.


-Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20060707/5f5280f6/attachment.htm 

From brett at python.org  Fri Jul  7 19:04:55 2006
From: brett at python.org (Brett Cannon)
Date: Fri, 7 Jul 2006 10:04:55 -0700
Subject: [Python-Dev] doc for new restricted execution design for Python
In-Reply-To: <44AE1256.1000509@canterbury.ac.nz>
References: <20060705154750.xzn3ft0hq2gw0koc@login.werra.lunarpages.com>
	<bbaeab100607051611pf70f63drd1c4a52d30d8f544@mail.gmail.com>
	<44AC60D0.1040508@canterbury.ac.nz>
	<bbaeab100607061027p7d742287n441b153370fef2e3@mail.gmail.com>
	<44AE1256.1000509@canterbury.ac.nz>
Message-ID: <bbaeab100607071004g7f41efb0t9f3811e9ac73126e@mail.gmail.com>

On 7/7/06, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
>
> Another thing I perhaps should point out is that
> I'm proposing the separation of open() and file()
> for *all* code, not just restricted code. So it's
> not a matter of "crippling" file() specially for
> restricted code.


Well, that's fine with me since I use open() for opening all of my files
anyway.  But I don't know if people want to make 'file' the only built-in
type who lacks a constructor on the type itself.

-Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20060707/50760867/attachment.html 

From pje at telecommunity.com  Fri Jul  7 19:25:43 2006
From: pje at telecommunity.com (Phillip J. Eby)
Date: Fri, 07 Jul 2006 13:25:43 -0400
Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
In-Reply-To: <2e1434c10607061856t6ef10e92lb8bec4f879cbfcc8@mail.gmail.co
 m>
References: <e8k58c$q7u$1@sea.gmane.org> <44AA1FF6.1050501@acm.org>
	<e8k58c$q7u$1@sea.gmane.org>
Message-ID: <5.1.1.6.0.20060706235548.01f22388@sparrow.telecommunity.com>

At 09:56 PM 7/6/2006 -0400, Kevin Jacobs <jacobs at bioinformed.com> wrote:
>Why not extend the interface to the locals builtin and add a __getitem__ 
>that returns a proxy to access locals defined in other lexical scopes via 
>__{get/set/del}attr_:
>
>def counter(num):
>     num = 1
>     def inc():
>         locals[1].num += 1
>         return outer.num
>     return inc
>
>
>Where, for CPython, locals[n] gives access to 
>NamespaceProxy(sys._getframe(n).f_locals).

That doesn't actually work, because sys._getframe(1) will give you inc()'s 
caller, *not* the frame where it was defined.


>   In addition to having a relatively pleasing and explicit syntax, this 
> may be a feasible method for allowing portable introspection into outer 
> scopes without having to export the whole frame object a la 
> sys._getframe(n).  I strongly suspect that Jython, IronPython, and PyPy 
> would have little difficulty supporting (and optimizing) this construct.
>
>Lacking core language support, it is easy to roll an object that does just 
>what I suggest.  Actual implementation is left to a more motivated reader, 
>of course.

While I hesitate to describe anything as "impossible", I will note that an 
implementation with the syntax you suggest is not likely to be possible 
without resorting to ctypes or a C module, because the frame doesn't have a 
reference to the function, only the code object, and it's the function 
objects that own the closure cells.

I think the only way you can reasonably accomplish rebinding in Python 
right now is to have a rebind(func, var=value) function, e.g.:

     def counter(num):
         def inc():
             rebind(inc, num=num+1)
             return num
         return inc

It doesn't allow augmented assignment, of course, and it also creates a 
circular reference between 'inc' and itself, requiring garbage collection 
to clean up.

Anyway, the actual implementation of such a rebind function would basically 
be a bit of syntax sugar over this cookbook recipe:

http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/440515

You'd want to just use the keyword argument names to figure out which cells 
of the function needed changing.  And the whole thing would be really slow 
and complex.  You'd probably be better off using function attributes:

     def counter(num):
         def inc():
             inc.num+=1
             return inc.num
         inc.num = num
         return inc

It's not ideal, but *this* should probably be the pattern we recommend for 
rebinding, rather than "create a class" or "use an anonymous namespace 
argument".

It can even be prettied up a little bit with a decorator to set the 
function attributes.

     def counter(num):
         @uses(num=num)
         def inc():
             inc.num+=1
             return inc.num
         return inc

But that's about as good as it gets, which is nowhere near as good as:

     def counter(num):
         def inc():
             nonlocal num
             num+=1
             return num
         return inc

On the other hand, the rebind() syntax actually is slightly cleaner in some 
respects:

     def counter(num):
         def inc():
             rebind(num = num+1)
             return num
         return inc

Maybe we should just make rebind() a fast builtin.  ;)


From talin at acm.org  Fri Jul  7 20:17:52 2006
From: talin at acm.org (Talin)
Date: Fri, 07 Jul 2006 11:17:52 -0700
Subject: [Python-Dev] In defense of Capabilities [was: doc for new
 restricted execution design for Python]
In-Reply-To: <bbaeab100607070956p54260951k74e4ce0871f1606b@mail.gmail.com>
References: <20060705155149.eipxm6j6oycgkcs8@login.werra.lunarpages.com>	
	<bbaeab100607051626g6aef5ebfle46e7fdd9b71d00f@mail.gmail.com>	
	<44AC9BE3.4000301@acm.org>	
	<bbaeab100607060959w6f018b3dwe893d7c44f1db2f5@mail.gmail.com>	
	<44ADFB4F.5070207@acm.org>
	<bbaeab100607070956p54260951k74e4ce0871f1606b@mail.gmail.com>
Message-ID: <44AEA550.1030204@acm.org>

Brett Cannon wrote:
> On 7/6/06, Talin <talin at acm.org> wrote:
>> And if we can call it for every operation, then we don't have to spend
>> time hunting down all of the possible loopholes and ways in which 'file'
>> or other restricted objects might be accessed.
> 
> Not true.  You have to set this object restriction flag, right?  What
> happens if you don't set it on all of the proper classes/types?  You end up
> in the exact same situation you are with crippling; making sure you cover
> your ass with what you flag as  unsafe else you risk having something get
> passed you.

But that's a much simpler problem.

With the restricted flag, it isn't just *your* code that is prevented 
from using 'file' - it's *all* code. Only approved gateways that remove 
the restriction (by setting the interpreter state) can perform 
operations on file objects without blowing up.

This means that if you call some random library function that attempts 
to open a file, it won't work, because the random library function is 
still running in restricted mode.

Similarly, if you have a reference to some externally created object 
that has a reference to a file (or the file class) somewhere in it's 
inheritance hierarchy, any attempt to access that object will fail.

Without this, you would have to chase down every bit of library code 
that opens file, or has a reference to a file.

What I am proposing shares some aspects of both the crippling and the 
capability model:

It's similar to crippling in the sense that you're protecting the object 
itself, not access to the object. So you avoid the problem of trying to 
figure out all of the possible ways an object can be accessed.

However, where it resembles capabilities is that its an 'all or nothing' 
approach - that is, you either have access to file, or you don't. Unlike 
the crippling model where fine-grained access control is implemented by 
modifying individual methods of the crippled object, in this scheme we 
cripple the object *entirely*, and then provide fine-grained access 
control via wrappers. Those wrappers, in turn, act just like 
capabilities - you can have different wrappers that have different sets 
of access permissions.

So it provides the advantage of the capability approach in that the set 
of restrictions can be extended or modified by writing new wrappers.

Thus, by providing an extremely simple but unbreakable check at the 
interpreter level, we can then write classes that build on top of that a 
richer and more sophisticated set of permissions, while still 
maintaining a strong barrier to unauthorized actions.

-- Talin

From brett at python.org  Fri Jul  7 21:39:43 2006
From: brett at python.org (Brett Cannon)
Date: Fri, 7 Jul 2006 12:39:43 -0700
Subject: [Python-Dev] In defense of Capabilities [was: doc for new
	restricted execution design for Python]
In-Reply-To: <44AEA550.1030204@acm.org>
References: <20060705155149.eipxm6j6oycgkcs8@login.werra.lunarpages.com>
	<bbaeab100607051626g6aef5ebfle46e7fdd9b71d00f@mail.gmail.com>
	<44AC9BE3.4000301@acm.org>
	<bbaeab100607060959w6f018b3dwe893d7c44f1db2f5@mail.gmail.com>
	<44ADFB4F.5070207@acm.org>
	<bbaeab100607070956p54260951k74e4ce0871f1606b@mail.gmail.com>
	<44AEA550.1030204@acm.org>
Message-ID: <bbaeab100607071239j5b413f4cy991111a87c8de9fe@mail.gmail.com>

On 7/7/06, Talin <talin at acm.org> wrote:
>
> Brett Cannon wrote:
> > On 7/6/06, Talin <talin at acm.org> wrote:
> >> And if we can call it for every operation, then we don't have to spend
> >> time hunting down all of the possible loopholes and ways in which
> 'file'
> >> or other restricted objects might be accessed.
> >
> > Not true.  You have to set this object restriction flag, right?  What
> > happens if you don't set it on all of the proper classes/types?  You end
> up
> > in the exact same situation you are with crippling; making sure you
> cover
> > your ass with what you flag as  unsafe else you risk having something
> get
> > passed you.
>
> But that's a much simpler problem.
>
> With the restricted flag, it isn't just *your* code that is prevented
> from using 'file' - it's *all* code. Only approved gateways that remove
> the restriction (by setting the interpreter state) can perform
> operations on file objects without blowing up.
>
> This means that if you call some random library function that attempts
> to open a file, it won't work, because the random library function is
> still running in restricted mode.


Right, but that happens with either approach being proposed anyway.

Similarly, if you have a reference to some externally created object
> that has a reference to a file (or the file class) somewhere in it's
> inheritance hierarchy, any attempt to access that object will fail.
>
> Without this, you would have to chase down every bit of library code
> that opens file, or has a reference to a file.


Not true.  With the library code running under a sandboxed interpreter the
checks by either implementation will still be in effect.  And if a proxy is
returned by open() instead of 'file' with stuff crippled then the worries
alleviated even further.

What I am proposing shares some aspects of both the crippling and the
> capability model:
>
> It's similar to crippling in the sense that you're protecting the object
> itself, not access to the object. So you avoid the problem of trying to
> figure out all of the possible ways an object can be accessed.
>
> However, where it resembles capabilities is that its an 'all or nothing'
> approach - that is, you either have access to file, or you don't. Unlike
> the crippling model where fine-grained access control is implemented by
> modifying individual methods of the crippled object, in this scheme we
> cripple the object *entirely*, and then provide fine-grained access
> control via wrappers. Those wrappers, in turn, act just like
> capabilities - you can have different wrappers that have different sets
> of access permissions.
>
> So it provides the advantage of the capability approach in that the set
> of restrictions can be extended or modified by writing new wrappers.
>
> Thus, by providing an extremely simple but unbreakable check at the
> interpreter level, we can then write classes that build on top of that a
> richer and more sophisticated set of permissions, while still
> maintaining a strong barrier to unauthorized actions.




I guess I am just not seeing where your approach is better than preventing
the constructor in 'file' and having open() return the 'file' object or
proxy object.  With your approach 'file' would be flagged, but with the
other you just put the same check in 'file's constructor.  With both you
would probably also want open() to be a factory function anyway.  So I don't
see where you gain simplicity or more security.  It seems like you are
pushing the check into the eval loop, but you still require the flagging of
objects as unsafe.  Going with the other two proposals you don't burden the
eval loop with the check but the objects that you would have flagged in the
first place.

It just seems like we are pushing around the flagging of unsafe stuff and
that doesn't feel like it buys us much.

-Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20060707/698a6016/attachment.htm 

From guido at python.org  Fri Jul  7 22:08:52 2006
From: guido at python.org (Guido van Rossum)
Date: Fri, 7 Jul 2006 22:08:52 +0200
Subject: [Python-Dev] Musings on concurrency and scoping ("replacing"
	Javascript)
In-Reply-To: <Pine.LNX.4.58.0607061642420.22834@server1.LFW.org>
References: <bbaeab100606211733x3dd58226g15e2919e8e2cdd7e@mail.gmail.com>
	<Pine.LNX.4.58.0607061642420.22834@server1.LFW.org>
Message-ID: <ca471dc20607071308v2a4e9885r7e0b09551df5711@mail.gmail.com>

On 7/7/06, Ka-Ping Yee <python-dev at zesty.ca> wrote:
> I've been doing a bunch of Firefox extension programming in Javascript
> and suddenly a few of the recent topics here came together in my head
> in a silent kapow of thoughts.  This is kind of a side note to the
> security discussion, but they're all interconnected: network
> programming, concurrency, lexical scoping, security.

Hm... I wonder if this style has become so popular in JS because it's
all they have? I find callback-style programming pretty inscrutable
pretty soon.

> Client-side web scripting tends to have a callback/continuation-ish
> concurrency style because it has to deal with network transactions
> (which can stall for long periods of time) in a user interface that
> is expected to stay always responsive.  The Firefox API is full of
> listeners/observers, events, and continuation-like things.  So one
> thing to consider is that, when Python is used for these purposes,
> it may be written in a specialized style.
>
> As i write JavaScript in this style i find i use nested functions
> a lot.  When i want to set up a callback that uses variables in the
> current context, the natural thing to do is to define a new function
> in the local namespace.  And if that function has to also provide a
> callback, then it has another function nested within it and so on.
>
>     function spam() {
>         var local_A = do_work();
>         do_network_transaction(
>             new function(result_1) {
>                 var local_B = do_work(result_1);
>                 do_network_transaction(
>                     new function(result_2) {
>                         do_work(local_A, local_B, result_1, result_2);
>                         ...
>                     }
>                 );
>             }
>         );
>     }

How can you ever keep track of when a '}' must be followed by a ';' ?

> So it is a possible consequence of embedding Python in Firefox that
> people will be using nested functions and lexical scoping in Python
> more often, which makes the recent discussion about access to
> enclosing scopes more significant.
>
> This is even related to security as well.  Namespaces and lexical
> scoping are a natural and visually apparent way to limit access.
> If, for example, result_1 and result_2 in the above example are
> security-sensitive objects like secret keys or networking functions,
> you can see just by inspection that they cannot leak outside of
> spam() except by directly being passed in an outgoing function call.
>
> The standard Pythonic response to nested functions is to translate
> them into classes.  But the nested function style has two advantages:
>
>     1.  Variables are more appropriately scoped; they exist
>         only where they are meaningful.  (In a class, all the
>         variables would be mixed into one namespace, where
>         some of them would be invalid some of the time.)

This doesn't strike me as very important. The same can happen in a
local scope and doesn 't seem to bother anyone there.

>     2.  Local variables are private.  (Class instances don't
>         get their own private namespaces to play in.)

Hmm... I wouldn't be so sure of that. Exception handlers have access
to locals of stack frames above and below the current frame.

> The first becomes more significant when in a more continuation-y
> style because it helps keep the various continuations from
> interfering with each other.  The second becomes more significant
> if you care about restricting untrusted Python code.

I would rather come up with a more Pythonic style of asynchronous
event handling, e.g. based on yield as shown by Phillip.

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From guido at python.org  Fri Jul  7 22:18:35 2006
From: guido at python.org (Guido van Rossum)
Date: Fri, 7 Jul 2006 22:18:35 +0200
Subject: [Python-Dev] introducing __dir__?
In-Reply-To: <44AE4BC1.1000203@gmail.com>
References: <1d85506f0607061022v76806e5dt6b59f5771ef3f1e3@mail.gmail.com>
	<200607061341.56945.fdrake@acm.org> <44AE4BC1.1000203@gmail.com>
Message-ID: <ca471dc20607071318h50a66423sc0f279052bf322d5@mail.gmail.com>

+1 here too. This could be added easily to Python 2.6.

--Guido

On 7/7/06, Nick Coghlan <ncoghlan at gmail.com> wrote:
> Fred L. Drake, Jr. wrote:
> > On Thursday 06 July 2006 13:22, tomer filiba wrote:
> >  > my suggestion is simple -- replace this mechanism with a __dir__ -
> >  > a special method that returns the list of attributes of the object.
> >  >
> >  > rationale:
> >  > * remove deprecated __methods__, etc.
> >  > * symmetry -- just like hex() calls __hex__, etc.
> >  > * __methods__ and __members__ are lists rather than callable
> >  > objects, which means they cannot be updated on-demand
> >
> > +1
>
> +1 here, too.
>
> It would also allow objects which override __getattribute__ and/or __getattr__
> to make dir() provide a sane answer (or raise an exception to indicate that a
> sane answer isn't possible). (This was something that actually came up when
> trying to implement a namespace object that *didn't* automatically fall back
> to its class namespace for Python level attribute access)
>
> For backwards compatibility, dir() could still fall back to the current
> mechanism if __dir__ isn't found.
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
> ---------------------------------------------------------------
>              http://www.boredomandlaziness.org
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org
>


-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From guido at python.org  Fri Jul  7 22:24:22 2006
From: guido at python.org (Guido van Rossum)
Date: Fri, 7 Jul 2006 22:24:22 +0200
Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
In-Reply-To: <r01050400-1039-AEFC2D140DA011DBB8DF001124365170@10.0.0.24>
References: <e8k58c$q7u$1@sea.gmane.org>
	<r01050400-1039-AEFC2D140DA011DBB8DF001124365170@10.0.0.24>
Message-ID: <ca471dc20607071324m5a8af44eid864e8c57786cacb@mail.gmail.com>

On 7/7/06, Just van Rossum <just at letterror.com> wrote:
> Why couldn't at least augmented assignment be implicitly rebinding?

Well, personally I'm for allowing full rebinding semantics but only
when a 'global' (or 'nonlocal') statement is used first. Making
augmented assignment automatically imply 'global' etc. seems too
magical to me.

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From guido at python.org  Fri Jul  7 22:43:19 2006
From: guido at python.org (Guido van Rossum)
Date: Fri, 7 Jul 2006 22:43:19 +0200
Subject: [Python-Dev] In defense of Capabilities [was: doc for new
	restricted execution design for Python]
In-Reply-To: <bbaeab100607071239j5b413f4cy991111a87c8de9fe@mail.gmail.com>
References: <20060705155149.eipxm6j6oycgkcs8@login.werra.lunarpages.com>
	<bbaeab100607051626g6aef5ebfle46e7fdd9b71d00f@mail.gmail.com>
	<44AC9BE3.4000301@acm.org>
	<bbaeab100607060959w6f018b3dwe893d7c44f1db2f5@mail.gmail.com>
	<44ADFB4F.5070207@acm.org>
	<bbaeab100607070956p54260951k74e4ce0871f1606b@mail.gmail.com>
	<44AEA550.1030204@acm.org>
	<bbaeab100607071239j5b413f4cy991111a87c8de9fe@mail.gmail.com>
Message-ID: <ca471dc20607071343j1ba9ac7bj33ebc9071e5ebbf7@mail.gmail.com>

On 7/7/06, Brett Cannon <brett at python.org> wrote:
> I guess I am just not seeing where your approach is better than preventing
> the constructor in 'file' and having open() return the 'file' object or
> proxy object.  With your approach 'file' would be flagged, but with the
> other you just put the same check in 'file's constructor.  With both you
> would probably also want open() to be a factory function anyway.  So I don't
> see where you gain simplicity or more security.  It seems like you are
> pushing the check into the eval loop, but you still require the flagging of
> objects as unsafe.  Going with the other two proposals you don't burden the
> eval loop with the check but the objects that you would have flagged in the
> first place.
>
> It just seems like we are pushing around the flagging of unsafe stuff and
> that doesn't feel like it buys us much.

At the risk of repeating someone's point or making no sense (I'm only
following this with half an ear) I would like to point out that as
long as there's C code involved, we can have the equivalent of private
constructors in C++/Java. This means classes that cannot be
constructed by calling the class from Python. C code has to use some
other API to create an instance, bypassing this check. It seems
reasonable to me, even if most popular types *can* be constructed.
There are other types that have this property, e.g. list iterators.
Try type(iter(list()))().

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From brett at python.org  Fri Jul  7 22:55:35 2006
From: brett at python.org (Brett Cannon)
Date: Fri, 7 Jul 2006 13:55:35 -0700
Subject: [Python-Dev] In defense of Capabilities [was: doc for new
	restricted execution design for Python]
In-Reply-To: <ca471dc20607071343j1ba9ac7bj33ebc9071e5ebbf7@mail.gmail.com>
References: <20060705155149.eipxm6j6oycgkcs8@login.werra.lunarpages.com>
	<bbaeab100607051626g6aef5ebfle46e7fdd9b71d00f@mail.gmail.com>
	<44AC9BE3.4000301@acm.org>
	<bbaeab100607060959w6f018b3dwe893d7c44f1db2f5@mail.gmail.com>
	<44ADFB4F.5070207@acm.org>
	<bbaeab100607070956p54260951k74e4ce0871f1606b@mail.gmail.com>
	<44AEA550.1030204@acm.org>
	<bbaeab100607071239j5b413f4cy991111a87c8de9fe@mail.gmail.com>
	<ca471dc20607071343j1ba9ac7bj33ebc9071e5ebbf7@mail.gmail.com>
Message-ID: <bbaeab100607071355r67b291b0t285774667025e165@mail.gmail.com>

On 7/7/06, Guido van Rossum <guido at python.org> wrote:
>
> On 7/7/06, Brett Cannon <brett at python.org> wrote:
> > I guess I am just not seeing where your approach is better than
> preventing
> > the constructor in 'file' and having open() return the 'file' object or
> > proxy object.  With your approach 'file' would be flagged, but with the
> > other you just put the same check in 'file's constructor.  With both you
> > would probably also want open() to be a factory function anyway.  So I
> don't
> > see where you gain simplicity or more security.  It seems like you are
> > pushing the check into the eval loop, but you still require the flagging
> of
> > objects as unsafe.  Going with the other two proposals you don't burden
> the
> > eval loop with the check but the objects that you would have flagged in
> the
> > first place.
> >
> > It just seems like we are pushing around the flagging of unsafe stuff
> and
> > that doesn't feel like it buys us much.
>
> At the risk of repeating someone's point or making no sense (I'm only
> following this with half an ear) I would like to point out that as
> long as there's C code involved, we can have the equivalent of private
> constructors in C++/Java. This means classes that cannot be
> constructed by calling the class from Python. C code has to use some
> other API to create an instance, bypassing this check. It seems
> reasonable to me, even if most popular types *can* be constructed.
> There are other types that have this property, e.g. list iterators.
> Try type(iter(list()))().


 Good point.  C code could circumvent the bit check by doing all of the work
behind the scenes without pushing the object on the stack.  But if the check
is in the C code for the object itself it is much harder to get around.

-Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20060707/fa8a7838/attachment-0001.html 

From python-dev at zesty.ca  Fri Jul  7 23:45:13 2006
From: python-dev at zesty.ca (Ka-Ping Yee)
Date: Fri, 7 Jul 2006 16:45:13 -0500 (CDT)
Subject: [Python-Dev] Musings on concurrency and scoping ("replacing"
 Javascript)
In-Reply-To: <ca471dc20607071308v2a4e9885r7e0b09551df5711@mail.gmail.com>
References: <bbaeab100606211733x3dd58226g15e2919e8e2cdd7e@mail.gmail.com>
	<Pine.LNX.4.58.0607061642420.22834@server1.LFW.org>
	<ca471dc20607071308v2a4e9885r7e0b09551df5711@mail.gmail.com>
Message-ID: <Pine.LNX.4.58.0607071632440.22834@server1.LFW.org>

On Fri, 7 Jul 2006, Guido van Rossum wrote:
> On 7/7/06, Ka-Ping Yee <python-dev at zesty.ca> wrote:
> > I've been doing a bunch of Firefox extension programming in Javascript
> > and suddenly a few of the recent topics here came together in my head
> > in a silent kapow of thoughts.  This is kind of a side note to the
> > security discussion, but they're all interconnected: network
> > programming, concurrency, lexical scoping, security.
>
> Hm... I wonder if this style has become so popular in JS because it's
> all they have? I find callback-style programming pretty inscrutable
> pretty soon.

Yes, i would love to be able to use generators instead.  Maybe
this will be a strong selling point for Python when it's ready for
browser scripting.  (There may be a fair amount of glue required to
cope with the existing callback-based Firefox APIs though.)

> How can you ever keep track of when a '}' must be followed by a ';' ?

Hee hee -- dizzying, eh?  Actually there are no occurrences of "};"
in that example, just ");".  In JavaScript the semicolons are optional
anyway; my personal style is to use semicolons after statements but
not after function declarations.

> >     1.  Variables are more appropriately scoped; they exist
> >         only where they are meaningful.  (In a class, all the
> >         variables would be mixed into one namespace, where
> >         some of them would be invalid some of the time.)
>
> This doesn't strike me as very important. The same can happen in a
> local scope and doesn't seem to bother anyone there.

That's true, but then we're talking about existing coding styles
where people aren't concerned about minimizing security risks.
Scopes are a very powerful tool for limiting access, but you might
not notice what you're missing if you haven't been faced with a
security problem and tried using them to solve it.  (It's a bit
like generators before we had them and got used to how handy they
were: substitute scopes => generators, limiting access => flow control,
security problem => concurrency problem.)

> >     2.  Local variables are private.  (Class instances don't
> >         get their own private namespaces to play in.)
>
> Hmm... I wouldn't be so sure of that. Exception handlers have access
> to locals of stack frames above and below the current frame.

True.  It would be more accurate to say that local variables are
usually thought of as private, and therefore could be enforced as
private without breaking (hardly) anything.  Making instance members
private would break tons of stuff.


-- ?!ng

From skip at pobox.com  Sat Jul  8 00:10:04 2006
From: skip at pobox.com (skip at pobox.com)
Date: Fri, 7 Jul 2006 17:10:04 -0500
Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
In-Reply-To: <ca471dc20607071324m5a8af44eid864e8c57786cacb@mail.gmail.com>
References: <e8k58c$q7u$1@sea.gmane.org>
	<r01050400-1039-AEFC2D140DA011DBB8DF001124365170@10.0.0.24>
	<ca471dc20607071324m5a8af44eid864e8c57786cacb@mail.gmail.com>
Message-ID: <17582.56252.360308.228430@montanaro.dyndns.org>


    Guido> Well, personally I'm for allowing full rebinding semantics but
    Guido> only when a 'global' (or 'nonlocal') statement is used
    Guido> first. Making augmented assignment automatically imply 'global'
    Guido> etc. seems too magical to me.

So, if I understand correctly, in the presence of a global statement search
just goes up the lexical chain looking for the first occurrence of the
variable to modify?

    x = 0
    def f():
        x = 1
        def g():
            global x
            x = 2
            print x
        g()
        print x
    f()
    print x

Today it prints

    2
    1
    2

You're suggesting it will print 

    2
    2
    0

?

Sounds reasonable to me.  If we're talking py3k I'd chuck "global" as a
keyword though and replace it with something like "outer".

Skip

From bob at redivi.com  Sat Jul  8 00:22:53 2006
From: bob at redivi.com (Bob Ippolito)
Date: Fri, 7 Jul 2006 15:22:53 -0700
Subject: [Python-Dev] Musings on concurrency and scoping ("replacing"
	Javascript)
In-Reply-To: <ca471dc20607071308v2a4e9885r7e0b09551df5711@mail.gmail.com>
References: <bbaeab100606211733x3dd58226g15e2919e8e2cdd7e@mail.gmail.com>
	<Pine.LNX.4.58.0607061642420.22834@server1.LFW.org>
	<ca471dc20607071308v2a4e9885r7e0b09551df5711@mail.gmail.com>
Message-ID: <03951EDD-187C-4751-B158-4C39C358CC3F@redivi.com>


On Jul 7, 2006, at 1:08 PM, Guido van Rossum wrote:

> On 7/7/06, Ka-Ping Yee <python-dev at zesty.ca> wrote:
>> I've been doing a bunch of Firefox extension programming in  
>> Javascript
>> and suddenly a few of the recent topics here came together in my head
>> in a silent kapow of thoughts.  This is kind of a side note to the
>> security discussion, but they're all interconnected: network
>> programming, concurrency, lexical scoping, security.
>
> Hm... I wonder if this style has become so popular in JS because it's
> all they have? I find callback-style programming pretty inscrutable
> pretty soon.

You really don't have any choice without continuations or some built- 
in concurrency primitive. Callbacks are slightly less painful in  
JavaScript because you can define them in-line instead of naming it  
first.

>> Client-side web scripting tends to have a callback/continuation-ish
>> concurrency style because it has to deal with network transactions
>> (which can stall for long periods of time) in a user interface that
>> is expected to stay always responsive.  The Firefox API is full of
>> listeners/observers, events, and continuation-like things.  So one
>> thing to consider is that, when Python is used for these purposes,
>> it may be written in a specialized style.
>>
>> As i write JavaScript in this style i find i use nested functions
>> a lot.  When i want to set up a callback that uses variables in the
>> current context, the natural thing to do is to define a new function
>> in the local namespace.  And if that function has to also provide a
>> callback, then it has another function nested within it and so on.
>>
>>     function spam() {
>>         var local_A = do_work();
>>         do_network_transaction(
>>             new function(result_1) {
>>                 var local_B = do_work(result_1);
>>                 do_network_transaction(
>>                     new function(result_2) {
>>                         do_work(local_A, local_B, result_1,  
>> result_2);
>>                         ...
>>                     }
>>                 );
>>             }
>>         );
>>     }
>
> How can you ever keep track of when a '}' must be followed by a ';' ?

"}\n" is the same as "};" as far as the JavaScript spec goes, you can  
do either or both.

-bob




From scott+python-dev at scottdial.com  Sat Jul  8 00:42:03 2006
From: scott+python-dev at scottdial.com (Scott Dial)
Date: Fri, 07 Jul 2006 18:42:03 -0400
Subject: [Python-Dev] 2.5 and beyond
In-Reply-To: <ee2a432c0606300005g256b3391na6e684123d3e9e93@mail.gmail.com>
References: <ee2a432c0606300005g256b3391na6e684123d3e9e93@mail.gmail.com>
Message-ID: <44AEE33B.2020702@scottdial.com>

Neal Norwitz wrote:
> The current list of serious bugs are in the PEP:
> 
>   http://www.python.org/dev/peps/pep-0356/
> 
> If there are any bugs you think should be considered show stoppers,
> mail them to the list and I will update the PEP.

http://www.python.org/sf/1519018

I believe this regression in syntax checking needs to be addressed.

-- 
Scott Dial
scott at scottdial.com
scodial at indiana.edu

From greg.ewing at canterbury.ac.nz  Sat Jul  8 01:32:00 2006
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sat, 08 Jul 2006 11:32:00 +1200
Subject: [Python-Dev] doc for new restricted execution design for Python
In-Reply-To: <44AE4940.5020603@gmail.com>
References: <20060705154750.xzn3ft0hq2gw0koc@login.werra.lunarpages.com>
	<bbaeab100607051611pf70f63drd1c4a52d30d8f544@mail.gmail.com>
	<44AC60D0.1040508@canterbury.ac.nz>
	<bbaeab100607061027p7d742287n441b153370fef2e3@mail.gmail.com>
	<44AE1256.1000509@canterbury.ac.nz> <44AE4940.5020603@gmail.com>
Message-ID: <44AEEEF0.5030001@canterbury.ac.nz>

Nick Coghlan wrote:

> What would the signature of the file constructor be in that case?

If it's possible to call it at all, I think it would
have to take a file descriptor, or whatever the
platform's OS-level representation of an open file
is.

The other possibility is to just raise an exception
if you call file() from Python code.

--
Greg

From greg.ewing at canterbury.ac.nz  Sat Jul  8 01:40:34 2006
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sat, 08 Jul 2006 11:40:34 +1200
Subject: [Python-Dev] [slighly OT] Native speakers and hurting brains
In-Reply-To: <e8m44k$5qb$1@sea.gmane.org>
References: <20060705101821.7sqnxo7lu5cgg4ks@login.werra.lunarpages.com>
	<ca471dc20607051027y677c7127wfe098401c366dc74@mail.gmail.com>
	<e8m44k$5qb$1@sea.gmane.org>
Message-ID: <44AEF0F2.5040007@canterbury.ac.nz>

Boris Borcic wrote:

> I believe that in this case native linguistic intuition made the decision...

The reason has nothing to do with language. Guido didn't
want sum() to become an attractive nuisance by *appearing*
to be an obvious way of joining a list of strings, while
actually being a very inefficient way of doing that.

Considerable effort was put into trying to make sum()
smart enough to detect when you were using it on a
list of strings and do "".join behind the scenes, but
Guido decided in the end that it wasn't worth the
trouble, given that he only ever intended sum() to
be used on numbers in the first place.

--
Greg

From brett at python.org  Sat Jul  8 01:54:37 2006
From: brett at python.org (Brett Cannon)
Date: Fri, 7 Jul 2006 16:54:37 -0700
Subject: [Python-Dev] doc for new restricted execution design for Python
In-Reply-To: <44AEEEF0.5030001@canterbury.ac.nz>
References: <20060705154750.xzn3ft0hq2gw0koc@login.werra.lunarpages.com>
	<bbaeab100607051611pf70f63drd1c4a52d30d8f544@mail.gmail.com>
	<44AC60D0.1040508@canterbury.ac.nz>
	<bbaeab100607061027p7d742287n441b153370fef2e3@mail.gmail.com>
	<44AE1256.1000509@canterbury.ac.nz> <44AE4940.5020603@gmail.com>
	<44AEEEF0.5030001@canterbury.ac.nz>
Message-ID: <bbaeab100607071654t4e5c5e84nb0b27f056baaa989@mail.gmail.com>

On 7/7/06, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
>
> Nick Coghlan wrote:
>
> > What would the signature of the file constructor be in that case?
>
> If it's possible to call it at all, I think it would
> have to take a file descriptor, or whatever the
> platform's OS-level representation of an open file
> is.
>
> The other possibility is to just raise an exception
> if you call file() from Python code.


I am going with raising an exception.

-Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20060707/09f05947/attachment.html 

From ncoghlan at gmail.com  Sat Jul  8 03:39:04 2006
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 08 Jul 2006 11:39:04 +1000
Subject: [Python-Dev] In defense of Capabilities [was: doc for
 new	restricted execution design for Python]
In-Reply-To: <bbaeab100607071355r67b291b0t285774667025e165@mail.gmail.com>
References: <20060705155149.eipxm6j6oycgkcs8@login.werra.lunarpages.com>	<bbaeab100607051626g6aef5ebfle46e7fdd9b71d00f@mail.gmail.com>	<44AC9BE3.4000301@acm.org>	<bbaeab100607060959w6f018b3dwe893d7c44f1db2f5@mail.gmail.com>	<44ADFB4F.5070207@acm.org>	<bbaeab100607070956p54260951k74e4ce0871f1606b@mail.gmail.com>	<44AEA550.1030204@acm.org>	<bbaeab100607071239j5b413f4cy991111a87c8de9fe@mail.gmail.com>	<ca471dc20607071343j1ba9ac7bj33ebc9071e5ebbf7@mail.gmail.com>
	<bbaeab100607071355r67b291b0t285774667025e165@mail.gmail.com>
Message-ID: <44AF0CB8.5010607@gmail.com>

Brett Cannon wrote:
>  Good point.  C code could circumvent the bit check by doing all of the 
> work behind the scenes without pushing the object on the stack.  But if 
> the check is in the C code for the object itself it is much harder to 
> get around.

C code can circumvent the bit check by calling fopen() directly and pushing 
something onto the stack that isn't even recognised by the interpreter as a 
file object :)

You *have* to trust C code completely before importing it, because it has 
access to the platform C library and can do whatever the heck it wants.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
             http://www.boredomandlaziness.org

From brett at python.org  Sat Jul  8 04:01:27 2006
From: brett at python.org (Brett Cannon)
Date: Fri, 7 Jul 2006 19:01:27 -0700
Subject: [Python-Dev] In defense of Capabilities [was: doc for new
	restricted execution design for Python]
In-Reply-To: <44AF0CB8.5010607@gmail.com>
References: <20060705155149.eipxm6j6oycgkcs8@login.werra.lunarpages.com>
	<44AC9BE3.4000301@acm.org>
	<bbaeab100607060959w6f018b3dwe893d7c44f1db2f5@mail.gmail.com>
	<44ADFB4F.5070207@acm.org>
	<bbaeab100607070956p54260951k74e4ce0871f1606b@mail.gmail.com>
	<44AEA550.1030204@acm.org>
	<bbaeab100607071239j5b413f4cy991111a87c8de9fe@mail.gmail.com>
	<ca471dc20607071343j1ba9ac7bj33ebc9071e5ebbf7@mail.gmail.com>
	<bbaeab100607071355r67b291b0t285774667025e165@mail.gmail.com>
	<44AF0CB8.5010607@gmail.com>
Message-ID: <bbaeab100607071901n4871b058gcf6821b82cc8c568@mail.gmail.com>

On 7/7/06, Nick Coghlan <ncoghlan at gmail.com> wrote:
>
> Brett Cannon wrote:
> >  Good point.  C code could circumvent the bit check by doing all of the
> > work behind the scenes without pushing the object on the stack.  But if
> > the check is in the C code for the object itself it is much harder to
> > get around.
>
> C code can circumvent the bit check by calling fopen() directly and
> pushing
> something onto the stack that isn't even recognised by the interpreter as
> a
> file object :)


Right, but you can take measures to prevent accidental circumvention.

You *have* to trust C code completely before importing it, because it has
> access to the platform C library and can do whatever the heck it wants.


Yep.

-Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20060707/2a96118f/attachment.htm 

From guido at python.org  Sat Jul  8 05:08:40 2006
From: guido at python.org (Guido van Rossum)
Date: Sat, 8 Jul 2006 05:08:40 +0200
Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
In-Reply-To: <17582.56252.360308.228430@montanaro.dyndns.org>
References: <e8k58c$q7u$1@sea.gmane.org>
	<r01050400-1039-AEFC2D140DA011DBB8DF001124365170@10.0.0.24>
	<ca471dc20607071324m5a8af44eid864e8c57786cacb@mail.gmail.com>
	<17582.56252.360308.228430@montanaro.dyndns.org>
Message-ID: <ca471dc20607072008y4744ca7aid568ba414041372f@mail.gmail.com>

On 7/8/06, skip at pobox.com <skip at pobox.com> wrote:
>
>     Guido> Well, personally I'm for allowing full rebinding semantics but
>     Guido> only when a 'global' (or 'nonlocal') statement is used
>     Guido> first. Making augmented assignment automatically imply 'global'
>     Guido> etc. seems too magical to me.
>
> So, if I understand correctly, in the presence of a global statement search
> just goes up the lexical chain looking for the first occurrence of the
> variable to modify?
>
>     x = 0
>     def f():
>         x = 1
>         def g():
>             global x
>             x = 2
>             print x
>         g()
>         print x
>     f()
>     print x
>
> Today it prints
>
>     2
>     1
>     2
>
> You're suggesting it will print
>
>     2
>     2
>     0
>
> ?

Right. And if the search finds no scope that defines x, it's a
compile-time error (that's also new).

> Sounds reasonable to me.  If we're talking py3k I'd chuck "global" as a
> keyword though and replace it with something like "outer".

This is still under debate. I don't think we ought to change this in
Python 2.x until we've settled on the 3.x syntax and semantics;
eventually (in Python 2.9 or so :-) we can backport it with a
__future__ statement.

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From talin at acm.org  Sat Jul  8 05:29:40 2006
From: talin at acm.org (Talin)
Date: Fri, 07 Jul 2006 20:29:40 -0700
Subject: [Python-Dev] In defense of Capabilities [was: doc for
 new	restricted execution design for Python]
In-Reply-To: <bbaeab100607071355r67b291b0t285774667025e165@mail.gmail.com>
References: <20060705155149.eipxm6j6oycgkcs8@login.werra.lunarpages.com>	<bbaeab100607051626g6aef5ebfle46e7fdd9b71d00f@mail.gmail.com>	<44AC9BE3.4000301@acm.org>	<bbaeab100607060959w6f018b3dwe893d7c44f1db2f5@mail.gmail.com>	<44ADFB4F.5070207@acm.org>	<bbaeab100607070956p54260951k74e4ce0871f1606b@mail.gmail.com>	<44AEA550.1030204@acm.org>	<bbaeab100607071239j5b413f4cy991111a87c8de9fe@mail.gmail.com>	<ca471dc20607071343j1ba9ac7bj33ebc9071e5ebbf7@mail.gmail.com>
	<bbaeab100607071355r67b291b0t285774667025e165@mail.gmail.com>
Message-ID: <44AF26A4.2040201@acm.org>

Brett Cannon wrote:
> On 7/7/06, Guido van Rossum <guido at python.org> wrote:
> 
>>
>> On 7/7/06, Brett Cannon <brett at python.org> wrote:
>> > I guess I am just not seeing where your approach is better than
>> preventing
>> > the constructor in 'file' and having open() return the 'file' object or
>> > proxy object.  With your approach 'file' would be flagged, but with the
>> > other you just put the same check in 'file's constructor.  With both 
>> you
>> > would probably also want open() to be a factory function anyway.  So I
>> don't
>> > see where you gain simplicity or more security.  It seems like you are
>> > pushing the check into the eval loop, but you still require the 
>> flagging
>> of
>> > objects as unsafe.  Going with the other two proposals you don't burden
>> the
>> > eval loop with the check but the objects that you would have flagged in
>> the
>> > first place.
>> >
>> > It just seems like we are pushing around the flagging of unsafe stuff
>> and
>> > that doesn't feel like it buys us much.
>>
>> At the risk of repeating someone's point or making no sense (I'm only
>> following this with half an ear) I would like to point out that as
>> long as there's C code involved, we can have the equivalent of private
>> constructors in C++/Java. This means classes that cannot be
>> constructed by calling the class from Python. C code has to use some
>> other API to create an instance, bypassing this check. It seems
>> reasonable to me, even if most popular types *can* be constructed.
>> There are other types that have this property, e.g. list iterators.
>> Try type(iter(list()))().
> 
> 
> 
> Good point.  C code could circumvent the bit check by doing all of the work
> behind the scenes without pushing the object on the stack.  But if the 
> check
> is in the C code for the object itself it is much harder to get around.
> 
> -Brett

I may be confused (I usually am), but I think you are misinterpreting 
Guido's point. I think what he is saying is not "you should beware of C 
code getting around the checks" but rather he is pointing out that C 
code that gets around the checks can be a useful part of the system - 
thus the notion of "private constructors", in other words methods for 
creating an object that are normally inaccessible to Python code.

As to your point: I think I am beginning to understand, so let me 
reiterate to see if I have it right.

In the scenario you describe, the open() function is replaced by a 
function that returns a proxy that has limits on what it is allowed to 
do. (Ideally, it would return one of several different types of proxies 
based on the input file path - so a file handle to /tmp would have a 
different set of restrictions than a file handle to /home/me.)

Somewhere inside this proxy is the 'real' file handle. We need to insure 
that the proxy is air-tight, so that it can't 'leak' to the outside 
world. (The original motivation for my scheme was the belief that 
air-tightness couldn't be achieved, however the point that Guido makes 
above is beginning to make me believe otherwise.)

The proxy also has to be able to support all of the methods that the 
regular file handle can - because we're going to want to pass that proxy 
over to other subsystems that don't know that they are dealing with a 
proxy - such as XML parsers or config file readers. Because the 
permission checks are implemented in the proxy, this means that library 
code using the proxy has exactly the same access restrictions as the 
sandboxed code.

If you want any library code to be able to have additional privileges 
beyond what the sandboxed code can do, you'll need to pass them the 
'real' file handle, something which can only be done by either the 
proxy, or by a C 'gateway' function. This is not a problem as long as 
the library code doesn't attempt to hold on to the file handle 
(otherwise then the sandboxed code could potentially grab it from the 
library's objects.) So for any case where a library needs additional 
privileges, you'll need to add additional methods to the proxy or create 
the gateway functions to deal with that.

So if it truly is the case that the file handle cannot leak into the 
outside world, then you are correct - there's no need to put a check 
into the interpreter. My motivation was based on the belief that there 
would always be a way to get around that, so instead the notion was to 
'poison' these protected objects so that even if they did leak out, 
attempting to use them in a restricted environment would fail.

Moreover, there would be no need to even bother preventing such leakage 
- you wouldn't have to change the behavior of __subclasses__ and so on - 
which means that the 'restricted' environment would look nearly 
identical to the normal Python programming environment, i.e. you are 
free to inspect and use __subclasses__ or any other inspection feature 
as long as the result doesn't contain any poisoned objects.

(BTW, I'm going to dub my idea the 'poisoned object' scheme, just so we 
have a label.)

While I was typing this, I did realize a drawback to poisoned objects, 
which I will illustrate by the following example:

Suppose we want to grant to the sandboxed program permission to read and 
write cofiguration properties. We don't want to give them arbitrary 
write access to the file, instead we want to force the sandbox code to 
only access that file by setting and getting properties.

This is an example where a subsystem would require elevated privileges 
compared to the main program - the config file reader / writer needs to 
be able to read & write the file as a text stream, but we don't want to 
allow the sandboxed program to just write arbitrary data to it.

The only way to enforce this restriction is to re-wrap the 'real' file 
handle - in other words, replace the 'file-like object' wrapper with a 
'config-like object' wrapper.

Merely passing the poisoned file handle to 'config' doesn't work, 
because 'config' doesn't know how to safely handle it (only the C 
gateway code can shift the interpreter into a state where poisoned 
objects can be handled safely.)

Passing the file-like proxy to 'config' doesn't work either, because the 
proxy doesn't allow arbitrary writes.

The only thing that you really can do is write another wrapper - which 
is exactly what you would have to do in the non-poison case.

-- Talin

From python-dev at zesty.ca  Sat Jul  8 06:28:06 2006
From: python-dev at zesty.ca (Ka-Ping Yee)
Date: Fri, 7 Jul 2006 23:28:06 -0500 (CDT)
Subject: [Python-Dev] In defense of Capabilities [was: doc for new
 restricted execution design for Python]
In-Reply-To: <44AF26A4.2040201@acm.org>
References: <20060705155149.eipxm6j6oycgkcs8@login.werra.lunarpages.com>
	<bbaeab100607051626g6aef5ebfle46e7fdd9b71d00f@mail.gmail.com>
	<44AC9BE3.4000301@acm.org>
	<bbaeab100607060959w6f018b3dwe893d7c44f1db2f5@mail.gmail.com>
	<44ADFB4F.5070207@acm.org>
	<bbaeab100607070956p54260951k74e4ce0871f1606b@mail.gmail.com>
	<44AEA550.1030204@acm.org>
	<bbaeab100607071239j5b413f4cy991111a87c8de9fe@mail.gmail.com>
	<ca471dc20607071343j1ba9ac7bj33ebc9071e5ebbf7@mail.gmail.com>
	<bbaeab100607071355r67b291b0t285774667025e165@mail.gmail.com>
	<44AF26A4.2040201@acm.org>
Message-ID: <Pine.LNX.4.58.0607072321000.22834@server1.LFW.org>

On Fri, 7 Jul 2006, Talin wrote:
> While I was typing this, I did realize a drawback to poisoned objects,
> which I will illustrate by the following example:
>
> Suppose we want to grant to the sandboxed program permission to read and
> write cofiguration properties. We don't want to give them arbitrary
> write access to the file, instead we want to force the sandbox code to
> only access that file by setting and getting properties.
>
> This is an example where a subsystem would require elevated privileges
> compared to the main program - the config file reader / writer needs to
> be able to read & write the file as a text stream, but we don't want to
> allow the sandboxed program to just write arbitrary data to it.

The situation you're describing here is a classic case of one
component keeping a closely held authority while using it to
provide some limited capability to some other component.  This
comes up quite often when you're trying to write secure code.

If you want to be able to write that subsystem in Python, then
we will need a way to create airtight Python objects (i.e. objects
that only leak what they explicitly choose to leak).

So this goes back to the big question of goals:

    Do we want to be able to protect one piece of Python code
    from another piece of Python code?

I'd like the answer to be yes.  It sounded for a while like this
was not part of Brett's plan, though.  Now i'm not so sure.  It
sounds like you're also interested in having the answer be yes?

Let's keep talking about and playing with more examples -- i think
they'll help us understand what goals we should aim for and what
pitfalls to anticipate before we nail down too many details.


-- ?!ng

From guido at python.org  Sat Jul  8 06:45:08 2006
From: guido at python.org (Guido van Rossum)
Date: Sat, 8 Jul 2006 06:45:08 +0200
Subject: [Python-Dev] In defense of Capabilities [was: doc for new
	restricted execution design for Python]
In-Reply-To: <Pine.LNX.4.58.0607072321000.22834@server1.LFW.org>
References: <20060705155149.eipxm6j6oycgkcs8@login.werra.lunarpages.com>
	<bbaeab100607060959w6f018b3dwe893d7c44f1db2f5@mail.gmail.com>
	<44ADFB4F.5070207@acm.org>
	<bbaeab100607070956p54260951k74e4ce0871f1606b@mail.gmail.com>
	<44AEA550.1030204@acm.org>
	<bbaeab100607071239j5b413f4cy991111a87c8de9fe@mail.gmail.com>
	<ca471dc20607071343j1ba9ac7bj33ebc9071e5ebbf7@mail.gmail.com>
	<bbaeab100607071355r67b291b0t285774667025e165@mail.gmail.com>
	<44AF26A4.2040201@acm.org>
	<Pine.LNX.4.58.0607072321000.22834@server1.LFW.org>
Message-ID: <ca471dc20607072145y5df22ea7h1b7fb4414fbd11b9@mail.gmail.com>

On 7/8/06, Ka-Ping Yee <python-dev at zesty.ca> wrote:
> The situation you're describing here is a classic case of one
> component keeping a closely held authority while using it to
> provide some limited capability to some other component.  This
> comes up quite often when you're trying to write secure code.
>
> If you want to be able to write that subsystem in Python, then
> we will need a way to create airtight Python objects (i.e. objects
> that only leak what they explicitly choose to leak).
>
> So this goes back to the big question of goals:
>
>     Do we want to be able to protect one piece of Python code
>     from another piece of Python code?
>
> I'd like the answer to be yes.  It sounded for a while like this
> was not part of Brett's plan, though.  Now i'm not so sure.  It
> sounds like you're also interested in having the answer be yes?
>
> Let's keep talking about and playing with more examples -- i think
> they'll help us understand what goals we should aim for and what
> pitfalls to anticipate before we nail down too many details.

I'd like the answer to be no, because I don't believe that we can
trust the VM to provide sufficient barriers. The old pre-2.2
restricted execution mode tried to do this but 2.2 punched a million
holes in it. Python isn't designed for this (it doesn't even enforce
private attributes). I guess this is also the main reason I'm
skeptical about capabilities for Python.

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From shane at hathawaymix.org  Sat Jul  1 17:09:25 2006
From: shane at hathawaymix.org (Shane Hathaway)
Date: Sat, 01 Jul 2006 09:09:25 -0600
Subject: [Python-Dev] ImportWarning flood
In-Reply-To: <200607011758.03756.anthony@interlink.com.au>
References: <ca471dc20606251751q36f2accbr3ff3fe8fbd24b20c@mail.gmail.com>
	<44A5D5DA.2060601@hathawaymix.org>
	<ca471dc20606301955i270d466ka1f3076d8390df28@mail.gmail.com>
	<200607011758.03756.anthony@interlink.com.au>
Message-ID: <44A69025.9030503@hathawaymix.org>

Anthony Baxter wrote:
> On Saturday 01 July 2006 12:55, Guido van Rossum wrote:
>> It's up to the release manager now to decide whether the pitchforks
>> at Google or the pitchforks in the larger Python community are
>> sharper. ;-)
> 
> At this point, I think removing the warning code is the prudent 
> course. If someone wanted to find an easy and safe way to make it 
> only be triggered when the import fails, it could stay in.

I created a patch (#1515361) intended to do just that.

https://sourceforge.net/tracker/index.php?func=detail&aid=1515361&group_id=5470&atid=305470

Unfortunately, something appears to be misconfigured in my ISP's mail
server, and as a result, my messages aren't making it to this list.  So
you may not have noticed Guido's re-posting of my message announcing the
patch.

Shane


From tomerfiliba at gmail.com  Sat Jul  8 18:49:46 2006
From: tomerfiliba at gmail.com (tomer filiba)
Date: Sat, 8 Jul 2006 18:49:46 +0200
Subject: [Python-Dev] exception too expensive?
Message-ID: <1d85506f0607080949v1853d38u28c952954afb02d1@mail.gmail.com>

i thought avoiding a second dict lookup should be faster, but it turned out
to be completely wrong. it's only marginally faster, but if an exception
occurs,
it's x10 slower.

#
# the code
#
>>> import time
>>> b = dict((i, 7) for i in range(1000))
>>> def try_lookup(k):
...     try:
...         return b[k]
...     except KeyError:
...         return 17
...
>>> def if_lookup(k):
...    if k in b:
...        return b[k]
...    return 17
...
>>> def test(f, k, count=100000):
...     t=time.time()
...     while count:
...         count -=1
...         f(k)
...     return time.time()-t
...

#
# try_lookup with valid key
#
>>> test(try_lookup, 56, 1000000)
0.6099998950958252
>>> test(try_lookup, 56, 1000000)
0.60899996757507324
>>> test(try_lookup, 56, 1000000)
0.6099998950958252

~0.61 sec

#
# if_lookup with valid key
#
>>> test(if_lookup, 56, 1000000)
0.68799996376037598
>>> test(if_lookup, 56, 1000000)
0.68700003623962402
>>> test(if_lookup, 56, 1000000)
0.67200016975402832

~0.68 sec

#
# try_lookup with invalid key
#
>>> test(try_lookup, 9456, 1000000)
7.062000036239624
>>> test(try_lookup, 9456, 1000000)
6.812000036239624
>>> test(try_lookup, 9456, 1000000)
6.8440001010894775

~6.90 sec

#
# if_lookup with invalid key
#
>>> test(if_lookup, 9456, 1000000)
0.625
>>> test(if_lookup, 9456, 1000000)
0.64100003242492676
>>> test(if_lookup, 9456, 1000000)
0.65599989891052246

~0.64 sec

======

before someone says "why don't you use dict.get", it's not applicable in
my code. i have a cache -- either object already exists in the cache, or i
create and store it in the cache -- so get() and setdefault() are not
useful.

so first of all, i'd recommend not using the try_lookup method; use the
if_lookup instead... unless you can be sure the changes a KeyError
will be raised are marginal.

second, isn't there anything that can be done to improve the performance
of exceptions? imo, exceptions should be cheap. i understand it has to
do with creating the exception instance everytime, but can't something
be done to improve that? for example, hold a cache exception instances
and just memcpy them when needed (just a wild idea)?


-tomer
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20060708/74dcbdd6/attachment.htm 

From fredrik at pythonware.com  Sat Jul  8 18:54:50 2006
From: fredrik at pythonware.com (Fredrik Lundh)
Date: Sat, 08 Jul 2006 18:54:50 +0200
Subject: [Python-Dev] exception too expensive?
In-Reply-To: <1d85506f0607080949v1853d38u28c952954afb02d1@mail.gmail.com>
References: <1d85506f0607080949v1853d38u28c952954afb02d1@mail.gmail.com>
Message-ID: <e8oo0o$525$1@sea.gmane.org>

tomer filiba wrote:

> i thought avoiding a second dict lookup should be faster, but it turned out
> to be completely wrong. it's only marginally faster, but if an exception 
> occurs, it's x10 slower.

maybe you should take this to comp.lang.python instead...

</F>


From raymond.hettinger at verizon.net  Sat Jul  8 23:37:38 2006
From: raymond.hettinger at verizon.net (Raymond Hettinger)
Date: Sat, 08 Jul 2006 21:37:38 -0000
Subject: [Python-Dev] exception too expensive?
References: <1d85506f0607080949v1853d38u28c952954afb02d1@mail.gmail.com>
Message-ID: <003401c68b43$b4ddeaf0$dc00000a@RaymondLaptop1>

> i thought avoiding a second dict lookup should be faster, but it turned out
> to be completely wrong.

Unless you have an expensive hash function, it is almost never worth it to try 
to avoid a second lookup.  Because of memory cache effects, the second lookup is 
dirt cheap (a cache miss is about as expensive as a floating point division). 
See dictnotes.txt for a few thoughts on the subject.


Raymond


From tim.peters at gmail.com  Sun Jul  9 03:31:26 2006
From: tim.peters at gmail.com (Tim Peters)
Date: Sat, 8 Jul 2006 21:31:26 -0400
Subject: [Python-Dev] "Missing" 2.5 feature
Message-ID: <1f7befae0607081831x4ff28307j8778f552c63be95@mail.gmail.com>

Back in:

    http://mail.python.org/pipermail/python-dev/2005-March/051856.html

I made a pitch for adding:

    sys._current_frames()

to 2.5, which would return a dict mapping each thread's id to that
thread's current (Python) frame.  As noted there, an extension module
exists along these lines that's used at least by the popular Zope
DeadlockDebugger product, but it's not possible to do it correctly in
an extension.  The latter is why it needs to be in the core (so long
as it's in an extension, it risks segfaulting, because the core's
internal `head_mutex` lock isn't exposed).

I forgot about this but was recently reminded.  How much opposition
would there be to sneaking this into 2.5b2?  It would consist of
adding a relatively simple new function, docs, and tests; since it
wouldn't _change_ any existing code, it would have a hard time
breaking anything that currently works.

From brett at python.org  Sun Jul  9 04:44:40 2006
From: brett at python.org (Brett Cannon)
Date: Sat, 8 Jul 2006 19:44:40 -0700
Subject: [Python-Dev] In defense of Capabilities [was: doc for new
	restricted execution design for Python]
In-Reply-To: <44AF26A4.2040201@acm.org>
References: <20060705155149.eipxm6j6oycgkcs8@login.werra.lunarpages.com>
	<44AC9BE3.4000301@acm.org>
	<bbaeab100607060959w6f018b3dwe893d7c44f1db2f5@mail.gmail.com>
	<44ADFB4F.5070207@acm.org>
	<bbaeab100607070956p54260951k74e4ce0871f1606b@mail.gmail.com>
	<44AEA550.1030204@acm.org>
	<bbaeab100607071239j5b413f4cy991111a87c8de9fe@mail.gmail.com>
	<ca471dc20607071343j1ba9ac7bj33ebc9071e5ebbf7@mail.gmail.com>
	<bbaeab100607071355r67b291b0t285774667025e165@mail.gmail.com>
	<44AF26A4.2040201@acm.org>
Message-ID: <bbaeab100607081944l5bc63fb7sfee76954c413fbff@mail.gmail.com>

On 7/7/06, Talin <talin at acm.org> wrote:
>
> Brett Cannon wrote:
> > On 7/7/06, Guido van Rossum <guido at python.org> wrote:
> >
> >>
> >> On 7/7/06, Brett Cannon <brett at python.org> wrote:
> >> > I guess I am just not seeing where your approach is better than
> >> preventing
> >> > the constructor in 'file' and having open() return the 'file' object
> or
> >> > proxy object.  With your approach 'file' would be flagged, but with
> the
> >> > other you just put the same check in 'file's constructor.  With both
> >> you
> >> > would probably also want open() to be a factory function anyway.  So
> I
> >> don't
> >> > see where you gain simplicity or more security.  It seems like you
> are
> >> > pushing the check into the eval loop, but you still require the
> >> flagging
> >> of
> >> > objects as unsafe.  Going with the other two proposals you don't
> burden
> >> the
> >> > eval loop with the check but the objects that you would have flagged
> in
> >> the
> >> > first place.
> >> >
> >> > It just seems like we are pushing around the flagging of unsafe stuff
> >> and
> >> > that doesn't feel like it buys us much.
> >>
> >> At the risk of repeating someone's point or making no sense (I'm only
> >> following this with half an ear) I would like to point out that as
> >> long as there's C code involved, we can have the equivalent of private
> >> constructors in C++/Java. This means classes that cannot be
> >> constructed by calling the class from Python. C code has to use some
> >> other API to create an instance, bypassing this check. It seems
> >> reasonable to me, even if most popular types *can* be constructed.
> >> There are other types that have this property, e.g. list iterators.
> >> Try type(iter(list()))().
> >
> >
> >
> > Good point.  C code could circumvent the bit check by doing all of the
> work
> > behind the scenes without pushing the object on the stack.  But if the
> > check
> > is in the C code for the object itself it is much harder to get around.
> >
> > -Brett
>
> I may be confused (I usually am), but I think you are misinterpreting
> Guido's point. I think what he is saying is not "you should beware of C
> code getting around the checks" but rather he is pointing out that C
> code that gets around the checks can be a useful part of the system -
> thus the notion of "private constructors", in other words methods for
> creating an object that are normally inaccessible to Python code.
>
> As to your point: I think I am beginning to understand, so let me
> reiterate to see if I have it right.
>
> In the scenario you describe, the open() function is replaced by a
> function that returns a proxy that has limits on what it is allowed to
> do. (Ideally, it would return one of several different types of proxies
> based on the input file path - so a file handle to /tmp would have a
> different set of restrictions than a file handle to /home/me.)


Yep.

Somewhere inside this proxy is the 'real' file handle. We need to insure
> that the proxy is air-tight, so that it can't 'leak' to the outside
> world. (The original motivation for my scheme was the belief that
> air-tightness couldn't be achieved, however the point that Guido makes
> above is beginning to make me believe otherwise.)


The proxy is air-tight by being written in C and holding on to the
reference to the file object in the struct of the proxy.

The proxy also has to be able to support all of the methods that the
> regular file handle can - because we're going to want to pass that proxy
> over to other subsystems that don't know that they are dealing with a
> proxy - such as XML parsers or config file readers. Because the
> permission checks are implemented in the proxy, this means that library
> code using the proxy has exactly the same access restrictions as the
> sandboxed code.


Yep.  Or at least the methods that will be needed (e.g., a read-only proxy
only needs the read()-like methods).

If you want any library code to be able to have additional privileges
> beyond what the sandboxed code can do, you'll need to pass them the
> 'real' file handle, something which can only be done by either the
> proxy, or by a C 'gateway' function. This is not a problem as long as
> the library code doesn't attempt to hold on to the file handle
> (otherwise then the sandboxed code could potentially grab it from the
> library's objects.) So for any case where a library needs additional
> privileges, you'll need to add additional methods to the proxy or create
> the gateway functions to deal with that.


Basically.  A library shouldn't need additional abilities.  If they do they
are not following the security restrictions so it shouldn't work.  It should
be a rarity that a library really needs to sneak around the security of a
proxy to the point of we don't even consider it.  Duck typing should be
enough for this not to be a problem.

So if it truly is the case that the file handle cannot leak into the
> outside world, then you are correct - there's no need to put a check
> into the interpreter. My motivation was based on the belief that there
> would always be a way to get around that, so instead the notion was to
> 'poison' these protected objects so that even if they did leak out,
> attempting to use them in a restricted environment would fail.


At some point we have to trust something.  If you control the constructor,
you shouldn't have to worry about anything getting out since you won't be
able to make anything out of it.

Moreover, there would be no need to even bother preventing such leakage
> - you wouldn't have to change the behavior of __subclasses__ and so on -
> which means that the 'restricted' environment would look nearly
> identical to the normal Python programming environment, i.e. you are
> free to inspect and use __subclasses__ or any other inspection feature
> as long as the result doesn't contain any poisoned objects.


Yep.  I would like to minimize the impact on Python code and what abilities
they have.

(BTW, I'm going to dub my idea the 'poisoned object' scheme, just so we
> have a label.)
>
> While I was typing this, I did realize a drawback to poisoned objects,
> which I will illustrate by the following example:
>
> Suppose we want to grant to the sandboxed program permission to read and
> write cofiguration properties. We don't want to give them arbitrary
> write access to the file, instead we want to force the sandbox code to
> only access that file by setting and getting properties.
>
> This is an example where a subsystem would require elevated privileges
> compared to the main program - the config file reader / writer needs to
> be able to read & write the file as a text stream, but we don't want to
> allow the sandboxed program to just write arbitrary data to it.
>
> The only way to enforce this restriction is to re-wrap the 'real' file
> handle - in other words, replace the 'file-like object' wrapper with a
> 'config-like object' wrapper.


Yep.  But that is just a general issue that you are going to have with any
security system.  It all comes down to where you put your foot down in terms
or restricting access to a resource.

Merely passing the poisoned file handle to 'config' doesn't work,
> because 'config' doesn't know how to safely handle it (only the C
> gateway code can shift the interpreter into a state where poisoned
> objects can be handled safely.)
>
> Passing the file-like proxy to 'config' doesn't work either, because the
> proxy doesn't allow arbitrary writes.
>
> The only thing that you really can do is write another wrapper - which
> is exactly what you would have to do in the non-poison case.


Yep.

-Brett

-- Talin
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/brett%40python.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20060708/7564ef57/attachment-0001.htm 

From brett at python.org  Sun Jul  9 04:48:38 2006
From: brett at python.org (Brett Cannon)
Date: Sat, 8 Jul 2006 19:48:38 -0700
Subject: [Python-Dev] In defense of Capabilities [was: doc for new
	restricted execution design for Python]
In-Reply-To: <ca471dc20607072145y5df22ea7h1b7fb4414fbd11b9@mail.gmail.com>
References: <20060705155149.eipxm6j6oycgkcs8@login.werra.lunarpages.com>
	<44ADFB4F.5070207@acm.org>
	<bbaeab100607070956p54260951k74e4ce0871f1606b@mail.gmail.com>
	<44AEA550.1030204@acm.org>
	<bbaeab100607071239j5b413f4cy991111a87c8de9fe@mail.gmail.com>
	<ca471dc20607071343j1ba9ac7bj33ebc9071e5ebbf7@mail.gmail.com>
	<bbaeab100607071355r67b291b0t285774667025e165@mail.gmail.com>
	<44AF26A4.2040201@acm.org>
	<Pine.LNX.4.58.0607072321000.22834@server1.LFW.org>
	<ca471dc20607072145y5df22ea7h1b7fb4414fbd11b9@mail.gmail.com>
Message-ID: <bbaeab100607081948q5aee5485o797a32a06e5c2daf@mail.gmail.com>

On 7/7/06, Guido van Rossum <guido at python.org> wrote:
>
> On 7/8/06, Ka-Ping Yee <python-dev at zesty.ca> wrote:
> > The situation you're describing here is a classic case of one
> > component keeping a closely held authority while using it to
> > provide some limited capability to some other component.  This
> > comes up quite often when you're trying to write secure code.
> >
> > If you want to be able to write that subsystem in Python, then
> > we will need a way to create airtight Python objects (i.e. objects
> > that only leak what they explicitly choose to leak).
> >
> > So this goes back to the big question of goals:
> >
> >     Do we want to be able to protect one piece of Python code
> >     from another piece of Python code?
> >
> > I'd like the answer to be yes.  It sounded for a while like this
> > was not part of Brett's plan, though.  Now i'm not so sure.  It
> > sounds like you're also interested in having the answer be yes?
> >
> > Let's keep talking about and playing with more examples -- i think
> > they'll help us understand what goals we should aim for and what
> > pitfalls to anticipate before we nail down too many details.
>
> I'd like the answer to be no, because I don't believe that we can
> trust the VM to provide sufficient barriers. The old pre-2.2
> restricted execution mode tried to do this but 2.2 punched a million
> holes in it. Python isn't designed for this (it doesn't even enforce
> private attributes). I guess this is also the main reason I'm
> skeptical about capabilities for Python.


My plan is no.  As Guido said, getting this right is  feasibly
questionable.  I do not plan on trying to have security proxies or such
implemented in Python code; it will need to be in C.  If someone comes along
and manages to find a way to make Python work without significantly changing
the languages, great, and we can toss out my security implementation for
that.

But as of right now, I am not planning on making Python code safe to run in
Python code.

-Brett

--
> --Guido van Rossum (home page: http://www.python.org/~guido/)
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/brett%40python.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20060708/1edabd3c/attachment.html 

From brett at python.org  Sun Jul  9 05:01:32 2006
From: brett at python.org (Brett Cannon)
Date: Sat, 8 Jul 2006 20:01:32 -0700
Subject: [Python-Dev] "Missing" 2.5 feature
In-Reply-To: <1f7befae0607081831x4ff28307j8778f552c63be95@mail.gmail.com>
References: <1f7befae0607081831x4ff28307j8778f552c63be95@mail.gmail.com>
Message-ID: <bbaeab100607082001g3d56c5e3sb1f7e86a67955460@mail.gmail.com>

On 7/8/06, Tim Peters <tim.peters at gmail.com> wrote:
>
> Back in:
>
>     http://mail.python.org/pipermail/python-dev/2005-March/051856.html
>
> I made a pitch for adding:
>
>     sys._current_frames()
>
> to 2.5, which would return a dict mapping each thread's id to that
> thread's current (Python) frame.  As noted there, an extension module
> exists along these lines that's used at least by the popular Zope
> DeadlockDebugger product, but it's not possible to do it correctly in
> an extension.  The latter is why it needs to be in the core (so long
> as it's in an extension, it risks segfaulting, because the core's
> internal `head_mutex` lock isn't exposed).
>
> I forgot about this but was recently reminded.  How much opposition
> would there be to sneaking this into 2.5b2?  It would consist of
> adding a relatively simple new function, docs, and tests; since it
> wouldn't _change_ any existing code, it would have a hard time
> breaking anything that currently works.


Well, my understanding is that beta is feature freeze, so I am going to say
we shouldn't do this.  Obviously this can go in day one when 2.6 is started.

Anyway, this is a true test of how rigid the rules are!  Can Neal and
Anthony going to say no to Uncle Timmy?  =)

-Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20060708/a8240a7a/attachment.htm 

From talin at acm.org  Sun Jul  9 05:15:18 2006
From: talin at acm.org (Talin)
Date: Sat, 08 Jul 2006 20:15:18 -0700
Subject: [Python-Dev] In defense of Capabilities [was: doc for
 new	restricted execution design for Python]
In-Reply-To: <bbaeab100607081948q5aee5485o797a32a06e5c2daf@mail.gmail.com>
References: <20060705155149.eipxm6j6oycgkcs8@login.werra.lunarpages.com>	<44ADFB4F.5070207@acm.org>	<bbaeab100607070956p54260951k74e4ce0871f1606b@mail.gmail.com>	<44AEA550.1030204@acm.org>	<bbaeab100607071239j5b413f4cy991111a87c8de9fe@mail.gmail.com>	<ca471dc20607071343j1ba9ac7bj33ebc9071e5ebbf7@mail.gmail.com>	<bbaeab100607071355r67b291b0t285774667025e165@mail.gmail.com>	<44AF26A4.2040201@acm.org>	<Pine.LNX.4.58.0607072321000.22834@server1.LFW.org>	<ca471dc20607072145y5df22ea7h1b7fb4414fbd11b9@mail.gmail.com>
	<bbaeab100607081948q5aee5485o797a32a06e5c2daf@mail.gmail.com>
Message-ID: <44B074C6.3070701@acm.org>

Brett Cannon wrote:
> On 7/7/06, Guido van Rossum <guido at python.org> wrote:
>> On 7/8/06, Ka-Ping Yee <python-dev at zesty.ca> wrote:
>> > I'd like the answer to be yes.  It sounded for a while like this
>> > was not part of Brett's plan, though.  Now i'm not so sure.  It
>> > sounds like you're also interested in having the answer be yes?
>> >
>> > Let's keep talking about and playing with more examples -- i think
>> > they'll help us understand what goals we should aim for and what
>> > pitfalls to anticipate before we nail down too many details.
>>
>> I'd like the answer to be no, because I don't believe that we can
>> trust the VM to provide sufficient barriers. The old pre-2.2
>> restricted execution mode tried to do this but 2.2 punched a million
>> holes in it. Python isn't designed for this (it doesn't even enforce
>> private attributes). I guess this is also the main reason I'm
>> skeptical about capabilities for Python.
> 
> My plan is no.  As Guido said, getting this right is  feasibly
> questionable.  I do not plan on trying to have security proxies or such
> implemented in Python code; it will need to be in C.  If someone comes 
> along
> and manages to find a way to make Python work without significantly 
> changing
> the languages, great, and we can toss out my security implementation for
> that.
> 
> But as of right now, I am not planning on making Python code safe to run in
> Python code.

It might be possible for the code *outside* the sandbox to create new 
security policies written in Python.

Lets start with the concept of a generic "protection" wrapper - its a C 
proxy object which can wrap around any Python object, and which can 
restrict access to a specific set of methods. So for example:

    protected_object = protect(myObject, methods=set('open','close'))

'protect' creates a C proxy which restricts access to the object, 
allowing only those methods listed to be called.

Now, lets create a security policy, written in Python. The policy is 
essentially a factory which creates wrapped objects:

    class MyPolicy:
       # Ask the policy to create a file object
       def file( path, perms ):
          if perms == 'r':
             # Trivial example, a real proxy would be more
             # sophisticated, and probably configurable.
             return protect( file( path, perms ),
                             methods=set('open', 'read', 'close') )
             raise SecurityException

Now, when we create our sandbox, we pass in the policy:

    sb = Sandbox( MyPolicy() )

The sandbox calls 'protect' on the policy object, preventing it from 
being inspected or called inappropriately.

-- Talin

From raymond.hettinger at verizon.net  Sun Jul  9 05:16:34 2006
From: raymond.hettinger at verizon.net (Raymond Hettinger)
Date: Sun, 09 Jul 2006 03:16:34 -0000
Subject: [Python-Dev] "Missing" 2.5 feature
References: <1f7befae0607081831x4ff28307j8778f552c63be95@mail.gmail.com>
Message-ID: <00b601c68b73$15849c30$dc00000a@RaymondLaptop1>


> I forgot about this but was recently reminded.  How much opposition
> would there be to sneaking this into 2.5b2?  It would consist of
> adding a relatively simple new function, docs, and tests; since it
> wouldn't _change_ any existing code, it would have a hard time
> breaking anything that currently works.

+1


Raymond


From g.brandl at gmx.net  Sun Jul  9 08:38:42 2006
From: g.brandl at gmx.net (Georg Brandl)
Date: Sun, 09 Jul 2006 08:38:42 +0200
Subject: [Python-Dev] "Missing" 2.5 feature
In-Reply-To: <1f7befae0607081831x4ff28307j8778f552c63be95@mail.gmail.com>
References: <1f7befae0607081831x4ff28307j8778f552c63be95@mail.gmail.com>
Message-ID: <e8q86v$r9d$1@sea.gmane.org>

Tim Peters wrote:
> Back in:
> 
>     http://mail.python.org/pipermail/python-dev/2005-March/051856.html
> 
> I made a pitch for adding:
> 
>     sys._current_frames()
> 
> to 2.5, which would return a dict mapping each thread's id to that
> thread's current (Python) frame.  As noted there, an extension module
> exists along these lines that's used at least by the popular Zope
> DeadlockDebugger product, but it's not possible to do it correctly in
> an extension.  The latter is why it needs to be in the core (so long
> as it's in an extension, it risks segfaulting, because the core's
> internal `head_mutex` lock isn't exposed).
> 
> I forgot about this but was recently reminded.  How much opposition
> would there be to sneaking this into 2.5b2?  It would consist of
> adding a relatively simple new function, docs, and tests; since it
> wouldn't _change_ any existing code, it would have a hard time
> breaking anything that currently works.

If you just check it in, using an appropriate log message[1] I think nobody
would object. <wink>

Georg

[1] Thinking of "Whitespace normalization"...


From anthony at interlink.com.au  Sun Jul  9 09:05:14 2006
From: anthony at interlink.com.au (Anthony Baxter)
Date: Sun, 9 Jul 2006 17:05:14 +1000
Subject: [Python-Dev] "Missing" 2.5 feature
In-Reply-To: <1f7befae0607081831x4ff28307j8778f552c63be95@mail.gmail.com>
References: <1f7befae0607081831x4ff28307j8778f552c63be95@mail.gmail.com>
Message-ID: <200607091705.16027.anthony@interlink.com.au>

On Sunday 09 July 2006 11:31, Tim Peters wrote:
> Back in:
>
>    
> http://mail.python.org/pipermail/python-dev/2005-March/051856.html
>
> I made a pitch for adding:
>
>     sys._current_frames()
>
> to 2.5, which would return a dict mapping each thread's id to that
> thread's current (Python) frame.  As noted there, an extension
> module exists along these lines that's used at least by the popular
> Zope DeadlockDebugger product, but it's not possible to do it
> correctly in an extension.  The latter is why it needs to be in the
> core (so long as it's in an extension, it risks segfaulting,
> because the core's internal `head_mutex` lock isn't exposed).
>
> I forgot about this but was recently reminded.  How much opposition
> would there be to sneaking this into 2.5b2?  It would consist of
> adding a relatively simple new function, docs, and tests; since it
> wouldn't _change_ any existing code, it would have a hard time
> breaking anything that currently works.


Hm. Would it be a smaller change to expose head_mutex so that the 
external module could use it?

I'm really not keen on this seeming tide of new features <wink> that 
seem to be popping up. We're only a few days away from the second and 
final planned beta - it's getting _awfully_ late to be slotting in 
new features.

Anthony
-- 
Anthony Baxter     <anthony at interlink.com.au>
It's never too late to have a happy childhood.

From richard at commonground.com.au  Sun Jul  9 09:57:54 2006
From: richard at commonground.com.au (Richard Jones)
Date: Sun, 9 Jul 2006 09:57:54 +0200
Subject: [Python-Dev] "Missing" 2.5 feature
In-Reply-To: <200607091705.16027.anthony@interlink.com.au>
References: <1f7befae0607081831x4ff28307j8778f552c63be95@mail.gmail.com>
	<200607091705.16027.anthony@interlink.com.au>
Message-ID: <0171D5BD-AAA8-4E58-ADE6-C62F033415F7@commonground.com.au>

On 09/07/2006, at 9:05 AM, Anthony Baxter wrote:
> I'm really not keen on this seeming tide of new features <wink> that
> seem to be popping up. We're only a few days away from the second and
> final planned beta - it's getting _awfully_ late to be slotting in
> new features.

And besides, one person has already been told that there's absolutely  
no new features added after the beta freeze, ever.


      Richard


From g.brandl at gmx.net  Sun Jul  9 11:24:36 2006
From: g.brandl at gmx.net (Georg Brandl)
Date: Sun, 09 Jul 2006 11:24:36 +0200
Subject: [Python-Dev] "Missing" 2.5 feature
In-Reply-To: <0171D5BD-AAA8-4E58-ADE6-C62F033415F7@commonground.com.au>
References: <1f7befae0607081831x4ff28307j8778f552c63be95@mail.gmail.com>	<200607091705.16027.anthony@interlink.com.au>
	<0171D5BD-AAA8-4E58-ADE6-C62F033415F7@commonground.com.au>
Message-ID: <e8qhu0$grv$1@sea.gmane.org>

Richard Jones wrote:
> On 09/07/2006, at 9:05 AM, Anthony Baxter wrote:
>> I'm really not keen on this seeming tide of new features <wink> that
>> seem to be popping up. We're only a few days away from the second and
>> final planned beta - it's getting _awfully_ late to be slotting in
>> new features.
> 
> And besides, one person has already been told that there's absolutely  
> no new features added after the beta freeze, ever.

Last time I looked, features approved by the BDFL and/or release manager
were OK to go in.

Besides, quite a few bugfixes are likely to introduce more "new features"
than this ;)

Cheers,
Georg


From mwh at python.net  Sun Jul  9 11:56:25 2006
From: mwh at python.net (Michael Hudson)
Date: Sun, 09 Jul 2006 10:56:25 +0100
Subject: [Python-Dev] Musings on concurrency and scoping ("replacing"
 Javascript)
In-Reply-To: <Pine.LNX.4.58.0607061642420.22834@server1.LFW.org> (Ka-Ping
	Yee's message of "Thu, 6 Jul 2006 17:17:44 -0500 (CDT)")
References: <bbaeab100606211733x3dd58226g15e2919e8e2cdd7e@mail.gmail.com>
	<Pine.LNX.4.58.0607061642420.22834@server1.LFW.org>
Message-ID: <2modvz3xva.fsf@starship.python.net>

Ka-Ping Yee <python-dev at zesty.ca> writes:

> Client-side web scripting tends to have a callback/continuation-ish
> concurrency style because it has to deal with network transactions
> (which can stall for long periods of time) in a user interface that
> is expected to stay always responsive.  The Firefox API is full of
> listeners/observers, events, and continuation-like things.  So one
> thing to consider is that, when Python is used for these purposes,
> it may be written in a specialized style.

You could even call it a Twisted style :-)

Cheers,
mwh

-- 
  Considering that this thread is completely on-topic in the way only
  c.l.py threads can be, I think I can say that you should replace
  "Oblivion" with "Gravity", and increase your Radiohead quotient.
                                      -- Ben Wolfson, comp.lang.python

From tim.peters at gmail.com  Sun Jul  9 11:56:52 2006
From: tim.peters at gmail.com (Tim Peters)
Date: Sun, 9 Jul 2006 05:56:52 -0400
Subject: [Python-Dev] "Missing" 2.5 feature
In-Reply-To: <200607091705.16027.anthony@interlink.com.au>
References: <1f7befae0607081831x4ff28307j8778f552c63be95@mail.gmail.com>
	<200607091705.16027.anthony@interlink.com.au>
Message-ID: <1f7befae0607090256h22cf00b7q1a4a6d099588d300@mail.gmail.com>

[Anthony Baxter]
> Hm. Would it be a smaller change to expose head_mutex so that the
> external module could use it?

No, in part because `head_mutex` may not exist (depends on the build
type).  What an external module would actually need is 3 new public C
API functions, callable workalikes for pystate.c's internal
HEAD_{INIT, LOCK, UNLOCK} macros.  Then the only clear use for those
would be to help implement the proposed new function -- bad tradeoff.

> I'm really not keen on this seeming tide of new features <wink> that
> seem to be popping up.

Well, suck it up, cuz that never ends ;-)

> We're only a few days away from the second and final planned beta - it's
> getting _awfully_ late to be slotting in new features.

Yup!  OTOH, I'm not proposing to change the behavior of existing code
in any way, just adding a new C function comprising about a dozen
lines of straightforward code (it can be that simple in the core
because it has direct access to the relevant internals then).  The
biggest risk I see is that I might screw up the LaTeX -- which is a
risk.  There's no realistic chance that this would suffer
platform-specific compiler or runtime problems, or break other code
(it only needs read access to the relevant internals, it never writes
to them).

Of course that doesn't mean you can't say no.  It only means you shouldn't :-)

From ark at acm.org  Sun Jul  9 17:07:56 2006
From: ark at acm.org (Andrew Koenig)
Date: Sun, 9 Jul 2006 11:07:56 -0400
Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
In-Reply-To: <17582.56252.360308.228430@montanaro.dyndns.org>
Message-ID: <001d01c6a369$78a59950$6402a8c0@arkdesktop>

> So, if I understand correctly, in the presence of a global statement
> search
> just goes up the lexical chain looking for the first occurrence of the
> variable to modify?
> 
>     x = 0
>     def f():
>         x = 1
>         def g():
>             global x
>             x = 2
>             print x
>         g()
>         print x
>     f()
>     print x
> 
> Today it prints
> 
>     2
>     1
>     2
> 
> You're suggesting it will print
> 
>     2
>     2
>     0
> 
> ?

Sounds right to me.

> Sounds reasonable to me.  If we're talking py3k I'd chuck "global" as a
> keyword though and replace it with something like "outer".

I must say that I don't like "outer" any more than I like "global."  The
problem is that in both cases we are selecting the *inner*most definition
that isn't in the current scope.




From brett at python.org  Sun Jul  9 19:52:35 2006
From: brett at python.org (Brett Cannon)
Date: Sun, 9 Jul 2006 10:52:35 -0700
Subject: [Python-Dev] In defense of Capabilities [was: doc for new
	restricted execution design for Python]
In-Reply-To: <44B074C6.3070701@acm.org>
References: <20060705155149.eipxm6j6oycgkcs8@login.werra.lunarpages.com>
	<44AEA550.1030204@acm.org>
	<bbaeab100607071239j5b413f4cy991111a87c8de9fe@mail.gmail.com>
	<ca471dc20607071343j1ba9ac7bj33ebc9071e5ebbf7@mail.gmail.com>
	<bbaeab100607071355r67b291b0t285774667025e165@mail.gmail.com>
	<44AF26A4.2040201@acm.org>
	<Pine.LNX.4.58.0607072321000.22834@server1.LFW.org>
	<ca471dc20607072145y5df22ea7h1b7fb4414fbd11b9@mail.gmail.com>
	<bbaeab100607081948q5aee5485o797a32a06e5c2daf@mail.gmail.com>
	<44B074C6.3070701@acm.org>
Message-ID: <bbaeab100607091052g4a84a5c8gedea016012ed8c08@mail.gmail.com>

On 7/8/06, Talin <talin at acm.org> wrote:
>
> Brett Cannon wrote:
> > On 7/7/06, Guido van Rossum <guido at python.org> wrote:
> >> On 7/8/06, Ka-Ping Yee <python-dev at zesty.ca> wrote:
> >> > I'd like the answer to be yes.  It sounded for a while like this
> >> > was not part of Brett's plan, though.  Now i'm not so sure.  It
> >> > sounds like you're also interested in having the answer be yes?
> >> >
> >> > Let's keep talking about and playing with more examples -- i think
> >> > they'll help us understand what goals we should aim for and what
> >> > pitfalls to anticipate before we nail down too many details.
> >>
> >> I'd like the answer to be no, because I don't believe that we can
> >> trust the VM to provide sufficient barriers. The old pre-2.2
> >> restricted execution mode tried to do this but 2.2 punched a million
> >> holes in it. Python isn't designed for this (it doesn't even enforce
> >> private attributes). I guess this is also the main reason I'm
> >> skeptical about capabilities for Python.
> >
> > My plan is no.  As Guido said, getting this right is  feasibly
> > questionable.  I do not plan on trying to have security proxies or such
> > implemented in Python code; it will need to be in C.  If someone comes
> > along
> > and manages to find a way to make Python work without significantly
> > changing
> > the languages, great, and we can toss out my security implementation for
> > that.
> >
> > But as of right now, I am not planning on making Python code safe to run
> in
> > Python code.
>
> It might be possible for the code *outside* the sandbox to create new
> security policies written in Python.
>
> Lets start with the concept of a generic "protection" wrapper - its a C
> proxy object which can wrap around any Python object, and which can
> restrict access to a specific set of methods. So for example:
>
>     protected_object = protect(myObject, methods=set('open','close'))
>
> 'protect' creates a C proxy which restricts access to the object,
> allowing only those methods listed to be called.
>
> Now, lets create a security policy, written in Python. The policy is
> essentially a factory which creates wrapped objects:
>
>     class MyPolicy:
>        # Ask the policy to create a file object
>        def file( path, perms ):
>           if perms == 'r':
>              # Trivial example, a real proxy would be more
>              # sophisticated, and probably configurable.
>              return protect( file( path, perms ),
>                              methods=set('open', 'read', 'close') )
>              raise SecurityException
>
> Now, when we create our sandbox, we pass in the policy:
>
>     sb = Sandbox( MyPolicy() )
>
> The sandbox calls 'protect' on the policy object, preventing it from
> being inspected or called inappropriately.


Using a factory method callback, one could store the PyCodeObject in a C
proxy object that just acts as a complete delegate, forwarding all method
calls to the internally stored PyCodeObject.  That would work.


For this initial implementation, though, I am not going to try to support
this.   We can always add support like this later since it doesn't
fundamentally break or change anything that is already planned.  Let's focus
on getting even more basic stuff working before we start to get too fancy.

-Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20060709/c6bbbb68/attachment.html 

From python-dev at zesty.ca  Sun Jul  9 20:50:00 2006
From: python-dev at zesty.ca (Ka-Ping Yee)
Date: Sun, 9 Jul 2006 13:50:00 -0500 (CDT)
Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
In-Reply-To: <001d01c6a369$78a59950$6402a8c0@arkdesktop>
References: <001d01c6a369$78a59950$6402a8c0@arkdesktop>
Message-ID: <Pine.LNX.4.58.0607091349230.22834@server1.LFW.org>

On Sun, 9 Jul 2006, Andrew Koenig wrote:
> > Sounds reasonable to me.  If we're talking py3k I'd chuck "global" as a
> > keyword though and replace it with something like "outer".
>
> I must say that I don't like "outer" any more than I like "global."  The
> problem is that in both cases we are selecting the *inner*most definition
> that isn't in the current scope.

That's why "nonlocal" is a better choice in my opinion.


-- ?!ng

From kbk at shore.net  Sun Jul  9 21:55:43 2006
From: kbk at shore.net (Kurt B. Kaiser)
Date: Sun, 9 Jul 2006 15:55:43 -0400 (EDT)
Subject: [Python-Dev] Weekly Python Patch/Bug Summary
Message-ID: <200607091955.k69Jthl5015458@bayview.thirdcreek.com>

Patch / Bug Summary
___________________

Patches :  393 open (+15) /  3315 closed (+17) /  3708 total (+32)
Bugs    :  908 open (+22) /  5975 closed (+49) /  6883 total (+71)
RFE     :  223 open ( -1) /   229 closed ( +2) /   452 total ( +1)

New / Reopened Patches
______________________

test_grp.py doesn't skip special NIS entry, fails  (2006-06-22)
       http://python.org/sf/1510987  opened by  Mat Martineau

socket.gethostbyaddr fix for machines with incomplete DNS  (2006-06-23)
       http://python.org/sf/1511317  opened by  Charles Schwieters

Speed up decimal object construction  (2006-06-25)
CLOSED http://python.org/sf/1511853  opened by  Nick Coghlan

ast branch changed interactive module name  (2006-06-25)
       http://python.org/sf/1512007  opened by  Jp Calderone

Improves an error message from setattr  (2006-06-26)
       http://python.org/sf/1512942  opened by  Alexander Belopolsky

[FIX][Bug 1512695] cPickle.loads seg. faults  (2006-06-27)
CLOSED http://python.org/sf/1513206  opened by  Faik Uygur

Remove reduce()  (2006-06-27)
CLOSED http://python.org/sf/1513249  opened by  Collin Winter

Clean up usage of map() in the stdlib  (2006-06-27)
       http://python.org/sf/1513299  opened by  Collin Winter

new turtle module  (2006-06-28)
       http://python.org/sf/1513695  opened by  Lingl

docs for xturtle.py  (2006-06-28)
CLOSED http://python.org/sf/1513699  opened by  Lingl

Move reduce() to functools  (2006-06-28)
       http://python.org/sf/1513870  opened by  Collin Winter

mailbox (Maildir): avoid losing messages on name clash  (2006-06-29)
       http://python.org/sf/1514543  opened by  David Watson

mailbox: use fsync() to ensure data is really on disk  (2006-06-29)
       http://python.org/sf/1514544  opened by  David Watson

proposal for a new circle method  (2006-06-29)
CLOSED http://python.org/sf/1514737  opened by  Lingl

1515163 fix - traceback and str exc  (2006-06-30)
       http://python.org/sf/1515343  opened by  Jim Jewett

urllib2 redirection fix  (2006-07-02)
       http://python.org/sf/1515745  opened by  Petr Gladkikh

tarfile.py fix for #1471427 and updates  (2006-05-09)
       http://python.org/sf/1484695  reopened by  gustaebel

tarfile.py fix for #1471427 and updates  (2006-05-09)
       http://python.org/sf/1484695  reopened by  gustaebel

Remove deprecated functions from operator  (2006-07-03)
       http://python.org/sf/1516309  opened by  Collin Winter

Module uuid: reduce pickle footprint  (2006-07-03)
       http://python.org/sf/1516327  opened by  Michael Amrhein

Module uuid: functions for retrieving MAC addres  (2006-07-03)
       http://python.org/sf/1516330  opened by  Michael Amrhein

Remove sys.exitfunc  (2006-07-03)
       http://python.org/sf/1516375  opened by  Collin Winter

OpenVMS patches Modules directory  (2006-07-04)
       http://python.org/sf/1516912  opened by  Pi?ronne Jean-Fran?ois

Fix for Lib/test/crashers/gc_inspection.py  (2006-07-04)
       http://python.org/sf/1517042  opened by  Collin Winter

Patch to commands.py to enable callback support   (2006-07-05)
       http://python.org/sf/1517586  opened by  Brad Doctor

Patch for ctypes to enable custom objects in function calls  (2006-07-05)
       http://python.org/sf/1517790  opened by  Thomas Heller

Patch for bug #1514451  (2006-07-06)
       http://python.org/sf/1517891  opened by  Thomas Lee

assert for NULL in Py_INCREF Py_DECREF  (2006-07-06)
       http://python.org/sf/1517947  opened by  Rene Dudfield

ext/win-cookbook.html has a broken link to distutils  (2006-07-07)
       http://python.org/sf/1518772  opened by  Johannes Gijsbers

New ver. of 1102879: Fix for 926423: socket timeouts  (2006-07-07)
       http://python.org/sf/1519025  opened by  Tony Nelson

turtle.py: correcting begin_fill  (2006-07-09)
       http://python.org/sf/1519566  opened by  Lingl

Patches Closed
______________

MS Toolkit Compiler no longer available  (2006-06-20)
       http://python.org/sf/1509163  closed by  loewis

Patch for 1506758 - popen2/subprocess MAXFD MemoryErrors  (2006-06-15)
       http://python.org/sf/1506760  closed by  astrand

Speed up decimal object construction  (2006-06-25)
       http://python.org/sf/1511853  closed by  ncoghlan

[FIX][Bug 1512695] cPickle.loads seg. faults  (2006-06-27)
       http://python.org/sf/1513206  closed by  gbrandl

Remove reduce()  (2006-06-27)
       http://python.org/sf/1513249  closed by  collinwinter

docs for xturtle.py  (2006-06-28)
       http://python.org/sf/1513699  closed by  loewis

Make thread stack size runtime tunable  (03/20/06)
       http://python.org/sf/1454481  closed by  sf-robot

proposal for a new circle method  (2006-06-30)
       http://python.org/sf/1514737  closed by  loewis

1515169 fix (ImportWarning flood)  (2006-06-30)
       http://python.org/sf/1515361  closed by  nnorwitz

Alternative fix for ImportWarning (fix for 1515169)  (2006-07-01)
       http://python.org/sf/1515609  closed by  nnorwitz

telnetlib timeout fix (bug 822974)  (2003-10-17)
       http://python.org/sf/825417  closed by  loewis

UUID module for Python  (2005-11-29)
       http://python.org/sf/1368955  closed by  gbrandl

IDLE L&F on MacOSX   (2006-05-19)
       http://python.org/sf/1491759  closed by  ronaldoussoren

'import foo as bar' not checking for 'as'  (2006-07-07)
       http://python.org/sf/1518569  closed by  twouters

Fix for 926423: socket timeouts + Ctrl-C don't play nice  (2005-01-15)
       http://python.org/sf/1102879  closed by  gbrandl

New / Reopened Bugs
___________________

Setting category fails with -W switch  (2006-06-22)
CLOSED http://python.org/sf/1510580  opened by  A.M. Kuchling

logging fileConfig swallows handler exception   (2006-06-18)
CLOSED http://python.org/sf/1508253  reopened by  tdir

method format of logging.Formatter caches incorrectly  (2006-06-06)
CLOSED http://python.org/sf/1501699  reopened by  blorbeer

CSV regression in 2.5a1: multi-line cells  (2006-04-06)
CLOSED http://python.org/sf/1465014  reopened by  andrewmcnamara

2.5b1 windows won't install or admit failure  (2006-06-22)
       http://python.org/sf/1510984  opened by  Jim Jewett

codec_getstreamcodec passes extra None  (2006-06-24)
CLOSED http://python.org/sf/1511381  opened by  Hye-Shik Chang

xml.sax.expatreader is missing  (2006-06-23)
       http://python.org/sf/1511497  opened by  Wummel

Python gettext doesn't support libglade (again)  (2006-06-24)
CLOSED http://python.org/sf/1511736  opened by  Shmyrev Nick

vague xref in description of sorted() builtin  (2006-06-24)
CLOSED http://python.org/sf/1511911  opened by  rurpy

Can't use sockets in 2.5b1  (2006-06-24)
CLOSED http://python.org/sf/1511964  opened by  Bryan O'Sullivan

Glitches in What's New for beta 1  (2006-06-25)
CLOSED http://python.org/sf/1511998  opened by  Nick Coghlan

OSX: debugger hangs IDLE  (2006-06-25)
       http://python.org/sf/1512124  opened by  Aahz

mailbox (2.5b1): locking doesn't work (esp. on FreeBSD)  (2006-06-25)
       http://python.org/sf/1512163  opened by  David Watson

builtin enumerate overflows  (2006-06-26)
       http://python.org/sf/1512504  opened by  James Harlow

Install on Windows Vista locks up  (2006-06-26)
       http://python.org/sf/1512604  opened by  Michael Lucas

cPickle.loads() seg.faults when interrupted with keyboard  (2006-06-26)
CLOSED http://python.org/sf/1512695  opened by  Faik Uygur

module wave does no rounding  (2006-06-26)
       http://python.org/sf/1512791  opened by  Greg Kochanski

Incorrect lineno's in code objects  (2006-06-26)
       http://python.org/sf/1512814  opened by  Thomas Wouters

'make install' failure on FreeBSD 5.3  (2006-06-26)
CLOSED http://python.org/sf/1513032  opened by  David Watson

socket close() hangs client if used without prior shutdown()  (2006-06-27)
CLOSED http://python.org/sf/1513223  opened by  Irmen de Jong

xml.sax.ParseException weirdness in python 2.5b1  (2006-06-27)
       http://python.org/sf/1513611  opened by  Marien Zwart

os.access reports incorrect write permissions in Windows  (2006-06-27)
CLOSED http://python.org/sf/1513646  opened by  Yi S. Ding

Change the name of a variable causes an exception  (2006-06-28)
       http://python.org/sf/1513802  opened by  Arun

mimetools message's To field can't be changed  (2006-06-28)
       http://python.org/sf/1513913  opened by  Albert Strasheim

Typo in signal.alarm() description  (2006-06-29)
CLOSED http://python.org/sf/1514404  opened by  Stephen Shirley

Missing module code does spurious file search  (2006-06-29)
       http://python.org/sf/1514420  opened by  Nick Maclaren

NaN comparison in Decimal broken  (2006-06-29)
       http://python.org/sf/1514428  opened by  Nick Maclaren

zipfile "append" mode should create a file if not present  (2006-06-29)
       http://python.org/sf/1514451  opened by  Ritesh Raj Sarraf

String Methods 2.3.6.1 not in TOC  (2006-06-29)
       http://python.org/sf/1514540  opened by  Terry J. Reedy

radians() doesn't work correctly  (2006-06-29)
CLOSED http://python.org/sf/1514685  opened by  Lingl

setup in Pen.__init__()  (2006-06-29)
CLOSED http://python.org/sf/1514703  opened by  Lingl

turtle.py: setup() call in Pen.__init__()  (2006-06-29)
CLOSED http://python.org/sf/1514725  opened by  Lingl

turtle.py docs out of date  (2006-06-29)
CLOSED http://python.org/sf/1514730  opened by  Lingl

site.py can break the location of the python library  (2006-06-29)
       http://python.org/sf/1514734  opened by  Mike Meyer

sgmllib should recover from unmatched quotes  (2006-06-30)
       http://python.org/sf/1515142  opened by  Sam Ruby

traceback now masks some string exceptions  (2006-06-30)
       http://python.org/sf/1515163  opened by  Jim Jewett

version says beta  (2006-06-30)
       http://python.org/sf/1515164  opened by  Jim Jewett

ImportWarning should be removed  (2006-06-30)
       http://python.org/sf/1515169  opened by  James Y Knight

wrong handling of character buffers in stringobject  (2006-07-01)
       http://python.org/sf/1515471  opened by  Georg Brandl

Exponential behavior in regular expression  (2006-07-02)
CLOSED http://python.org/sf/1515829  opened by  Erik Demaine

socket timeout inheritance on accept  (2006-07-02)
       http://python.org/sf/1515839  opened by  Jari Kirma

2.3.6.4 Mutable Sequence Types clarification  (2006-07-02)
CLOSED http://python.org/sf/1515932  opened by  Alan

bdist_msi fails (egg-info)  (2006-07-02)
CLOSED http://python.org/sf/1515998  opened by  Paul Moore

Under OS X, compiling against local readline fails  (2006-07-02)
       http://python.org/sf/1516068  opened by  Nelson Arzola

inspect.py: still infinite recursion inspecting frames  (2006-07-03)
       http://python.org/sf/1516184  opened by  Don Quijote

Decimal should allow leading or trailing spaces.  (2006-07-03)
       http://python.org/sf/1516613  opened by  Ronaldo Francisco Maia

warnings.py still refers to OverflowWarning  (2006-07-04)
CLOSED http://python.org/sf/1516910  opened by  John Machin

test_logging fails with reference count-checking  (2006-07-04)
       http://python.org/sf/1516995  opened by  Collin Winter

Getting an error message import site failed -v traceback.  (2006-07-05)
       http://python.org/sf/1517370  opened by  brijesh kumar

memory leak threading or socketserver module  (2006-07-05)
       http://python.org/sf/1517495  opened by  Ids van der Molen

filter() implementation does not match docs  (2006-07-05)
CLOSED http://python.org/sf/1517509  opened by  Collin Winter

Interpreter crash: filter() + gc.get_referrers()  (2006-07-05)
       http://python.org/sf/1517663  opened by  Collin Winter

IDLE on macosx keybindings need work  (2006-07-06)
       http://python.org/sf/1517990  opened by  Ronald Oussoren

IDLE: config-main.def contains windows-specific settings  (2006-07-06)
       http://python.org/sf/1517993  opened by  Ronald Oussoren

IDLE (macosx): Class and Path browsers show Tk menu  (2006-07-06)
       http://python.org/sf/1517996  opened by  Ronald Oussoren

c_void_pointer should accept a long pointer > 0x7fffffff  (2006-07-06)
       http://python.org/sf/1518190  opened by  Thomas Heller

cookielib doku  (2006-07-06)
CLOSED http://python.org/sf/1518346  opened by  Christoph Zwerschke

re '\' char interpretation problem  (2006-07-06)
CLOSED http://python.org/sf/1518406  opened by  ollie oldham

'\' problem in re.sub  (2006-07-06)
CLOSED http://python.org/sf/1518453  opened by  ilan29

'import foo as bar' not checking for 'as'  (2006-07-07)
CLOSED http://python.org/sf/1518569  opened by  Thomas Wouters

Website updating part of PEP101/102 out of date  (2006-07-07)
       http://python.org/sf/1518617  opened by  Johannes Gijsbers

optparse.parse_args() ret value seems to be a dict but isn't  (2006-07-07)
       http://python.org/sf/1518621  opened by  rhunger

import_as_name and dotted_as_name fail to validate syntax  (2006-07-07)
CLOSED http://python.org/sf/1519018  opened by  Scott Dial

incorrect locale.strcoll() return in Windows  (2006-07-07)
       http://python.org/sf/1519069  opened by  Brian Matherly

too many files?  (2006-07-09)
       http://python.org/sf/1519452  opened by  Joe Brown

turtle.py Docs still incomplete  (2006-07-09)
       http://python.org/sf/1519571  opened by  Lingl

Unmatched Group issue  (2006-07-09)
       http://python.org/sf/1519638  opened by  Bobby Xiao

Bugs Closed
___________

Setting category fails with -W switch  (2006-06-22)
       http://python.org/sf/1510580  closed by  bcannon

logging fileConfig swallows handler exception   (2006-06-18)
       http://python.org/sf/1508253  closed by  vsajip

Logging doesn't report the correct filename or line numbers  (2006-04-14)
       http://python.org/sf/1470422  closed by  vsajip

method format of logging.Formatter caches incorrectly  (06/06/06)
       http://python.org/sf/1501699  closed by  sf-robot

method format of logging.Formatter caches incorrectly  (2006-06-06)
       http://python.org/sf/1501699  closed by  vsajip

CSV regression in 2.5a1: multi-line cells  (2006-04-05)
       http://python.org/sf/1465014  closed by  goodger

If MAXFD too high, popen2/subprocess produce MemoryErrors  (2006-06-15)
       http://python.org/sf/1506758  closed by  astrand

IP6 address parsing i sgmllib  (2003-12-03)
       http://python.org/sf/853506  closed by  fdrake

codec_getstreamcodec passes extra None  (2006-06-24)
       http://python.org/sf/1511381  closed by  perky

Python gettext doesn't support libglade (again)  (2006-06-24)
       http://python.org/sf/1511736  closed by  loewis

vague xref in description of sorted() builtin  (2006-06-24)
       http://python.org/sf/1511911  closed by  akuchling

Can't use sockets in 2.5b1  (2006-06-24)
       http://python.org/sf/1511964  closed by  bos

Glitches in What's New for beta 1  (2006-06-25)
       http://python.org/sf/1511998  closed by  akuchling

package manager page outdated link  (2004-03-24)
       http://python.org/sf/922690  closed by  ronaldoussoren

"-framework Python" for building modules is bad  (2004-01-29)
       http://python.org/sf/887242  closed by  ronaldoussoren

PackageManager does not clean up after itself  (2003-11-07)
       http://python.org/sf/838144  closed by  ronaldoussoren

PackageManager maintainer missing documentation  (2003-07-28)
       http://python.org/sf/779154  closed by  ronaldoussoren

IDE preferences cleanup  (2003-06-22)
       http://python.org/sf/758566  closed by  ronaldoussoren

cPickle.loads() seg.faults when interrupted with keyboard  (2006-06-26)
       http://python.org/sf/1512695  closed by  nnorwitz

'make install' failure on FreeBSD 5.3  (2006-06-26)
       http://python.org/sf/1513032  closed by  nnorwitz

socket close() hangs client if used without prior shutdown()  (2006-06-27)
       http://python.org/sf/1513223  closed by  loewis

os.access reports incorrect write permissions in Windows  (2006-06-28)
       http://python.org/sf/1513646  closed by  loewis

OS/X on i386 framework build  (03/16/06)
       http://python.org/sf/1451717  closed by  sf-robot

Typo in signal.alarm() description  (2006-06-30)
       http://python.org/sf/1514404  closed by  quiver

sgmllib should allow angle brackets in quoted values  (2006-06-11)
       http://python.org/sf/1504333  closed by  fdrake

radians() doesn't work correctly  (2006-06-29)
       http://python.org/sf/1514685  closed by  loewis

setup in Pen.__init__()  (2006-06-29)
       http://python.org/sf/1514703  closed by  loewis

turtle.py: setup() call in Pen.__init__()  (2006-06-30)
       http://python.org/sf/1514725  closed by  loewis

turtle.py docs out of date  (2006-06-30)
       http://python.org/sf/1514730  closed by  loewis

incorrect LOAD/STORE_GLOBAL generation  (2006-06-06)
       http://python.org/sf/1501934  closed by  nascheme

Exponential behavior in regular expression  (2006-07-02)
       http://python.org/sf/1515829  closed by  tim_one

2.3.6.4 Mutable Sequence Types clarification  (2006-07-02)
       http://python.org/sf/1515932  closed by  akuchling

bdist_msi fails (egg-info)  (2006-07-02)
       http://python.org/sf/1515998  closed by  loewis

float/atof have become locale aware  (2006-01-29)
       http://python.org/sf/1417699  closed by  loewis

Bug in dbm - long strings in keys and values  (2003-10-21)
       http://python.org/sf/827760  closed by  loewis

Telnet.read_until() timeout parameter misleading   (2003-10-13)
       http://python.org/sf/822974  closed by  loewis

Python segfaults when freeing &quot;deep&quot; objects  (2004-04-01)
       http://python.org/sf/927248  closed by  loewis

warnings.py still refers to OverflowWarning  (2006-07-04)
       http://python.org/sf/1516910  closed by  gbrandl

2.5 rev 45972 fails build on Mac-Intel  (2006-05-11)
       http://python.org/sf/1487105  closed by  nnorwitz

filter() implementation does not match docs  (2006-07-05)
       http://python.org/sf/1517509  closed by  rhettinger

zipfile appends broken in 2.5b1  (2006-07-06)
       http://python.org/sf/1517887  deleted by  krumms

sqlite3.dll only installed with Tcl/Tk  (2006-07-05)
       http://python.org/sf/1517388  closed by  loewis

cookielib doku port cookie-attribute  (2006-07-06)
       http://python.org/sf/1518346  closed by  gbrandl

re '\' char interpretation problem  (2006-07-06)
       http://python.org/sf/1518406  closed by  niemeyer

'\' problem in re.sub  (2006-07-07)
       http://python.org/sf/1518453  closed by  gbrandl

import_as_name and dotted_as_name fail to validate syntax  (2006-07-07)
       http://python.org/sf/1519018  closed by  nnorwitz

New / Reopened RFE
__________________

Add Windows 9x/ME (lack of) support information to README.TX  (2006-06-22)
       http://python.org/sf/1510853  opened by  ajaksu

RFE Closed
__________

Add some dicts to datetime module  (2006-06-14)
       http://python.org/sf/1506340  closed by  nnorwitz

Add str.partition  (2006-01-08)
       http://python.org/sf/1399936  closed by  nnorwitz


From nas at arctrix.com  Sun Jul  9 22:54:56 2006
From: nas at arctrix.com (Neil Schemenauer)
Date: Sun, 9 Jul 2006 20:54:56 +0000 (UTC)
Subject: [Python-Dev] Unary minus bug
Message-ID: <e8rqf0$q0h$1@sea.gmane.org>

The bug was reported by Armin in SF #1333982:

    the literal -2147483648 (i.e. the value of -sys.maxint-1) gives
    a long in 2.5, but an int in <= 2.4.

I have a fix but I wonder if it's the right thing to do.  I suppose
returning a long has the chance of breaking someone code.  Here's
the test we currently have for a related case:

    [...]
    # 32-bit machine
    all_one_bits = '0xffffffff'
    self.assertEqual(eval(all_one_bits), 4294967295L)
    self.assertEqual(eval("-" + all_one_bits), -4294967295L)

I guess I would add self.assertTrue(isinstance(eval("-2147483648"), int))
to that set of tests.

  Neil


From guido at python.org  Sun Jul  9 23:20:07 2006
From: guido at python.org (Guido van Rossum)
Date: Sun, 9 Jul 2006 23:20:07 +0200
Subject: [Python-Dev] Unary minus bug
In-Reply-To: <e8rqf0$q0h$1@sea.gmane.org>
References: <e8rqf0$q0h$1@sea.gmane.org>
Message-ID: <ca471dc20607091420s59e60438q3619639020e32edb@mail.gmail.com>

I think it ought to be an int, like before.

--Guido

On 7/9/06, Neil Schemenauer <nas at arctrix.com> wrote:
> The bug was reported by Armin in SF #1333982:
>
>     the literal -2147483648 (i.e. the value of -sys.maxint-1) gives
>     a long in 2.5, but an int in <= 2.4.
>
> I have a fix but I wonder if it's the right thing to do.  I suppose
> returning a long has the chance of breaking someone code.  Here's
> the test we currently have for a related case:
>
>     [...]
>     # 32-bit machine
>     all_one_bits = '0xffffffff'
>     self.assertEqual(eval(all_one_bits), 4294967295L)
>     self.assertEqual(eval("-" + all_one_bits), -4294967295L)
>
> I guess I would add self.assertTrue(isinstance(eval("-2147483648"), int))
> to that set of tests.
>
>   Neil
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org
>


-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From tim.peters at gmail.com  Sun Jul  9 23:54:53 2006
From: tim.peters at gmail.com (Tim Peters)
Date: Sun, 9 Jul 2006 17:54:53 -0400
Subject: [Python-Dev] Unary minus bug
In-Reply-To: <e8rqf0$q0h$1@sea.gmane.org>
References: <e8rqf0$q0h$1@sea.gmane.org>
Message-ID: <1f7befae0607091454g4945e866ifb36ff6292fa489@mail.gmail.com>

[Neil Schemenauer]
> The bug was reported by Armin in SF #1333982:
>
>     the literal -2147483648 (i.e. the value of -sys.maxint-1) gives
>     a long in 2.5, but an int in <= 2.4.

That actually depends on how far back you go.  It was also a long "at
the start".  IIRC, Fred or I added hackery to make it an int, because
back then there was a sharp distinction between int and long and a
surprising number of people complained about this case.  Strictly
speaking,  -2147483648 is not an integer literal (see the grammar --
Python has only positive integer literals).

> I have a fix but I wonder if it's the right thing to do.

Practicality beats purity here, IMO -- +1.

> I suppose returning a long has the chance of breaking someone code.  Here's
> the test we currently have for a related case:
>
>     [...]
>     # 32-bit machine
>     all_one_bits = '0xffffffff'
>     self.assertEqual(eval(all_one_bits), 4294967295L)
>     self.assertEqual(eval("-" + all_one_bits), -4294967295L)
>
> I guess I would add self.assertTrue(isinstance(eval("-2147483648"), int))
> to that set of tests.

Or, more obscure but maybe better (since this also tests the similar
endcase on sizeof(long) == 8 boxes):

    minint = -sys.maxint-1
    self.assert_(isinstance(minint, int))
    self.assert_(isinstance(eval(str(minint)), int))

From nnorwitz at gmail.com  Mon Jul 10 00:02:06 2006
From: nnorwitz at gmail.com (Neal Norwitz)
Date: Sun, 9 Jul 2006 15:02:06 -0700
Subject: [Python-Dev] Unary minus bug
In-Reply-To: <e8rqf0$q0h$1@sea.gmane.org>
References: <e8rqf0$q0h$1@sea.gmane.org>
Message-ID: <ee2a432c0607091502q1c03f13fs546d1e2a8ac90709@mail.gmail.com>

Do we care about this (after your checkin and with my fix to make
32-63 bit values ints rather than longs):

# 64 bit box
>>> minint = str(-sys.maxint - 1)
>>> minint
'-9223372036854775808'
>>> eval(minint)
-9223372036854775808
>>> eval('-(%s)' % minint[1:])
-9223372036854775808L

n
--
On 7/9/06, Neil Schemenauer <nas at arctrix.com> wrote:
> The bug was reported by Armin in SF #1333982:
>
>     the literal -2147483648 (i.e. the value of -sys.maxint-1) gives
>     a long in 2.5, but an int in <= 2.4.
>
> I have a fix but I wonder if it's the right thing to do.  I suppose
> returning a long has the chance of breaking someone code.  Here's
> the test we currently have for a related case:
>
>     [...]
>     # 32-bit machine
>     all_one_bits = '0xffffffff'
>     self.assertEqual(eval(all_one_bits), 4294967295L)
>     self.assertEqual(eval("-" + all_one_bits), -4294967295L)
>
> I guess I would add self.assertTrue(isinstance(eval("-2147483648"), int))
> to that set of tests.
>
>   Neil
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/nnorwitz%40gmail.com
>

From nas at arctrix.com  Mon Jul 10 00:18:36 2006
From: nas at arctrix.com (Neil Schemenauer)
Date: Sun, 9 Jul 2006 16:18:36 -0600
Subject: [Python-Dev] Unary minus bug
In-Reply-To: <ee2a432c0607091502q1c03f13fs546d1e2a8ac90709@mail.gmail.com>
References: <e8rqf0$q0h$1@sea.gmane.org>
	<ee2a432c0607091502q1c03f13fs546d1e2a8ac90709@mail.gmail.com>
Message-ID: <20060709221836.GA20195@mems-exchange.org>

On Sun, Jul 09, 2006 at 03:02:06PM -0700, Neal Norwitz wrote:
> Do we care about this (after your checkin and with my fix to make
> 32-63 bit values ints rather than longs):
> 
> # 64 bit box
> >>>minint = str(-sys.maxint - 1)
> >>>minint
> '-9223372036854775808'
> >>>eval(minint)
> -9223372036854775808
> >>>eval('-(%s)' % minint[1:])
> -9223372036854775808L

I don't think we care.  Python <= 2.4 has the same behavior (i.e.
adding parens defeats the constant folding since it's done at the
syntax tree level).

Where's your fix regarding 32-63 bit ints?  I'm not familiar with
that issue but my recent check-in appears to have broke the 64-bit
buildbots.  If no one can explain what's going on there then I guess
I will have to find a login on a 64-bit machine at debug it myself.

  Neil

From nnorwitz at gmail.com  Mon Jul 10 00:23:26 2006
From: nnorwitz at gmail.com (Neal Norwitz)
Date: Sun, 9 Jul 2006 15:23:26 -0700
Subject: [Python-Dev] Unary minus bug
In-Reply-To: <20060709221836.GA20195@mems-exchange.org>
References: <e8rqf0$q0h$1@sea.gmane.org>
	<ee2a432c0607091502q1c03f13fs546d1e2a8ac90709@mail.gmail.com>
	<20060709221836.GA20195@mems-exchange.org>
Message-ID: <ee2a432c0607091523x28f0b9abwe7b7b827e5d9241c@mail.gmail.com>

On 7/9/06, Neil Schemenauer <nas at arctrix.com> wrote:
> On Sun, Jul 09, 2006 at 03:02:06PM -0700, Neal Norwitz wrote:
> > Do we care about this (after your checkin and with my fix to make
> > 32-63 bit values ints rather than longs):
> >
> > # 64 bit box
> > >>>minint = str(-sys.maxint - 1)
> > >>>minint
> > '-9223372036854775808'
> > >>>eval(minint)
> > -9223372036854775808
> > >>>eval('-(%s)' % minint[1:])
> > -9223372036854775808L
>
> I don't think we care.  Python <= 2.4 has the same behavior (i.e.
> adding parens defeats the constant folding since it's done at the
> syntax tree level).

Ok, unless someone care's enough to make a patch, I'm removing this
from the outstanding issues for 2.5.

> Where's your fix regarding 32-63 bit ints?  I'm not familiar with

I just had to finish the test.  It's checked in now and should fix the
buildbot problems.  Your test exposed the issue, but didn't cause it.
It was caused
by the optimization of converting strings to ints.  My approach to fix it was
kinda hacky (just add the proper table for 64-bit machines).  I guess
ideally we would generate this table based off sizeof(long).  Probably
should have added
a TODO.

n

From tim.peters at gmail.com  Mon Jul 10 00:44:27 2006
From: tim.peters at gmail.com (Tim Peters)
Date: Sun, 9 Jul 2006 18:44:27 -0400
Subject: [Python-Dev] "Missing" 2.5 feature
In-Reply-To: <1f7befae0607090256h22cf00b7q1a4a6d099588d300@mail.gmail.com>
References: <1f7befae0607081831x4ff28307j8778f552c63be95@mail.gmail.com>
	<200607091705.16027.anthony@interlink.com.au>
	<1f7befae0607090256h22cf00b7q1a4a6d099588d300@mail.gmail.com>
Message-ID: <1f7befae0607091544l7966feecsaaca705b3eeefaa7@mail.gmail.com>

Just to make life harder ;-), I should note that code, docs and tests
for sys._current_frames() are done, on the tim-current_frames branch.
All tests pass, and there are no leaks in the new code.  It's just a
NEWS blurb away from being just another hectic release memory :-)

From talin at acm.org  Mon Jul 10 01:27:27 2006
From: talin at acm.org (Talin)
Date: Sun, 09 Jul 2006 16:27:27 -0700
Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
In-Reply-To: <Pine.LNX.4.58.0607091349230.22834@server1.LFW.org>
References: <001d01c6a369$78a59950$6402a8c0@arkdesktop>
	<Pine.LNX.4.58.0607091349230.22834@server1.LFW.org>
Message-ID: <44B190DF.3000805@acm.org>

Ka-Ping Yee wrote:
> On Sun, 9 Jul 2006, Andrew Koenig wrote:
> 
>>>Sounds reasonable to me.  If we're talking py3k I'd chuck "global" as a
>>>keyword though and replace it with something like "outer".
>>
>>I must say that I don't like "outer" any more than I like "global."  The
>>problem is that in both cases we are selecting the *inner*most definition
>>that isn't in the current scope.
> 
> 
> That's why "nonlocal" is a better choice in my opinion.

Some alternatives:

     use x
     using x
     with x     -- recycle a keyword?
     reuse x
     use extant x
     share x
     common x
     same x
     borrow x
     existing x

Although, to be perfectly honest, the longer this discussion goes on, 
the more that I find that I'm not buying Guido's argument about it being 
better to define this at the point of use rather than at the point of 
definition. I agree with him that "point of use" is more Pythonic, but 
I'm also beginning to believe that there are some good reasons why many 
other languages do it the other way.

Part of the reason why its so hard to name this feature is that it's 
real name is something like "Hey, Python, you know that cool funky thing 
you do with defining variables in the same scope as they are assigned? 
Well, don't do that here."

-- Talin

From talin at acm.org  Mon Jul 10 01:51:45 2006
From: talin at acm.org (Talin)
Date: Sun, 09 Jul 2006 16:51:45 -0700
Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
In-Reply-To: <44B190DF.3000805@acm.org>
References: <001d01c6a369$78a59950$6402a8c0@arkdesktop>	<Pine.LNX.4.58.0607091349230.22834@server1.LFW.org>
	<44B190DF.3000805@acm.org>
Message-ID: <44B19691.4050509@acm.org>

Talin wrote:
> Some alternatives:
> 
>      use x
>      using x
>      with x     -- recycle a keyword?
>      reuse x
>      use extant x
>      share x
>      common x
>      same x
>      borrow x
>      existing x
> 
> Although, to be perfectly honest, the longer this discussion goes on, 
> the more that I find that I'm not buying Guido's argument about it being 
> better to define this at the point of use rather than at the point of 
> definition. I agree with him that "point of use" is more Pythonic, but 
> I'm also beginning to believe that there are some good reasons why many 
> other languages do it the other way.
> 
> Part of the reason why its so hard to name this feature is that it's 
> real name is something like "Hey, Python, you know that cool funky thing 
> you do with defining variables in the same scope as they are assigned? 
> Well, don't do that here."

(Followup to my own comment)

There are really 3 places where you can indicate that a variable is to 
be reused instead of redefined: 1) The point of definition in the outer 
scope, 2) A declaration in the inner scope, and 3) The actual point of 
assignment.

#1 is what I've been pushing for, #2 is what most of the discussion has 
been about, #3 has been talked about a little bit in the context of an 
augmented assignment operator.

I actually like #3 a little better than #2, but not with a new operator. 
I'm thinking more along the lines of a keyword that modifies and 
assignment statement:

    rebind x = 10

Other possible keywords are: modify, mutate, change, update, change, etc...

My gut feeling is that most code that wants to use this feature only 
wants to use it in a few places. A good example is fcgi.py (implements 
WSGI for FastCGI), where they use a mutable array to store a flag 
indicating whether or not the headers have already been sent.

-- Talin

From guido at python.org  Mon Jul 10 05:39:01 2006
From: guido at python.org (Guido van Rossum)
Date: Sun, 9 Jul 2006 20:39:01 -0700
Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
In-Reply-To: <44B19691.4050509@acm.org>
References: <001d01c6a369$78a59950$6402a8c0@arkdesktop>
	<Pine.LNX.4.58.0607091349230.22834@server1.LFW.org>
	<44B190DF.3000805@acm.org> <44B19691.4050509@acm.org>
Message-ID: <ca471dc20607092039g5cf2ea54h5225925ce877145e@mail.gmail.com>

On 7/9/06, Talin <talin at acm.org> wrote:
> Talin wrote:
> > Some alternatives:
> >
> >      use x
> >      using x
> >      with x     -- recycle a keyword?
> >      reuse x
> >      use extant x
> >      share x
> >      common x
> >      same x
> >      borrow x
> >      existing x

Of these, I like reuse, share, common slightly better than the rest.

> > Although, to be perfectly honest, the longer this discussion goes on,
> > the more that I find that I'm not buying Guido's argument about it being
> > better to define this at the point of use rather than at the point of
> > definition. I agree with him that "point of use" is more Pythonic, but
> > I'm also beginning to believe that there are some good reasons why many
> > other languages do it the other way.

Well, I still don't like point of definition. It means that something
far away can change the interpretation of something local. The reason
other languages do it differently is that variable declarations are
obligatory. It's really unacceptable that when I see

 def foo():
   x = 12

I would have to search the entire module for a possible definition of
a global named 'x' to see whether the x assigned to here is local or
not.

> > Part of the reason why its so hard to name this feature is that it's
> > real name is something like "Hey, Python, you know that cool funky thing
> > you do with defining variables in the same scope as they are assigned?
> > Well, don't do that here."

Sorry, that sounds like a B.S. argument. You can translate most
language constructs into long sentences if you want to.

> (Followup to my own comment)
>
> There are really 3 places where you can indicate that a variable is to
> be reused instead of redefined: 1) The point of definition in the outer
> scope, 2) A declaration in the inner scope, and 3) The actual point of
> assignment.
>
> #1 is what I've been pushing for, #2 is what most of the discussion has
> been about, #3 has been talked about a little bit in the context of an
> augmented assignment operator.
>
> I actually like #3 a little better than #2, but not with a new operator.
> I'm thinking more along the lines of a keyword that modifies and
> assignment statement:
>
>     rebind x = 10
>
> Other possible keywords are: modify, mutate, change, update, change, etc...

A problem with this is the ambiguity if some assignments to x use the
rebind keyword and others don't. Then there's a local x and also a
nonlocal x. When x is used in an expression (even on the RHS of a
rebind statement!) it would be interpreted as the local one.

> My gut feeling is that most code that wants to use this feature only
> wants to use it in a few places. A good example is fcgi.py (implements
> WSGI for FastCGI), where they use a mutable array to store a flag
> indicating whether or not the headers have already been sent.

Then let's allow

  nonlocal x = 12

as a shortcut for

  nonlocal x
  x = 12

Bash users should be familiar with this from e.g. export FOO=bar.

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From nnorwitz at gmail.com  Mon Jul 10 05:45:05 2006
From: nnorwitz at gmail.com (Neal Norwitz)
Date: Sun, 9 Jul 2006 20:45:05 -0700
Subject: [Python-Dev] "Missing" 2.5 feature
In-Reply-To: <1f7befae0607091544l7966feecsaaca705b3eeefaa7@mail.gmail.com>
References: <1f7befae0607081831x4ff28307j8778f552c63be95@mail.gmail.com>
	<200607091705.16027.anthony@interlink.com.au>
	<1f7befae0607090256h22cf00b7q1a4a6d099588d300@mail.gmail.com>
	<1f7befae0607091544l7966feecsaaca705b3eeefaa7@mail.gmail.com>
Message-ID: <ee2a432c0607092045n95c73k8c9f46723de07670@mail.gmail.com>

On 7/9/06, Tim Peters <tim.peters at gmail.com> wrote:
> Just to make life harder ;-), I should note that code, docs and tests
> for sys._current_frames() are done, on the tim-current_frames branch.
> All tests pass, and there are no leaks in the new code.  It's just a
> NEWS blurb away from being just another hectic release memory :-)

There hasn't been much positive response (in the original thread or
here).  Given you forgot about it for over a year, how important can
it be? :-)

What's the justifcation to add this feature, but none of the others?
Can we defer this to 2.6?

n

From nnorwitz at gmail.com  Mon Jul 10 06:00:49 2006
From: nnorwitz at gmail.com (Neal Norwitz)
Date: Sun, 9 Jul 2006 21:00:49 -0700
Subject: [Python-Dev] xml issue in 2.5
Message-ID: <ee2a432c0607092100t70f75b0fi46d3975a8e28e75a@mail.gmail.com>

http://python.org/sf/1513611
xml.sax.ParseException weirdness in python 2.5b1.  The following code
doesn't work:

from xml.sax import make_parser, SAXParseException

parser = make_parser()
try:
    parser.parse(StringIO('invalid'))
except SAXParseException:
    print 'caught it!'

Any comments?

n

From martin at v.loewis.de  Mon Jul 10 08:09:22 2006
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Mon, 10 Jul 2006 08:09:22 +0200
Subject: [Python-Dev] xml issue in 2.5
In-Reply-To: <ee2a432c0607092100t70f75b0fi46d3975a8e28e75a@mail.gmail.com>
References: <ee2a432c0607092100t70f75b0fi46d3975a8e28e75a@mail.gmail.com>
Message-ID: <44B1EF12.3010603@v.loewis.de>

Neal Norwitz wrote:
> http://python.org/sf/1513611
> xml.sax.ParseException weirdness in python 2.5b1.  The following code
> doesn't work:
> 
> from xml.sax import make_parser, SAXParseException
> 
> parser = make_parser()
> try:
>    parser.parse(StringIO('invalid'))
> except SAXParseException:
>    print 'caught it!'
> 
> Any comments?

The problem can be simplified to this:

py> import xml.sax,xmlcore.sax,sys
py> sys.modules['xml.sax'] is sys.modules['xmlcore.sax']
False

One way to fix this would be to never refer to "xmlcore"
explicitly (i.e. import from xml.sax._exceptions in
expatreader), but I guess that would defeat the purpose
of the xmlcore renaming.

Regards,
Martin

From anthony at interlink.com.au  Mon Jul 10 09:32:26 2006
From: anthony at interlink.com.au (Anthony Baxter)
Date: Mon, 10 Jul 2006 17:32:26 +1000
Subject: [Python-Dev] TRUNK FREEZE for 2.5b2, tomorrow Tuesday 11th,
	00:00 UTC
Message-ID: <200607101732.28316.anthony@interlink.com.au>

The trunk is frozen from 00:00 UTC Tuesday the 11th of July. This is 
in about 16 hours or so time. As usual, unless you're a member of the 
release team, please don't checkin past that time until I send an 
email that the release is done. 

Thanks,
Anthony
-- 
Anthony Baxter     <anthony at interlink.com.au>
It's never too late to have a happy childhood.

From scott+python-dev at scottdial.com  Mon Jul 10 11:25:33 2006
From: scott+python-dev at scottdial.com (Scott Dial)
Date: Mon, 10 Jul 2006 05:25:33 -0400
Subject: [Python-Dev] "Missing" 2.5 feature
In-Reply-To: <1f7befae0607091544l7966feecsaaca705b3eeefaa7@mail.gmail.com>
References: <1f7befae0607081831x4ff28307j8778f552c63be95@mail.gmail.com>	<200607091705.16027.anthony@interlink.com.au>	<1f7befae0607090256h22cf00b7q1a4a6d099588d300@mail.gmail.com>
	<1f7befae0607091544l7966feecsaaca705b3eeefaa7@mail.gmail.com>
Message-ID: <44B21D0D.5090603@scottdial.com>

Tim Peters wrote:
> Just to make life harder ;-), I should note that code, docs and tests
> for sys._current_frames() are done, on the tim-current_frames branch.
> All tests pass, and there are no leaks in the new code.  It's just a
> NEWS blurb away from being just another hectic release memory :-)

Wouldn't this function be better named sys._getframes since we already 
have a sys._getframe for getting the current frame?

-- 
Scott Dial
scott at scottdial.com
scodial at indiana.edu

From 2006a at usenet.alexanderweb.de  Mon Jul 10 13:11:42 2006
From: 2006a at usenet.alexanderweb.de (Alexander Schremmer)
Date: Mon, 10 Jul 2006 13:11:42 +0200
Subject: [Python-Dev] "Missing" 2.5 feature
References: <1f7befae0607081831x4ff28307j8778f552c63be95@mail.gmail.com>
	<200607091705.16027.anthony@interlink.com.au>
	<1f7befae0607090256h22cf00b7q1a4a6d099588d300@mail.gmail.com>
	<1f7befae0607091544l7966feecsaaca705b3eeefaa7@mail.gmail.com>
	<ee2a432c0607092045n95c73k8c9f46723de07670@mail.gmail.com>
Message-ID: <hf41jagfxbu9.dlg@usenet.alexanderweb.de>

On Sun, 9 Jul 2006 20:45:05 -0700, Neal Norwitz wrote:

> There hasn't been much positive response (in the original thread or
> here).  Given you forgot about it for over a year, how important can
> it be? :-)

For me it would be very important because I often wonder where the threads
are currently working in my multithreaded apps. As a partial solution, I
wrote a "thread monitor" that tracks all thread's frames using a trace
function and generates tracebacks on demand. This slows down program
execution of course. With that function, it would be much simpler to see
the current state of the program (which is trivial in environments like the
JVM for example).

Kind regards,
Alexander


From skip at pobox.com  Mon Jul 10 14:49:39 2006
From: skip at pobox.com (skip at pobox.com)
Date: Mon, 10 Jul 2006 07:49:39 -0500
Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
In-Reply-To: <ca471dc20607092039g5cf2ea54h5225925ce877145e@mail.gmail.com>
References: <001d01c6a369$78a59950$6402a8c0@arkdesktop>
	<Pine.LNX.4.58.0607091349230.22834@server1.LFW.org>
	<44B190DF.3000805@acm.org> <44B19691.4050509@acm.org>
	<ca471dc20607092039g5cf2ea54h5225925ce877145e@mail.gmail.com>
Message-ID: <17586.19683.920761.293494@montanaro.dyndns.org>


    >> > Part of the reason why its so hard to name this feature is that
    >> > it's real name is something like "Hey, Python, you know that cool
    >> > funky thing you do with defining variables in the same scope as
    >> > they are assigned?  Well, don't do that here."

    Guido> Sorry, that sounds like a B.S. argument. You can translate most
    Guido> language constructs into long sentences if you want to.

I think Talin's got a point though.  It seems hard to find one short English
word that captures the essence of the desired behavior.  None of the words
in his list seem strongly suggestive of the meaning to me.  I suspect that
means one's ultimately as good (or as bad) as the rest.

how-about-"lookout"?-ly, y'rs,

Skip

From mattjfleming at googlemail.com  Mon Jul 10 14:50:22 2006
From: mattjfleming at googlemail.com (Matt Fleming)
Date: Mon, 10 Jul 2006 13:50:22 +0100
Subject: [Python-Dev] "Missing" 2.5 feature
In-Reply-To: <hf41jagfxbu9.dlg@usenet.alexanderweb.de>
References: <1f7befae0607081831x4ff28307j8778f552c63be95@mail.gmail.com>
	<200607091705.16027.anthony@interlink.com.au>
	<1f7befae0607090256h22cf00b7q1a4a6d099588d300@mail.gmail.com>
	<1f7befae0607091544l7966feecsaaca705b3eeefaa7@mail.gmail.com>
	<ee2a432c0607092045n95c73k8c9f46723de07670@mail.gmail.com>
	<hf41jagfxbu9.dlg@usenet.alexanderweb.de>
Message-ID: <5ff4a1e50607100550j6280929dpd27f18676945125a@mail.gmail.com>

On 10/07/06, Alexander Schremmer <2006a at usenet.alexanderweb.de> wrote:
> On Sun, 9 Jul 2006 20:45:05 -0700, Neal Norwitz wrote:
>
> > There hasn't been much positive response (in the original thread or
> > here).  Given you forgot about it for over a year, how important can
> > it be? :-)
>

I'm the SoC student working on the improvements for pdb, one of the
improvements is allowing thread debugging. I was in fact, going to use
the threadframe module if it was available on the system, having this
method in the Python core is an even better solution.

cheering-for-tim-ly yr's,
Matt


-- 
http://mattssanctuary.blogspot.com

From arigo at tunes.org  Mon Jul 10 17:13:53 2006
From: arigo at tunes.org (Armin Rigo)
Date: Mon, 10 Jul 2006 17:13:53 +0200
Subject: [Python-Dev] User's complaints
In-Reply-To: <ee2a432c0607041649i7ecb6cc1w8ebdf47f7d02097a@mail.gmail.com>
References: <ee2a432c0607041649i7ecb6cc1w8ebdf47f7d02097a@mail.gmail.com>
Message-ID: <20060710151353.GA30654@code0.codespeak.net>

Hi,

On Tue, Jul 04, 2006 at 04:49:13PM -0700, Neal Norwitz wrote:
> On 7/4/06, Guido van Rossum <guido at python.org> wrote:
> >
> >  From actual users of
> > the language I get more complaints about the breakneck speed of
> > Python's evolution than about the brokenness of the current language.

I'd like to report another (subjective) experience in favor of the
"Python is complete enough already" camp, from last year's EuroPython,
during Guido's keynote.  He announced he accepted two of the major 2.5
PEPs: the 'yield' extension, and I think the 'with' statement.  This
didn't draw much applause.  It certainly gave me the impression that
many changes in Python are advocated and welcomed by only a small
fraction of users.

I cannot be objective here, though, being myself firmly of the
impression that there are only so many syntactic features you can put in
a language before it stops being elegant and starts promoting obscure
code...

> PS.  One thing I tend to talk to users about is stability of the
> interpreter.  When I talk about crashing the interpreter, the most
> common first reaction I get is "you can crash the interpreter? How do
> you do that?"  I take that answer as a good sign. :-)

Indeed :-)  Getting some more python-dev discussions about
Lib/test/crashers/*.py would be nice too, though.


A bientot,

Armin

From theller at python.net  Mon Jul 10 18:07:50 2006
From: theller at python.net (Thomas Heller)
Date: Mon, 10 Jul 2006 18:07:50 +0200
Subject: [Python-Dev] Fix for Lib/test/leakers/test_gestalt.py
Message-ID: <e8tu0k$vbo$1@sea.gmane.org>

I think this fixes the leak in Lib/test/leakers/test_gestalt.py.

Index: Python/mactoolboxglue.c
===================================================================
--- Python/mactoolboxglue.c	(revision 50522)
+++ Python/mactoolboxglue.c	(working copy)
@@ -60,8 +60,9 @@
 			strncpy(buf, input, sizeof(buf) - 1);
 			buf[sizeof(buf) - 1] = '\0';
 		}
+		Py_DECREF(rv);
 	}
-	
+	Py_XDECREF(m);
 	return buf;
 }
 

I'm less sure about this one, but it looks like the DECREF
is also required:

Index: Python/mactoolboxglue.c
===================================================================
--- Python/mactoolboxglue.c	(revision 50522)
+++ Python/mactoolboxglue.c	(working copy)
@@ -60,8 +60,9 @@
 			strncpy(buf, input, sizeof(buf) - 1);
 			buf[sizeof(buf) - 1] = '\0';
 		}
+		Py_DECREF(rv);
 	}
-	
+	Py_XDECREF(m);
 	return buf;
 }
 

Thomas


From tim.peters at gmail.com  Mon Jul 10 18:30:30 2006
From: tim.peters at gmail.com (Tim Peters)
Date: Mon, 10 Jul 2006 12:30:30 -0400
Subject: [Python-Dev] "Missing" 2.5 feature
In-Reply-To: <ee2a432c0607092045n95c73k8c9f46723de07670@mail.gmail.com>
References: <1f7befae0607081831x4ff28307j8778f552c63be95@mail.gmail.com>
	<200607091705.16027.anthony@interlink.com.au>
	<1f7befae0607090256h22cf00b7q1a4a6d099588d300@mail.gmail.com>
	<1f7befae0607091544l7966feecsaaca705b3eeefaa7@mail.gmail.com>
	<ee2a432c0607092045n95c73k8c9f46723de07670@mail.gmail.com>
Message-ID: <1f7befae0607100930m10fb4ee3xc20d1bf9d410667c@mail.gmail.com>

[Neal Norwitz]
> There hasn't been much positive response (in the original thread or
> here).

Do note that there was little response of any kind, but all it got was
positive.  It's not sexy, but is essential for debugging deadlocks.
If you ask for positive response, you'll get some -- the use is
obvious for those who need it.  I didn't (and don't) think this needs
a "sales job".

> Given you forgot about it for over a year, how important can it be? :-)

Important enough that I stopped beating my wife to write the code ;-)

> What's the justifcation to add this feature, but none of the others?

Don't know which others you have in mind.  This is very simple code,
changes no existing behaviors, and has no potential to break other
code.  There is an extension module that tries to do the same, but
can't do so wholly safely.  It needs to be in the core to be done
correctly.  The new code, docs and tests are 100% ready to go.  How do
the others you have in mind stack up against those, point by point?

> Can we defer this to 2.6?

Of course, although waiting another ~18 months given the above seems
needlessly wasteful.

From brett at python.org  Mon Jul 10 20:17:54 2006
From: brett at python.org (Brett Cannon)
Date: Mon, 10 Jul 2006 11:17:54 -0700
Subject: [Python-Dev] Discussion on Lib/test/crashers/
Message-ID: <bbaeab100607101117r1eb8e0bbq3b2cafb751f58367@mail.gmail.com>

As I am sure some have noticed, as part of my dissertation I have been
trying to fix the various crashers.  I currently have some patches in SF for
some of the crashers.  The other ones Armin and I have been talking, while
others I have not started yet.  Review for the patches or help with fixing
the ones I have not gotten to would be great.

http://www.python.org/sf/1202533 has a potential fix for the
infinite_rec_*.py crashers.  PyObject_Call() basically needs a recursion
check.  But there were some issues with PyErr_NormalizeException() blowing
up while trying to normalize the exception, that also needs to watch out for
hitting the recursion limit and returning a RuntimeError that does not need
normalization itself.

http://www.python.org/sf/1377858 has a patch for weakref_in_del.py .  Turns
out if you create a new weakref in a __del__ for self the weakref is never
notified that self is gone and it no longer has a valid weakref to
anything.  Added a check after __del__ is called to make sure that if any
weakrefs were created that they get cleared.

http://www.python.org/sf/1303614 is a discussion currently between Armin and
I on how to handle del___dict__.py .  Looks like more careful reference
counting is needed, but Armin is worried about a performance hit.  There is
also a patch there to fix loosing_dict_ref.py .

dangerous_subclassing.py, modify_dict_attr.py, and nasty_eq_vs_dict.py I
have not gotten to yet.  I am ignoring bogus_code_obj.py, gc_inspection.py,
and recursive_call.py since they either require setting the recursion depth
higher or involve an extension module.

-Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20060710/496b5c1f/attachment.htm 

From tim.peters at gmail.com  Mon Jul 10 20:23:11 2006
From: tim.peters at gmail.com (Tim Peters)
Date: Mon, 10 Jul 2006 14:23:11 -0400
Subject: [Python-Dev] Add new PyErr_WarnEx() to 2.5?
In-Reply-To: <1f7befae0605310742y7ac1f7b3i6e9c2e5cc20affbe@mail.gmail.com>
References: <1f7befae0605310742y7ac1f7b3i6e9c2e5cc20affbe@mail.gmail.com>
Message-ID: <1f7befae0607101123x539c3b5fnf35f15ce042af51b@mail.gmail.com>

As noted in

   http://mail.python.org/pipermail/python-dev/2006-May/065478.html

it looks like we need a new Python C API function to make new warnings
from the struct module non-useless.  For example, runnning
test_zipfile on Windows now yields

"""
test_zipfile
C:\Code\python\lib\struct.py:63: DeprecationWarning: struct integer
overflow masking is deprecated
  return o.pack(*args)
"""

This is useless because the message points _into_ struct.py, not to
the code calling struct.pack().  Consequently I have no idea where the
offending pack() call is, and neither do you ;-)

The current PyErr_Warn() doesn't allow for passing a non-zero
`stacklevel` argument to warnings.warn(), so there's little the C code
in _struct.c can do to make the warning more useful.

From rhettinger at ewtllc.com  Mon Jul 10 22:11:10 2006
From: rhettinger at ewtllc.com (Raymond Hettinger)
Date: Mon, 10 Jul 2006 13:11:10 -0700
Subject: [Python-Dev] Discussion on Lib/test/crashers/
In-Reply-To: <bbaeab100607101117r1eb8e0bbq3b2cafb751f58367@mail.gmail.com>
References: <bbaeab100607101117r1eb8e0bbq3b2cafb751f58367@mail.gmail.com>
Message-ID: <44B2B45E.8030209@ewtllc.com>

Brett Cannon wrote:

> As I am sure some have noticed, as part of my dissertation I have been 
> trying to fix the various crashers.

Nice project.

One quick thought:  Any crasher that relies on gc.get_referrers() should 
not be considered a bug.  The codebase should not be convoluted, 
complexified, obfucated, altered, touched, or slowed down just prevent 
perverse, sadistic safe-cracking using this tool.  The docs are clear on 
this subject:

"Care must be taken when using objeccts returned by get_referrers() 
becasue some of them could still be under construction and hence in a 
temporarily invalid state.  Avoid using get_referrers() for any purpose 
other than debugging."



Raymond

From brett at python.org  Mon Jul 10 22:15:30 2006
From: brett at python.org (Brett Cannon)
Date: Mon, 10 Jul 2006 13:15:30 -0700
Subject: [Python-Dev] Discussion on Lib/test/crashers/
In-Reply-To: <44B2B45E.8030209@ewtllc.com>
References: <bbaeab100607101117r1eb8e0bbq3b2cafb751f58367@mail.gmail.com>
	<44B2B45E.8030209@ewtllc.com>
Message-ID: <bbaeab100607101315s7fae30b7j65a4922635e0195b@mail.gmail.com>

On 7/10/06, Raymond Hettinger <rhettinger at ewtllc.com> wrote:
>
> Brett Cannon wrote:
>
> > As I am sure some have noticed, as part of my dissertation I have been
> > trying to fix the various crashers.
>
> Nice project.
>
> One quick thought:  Any crasher that relies on gc.get_referrers() should
> not be considered a bug.


Right.  That is on the list of crashers I am ignoring.

  The codebase should not be convoluted,
> complexified, obfucated, altered, touched, or slowed down just prevent
> perverse, sadistic safe-cracking using this tool.  The docs are clear on
> this subject:
>
> "Care must be taken when using objeccts returned by get_referrers()
> becasue some of them could still be under construction and hence in a
> temporarily invalid state.  Avoid using get_referrers() for any purpose
> other than debugging."


And I am trying to make sure all the fixes are not superfluous but fix
actual issues in the code and behaviour of Python.

-Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20060710/b76f248c/attachment.htm 

From rhettinger at ewtllc.com  Mon Jul 10 22:16:06 2006
From: rhettinger at ewtllc.com (Raymond Hettinger)
Date: Mon, 10 Jul 2006 13:16:06 -0700
Subject: [Python-Dev] "Missing" 2.5 feature
In-Reply-To: <1f7befae0607100930m10fb4ee3xc20d1bf9d410667c@mail.gmail.com>
References: <1f7befae0607081831x4ff28307j8778f552c63be95@mail.gmail.com>	<200607091705.16027.anthony@interlink.com.au>	<1f7befae0607090256h22cf00b7q1a4a6d099588d300@mail.gmail.com>	<1f7befae0607091544l7966feecsaaca705b3eeefaa7@mail.gmail.com>	<ee2a432c0607092045n95c73k8c9f46723de07670@mail.gmail.com>
	<1f7befae0607100930m10fb4ee3xc20d1bf9d410667c@mail.gmail.com>
Message-ID: <44B2B586.1030102@ewtllc.com>

An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20060710/2cf7e4a5/attachment.htm 

From guido at python.org  Mon Jul 10 22:20:38 2006
From: guido at python.org (Guido van Rossum)
Date: Mon, 10 Jul 2006 13:20:38 -0700
Subject: [Python-Dev] "Missing" 2.5 feature
In-Reply-To: <44B2B586.1030102@ewtllc.com>
References: <1f7befae0607081831x4ff28307j8778f552c63be95@mail.gmail.com>
	<200607091705.16027.anthony@interlink.com.au>
	<1f7befae0607090256h22cf00b7q1a4a6d099588d300@mail.gmail.com>
	<1f7befae0607091544l7966feecsaaca705b3eeefaa7@mail.gmail.com>
	<ee2a432c0607092045n95c73k8c9f46723de07670@mail.gmail.com>
	<1f7befae0607100930m10fb4ee3xc20d1bf9d410667c@mail.gmail.com>
	<44B2B586.1030102@ewtllc.com>
Message-ID: <ca471dc20607101320j1583d18meaecf439674817ca@mail.gmail.com>

+1

On 7/10/06, Raymond Hettinger <rhettinger at ewtllc.com> wrote:
>
>  Tim Peters wrote:
>  [Neal Norwitz]
>
>
>  There hasn't been much positive response (in the original thread or
> here).
>
>  Do note that there was little response of any kind, but all it got was
> positive. It's not sexy, but is essential for debugging deadlocks.
> If you ask for positive response, you'll get some -- the use is
> obvious for those who need it. I didn't (and don't) think this needs
> a "sales job".
>
>
>
>  FWIW, I think this patch should go in.  The benefits are obvious and real.
> But, the imagined costs of a new feature during beta are illusory.
>
>  In this case, practicality beats pedantry.  For the users, it is a net win
> if this goes in.
>
>
>  Raymond
>
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/guido%40python.org
>
>
>


-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From python-dev at zesty.ca  Mon Jul 10 22:34:55 2006
From: python-dev at zesty.ca (Ka-Ping Yee)
Date: Mon, 10 Jul 2006 15:34:55 -0500 (CDT)
Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
In-Reply-To: <17586.19683.920761.293494@montanaro.dyndns.org>
References: <001d01c6a369$78a59950$6402a8c0@arkdesktop>
	<Pine.LNX.4.58.0607091349230.22834@server1.LFW.org>
	<44B190DF.3000805@acm.org> <44B19691.4050509@acm.org>
	<ca471dc20607092039g5cf2ea54h5225925ce877145e@mail.gmail.com>
	<17586.19683.920761.293494@montanaro.dyndns.org>
Message-ID: <Pine.LNX.4.58.0607101533350.22834@server1.LFW.org>

On Mon, 10 Jul 2006 skip at pobox.com wrote:
> I think Talin's got a point though.  It seems hard to find one short English
> word that captures the essence of the desired behavior.  None of the words
> in his list seem strongly suggestive of the meaning to me.  I suspect that
> means one's ultimately as good (or as bad) as the rest.

What's wrong with "nonlocal"?  I don't think i've seen an argument
against that one so far (from Talin or others).


-- ?!ng

From amk at amk.ca  Mon Jul 10 22:48:59 2006
From: amk at amk.ca (A.M. Kuchling)
Date: Mon, 10 Jul 2006 16:48:59 -0400
Subject: [Python-Dev] User's complaints
In-Reply-To: <20060710151353.GA30654@code0.codespeak.net>
References: <ee2a432c0607041649i7ecb6cc1w8ebdf47f7d02097a@mail.gmail.com>
	<20060710151353.GA30654@code0.codespeak.net>
Message-ID: <20060710204859.GA11451@rogue.amk.ca>

On Mon, Jul 10, 2006 at 05:13:53PM +0200, Armin Rigo wrote:
> didn't draw much applause.  It certainly gave me the impression that
> many changes in Python are advocated and welcomed by only a small
> fraction of users.

The benefits of changes are usually clear, but I don't think the costs
of changes are fully assessed.  python-dev considers the cost of
changes to CPython's implementation, but I don't think the cost
changes to Jython, IronPython, or PyPy are taken into account here.
PyPy probably has enough representatives here who would squawk if
something was difficult, but I don't think Jython or IronPython do.

I think if we assessed those costs fully, the bar for changes to the
language would be a good deal higher.

--amk

From jeremy at alum.mit.edu  Mon Jul 10 22:43:58 2006
From: jeremy at alum.mit.edu (Jeremy Hylton)
Date: Mon, 10 Jul 2006 16:43:58 -0400
Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
In-Reply-To: <Pine.LNX.4.58.0607101533350.22834@server1.LFW.org>
References: <001d01c6a369$78a59950$6402a8c0@arkdesktop>
	<Pine.LNX.4.58.0607091349230.22834@server1.LFW.org>
	<44B190DF.3000805@acm.org> <44B19691.4050509@acm.org>
	<ca471dc20607092039g5cf2ea54h5225925ce877145e@mail.gmail.com>
	<17586.19683.920761.293494@montanaro.dyndns.org>
	<Pine.LNX.4.58.0607101533350.22834@server1.LFW.org>
Message-ID: <e8bf7a530607101343g77148a93o39d0f671adb99e4a@mail.gmail.com>

On 7/10/06, Ka-Ping Yee <python-dev at zesty.ca> wrote:
> On Mon, 10 Jul 2006 skip at pobox.com wrote:
> > I think Talin's got a point though.  It seems hard to find one short English
> > word that captures the essence of the desired behavior.  None of the words
> > in his list seem strongly suggestive of the meaning to me.  I suspect that
> > means one's ultimately as good (or as bad) as the rest.
>
> What's wrong with "nonlocal"?  I don't think i've seen an argument
> against that one so far (from Talin or others).

It's a made-up word.  You won't find it in the dictionary and the
google define: query sends me to a wikipedia page about quantum
mechanics.  It also expresses itself in the negative form "not local"
as opposed to the positive form like global "this is a global."
Finally, I think it sounds yucky.

To express this email in the positive form:
1. Reserved words should be real words.
2. The meaning of the word should be clear.
3. "Put statements in positive form."  (Strunk & White)
4. The word should sound good.

global meets all of these requirements.  "free" was the word I
remember preferring from earlier discussions, but I think it fails #2.
 (Too much confusion about freeing memory, for example.)

Jeremy

>
>
> -- ?!ng
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/jeremy%40alum.mit.edu
>

From anthony at interlink.com.au  Mon Jul 10 22:45:02 2006
From: anthony at interlink.com.au (Anthony Baxter)
Date: Tue, 11 Jul 2006 06:45:02 +1000
Subject: [Python-Dev] "Missing" 2.5 feature
In-Reply-To: <44B2B586.1030102@ewtllc.com>
References: <1f7befae0607081831x4ff28307j8778f552c63be95@mail.gmail.com>
	<1f7befae0607100930m10fb4ee3xc20d1bf9d410667c@mail.gmail.com>
	<44B2B586.1030102@ewtllc.com>
Message-ID: <200607110645.07875.anthony@interlink.com.au>

On Tuesday 11 July 2006 06:16, Raymond Hettinger wrote:
> FWIW, I think this patch should go in. The benefits are
> obvious and real.

Yep. I'm going to check it in, unless someone else beats me to it in 
the next couple of hours before the b2 freeze.

> But, the imagined costs of a new feature 
> during beta are illusory.

This, I cannot agree with. The costs and risks of just continuing to 
add new features all through the release process are high. The 
chances of us releasing a broken Python increases dramatically. Then 
we have to do emergency bugfix releases. And I'm sorry, but the 
release process is not something that's zero work. Sure, it's zero 
work _for_ _you_ <wink>, but not for Martin, Fred and myself.


> In this case, practicality beats pedantry.

I don't think trying to produce the most stable and bugfree Python 
possible could in _anyway_ be considered "pedantry", and it makes me 
quite grumpy to have it described in that way.


> For the users, it 
> is a net win if this goes in.

In the case of this feature, that's true.

Anthony
-- 
Anthony Baxter     <anthony at interlink.com.au>
It's never too late to have a happy childhood.

From mbk.lists at gmail.com  Mon Jul 10 22:51:58 2006
From: mbk.lists at gmail.com (Mike Krell)
Date: Mon, 10 Jul 2006 13:51:58 -0700
Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
In-Reply-To: <Pine.LNX.4.58.0607101533350.22834@server1.LFW.org>
References: <001d01c6a369$78a59950$6402a8c0@arkdesktop>
	<Pine.LNX.4.58.0607091349230.22834@server1.LFW.org>
	<44B190DF.3000805@acm.org> <44B19691.4050509@acm.org>
	<ca471dc20607092039g5cf2ea54h5225925ce877145e@mail.gmail.com>
	<17586.19683.920761.293494@montanaro.dyndns.org>
	<Pine.LNX.4.58.0607101533350.22834@server1.LFW.org>
Message-ID: <da7032ce0607101351p159b4333mac0f566b8d0e500@mail.gmail.com>

> What's wrong with "nonlocal"?  I don't think i've seen an argument
> against that one so far (from Talin or others).

It sounds a bit awkward to me.  Also, it would be nice if the keyword
indicated which scope was operative.

If I've followed the discussions correctly, I think the parent scope
would be operative, so I humbly suggest "parent".

   Mike

From tim.peters at gmail.com  Mon Jul 10 22:52:30 2006
From: tim.peters at gmail.com (Tim Peters)
Date: Mon, 10 Jul 2006 16:52:30 -0400
Subject: [Python-Dev] "Missing" 2.5 feature
In-Reply-To: <200607110645.07875.anthony@interlink.com.au>
References: <1f7befae0607081831x4ff28307j8778f552c63be95@mail.gmail.com>
	<1f7befae0607100930m10fb4ee3xc20d1bf9d410667c@mail.gmail.com>
	<44B2B586.1030102@ewtllc.com>
	<200607110645.07875.anthony@interlink.com.au>
Message-ID: <1f7befae0607101352j6ff60371m988dae24266740@mail.gmail.com>

[Raymond]
>> FWIW, I think this patch should go in. The benefits are
>> obvious and real.

[Anthony Baxter]
> Yep. I'm going to check it in, unless someone else beats me to it in
> the next couple of hours before the b2 freeze.

I'll merge it from my branch right after I send this email.  It still
needs a NEWS blurb, which I'll write.

Thanks!  You made the right decision :-)

>> But, the imagined costs of a new feature during beta are illusory.

> This, I cannot agree with. The costs and risks of just continuing to
> add new features all through the release process are high. The
> chances of us releasing a broken Python increases dramatically. Then
> we have to do emergency bugfix releases. And I'm sorry, but the
> release process is not something that's zero work. Sure, it's zero
> work _for_ _you_ <wink>, but not for Martin, Fred and myself.

I agree.  For example, even here (i.e., a pure, simple "new feature"),
as I noted from the start, it's possible that my doc changes broke the
LaTeX somehow, and that would be disruptive.

>> In this case, practicality beats pedantry.

> I don't think trying to produce the most stable and bugfree Python
> possible could in _anyway_ be considered "pedantry", and it makes me
> quite grumpy to have it described in that way.

He meant that "no new features", while a useful guideline, can be
counterproductive if followed slavishly.

>> For the users, it is a net win if this goes in.

> In the case of this feature, that's true.

More to the point, at this stage in the release process this specific
little new feature carries almost no risk of loss for anyone.

From anthony at interlink.com.au  Mon Jul 10 23:02:02 2006
From: anthony at interlink.com.au (Anthony Baxter)
Date: Tue, 11 Jul 2006 07:02:02 +1000
Subject: [Python-Dev] "Missing" 2.5 feature
In-Reply-To: <1f7befae0607101352j6ff60371m988dae24266740@mail.gmail.com>
References: <1f7befae0607081831x4ff28307j8778f552c63be95@mail.gmail.com>
	<200607110645.07875.anthony@interlink.com.au>
	<1f7befae0607101352j6ff60371m988dae24266740@mail.gmail.com>
Message-ID: <200607110702.04021.anthony@interlink.com.au>

On Tuesday 11 July 2006 06:52, Tim Peters wrote:
> > I don't think trying to produce the most stable and bugfree
> > Python possible could in _anyway_ be considered "pedantry", and
> > it makes me quite grumpy to have it described in that way.
>
> He meant that "no new features", while a useful guideline, can be
> counterproductive if followed slavishly.

I'm not taking a slavish "no new features" line. I _am_ saying that 
any new features, post beta, require a good justification and a clear 
understanding of the risks that are added by the new code. In this 
case, the tradeoff is fine. Simply saying code is very low risk isn't 
enough - there also has to be a positive reason for the code going 
in. The ability to debug deadlocks is a good thing, and the clincher 
(once I sat and thought about it a bit) is that there is _already_ a 
module out there that attempts to do this, albeit in a buggy fashion. 
This is pretty clear indication that there is a demand for the 
feature.

Similarly, the PyErr_WarnEx() is _probably_ a good thing to add in, 
because otherwise we can't do anything about the struct warning. But 
that really will have to wait until post-beta2 at this point. 

Anthony
-- 
Anthony Baxter     <anthony at interlink.com.au>
It's never too late to have a happy childhood.

From brett at python.org  Mon Jul 10 23:10:07 2006
From: brett at python.org (Brett Cannon)
Date: Mon, 10 Jul 2006 14:10:07 -0700
Subject: [Python-Dev] Klocwork analysis of source if we want it
Message-ID: <bbaeab100607101410o170186ffy800d54de8d969d7b@mail.gmail.com>

http://www.klocwork.com/company/releases/06_26_06.asp

Looks like Klocowork is doing the same thing as Coverity and providing free
static analysis of source for open source projects.  Doubt we want this
*and* Coverity, but figured wouldn't hurt to let people know about it.

-Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20060710/7d11306d/attachment.html 

From rhettinger at ewtllc.com  Mon Jul 10 23:12:32 2006
From: rhettinger at ewtllc.com (Raymond Hettinger)
Date: Mon, 10 Jul 2006 14:12:32 -0700
Subject: [Python-Dev] "Missing" 2.5 feature
In-Reply-To: <200607110645.07875.anthony@interlink.com.au>
References: <1f7befae0607081831x4ff28307j8778f552c63be95@mail.gmail.com>
	<1f7befae0607100930m10fb4ee3xc20d1bf9d410667c@mail.gmail.com>
	<44B2B586.1030102@ewtllc.com>
	<200607110645.07875.anthony@interlink.com.au>
Message-ID: <44B2C2C0.3020305@ewtllc.com>

An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20060710/e8ba99fd/attachment.htm 

From theller at python.net  Mon Jul 10 23:51:28 2006
From: theller at python.net (Thomas Heller)
Date: Mon, 10 Jul 2006 23:51:28 +0200
Subject: [Python-Dev] Fix for Lib/test/leakers/test_gestalt.py
In-Reply-To: <e8tu0k$vbo$1@sea.gmane.org>
References: <e8tu0k$vbo$1@sea.gmane.org>
Message-ID: <e8ui4t$7t4$1@sea.gmane.org>

Thomas Heller schrieb:
> I think this fixes the leak in Lib/test/leakers/test_gestalt.py.
> 
> Index: Python/mactoolboxglue.c
> ===================================================================
> --- Python/mactoolboxglue.c	(revision 50522)
> +++ Python/mactoolboxglue.c	(working copy)
> @@ -60,8 +60,9 @@
>  			strncpy(buf, input, sizeof(buf) - 1);
>  			buf[sizeof(buf) - 1] = '\0';
>  		}
> +		Py_DECREF(rv);
>  	}
> -	
> +	Py_XDECREF(m);
>  	return buf;
>  }
>  
> 
> I'm less sure about this one, but it looks like the DECREF
> is also required:

(Here's the patch that I *really* meant)

Index: Mac/Modules/macosmodule.c
===================================================================
--- Mac/Modules/macosmodule.c	(Revision 50515)
+++ Mac/Modules/macosmodule.c	(Arbeitskopie)
@@ -375,6 +375,7 @@
 				/* And try again... */
 				h = GetResource('Estr', err);
 			}
+			Py_DECREF(m);
 		}
 	}
 	/*

Thomas


From python-dev at zesty.ca  Mon Jul 10 23:53:01 2006
From: python-dev at zesty.ca (Ka-Ping Yee)
Date: Mon, 10 Jul 2006 16:53:01 -0500 (CDT)
Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
In-Reply-To: <e8bf7a530607101343g77148a93o39d0f671adb99e4a@mail.gmail.com>
References: <001d01c6a369$78a59950$6402a8c0@arkdesktop> 
	<Pine.LNX.4.58.0607091349230.22834@server1.LFW.org>
	<44B190DF.3000805@acm.org> <44B19691.4050509@acm.org>
	<ca471dc20607092039g5cf2ea54h5225925ce877145e@mail.gmail.com>
	<17586.19683.920761.293494@montanaro.dyndns.org> 
	<Pine.LNX.4.58.0607101533350.22834@server1.LFW.org>
	<e8bf7a530607101343g77148a93o39d0f671adb99e4a@mail.gmail.com>
Message-ID: <Pine.LNX.4.58.0607101638090.22834@server1.LFW.org>

On 7/10/06, Ka-Ping Yee <python-dev at zesty.ca> wrote:
> What's wrong with "nonlocal"?  I don't think i've seen an argument
> against that one so far (from Talin or others).

On Mon, 10 Jul 2006, Jeremy Hylton wrote:
> It's a made-up word.  You won't find it in the dictionary and the
> google define: query sends me to a wikipedia page about quantum
> mechanics.

Two million Google hits for "nonlocal" seems like plenty.
The explanation of "nonlocal" is pretty straightforward --

    If the definition of f() contains an assignment to x, then
    x is a local variable in f, unless x is declared nonlocal.

> To express this email in the positive form:
> 1. Reserved words should be real words.
> 2. The meaning of the word should be clear.
> 3. "Put statements in positive form."  (Strunk & White)
> 4. The word should sound good.
>
> global meets all of these requirements.

But it's the wrong word.  "Purple" also meets all of these requirements.

I'd rather accurately express the concept in the negative (the
true meaning really is in the negative: "don't make a new binding"),
than choose a simple-sounding word that is essentially a lie.

    x = 1

    def f(x):
        print x

        def g():
            global x
            x = 3
            print x

The "x" used in g is not global at all -- it belongs to f.


-- ?!ng

From guido at python.org  Tue Jul 11 00:24:29 2006
From: guido at python.org (Guido van Rossum)
Date: Mon, 10 Jul 2006 15:24:29 -0700
Subject: [Python-Dev] User's complaints
In-Reply-To: <20060710204859.GA11451@rogue.amk.ca>
References: <ee2a432c0607041649i7ecb6cc1w8ebdf47f7d02097a@mail.gmail.com>
	<20060710151353.GA30654@code0.codespeak.net>
	<20060710204859.GA11451@rogue.amk.ca>
Message-ID: <ca471dc20607101524k1bcb708fne8a73081af556cdc@mail.gmail.com>

Not to mention the cost to documentation and books everywhere --
updating our own docs is only the tip of the iceberg. And then
updating the users' brains is an even bigger job... (Though at least
it is highly parallellizable. :-)

--Guido

On 7/10/06, A.M. Kuchling <amk at amk.ca> wrote:
> On Mon, Jul 10, 2006 at 05:13:53PM +0200, Armin Rigo wrote:
> > didn't draw much applause.  It certainly gave me the impression that
> > many changes in Python are advocated and welcomed by only a small
> > fraction of users.
>
> The benefits of changes are usually clear, but I don't think the costs
> of changes are fully assessed.  python-dev considers the cost of
> changes to CPython's implementation, but I don't think the cost
> changes to Jython, IronPython, or PyPy are taken into account here.
> PyPy probably has enough representatives here who would squawk if
> something was difficult, but I don't think Jython or IronPython do.
>
> I think if we assessed those costs fully, the bar for changes to the
> language would be a good deal higher.
>
> --amk
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org
>


-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From almann.goo at gmail.com  Tue Jul 11 01:11:27 2006
From: almann.goo at gmail.com (Almann T. Goo)
Date: Mon, 10 Jul 2006 19:11:27 -0400
Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
In-Reply-To: <e8bf7a530607101343g77148a93o39d0f671adb99e4a@mail.gmail.com>
References: <001d01c6a369$78a59950$6402a8c0@arkdesktop>
	<Pine.LNX.4.58.0607091349230.22834@server1.LFW.org>
	<44B190DF.3000805@acm.org> <44B19691.4050509@acm.org>
	<ca471dc20607092039g5cf2ea54h5225925ce877145e@mail.gmail.com>
	<17586.19683.920761.293494@montanaro.dyndns.org>
	<Pine.LNX.4.58.0607101533350.22834@server1.LFW.org>
	<e8bf7a530607101343g77148a93o39d0f671adb99e4a@mail.gmail.com>
Message-ID: <7e9b97090607101611k6e1718bbi13edf82b865e8406@mail.gmail.com>

On 7/10/06, Jeremy Hylton <jeremy at alum.mit.edu> wrote:
>
> On 7/10/06, Ka-Ping Yee <python-dev at zesty.ca> wrote:
> > On Mon, 10 Jul 2006 skip at pobox.com wrote:
> > > I think Talin's got a point though.  It seems hard to find one short
> English
> > > word that captures the essence of the desired behavior.  None of the
> words
> > > in his list seem strongly suggestive of the meaning to me.  I suspect
> that
> > > means one's ultimately as good (or as bad) as the rest.
> >
> > What's wrong with "nonlocal"?  I don't think i've seen an argument
> > against that one so far (from Talin or others).
>
> It's a made-up word.  You won't find it in the dictionary and the
> google define: query sends me to a wikipedia page about quantum
> mechanics.  It also expresses itself in the negative form "not local"
> as opposed to the positive form like global "this is a global."
> Finally, I think it sounds yucky.
>
> To express this email in the positive form:
> 1. Reserved words should be real words.
> 2. The meaning of the word should be clear.
> 3. "Put statements in positive form."  (Strunk & White)
> 4. The word should sound good.
>
> global meets all of these requirements.  "free" was the word I
> remember preferring from earlier discussions, but I think it fails #2.
> (Too much confusion about freeing memory, for example.)



I remember previous discussions also referring to spelling this as "outer"
which IMO passes #2 as well as the other, although arguably #4 is subjective
;-).

-Almann

-- 
Almann T. Goo
almann.goo at gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20060710/5d24e320/attachment.html 

From greg.ewing at canterbury.ac.nz  Tue Jul 11 01:32:22 2006
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Tue, 11 Jul 2006 11:32:22 +1200
Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
In-Reply-To: <ca471dc20607092039g5cf2ea54h5225925ce877145e@mail.gmail.com>
References: <001d01c6a369$78a59950$6402a8c0@arkdesktop>
	<Pine.LNX.4.58.0607091349230.22834@server1.LFW.org>
	<44B190DF.3000805@acm.org> <44B19691.4050509@acm.org>
	<ca471dc20607092039g5cf2ea54h5225925ce877145e@mail.gmail.com>
Message-ID: <44B2E386.50101@canterbury.ac.nz>

Guido van Rossum wrote:

> Then let's allow
> 
>   nonlocal x = 12
> 
> as a shortcut for
> 
>   nonlocal x
>   x = 12

I thought you didn't like that, because in

   nonlocal x = 12
   x = 42

it's not clear whether these are talking about the same
x or not.

--
Greg

From greg.ewing at canterbury.ac.nz  Tue Jul 11 01:50:35 2006
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Tue, 11 Jul 2006 11:50:35 +1200
Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
In-Reply-To: <da7032ce0607101351p159b4333mac0f566b8d0e500@mail.gmail.com>
References: <001d01c6a369$78a59950$6402a8c0@arkdesktop>
	<Pine.LNX.4.58.0607091349230.22834@server1.LFW.org>
	<44B190DF.3000805@acm.org> <44B19691.4050509@acm.org>
	<ca471dc20607092039g5cf2ea54h5225925ce877145e@mail.gmail.com>
	<17586.19683.920761.293494@montanaro.dyndns.org>
	<Pine.LNX.4.58.0607101533350.22834@server1.LFW.org>
	<da7032ce0607101351p159b4333mac0f566b8d0e500@mail.gmail.com>
Message-ID: <44B2E7CB.7040405@canterbury.ac.nz>

Mike Krell wrote:

> If I've followed the discussions correctly, I think the parent scope
> would be operative, so I humbly suggest "parent".

-1, this is far too commonly used as a variable name.

--
Greg

From tim.peters at gmail.com  Tue Jul 11 02:46:16 2006
From: tim.peters at gmail.com (Tim Peters)
Date: Mon, 10 Jul 2006 20:46:16 -0400
Subject: [Python-Dev] "Missing" 2.5 feature
In-Reply-To: <44B21D0D.5090603@scottdial.com>
References: <1f7befae0607081831x4ff28307j8778f552c63be95@mail.gmail.com>
	<200607091705.16027.anthony@interlink.com.au>
	<1f7befae0607090256h22cf00b7q1a4a6d099588d300@mail.gmail.com>
	<1f7befae0607091544l7966feecsaaca705b3eeefaa7@mail.gmail.com>
	<44B21D0D.5090603@scottdial.com>
Message-ID: <1f7befae0607101746x14f52816jb64dc1f7abfdf900@mail.gmail.com>

[Scott Dial]
> Wouldn't this function be better named sys._getframes since we already
> have a sys._getframe for getting the current frame?

    http://mail.python.org/pipermail/python-dev/2005-March/051887.html

The first & only name suggested won.  As it says there, I usually have
no appetite for naming arguments.

From barry at python.org  Tue Jul 11 03:52:34 2006
From: barry at python.org (Barry Warsaw)
Date: Mon, 10 Jul 2006 21:52:34 -0400
Subject: [Python-Dev] Support for PyGetSetDefs in pydoc
Message-ID: <A027CA35-EE77-4658-BA59-E849368C440B@python.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Patch #1520294 adds support for attributes defined with PyGetSetDef  
in extension modules to pydoc, specifically so things like help 
(array.array.typecode) gives something useful, like the attribute's  
docstring for instance.

Along the way, I added types.GetSetterType and inspect.isgetsetter(),  
along with tests and docs.  Now I definitely think that the pydoc  
change should go into Python 2.5 and be backported to Python 2.4, but  
it's a bit of a question whether the types.py and inspect.py changes  
should go into Python 2.5 now that we're in beta.

I personal think they're worthwhile changes to go it, but I won't  
argue too hard for them since the pydoc fixes can be implemented with  
local changes to pydoc.py alone.

Thoughts?
- -Barry

https://sourceforge.net/tracker/? 
func=detail&atid=305470&aid=1520294&group_id=5470

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.3 (Darwin)

iQCVAwUBRLMEYnEjvBPtnXfVAQKB8gQAmkk0V0B0xdRD/Xgko4vD0JmvFiXZL3EO
bEzE9Wtn8u54JB/EXLdocjeFLjlOzlSzalLKGvxgQHyKGP0F8CSBN6mIreTsztct
wl58FyPro3mkwsamUrkPzf/xSZk/4tO81oXPkuzqANbv+0WEnD5MmdSpVqz7jQIo
qqbAqah2ha4=
=kxSp
-----END PGP SIGNATURE-----

From jeremy at alum.mit.edu  Tue Jul 11 04:54:22 2006
From: jeremy at alum.mit.edu (Jeremy Hylton)
Date: Mon, 10 Jul 2006 22:54:22 -0400
Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
In-Reply-To: <Pine.LNX.4.58.0607101638090.22834@server1.LFW.org>
References: <001d01c6a369$78a59950$6402a8c0@arkdesktop>
	<Pine.LNX.4.58.0607091349230.22834@server1.LFW.org>
	<44B190DF.3000805@acm.org> <44B19691.4050509@acm.org>
	<ca471dc20607092039g5cf2ea54h5225925ce877145e@mail.gmail.com>
	<17586.19683.920761.293494@montanaro.dyndns.org>
	<Pine.LNX.4.58.0607101533350.22834@server1.LFW.org>
	<e8bf7a530607101343g77148a93o39d0f671adb99e4a@mail.gmail.com>
	<Pine.LNX.4.58.0607101638090.22834@server1.LFW.org>
Message-ID: <e8bf7a530607101954s7e06707wd6509b048963f6c9@mail.gmail.com>

On 7/10/06, Ka-Ping Yee <python-dev at zesty.ca> wrote:
> On 7/10/06, Ka-Ping Yee <python-dev at zesty.ca> wrote:
> > What's wrong with "nonlocal"?  I don't think i've seen an argument
> > against that one so far (from Talin or others).
>
> On Mon, 10 Jul 2006, Jeremy Hylton wrote:
> > It's a made-up word.  You won't find it in the dictionary and the
> > google define: query sends me to a wikipedia page about quantum
> > mechanics.
>
> Two million Google hits for "nonlocal" seems like plenty.
> The explanation of "nonlocal" is pretty straightforward --
>
>     If the definition of f() contains an assignment to x, then
>     x is a local variable in f, unless x is declared nonlocal.
>
> > To express this email in the positive form:
> > 1. Reserved words should be real words.
> > 2. The meaning of the word should be clear.
> > 3. "Put statements in positive form."  (Strunk & White)
> > 4. The word should sound good.
> >
> > global meets all of these requirements.
>
> But it's the wrong word.  "Purple" also meets all of these requirements.

No.  Don't be silly.

The current use of global meets the requirements.  The meaning there
is clear, because global means global namespace.  If purple were a
keyword, I wouldn't know what it means.  I was not proposing the use
of global for some other meaning (and thought I made that clear in the
remainder of the message).

Jeremy

>
> I'd rather accurately express the concept in the negative (the
> true meaning really is in the negative: "don't make a new binding"),
> than choose a simple-sounding word that is essentially a lie.
>
>     x = 1
>
>     def f(x):
>         print x
>
>         def g():
>             global x
>             x = 3
>             print x
>
> The "x" used in g is not global at all -- it belongs to f.
>
>
> -- ?!ng
>

From mbarnes at redhat.com  Tue Jul 11 05:52:28 2006
From: mbarnes at redhat.com (Matthew Barnes)
Date: Mon, 10 Jul 2006 23:52:28 -0400
Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
In-Reply-To: <e8bf7a530607101343g77148a93o39d0f671adb99e4a@mail.gmail.com>
References: <001d01c6a369$78a59950$6402a8c0@arkdesktop>
	<Pine.LNX.4.58.0607091349230.22834@server1.LFW.org>
	<44B190DF.3000805@acm.org> <44B19691.4050509@acm.org>
	<ca471dc20607092039g5cf2ea54h5225925ce877145e@mail.gmail.com>
	<17586.19683.920761.293494@montanaro.dyndns.org>
	<Pine.LNX.4.58.0607101533350.22834@server1.LFW.org>
	<e8bf7a530607101343g77148a93o39d0f671adb99e4a@mail.gmail.com>
Message-ID: <1152589949.14276.16.camel@localhost.localdomain>

On Mon, 2006-07-10 at 16:43 -0400, Jeremy Hylton wrote:
> To express this email in the positive form:
> 1. Reserved words should be real words.
> 2. The meaning of the word should be clear.
> 3. "Put statements in positive form."  (Strunk & White)
> 4. The word should sound good.

As I've been following this thread I find that the word "extern" keeps
coming to mind.  It's debatable whether "extern" passes #1, although my
dictionary has an entry for it.  But more importantly, there seems to be
fairly strong parallels between what we're discussing here and its
meaning in C/C++ (i.e. the symbol is defined outside of the current
scope).  So I think "extern" easily passes #2 for C/C++ programmers, and
I think others can probably guess that extern == external (my wife did,
at least).

I haven't seen this suggested yet so I thought I'd just throw it out
there.

Matt


From python-dev at zesty.ca  Tue Jul 11 05:58:32 2006
From: python-dev at zesty.ca (Ka-Ping Yee)
Date: Mon, 10 Jul 2006 22:58:32 -0500 (CDT)
Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
In-Reply-To: <e8bf7a530607101954s7e06707wd6509b048963f6c9@mail.gmail.com>
References: <001d01c6a369$78a59950$6402a8c0@arkdesktop> 
	<Pine.LNX.4.58.0607091349230.22834@server1.LFW.org>
	<44B190DF.3000805@acm.org> <44B19691.4050509@acm.org>
	<ca471dc20607092039g5cf2ea54h5225925ce877145e@mail.gmail.com>
	<17586.19683.920761.293494@montanaro.dyndns.org> 
	<Pine.LNX.4.58.0607101533350.22834@server1.LFW.org> 
	<e8bf7a530607101343g77148a93o39d0f671adb99e4a@mail.gmail.com> 
	<Pine.LNX.4.58.0607101638090.22834@server1.LFW.org>
	<e8bf7a530607101954s7e06707wd6509b048963f6c9@mail.gmail.com>
Message-ID: <Pine.LNX.4.58.0607102257490.22834@server1.LFW.org>

On Mon, 10 Jul 2006, Jeremy Hylton wrote:
> The current use of global meets the requirements.  The meaning there
> is clear, because global means global namespace.  If purple were a
> keyword, I wouldn't know what it means.  I was not proposing the use
> of global for some other meaning (and thought I made that clear in the
> remainder of the message).

My apologies.  I misunderstood you -- i thought you were suggesting
it as an alternative to "nonlocal".


-- ?!ng

From benji at zope.com  Tue Jul 11 05:36:07 2006
From: benji at zope.com (Benji York)
Date: Mon, 10 Jul 2006 23:36:07 -0400
Subject: [Python-Dev] Doctest and Footnotes
Message-ID: <44B31CA7.7080906@zope.com>

A coworker of mine (Gary Poster) had a really good idea a couple weeks 
ago: teach doctest about ReST-style footnotes.  I implemented it over 
the weekend and brought it to Tim Peter's attention today.  Tim 
generally liked the idea and suggested I bring it up here.

Here's the idea: when a footnote is referenced in prose, execute the 
code associated with the footnote at that point.  For example:

"""
After initializing the system [#init]_ it is possible to retrieve status 
information:

     >>> system.status()
     'good to go'

[snip some of the doctest]

.. [#init] Initialize the system:
     >>> system = System()
     >>> system.init()
"""

The footnote code is executed every time the footnote is referenced, and 
is /not/ executed at any other time (i.e. it's not executed at the point 
the footnote is defined).  A warning is generated if a footnote (that 
includes code) is defined but never used.

This would allow moving repetitive or verbose code (e.g. tests for 
corner cases) into footnotes so they won't hinder the documentation 
aspect of a test.  It also allows defining reusable bits of setup code, 
freeing the doctest author to structure the prose as they wish instead 
of being constrained by having to place bits of code with common 
environmental needs together.

I've implemented this in a branch of zope.testing (which contains a copy 
of a post-Python 2.4.3 version of doctest (http://tinyurl.com/nekam). 
The behavior is controlled by an optionflag, much like ELLIPSIS or 
NORMALIZE_WHITESPACE.  Tim has given me a few pointers on improvements 
to make, which I'll work on this week.

Thoughts/questions?
-- 
Benji York
Senior Software Engineer
Zope Corporation

From skip at pobox.com  Tue Jul 11 06:44:25 2006
From: skip at pobox.com (skip at pobox.com)
Date: Mon, 10 Jul 2006 23:44:25 -0500
Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
In-Reply-To: <da7032ce0607101351p159b4333mac0f566b8d0e500@mail.gmail.com>
References: <001d01c6a369$78a59950$6402a8c0@arkdesktop>
	<Pine.LNX.4.58.0607091349230.22834@server1.LFW.org>
	<44B190DF.3000805@acm.org> <44B19691.4050509@acm.org>
	<ca471dc20607092039g5cf2ea54h5225925ce877145e@mail.gmail.com>
	<17586.19683.920761.293494@montanaro.dyndns.org>
	<Pine.LNX.4.58.0607101533350.22834@server1.LFW.org>
	<da7032ce0607101351p159b4333mac0f566b8d0e500@mail.gmail.com>
Message-ID: <17587.11433.405626.356611@montanaro.dyndns.org>


    >> What's wrong with "nonlocal"?  I don't think i've seen an argument
    >> against that one so far (from Talin or others).

    Mike> It sounds a bit awkward to me.  Also, it would be nice if the
    Mike> keyword indicated which scope was operative.

Sounds awkward to me as well.  Couldn't put my finger on it until seeing
Jeremy's post though.  I don't think the keyword should indicate a scope.
I'd prefer it if LOAD_<WHATEVER> just percolated its way up the chain of
cells (or could be identified at compile time by inspecting the AST as I
think Guido intends) without the programmer having to name the binding
scope.

Maybe if it names the function containing the variable to bind.  Nothing
static (like "up 3 levels") though.

    Mike> If I've followed the discussions correctly, I think the parent
    Mike> scope would be operative, so I humbly suggest "parent".

Since it might not just be in the immediate parent scope, how about
"ancestor"? <0.5 wink>

Skip


From pje at telecommunity.com  Tue Jul 11 06:47:21 2006
From: pje at telecommunity.com (Phillip J. Eby)
Date: Tue, 11 Jul 2006 00:47:21 -0400
Subject: [Python-Dev] Doctest and Footnotes
In-Reply-To: <44B31CA7.7080906@zope.com>
Message-ID: <5.1.1.6.0.20060711004320.01eed7d0@sparrow.telecommunity.com>

At 11:36 PM 7/10/2006 -0400, Benji York wrote:
>The footnote code is executed every time the footnote is referenced, and
>is /not/ executed at any other time (i.e. it's not executed at the point
>the footnote is defined).  A warning is generated if a footnote (that
>includes code) is defined but never used.
>
>This would allow moving repetitive or verbose code (e.g. tests for
>corner cases) into footnotes so they won't hinder the documentation
>aspect of a test.  It also allows defining reusable bits of setup code,
>freeing the doctest author to structure the prose as they wish instead
>of being constrained by having to place bits of code with common
>environmental needs together.
>
>I've implemented this in a branch of zope.testing (which contains a copy
>of a post-Python 2.4.3 version of doctest (http://tinyurl.com/nekam).
>The behavior is controlled by an optionflag, much like ELLIPSIS or
>NORMALIZE_WHITESPACE.  Tim has given me a few pointers on improvements
>to make, which I'll work on this week.
>
>Thoughts/questions?

It would be nice if tracebacks in the footnote show the invoking context, 
so that if a footnote is invoked from more than one place, you'll be able 
to tell which one from any errors that occur.  I haven't looked at the 
code, so perhaps you've already done this.

My other thought would be that having a patch that works against the 2.5 
version of doctest would be good, as 2.5 has fixes to make doctest work 
with zipped (or egged) doctest files.

Apart from those two comments, it sounds like an interesting idea.


From mbk.lists at gmail.com  Tue Jul 11 07:51:32 2006
From: mbk.lists at gmail.com (Mike Krell)
Date: Mon, 10 Jul 2006 22:51:32 -0700
Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
In-Reply-To: <17587.11433.405626.356611@montanaro.dyndns.org>
References: <001d01c6a369$78a59950$6402a8c0@arkdesktop>
	<Pine.LNX.4.58.0607091349230.22834@server1.LFW.org>
	<44B190DF.3000805@acm.org> <44B19691.4050509@acm.org>
	<ca471dc20607092039g5cf2ea54h5225925ce877145e@mail.gmail.com>
	<17586.19683.920761.293494@montanaro.dyndns.org>
	<Pine.LNX.4.58.0607101533350.22834@server1.LFW.org>
	<da7032ce0607101351p159b4333mac0f566b8d0e500@mail.gmail.com>
	<17587.11433.405626.356611@montanaro.dyndns.org>
Message-ID: <da7032ce0607102251i33e6dcf8ic40fbfbc0e362f0f@mail.gmail.com>

On 7/10/06, skip at pobox.com <skip at pobox.com> wrote:

> I don't think the keyword should indicate a scope.
> I'd prefer it if LOAD_<WHATEVER> just percolated its way up the chain of
> cells (or could be identified at compile time by inspecting the AST as I
> think Guido intends) without the programmer having to name the binding
> scope.

I agree completely.  I realize that probably wasn't clear when I said
"it would be nice if the keyword indicated which scope was operative".
 I was really only trying to proffer a keyword which would hopefully
suggest to a newbie that they should percolate up the chain of scopes.
 In this sense, the word "outer" works for me, but I'm sympathetic to
Andrew Koenig's argument that "outer" is confusing because the
innermost binding is actually affected.

> Since it might not just be in the immediate parent scope, how about
> "ancestor"? <0.5 wink>

That thought had occurred to me, but then I beat it down with a stick :-)

My rationale was that the affected binding is the one seen by the
parent, even if the parent did not create the binding itself.

Greg Ewing's point about parent being a common variable name is
well-taken, though.

   Mike

From talin at acm.org  Tue Jul 11 08:05:19 2006
From: talin at acm.org (Talin)
Date: Mon, 10 Jul 2006 23:05:19 -0700
Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
In-Reply-To: <Pine.LNX.4.58.0607101533350.22834@server1.LFW.org>
References: <001d01c6a369$78a59950$6402a8c0@arkdesktop>	<Pine.LNX.4.58.0607091349230.22834@server1.LFW.org>	<44B190DF.3000805@acm.org>
	<44B19691.4050509@acm.org>	<ca471dc20607092039g5cf2ea54h5225925ce877145e@mail.gmail.com>	<17586.19683.920761.293494@montanaro.dyndns.org>
	<Pine.LNX.4.58.0607101533350.22834@server1.LFW.org>
Message-ID: <44B33F9F.2000804@acm.org>

Ka-Ping Yee wrote:
> On Mon, 10 Jul 2006 skip at pobox.com wrote:
> 
>>I think Talin's got a point though.  It seems hard to find one short English
>>word that captures the essence of the desired behavior.  None of the words
>>in his list seem strongly suggestive of the meaning to me.  I suspect that
>>means one's ultimately as good (or as bad) as the rest.
> 
> 
> What's wrong with "nonlocal"?  I don't think i've seen an argument
> against that one so far (from Talin or others).

Well, I just think that a fix for "an aesthetic wart" should be, well, 
aesthetic :)

I also think that it won't be a complete disaster if we do nothing at 
all - there *are* existing ways to deal with this problem; there are 
even some which aren't hackish and non-obvious. For example, its easy 
enough to create an object which acts as an artificial scope:

    def x():
       scope = object()
       scope.x = 1
       def y():
          scope.x = 2

To my mind, the above code looks about as elegant and efficient as most 
of the proposals put forward so far, and it already works.

How much are we really saving here by building this feature into the 
language?

-- Talin

From talin at acm.org  Tue Jul 11 08:29:10 2006
From: talin at acm.org (Talin)
Date: Mon, 10 Jul 2006 23:29:10 -0700
Subject: [Python-Dev] In defense of Capabilities [was: doc for new
 restricted execution design for Python]
In-Reply-To: <bbaeab100607091052g4a84a5c8gedea016012ed8c08@mail.gmail.com>
References: <20060705155149.eipxm6j6oycgkcs8@login.werra.lunarpages.com>	
	<44AEA550.1030204@acm.org>	
	<bbaeab100607071239j5b413f4cy991111a87c8de9fe@mail.gmail.com>	
	<ca471dc20607071343j1ba9ac7bj33ebc9071e5ebbf7@mail.gmail.com>	
	<bbaeab100607071355r67b291b0t285774667025e165@mail.gmail.com>	
	<44AF26A4.2040201@acm.org>	
	<Pine.LNX.4.58.0607072321000.22834@server1.LFW.org>	
	<ca471dc20607072145y5df22ea7h1b7fb4414fbd11b9@mail.gmail.com>	
	<bbaeab100607081948q5aee5485o797a32a06e5c2daf@mail.gmail.com>	
	<44B074C6.3070701@acm.org>
	<bbaeab100607091052g4a84a5c8gedea016012ed8c08@mail.gmail.com>
Message-ID: <44B34536.7040907@acm.org>

Brett Cannon wrote:
> Using a factory method callback, one could store the PyCodeObject in a C
> proxy object that just acts as a complete delegate, forwarding all method
> calls to the internally stored PyCodeObject.  That would work.
> 
> 
> For this initial implementation, though, I am not going to try to support
> this.   We can always add support like this later since it doesn't
> fundamentally break or change anything that is already planned.  Let's 
> focus
> on getting even more basic stuff working before we start to get too fancy.
> 
> -Brett

Thinking about this some more, I've come up with a simpler idea for a 
generic wrapper class. The wrapper consists of two parts: a decorator to 
indicate that a given method is 'public', and a C 'guard' wrapper that 
insures that only 'public' members can be accessed.

So for example:

    from Sandbox import guard, public

    class FileProxy:
       # Public method, no decorator
       @public
       def read( length ):
          ...

       @public
       def seek( offset ):
          ...

       # Private method, no decorator
       def open( path ):
          ...

    # Construct an instance of FileProxy, and wrap a
    # guard object around it. Any attempt to access a non-public
    # attribute will raise an exception (or perhaps simply report
    # that the attribute doesn't exist.)
    fh = guard( FileProxy() )

Now, from my point of view this *is* 'the basic stuff'. In other words, 
this right here is the fundamental sandbox mechanism, and everything 
else is built on top of it.

Now, the C 'guard' function is only a low-level means to insure that 
no-one can mess with the object; It is not intended to be the actual 
restriction policy itself. Those are placed in the wrapper classes, just 
as in your proposed scheme.

(This goes back to my basic premise, that a simple "yes/no" security 
feature can be used to build up much more complex and subtle security 
features.)

The only real complexity of this, as I see it, is that methods to 
references will have to be themselves wrapped.

In other words, if I say 'fh.read' without the (), what I get back can't 
be the actual read function object - that would be too easy to fiddle 
with. What I'd have to get is a wrapped version of the method that is a 
callable.

One relatively simply way to deal with this is to have the 'public' 
decorator create a C wrapper for the function object, and store it as an 
attribute of the function. The class wrapper then simply looks up the 
attribute, and if it has an attribute wrapper, returns that, otherwise 
it fails.

(Although, I've often wished for Python to have a variant of __call__ 
that could be used to override individual methods, i.e.:

     __call_method__( self, methodname, *args )

This would make the guard wrapper much easier to construct, since we 
could restrict the methods only to being called, and not allow 
references to methods to be taken.)

-- Talin

From martin at v.loewis.de  Tue Jul 11 08:30:06 2006
From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=)
Date: Tue, 11 Jul 2006 08:30:06 +0200
Subject: [Python-Dev] Klocwork analysis of source if we want it
In-Reply-To: <bbaeab100607101410o170186ffy800d54de8d969d7b@mail.gmail.com>
References: <bbaeab100607101410o170186ffy800d54de8d969d7b@mail.gmail.com>
Message-ID: <44B3456E.3000109@v.loewis.de>

Brett Cannon wrote:
> http://www.klocwork.com/company/releases/06_26_06.asp
> 
> Looks like Klocowork is doing the same thing as Coverity and providing
> free static analysis of source for open source projects.  Doubt we want
> this *and* Coverity, but figured wouldn't hurt to let people know about it.

See http://python.org/sf/1484556
ISTM that it generates too many false positives to be useful.

Regards,
Martin

From fredrik at pythonware.com  Tue Jul 11 08:42:02 2006
From: fredrik at pythonware.com (Fredrik Lundh)
Date: Tue, 11 Jul 2006 08:42:02 +0200
Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
In-Reply-To: <e8bf7a530607101343g77148a93o39d0f671adb99e4a@mail.gmail.com>
References: <001d01c6a369$78a59950$6402a8c0@arkdesktop>	<Pine.LNX.4.58.0607091349230.22834@server1.LFW.org>	<44B190DF.3000805@acm.org>
	<44B19691.4050509@acm.org>	<ca471dc20607092039g5cf2ea54h5225925ce877145e@mail.gmail.com>	<17586.19683.920761.293494@montanaro.dyndns.org>	<Pine.LNX.4.58.0607101533350.22834@server1.LFW.org>
	<e8bf7a530607101343g77148a93o39d0f671adb99e4a@mail.gmail.com>
Message-ID: <e8vh7p$m7k$1@sea.gmane.org>

Jeremy Hylton wrote:

> To express this email in the positive form:
> 1. Reserved words should be real words.
> 2. The meaning of the word should be clear.
> 3. "Put statements in positive form."  (Strunk & White)
> 4. The word should sound good.

agreed.  a word should describe what a thing is, not what it isn't.

not entirely sure about "global", though; things like "outer" and 
"extern(al)" might be better (especially if we could ignore C, which I 
don't think we can).

cannot think of prior art here, but there has to be some.  anyone ?


</F>


From fredrik at pythonware.com  Tue Jul 11 08:45:49 2006
From: fredrik at pythonware.com (Fredrik Lundh)
Date: Tue, 11 Jul 2006 08:45:49 +0200
Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
In-Reply-To: <44B33F9F.2000804@acm.org>
References: <001d01c6a369$78a59950$6402a8c0@arkdesktop>	<Pine.LNX.4.58.0607091349230.22834@server1.LFW.org>	<44B190DF.3000805@acm.org>	<44B19691.4050509@acm.org>	<ca471dc20607092039g5cf2ea54h5225925ce877145e@mail.gmail.com>	<17586.19683.920761.293494@montanaro.dyndns.org>	<Pine.LNX.4.58.0607101533350.22834@server1.LFW.org>
	<44B33F9F.2000804@acm.org>
Message-ID: <e8vhes$m7k$2@sea.gmane.org>

Talin wrote:

> I also think that it won't be a complete disaster if we do nothing at 
> all - there *are* existing ways to deal with this problem; there are 
> even some which aren't hackish and non-obvious. For example, its easy 
> enough to create an object which acts as an artificial scope:
> 
>     def x():
>        scope = object()
>        scope.x = 1
>        def y():
>           scope.x = 2
> 
> To my mind, the above code looks about as elegant and efficient as most 
> of the proposals put forward so far, and it already works.
> 
> How much are we really saving here by building this feature into the 
> language?

I don't think anyone commented on my "mutable" post, but as I mentioned 
in that post, if you look at the use cases for this, what most use cases 
really need is *mutation*, not *rebinding*.

</F>


From talin at acm.org  Tue Jul 11 08:53:35 2006
From: talin at acm.org (Talin)
Date: Mon, 10 Jul 2006 23:53:35 -0700
Subject: [Python-Dev] easy_install
Message-ID: <44B34AEF.8030008@acm.org>

Here's something to discuss:

First, let me say that I love easy_install. I absolutely "just works" 
and does what I want, and makes it really simple to install whatever bit 
of Python code I need.

At the same time, however, I get kind of scared when I hear people on 
the list discussing the various hacks needed to get setuputils and 
distutils et al all playing nice with each other (monkeypatching, etc.)

Having done a small bit of fiddling with distutils myself (as a user, I 
mean), I can see that while there's a terrific amount of effort put into 
it, its also not for the feignt of heart. That's not entirely distutil's 
fault - I gather that it's dealing with a lot of accumulated cruft (I 
imagine things like different and strange ways of archiving modules, 
dynamic modifications to path, all that sort of thing.)

It seems to me that if someone was going to spend some energy on this 
list coming up with proposals to improve Python, the thing that would 
have the most positive benefit in the long run (with the possible 
exception of Brett's work on rexec) would be a unified and clean vision 
of the whole import / package / download architecture.

Now, speaking from complete ignorance here, I might be way off base - it 
may be that this matter is well in hand, perhaps on some other mailing 
list. I don't know. In any case, I wanted to throw this out there...

-- Talin

From nnorwitz at gmail.com  Tue Jul 11 08:54:28 2006
From: nnorwitz at gmail.com (Neal Norwitz)
Date: Mon, 10 Jul 2006 23:54:28 -0700
Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
In-Reply-To: <e8vh7p$m7k$1@sea.gmane.org>
References: <001d01c6a369$78a59950$6402a8c0@arkdesktop>
	<Pine.LNX.4.58.0607091349230.22834@server1.LFW.org>
	<44B190DF.3000805@acm.org> <44B19691.4050509@acm.org>
	<ca471dc20607092039g5cf2ea54h5225925ce877145e@mail.gmail.com>
	<17586.19683.920761.293494@montanaro.dyndns.org>
	<Pine.LNX.4.58.0607101533350.22834@server1.LFW.org>
	<e8bf7a530607101343g77148a93o39d0f671adb99e4a@mail.gmail.com>
	<e8vh7p$m7k$1@sea.gmane.org>
Message-ID: <ee2a432c0607102354l5f49f1c7p383fc367b5c32031@mail.gmail.com>

On 7/10/06, Fredrik Lundh <fredrik at pythonware.com> wrote:
> Jeremy Hylton wrote:
>
> > To express this email in the positive form:
> > 1. Reserved words should be real words.
> > 2. The meaning of the word should be clear.
> > 3. "Put statements in positive form."  (Strunk & White)
> > 4. The word should sound good.
>
> agreed.  a word should describe what a thing is, not what it isn't.

Just looking at a thesaurus, came up with expose.  Not to mention
outward, remote, peripheral and without.  We already have a 'with'
keyword, why not 'without'?  Oh, I guess it fails #3...and #2...and #4
too.  Screw it, just go with alien. :-)

http://thesaurus.reference.com/browse/outer

n

From astrand at cendio.se  Tue Jul 11 09:05:47 2006
From: astrand at cendio.se (=?iso-8859-1?Q?Peter_=C5strand?=)
Date: Tue, 11 Jul 2006 09:05:47 +0200 (CEST)
Subject: [Python-Dev] subprocess.CalledProcessError.errno (#1223937)
Message-ID: <Pine.LNX.4.64.0607110858510.8438@maggie.lkpg.cendio.se>


I intend to fix bug #1223937: subprocess.py abuse of errno. I thought this was 
going to to tricky, to maintain backwards compatibility, but then I realized 
that check_call() and CalledProcessError() are not available in any released 
version of Python, so I guess it's safe to change them.

See http://www.python.org/sf?id=1223937 for my suggested patch. Any chance this 
can go into 2.5? Yeah, I know I'm late :)

Regards,
-- 
Peter ?strand		ThinLinc Chief Developer
Cendio			http://www.cendio.se
Teknikringen 3
583 30 Link?ping        Phone: +46-13-21 46 00

From fredrik at pythonware.com  Tue Jul 11 09:15:20 2006
From: fredrik at pythonware.com (Fredrik Lundh)
Date: Tue, 11 Jul 2006 09:15:20 +0200
Subject: [Python-Dev] subprocess.CalledProcessError.errno (#1223937)
In-Reply-To: <Pine.LNX.4.64.0607110858510.8438@maggie.lkpg.cendio.se>
References: <Pine.LNX.4.64.0607110858510.8438@maggie.lkpg.cendio.se>
Message-ID: <e8vj67$rja$1@sea.gmane.org>

Peter ?strand wrote:

> I intend to fix bug #1223937: subprocess.py abuse of errno. I thought 
> this was going to to tricky, to maintain backwards compatibility, but 
> then I realized that check_call() and CalledProcessError() are not 
> available in any released version of Python, so I guess it's safe to 
> change them.
> 
> See http://www.python.org/sf?id=1223937 for my suggested patch. Any 
> chance this can go into 2.5? Yeah, I know I'm late :)

that's up to Anthony to decide, but I'd say this this qualifies as a bug 
fix.

</F>


From fuzzyman at voidspace.org.uk  Tue Jul 11 10:14:22 2006
From: fuzzyman at voidspace.org.uk (Fuzzyman)
Date: Tue, 11 Jul 2006 09:14:22 +0100
Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
In-Reply-To: <44B33F9F.2000804@acm.org>
References: <001d01c6a369$78a59950$6402a8c0@arkdesktop>	<Pine.LNX.4.58.0607091349230.22834@server1.LFW.org>	<44B190DF.3000805@acm.org>	<44B19691.4050509@acm.org>	<ca471dc20607092039g5cf2ea54h5225925ce877145e@mail.gmail.com>	<17586.19683.920761.293494@montanaro.dyndns.org>	<Pine.LNX.4.58.0607101533350.22834@server1.LFW.org>
	<44B33F9F.2000804@acm.org>
Message-ID: <44B35DDE.5000405@voidspace.org.uk>

Talin wrote:

>Ka-Ping Yee wrote:
>  
>
>>On Mon, 10 Jul 2006 skip at pobox.com wrote:
>>
>>    
>>
>>>I think Talin's got a point though.  It seems hard to find one short English
>>>word that captures the essence of the desired behavior.  None of the words
>>>in his list seem strongly suggestive of the meaning to me.  I suspect that
>>>means one's ultimately as good (or as bad) as the rest.
>>>      
>>>
>>What's wrong with "nonlocal"?  I don't think i've seen an argument
>>against that one so far (from Talin or others).
>>    
>>
>
>Well, I just think that a fix for "an aesthetic wart" should be, well, 
>aesthetic :)
>
>I also think that it won't be a complete disaster if we do nothing at 
>all - there *are* existing ways to deal with this problem; there are 
>even some which aren't hackish and non-obvious. For example, its easy 
>enough to create an object which acts as an artificial scope:
>
>    def x():
>       scope = object()
>       scope.x = 1
>       def y():
>          scope.x = 2
>
>To my mind, the above code looks about as elegant and efficient as most 
>of the proposals put forward so far, and it already works.
>
>How much are we really saving here by building this feature into the 
>language?
>  
>
 >>> def x():
  ...:    scope = object()
  ...:    scope.x = 1
  ...:    def y():
  ...:        scope.x = 2
>>> x()
---------------------------------------------------------------------------
exceptions.AttributeError                            Traceback (most
recent call
 last)<ipython console>

I've often found it a nuisance that you can't instantiate an 'object',
to use as a mutable 'namespace', but instead have to define an arbitrary
empty class.

What happened to the 'namespace' proposal ?

Michael Foord
http://www.voidspace.org.uk/python/index.shtml

>-- Talin
>_______________________________________________
>Python-Dev mailing list
>Python-Dev at python.org
>http://mail.python.org/mailman/listinfo/python-dev
>Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk
>
>  
>


From mwh at python.net  Tue Jul 11 10:21:12 2006
From: mwh at python.net (Michael Hudson)
Date: Tue, 11 Jul 2006 09:21:12 +0100
Subject: [Python-Dev] User's complaints
In-Reply-To: <20060710204859.GA11451@rogue.amk.ca> (A. M. Kuchling's message
	of "Mon, 10 Jul 2006 16:48:59 -0400")
References: <ee2a432c0607041649i7ecb6cc1w8ebdf47f7d02097a@mail.gmail.com>
	<20060710151353.GA30654@code0.codespeak.net>
	<20060710204859.GA11451@rogue.amk.ca>
Message-ID: <2m3bd84knb.fsf@starship.python.net>

"A.M. Kuchling" <amk at amk.ca> writes:

> On Mon, Jul 10, 2006 at 05:13:53PM +0200, Armin Rigo wrote:
>> didn't draw much applause.  It certainly gave me the impression that
>> many changes in Python are advocated and welcomed by only a small
>> fraction of users.
>
> The benefits of changes are usually clear, but I don't think the costs
> of changes are fully assessed.  python-dev considers the cost of
> changes to CPython's implementation, but I don't think the cost
> changes to Jython, IronPython, or PyPy are taken into account here.

I don't want to attempt to speak for Armin, but I don't think that was
quite the point he was trying to make.  The cost of implementation
isn't really that high -- most of the changes from 2.3 to 2.4 were
implemented for PyPy in about a week and we have implementations of
most of the 2.5 features already, and you can also consider the
decorators discussion where each of the proposed syntaxes had solid
tested implementations.

At least from my POV, the cost of bad design, whether through simple
lack of sufficient thought or aggregation of features into cruft, is
much much higher.

> PyPy probably has enough representatives here who would squawk if
> something was difficult, but I don't think Jython or IronPython do.

People implementing Python should accept the fact that Python is a
changing language, IMHO.

> I think if we assessed those costs fully, the bar for changes to the
> language would be a good deal higher.

I think if you assessed these other costs fully, the bar would be
higher still.

Cheers,
mwh

-- 
  This is the fixed point problem again; since all some implementors
  do is implement the compiler and libraries for compiler writing, the
  language becomes good at writing compilers and not much else!
                                 -- Brian Rogoff, comp.lang.functional

From sreeram at tachyontech.net  Tue Jul 11 10:56:54 2006
From: sreeram at tachyontech.net (K.S.Sreeram)
Date: Tue, 11 Jul 2006 14:26:54 +0530
Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
In-Reply-To: <44B33F9F.2000804@acm.org>
References: <001d01c6a369$78a59950$6402a8c0@arkdesktop>	<Pine.LNX.4.58.0607091349230.22834@server1.LFW.org>	<44B190DF.3000805@acm.org>	<44B19691.4050509@acm.org>	<ca471dc20607092039g5cf2ea54h5225925ce877145e@mail.gmail.com>	<17586.19683.920761.293494@montanaro.dyndns.org>	<Pine.LNX.4.58.0607101533350.22834@server1.LFW.org>
	<44B33F9F.2000804@acm.org>
Message-ID: <44B367D6.9030600@tachyontech.net>

Talin wrote:
>     def x():
>        scope = object()
>        scope.x = 1
>        def y():
>           scope.x = 2

'object' doesn't let you set custom attributes.
Here's what I normally use in my code...

class Data : pass

def x() :
    d = Data()
    d.x = 1
    def y() :
        d.x += 1


Regards
Sreeram


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 252 bytes
Desc: OpenPGP digital signature
Url : http://mail.python.org/pipermail/python-dev/attachments/20060711/eb5ca5be/attachment.pgp 

From ncoghlan at gmail.com  Tue Jul 11 11:44:36 2006
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 11 Jul 2006 19:44:36 +1000
Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
In-Reply-To: <44B35DDE.5000405@voidspace.org.uk>
References: <001d01c6a369$78a59950$6402a8c0@arkdesktop>	<Pine.LNX.4.58.0607091349230.22834@server1.LFW.org>	<44B190DF.3000805@acm.org>	<44B19691.4050509@acm.org>	<ca471dc20607092039g5cf2ea54h5225925ce877145e@mail.gmail.com>	<17586.19683.920761.293494@montanaro.dyndns.org>	<Pine.LNX.4.58.0607101533350.22834@server1.LFW.org>	<44B33F9F.2000804@acm.org>
	<44B35DDE.5000405@voidspace.org.uk>
Message-ID: <44B37304.9090106@gmail.com>

Fuzzyman wrote:
> I've often found it a nuisance that you can't instantiate an 'object',
> to use as a mutable 'namespace', but instead have to define an arbitrary
> empty class.
> 
> What happened to the 'namespace' proposal ?

The code and the pre-PEP [1] are still out there, but Carlos, Steve and I all 
got distracted by other things. It really needs to go back to c.l.p to thrash 
out some of the name collision prblems (e.g. should *all* access to methods, 
even special methods, be via type(x)?).

Cheers,
Nick.

[1] http://namespace.python-hosting.com/wiki/NamespacePep

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
             http://www.boredomandlaziness.org

From talin at acm.org  Tue Jul 11 12:36:09 2006
From: talin at acm.org (Talin)
Date: Tue, 11 Jul 2006 03:36:09 -0700
Subject: [Python-Dev] Capabilities / Restricted Execution
Message-ID: <44B37F19.7090703@acm.org>

I thought a little bit more about Guido's comment that you can hide 
Python objects in a C wrapper class. However, as I was trying to sleep, 
I realized that you don't even need C to do it.

The trick is to store the object reference as a closure variable. 
Assuming that there's no mechanism to directly access such variables, 
you can effectively have 'private' variables.

Here's an pure-Python implementation of a wrapper that restricts access 
to all class members except those marked with the 'public' decorator. 
Note in the test cases below that even '__class__' and '__dict__' are 
inaccessible.
------------------------------------------------------------------

# Exception to throw when we violate restrictions
class GuardException( Exception ): pass

# Decorator function
def public(func):
     func.f_public = True
     return func

# Wrapper function
def guard(content):
     class Guard(object):
         def __getattribute__(self, name):
             # This will throw an AttributeException if the
             # attribute doesn't exist, which is what
             # we want.
             attr = object.__getattribute__(content,name)
             if hasattr( attr, 'im_func' ):
                 if hasattr( attr.im_func, 'f_public' ):
                     # We can't return the bound method directly,
                     # instead we create a proxy for it.
                     def proxy_attr( *args, **kwargs ):
                         return attr( *args, **kwargs )
                     return proxy_attr

             # Attribute exists but has no proxy
             raise GuardException( name )

     return Guard()

# Test class
class Test(object):
     def __init__(self,name):
         self.name = name

     @public
     def get_name(self):
         return self.name

     def set_name(self,name):
         self,name = name

# Test objects. First is unwrapped, second is wrapped.
t1 = Test("alpha")
t2 = guard(Test("beta"))

# These functions work correctly
print t1.name
print t2.get_name()

# This gets AttributeError because there's no such attribute
print t2.unk()

# These generate GuardException
print t2.set_name()
print t2.__dict__
print t2.__class__
print t2.name

------------------------------------------------------

-- Talin

From benji at benjiyork.com  Tue Jul 11 13:33:10 2006
From: benji at benjiyork.com (Benji York)
Date: Tue, 11 Jul 2006 07:33:10 -0400
Subject: [Python-Dev] Doctest and Footnotes
In-Reply-To: <5.1.1.6.0.20060711004320.01eed7d0@sparrow.telecommunity.com>
References: <5.1.1.6.0.20060711004320.01eed7d0@sparrow.telecommunity.com>
Message-ID: <44B38C76.5050502@benjiyork.com>

Phillip J. Eby wrote:
> It would be nice if tracebacks in the footnote show the invoking context

Yep.  Someone (Jim Fulton I think) had suggested that to me.  I'll look 
into it.

> My other thought would be that having a patch that works against the 2.5 
> version of doctest would be good

My intent is to update zope.testing to the 2.5 version of doctest and 
generate a new patch against that.

It would be even better if there were a stand-alone release of doctest 
instead, or (another Jim suggestion) if doctest were made extensible.  I 
have another doctest/ReST idea I'd like to try some time (using ReST 
tables for FIT), so if there's interest perhaps I'll try my hand at a 
plugin system of some sort. (wish I could use zope.component :)
--
Benji York


From gmccaughan at synaptics-uk.com  Tue Jul 11 13:09:51 2006
From: gmccaughan at synaptics-uk.com (Gareth McCaughan)
Date: Tue, 11 Jul 2006 12:09:51 +0100
Subject: [Python-Dev] 2.5 and beyond
In-Reply-To: <ca471dc20607032156t4d281194u3cd8681e59442426@mail.gmail.com>
References: <9B706ACDA57B23438AAE4822369672AF04062AD1@RED-MSG-10.redmond.corp.microsoft.com>
	<20060703171546.1108.JCARLSON@uci.edu>
	<ca471dc20607032156t4d281194u3cd8681e59442426@mail.gmail.com>
Message-ID: <200607111209.55177.gmccaughan@synaptics-uk.com>

> I can say it stronger. Any resemblance between Python and Scheme or
> Lisp is purely a coincidence. Neither language is in Python's
> ancestry, at least not explicitly; I'd never used or tried to learn
> Scheme when I started Python (still haven't) and my Lisp experience
> was limited to copying Emacs startup code from friends (still is).

Python resembles Lisp like an octopus eye resembles a mammalian eye:
they have lots in common because they're both pretty good solutions
to similar problems. Deciding whether it's Python or Lisp that has
the retina fitted back-to-front is left as an exercise for the reader.

-- 
g


From bborcic at gmail.com  Tue Jul 11 14:43:52 2006
From: bborcic at gmail.com (Boris Borcic)
Date: Tue, 11 Jul 2006 14:43:52 +0200
Subject: [Python-Dev] [slighly OT] Native speakers and hurting brains
In-Reply-To: <44AEF0F2.5040007@canterbury.ac.nz>
References: <20060705101821.7sqnxo7lu5cgg4ks@login.werra.lunarpages.com>	<ca471dc20607051027y677c7127wfe098401c366dc74@mail.gmail.com>	<e8m44k$5qb$1@sea.gmane.org>
	<44AEF0F2.5040007@canterbury.ac.nz>
Message-ID: <e906f3$sdo$1@sea.gmane.org>

Greg Ewing wrote:
> Boris Borcic wrote:
> 
>> I believe that in this case native linguistic intuition made the decision...
> 
> The reason has nothing to do with language. Guido didn't
> want sum() to become an attractive nuisance by *appearing*
> to be an obvious way of joining a list of strings, while
> actually being a very inefficient way of doing that.

sum() *is* exactly an attractive nuisance by *appearing* to be an obvious way of 
chaining strings in a list (without actually being one).

> Considerable effort was put into trying to make sum()
> smart enough to detect when you were using it on a
> list of strings and do "".join behind the scenes, but
> Guido decided in the end that it wasn't worth the
> trouble, given that he only ever intended sum() to
> be used on numbers in the first place.

That's not quite conform to the record. According to py-dev archives it happened 
on April 2003 with a thread "Fwd: summing a bunch of numbers (or "whatevers")" 
initiated by Alex Martelli where he actually proposed a working implementation 
of sum() in C, that did short-circuit the case of strings to ''.join. That was 
Sat 19th of April.

Debate ensued, and by late Sunday 20th around 11PM;, the honorable author of the 
Zen of Python had killed that use case for sum() with "sum(sequence_of_strings) 
hurts my brain". (Hello Tim, so what about  < sqrt(':(') >  ?)

Guido's first intervention in the thread was the next morning, and the two very 
first lines of his intervention where :

"OK, let me summarize and pronounce.

  sum(sequence_of_strings) is out...."

I admit that there is a step of arguable interpretation from these recorded 
facts to my diagnostic, but the latter is compatible with the facts. Your 
version otoh looks more robust in the role of eg creation myth.

Best Regards, Boris Borcic
--
"assert 304 in 340343, P424D15E_M15M47CH"



From fredrik at pythonware.com  Tue Jul 11 15:06:09 2006
From: fredrik at pythonware.com (Fredrik Lundh)
Date: Tue, 11 Jul 2006 15:06:09 +0200
Subject: [Python-Dev] [slighly OT] Native speakers and hurting brains
References: <20060705101821.7sqnxo7lu5cgg4ks@login.werra.lunarpages.com>	<ca471dc20607051027y677c7127wfe098401c366dc74@mail.gmail.com>	<e8m44k$5qb$1@sea.gmane.org><44AEF0F2.5040007@canterbury.ac.nz>
	<e906f3$sdo$1@sea.gmane.org>
Message-ID: <e907o1$1i5$1@sea.gmane.org>

Boris Borcic wrote:

>>> I believe that in this case native linguistic intuition made the decision...
>>
>> The reason has nothing to do with language. Guido didn't
>> want sum() to become an attractive nuisance by *appearing*
>> to be an obvious way of joining a list of strings, while
>> actually being a very inefficient way of doing that.
>
> sum() *is* exactly an attractive nuisance by *appearing* to be an obvious way of
> chaining strings in a list (without actually being one).

in what language the word "sum" an appropriate synonym for "concatenate" ?

</F> 




From gmccaughan at synaptics-uk.com  Tue Jul 11 15:29:21 2006
From: gmccaughan at synaptics-uk.com (Gareth McCaughan)
Date: Tue, 11 Jul 2006 14:29:21 +0100
Subject: [Python-Dev] [slighly OT] Native speakers and hurting brains
In-Reply-To: <e906f3$sdo$1@sea.gmane.org>
References: <20060705101821.7sqnxo7lu5cgg4ks@login.werra.lunarpages.com>
	<44AEF0F2.5040007@canterbury.ac.nz> <e906f3$sdo$1@sea.gmane.org>
Message-ID: <200607111429.22964.gmccaughan@synaptics-uk.com>

(Attention conservation notice: the following is concerned almost entirely
with exegesis of an old python-dev thread. Those interested in improving Python
and not in history and exegesis should probably ignore it.)

On Tuesday 2006-07-11 13:43, Boris Borcic wrote:

> >> I believe that in this case native linguistic intuition made the decision...
> > 
> > The reason has nothing to do with language. Guido didn't
> > want sum() to become an attractive nuisance by *appearing*
> > to be an obvious way of joining a list of strings, while
> > actually being a very inefficient way of doing that.
> 
> sum() *is* exactly an attractive nuisance by *appearing* to be an obvious way of 
> chaining strings in a list (without actually being one).

It is a *short-term* attractive nuisance; anyone who tries it
will quickly find that it can't be used to "sum" strings;
whereas if this functionality of sum() had been retained
then users would much more readily have been led to use it
*and leave it in their programs*, to the detriment of their
efficiency.

> > Considerable effort was put into trying to make sum()
> > smart enough to detect when you were using it on a
> > list of strings and do "".join behind the scenes, but
> > Guido decided in the end that it wasn't worth the
> > trouble, given that he only ever intended sum() to
> > be used on numbers in the first place.
> 
> That's not quite conform to the record. According to py-dev archives it happened 
> on April 2003 with a thread "Fwd: summing a bunch of numbers (or "whatevers")" 
> initiated by Alex Martelli where he actually proposed a working implementation 
> of sum() in C, that did short-circuit the case of strings to ''.join. That was 
> Sat 19th of April.
> 
> Debate ensued, and by late Sunday 20th around 11PM;, the honorable author of the 
> Zen of Python had killed that use case for sum() with "sum(sequence_of_strings) 
> hurts my brain". (Hello Tim, so what about  < sqrt(':(') >  ?)

Hardly "killed", since discussion of sum(stringseq) continued after
Tim's comment, and since Tim was neither the only, nor the first, nor
(I think) the last, person to object to sum(stringseq).

> Guido's first intervention in the thread was the next morning, and the two very 
> first lines of his intervention where :
> 
> "OK, let me summarize and pronounce.
> 
>   sum(sequence_of_strings) is out...."
> 
> I admit that there is a step of arguable interpretation from these recorded 
> facts to my diagnostic, but the latter is compatible with the facts. Your 
> version otoh looks more robust in the role of eg creation myth.

Your interpretation is only "compatible with the facts" by means
of the hypothesis that Guido, despite saying "OK, let me summarize
and pronounce" (which, incidentally, was the *actual* opening of
that message, which was not Guido's first intervention in the
thread, but who cares about facts?), had only read Tim Peters's
message and not the rest of the thread. Including, for instance,
one of the last messages before Guido's, from Alex, reporting that
his sum() concatenated strings twice as slowly as ''.join. It also
requires one to ignore the fact that Guido said (in that same
message that you described as his "first intervention")

  |                       OTOH if we provide *any* way of providing a
  | different starting point, some creative newbie is going to use
  | sum(list_of_strings, "") instead of "".join(), and be hurt by the
  | performance months later.

and the fact that in Guido's *actual* "first intervention" he
said that he found sum(stringseq) "weird" and would continue to
use ''.join. (Remark: Guido is not a native speaker of English,
though he's a very competent one.) For that matter, Tim's
comment about sum(stringseq) hurting his brain was in direct
response to another part of that same message from Guido in
which he said that "the name sum() strongly suggests that it's
summing up numbers". (Remark: Guido still isn't a native speaker
of English.)

Perhaps the fact that "sum" isn't a natural English term for
concatenation of strings -- which, indeed, it isn't -- was one
of his reasons; it demonstrably wasn't the only one; and it
seems to have been the non-native speaker Guido himself who
introduced that consideration to the discussion.

It appears to me that your interpretation is difficult to reconcile
with the facts and is based on an incorrect and narrow reading of the
original texts. What was that you were saying about creation myths?

(I agree that Greg's interpretation is also not well supported
by that thread; I don't know whether Guido later said anything
that would determine how much truth there is in it. Since sum()
was Alex Martelli's invention, it seems unlikely that Greg's
quite right.)

-- 
g


From bborcic at gmail.com  Tue Jul 11 15:47:25 2006
From: bborcic at gmail.com (Boris Borcic)
Date: Tue, 11 Jul 2006 15:47:25 +0200
Subject: [Python-Dev] [slighly OT] Native speakers and hurting brains
In-Reply-To: <200607111429.22964.gmccaughan@synaptics-uk.com>
References: <20060705101821.7sqnxo7lu5cgg4ks@login.werra.lunarpages.com>	<44AEF0F2.5040007@canterbury.ac.nz>
	<e906f3$sdo$1@sea.gmane.org>
	<200607111429.22964.gmccaughan@synaptics-uk.com>
Message-ID: <e90a69$b6c$1@sea.gmane.org>

Gareth McCaughan wrote:
> (Attention conservation notice: the following is concerned almost entirely
> with exegesis of an old python-dev thread. Those interested in improving Python
> and not in history and exegesis should probably ignore it.)
> 
> On Tuesday 2006-07-11 13:43, Boris Borcic wrote:
> 
>>>> I believe that in this case native linguistic intuition made the decision...
>>> The reason has nothing to do with language. Guido didn't
>>> want sum() to become an attractive nuisance by *appearing*
>>> to be an obvious way of joining a list of strings, while
>>> actually being a very inefficient way of doing that.
>> sum() *is* exactly an attractive nuisance by *appearing* to be an obvious way of 
>> chaining strings in a list (without actually being one).
> 
> It is a *short-term* attractive nuisance; anyone who tries it
> will quickly find that it can't be used to "sum" strings;
> whereas if this functionality of sum() had been retained
> then users would much more readily have been led to use it
> *and leave it in their programs*, to the detriment of their
> efficiency.
> 
>>> Considerable effort was put into trying to make sum()
>>> smart enough to detect when you were using it on a
>>> list of strings and do "".join behind the scenes, but
>>> Guido decided in the end that it wasn't worth the
>>> trouble, given that he only ever intended sum() to
>>> be used on numbers in the first place.
>> That's not quite conform to the record. According to py-dev archives it happened 
>> on April 2003 with a thread "Fwd: summing a bunch of numbers (or "whatevers")" 
>> initiated by Alex Martelli where he actually proposed a working implementation 
>> of sum() in C, that did short-circuit the case of strings to ''.join. That was 
>> Sat 19th of April.
>>
>> Debate ensued, and by late Sunday 20th around 11PM;, the honorable author of the 
>> Zen of Python had killed that use case for sum() with "sum(sequence_of_strings) 
>> hurts my brain". (Hello Tim, so what about  < sqrt(':(') >  ?)
> 
> Hardly "killed", since discussion of sum(stringseq) continued after
> Tim's comment, and since Tim was neither the only, nor the first, nor
> (I think) the last, person to object to sum(stringseq).
> 
>> Guido's first intervention in the thread was the next morning, and the two very 
>> first lines of his intervention where :
>>
>> "OK, let me summarize and pronounce.
>>
>>   sum(sequence_of_strings) is out...."
>>
>> I admit that there is a step of arguable interpretation from these recorded 
>> facts to my diagnostic, but the latter is compatible with the facts. Your 
>> version otoh looks more robust in the role of eg creation myth.
> 
> Your interpretation is only "compatible with the facts" by means
> of the hypothesis that Guido, despite saying "OK, let me summarize
> and pronounce" (which, incidentally, was the *actual* opening of
> that message, which was not Guido's first intervention in the
> thread, but who cares about facts?), had only read Tim Peters's
> message and not the rest of the thread. Including, for instance,
> one of the last messages before Guido's, from Alex, reporting that
> his sum() concatenated strings twice as slowly as ''.join. It also
> requires one to ignore the fact that Guido said (in that same
> message that you described as his "first intervention")
> 
>   |                       OTOH if we provide *any* way of providing a
>   | different starting point, some creative newbie is going to use
>   | sum(list_of_strings, "") instead of "".join(), and be hurt by the
>   | performance months later.
> 
> and the fact that in Guido's *actual* "first intervention" he
> said that he found sum(stringseq) "weird" and would continue to
> use ''.join. (Remark: Guido is not a native speaker of English,
> though he's a very competent one.) For that matter, Tim's
> comment about sum(stringseq) hurting his brain was in direct
> response to another part of that same message from Guido in
> which he said that "the name sum() strongly suggests that it's
> summing up numbers". (Remark: Guido still isn't a native speaker
> of English.)
> 
> Perhaps the fact that "sum" isn't a natural English term for
> concatenation of strings -- which, indeed, it isn't -- was one
> of his reasons; it demonstrably wasn't the only one; and it
> seems to have been the non-native speaker Guido himself who
> introduced that consideration to the discussion.
> 
> It appears to me that your interpretation is difficult to reconcile
> with the facts and is based on an incorrect and narrow reading of the
> original texts. What was that you were saying about creation myths?
> 
> (I agree that Greg's interpretation is also not well supported
> by that thread; I don't know whether Guido later said anything
> that would determine how much truth there is in it. Since sum()
> was Alex Martelli's invention, it seems unlikely that Greg's
> quite right.)
> 


From bborcic at gmail.com  Tue Jul 11 15:54:24 2006
From: bborcic at gmail.com (Boris Borcic)
Date: Tue, 11 Jul 2006 15:54:24 +0200
Subject: [Python-Dev] [slighly OT] Native speakers and hurting brains
In-Reply-To: <200607111429.22964.gmccaughan@synaptics-uk.com>
References: <20060705101821.7sqnxo7lu5cgg4ks@login.werra.lunarpages.com>	<44AEF0F2.5040007@canterbury.ac.nz>
	<e906f3$sdo$1@sea.gmane.org>
	<200607111429.22964.gmccaughan@synaptics-uk.com>
Message-ID: <e90ajd$cq0$1@sea.gmane.org>

I wish to apologize for mistakenly pushing the "send" button on an untouched 
copy of Gereth McGaughan's reply, in the case my early cancel at gmane did not 
stop the propagation.

I'll profit just to add (bringing this to a conclusion)

Gareth McCaughan wrote:
> (...was not Guido's first intervention in the
> thread, but who cares about facts?)

I do, and I stand corrected.

Best regards, Boris Borcic
--
"On na?t tous les m?tres du m?me monde"


From stefan.rank at ofai.at  Tue Jul 11 15:55:46 2006
From: stefan.rank at ofai.at (Stefan Rank)
Date: Tue, 11 Jul 2006 15:55:46 +0200
Subject: [Python-Dev] urllib.quote and unicode bug resuscitation attempt
Message-ID: <44B3ADE2.7040108@ofai.at>

Hi,

urllib.quote fails on unicode strings and in an unhelpful way::

   Python 2.4.3 (#69, Mar 29 2006, 17:35:34) [MSC v.1310 32 bit (Intel)] 
on win32
   Type "help", "copyright", "credits" or "license" for more information.
   >>> import urllib
   >>> urllib.quote('a\xf1a')
   'a%F1a'
   >>> urllib.quote(u'ana')
   'ana'
   >>> urllib.quote(u'a\xf1a')
   Traceback (most recent call last):
     File "<stdin>", line 1, in ?
     File "C:\Python24\lib\urllib.py", line 1117, in quote
       res = map(safe_map.__getitem__, s)
   KeyError: u'\xf1'

There is a (closed) tracker item, dated 2000-10-12,
http://sourceforge.net/tracker/?group_id=5470&atid=105470&aid=216716&func=detail
and there was a note added to PEP-42 by Guido.

According to a message I found on quixote-users,
http://mail.mems-exchange.org/durusmail/quixote-users/5363/
it might have worked prior to 2.4.2.
(I guess that this changed because of ascii now being the default encoding?)

BTW, a patch by rhettinger from 8 months or so ago allows urllib.unquote 
to operate transparently on unicode strings::

   >>> urllib.unquote('a%F1a')
   'a\xf1a'
   >>> urllib.unquote(u'a%F1a')
   u'a\xf1a'

I suggest to add (after 2.5 I assume) one of the following to the 
beginning of urllib.quote to either fail early and consistently on 
unicode arguments and improve the error message::

   if isinstance(s, unicode):
       raise TypeError("quote needs a byte string argument, not unicode,"
                       " use `argument.encode('utf-8')` first.")

or to do The Right Thing (tm), which is utf-8 encoding::

   if isinstance(s, unicode):
       s = s.encode('utf-8')

as suggested in
http://www.w3.org/International/O-URL-code.html
and rfc3986.

cheers,
stefan


From bborcic at gmail.com  Tue Jul 11 16:07:20 2006
From: bborcic at gmail.com (Boris Borcic)
Date: Tue, 11 Jul 2006 16:07:20 +0200
Subject: [Python-Dev] [slighly OT] Native speakers and hurting brains
In-Reply-To: <e907o1$1i5$1@sea.gmane.org>
References: <20060705101821.7sqnxo7lu5cgg4ks@login.werra.lunarpages.com>	<ca471dc20607051027y677c7127wfe098401c366dc74@mail.gmail.com>	<e8m44k$5qb$1@sea.gmane.org><44AEF0F2.5040007@canterbury.ac.nz>	<e906f3$sdo$1@sea.gmane.org>
	<e907o1$1i5$1@sea.gmane.org>
Message-ID: <e90bbm$g10$1@sea.gmane.org>

Fredrik Lundh wrote:
> 
> in what language the word "sum" an appropriate synonym for "concatenate" ?

any that admits a+b to mean ''.join([a,b]), I'd say.

- BB


From fredrik at pythonware.com  Tue Jul 11 16:16:49 2006
From: fredrik at pythonware.com (Fredrik Lundh)
Date: Tue, 11 Jul 2006 16:16:49 +0200
Subject: [Python-Dev] [slighly OT] Native speakers and hurting brains
References: <20060705101821.7sqnxo7lu5cgg4ks@login.werra.lunarpages.com>	<ca471dc20607051027y677c7127wfe098401c366dc74@mail.gmail.com>	<e8m44k$5qb$1@sea.gmane.org><44AEF0F2.5040007@canterbury.ac.nz>	<e906f3$sdo$1@sea.gmane.org><e907o1$1i5$1@sea.gmane.org>
	<e90bbm$g10$1@sea.gmane.org>
Message-ID: <e90bsh$i6n$1@sea.gmane.org>

Boris Borcic wrote:

>> in what language [is] the word "sum" an appropriate synonym for "concatenate" ?
>
> any that admits a+b to mean ''.join([a,b]), I'd say.

and what human language would that be ?

</F> 




From skip at pobox.com  Tue Jul 11 16:35:55 2006
From: skip at pobox.com (skip at pobox.com)
Date: Tue, 11 Jul 2006 09:35:55 -0500
Subject: [Python-Dev] urllib.quote and unicode bug resuscitation attempt
In-Reply-To: <44B3ADE2.7040108@ofai.at>
References: <44B3ADE2.7040108@ofai.at>
Message-ID: <17587.46923.677564.898184@montanaro.dyndns.org>


    Stefan> According to a message I found on quixote-users,
    Stefan> http://mail.mems-exchange.org/durusmail/quixote-users/5363/ it
    Stefan> might have worked prior to 2.4.2.

Confirmed with 2.3.5.

    Stefan>    if isinstance(s, unicode):
    Stefan>        s = s.encode('utf-8')

    Stefan> as suggested in
    Stefan> http://www.w3.org/International/O-URL-code.html
    Stefan> and rfc3986.

Seems like the right way to do it to me.

Skip

From robinbryce at gmail.com  Tue Jul 11 16:46:44 2006
From: robinbryce at gmail.com (Robin Bryce)
Date: Tue, 11 Jul 2006 15:46:44 +0100
Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
In-Reply-To: <ee2a432c0607102354l5f49f1c7p383fc367b5c32031@mail.gmail.com>
References: <001d01c6a369$78a59950$6402a8c0@arkdesktop>
	<Pine.LNX.4.58.0607091349230.22834@server1.LFW.org>
	<44B190DF.3000805@acm.org> <44B19691.4050509@acm.org>
	<ca471dc20607092039g5cf2ea54h5225925ce877145e@mail.gmail.com>
	<17586.19683.920761.293494@montanaro.dyndns.org>
	<Pine.LNX.4.58.0607101533350.22834@server1.LFW.org>
	<e8bf7a530607101343g77148a93o39d0f671adb99e4a@mail.gmail.com>
	<e8vh7p$m7k$1@sea.gmane.org>
	<ee2a432c0607102354l5f49f1c7p383fc367b5c32031@mail.gmail.com>
Message-ID: <bcf87d920607110746n70f7cd16lf3ae057edc003f73@mail.gmail.com>

outbound x = 1
x = 2

evaluating using Jeremy Hilton's' list:

1. is a real word
2. For me - in python - it would mean: Is found in 'outer' scope and
is already bound.
  And the literal meaning of 'outbound 'headed away' [1] is pretty
darn close to what I mean when I spell the usual mutables kluge.

3 statement is positive form
4. I like it

could not find a use of outbound in python source (2.4.3)

[1] http://dictionary.reference.com/search?q=outbound


Robin

From fuzzyman at voidspace.org.uk  Tue Jul 11 16:55:20 2006
From: fuzzyman at voidspace.org.uk (Fuzzyman)
Date: Tue, 11 Jul 2006 15:55:20 +0100
Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
In-Reply-To: <bcf87d920607110746n70f7cd16lf3ae057edc003f73@mail.gmail.com>
References: <001d01c6a369$78a59950$6402a8c0@arkdesktop>	<Pine.LNX.4.58.0607091349230.22834@server1.LFW.org>	<44B190DF.3000805@acm.org>
	<44B19691.4050509@acm.org>	<ca471dc20607092039g5cf2ea54h5225925ce877145e@mail.gmail.com>	<17586.19683.920761.293494@montanaro.dyndns.org>	<Pine.LNX.4.58.0607101533350.22834@server1.LFW.org>	<e8bf7a530607101343g77148a93o39d0f671adb99e4a@mail.gmail.com>	<e8vh7p$m7k$1@sea.gmane.org>	<ee2a432c0607102354l5f49f1c7p383fc367b5c32031@mail.gmail.com>
	<bcf87d920607110746n70f7cd16lf3ae057edc003f73@mail.gmail.com>
Message-ID: <44B3BBD8.7050307@voidspace.org.uk>

Robin Bryce wrote:

>outbound x = 1
>  
>
FWINW, to me 'nonlocal' clearly and immediately tells you what you need
to know about the variable.

'outbound' has no programming associations for me (it makes me think of
'outward bound' and roaming the great outdoors). So negative or not I'm
+1 on nonlocal if we really need this... (Hey, and I'm a native.)

Michael Foord
http://www.voidspace.org.uk/python/index.shtml


>x = 2
>
>evaluating using Jeremy Hilton's' list:
>
>1. is a real word
>2. For me - in python - it would mean: Is found in 'outer' scope and
>is already bound.
>  And the literal meaning of 'outbound 'headed away' [1] is pretty
>darn close to what I mean when I spell the usual mutables kluge.
>
>3 statement is positive form
>4. I like it
>
>could not find a use of outbound in python source (2.4.3)
>
>[1] http://dictionary.reference.com/search?q=outbound
>
>
>Robin
>_______________________________________________
>Python-Dev mailing list
>Python-Dev at python.org
>http://mail.python.org/mailman/listinfo/python-dev
>Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk
>
>  
>


From pje at telecommunity.com  Tue Jul 11 16:56:22 2006
From: pje at telecommunity.com (Phillip J. Eby)
Date: Tue, 11 Jul 2006 10:56:22 -0400
Subject: [Python-Dev] Capabilities / Restricted Execution
In-Reply-To: <44B37F19.7090703@acm.org>
Message-ID: <5.1.1.6.0.20060711105343.01eed878@sparrow.telecommunity.com>

At 03:36 AM 7/11/2006 -0700, Talin wrote:
>I thought a little bit more about Guido's comment that you can hide
>Python objects in a C wrapper class. However, as I was trying to sleep,
>I realized that you don't even need C to do it.
>
>The trick is to store the object reference as a closure variable.
>Assuming that there's no mechanism to directly access such variables,
>you can effectively have 'private' variables.

A function's func_closure contains cell objects that hold the 
variables.  These are readable if you can set the func_closure of some 
function of your own.  If the overall plan includes the ability to restrict 
func_closure setting (or reading) in a restricted interpreter, then you 
might be okay.


From anthony at python.org  Tue Jul 11 17:30:35 2006
From: anthony at python.org (Anthony Baxter)
Date: Wed, 12 Jul 2006 01:30:35 +1000
Subject: [Python-Dev] Subject: RELEASED Python 2.5 (beta 2)
Message-ID: <200607120130.49083.anthony@python.org>

On behalf of the Python development team and the Python
community, I'm happy to announce the second BETA release
of Python 2.5.

This is an *beta* release of Python 2.5. As such, it is not
suitable for a production environment. It is being released
to solicit feedback and hopefully discover bugs, as well as
allowing you to determine how changes in 2.5 might impact
you. If you find things broken or incorrect, please log a
bug on Sourceforge.

In particular, note that changes to improve Python's support
of 64 bit systems might require authors of C extensions
to change their code. More information (as well as source
distributions and Windows installers) are available from the
2.5 website:

    http://www.python.org/2.5/

A Universal Mac OSX Installer will be available shortly - in
the meantime, Mac users can build from the source tarballs.

Since the first beta, a large number of bug fixes have been
made to Python 2.5 - see the release notes (available from
the 2.5 webpage) for the full details.

There has been one very small new feature added - the
sys._current_frames() function was added. This is extremely
useful for tracking down deadlocks and related problems - 
a similar technique is already used in the popular 
DeadlockDebugger extension for Zope. It is not possible to 
do this sort of debugging from outside the Python core safely 
and robustly, which is why we've snuck this in after the 
feature freeze.

As of this release, Python 2.5 is now in *feature freeze*.
Unless absolutely necessary, no functionality changes will
be made between now and the final release of Python 2.5.

The plan is for this to be the final beta release. We should
now move to one or more release candidates, leading to
a 2.5 final release early August. PEP 356 includes the
schedule and will be updated as the schedule evolves. At
this point, any testing you can do would be greatly, greatly
appreciated.

The new features in Python 2.5 are described in Andrew
Kuchling's What's New In Python 2.5. It's available from
the 2.5 web page.

Amongst the language features added include conditional
expressions, the with statement, the merge of try/except
and try/finally into try/except/finally, enhancements to
generators to produce a coroutine kind of functionality, and
a brand new AST-based compiler implementation.

New modules added include hashlib, ElementTree, sqlite3,
wsgiref and ctypes. In addition, a new profiling module
"cProfile" was added.

Enjoy this new release,
Anthony

Anthony Baxter
anthony at python.org
Python Release Manager
(on behalf of the entire python-dev team)
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 191 bytes
Desc: not available
Url : http://mail.python.org/pipermail/python-dev/attachments/20060712/d60b96c4/attachment.pgp 

From bborcic at gmail.com  Tue Jul 11 17:35:10 2006
From: bborcic at gmail.com (Boris Borcic)
Date: Tue, 11 Jul 2006 17:35:10 +0200
Subject: [Python-Dev] [slighly OT] Native speakers and hurting brains
Message-ID: <e90ggi$4q6$1@sea.gmane.org>

Fredrik Lundh wrote:
 > Boris Borcic wrote:
 >
 >>> in what language [is] the word "sum" an appropriate synonym for "concatenate" ?

 >> any that admits a+b to mean ''.join([a,b]), I'd say.
 >
 > and what human language would that be ?

Let's admit the answer is 'none' (and I apologize for accusing only English),
what is the impact on the idea that natural language intuition is a first rank
suspect, to explain the double standard in the Python treatment of str1+str2 and
sum([str1,str2]) ?

As for 'concatenate'. To my linguistic intuition it is an ugly elitist jargon
word made up for the function, like "we mean to say 'to chain' but don't want to
say it to your ears, unless you had latin classes". Admitting such a word as the
legal standard eliminates the possibility of synonyms. Makes me think, maybe
*that* word is the root of the problem, for being too ugly to find its way into
Python in the first place.

Cheers, BB


From bborcic at gmail.com  Tue Jul 11 18:21:20 2006
From: bborcic at gmail.com (Boris Borcic)
Date: Tue, 11 Jul 2006 18:21:20 +0200
Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
In-Reply-To: <e8ldjg$eq7$1@sea.gmane.org>
References: <e8k58c$q7u$1@sea.gmane.org>	<r01050400-1039-AEFC2D140DA011DBB8DF001124365170@[10.0.0.24]>
	<e8ldjg$eq7$1@sea.gmane.org>
Message-ID: <e90j6o$fij$1@sea.gmane.org>

Fredrik Lundh wrote:
> Just van Rossum wrote:
> 
>> Why couldn't at least augmented assignment be implicitly rebinding? It
>> has been suggested before (in the context of a rebinding operator), but
>> I'm wondering, is this also off the table?
>>
>>     def counter(num):
>>         def inc():
>>             num += 1
>>             return num
>>         return inc
>>
>> Reads very natural to me. It's likely the most frequent example of what
>> people try before they learn that rebinding to outer scopes isn't
>> allowed. It could Just Work.
> 
> note that most examples of this type already work, if the target type is 
> mutable, and implement the right operations:
> 
>       def counter(num):
>           num = mutable_int(num)
>           def inc():
>               num += 1
>               return num
>           return inc

I agree with you (and argued it in "scopes vs augmented assignment vs sets" 
recently) that mutating would be sufficient /if/ the compiler would view 
augmented assignment as mutations operators : which it doesn't as far as 
concerns scopes where a variable appears as target only of /augmented/ assignments.

Currently, the code you propose above will not work, and whatever your 
mutable_int() it will result in

UnboundLocalError: local variable 'num' referenced before assignment

What probably trips you is that the compiler thus makes a choice of 
interpretation that has no use cases.


Cheers, BB


From guido at python.org  Tue Jul 11 18:35:48 2006
From: guido at python.org (Guido van Rossum)
Date: Tue, 11 Jul 2006 09:35:48 -0700
Subject: [Python-Dev] [slighly OT] Native speakers and hurting brains
In-Reply-To: <e90ggi$4q6$1@sea.gmane.org>
References: <e90ggi$4q6$1@sea.gmane.org>
Message-ID: <ca471dc20607110935g333c6823m2400c98e2a37f465@mail.gmail.com>

Please end this thread. Now. Really.

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From anthony at interlink.com.au  Tue Jul 11 19:03:56 2006
From: anthony at interlink.com.au (Anthony Baxter)
Date: Wed, 12 Jul 2006 03:03:56 +1000
Subject: [Python-Dev] TRUNK is UNFROZEN.
Message-ID: <200607120303.59555.anthony@interlink.com.au>

beta2 is done, so trunk is unfrozen. Remember, we're still in feature 
freeze, so new features need approval before being committed. 

Thanks!
Anthony
-- 
Anthony Baxter     <anthony at interlink.com.au>
It's never too late to have a happy childhood.

From anthony at interlink.com.au  Tue Jul 11 19:05:44 2006
From: anthony at interlink.com.au (Anthony Baxter)
Date: Wed, 12 Jul 2006 03:05:44 +1000
Subject: [Python-Dev] Minor: Unix icons for 2.5?
Message-ID: <200607120305.45714.anthony@interlink.com.au>

There's an open PEP-356 issue for "update the icons to the newer 
shinier ones" for Unix. As far as I can see, there's the 14x15 GIF 
images used for Idle and the documentation. Note that for me at least, 
idle comes up without an icon _anyway_. 

Are there any others I missed?

Anthony
-- 
Anthony Baxter     <anthony at interlink.com.au>
It's never too late to have a happy childhood.

From g.brandl at gmx.net  Tue Jul 11 19:25:49 2006
From: g.brandl at gmx.net (Georg Brandl)
Date: Tue, 11 Jul 2006 19:25:49 +0200
Subject: [Python-Dev] Minor: Unix icons for 2.5?
In-Reply-To: <200607120305.45714.anthony@interlink.com.au>
References: <200607120305.45714.anthony@interlink.com.au>
Message-ID: <e90mut$sns$1@sea.gmane.org>

Anthony Baxter wrote:
> There's an open PEP-356 issue for "update the icons to the newer 
> shinier ones" for Unix. As far as I can see, there's the 14x15 GIF 
> images used for Idle and the documentation. Note that for me at least, 
> idle comes up without an icon _anyway_. 
> 
> Are there any others I missed?

In case we add a Python .desktop file (as proposed in patch #1353344),
we'll need some PNGs in /usr/share/icons. A patch for Makefile.pre.in
is attached.

Georg


Index: Misc/python.desktop.in
===================================================================
--- Misc/python.desktop.in      (Revision 0)
+++ Misc/python.desktop.in      (Revision 0)
@@ -0,0 +1,14 @@
+[Desktop Entry]
+Encoding=UTF-8
+Name=Python Programming Language
+Name[sv]=Programspr??ket Python
+Name[de]=Die Programmiersprache Python
+Comment=Program in Python
+Comment[sv]=Programmera med Python
+Comment[de]=Programmieren in Python
+Exec=python_VER_ %f
+Terminal=true
+Type=Application
+Icon=pycon.png
+Categories=Application;Development;ConsoleOnly
+MimeType=text/x-python;application/x-python
Index: Makefile.pre.in
===================================================================
--- Makefile.pre.in     (Revision 47289)
+++ Makefile.pre.in     (Arbeitskopie)
@@ -89,6 +89,8 @@
 INCLUDEDIR=    @includedir@
 CONFINCLUDEDIR=        $(exec_prefix)/include
 SCRIPTDIR=     $(prefix)/lib
+DESKTOPDIR=     $(prefix)/share/applications
+ICONDIR=        $(prefix)/share/icons

 # Detailed destination directories
 BINLIBDEST=    $(LIBDIR)/python$(VERSION)
@@ -614,7 +616,7 @@
                $(TESTPYTHON) $(TESTPROG) $(MEMTESTOPTS)

 # Install everything
-install:       @FRAMEWORKINSTALLFIRST@ altinstall bininstall maninstall
@FRAMEWORKINSTALLLAST@
+install:       @FRAMEWORKINSTALLFIRST@ altinstall bininstall maninstall
desktopinstall @FRAMEWORKINSTALLLAST@

 # Install almost everything without disturbing previous versions
 altinstall:    @FRAMEWORKALTINSTALLFIRST@ altbininstall libinstall inclinstall
libainstall \
@@ -687,6 +689,25 @@
        $(INSTALL_DATA) $(srcdir)/Misc/python.man \
                $(DESTDIR)$(MANDIR)/man1/python.1

+# Install the .desktop file and the icons
+desktopinstall:
+       @for i in $(DESKTOPDIR) $(ICONDIR) $(ICONDIR)/hicolor
$(ICONDIR)/hicolor/16x16 $(ICONDIR)/hicolor/32x32 $(ICONDIR)/hicolor/48x48; \
+       do \
+               if test ! -d $(DESTDIR)$$i; then \
+                       echo "Creating directory $$i"; \
+                       $(INSTALL) -d -m $(DIRMODE) $(DESTDIR)$$i; \
+               else    true; \
+               fi; \
+       done
+       sed 's/_VER_/$(VERSION)$(EXE)/' $(srcdir)/Misc/python.desktop.in >
$(srcdir)/Misc/python.desktop
+       $(INSTALL_DATA) $(srcdir)/Misc/python.desktop
$(DESTDIR)$(DESKTOPDIR)/python.desktop
+       for i in 16x16 32x32 48x48; \
+       do \
+               $(INSTALL_DATA) $(srcdir)/Misc/py-$$i.png
$(DESTDIR)$(ICONDIR)/hicolor/$$i/py.png; \
+               $(INSTALL_DATA) $(srcdir)/Misc/pyc-$$i.png
$(DESTDIR)$(ICONDIR)/hicolor/$$i/pyc.png; \
+               $(INSTALL_DATA) $(srcdir)/Misc/pycon-$$i.png
$(DESTDIR)$(ICONDIR)/hicolor/$$i/pycon.png; \
+       done
+
 # Install the library
 PLATDIR=       plat-$(MACHDEP)
 EXTRAPLATDIR= @EXTRAPLATDIR@
@@ -1076,7 +1097,7 @@
 # Declare targets that aren't real files
 .PHONY: all sharedmods oldsharedmods test quicktest memtest
 .PHONY: install altinstall oldsharedinstall bininstall altbininstall
-.PHONY: maninstall libinstall inclinstall libainstall sharedinstall
+.PHONY: maninstall libinstall inclinstall libainstall sharedinstall desktopinstall
 .PHONY: frameworkinstall frameworkinstallframework frameworkinstallstructure
 .PHONY: frameworkinstallmaclib frameworkinstallapps frameworkinstallunixtools
 .PHONY: frameworkaltinstallunixtools recheck autoconf clean clobber distclean


From scott+python-dev at scottdial.com  Tue Jul 11 19:30:17 2006
From: scott+python-dev at scottdial.com (Scott Dial)
Date: Tue, 11 Jul 2006 13:30:17 -0400
Subject: [Python-Dev] Capabilities / Restricted Execution
In-Reply-To: <5.1.1.6.0.20060711105343.01eed878@sparrow.telecommunity.com>
References: <5.1.1.6.0.20060711105343.01eed878@sparrow.telecommunity.com>
Message-ID: <44B3E029.5070604@scottdial.com>

Phillip J. Eby wrote:
> A function's func_closure contains cell objects that hold the 
> variables.  These are readable if you can set the func_closure of some 
> function of your own.  If the overall plan includes the ability to restrict 
> func_closure setting (or reading) in a restricted interpreter, then you 
> might be okay.

Except this function (__getattribute__) has been trapped inside of a
class which does not expose it as an attribute. So, you shouldn't be
able to get to the func_closure attribute of the __getattribute__
function for an instance of the Guard class. I can't come up with a way
to defeat this protection, at least. If you have a way, then I'd be
interested to hear it.

-- 
Scott Dial
scott at scottdial.com
scodial at indiana.edu


From rowen at cesmail.net  Tue Jul 11 19:32:01 2006
From: rowen at cesmail.net (Russell E. Owen)
Date: Tue, 11 Jul 2006 10:32:01 -0700
Subject: [Python-Dev] get for lists and tuples?
Message-ID: <rowen-735DB8.10320111072006@sea.gmane.org>

I'd like to have the get method available for lists and tuples. (I 
figured this must have been discussed before but can't recall it and 
didn't turn anything up on google).

It's obviously not a use-all-the-time method (or it'd already be there), 
but I find myself wanting it often enough to justify it in my own mind 
(and curse this omission, relative to dict).

Basically I run into it when parsing data of variable length (where the 
extra elements have some obvious default value), including config files, 
sys.argv (for simple command scripts), that sort of thing.

Yes a 4-liner does the job, but "get" would be a much clearer way to 
write it.

Anyway, I'm just testing the waters. If it's not heresy then I'd like to 
do what I can to make it happen.

-- Russell


From fdrake at acm.org  Tue Jul 11 19:33:07 2006
From: fdrake at acm.org (Fred L. Drake, Jr.)
Date: Tue, 11 Jul 2006 13:33:07 -0400
Subject: [Python-Dev] Minor: Unix icons for 2.5?
In-Reply-To: <e90mut$sns$1@sea.gmane.org>
References: <200607120305.45714.anthony@interlink.com.au>
	<e90mut$sns$1@sea.gmane.org>
Message-ID: <200607111333.07453.fdrake@acm.org>

Anthony Baxter wrote:
 > There's an open PEP-356 issue for "update the icons to the newer
 > shinier ones" for Unix. As far as I can see, there's the 14x15 GIF
 > images used for Idle and the documentation. Note that for me at least,
 > idle comes up without an icon _anyway_.

A pyfav.(gif|png) replacement would be quite welcome!

On Tuesday 11 July 2006 13:25, Georg Brandl wrote:
 > In case we add a Python .desktop file (as proposed in patch #1353344),
 > we'll need some PNGs in /usr/share/icons. A patch for Makefile.pre.in
 > is attached.

I know the .desktop files have become fairly standard, but are these our 
responsibility or does that rest with the distributions/integrators?  (I'm 
not objecting, but I'm not sure what the right thing really is since Python 
is an interpreter, not a desktop application.)


  -Fred

-- 
Fred L. Drake, Jr.   <fdrake at acm.org>

From alexander.belopolsky at gmail.com  Tue Jul 11 19:34:49 2006
From: alexander.belopolsky at gmail.com (Alexander Belopolsky)
Date: Tue, 11 Jul 2006 17:34:49 +0000 (UTC)
Subject: [Python-Dev] Doctest and Footnotes
References: <44B31CA7.7080906@zope.com>
Message-ID: <loom.20060711T193109-356@post.gmane.org>

Benji York <benji <at> zope.com> writes:


> 
> Here's the idea: when a footnote is referenced in prose, execute the 
> code associated with the footnote at that point.  For example:
>

Another natural place for the referenced code is the __test__ dictionary.
Using that has an advantage of not clobbering the display in the default
pydoc viewer.


From g.brandl at gmx.net  Tue Jul 11 19:39:56 2006
From: g.brandl at gmx.net (Georg Brandl)
Date: Tue, 11 Jul 2006 19:39:56 +0200
Subject: [Python-Dev] get for lists and tuples?
In-Reply-To: <rowen-735DB8.10320111072006@sea.gmane.org>
References: <rowen-735DB8.10320111072006@sea.gmane.org>
Message-ID: <e90npc$vkl$1@sea.gmane.org>

Russell E. Owen wrote:
> I'd like to have the get method available for lists and tuples. (I 
> figured this must have been discussed before but can't recall it and 
> didn't turn anything up on google).
> 
> It's obviously not a use-all-the-time method (or it'd already be there), 
> but I find myself wanting it often enough to justify it in my own mind 
> (and curse this omission, relative to dict).
> 
> Basically I run into it when parsing data of variable length (where the 
> extra elements have some obvious default value), including config files, 
> sys.argv (for simple command scripts), that sort of thing.
> 
> Yes a 4-liner does the job, but "get" would be a much clearer way to 
> write it.

A 4-liner?

x = (list[i:i+1] or (default,))[0]

is just one line ;) Honestly, often enough you can work around with slices
in one or the other way.

> Anyway, I'm just testing the waters. If it's not heresy then I'd like to 
> do what I can to make it happen.

IMO there's almost no chance this can go into 2.5.

Georg


From g.brandl at gmx.net  Tue Jul 11 19:42:53 2006
From: g.brandl at gmx.net (Georg Brandl)
Date: Tue, 11 Jul 2006 19:42:53 +0200
Subject: [Python-Dev] Minor: Unix icons for 2.5?
In-Reply-To: <200607111333.07453.fdrake@acm.org>
References: <200607120305.45714.anthony@interlink.com.au>	<e90mut$sns$1@sea.gmane.org>
	<200607111333.07453.fdrake@acm.org>
Message-ID: <e90nut$vkl$2@sea.gmane.org>

Fred L. Drake, Jr. wrote:
> Anthony Baxter wrote:
>  > There's an open PEP-356 issue for "update the icons to the newer
>  > shinier ones" for Unix. As far as I can see, there's the 14x15 GIF
>  > images used for Idle and the documentation. Note that for me at least,
>  > idle comes up without an icon _anyway_.
> 
> A pyfav.(gif|png) replacement would be quite welcome!
> 
> On Tuesday 11 July 2006 13:25, Georg Brandl wrote:
>  > In case we add a Python .desktop file (as proposed in patch #1353344),
>  > we'll need some PNGs in /usr/share/icons. A patch for Makefile.pre.in
>  > is attached.
> 
> I know the .desktop files have become fairly standard, but are these our 
> responsibility or does that rest with the distributions/integrators?  (I'm 
> not objecting, but I'm not sure what the right thing really is since Python 
> is an interpreter, not a desktop application.)

I'm also not very sure, as I myself would never start python via a menu entry.
But for newcomers, it could be as it is for Windows users: A menu entry that
starts the interpreter in a terminal window, and maybe one for IDLE too.

Georg


From anthony at interlink.com.au  Tue Jul 11 19:50:47 2006
From: anthony at interlink.com.au (Anthony Baxter)
Date: Wed, 12 Jul 2006 03:50:47 +1000
Subject: [Python-Dev] get for lists and tuples?
In-Reply-To: <e90npc$vkl$1@sea.gmane.org>
References: <rowen-735DB8.10320111072006@sea.gmane.org>
	<e90npc$vkl$1@sea.gmane.org>
Message-ID: <200607120350.51728.anthony@interlink.com.au>

On Wednesday 12 July 2006 03:39, Georg Brandl wrote:
> > Anyway, I'm just testing the waters. If it's not heresy then I'd
> > like to do what I can to make it happen.
>
> IMO there's almost no chance this can go into 2.5.

"almost"?

I'll go you one better. No way at all will it be in 2.5. And I'd be a 
firm -1 on adding in to 2.6 as well. It encourages bad coding. You 
shouldn't be searching lists and tuples like that unless you know 
what you're doing.

Anthony

-- 
Anthony Baxter     <anthony at interlink.com.au>
It's never too late to have a happy childhood.

From benji at zope.com  Tue Jul 11 19:57:23 2006
From: benji at zope.com (Benji York)
Date: Tue, 11 Jul 2006 13:57:23 -0400
Subject: [Python-Dev] Doctest and Footnotes
In-Reply-To: <loom.20060711T193109-356@post.gmane.org>
References: <44B31CA7.7080906@zope.com>
	<loom.20060711T193109-356@post.gmane.org>
Message-ID: <44B3E683.2060004@zope.com>

Alexander Belopolsky wrote:
> Benji York <benji <at> zope.com> writes:
>>Here's the idea: when a footnote is referenced in prose, execute the 
>>code associated with the footnote at that point.  For example:
>>
> 
> Another natural place for the referenced code is the __test__ dictionary.
> Using that has an advantage of not clobbering the display in the default
> pydoc viewer.

I'm not quite sure what you're suggesting.  A guess: put the code that 
isn't to be seen in the __test__ dict with a string key being the name 
of the footnote?  I don't think a ReST processor would like that much. 
It would see references to footnotes that are never defined.  Or perhaps 
you're suggesting a non-ReST mechanism for the references?

Also, for many of the use-cases we have, we do want the code in the 
test, just not in such a prominent place, and not repeated more than once.
-- 
Benji York
Senior Software Engineer
Zope Corporation

From brett at python.org  Tue Jul 11 19:56:47 2006
From: brett at python.org (Brett Cannon)
Date: Tue, 11 Jul 2006 10:56:47 -0700
Subject: [Python-Dev] In defense of Capabilities [was: doc for new
	restricted execution design for Python]
In-Reply-To: <44B34536.7040907@acm.org>
References: <20060705155149.eipxm6j6oycgkcs8@login.werra.lunarpages.com>
	<ca471dc20607071343j1ba9ac7bj33ebc9071e5ebbf7@mail.gmail.com>
	<bbaeab100607071355r67b291b0t285774667025e165@mail.gmail.com>
	<44AF26A4.2040201@acm.org>
	<Pine.LNX.4.58.0607072321000.22834@server1.LFW.org>
	<ca471dc20607072145y5df22ea7h1b7fb4414fbd11b9@mail.gmail.com>
	<bbaeab100607081948q5aee5485o797a32a06e5c2daf@mail.gmail.com>
	<44B074C6.3070701@acm.org>
	<bbaeab100607091052g4a84a5c8gedea016012ed8c08@mail.gmail.com>
	<44B34536.7040907@acm.org>
Message-ID: <bbaeab100607111056h6d29b0b1v3d3773274be3cda8@mail.gmail.com>

On 7/10/06, Talin <talin at acm.org> wrote:
>
> Brett Cannon wrote:
> > Using a factory method callback, one could store the PyCodeObject in a C
> > proxy object that just acts as a complete delegate, forwarding all
> method
> > calls to the internally stored PyCodeObject.  That would work.
> >
> >
> > For this initial implementation, though, I am not going to try to
> support
> > this.   We can always add support like this later since it doesn't
> > fundamentally break or change anything that is already planned.  Let's
> > focus
> > on getting even more basic stuff working before we start to get too
> fancy.
> >
> > -Brett
>
> Thinking about this some more, I've come up with a simpler idea for a
> generic wrapper class. The wrapper consists of two parts: a decorator to
> indicate that a given method is 'public', and a C 'guard' wrapper that
> insures that only 'public' members can be accessed.
>
> So for example:
>
>     from Sandbox import guard, public
>
>     class FileProxy:
>        # Public method, no decorator


Is that supposed to say "no decorator", even though there is one?

       @public
>        def read( length ):
>           ...
>
>        @public
>        def seek( offset ):
>           ...
>
>        # Private method, no decorator
>        def open( path ):
>           ...
>
>     # Construct an instance of FileProxy, and wrap a
>     # guard object around it. Any attempt to access a non-public
>     # attribute will raise an exception (or perhaps simply report
>     # that the attribute doesn't exist.)
>     fh = guard( FileProxy() )


But wouldn't you still need to protect things like object.__metaclasses__()
so that you can't get access to the class and mutate it?  And if you use a
custom metaclass to avoid having object expose the reference you then would
need to protect the metaclass.

I guess I would need to see an implementation to see if there is any way to
get around security protections since it is still Python code.

Now, from my point of view this *is* 'the basic stuff'. In other words,
> this right here is the fundamental sandbox mechanism, and everything
> else is built on top of it.
>
> Now, the C 'guard' function is only a low-level means to insure that
> no-one can mess with the object; It is not intended to be the actual
> restriction policy itself. Those are placed in the wrapper classes, just
> as in your proposed scheme.
>
> (This goes back to my basic premise, that a simple "yes/no" security
> feature can be used to build up much more complex and subtle security
> features.)


Yeah, this tends to be true.

-Brett

The only real complexity of this, as I see it, is that methods to
> references will have to be themselves wrapped.
>
> In other words, if I say 'fh.read' without the (), what I get back can't
> be the actual read function object - that would be too easy to fiddle
> with. What I'd have to get is a wrapped version of the method that is a
> callable.
>
> One relatively simply way to deal with this is to have the 'public'
> decorator create a C wrapper for the function object, and store it as an
> attribute of the function. The class wrapper then simply looks up the
> attribute, and if it has an attribute wrapper, returns that, otherwise
> it fails.
>
> (Although, I've often wished for Python to have a variant of __call__
> that could be used to override individual methods, i.e.:
>
>      __call_method__( self, methodname, *args )
>
> This would make the guard wrapper much easier to construct, since we
> could restrict the methods only to being called, and not allow
> references to methods to be taken.)
>
> -- Talin
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/brett%40python.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20060711/b11958de/attachment.html 

From pje at telecommunity.com  Tue Jul 11 20:09:37 2006
From: pje at telecommunity.com (Phillip J. Eby)
Date: Tue, 11 Jul 2006 14:09:37 -0400
Subject: [Python-Dev] Capabilities / Restricted Execution
In-Reply-To: <44B3E029.5070604@scottdial.com>
References: <5.1.1.6.0.20060711105343.01eed878@sparrow.telecommunity.com>
	<5.1.1.6.0.20060711105343.01eed878@sparrow.telecommunity.com>
Message-ID: <5.1.1.6.0.20060711140809.03aa2818@sparrow.telecommunity.com>

At 01:30 PM 7/11/2006 -0400, Scott Dial wrote:
>Phillip J. Eby wrote:
> > A function's func_closure contains cell objects that hold the
> > variables.  These are readable if you can set the func_closure of some
> > function of your own.  If the overall plan includes the ability to 
> restrict
> > func_closure setting (or reading) in a restricted interpreter, then you
> > might be okay.
>
>Except this function (__getattribute__) has been trapped inside of a
>class which does not expose it as an attribute. So, you shouldn't be
>able to get to the func_closure attribute of the __getattribute__
>function for an instance of the Guard class.

That doesn't matter, because it's the *returned* function's func_closure 
that's at issue.  That is, proxy_attr.func_closure[0] is the cell for the 
'attr' value.


From talin at acm.org  Tue Jul 11 20:10:34 2006
From: talin at acm.org (Talin)
Date: Tue, 11 Jul 2006 11:10:34 -0700
Subject: [Python-Dev] Capabilities / Restricted Execution
In-Reply-To: <44B3E029.5070604@scottdial.com>
References: <5.1.1.6.0.20060711105343.01eed878@sparrow.telecommunity.com>
	<44B3E029.5070604@scottdial.com>
Message-ID: <44B3E99A.4080000@acm.org>

Scott Dial wrote:
> Phillip J. Eby wrote:
> 
>>A function's func_closure contains cell objects that hold the 
>>variables.  These are readable if you can set the func_closure of some 
>>function of your own.  If the overall plan includes the ability to restrict 
>>func_closure setting (or reading) in a restricted interpreter, then you 
>>might be okay.
> 
> 
> Except this function (__getattribute__) has been trapped inside of a
> class which does not expose it as an attribute. So, you shouldn't be
> able to get to the func_closure attribute of the __getattribute__
> function for an instance of the Guard class. I can't come up with a way
> to defeat this protection, at least. If you have a way, then I'd be
> interested to hear it.

I've thought of several ways to break it already. Some are repairable, 
I'm not sure that they all are.

For example, neither of the following statements blows up:

    print t2.get_name.func_closure[0]
    print object.__getattribute__( t2, '__dict__' )

Still, its perhaps a useful basis for experimentation.

-- Talin

From alexander.belopolsky at gmail.com  Tue Jul 11 20:12:19 2006
From: alexander.belopolsky at gmail.com (Alexander Belopolsky)
Date: Tue, 11 Jul 2006 14:12:19 -0400
Subject: [Python-Dev] Doctest and Footnotes
In-Reply-To: <44B3E683.2060004@zope.com>
References: <44B31CA7.7080906@zope.com>
	<loom.20060711T193109-356@post.gmane.org> <44B3E683.2060004@zope.com>
Message-ID: <d38f5330607111112x74d5a57fl9d9d105cd52f075c@mail.gmail.com>

On 7/11/06, Benji York <benji at zope.com> wrote:
[snip]
> I'm not quite sure what you're suggesting.  A guess: put the code that
> isn't to be seen in the __test__ dict with a string key being the name
> of the footnote?

That's right.

>  I don't think a ReST processor would like that much.
> It would see references to footnotes that are never defined.  Or perhaps
> you're suggesting a non-ReST mechanism for the references?
>

I don't know how ReST processor is used.  If you just filter the
output of pydoc through a ReST processor, then you are right about
undefined references. If, however, ReST processing is implemented
inside pydoc, I don't see any problem in implementing __test__ lookup.

> Also, for many of the use-cases we have, we do want the code in the
> test, just not in such a prominent place, and not repeated more than once.

I my use-cases, testing code is clobbering the documentation, but
there is no easy way to move it outside of the docstrings without
breaking the order of evaluation.  I don't use a ReST processor, by I
can read ReST formatted text with little difficulty. I would greatly
appreciate if I could clean the docstrings without loosing the tests.

BTW, another feature that I would greatly appreciate would be a
unittest wrapper which makes each docstring in a separate test case.
Also __new__ and __init__ method docstrings is the natural place to
put set-up code.

From fred at zope.com  Tue Jul 11 20:17:00 2006
From: fred at zope.com (Fred Drake)
Date: Tue, 11 Jul 2006 14:17:00 -0400
Subject: [Python-Dev] Doctest and Footnotes
In-Reply-To: <d38f5330607111112x74d5a57fl9d9d105cd52f075c@mail.gmail.com>
References: <44B31CA7.7080906@zope.com> <44B3E683.2060004@zope.com>
	<d38f5330607111112x74d5a57fl9d9d105cd52f075c@mail.gmail.com>
Message-ID: <200607111417.01500.fred@zope.com>

On Tuesday 11 July 2006 14:12, Alexander Belopolsky wrote:
 > Also __new__ and __init__ method docstrings is the natural place to
 > put set-up code.

Maybe, if all the tests required the same setup code.  That's often not the 
case.


  -Fred

-- 
Fred L. Drake, Jr.  <fred at zope.com>
Zope Corporation

From g.brandl at gmx.net  Tue Jul 11 20:17:35 2006
From: g.brandl at gmx.net (Georg Brandl)
Date: Tue, 11 Jul 2006 20:17:35 +0200
Subject: [Python-Dev] get for lists and tuples?
In-Reply-To: <200607120350.51728.anthony@interlink.com.au>
References: <rowen-735DB8.10320111072006@sea.gmane.org>	<e90npc$vkl$1@sea.gmane.org>
	<200607120350.51728.anthony@interlink.com.au>
Message-ID: <e90q00$937$1@sea.gmane.org>

Anthony Baxter wrote:
> On Wednesday 12 July 2006 03:39, Georg Brandl wrote:
>> > Anyway, I'm just testing the waters. If it's not heresy then I'd
>> > like to do what I can to make it happen.
>>
>> IMO there's almost no chance this can go into 2.5.
> 
> "almost"?
> 
> I'll go you one better. No way at all will it be in 2.5. And I'd be a 
> firm -1 on adding in to 2.6 as well. It encourages bad coding. You 
> shouldn't be searching lists and tuples like that unless you know 
> what you're doing.

NB: I did not support adding that method, only notified the OP of the
2.5 situation.

Georg


From rhettinger at ewtllc.com  Tue Jul 11 20:19:48 2006
From: rhettinger at ewtllc.com (Raymond Hettinger)
Date: Tue, 11 Jul 2006 11:19:48 -0700
Subject: [Python-Dev] get for lists and tuples?
In-Reply-To: <rowen-735DB8.10320111072006@sea.gmane.org>
References: <rowen-735DB8.10320111072006@sea.gmane.org>
Message-ID: <44B3EBC4.3090703@ewtllc.com>

Russell E. Owen wrote:

>I'd like to have the get method available for lists and tuples. (I 
>figured this must have been discussed before but can't recall it and 
>didn't turn anything up on google).
>
>It's obviously not a use-all-the-time method (or it'd already be there), 
>but I find myself wanting it often enough to justify it in my own mind 
>(and curse this omission, relative to dict).
>
>Basically I run into it when parsing data of variable length (where the 
>extra elements have some obvious default value), including config files, 
>sys.argv (for simple command scripts), that sort of thing.
>  
>
How about:

    optarg = argv[4] if len(argv)>4 else 'default'


>Yes a 4-liner does the job, but "get" would be a much clearer way to 
>write it.
>
>Anyway, I'm just testing the waters. If it's not heresy then I'd like to 
>do what I can to make it happen.
>  
>

Perhaps in an alternate universe ;-)


Raymond


From pje at telecommunity.com  Tue Jul 11 20:29:35 2006
From: pje at telecommunity.com (Phillip J. Eby)
Date: Tue, 11 Jul 2006 14:29:35 -0400
Subject: [Python-Dev] In defense of Capabilities [was: doc for new
 restricted execution design for Python]
In-Reply-To: <bbaeab100607111056h6d29b0b1v3d3773274be3cda8@mail.gmail.co
 m>
References: <44B34536.7040907@acm.org>
	<20060705155149.eipxm6j6oycgkcs8@login.werra.lunarpages.com>
	<ca471dc20607071343j1ba9ac7bj33ebc9071e5ebbf7@mail.gmail.com>
	<bbaeab100607071355r67b291b0t285774667025e165@mail.gmail.com>
	<44AF26A4.2040201@acm.org>
	<Pine.LNX.4.58.0607072321000.22834@server1.LFW.org>
	<ca471dc20607072145y5df22ea7h1b7fb4414fbd11b9@mail.gmail.com>
	<bbaeab100607081948q5aee5485o797a32a06e5c2daf@mail.gmail.com>
	<44B074C6.3070701@acm.org>
	<bbaeab100607091052g4a84a5c8gedea016012ed8c08@mail.gmail.com>
	<44B34536.7040907@acm.org>
Message-ID: <5.1.1.6.0.20060711141257.03e8de00@sparrow.telecommunity.com>

At 10:56 AM 7/11/2006 -0700, Brett Cannon wrote:
>On 7/10/06, Talin <<mailto:talin at acm.org>talin at acm.org> wrote:
>>(Although, I've often wished for Python to have a variant of __call__
>>that could be used to override individual methods, i.e.:
>>
>>      __call_method__( self, methodname, *args )

As with so many other things in this discussion, this was already invented 
by the Zope folks just shy of a decade ago.  ;-)  __call_method__ is 
actually a feature of ExtensionClasses, although you can of course 
implement it yourself now atop new-style classes.

For other things that Zope has already done in the area of restricted 
execution R&D, see:

http://svn.zope.org/Zope3/trunk/src/zope/security/untrustedinterpreter.txt?view=auto

The surrounding directory has other documents regarding their approach.

I haven't been following this discussion closely, but a lot of the things 
mentioned in this thread seem to have a lot in common with stuff the Zope 
folks have had in production for "untrusted" Python execution for some time 
now, working with current versions of Python.  It would be a shame to 
reinvent all the same wheels, especially since their code is nicely 
documented complete with extensive doctests and explanations of the approach.


From pje at telecommunity.com  Tue Jul 11 20:32:17 2006
From: pje at telecommunity.com (Phillip J. Eby)
Date: Tue, 11 Jul 2006 14:32:17 -0400
Subject: [Python-Dev] Doctest and Footnotes
In-Reply-To: <d38f5330607111112x74d5a57fl9d9d105cd52f075c@mail.gmail.com
 >
References: <44B3E683.2060004@zope.com> <44B31CA7.7080906@zope.com>
	<loom.20060711T193109-356@post.gmane.org>
	<44B3E683.2060004@zope.com>
Message-ID: <5.1.1.6.0.20060711143002.03e9c800@sparrow.telecommunity.com>

At 02:12 PM 7/11/2006 -0400, Alexander Belopolsky wrote:
>On 7/11/06, Benji York <benji at zope.com> wrote:
>[snip]
> > I'm not quite sure what you're suggesting.  A guess: put the code that
> > isn't to be seen in the __test__ dict with a string key being the name
> > of the footnote?
>
>That's right.
>
> >  I don't think a ReST processor would like that much.
> > It would see references to footnotes that are never defined.  Or perhaps
> > you're suggesting a non-ReST mechanism for the references?
> >
>
>I don't know how ReST processor is used.  If you just filter the
>output of pydoc through a ReST processor, then you are right about
>undefined references.

A common use case for doctests in Zope is separate text files that are used 
as documentation.  I also use this approach for generating documentation 
myself, e.g.:

http://chandler.osafoundation.org/docs/0.7/parcel-schema-guide.html
http://chandler.osafoundation.org/docs/0.7/running-code-at-startup.html
http://peak.telecommunity.com/DevCenter/BytecodeAssembler

To name just a few.


From brett at python.org  Tue Jul 11 20:35:23 2006
From: brett at python.org (Brett Cannon)
Date: Tue, 11 Jul 2006 11:35:23 -0700
Subject: [Python-Dev] In defense of Capabilities [was: doc for new
	restricted execution design for Python]
In-Reply-To: <5.1.1.6.0.20060711141257.03e8de00@sparrow.telecommunity.com>
References: <20060705155149.eipxm6j6oycgkcs8@login.werra.lunarpages.com>
	<bbaeab100607071355r67b291b0t285774667025e165@mail.gmail.com>
	<44AF26A4.2040201@acm.org>
	<Pine.LNX.4.58.0607072321000.22834@server1.LFW.org>
	<ca471dc20607072145y5df22ea7h1b7fb4414fbd11b9@mail.gmail.com>
	<bbaeab100607081948q5aee5485o797a32a06e5c2daf@mail.gmail.com>
	<44B074C6.3070701@acm.org>
	<bbaeab100607091052g4a84a5c8gedea016012ed8c08@mail.gmail.com>
	<44B34536.7040907@acm.org>
	<5.1.1.6.0.20060711141257.03e8de00@sparrow.telecommunity.com>
Message-ID: <bbaeab100607111135i2831704xd092ccf456023e1a@mail.gmail.com>

On 7/11/06, Phillip J. Eby <pje at telecommunity.com> wrote:
>
> At 10:56 AM 7/11/2006 -0700, Brett Cannon wrote:
> >On 7/10/06, Talin <<mailto:talin at acm.org>talin at acm.org> wrote:
> >>(Although, I've often wished for Python to have a variant of __call__
> >>that could be used to override individual methods, i.e.:
> >>
> >>      __call_method__( self, methodname, *args )
>
> As with so many other things in this discussion, this was already invented
> by the Zope folks just shy of a decade ago.  ;-)  __call_method__ is
> actually a feature of ExtensionClasses, although you can of course
> implement it yourself now atop new-style classes.
>
> For other things that Zope has already done in the area of restricted
> execution R&D, see:
>
>
> http://svn.zope.org/Zope3/trunk/src/zope/security/untrustedinterpreter.txt?view=auto
>
> The surrounding directory has other documents regarding their approach.
>
> I haven't been following this discussion closely, but a lot of the things
> mentioned in this thread seem to have a lot in common with stuff the Zope
> folks have had in production for "untrusted" Python execution for some
> time
> now, working with current versions of Python.  It would be a shame to
> reinvent all the same wheels, especially since their code is nicely
> documented complete with extensive doctests and explanations of the
> approach.


Taking a proxy approach is just being discussed; it has not been decided as
the proper solution.  I have been considering just preventing direct 'file'
access and using open() to act as a delegate for opening files.  That
approach has nothing to do with proxies.

-Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20060711/88cfec06/attachment.html 

From alexander.belopolsky at gmail.com  Tue Jul 11 20:37:37 2006
From: alexander.belopolsky at gmail.com (Alexander Belopolsky)
Date: Tue, 11 Jul 2006 14:37:37 -0400
Subject: [Python-Dev] Doctest and Footnotes
In-Reply-To: <200607111417.01500.fred@zope.com>
References: <44B31CA7.7080906@zope.com> <44B3E683.2060004@zope.com>
	<d38f5330607111112x74d5a57fl9d9d105cd52f075c@mail.gmail.com>
	<200607111417.01500.fred@zope.com>
Message-ID: <d38f5330607111137oc3934c7r6b82e98bb40d8d3c@mail.gmail.com>

On 7/11/06, Fred Drake <fred at zope.com> wrote:
> On Tuesday 11 July 2006 14:12, Alexander Belopolsky wrote:
>  > Also __new__ and __init__ method docstrings is the natural place to
>  > put set-up code.
>
> Maybe, if all the tests required the same setup code.  That's often not the
> case.

That's true, but you cannot test an object method without creating the
object first.  For the main cases you would want the object definition
close to the test for the benefit of people who do pydoc Foo.bar, but
for corner cases it is better to have a predefined set of exotic
objects available under descriptive names.  I am also advocating
__init__ docstring because in my use-cases it rarely much of
documentation because object construction is documented in the class
docstring, therefore test set-up code placed in __init__ docstring is
unlikely to clobber often-used documentation.

From jjl at pobox.com  Tue Jul 11 20:43:22 2006
From: jjl at pobox.com (John J Lee)
Date: Tue, 11 Jul 2006 19:43:22 +0100 (GMT Standard Time)
Subject: [Python-Dev] urllib.quote and unicode bug resuscitation attempt
In-Reply-To: <44B3ADE2.7040108@ofai.at>
References: <44B3ADE2.7040108@ofai.at>
Message-ID: <Pine.WNT.4.64.0607111906390.464@shaolin>

On Tue, 11 Jul 2006, Stefan Rank wrote:

> urllib.quote fails on unicode strings and in an unhelpful way::
[...]
>   >>> urllib.quote(u'a\xf1a')
>   Traceback (most recent call last):
>     File "<stdin>", line 1, in ?
>     File "C:\Python24\lib\urllib.py", line 1117, in quote
>       res = map(safe_map.__getitem__, s)
>   KeyError: u'\xf1'

More helpful than silently producing the wrong answer.


[...]
> I suggest to add (after 2.5 I assume) one of the following to the
> beginning of urllib.quote to either fail early and consistently on
> unicode arguments and improve the error message::
>
>   if isinstance(s, unicode):
>       raise TypeError("quote needs a byte string argument, not unicode,"
>                       " use `argument.encode('utf-8')` first.")

Won't this break existing code that catches the KeyError, for no big 
benefit?  If nobody is yet sure what the Right Thing is (see below), I 
think we should not change this yet.


> or to do The Right Thing (tm), which is utf-8 encoding::
>
>   if isinstance(s, unicode):
>       s = s.encode('utf-8')
>
> as suggested in
> http://www.w3.org/International/O-URL-code.html
> and rfc3986.

You seem quite confident of that.  You may be correct, but have you read 
all of the following?  (not trying to claim superior knowledge by asking 
that, I just dunno what the right thing is yet: I haven't yet read RFC 
2617 or got my head around what the unicode issues are or how they should 
apply to the Python stdlib)

http://www.ietf.org/rfc/rfc2617.txt

http://www.ietf.org/rfc/rfc2616.txt

http://en.wikipedia.org/wiki/Percent-encoding

http://mail.python.org/pipermail/python-dev/2004-September/048944.html


Also note the recent discussions here about a module named "uriparse" or 
"urischemes", which fits in to this somewhere.  It would be good to make 
all the following changes in a single Python release (2.6, with luck):

  - extend / modify urllib and urllib2 to handle unicode input

  - address the urllib.quote issue you raise above (+ consider the other
    utility functions in that module)

  - add the urischemes module


In summary, I agree that your suggested fix (and all of the rest I refer 
to above) should wait for 2.6, unless somebody (Martin?) who understands 
all these issues is quite confident your suggested change is OK. 
Presumably the release managers wouldn't allow it in 2.5 anyway.


John

From fred at zope.com  Tue Jul 11 20:52:29 2006
From: fred at zope.com (Fred Drake)
Date: Tue, 11 Jul 2006 14:52:29 -0400
Subject: [Python-Dev] Doctest and Footnotes
In-Reply-To: <d38f5330607111137oc3934c7r6b82e98bb40d8d3c@mail.gmail.com>
References: <44B31CA7.7080906@zope.com> <200607111417.01500.fred@zope.com>
	<d38f5330607111137oc3934c7r6b82e98bb40d8d3c@mail.gmail.com>
Message-ID: <200607111452.30118.fred@zope.com>

On Tuesday 11 July 2006 14:37, Alexander Belopolsky wrote:
 > That's true, but you cannot test an object method without creating the
 > object first.  

True.  How the object is created can vary; if the creation affects the 
expected behavior in any way, you'll need be careful about how the 
constructor is called for that test.  In fact, __init__ isn't always the 
desired constructor; some other class method might be used.

 > For the main cases you would want the object definition 
 > close to the test for the benefit of people who do pydoc Foo.bar, but
 > for corner cases it is better to have a predefined set of exotic
 > objects available under descriptive names.  

This is subjective.  That's certainly one way of organizing the tests, and 
I've no problem with it, but some of us use a more narrative approach, and we 
don't want our test structure to be constrained by the layout of the source 
code.

Gary's idea about using footnotes to aid in structuring tests is only about 
enabling a particular approach to structure, not about requiring it.  If it 
doesn't fit your needs, there's certainly no need to use it.  For more 
elaborate test structures, I suspect it will prove quite useful.  I also 
expect it will be more helpful for tests in separate text files than for 
tests embedded in source code.


  -Fred

-- 
Fred L. Drake, Jr.  <fred at zope.com>
Zope Corporation

From pje at telecommunity.com  Tue Jul 11 20:59:10 2006
From: pje at telecommunity.com (Phillip J. Eby)
Date: Tue, 11 Jul 2006 14:59:10 -0400
Subject: [Python-Dev] In defense of Capabilities [was: doc for new
 restricted execution design for Python]
In-Reply-To: <bbaeab100607111135i2831704xd092ccf456023e1a@mail.gmail.com
 >
References: <5.1.1.6.0.20060711141257.03e8de00@sparrow.telecommunity.com>
	<20060705155149.eipxm6j6oycgkcs8@login.werra.lunarpages.com>
	<bbaeab100607071355r67b291b0t285774667025e165@mail.gmail.com>
	<44AF26A4.2040201@acm.org>
	<Pine.LNX.4.58.0607072321000.22834@server1.LFW.org>
	<ca471dc20607072145y5df22ea7h1b7fb4414fbd11b9@mail.gmail.com>
	<bbaeab100607081948q5aee5485o797a32a06e5c2daf@mail.gmail.com>
	<44B074C6.3070701@acm.org>
	<bbaeab100607091052g4a84a5c8gedea016012ed8c08@mail.gmail.com>
	<44B34536.7040907@acm.org>
	<5.1.1.6.0.20060711141257.03e8de00@sparrow.telecommunity.com>
Message-ID: <5.1.1.6.0.20060711145232.03c8c0e0@sparrow.telecommunity.com>

At 11:35 AM 7/11/2006 -0700, Brett Cannon wrote:
>On 7/11/06, Phillip J. Eby 
><<mailto:pje at telecommunity.com>pje at telecommunity.com> wrote:
>>At 10:56 AM 7/11/2006 -0700, Brett Cannon wrote:
>> >On 7/10/06, Talin 
>> <<mailto:talin at acm.org><mailto:talin at acm.org>talin at acm.org> wrote:
>> >>(Although, I've often wished for Python to have a variant of __call__
>> >>that could be used to override individual methods, i.e.:
>> >>
>> >>      __call_method__( self, methodname, *args )
>>
>>As with so many other things in this discussion, this was already invented
>>by the Zope folks just shy of a decade ago.  ;-)  __call_method__ is
>>actually a feature of ExtensionClasses, although you can of course
>>implement it yourself now atop new-style classes.
>>
>>For other things that Zope has already done in the area of restricted
>>execution R&D, see:
>>
>><http://svn.zope.org/Zope3/trunk/src/zope/security/untrustedinterpreter.txt?view=auto>http://svn.zope.org/Zope3/trunk/src/zope/security/untrustedinterpreter.txt?view=auto
>>
>>The surrounding directory has other documents regarding their approach.
>>
>>I haven't been following this discussion closely, but a lot of the things
>>mentioned in this thread seem to have a lot in common with stuff the Zope
>>folks have had in production for "untrusted" Python execution for some time
>>now, working with current versions of Python.  It would be a shame to
>>reinvent all the same wheels, especially since their code is nicely
>>documented complete with extensive doctests and explanations of the approach.
>
>Taking a proxy approach is just being discussed; it has not been decided 
>as the proper solution.

My point is more to the study of the requirements for untrusted execution, 
and what options are open within the existing CPython architecture.  Many 
aspects of the Zope untrusted interpreter do *not* involve proxies; there 
are quite a few simple types that Zope's untrusted interpreter allows 
direct access to.  See the section on "Basic objects" in the link above.


From jimjjewett at gmail.com  Tue Jul 11 22:40:10 2006
From: jimjjewett at gmail.com (Jim Jewett)
Date: Tue, 11 Jul 2006 16:40:10 -0400
Subject: [Python-Dev] PEP 356: python.org/sf/1515343 resolution
Message-ID: <fb6fbf560607111340i73adc8fcu61ea57927b817778@mail.gmail.com>

python.org/sf/1515343 fixes a regression against 2.4, which masks
exceptions, and should therefore go in before release.  The tracker
has a patch to the test cases, and two alternative fixes.  (One
minimal, the other less ugly.)

Since this only affects string exceptions with a value, it might be
acceptable to just say "won't ever fix", but then it should be added
to the upgrade news.

-jJ

From martin at v.loewis.de  Tue Jul 11 23:16:21 2006
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Tue, 11 Jul 2006 23:16:21 +0200
Subject: [Python-Dev] urllib.quote and unicode bug resuscitation attempt
In-Reply-To: <44B3ADE2.7040108@ofai.at>
References: <44B3ADE2.7040108@ofai.at>
Message-ID: <44B41525.3040004@v.loewis.de>

Stefan Rank wrote:
> I suggest to add (after 2.5 I assume) one of the following to the 
> beginning of urllib.quote to either fail early and consistently on 
> unicode arguments and improve the error message::
> 
>    if isinstance(s, unicode):
>        raise TypeError("quote needs a byte string argument, not unicode,"
>                        " use `argument.encode('utf-8')` first.")
> 
> or to do The Right Thing (tm), which is utf-8 encoding::

The right thing to do is IRIs. This is more complicated than encoding
the Unicode string as UTF-8, though: for the host part of the URL, you
have to encode it with IDNA (and there are additional complicated rules
in place, e.g. when the Unicode string already contains %).

Contributions are welcome, as long as they fix this entire issue "for
good" (i.e. in all URL-processing code, and considering all relevant
RFCs).

Regards,
Martin

From tjreedy at udel.edu  Tue Jul 11 23:23:51 2006
From: tjreedy at udel.edu (Terry Reedy)
Date: Tue, 11 Jul 2006 17:23:51 -0400
Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
References: <e8k58c$q7u$1@sea.gmane.org>	<r01050400-1039-AEFC2D140DA011DBB8DF001124365170@[10.0.0.24]><e8ldjg$eq7$1@sea.gmane.org>
	<e90j6o$fij$1@sea.gmane.org>
Message-ID: <e914t8$gsv$1@sea.gmane.org>


"Boris Borcic" <bborcic at gmail.com> wrote in message 
news:e90j6o$fij$1 at sea.gmane.org...
> I agree with you (and argued it in "scopes vs augmented assignment vs 
> sets"
> recently) that mutating would be sufficient /if/ the compiler would view
> augmented assignment as mutations operators :

Mutation is an operation on objects.  Binding is an operation on 
namespaces.  The difference between objects and namespaces (and the actions 
thereupon) is fundamental to Python.  Asking the interpreter to view one 
thing as something else which it isn't can only lead to more confusion.  In 
particular, asking that arithmetic operations on immutable numbers be seen 
as mutations seems wacky to me.

> which it doesn't as far as  concerns scopes where a variable
> appears as target only of /augmented/ assignments.

The interpreter/compiler, as far as I can think, never views binding as 
mutation, nor should it.  The request that it do so makes me wonder whether 
it might have been a mistake to allow mutable objects in an augmented 
assignment to choose to implement the associated operation as an in-place 
mutation.

Terry Jan Reedy





From martin at v.loewis.de  Tue Jul 11 23:28:58 2006
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Tue, 11 Jul 2006 23:28:58 +0200
Subject: [Python-Dev] Minor: Unix icons for 2.5?
In-Reply-To: <e90mut$sns$1@sea.gmane.org>
References: <200607120305.45714.anthony@interlink.com.au>
	<e90mut$sns$1@sea.gmane.org>
Message-ID: <44B4181A.1070506@v.loewis.de>

Georg Brandl wrote:
> In case we add a Python .desktop file (as proposed in patch #1353344),
> we'll need some PNGs in /usr/share/icons. A patch for Makefile.pre.in
> is attached.

Independent of whether this should be done at all, I have a comment on
the patch. Instead of invoking sed, I would use autoconf machinery here

> +Exec=python_VER_ %f

This would become

Exec=python at VERSION@@EXEEXT@ %f

> +       sed 's/_VER_/$(VERSION)$(EXE)/' $(srcdir)/Misc/python.desktop.in >
> $(srcdir)/Misc/python.desktop

This could go away, and configure should grow

AC_CONFIG_FILES(Makefile.pre Modules/Setup.config Misc/python.desktop)



Regards,
Martin

From bjourne at gmail.com  Tue Jul 11 23:39:33 2006
From: bjourne at gmail.com (=?ISO-8859-1?Q?BJ=F6rn_Lindqvist?=)
Date: Tue, 11 Jul 2006 23:39:33 +0200
Subject: [Python-Dev] Minor: Unix icons for 2.5?
In-Reply-To: <200607111333.07453.fdrake@acm.org>
References: <200607120305.45714.anthony@interlink.com.au>
	<e90mut$sns$1@sea.gmane.org> <200607111333.07453.fdrake@acm.org>
Message-ID: <740c3aec0607111439h691c4450i9aa52fa00eb73072@mail.gmail.com>

> I know the .desktop files have become fairly standard, but are these our
> responsibility or does that rest with the distributions/integrators?  (I'm
> not objecting, but I'm not sure what the right thing really is since Python
> is an interpreter, not a desktop application.)

The same anal argument could of course be made for every piece of
software. Is Emacs a desktop application or a lisp interpreter? Is vim
a desktop app? Are games? There is no reason add yet another task to
to multiple distributors/integrators (that they won't do) when the
problem can be fixed at the source.

-- 
mvh Bj?rn

From g.brandl at gmx.net  Tue Jul 11 23:43:24 2006
From: g.brandl at gmx.net (Georg Brandl)
Date: Tue, 11 Jul 2006 23:43:24 +0200
Subject: [Python-Dev] Minor: Unix icons for 2.5?
In-Reply-To: <44B4181A.1070506@v.loewis.de>
References: <200607120305.45714.anthony@interlink.com.au>	<e90mut$sns$1@sea.gmane.org>
	<44B4181A.1070506@v.loewis.de>
Message-ID: <e9161s$k88$1@sea.gmane.org>

Martin v. L?wis wrote:
> Georg Brandl wrote:
>> In case we add a Python .desktop file (as proposed in patch #1353344),
>> we'll need some PNGs in /usr/share/icons. A patch for Makefile.pre.in
>> is attached.
> 
> Independent of whether this should be done at all, I have a comment on
> the patch. Instead of invoking sed, I would use autoconf machinery here
> 
>> +Exec=python_VER_ %f
> 
> This would become
> 
> Exec=python at VERSION@@EXEEXT@ %f
> 
>> +       sed 's/_VER_/$(VERSION)$(EXE)/' $(srcdir)/Misc/python.desktop.in >
>> $(srcdir)/Misc/python.desktop
> 
> This could go away, and configure should grow
> 
> AC_CONFIG_FILES(Makefile.pre Modules/Setup.config Misc/python.desktop)

Ah, thanks. I'm not very fit with autoconf magic.

Georg


From michael at ellerman.id.au  Wed Jul 12 02:30:17 2006
From: michael at ellerman.id.au (Michael Ellerman)
Date: Wed, 12 Jul 2006 10:30:17 +1000
Subject: [Python-Dev] User's complaints
In-Reply-To: <ee2a432c0607041649i7ecb6cc1w8ebdf47f7d02097a@mail.gmail.com>
References: <ee2a432c0607041649i7ecb6cc1w8ebdf47f7d02097a@mail.gmail.com>
Message-ID: <dc1166600607111730i44912cedqcea01d918f2b0a85@mail.gmail.com>

On 7/5/06, Neal Norwitz <nnorwitz at gmail.com> wrote:
> On 7/4/06, Guido van Rossum <guido at python.org> wrote:
> >
> >  From actual users of
> > the language I get more complaints about the breakneck speed of
> > Python's evolution than about the brokenness of the current language.
>
> Guido,
>
> I'm really interested in your perspective here.  I assume you hear far
> more "average" complaints from Joe Random User.  Can you help give the
> rest of us an idea about the top 10 complaints/problems people have?
> I realize this will be subjective, that's ok.  Perhaps we should try
> to focus our energies on some of these issues.

Well here's one I stumbled across the other day. I don't know if it's
legit, but it's still bad PR:

http://www.gbch.net/gjb/blog/software/discuss/python-sucks.html

For the impatient, he's not at all bothered about the lack of obscure
language feature X.

cheers

From brett at python.org  Sat Jul  8 00:51:11 2006
From: brett at python.org (Brett Cannon)
Date: Fri, 7 Jul 2006 15:51:11 -0700
Subject: [Python-Dev] second draft of sandboxing design doc
Message-ID: <bbaeab100607071551l173524dt129c85168a10f076@mail.gmail.com>

OK, lots of revisions.  The approach to handling 'file' I have left up in
the air.  Biggest change is switching to "unprotected" and "sandboxed" for
terms when referring to the interpreter.  Also added a Threat Model section
to explain assumptions about the basics of the interpreter.  Hopefully it is
also more clear about the competing approaches for dealing with 'file'.

I am planning on starting work next week on implementation, but I will start
with the least controversial and work my way up.

One thing that is open that I would like some feedback on immediately is
whether people would rather pass in PyObjects or C level types to the
API.  The latter makes implementing the Python wrapper much easier, but
makes embedding a more heavy-handed.  People have a preference or think that
the Python API will be used more often than the C one?  I am leaning towards
making the C API simpler by using C level types (const char *, etc.) and
just deal with the Python wrappings requiring more rejiggering between
types.

Once again, I have a branch going (bcannon-sandboxing) where the work is
going to be done and where this doc lives.  I don't plan on doing another
post of this doc until another major revision.

---------------------------------------------------------------------------------------------------------------

      Restricted Execution for Python
#######################################

About This Document
=============================

This document is meant to lay out the general design for re-introducing
a sandboxing model for Python.  This document should provide one with
enough information to understand the goals for sandboxing, what
considerations were made for the design, and the actual design itself.
Design decisions should be clear and explain not only why they were
chosen but possible drawbacks from taking a specific approach.

If any of the above is found not to be true, please email me at
brett at python.org and let me know what problems you are having with the
document.


XXX TO DO
=============================

* threading needs protection?
* python-dev convince me that hiding 'file' possible?
    + based on that, handle code objects
    + also decide how to handle sockets
    + perhaps go with crippling but try best effort on hiding reference and
if
      best effort holds up eventually shift over to capabilities system
* resolve to IP at call time to prevent DNS man-in-the-middle attacks when
  allowing a specific host name?
* what network info functions are allowed by default?
* does the object.__subclasses__() trick work across interpreters, or is it
  unique per interpreter?
* figure out default whitelist of extension modules
* check default accessible objects for file path exposure
* helper functions to get at StringIO instances for stdin, stdout, and
friends?
* decide on what type of objects (e.g., PyStringObject or const char *) are
to
  be passed in
* all built-ins properly protected?
* exactly how to tell whether argument to open() is a path, IP, or host name
  (third argument, 'n' prefix for networking, format of path, ...)
* API at the Python level
* for extension module protection, allow for wildcard allowance
  (e.g., ``xml.*``)


Goal
=============================

A good sandboxing model provides enough protection to prevent malicious
harm to come to the system, and no more.  Barriers should be minimized
so as to allow most code that does not do anything that would be
regarded as harmful to run unmodified.  But the protections need to be
thorough enough to prevent any unintended changes or information of the
system to come about.

An important point to take into consideration when reading this
document is to realize it is part of my (Brett Cannon's) Ph.D.
dissertation.  This means it is heavily geared toward sandboxing when
the interpreter is working with Python code embedded in a web page as
viewed in Firefox.  While great strides have been taken to keep the
design general enough so as to allow all previous uses of the 'rexec'
module [#rexec]_ to be able to use the new design, it is not the
focused goal.  This means if a design decision must be made for the
embedded use case compared to sandboxing Python code in a pure Python
application, the former will win out over the latter.

Throughout this document, the term "resource" is used to represent
anything that deserves possible protection.  This includes things that
have a physical representation (e.g., memory) to things that are more
abstract and specific to the interpreter (e.g., sys.path).

When referring to the state of an interpreter, it is either
"unprotected" or "sandboxed".  A unprotected interpreter has no
restrictions imposed upon any resource.  A sandboxed interpreter has at
least one, possibly more, resource with restrictions placed upon it to
prevent unsafe code  that is running within the interpreter to cause
harm to the system.


.. contents::


Use Cases
/////////////////////////////

All use cases are based on how many sandboxed interpreters are running
in a single process and whether an unprotected interpreter is also
running.  The use cases can be broken down into two categories: when
the interpreter is embedded and only using sandboxed interpreters, and
when pure Python code is running in an unprotected interpreter and uses
sandboxed interpreters.


When the Interpreter Is Embedded
================================

Single Sandboxed Interpreter
----------------------------

This use case is when an application embeds the interpreter and never
has more than one interpreter running which happens to be sandboxed.


Multiple Sandboxed Interpreters
-------------------------------

When multiple interpreters, all sandboxed at varying levels, need to be
running within a single application.  This is the key use case that
this proposed design is targeted for.


Stand-Alone Python
=============================

When someone has written a Python program that wants to execute Python
code in an sandboxed interpreter(s).  This is the use case that 'rexec'
attempted to fulfill.


Issues to Consider
=============================

Common to all use cases, resources that the interpreter requires to
function at a level below user code cannot be exposed to a sandboxed
interpreter.  For instance, the interpreter might need to stat a file
to see if it is possible to import.  If the ability to stat a file is
not allowed to a sandboxed interpreter, it should not be allowed to
perform that action, regardless of whether the interpreter at a level
below user code needs that ability.

When multiple interpreters are involved (sandboxed or not), not
allowing an interpreter to gain access to resources available in other
interpreters without explicit permission must be enforced.


Resources to Protect
/////////////////////////////

It is important to make sure that the proper resources are protected
from a sandboxed interpreter.  If you don't there is no point to sandboxing.

Filesystem
===================

All facets of the filesystem must be protected.  This means restricting
reading and writing to the filesystem (e.g., files, directories, etc.).
It should be allowed in controlled situations where allowing access to
the filesystem is desirable, but that should be an explicit allowance.

There must also be protection to prevent revealing any information
about the filesystem.  Disclosing information on the filesystem could
allow one to infer what OS the interpreter is running on, for instance.


Memory
===================

Memory should be protected.  It is a limited resource on the system
that can have an impact on other running programs if it is exhausted.
Being able to restrict the use of memory would help alleviate issues
from denial-of-service (DoS) attacks on the system.


Networking
===================

Networking is somewhat like the filesystem in terms of wanting similar
protections.  You do not want to let unsafe code make socket
connections unhindered or accept them to do possibly nefarious things.
You also want to prevent finding out information about the network your
are connected to.


Interpreter
===================

One must make sure that the interpreter is not harmed in any way from
sandboxed code.  This usually takes the form of crashing the program
that the interpreter is embedded in or the unprotected interpreter that
started the sandbox interpreter.  Executing hostile bytecode that might
lead to undesirable effects is another possible issue.

There is also the issue of taking it over.  One should not able to gain
escalated privileges in any way without explicit permission.


Types of Security
///////////////////////////////////////

As with most things, there are multiple approaches one can take to
tackle a problem.  Security is no exception.  In general there seem to
be two approaches to protecting resources.


Resource Hiding
=============================

By never giving code a chance to access a resource, you prevent it from
being (ab)used.  This is the idea behind resource hiding; you can't
misuse something you don't have in the first place.

The most common implementation of resource hiding is capabilities.  In
this type of system a resource's reference acts as a ticket that
represents the right to use the resource.  Once code has a reference it
is considered to have full use of resource that reference represents
and no further security checks are directly performed (using delegates
and other structured ways one can actually have a security check for
each access of a resource, but this is not a default behaviour).

As an example, consider the 'file' type as a resource we want to
protect.  That would mean that we did not want a reference to the
'file' type to ever be accessible without explicit permission.  If one
wanted to provide read-only access to a temp file, you could have
open() perform a check on the permissions of the current interpreter,
and if it is allowed to, return a proxy object for the file that only
allows reading from it.  The 'file' instance for the proxy would need
to be properly hidden so that the reference was not reachable from
outside so that 'file' access could still be controlled.

Python, as it stands now, unfortunately does not work well for a pure
capabilities system.  Capabilities require the prohibition of certain
abilities, such as "direct access to another's private state"
[#paradigm regained]_.  This obviously is not possible in Python since,
at least at the Python level, there is no such thing as private state
that is persistent (one could argue that local variables that are not
cell variables for lexical scopes are private, but since they do not
survive after a function call they are not usable for keeping
persistent state).  One can hide references at the C level by storing
it in the struct for the instance of a type and not providing a
function to access that attribute.

Python's introspection abilities also do not help make implementing
capabilities that much easier.  Consider how one could access 'file'
even when it is deleted from __builtin__.  You can still get to the
reference for 'file' through the sequence returned by
``object.__subclasses__()``.


Resource Crippling
=============================

Another approach to security is to not worry about controlling access
to the reference of a resource.  One can have a resource perform a
security check every time someone tries to use a method on that
resource.  This pushes the security check to a lower level; from a
reference level to the method level.

By performing the security check every time a resource's method is
called the worry of a specific resource's reference leaking out to
insecure code is alleviated.  This does add extra overhead, though, by
having to do so many security checks.  It also does not handle the
situation where an unexpected exposure of a type occurs that has not
been properly crippled.

FreeBSD's jail system provides a protection scheme similar to this.
Various system calls allow for basic usage, but knowing or having
access to the system call is not enough to grant usage.  Every call to
a system call requires checking that the proper rights have been
granted to the use in order to allow for the system call to perform
its action.

An even better example in FreeBSD's jail system is its protection of
sockets.  One can only bind a single IP address to a jail.  Any attempt
to do more or perform uses with the one IP address that is granted is
prevented.  The check is performed at every call involving the one
granted IP address.

Using 'file' as the example again, one could cripple the type so that
instantiation is not possible for the type in Python.  One could also
provide a permission check on each call to a unsafe method call and
thus allow the type to be used in normal situations (such as type
checking), but still feel safe that illegal operations are not
performed.  Regardless of which approach you take, you do not need to
worry about a reference to the type being exposed unexpectedly since
the reference is not the security check but the actual method calls.


Comparison of the Two Approaches
================================

>From the perspective of Python, the two approaches differ on what would
be the most difficult thing to analyze from a security standpoint: all
of the ways to gain access to various types from a sandboxed
interpreter with no imports, or finding all of the types that can lead
to possibly dangerous actions and thus need to be crippled.

Some Python developers, such as Armin Rigo, feel that truly hiding
objects in Python is "quite hard" [#armin-hiding]_.  This sentiment
means that making a pure capabilities system in Python that is secure
is not possible as people would continue to find new ways to get a hold
of the reference to a protected resource.

Others feel that by not going the capabilities route we will be
constantly chasing down new types that require crippling.  The thinking
is that if we cannot control the references for 'file', how are we to
know what other types might become exposed later on and thus require
more crippling?

It essentially comes down to what is harder to do: find all the ways to
access the types in Python in a sandboxed interpreter with no imported
modules, or to go through the Python code base and find all types that
should be crippled?


The 'rexec' Module
///////////////////////////////////////

The 'rexec' module [#rexec]_ was the original attempt at providing a
sandbox environment for Python code to run in.  It's design was based
on Safe-Tcl which was essentially a capabilities system [#safe-tcl]_.
Safe-Tcl allowed you to launch a separate interpreter where its global
functions were specified at creation time.  This prevented one from
having any abilities that were not explicitly provided.

For 'rexec', the Safe-Tcl model was tweaked to better match Python's
situation.  An RExec object represented a sandboxed environment.
Imports were checked against a whitelist of modules.  You could also
restrict the type of modules to import based on whether they were
Python source, bytecode, or C extensions.  Built-ins were allowed
except for a blacklist of built-ins to not provide.  One could restrict
whether stdin, stdout, and stderr were provided or not on a per-RExec
basis.  Several other protections were provided; see documentation for
the complete list.

The ultimate undoing of the 'rexec' module was how access to objects
that in normal Python require no imports to reach was handled.
Importing modules requires a direct action, and thus can be protected
against directly in the import machinery.  But for built-ins, they are
accessible by default and require no direct action to access in normal
Python; you just use their name since they are provided in all
namespaces.

For instance, in a sandboxed interpreter, one only had to
``del __builtins__`` to gain access to the full set of built-ins.
Another way is through using the gc module:
``gc.get_referrers(''.__class__.__bases__[0])[6]['file']``.  While both
of these could be fixed (the former was a bug in 'rexec' that was fixed
and the latter could be handled by not allowing 'gc' to be imported),
they are examples of things that do not require proactive actions on
the part of the programmer in normal Python to gain access to a
resource.  This was an unfortunate side-effect of having all of that
wonderful reflection in Python.

There is also the issue that 'rexec' was written in Python which
provides its own problems based on reflection and the ability to modify
the code at run-time without security protection.

Much has been learned since 'rexec' was written about how Python tends
to be used and where security issues tend to appear.  Essentially
Python's dynamic nature does not lend itself very well to a security
implementation that does not require a constant checking of
permissions.


Threat Model
///////////////////////////////////////

Below is a list of what the security implementation assumes, along with
what section of this document that addresses that part of the security
model (if not already true in Python by default).  The term "bare" when
in regards to an interpreter means an interpreter that has not
performed a single import of a module.  Also, all comments refer to a
sandboxed interpreter unless otherwise explicitly stated.

This list does not address specifics such as how 'file' will be
protected or whether memory should be protected.  This list is meant to
make clear at a more basic level what the security model is assuming is
true.

* The Python interpreter itself is always trusted.
* The Python interpreter cannot be crashed by valid Python source code
  in a bare interpreter.
* Python source code is always considered safe.
* Python bytecode is always considered dangerous [`Hostile Bytecode`_].
* C extension modules are inherently considered dangerous
  [`Extension Module Importation`_].
    + Explicit trust of a C extension module is possible.
* Sandboxed interpreters running in the same process inherently cannot
  communicate with each other.
    + Communication through C extension modules is possible because of
      the technical need to share extension module instances between
      interpreters.
* Sandboxed interpreters running in the same process inherently cannot
  share objects.
    + Sharing objects through C extension modules is possible because
      of the technical need to share extension module instances between
      interpreters.
* When starting a sandboxed interpreter, it starts with a fresh
  built-in and global namespace that is not shared with the interpreter
  that started it.
* Objects in the default built-in namespace should be safe to use
  [`Reading/Writing Files`_, `Stdin, Stdout, and Stderr`_].
    + Either hide the dangerous ones or cripple them so they can cause
      no harm.

There are also some features that might be desirable, but are not being
addressed by this security model.

* Communication in any direction between an unprotected interpreter and
  a sandboxed interpreter it created.


The Proposed Approach
///////////////////////////////////////

In light of where 'rexec' succeeded and failed along with what is known
about the two main approaches to security and how Python tends to
operate, the following is a proposal on how to secure Python for
sandboxing.


Implementation Details
===============================

Support for sandboxed interpreters will require a compilation flag.
This allows the more common case of people not caring about protections
to not take a performance hit.  And even when Python is compiled for
sandboxed interpreter restrictions, when the running interpreter *is*
unprotected, there will be no accidental triggers of protections.  This
means that developers should be liberal with the security protections
without worrying about there being issues for interpreters that do not
need/want the protection.

At the Python level, the __sandboxed__ built-in will be set based on
whether the interpreter is sandboxed or not.  This will be set for
*all* interpreters, regardless of whether sandboxed interpreter support
was compiled in or not.

For setting what is to be protected, the PyThreadState for the
sandboxed interpreter must be passed in.  This makes the protection
very explicit and helps make sure you set protections for the exact
interpreter you mean to.  All functions that set protections begin with
the prefix ``PySandbox_Set*()``.  These functions are meant to only
work with sandboxed interpreters that have not been used yet to execute
any Python code.  The calls must be made by the code creating and
handling the sandboxed interpreter *before* the sandboxed interpreter
is used to execute any Python code.

The functions for checking for permissions are actually macros that
take in at least an error return value for the function calling the
macro.  This allows the macro to return on behalf of the caller if the
check fails and cause the SandboxError exception to be propagated
automatically.  This helps eliminate any coding errors from incorrectly
checking a return value on a rights-checking function call.  For the
rare case where this functionality is disliked, just make the check in
a utility function and check that function's return value (but this is
strongly discouraged!).

Functions that check that an operation is allowed implicitly operate on
the currently running interpreter as returned by
``PyInterpreter_Get()`` and are to be used by any code (the
interpreter, extension modules, etc.) that needs to check for
permission to execute.  They have the common prefix of
`PySandbox_Allowed*()``.


API
--------------

* PyThreadState* PySandbox_NewInterpreter()
    Return a new interpreter that is considered sandboxed.  There is no
    corresponding ``PySandbox_EndInterpreter()`` as
    ``Py_EndInterpreter()`` will be taught how to handle sandboxed
    interpreters.  ``NULL`` is returned on error.

* PySandbox_Allowed(error_return)
    Macro that has the caller return with 'error_return' if the
    interpreter is unprotected, otherwise do nothing.


Memory
=============================

Protection
--------------

A memory cap will be allowed.

Modification to pymalloc will be needed to properly keep track of the
allocation and freeing of memory.  Same goes for the macros around the
system malloc/free system calls.  This provides a platform-independent
system for protection of memory instead of relying on the operating
system to provide a service for capping memory usage of a process.  It
also allows the protection to be at the interpreter level instead of at
the process level.


Why
--------------

Protecting excessive memory usage allows one to make sure that a DoS
attack against the system's memory is prevented.


Possible Security Flaws
-----------------------

If code makes direct calls to malloc/free instead of using the proper
``PyMem_*()``
macros then the security check will be circumvented.  But C code is
*supposed* to use the proper macros or pymalloc and thus this issue is
not with the security model but with code not following Python coding
standards.


API
--------------

* int PySandbox_SetMemoryCap(PyThreadState *, integer)
    Set the memory cap for an sandboxed interpreter.  If the
    interpreter is not running an sandboxed interpreter, return a false
    value.

* PySandbox_AllowedMemoryAlloc(integer, error_return)
    Macro to increase the amount of memory that is reported that the
    running sandboxed interpreter is using.  If the increase puts the
    total count passed the set limit, raise an SandboxError exception
    and cause the calling function to return with the value of
    'error_return', otherwise do nothing.

* PySandbox_AllowedMemoryFree(integer, error_return)
    Macro to decrease the current running interpreter's allocated
    memory.  If this puts the memory used to below 0, raise a
    SandboxError exception and return 'error_return', otherwise do
    nothing.


Reading/Writing Files
=============================

Protection
--------------

XXX

To open a file, one will have to use open().  This will make open() a
factory function that controls reference access to the 'file' type in
terms of creating new instances.  When an attempted file opening fails
(either because the path does not exist or of security reasons),
SandboxError will be raised.  The same exception must be raised to
prevent filesystem information being gleaned from the type of exception
returned (i.e., returning IOError if a path does not exist tells the
user something about that file path).

What open() returns may not be an instance of 'file' but a proxy that
provides the security measures needed.  While this might break code
that uses type checking to make sure a 'file' object is used, taking a
duck typing approach would be better.  This is not only more Pythonic
but would also allow the code to use a StringIO instance.

It has been suggested to allow for a passed-in callback to be called
when a specific path is to be opened.  While this provides good
flexibility in terms of allowing custom proxies with more fine-grained
security (e.g., capping the amount of disk write), this has been deemed
unneeded in the initial security model and thus is not being considered
at this time.

Why
--------------

Allowing anyone to be able to arbitrarily read, write, or learn about
the layout of your filesystem is extremely dangerous.  It can lead to
loss of data or data being exposed to people whom should not have
access.


Possible Security Flaws
-----------------------

XXX


API
--------------

* int PySandbox_SetAllowedFile(PyThreadState *, string path,
                                string mode)
    Add a file that is allowed to be opened in 'mode' by the 'file'
    object.  If the interpreter is not sandboxed then return a false
    value.

* PySandbox_AllowedPath(string path, string mode, error_return)
    Macro that causes the caller to return with 'error_return' and
    raise SandboxError as the exception if the specified path with
    'mode' is not allowed, otherwise do nothing.


Extension Module Importation
============================

Protection
--------------

A whitelist of extension modules that may be imported must be provided.
A default set is given for stdlib modules known to be safe.

A check in the import machinery will check that a specified module name
is allowed based on the type of module (Python source, Python bytecode,
or extension module).  Python bytecode files are never directly
imported because of the possibility of hostile bytecode being present.
Python source is always considered safe based on the assumption that
all resource harm is eventually done at the C level, thus Python source
code directly cannot cause harm without help of C extension modules.
Thus only C extension modules need to be checked against the whitelist.

The requested extension module name is checked in order to make sure
that it is on the whitelist if it is a C extension module.  If the name
is not correct a SandboxError exception is raised.  Otherwise the
import is allowed.

Even if a Python source code module imports a C extension module in an
unprotected interpreter it is not a problem since the Python source
code module is reloaded in the sandboxed interpreter.  When that Python
source module is freshly imported the normal import check will be
triggered to prevent the C extension module from becoming available to
the sandboxed interpreter.

For the 'os' module, a special sandboxed version will be used if the
proper C extension module providing the correct abilities is not
allowed.  This will default to '/' as the path separator and provide as
much reasonable abilities as possible from a pure Python module.

The 'sys' module is specially addressed in
`Changing the Behaviour of the Interpreter`_.

By default, the whitelisted modules are:

* XXX


Why
--------------

Because C code is considered unsafe, its use should be regulated.  By
using a whitelist it allows one to explicitly decide that a C extension
module is considered safe.


Possible Security Flaws
-----------------------

If a whitelisted C extension module imports a non-whitelisted C
extension module and makes it an attribute of the whitelisted module
there will be a breach in security.  Luckily this a rarity in
extension modules.

There is also the issue of a C extension module calling the C API of a
non-whitelisted C extension module.

Lastly, if a whitelisted C extension module is loaded in an unprotected
interpreter and then loaded into a sandboxed interpreter then there is
no checks during module initialization for possible security issues in
the sandboxed interpreter that would have occurred had the sandboxed
interpreter done the initial import.

All of these issues can be handled by never blindly whitelisting a C
extension module.  Added support for dealing with C extension modules
comes in the form of `Extension Module Crippling`_.


API
--------------

* int PySandbox_SetModule(PyThreadState *, string module_name)
    Allow the sandboxed interpreter to import 'module_name'.  If the
    interpreter is not sandboxed, return a false value.  Absolute
    import paths must be specified.

* int PySandbox_BlockModule(PyThreadState *, string module_name)
    Remove the specified module from the whitelist.  Used to remove
    modules that are allowed by default.  Return a false value if
    called on an unprotected interpreter.

* PySandbox_AllowedModule(string module_name, error_return)
    Macro that causes the caller to return with 'error_return' and sets
    the exception SandboxError if the specified module cannot be
    imported, otherwise does nothing.


Extension Module Crippling
==========================

Protection
--------------

By providing a C API for checking for allowed abilities, modules that
have some useful functionality can do proper security checks for those
functions that could provide insecure abilities while allowing safe
code to be used (and thus not fully deny importation).


Why
--------------

Consider a module that provides a string processing ability.  If that
module provides a single convenience function that reads its input
string from a file (with a specified path), the whole module should not
be blocked from being used, just that convenience function.  By
whitelisting the module but having a security check on the one problem
function, the user can still gain access to the safe functions.  Even
better, the unsafe function can be allowed if the security checks pass.


Possible Security Flaws
-----------------------

If a C extension module developer incorrectly implements the security
checks for the unsafe functions it could lead to undesired abilities.


API
--------------

Use PySandbox_Allowed() to protect unsafe code from being executed.


Hostile Bytecode
=============================

Protection
--------------

XXX


Why
--------------

Without implementing a bytecode verification tool, there is no way of
making sure that bytecode does not jump outside its bounds, thus
possibly executing malicious code.  It also presents the possibility of
crashing the interpreter.


Possible Security Flaws
-----------------------

None known.


API
--------------

N/A


Changing the Behaviour of the Interpreter
=========================================

Protection
--------------

Only a subset of the 'sys' module will be made available to sandboxed
interpreters.  Things to allow from the sys module:

* byteorder (?)
* copyright
* displayhook
* excepthook
* __displayhook__
* __excepthook__
* exc_info
* exc_clear
* exit
* getdefaultencoding
* _getframe (?)
* hexversion
* last_type
* last_value
* last_traceback
* maxint (?)
* maxunicode (?)
* modules
* stdin  # See `Stdin, Stdout, and Stderr`_.
* stdout
* stderr
* version


Why
--------------

Filesystem information must be removed.  Any settings that could
possibly lead to a DoS attack (e.g., sys.setrecursionlimit()) or risk
crashing the interpreter must also be removed.


Possible Security Flaws
-----------------------

Exposing something that could lead to future security problems (e.g., a
way to crash the interpreter).


API
--------------

None.


Socket Usage
=============================

Protection
--------------

Allow sending and receiving data to/from specific IP addresses on
specific ports.

open() is to be used as a factory function to open a network
connection.  If the connection is not possible (either because of an
invalid address or security reasons), SandboxError is raised.

A socket object may not be returned by the call.  A proxy to handle
security might be returned instead.

XXX


Why
--------------

Allowing arbitrary sending of data over sockets can lead to DoS attacks
on the network and other machines.  Limiting accepting data prevents
your machine from being attacked by accepting malicious network
connections.  It also allows you to know exactly where communication is
going to and coming from.


Possible Security Flaws
-----------------------

If someone managed to influence the used DNS server to influence what
IP addresses were used after a DNS lookup.


API
--------------

* int PySandbox_SetIPAddress(PyThreadState *, string IP, integer port)
    Allow the sandboxed interpreter to send/receive to the specified
    'IP' address on the specified 'port'.  If the interpreter is not
    sandboxed, return a false value.

* PySandbox_AllowedIPAddress(string IP, integer port, error_return)
    Macro to verify that the specified 'IP' address on the specified
    'port' is allowed to be communicated with.  If not, cause the
    caller to return with 'error_return' and SandboxError exception
    set, otherwise do nothing.

* int PySandbox_SetHost(PyThreadState *, string host, integer port)
    Allow the sandboxed interpreter to send/receive to the specified
    'host' on the specified 'port'.  If the interpreter is not
    sandboxed, return a false value.

* PySandbox_AllowedHost(string host, integer port, error_return)
    Check that the specified 'host' on the specified 'port' is allowed
    to be communicated with.  If not, set a SandboxError exception and
    cause the caller to return 'error_return', otherwise do nothing.


Network Information
=============================

Protection
--------------

Limit what information can be gleaned about the network the system is
running on.  This does not include restricting information on IP
addresses and hosts that are have been explicitly allowed for the
sandboxed interpreter to communicate with.

XXX


Why
--------------

With enough information from the network several things could occur.
One is that someone could possibly figure out where your machine is on
the Internet.  Another is that enough information about the network you
are connected to could be used against it in an attack.


Possible Security Flaws
-----------------------

As long as usage is restricted to only what is needed to work with
allowed addresses, there are no security issues to speak of.


API
--------------

* int PySandbox_SetNetworkInfo(PyThreadState *)
    Allow the sandboxed interpreter to get network information
    regardless of whether the IP or host address is explicitly allowed.
    If the interpreter is not sandboxed, return a false value.

* PySandbox_AllowedNetworkInfo(error_return)
    Macro that will return 'error_return' for the caller and set a
    SandboxError exception if the sandboxed interpreter does not allow
    checking for arbitrary network information, otherwise do nothing.


Filesystem Information
=============================

Protection
--------------

Do not allow information about the filesystem layout from various parts
of Python to be exposed.  This means blocking exposure at the Python
level to:

* __file__ attribute on modules
* __path__ attribute on packages
* co_filename attribute on code objects
* XXX


Why
--------------

Exposing information about the filesystem is not allowed.  You can
figure out what operating system one is on which can lead to
vulnerabilities specific to that operating system being exploited.


Possible Security Flaws
-----------------------

Not finding every single place where a file path is exposed.


API
--------------

* int PySandbox_SetFilesystemInfo(PyThreadState *)
    Allow the sandboxed interpreter to expose filesystem information.
    If the passed-in interpreter is not sandboxed, return NULL.

* PySandbox_AllowedFilesystemInfo(error_return)
    Macro that checks if exposing filesystem information is allowed.
    If it is not, cause the caller to return with the value of
    'error_return' and raise SandboxError, otherwise do nothing.


Stdin, Stdout, and Stderr
=============================

Protection
--------------

By default, sys.__stdin__, sys.__stdout__, and sys.__stderr__ will be
set to instances of StringIO.  Explicit allowance of the process'
stdin, stdout, and stderr is possible.

This will protect the 'print' statement, and the built-ins input() and
raw_input().


Why
--------------

Interference with stdin, stdout, or stderr should not be allowed unless
desired.  No one wants uncontrolled output sent to their screen.


Possible Security Flaws
-----------------------

Unless StringIO instances can be used maliciously, none to speak of.


API
--------------

* int PySandbox_SetTrueStdin(PyThreadState *)
  int PySandbox_SetTrueStdout(PyThreadState *)
  int PySandbox_SetTrueStderr(PyThreadState *)
    Set the specific stream for the interpreter to the true version of
    the stream and not to the default instance of StringIO.  If the
    interpreter is not sandboxed, return a false value.


Adding New Protections
=============================

.. note:: This feature has the lowest priority and thus will be the
          last feature implemented (if ever).

Protection
--------------

Allow for extensibility in the security model by being able to add new
types of checks.  This allows not only for Python to add new security
protections in a backwards-compatible fashion, but to also have
extension modules add their own as well.

An extension module can introduce a group for its various values to
check, with a type being a specific value within a group.  The "Python"
group is specifically reserved for use by the Python core itself.


Why
--------------

We are all human.  There is the possibility that a need for a new type
of protection for the interpreter will present itself and thus need
support.  By providing an extensible way to add new protections it
helps to future-proof the system.

It also allows extension modules to present their own set of security
protections.  That way one extension module can use the protection
scheme presented by another that it is dependent upon.


Possible Security Flaws
------------------------

Poor definitions by extension module users of how their protections
should be used would allow for possible exploitation.


API
--------------

+ Bool
    * int PySandbox_SetExtendedFlag(PyThreadState *, string group,
                                    string type)
        Set a group-type to be true.  Expected use is for when a binary
        possibility of something is needed and that the default is to
        not allow use of the resource (e.g., network information).
        Returns a false value if used on an unprotected interpreter.

    * PySandbox_AllowedExtendedFlag(string group, string type,
                                    error_return)
        Macro that if the group-type is not set to true, cause the
        caller to return with 'error_return' with SandboxError
        exception raised.  For unprotected interpreters the check does
        nothing.

+ Numeric Range
    * int PySandbox_SetExtendedCap(PyThreadState *, string group,
                                    string type, integer cap)
        Set a group-type to a capped value, 'cap', with the initial
        allocated value set to 0.  Expected use is when a resource has
        a capped amount of use (e.g., memory).  Returns a false value
        if the interpreter is not sandboxed.

    * PySandbox_AllowedExtendedAlloc(integer increase, error_return)
        Macro to raise the amount of a resource is used by 'increase'.
        If the increase pushes the resource allocation past the set
        cap, then return 'error_return' and set SandboxError as the
        exception, otherwise do nothing.

    * PySandbox_AllowedExtendedFree(integer decrease, error_return)
        Macro to lower the amount a resource is used by 'decrease'.  If
        the decrease pushes the allotment to below 0 then have the
        caller return 'error_return' and set SandboxError as the
        exception, otherwise do nothing.


+ Membership
    * int PySandbox_SetExtendedMembership(PyThreadState *,
                                            string group, string type,
                                            string member)
        Add a string, 'member',  to be considered a member of a
        group-type (e.g., allowed file paths).  If the interpreter is not
        an sandboxed interpreter, return a false value.

    * PySandbox_AllowedExtendedMembership(string group, string type,
                                            string member,
                                            error_return)
        Macro that checks 'member' is a member of the values set for
        the group-type.  If it is not, then have the caller return
        'error_return' and set an exception for SandboxError, otherwise
        does nothing.

+ Specific Value
    * int PySandbox_SetExtendedValue(PyThreadState *, string group,
                                        string type, string value)
        Set a group-type to 'value'.  If the interpreter is not
        sandboxed, return NULL.

    * PySandbox_AllowedExtendedValue(string group, string type,
                                        string value, error_return)
        Macro to check that the group-type is set to 'value'.  If it is
        not, then have the caller return 'error_return' and set an
        exception for SandboxError, otherwise do nothing.


Python API
=============================

__sandboxed__
--------------

A built-in that flags whether the interpreter currently running is
sandboxed or not.  Set to a 'bool' value that is read-only.  To mimic
working of __debug__.


sandbox module
--------------

XXX


References
///////////////////////////////////////

.. [#rexec] The 'rexec' module
   (http://docs.python.org/lib/module-rexec.html)

.. [#safe-tcl] The Safe-Tcl Security Model
   (http://research.sun.com/technical-reports/1997/abstract-60.html)

.. [#ctypes] 'ctypes' module
   (http://docs.python.org/dev/lib/module-ctypes.html)

.. [#paradigm regained] "Paradigm Regained:
                         Abstraction Mechanisms for Access Control"
   (http://erights.org/talks/asian03/paradigm-revised.pdf)

.. [#armin-hiding] [Python-Dev] what can we do to hide the 'file' type?
   (http://mail.python.org/pipermail/python-dev/2006-July/067076.html)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20060707/a32678f1/attachment-0001.html 

From christopher.perkins at pw.utc.com  Mon Jul 10 19:36:02 2006
From: christopher.perkins at pw.utc.com (Perkins, Christopher)
Date: Mon, 10 Jul 2006 13:36:02 -0400
Subject: [Python-Dev]  Autoloading? (Making Queue.Queue easier to use)
Message-ID: <4B9D477017AEEC4188C18CB8DA8364B606156A02@pusehe0l.eh.pweh.com>

John,

I see what you are doing with the algorithm now, and I can easily re-factor
it.  What I am having issues with is how structured it is.  5 minute
windows?  Then running hours will always be recorded in 1/12th time steps.

Would it not be more accurate to record engine time where the breaker was
closed?

-chris

From skip at pobox.com  Wed Jul 12 02:54:03 2006
From: skip at pobox.com (skip at pobox.com)
Date: Tue, 11 Jul 2006 19:54:03 -0500
Subject: [Python-Dev] User's complaints
In-Reply-To: <dc1166600607111730i44912cedqcea01d918f2b0a85@mail.gmail.com>
References: <ee2a432c0607041649i7ecb6cc1w8ebdf47f7d02097a@mail.gmail.com>
	<dc1166600607111730i44912cedqcea01d918f2b0a85@mail.gmail.com>
Message-ID: <17588.18475.171696.506911@montanaro.dyndns.org>


    Michael> Well here's one I stumbled across the other day. I don't know
    Michael> if it's legit, but it's still bad PR:

    Michael> http://www.gbch.net/gjb/blog/software/discuss/python-sucks.html

    Michael> For the impatient, he's not at all bothered about the lack of
    Michael> obscure language feature X.

The way I used to format dates using time.strftime does indeed no longer
work.

Python 2.3:

    >>> import time
    >>> time.strftime("%Y-%m-%d", (2005, 6, 4) + (0,)*6)
    '2005-06-04'

Python 2.4 or 2.5:

    >>> import time
    >>> time.strftime("%Y-%m-%d", (2005, 6, 4) + (0,)*6)
    Traceback (most recent call last):
      File "<stdin>", line 1, in ?
    ValueError: day of year out of range
    >>> time.strftime("%Y-%m-%d", (2005, 6, 4) + (1,)*6)
    '2005-06-04'

I don't actually run into this problem as I've pretty much converted to use
datetime in new code.  I also realize that's not documented as the way it
should be done, but I'm fairly certain it was common usage before the
datetime module came along.  Still, it is a bit annoying that the
(undocumented, but I think de facto) commonly used idiom no longer works.

(In fact, it always bothered me a bit that I had to even provide the unused
values.)

Skip

From brett at python.org  Wed Jul 12 03:05:21 2006
From: brett at python.org (Brett Cannon)
Date: Tue, 11 Jul 2006 18:05:21 -0700
Subject: [Python-Dev] User's complaints
In-Reply-To: <17588.18475.171696.506911@montanaro.dyndns.org>
References: <ee2a432c0607041649i7ecb6cc1w8ebdf47f7d02097a@mail.gmail.com>
	<dc1166600607111730i44912cedqcea01d918f2b0a85@mail.gmail.com>
	<17588.18475.171696.506911@montanaro.dyndns.org>
Message-ID: <bbaeab100607111805g5f6ca07cvf0ba9cb09b68bcfa@mail.gmail.com>

On 7/11/06, skip at pobox.com <skip at pobox.com> wrote:
>
>
>     Michael> Well here's one I stumbled across the other day. I don't know
>     Michael> if it's legit, but it's still bad PR:
>
>     Michael>
> http://www.gbch.net/gjb/blog/software/discuss/python-sucks.html
>
>     Michael> For the impatient, he's not at all bothered about the lack of
>     Michael> obscure language feature X.


That whole entry is a little overblown.  I mean the guy has a single issue
with a change and he gives up on the language after only the second one?
Try using any language that is under active development and show me stuff
that won't break.  I think it shows how hard we work to not break things
that this guy only got bit twice, and at least the second time was minor and
had a legit reason for the change.

The way I used to format dates using time.strftime does indeed no longer
> work.
>
> Python 2.3:
>
>     >>> import time
>     >>> time.strftime("%Y-%m-%d", (2005, 6, 4) + (0,)*6)
>     '2005-06-04'
>
> Python 2.4 or 2.5:
>
>     >>> import time
>     >>> time.strftime("%Y-%m-%d", (2005, 6, 4) + (0,)*6)
>     Traceback (most recent call last):
>       File "<stdin>", line 1, in ?
>     ValueError: day of year out of range
>     >>> time.strftime("%Y-%m-%d", (2005, 6, 4) + (1,)*6)
>     '2005-06-04'


That was done to fix buffer overflow issues when libc implementations didn't
do bound checks on the arguments to strftime() and would index too far (e.g.,
setting the month as 20 indexes too far and setting to 0 can be an issue as
well since the array indexing for the month name will be January, and thus
will most likely do ``mon - 1`` to get the index into the array).

I don't actually run into this problem as I've pretty much converted to use
> datetime in new code.  I also realize that's not documented as the way it
> should be done,


It is the last point in the first paragraph on time.strftime() discussing
what changed in Python 2.4 as to what the change was.  It's also in
Misc/NEWS .  Basically the guy didn't read the release notes or the docs to
see why that changed and that it was legitimate and needed for stability.

but I'm fairly certain it was common usage before the
> datetime module came along.  Still, it is a bit annoying that the
> (undocumented, but I think de facto) commonly used idiom no longer works.


As I said, stability called for the need to make the change.  It was
discussed on python-dev when it came up.

(In fact, it always bothered me a bit that I had to even provide the unused
> values.)


That would probably be fine to add in 2.6 .  Should be a huge issue.  You
could also change the function to detect values of 0 when provided and then
automatically use defaults in those cases that are within the proper range.

-Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20060711/6cb60fa7/attachment.html 

From skip at pobox.com  Wed Jul 12 03:57:18 2006
From: skip at pobox.com (skip at pobox.com)
Date: Tue, 11 Jul 2006 20:57:18 -0500
Subject: [Python-Dev] User's complaints
In-Reply-To: <bbaeab100607111805g5f6ca07cvf0ba9cb09b68bcfa@mail.gmail.com>
References: <ee2a432c0607041649i7ecb6cc1w8ebdf47f7d02097a@mail.gmail.com>
	<dc1166600607111730i44912cedqcea01d918f2b0a85@mail.gmail.com>
	<17588.18475.171696.506911@montanaro.dyndns.org>
	<bbaeab100607111805g5f6ca07cvf0ba9cb09b68bcfa@mail.gmail.com>
Message-ID: <17588.22270.591885.831508@montanaro.dyndns.org>

    Brett> That whole entry is a little overblown.  

Well, sure.  Think of it as a bug report with attitude. ;-)

    Brett> That was done to fix buffer overflow issues when libc
    Brett> implementations didn't do bound checks on the arguments to
    Brett> strftime() and would index too far...

That special case could simply be recognized and converted into one that
works couldn't it?  Documented or not, I believe it was the standard idiom
for formatting just a date before 2.4.

    http://python.org/sf/1520914

Keep or toss as you see fit.  Seems like breakage that could have been
avoided to me though.

Skip

From anthony at interlink.com.au  Wed Jul 12 04:42:09 2006
From: anthony at interlink.com.au (Anthony Baxter)
Date: Wed, 12 Jul 2006 12:42:09 +1000
Subject: [Python-Dev] urllib.quote and unicode bug resuscitation attempt
In-Reply-To: <44B41525.3040004@v.loewis.de>
References: <44B3ADE2.7040108@ofai.at> <44B41525.3040004@v.loewis.de>
Message-ID: <200607121242.12323.anthony@interlink.com.au>

On Wednesday 12 July 2006 07:16, Martin v. L?wis wrote:
> Stefan Rank wrote:
> > I suggest to add (after 2.5 I assume) one of the following to the
> > beginning of urllib.quote to either fail early and consistently
> > on unicode arguments and improve the error message::
> >
> >    if isinstance(s, unicode):
> >        raise TypeError("quote needs a byte string argument, not
> > unicode," " use `argument.encode('utf-8')` first.")
> >
> > or to do The Right Thing (tm), which is utf-8 encoding::
>
> The right thing to do is IRIs. This is more complicated than
> encoding the Unicode string as UTF-8, though: for the host part of
> the URL, you have to encode it with IDNA (and there are additional
> complicated rules in place, e.g. when the Unicode string already
> contains %).
>
> Contributions are welcome, as long as they fix this entire issue
> "for good" (i.e. in all URL-processing code, and considering all
> relevant RFCs).

For 2.5, should we at least detect that it's unicode and raise a 
useful error?


-- 
Anthony Baxter     <anthony at interlink.com.au>
It's never too late to have a happy childhood.

From brett at python.org  Wed Jul 12 04:50:05 2006
From: brett at python.org (Brett Cannon)
Date: Tue, 11 Jul 2006 19:50:05 -0700
Subject: [Python-Dev] changing time.strftime() to accept 0s (was: User's
	complaints)
Message-ID: <bbaeab100607111950o42ce6528xc3f2717292d2e58c@mail.gmail.com>

On 7/11/06, skip at pobox.com <skip at pobox.com> wrote:
>
>     Brett> That whole entry is a little overblown.
>
> Well, sure.  Think of it as a bug report with attitude. ;-)
>
>     Brett> That was done to fix buffer overflow issues when libc
>     Brett> implementations didn't do bound checks on the arguments to
>     Brett> strftime() and would index too far...
>
> That special case could simply be recognized and converted into one that
> works couldn't it?  Documented or not, I believe it was the standard idiom
> for formatting just a date before 2.4.
>
>     http://python.org/sf/1520914
>
> Keep or toss as you see fit.  Seems like breakage that could have been
> avoided to me though.


Right, but that would have required realizing how to prevent it at the
time.  =)

I can change it so that 0 is an acceptable value and internally handles
defaulting to something reasonable (accepting less than 9 values in the
tuple is a separate thing that I don't feel like bothering to implement).
It does possibly hide bugs where 0 was not meant to be passed, though.

If people think this is a reasonable thing to change, then Neal and Anthony,
do you want it to go into 2.5?

-Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20060711/7593f880/attachment.html 

From greg.ewing at canterbury.ac.nz  Wed Jul 12 05:05:37 2006
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Wed, 12 Jul 2006 15:05:37 +1200
Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
In-Reply-To: <1152589949.14276.16.camel@localhost.localdomain>
References: <001d01c6a369$78a59950$6402a8c0@arkdesktop>
	<Pine.LNX.4.58.0607091349230.22834@server1.LFW.org>
	<44B190DF.3000805@acm.org> <44B19691.4050509@acm.org>
	<ca471dc20607092039g5cf2ea54h5225925ce877145e@mail.gmail.com>
	<17586.19683.920761.293494@montanaro.dyndns.org>
	<Pine.LNX.4.58.0607101533350.22834@server1.LFW.org>
	<e8bf7a530607101343g77148a93o39d0f671adb99e4a@mail.gmail.com>
	<1152589949.14276.16.camel@localhost.localdomain>
Message-ID: <44B46701.3040305@canterbury.ac.nz>

Matthew Barnes wrote:
> its
> meaning in C/C++ (i.e. the symbol is defined outside of the current
> scope).

It means more than that -- it means defined outside
the current *file*. That's much too drastic for
what we want.

--
Greg

From ncoghlan at gmail.com  Wed Jul 12 05:20:27 2006
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 12 Jul 2006 13:20:27 +1000
Subject: [Python-Dev] [slighly OT] Native speakers and hurting brains
In-Reply-To: <e90bsh$i6n$1@sea.gmane.org>
References: <20060705101821.7sqnxo7lu5cgg4ks@login.werra.lunarpages.com>	<ca471dc20607051027y677c7127wfe098401c366dc74@mail.gmail.com>	<e8m44k$5qb$1@sea.gmane.org><44AEF0F2.5040007@canterbury.ac.nz>	<e906f3$sdo$1@sea.gmane.org><e907o1$1i5$1@sea.gmane.org>	<e90bbm$g10$1@sea.gmane.org>
	<e90bsh$i6n$1@sea.gmane.org>
Message-ID: <44B46A7B.8060509@gmail.com>

Fredrik Lundh wrote:
> Boris Borcic wrote:
> 
>>> in what language [is] the word "sum" an appropriate synonym for "concatenate" ?
>> any that admits a+b to mean ''.join([a,b]), I'd say.
> 
> and what human language would that be ?

Python :)

Sure-computers-can-speak-it-but-so-can-humans'ly yours,
Nick.


-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
             http://www.boredomandlaziness.org

From greg.ewing at canterbury.ac.nz  Wed Jul 12 05:31:13 2006
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Wed, 12 Jul 2006 15:31:13 +1200
Subject: [Python-Dev] [slighly OT] Native speakers and hurting brains
In-Reply-To: <e906f3$sdo$1@sea.gmane.org>
References: <20060705101821.7sqnxo7lu5cgg4ks@login.werra.lunarpages.com>
	<ca471dc20607051027y677c7127wfe098401c366dc74@mail.gmail.com>
	<e8m44k$5qb$1@sea.gmane.org> <44AEF0F2.5040007@canterbury.ac.nz>
	<e906f3$sdo$1@sea.gmane.org>
Message-ID: <44B46D01.4090809@canterbury.ac.nz>

Boris Borcic wrote:

> sum() *is* exactly an attractive nuisance by *appearing* to be an obvious way of 
> chaining strings in a list (without actually being one).

But at least it fails immediately, prompting you to
look in another direction.

> I admit that there is a step of arguable interpretation from these recorded 
> facts to my diagnostic, but the latter is compatible with the facts. Your 
> version otoh looks more robust in the role of eg creation myth.

I suppose you could call that a linguistic matter, but
I don't think it's exclusively a *native* one. I suspect
that only someone with a programmer's warped mind would
make the leap from "sum" to "string concatenation" --
whether they were a native English speaker or not.

Also I don't see that my version of events is inconsistent
with the messages you quoted either -- at least not so much
as to be relegated to a myth!

--
Greg

From greg.ewing at canterbury.ac.nz  Wed Jul 12 05:44:32 2006
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Wed, 12 Jul 2006 15:44:32 +1200
Subject: [Python-Dev] [slighly OT] Native speakers and hurting brains
In-Reply-To: <200607111429.22964.gmccaughan@synaptics-uk.com>
References: <20060705101821.7sqnxo7lu5cgg4ks@login.werra.lunarpages.com>
	<44AEF0F2.5040007@canterbury.ac.nz> <e906f3$sdo$1@sea.gmane.org>
	<200607111429.22964.gmccaughan@synaptics-uk.com>
Message-ID: <44B47020.8050906@canterbury.ac.nz>

Gareth McCaughan wrote:

> (I agree that Greg's interpretation is also not well supported
> by that thread;

I was perhaps a bit excessive in claiming that language
had nothing to do with it. What I meant was that it
wasn't the *only* consideration. If there hadn't been
any disadvantages, quite possibly the ability to use
sum() on strings would have been supported, or at
least not actively blocked.

But because of the disadvantage I mentioned, *together*
with the fact that it's an odd meaning for "sum" (whether
you're a native English speaker or not), the decision was
made to disallow it.

Does that version sound less like a creation myth?-)

--
Greg

From greg.ewing at canterbury.ac.nz  Wed Jul 12 05:45:21 2006
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Wed, 12 Jul 2006 15:45:21 +1200
Subject: [Python-Dev] [slighly OT] Native speakers and hurting brains
In-Reply-To: <e90bbm$g10$1@sea.gmane.org>
References: <20060705101821.7sqnxo7lu5cgg4ks@login.werra.lunarpages.com>
	<ca471dc20607051027y677c7127wfe098401c366dc74@mail.gmail.com>
	<e8m44k$5qb$1@sea.gmane.org> <44AEF0F2.5040007@canterbury.ac.nz>
	<e906f3$sdo$1@sea.gmane.org> <e907o1$1i5$1@sea.gmane.org>
	<e90bbm$g10$1@sea.gmane.org>
Message-ID: <44B47051.9090608@canterbury.ac.nz>

Boris Borcic wrote:
> Fredrik Lundh wrote:
> 
>>in what language the word "sum" an appropriate synonym for "concatenate" ?
> 
> any that admits a+b to mean ''.join([a,b]), I'd say.

Not the same thing. "a + b" is usually pronounced
"a plus b". Now, "plus" has a somewhat wider meaning
than "sum". It sounds quite in order to say things
like "I have a sandwich plus an apple in my lunch
box", but it would be odd to say "I have the sum
of a sandwich and an apple in my lunch box".

--
Greg

From martin at v.loewis.de  Wed Jul 12 07:53:24 2006
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Wed, 12 Jul 2006 07:53:24 +0200
Subject: [Python-Dev] urllib.quote and unicode bug resuscitation attempt
In-Reply-To: <200607121242.12323.anthony@interlink.com.au>
References: <44B3ADE2.7040108@ofai.at> <44B41525.3040004@v.loewis.de>
	<200607121242.12323.anthony@interlink.com.au>
Message-ID: <44B48E54.3060908@v.loewis.de>

Anthony Baxter wrote:
>> The right thing to do is IRIs. 
> 
> For 2.5, should we at least detect that it's unicode and raise a 
> useful error?

That can certainly be done, sure.

Martin

From anthony at interlink.com.au  Wed Jul 12 07:57:14 2006
From: anthony at interlink.com.au (Anthony Baxter)
Date: Wed, 12 Jul 2006 15:57:14 +1000
Subject: [Python-Dev] changing time.strftime() to accept 0s (was: User's
	complaints)
In-Reply-To: <bbaeab100607111950o42ce6528xc3f2717292d2e58c@mail.gmail.com>
References: <bbaeab100607111950o42ce6528xc3f2717292d2e58c@mail.gmail.com>
Message-ID: <200607121557.20462.anthony@interlink.com.au>

Making strftime accept 0s is fine to be checked in, since it's a 
regression (an understandable one, but what the hell). Making it 
accept less than 9 items and have useful defaults should wait for 2.6

From tjreedy at udel.edu  Wed Jul 12 08:29:32 2006
From: tjreedy at udel.edu (Terry Reedy)
Date: Wed, 12 Jul 2006 02:29:32 -0400
Subject: [Python-Dev] changing time.strftime() to accept 0s (was:
	User'scomplaints)
References: <bbaeab100607111950o42ce6528xc3f2717292d2e58c@mail.gmail.com>
	<200607121557.20462.anthony@interlink.com.au>
Message-ID: <e924sd$38g$1@sea.gmane.org>


"Anthony Baxter" <anthony at interlink.com.au> wrote in message 
news:200607121557.20462.anthony at interlink.com.au...
> Making strftime accept 0s is fine to be checked in, since it's a
> regression (an understandable one, but what the hell).

Once it is, someone should  put a note back on the complainant's blog page. 




From martin at v.loewis.de  Wed Jul 12 08:34:49 2006
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Wed, 12 Jul 2006 08:34:49 +0200
Subject: [Python-Dev] User's complaints
In-Reply-To: <17588.18475.171696.506911@montanaro.dyndns.org>
References: <ee2a432c0607041649i7ecb6cc1w8ebdf47f7d02097a@mail.gmail.com>	<dc1166600607111730i44912cedqcea01d918f2b0a85@mail.gmail.com>
	<17588.18475.171696.506911@montanaro.dyndns.org>
Message-ID: <44B49809.5050707@v.loewis.de>

skip at pobox.com wrote:
> The way I used to format dates using time.strftime does indeed no longer
> work.
> 
> Python 2.3:
> 
>     >>> import time
>     >>> time.strftime("%Y-%m-%d", (2005, 6, 4) + (0,)*6)
>     '2005-06-04'

Is there any specific reason you couldn't write

"%d-%02d-%02d" % (2005, 6, 4)

(i.e. not use strftime at all)? It seems strange to fake a time tuple
just to use that function, in particular if the time formatting should
not use any locale-specific output.

> I don't actually run into this problem as I've pretty much converted to use
> datetime in new code.  I also realize that's not documented as the way it
> should be done, but I'm fairly certain it was common usage before the
> datetime module came along.  Still, it is a bit annoying that the
> (undocumented, but I think de facto) commonly used idiom no longer works.

I guess this was caused by this change:

    /* Checks added to make sure strftime() does not crash Python by
       indexing blindly into some array for a textual representation
       by some bad index (fixes bug #897625).

       No check for year since handled in gettmarg().
    */

So this was changed in response to a bug report about a crash.


Regards,
Martin

From stefan.rank at ofai.at  Wed Jul 12 10:27:32 2006
From: stefan.rank at ofai.at (Stefan Rank)
Date: Wed, 12 Jul 2006 10:27:32 +0200
Subject: [Python-Dev] urllib.quote and unicode bug resuscitation attempt
In-Reply-To: <44B48E54.3060908@v.loewis.de>
References: <44B3ADE2.7040108@ofai.at>
	<44B41525.3040004@v.loewis.de>	<200607121242.12323.anthony@interlink.com.au>
	<44B48E54.3060908@v.loewis.de>
Message-ID: <44B4B274.4050602@ofai.at>

on 12.07.2006 07:53 Martin v. L?wis said the following:
> Anthony Baxter wrote:
>>> The right thing to do is IRIs. 
>> For 2.5, should we at least detect that it's unicode and raise a 
>> useful error?
> 
> That can certainly be done, sure.
> 
> Martin

That would be great.

And I agree that updating urllib.quote for unicode should be part of a 
grand plan that updates all of urllib[2] and introduces an irilib / 
urischemes / uriparse module in 2.6 as Martin and John J Lee suggested.
  =)

cheers,
stefan


From skip at pobox.com  Wed Jul 12 13:13:47 2006
From: skip at pobox.com (skip at pobox.com)
Date: Wed, 12 Jul 2006 06:13:47 -0500
Subject: [Python-Dev] User's complaints
In-Reply-To: <44B49809.5050707@v.loewis.de>
References: <ee2a432c0607041649i7ecb6cc1w8ebdf47f7d02097a@mail.gmail.com>
	<dc1166600607111730i44912cedqcea01d918f2b0a85@mail.gmail.com>
	<17588.18475.171696.506911@montanaro.dyndns.org>
	<44B49809.5050707@v.loewis.de>
Message-ID: <17588.55659.727250.840456@montanaro.dyndns.org>


    >> Python 2.3:
    >> 
    >> >>> import time
    >> >>> time.strftime("%Y-%m-%d", (2005, 6, 4) + (0,)*6)
    >> '2005-06-04'

    Martin> Is there any specific reason you couldn't write

    Martin> "%d-%02d-%02d" % (2005, 6, 4)

    Martin> (i.e. not use strftime at all)?

Sure, but that was just me being lazy typing at the interactive prompt.
%Y-%m-%d is just about the only date format I can ever remember without
consulting the strftime(3) man page. ;-)  Suppose I had used

    >>> time.strftime("%b %d, %Y", (2005, 6, 4) + (1,)*6)
    'Jun 04, 2005'

instead (switching to the all-ones default that still works)?

    Martin> So this was changed in response to a bug report about a crash.

Yeah, but it broke common (at the time) usage.

Skip

From g.brandl at gmx.net  Wed Jul 12 13:55:31 2006
From: g.brandl at gmx.net (Georg Brandl)
Date: Wed, 12 Jul 2006 13:55:31 +0200
Subject: [Python-Dev] Long options support
In-Reply-To: <e6okjb$3s0$1@sea.gmane.org>
References: <e6oht3$q2n$1@sea.gmane.org> <e6okjb$3s0$1@sea.gmane.org>
Message-ID: <e92nvj$udo$1@sea.gmane.org>

Fredrik Lundh wrote:
> Georg Brandl wrote:
> 
>> I've just closed a bug report wishing for long option support,
>> pointing to a patch sitting in the patch tracker implementing
>> this.
>>
>> Should we accept at least the very common options "--help" and
>> "--version" in 2.5?
> 
> Guido pronounced on this in May

Late it comes, but here is a patch for getopt.c implementing
this pronouncement. I think there's no need to wait for 2.6 with it,
or is there?

Cheers,
Georg


Index: Python/getopt.c
===================================================================
--- Python/getopt.c     (Revision 50599)
+++ Python/getopt.c     (Arbeitskopie)
@@ -24,6 +24,9 @@
  * davegottner at delphi.com.
  *---------------------------------------------------------------------------*/

+/* Modified to support --help and --version, as well as /? on Windows
+ * by Georg Brandl. */
+
 #include <stdio.h>
 #include <string.h>

@@ -43,15 +46,35 @@

        if (*opt_ptr == '\0') {

-               if (_PyOS_optind >= argc || argv[_PyOS_optind][0] != '-' ||
-                   argv[_PyOS_optind][1] == '\0' /* lone dash */ )
+               if (_PyOS_optind >= argc)
                        return -1;
+#ifdef MS_WINDOWS
+               else if (strcmp(argv[_PyOS_optind], "/?") == 0) {
+                       ++_PyOS_optind;
+                       return 'h';
+               }
+#endif

+               else if (argv[_PyOS_optind][0] != '-' ||
+                        argv[_PyOS_optind][1] == '\0' /* lone dash */ )
+                       return -1;
+
                else if (strcmp(argv[_PyOS_optind], "--") == 0) {
                        ++_PyOS_optind;
                        return -1;
                }

+               else if (strcmp(argv[_PyOS_optind], "--help") == 0) {
+                       ++_PyOS_optind;
+                       return 'h';
+               }
+
+               else if (strcmp(argv[_PyOS_optind], "--version") == 0) {
+                       ++_PyOS_optind;
+                       return 'V';
+               }
+
+
                opt_ptr = &argv[_PyOS_optind++][1];
        }

@@ -62,7 +85,7 @@
                if (_PyOS_opterr)
                        fprintf(stderr, "Unknown option: -%c\n", option);

-               return '?';
+               return '_';
        }

        if (*(ptr + 1) == ':') {
@@ -76,7 +99,7 @@
                                if (_PyOS_opterr)
                                        fprintf(stderr,
                            "Argument expected for the -%c option\n", option);
-                               return '?';
+                               return '_';
                        }

                        _PyOS_optarg = argv[_PyOS_optind++];
Index: Modules/main.c
===================================================================
--- Modules/main.c      (Revision 50599)
+++ Modules/main.c      (Arbeitskopie)
@@ -40,7 +40,7 @@
 static int  orig_argc;

 /* command line options */
-#define BASE_OPTS "c:dEhim:OQ:StuUvVW:xX"
+#define BASE_OPTS "c:dEhim:OQ:StuUvVW:xX?"

 #ifndef RISCOS
 #define PROGRAM_OPTS BASE_OPTS
@@ -62,7 +62,7 @@
 -c cmd : program passed in as string (terminates option list)\n\
 -d     : debug output from parser (also PYTHONDEBUG=x)\n\
 -E     : ignore environment variables (such as PYTHONPATH)\n\
--h     : print this help message and exit\n\
+-h     : print this help message and exit (also --help)\n\
 -i     : inspect interactively after running script, (also PYTHONINSPECT=x)\n\
          and force prompts, even if stdin does not appear to be a terminal\n\
 ";
@@ -78,7 +78,7 @@
 static char *usage_3 = "\
          see man page for details on internal buffering relating to '-u'\n\
 -v     : verbose (trace import statements) (also PYTHONVERBOSE=x)\n\
--V     : print the Python version number and exit\n\
+-V     : print the Python version number and exit (also --version)\n\
 -W arg : warning control (arg is action:message:category:module:lineno)\n\
 -x     : skip first line of source, allowing use of non-Unix forms of #!cmd\n\
 file   : program read from script file\n\
@@ -313,6 +313,7 @@
                        Py_UnicodeFlag++;
                        break;
                case 'h':
+               case '?':
                        help++;
                        break;
                case 'V':


From fredrik at pythonware.com  Wed Jul 12 15:41:04 2006
From: fredrik at pythonware.com (Fredrik Lundh)
Date: Wed, 12 Jul 2006 15:41:04 +0200
Subject: [Python-Dev] Long options support
References: <e6oht3$q2n$1@sea.gmane.org> <e6okjb$3s0$1@sea.gmane.org>
	<e92nvj$udo$1@sea.gmane.org>
Message-ID: <e92u65$l9i$1@sea.gmane.org>

Georg Brandl wrote:

> Late it comes, but here is a patch for getopt.c implementing
> this pronouncement. I think there's no need to wait for 2.6 with it,
> or is there?

check it in already.

</F> 




From bioinformed at gmail.com  Wed Jul 12 16:33:28 2006
From: bioinformed at gmail.com (Kevin Jacobs <jacobs@bioinformed.com>)
Date: Wed, 12 Jul 2006 10:33:28 -0400
Subject: [Python-Dev] Behavior change in subprocess.py
Message-ID: <2e1434c10607120733h3b2a0d86l1908518593d2eb49@mail.gmail.com>

During my testing of Python 2.5b2, I've found something that may be worthy
of discussion.  I suspect that recent GC and finalization changes have
altered the behavior of the Popen object in subprocess.py.  I am now getting
many many many finalization warnings in my code like:

Exception exceptions.AttributeError: "'NoneType' object has no attribute
'append'" in <bound method Popen.__del__ of <subprocess.Popen object at
0x2aaaab910950>> ignored

Is this a bug or a feature?  Personally, I'd like to see these messages
silenced, since it is being generated during interpreter shutdown.  The
following patch does the trick for me:

--- /usr/local/lib/python2.5/subprocess.py      2006-07-11 14:11:
59.000000000 -0400
+++ subprocess.py       2006-07-12 10:17:09.000000000 -0400
@@ -613,7 +613,7 @@
             return
         # In case the child hasn't been waited on, check if it's done.
         self.poll(_deadstate=sys.maxint)
-        if self.returncode is None:
+        if self.returncode is None and _active is not None:
             # Child is still running, keep us alive until we can wait on
it.
             _active.append(self)


Note that popen.py does something similar, though I am not convinced that
the test is right or if it is doing something more subtle:

    def __del__(self):
        # In case the child hasn't been waited on, check if it's done.
        self.poll(_deadstate=sys.maxint)
        if self.sts < 0:
            if _active:
                # Child is still running, keep us alive until we can wait on
it.
                _active.append(self)

Regards,
-Kevin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20060712/6b697648/attachment.htm 

From bborcic at gmail.com  Wed Jul 12 16:36:28 2006
From: bborcic at gmail.com (Boris Borcic)
Date: Wed, 12 Jul 2006 16:36:28 +0200
Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
In-Reply-To: <e914t8$gsv$1@sea.gmane.org>
References: <e8k58c$q7u$1@sea.gmane.org>	<r01050400-1039-AEFC2D140DA011DBB8DF001124365170@[10.0.0.24]><e8ldjg$eq7$1@sea.gmane.org>	<e90j6o$fij$1@sea.gmane.org>
	<e914t8$gsv$1@sea.gmane.org>
Message-ID: <e931eh$2c0$1@sea.gmane.org>

Terry Reedy wrote:
> "Boris Borcic" <bborcic at gmail.com> wrote in message 
> news:e90j6o$fij$1 at sea.gmane.org...
>> I agree with you (and argued it in "scopes vs augmented assignment vs 
>> sets"
>> recently) that mutating would be sufficient /if/ the compiler would view
>> augmented assignment as mutations operators :
> 
> Mutation is an operation on objects.  Binding is an operation on 
> namespaces.  The difference between objects and namespaces (and the actions 
> thereupon) is fundamental to Python.

If you want to put it that way, but I think it muddies the relevant water to 
confuse scopes with namespaces, or compiler with interpreter responsibilities.

Deciding whether an augmented assignment results in a bona-fide mutation or in 
the rebinding of an already bound name in an unchanging namespace : that's the 
responsibility of the interpreter. [Well, nowa-pypy-days I guess we should say, 
not even of the interpreter, but of the object space].

Deciding how names bind to lexical scopes, that's the responsibility of the 
compiler. When faced with such code as below [ while mutable_int() might as well 
equal (lambda x : x) as far as the compiler is concerned ]

 >       def counter(num):
 > >           num = mutable_int(num)
 > >           def inc():
 > >               num += 1
 > >               return num
 > >           return inc

It is disingenuous that the /scope-binding phase/ of the compiler design feigns 
to believe that my "num += 1" is really more like "num = 2" than it is like a 
mutation operator, given that :

         whether the statement ultimately results in REbinding or mutation
         is /irrelevant/ to the fact that any augmented assignment requires
         an autonomous initial binding of the variable to ever function.

- what is exactly the sort of assertions independent of run-time details that 
compilers are made to reason about.


> Asking the interpreter to view one 
> thing as something else which it isn't can only lead to more confusion.

I think its impossible to be more confused than setting things up so that the 
programmer of counter() above gets :

UnboundLocalError: local variable 'num' referenced before assignment

When what is really meant is (your drift no ?) :

<< We think allowing you what you want would make

a=b=c ; a += d ; b = b+d ; assert a == b

more difficult for newbies to understand and predict in context >>

> In 
> particular, asking that arithmetic operations on immutable numbers be seen 
> as mutations seems wacky to me.

Nobody really asked for this (I invite you to check), but even so I feel an 
exact analysis of the impact on use-cases would be worth its while.

> 
>> which it doesn't as far as  concerns scopes where a variable
>> appears as target only of /augmented/ assignments.
> 
> The interpreter/compiler, as far as I can think, never views binding as 
> mutation, nor should it.  The request that it do so

The request is not for the "compiler/interpreter" to view binding as mutation, 
it is for the /compiler/ to view /REbinding/ just like mutation - which it is as 
far as compiler responsibilities are concerned imo.

> makes me wonder whether 
> it might have been a mistake to allow mutable objects in an augmented 
> assignment to choose to implement the associated operation as an in-place 
> mutation.

I agree with you that removing this possibility would provide better consistency 
to your position.

Best regards, Boris Borcic
--
"C++ is a contradiction in terms" - Lorentz, Einstein, Poincar?


From guido at python.org  Wed Jul 12 17:26:15 2006
From: guido at python.org (Guido van Rossum)
Date: Wed, 12 Jul 2006 08:26:15 -0700
Subject: [Python-Dev] Autoloading? (Making Queue.Queue easier to use)
In-Reply-To: <4B9D477017AEEC4188C18CB8DA8364B606156A02@pusehe0l.eh.pweh.com>
References: <4B9D477017AEEC4188C18CB8DA8364B606156A02@pusehe0l.eh.pweh.com>
Message-ID: <ca471dc20607120826xffb2b59wfc12917a070b0619@mail.gmail.com>

Why do I have the feeling you sent this to the wrong list?

On 7/10/06, Perkins, Christopher <christopher.perkins at pw.utc.com> wrote:
> John,
>
> I see what you are doing with the algorithm now, and I can easily re-factor
> it.  What I am having issues with is how structured it is.  5 minute
> windows?  Then running hours will always be recorded in 1/12th time steps.
>
> Would it not be more accurate to record engine time where the breaker was
> closed?
>
> -chris
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org
>


-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From anthony at interlink.com.au  Wed Jul 12 17:25:38 2006
From: anthony at interlink.com.au (Anthony Baxter)
Date: Thu, 13 Jul 2006 01:25:38 +1000
Subject: [Python-Dev] Long options support
In-Reply-To: <e92nvj$udo$1@sea.gmane.org>
References: <e6oht3$q2n$1@sea.gmane.org> <e6okjb$3s0$1@sea.gmane.org>
	<e92nvj$udo$1@sea.gmane.org>
Message-ID: <200607130125.44037.anthony@interlink.com.au>

On Wednesday 12 July 2006 21:55, Georg Brandl wrote:
> >> Should we accept at least the very common options "--help" and
> >> "--version" in 2.5?
> >
> > Guido pronounced on this in May
>
> Late it comes, but here is a patch for getopt.c implementing
> this pronouncement. I think there's no need to wait for 2.6 with
> it, or is there?

Check it in.

Anthony

From barry at python.org  Wed Jul 12 20:56:36 2006
From: barry at python.org (Barry Warsaw)
Date: Wed, 12 Jul 2006 14:56:36 -0400
Subject: [Python-Dev] Support for PyGetSetDefs in pydoc
In-Reply-To: <A027CA35-EE77-4658-BA59-E849368C440B@python.org>
References: <A027CA35-EE77-4658-BA59-E849368C440B@python.org>
Message-ID: <EF6E6AF2-654F-4F53-91A8-A6C9B8A80745@python.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Jul 10, 2006, at 9:52 PM, Barry Warsaw wrote:

> Patch #1520294 adds support for attributes defined with PyGetSetDef  
> in extension modules to pydoc, specifically so things like help 
> (array.array.typecode) gives something useful, like the attribute's  
> docstring for instance.
>
> Along the way, I added types.GetSetterType and inspect.isgetsetter 
> (), along with tests and docs.  Now I definitely think that the  
> pydoc change should go into Python 2.5 and be backported to Python  
> 2.4, but it's a bit of a question whether the types.py and  
> inspect.py changes should go into Python 2.5 now that we're in beta.
>
> I personal think they're worthwhile changes to go it, but I won't  
> argue too hard for them since the pydoc fixes can be implemented  
> with local changes to pydoc.py alone.

Turns out there is at least one other type that I think should be  
better supported, but which I'm having a hard time figuring out the  
right way to do this.  Member descriptors are one such type, an  
example of which is datetime.timedelta.days

 >>> import datetime
 >>> type(datetime.timedelta.days)
<type 'member_descriptor'>

Do a help(datetime.timedelta.days) and you get nothing useful.

Now, I started down a similar path for solving this as I did the  
other day for getset descriptors, but I've run into a bit of a  
roadblock.  What you basically want to do is add a constant to the  
types module, then add an is-predicate to inspect.py, then add the  
necessary glue to pydoc.py.  The problem is that you have to have  
something readily available to call type() on to get a member  
descriptor type, and for that you have to go hunting around in  
Python's C layer.  Basically you're looking for a type initialized  
with PyObject_HEAD_INIT(NULL) /and/ has a tp_members slot.

There are a few of them around, such as the datetime.timedelta type  
mentioned above, but they all have a pretty serious problem.  You  
cannot import their module in types.py.  I'm not exactly sure why  
this is, but if you put

import datetime

in types.py, you'll get the "import site" failed error.  Run with -v  
and you get buried in the output:

'import site' failed; traceback:
Traceback (most recent call last):
   File "/Users/barry/projects/python/Lib/site.py", line 62, in <module>
     import os
   File "/Users/barry/projects/python/Lib/os.py", line 690, in <module>
     import copy_reg as _copy_reg
   File "/Users/barry/projects/python/Lib/copy_reg.py", line 7, in  
<module>
     from types import ClassType as _ClassType
   File "/Users/barry/projects/python/Lib/types.py", line 90, in  
<module>
     import datetime
ImportError: No module named datetime

I think the problem is that the platform-specific extension module  
directory isn't on sys.path yet, so it can't e.g. find datetime.so.   
Of course, datetime is perfectly importable later on.

As far as I can tell, there does not seem to be any candidate member  
descriptors that are built into the interpreter and guaranteed to be  
there early enough to import into types, but maybe I'm missing  
something.

I can think of a number of ways to address this, with varying levels  
of ickyness, but I wanted to see what you all thought.

For example, I could change inspect locally so that it gets the type  
of datetime.timedelta.days without adding a constant to types.py.  Or  
I could patch pydoc.py directly and leave even inspect.py out of it.   
Or I could create some stupid internal type in some stupid internal  
module who's only purpose would be to have a handle on member  
descriptors.  Or I could change datetime to be built-in.  (see what I  
mean about levels of ickyness? :).

I'm up for suggestions.  I think this would be worthwhile to address  
in Python 2.5 since I think it would be good to have an accurate  
representation of Python's built-in types in types.py.  Ultimately, I  
really care about teaching pydoc.help() about instances of these  
types so that users can get better help when they encounter them  
(such as might be the case in 3rd party extension modules).

Suggestions are welcome.
- -Barry

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.3 (Darwin)

iQCVAwUBRLVF6XEjvBPtnXfVAQJVmAP5ATS56TtrdUamo+CxdNHrP6j7zxwEef2t
fCM3XuEjv/Y+UH3goQIm34ENySjqYgC/whQqXGPWa/5GVc1MkhZo3W2BsA4NJ8yq
Za8+6+KsWaR625qj/mh/Fbw5I/2vOE3MClATuQ75aCJjLpSAwhxYqleYPTD7tBiF
P4zc+2OYedU=
=ikCK
-----END PGP SIGNATURE-----

From arigo at tunes.org  Wed Jul 12 22:29:08 2006
From: arigo at tunes.org (Armin Rigo)
Date: Wed, 12 Jul 2006 22:29:08 +0200
Subject: [Python-Dev] User's complaints
In-Reply-To: <bbaeab100607111805g5f6ca07cvf0ba9cb09b68bcfa@mail.gmail.com>
References: <ee2a432c0607041649i7ecb6cc1w8ebdf47f7d02097a@mail.gmail.com>
	<dc1166600607111730i44912cedqcea01d918f2b0a85@mail.gmail.com>
	<17588.18475.171696.506911@montanaro.dyndns.org>
	<bbaeab100607111805g5f6ca07cvf0ba9cb09b68bcfa@mail.gmail.com>
Message-ID: <20060712202908.GA13303@code0.codespeak.net>

Hi Brett,

On Tue, Jul 11, 2006 at 06:05:21PM -0700, Brett Cannon wrote:
> It is the last point in the first paragraph on time.strftime() discussing
> what changed in Python 2.4 as to what the change was.  It's also in
> Misc/NEWS .  Basically the guy didn't read the release notes or the docs to
> see why that changed and that it was legitimate and needed for stability.

Surely everybody should read and think carefully about each (longish)
NEWS file for each software package whenever they update their machines
or switch to one with newer software than they last used.

Or if they cannot bother, surely they should read at least Python's?

I guess I'm going to side with Greg Black on his blog entry.
Only two breakages is certainly nice, and I know that we all try quite
hard to minimize that; that's probably still two breakages too much.


Armin

From nnorwitz at gmail.com  Wed Jul 12 23:09:23 2006
From: nnorwitz at gmail.com (Neal Norwitz)
Date: Wed, 12 Jul 2006 14:09:23 -0700
Subject: [Python-Dev] User's complaints
In-Reply-To: <20060712202908.GA13303@code0.codespeak.net>
References: <ee2a432c0607041649i7ecb6cc1w8ebdf47f7d02097a@mail.gmail.com>
	<dc1166600607111730i44912cedqcea01d918f2b0a85@mail.gmail.com>
	<17588.18475.171696.506911@montanaro.dyndns.org>
	<bbaeab100607111805g5f6ca07cvf0ba9cb09b68bcfa@mail.gmail.com>
	<20060712202908.GA13303@code0.codespeak.net>
Message-ID: <ee2a432c0607121409h566f7ffbv54dc53ee08a1e026@mail.gmail.com>

On 7/12/06, Armin Rigo <arigo at tunes.org> wrote:
>
> Only two breakages is certainly nice, and I know that we all try quite
> hard to minimize that; that's probably still two breakages too much.

I agree, but some of this responsibility has to fall to users.
Sometimes these breakages are bugs, pure and simple.  Our tests don't
catch everything.  This is why it's really, really important to get as
many alpha/beta testers as possible.  Had the issues been brought up
before 2.4 was released, we could have addressed them.

Basically, if you care about these things, test before we release.
That way the issues can be addressed. If you don't care enough to test
and get bitten by a problem, quit whining, you had your chance.  I
have no sympathy for someone who is only willing to whine after the
fact, but unwilling to do any work.  That's really not helpful.

n
--
PS  People do this all the time in politics too.  They complain about
someone, when they couldn't even be bothered to vote.

From jimjjewett at gmail.com  Wed Jul 12 23:23:35 2006
From: jimjjewett at gmail.com (Jim Jewett)
Date: Wed, 12 Jul 2006 17:23:35 -0400
Subject: [Python-Dev] Restricted execution: what's the threat model?
Message-ID: <fb6fbf560607121423x95ab056odd2502826b171fd3@mail.gmail.com>

Ka-Ping Yee writes:

>   A.  The interpreter will not crash no matter what Python code
>       it is given to execute.

Why?

We don't want it to crash the embedding app (which might be another
python interpreter), but if the sandboxed interpreter itself crashes,
is that so bad?  The embedding app should just act as though that
interpreter exited, possibly with a status code.

>  B.  Python programs running in different interpreters embedded
>       in the same process cannot communicate with each other.

Why not?  Can't eavesdrop, yes.   Can't force a connection, so that
the other interpreter is free to ignore them.  Maybe even make it
lockable, like sockets -- but it isn't something worth promising.

>   C.  Python programs running in different interpreters embedded
>       in the same process cannot access each other's Python objects.

Note that Brett's assumption of shared extension modules violates this
-- but I'm not sure why he needs to assume that.  (Because of the
init-only-once semantics, I'm not even sure it is a good idea to share
them.)

>   D.  A given piece of Python code cannot access or communicate
>       with certain Python objects in the same interpreter.

Why not?  Is this just a way of allowing lightweight subinterpreters?
Or do you really mean that they can't replace or modify certain
objects, such as the permission-controlling code?

>   E.  A given piece of Python code can access only a limited set
>       of Python objects in the same interpreter.

Does this include objects it creates?  Or are you just saying that it
will behave as if its builtins were segregated, and not see changes
made by another interpreter?

-jJ

From bob at redivi.com  Wed Jul 12 23:50:13 2006
From: bob at redivi.com (Bob Ippolito)
Date: Wed, 12 Jul 2006 14:50:13 -0700
Subject: [Python-Dev] Restricted execution: what's the threat model?
In-Reply-To: <fb6fbf560607121423x95ab056odd2502826b171fd3@mail.gmail.com>
References: <fb6fbf560607121423x95ab056odd2502826b171fd3@mail.gmail.com>
Message-ID: <F5F62DAC-59D1-45CD-8E52-64A849AC853B@redivi.com>


On Jul 12, 2006, at 2:23 PM, Jim Jewett wrote:

> Ka-Ping Yee writes:
>
>>   A.  The interpreter will not crash no matter what Python code
>>       it is given to execute.
>
> Why?
>
> We don't want it to crash the embedding app (which might be another
> python interpreter), but if the sandboxed interpreter itself crashes,
> is that so bad?  The embedding app should just act as though that
> interpreter exited, possibly with a status code.

When he says crash, I'd have to imagine that he means of the segfault  
variety. Good luck saving the embedding app after that.

>>   C.  Python programs running in different interpreters embedded
>>       in the same process cannot access each other's Python objects.
>
> Note that Brett's assumption of shared extension modules violates this
> -- but I'm not sure why he needs to assume that.  (Because of the
> init-only-once semantics, I'm not even sure it is a good idea to share
> them.)

Well if you don't share them, you can't have them at all other than  
in the main trusted interpreter. C extensions can only be safely  
initialized once and they often cache objects in static variables...  
lots of C modules aren't even safe to use when combined with multiple  
interpreters and threads (e.g. PyGILState API), so I guess that  
perhaps the C API should be refined anyway.

-bob




From fredrik at pythonware.com  Thu Jul 13 00:13:51 2006
From: fredrik at pythonware.com (Fredrik Lundh)
Date: Thu, 13 Jul 2006 00:13:51 +0200
Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
In-Reply-To: <e90j6o$fij$1@sea.gmane.org>
References: <e8k58c$q7u$1@sea.gmane.org>	<r01050400-1039-AEFC2D140DA011DBB8DF001124365170@[10.0.0.24]>	<e8ldjg$eq7$1@sea.gmane.org>
	<e90j6o$fij$1@sea.gmane.org>
Message-ID: <e93s6t$5fu$1@sea.gmane.org>

Boris Borcic wrote:

>> note that most examples of this type already work, if the target type is 
>> mutable, and implement the right operations:
>>
>>       def counter(num):
>>           num = mutable_int(num)
>>           def inc():
>>               num += 1
>>               return num
>>           return inc
> 
> I agree with you (and argued it in "scopes vs augmented assignment vs sets" 
> recently) that mutating would be sufficient /if/ the compiler would view 
> augmented assignment as mutations operators

feel free to replace that += with an .add(1) method call; the point 
wasn't the behaviour of augmented assigment, the point was that that the 
most common use pattern involves *mutation* of the target object.

the syntax isn't that important, really.

</F>


From python-dev at zesty.ca  Thu Jul 13 00:38:44 2006
From: python-dev at zesty.ca (Ka-Ping Yee)
Date: Wed, 12 Jul 2006 17:38:44 -0500 (CDT)
Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
In-Reply-To: <e93s6t$5fu$1@sea.gmane.org>
References: <e8k58c$q7u$1@sea.gmane.org>
	<r01050400-1039-AEFC2D140DA011DBB8DF001124365170@[10.0.0.24]>
	<e8ldjg$eq7$1@sea.gmane.org> <e90j6o$fij$1@sea.gmane.org>
	<e93s6t$5fu$1@sea.gmane.org>
Message-ID: <Pine.LNX.4.58.0607121737230.31087@server1.LFW.org>

On Thu, 13 Jul 2006, Fredrik Lundh wrote:
> Boris Borcic wrote:
>
> >> note that most examples of this type already work, if the target type is
> >> mutable, and implement the right operations:
> >>
> >>       def counter(num):
> >>           num = mutable_int(num)
> >>           def inc():
> >>               num += 1
> >>               return num
> >>           return inc
[...]
> feel free to replace that += with an .add(1) method call; the point
> wasn't the behaviour of augmented assigment, the point was that that the
> most common use pattern involves *mutation* of the target object.

I don't think that's clear.  In:

    a = 1
    b = a
    a += 1

it doesn't seem common to me that one would expect b to become 2.
Sometimes you want mutation, sometimes you don't.  I don't think
that necessarily follows from whether you are accessing a free
variable.


-- ?!ng

From brett at python.org  Thu Jul 13 00:51:45 2006
From: brett at python.org (Brett Cannon)
Date: Wed, 12 Jul 2006 15:51:45 -0700
Subject: [Python-Dev] User's complaints
In-Reply-To: <20060712202908.GA13303@code0.codespeak.net>
References: <ee2a432c0607041649i7ecb6cc1w8ebdf47f7d02097a@mail.gmail.com>
	<dc1166600607111730i44912cedqcea01d918f2b0a85@mail.gmail.com>
	<17588.18475.171696.506911@montanaro.dyndns.org>
	<bbaeab100607111805g5f6ca07cvf0ba9cb09b68bcfa@mail.gmail.com>
	<20060712202908.GA13303@code0.codespeak.net>
Message-ID: <bbaeab100607121551g425281fcne3fa001e32471f0a@mail.gmail.com>

On 7/12/06, Armin Rigo <arigo at tunes.org> wrote:
>
> Hi Brett,
>
> On Tue, Jul 11, 2006 at 06:05:21PM -0700, Brett Cannon wrote:
> > It is the last point in the first paragraph on time.strftime()
> discussing
> > what changed in Python 2.4 as to what the change was.  It's also in
> > Misc/NEWS .  Basically the guy didn't read the release notes or the docs
> to
> > see why that changed and that it was legitimate and needed for
> stability.
>
> Surely everybody should read and think carefully about each (longish)
> NEWS file for each software package whenever they update their machines
> or switch to one with newer software than they last used.
>
> Or if they cannot bother, surely they should read at least Python's?


Obviously I don't expect people to read all of the docs; I know I don't.
But when something I am using starts to act differently in a program (and
especially a programming language), I do do at least a little
investigating.  I don't expect people to read Misc/NEWS upfront, but if
something suddnely starts to act differently I would expect them to at least
try to figure it out and hopefully report a bug if it is one.

I guess I'm going to side with Greg Black on his blog entry.
> Only two breakages is certainly nice, and I know that we all try quite
> hard to minimize that; that's probably still two breakages too much.


As Neal said, we are not perfect; bugs happen.  If we all gave up on a piece
of software after two bugs we would not be able to turn our computers.  =)

-Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20060712/f1d7446b/attachment.htm 

From brett at python.org  Thu Jul 13 00:58:50 2006
From: brett at python.org (Brett Cannon)
Date: Wed, 12 Jul 2006 15:58:50 -0700
Subject: [Python-Dev] Restricted execution: what's the threat model?
In-Reply-To: <fb6fbf560607121423x95ab056odd2502826b171fd3@mail.gmail.com>
References: <fb6fbf560607121423x95ab056odd2502826b171fd3@mail.gmail.com>
Message-ID: <bbaeab100607121558l3b34324cmbb83f3a53b7bad1a@mail.gmail.com>

On 7/12/06, Jim Jewett <jimjjewett at gmail.com> wrote:
>
> Ka-Ping Yee writes:
>
> >   A.  The interpreter will not crash no matter what Python code
> >       it is given to execute.
>
> Why?
>
> We don't want it to crash the embedding app (which might be another
> python interpreter), but if the sandboxed interpreter itself crashes,
> is that so bad?  The embedding app should just act as though that
> interpreter exited, possibly with a status code.


As Bob said, "crash" means segfaulting the Python proceess.  Can't exactly
save yourself from that one easily.  =)

>  B.  Python programs running in different interpreters embedded
> >       in the same process cannot communicate with each other.
>
> Why not?  Can't eavesdrop, yes.   Can't force a connection, so that
> the other interpreter is free to ignore them.  Maybe even make it
> lockable, like sockets -- but it isn't something worth promising.


>From an initial design point it is.  It is an assumption I want to be able
to make about the design.  If we can come up with a reasonable way for it to
work, great.  But to begin with, I am assuming objects created within an
interpreter will not be passed into another one.

>   C.  Python programs running in different interpreters embedded
> >       in the same process cannot access each other's Python objects.
>
> Note that Brett's assumption of shared extension modules violates this
> -- but I'm not sure why he needs to assume that.  (Because of the
> init-only-once semantics, I'm not even sure it is a good idea to share
> them.)


Security reasons.  If I can get  a Python object in other interpreter with
more rights to do something for me I have a serious problem.

Do realize that things assumed might have to be made true as much as
possible.  And in my Threat Model list, I have the caveat that C extension
modules are exempt from this.

>   D.  A given piece of Python code cannot access or communicate
> >       with certain Python objects in the same interpreter.
>
> Why not?  Is this just a way of allowing lightweight subinterpreters?
> Or do you really mean that they can't replace or modify certain
> objects, such as the permission-controlling code?


This one is not in my Threat Model.

>   E.  A given piece of Python code can access only a limited set
> >       of Python objects in the same interpreter.
>
> Does this include objects it creates?  Or are you just saying that it
> will behave as if its builtins were segregated, and not see changes
> made by another interpreter?


I am going with your latter assumption in my design.

-Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20060712/98b621f6/attachment.html 

From guido at python.org  Thu Jul 13 03:34:48 2006
From: guido at python.org (Guido van Rossum)
Date: Wed, 12 Jul 2006 18:34:48 -0700
Subject: [Python-Dev] User's complaints
In-Reply-To: <20060712202908.GA13303@code0.codespeak.net>
References: <ee2a432c0607041649i7ecb6cc1w8ebdf47f7d02097a@mail.gmail.com>
	<dc1166600607111730i44912cedqcea01d918f2b0a85@mail.gmail.com>
	<17588.18475.171696.506911@montanaro.dyndns.org>
	<bbaeab100607111805g5f6ca07cvf0ba9cb09b68bcfa@mail.gmail.com>
	<20060712202908.GA13303@code0.codespeak.net>
Message-ID: <ca471dc20607121834n4f1bd00ey63cae6dc9ebe1db3@mail.gmail.com>

On 7/12/06, Armin Rigo <arigo at tunes.org> wrote:
> I guess I'm going to side with Greg Black on his blog entry.

I seem to recall that that particular one wass *not* an accidental
bug. I believe I fell over exactly the problem that Greg Black
complained about (or almost the same; maybe my problem was setting the
month and day to zero and formatting only the time :-) and tried to
convince the author to change it; but was told that the new behavior
was never documented and that the new behavior was somehow better; and
I didn't fight it further then. Do I remember this correctly? Does
anybody else recall?

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From brett at python.org  Thu Jul 13 03:42:49 2006
From: brett at python.org (Brett Cannon)
Date: Wed, 12 Jul 2006 18:42:49 -0700
Subject: [Python-Dev] User's complaints
In-Reply-To: <ca471dc20607121834n4f1bd00ey63cae6dc9ebe1db3@mail.gmail.com>
References: <ee2a432c0607041649i7ecb6cc1w8ebdf47f7d02097a@mail.gmail.com>
	<dc1166600607111730i44912cedqcea01d918f2b0a85@mail.gmail.com>
	<17588.18475.171696.506911@montanaro.dyndns.org>
	<bbaeab100607111805g5f6ca07cvf0ba9cb09b68bcfa@mail.gmail.com>
	<20060712202908.GA13303@code0.codespeak.net>
	<ca471dc20607121834n4f1bd00ey63cae6dc9ebe1db3@mail.gmail.com>
Message-ID: <bbaeab100607121842h49569499pdd4520437629bb89@mail.gmail.com>

On 7/12/06, Guido van Rossum <guido at python.org> wrote:
>
> On 7/12/06, Armin Rigo <arigo at tunes.org> wrote:
> > I guess I'm going to side with Greg Black on his blog entry.
>
> I seem to recall that that particular one wass *not* an accidental
> bug. I believe I fell over exactly the problem that Greg Black
> complained about (or almost the same; maybe my problem was setting the
> month and day to zero and formatting only the time :-) and tried to
> convince the author to change it; but was told that the new behavior
> was never documented and that the new behavior was somehow better; and
> I didn't fight it further then. Do I remember this correctly? Does
> anybody else recall?


My recollection is that we found a way to cause a crash if improper values
were used.  We never said 0s were allowed in the docs and that it could mask
bugs if you did use them and we supported them (e.g., setting 0 for January
instead of 1 if you were thinking in terms of indexing at the time).  So we
said that values should be within the proper range and not above or below
them.

The python-dev Summary coverage can be found at
http://www.python.org/dev/summary/2004-02-01_2004-02-29/#time-strftime-now-checks-its-argument-s-bounds.

-Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20060712/afe7da0c/attachment.htm 

From guido at python.org  Thu Jul 13 06:46:21 2006
From: guido at python.org (Guido van Rossum)
Date: Wed, 12 Jul 2006 21:46:21 -0700
Subject: [Python-Dev] User's complaints
In-Reply-To: <bbaeab100607121842h49569499pdd4520437629bb89@mail.gmail.com>
References: <ee2a432c0607041649i7ecb6cc1w8ebdf47f7d02097a@mail.gmail.com>
	<dc1166600607111730i44912cedqcea01d918f2b0a85@mail.gmail.com>
	<17588.18475.171696.506911@montanaro.dyndns.org>
	<bbaeab100607111805g5f6ca07cvf0ba9cb09b68bcfa@mail.gmail.com>
	<20060712202908.GA13303@code0.codespeak.net>
	<ca471dc20607121834n4f1bd00ey63cae6dc9ebe1db3@mail.gmail.com>
	<bbaeab100607121842h49569499pdd4520437629bb89@mail.gmail.com>
Message-ID: <ca471dc20607122146l1ee5d426ie216e17647db4a6b@mail.gmail.com>

On 7/12/06, Brett Cannon <brett at python.org> wrote:
> On 7/12/06, Guido van Rossum <guido at python.org> wrote:
>
> > On 7/12/06, Armin Rigo <arigo at tunes.org> wrote:
> > > I guess I'm going to side with Greg Black on his blog entry.
> >
> > I seem to recall that that particular one wass *not* an accidental
> > bug. I believe I fell over exactly the problem that Greg Black
> > complained about (or almost the same; maybe my problem was setting the
> > month and day to zero and formatting only the time :-) and tried to
> > convince the author to change it; but was told that the new behavior
> > was never documented and that the new behavior was somehow better; and
> > I didn't fight it further then. Do I remember this correctly? Does
> > anybody else recall?
>
>
> My recollection is that we found a way to cause a crash if improper values
> were used.  We never said 0s were allowed in the docs and that it could mask
> bugs if you did use them and we supported them ( e.g., setting 0 for January
> instead of 1 if you were thinking in terms of indexing at the time).  So we
> said that values should be within the proper range and not above or below
> them.
>
> The python-dev Summary coverage can be found at
> http://www.python.org/dev/summary/2004-02-01_2004-02-29/#time-strftime-now-checks-its-argument-s-bounds

Thanks for confirming  memory! So it was an intentional regression;
"bugs happen" doesn't apply in this case. And an unfortunate
regression at that -- not because one guy writes a silly blog entry
about it, but because it breaks real code -- undocumented be damned.

Are we going to fix it, without allowing those crashes again?

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From anthony at interlink.com.au  Thu Jul 13 06:56:10 2006
From: anthony at interlink.com.au (Anthony Baxter)
Date: Thu, 13 Jul 2006 14:56:10 +1000
Subject: [Python-Dev] User's complaints
In-Reply-To: <ca471dc20607122146l1ee5d426ie216e17647db4a6b@mail.gmail.com>
References: <ee2a432c0607041649i7ecb6cc1w8ebdf47f7d02097a@mail.gmail.com>
	<bbaeab100607121842h49569499pdd4520437629bb89@mail.gmail.com>
	<ca471dc20607122146l1ee5d426ie216e17647db4a6b@mail.gmail.com>
Message-ID: <200607131456.16309.anthony@interlink.com.au>

On Thursday 13 July 2006 14:46, Guido van Rossum wrote:
> Thanks for confirming  memory! So it was an intentional regression;
> "bugs happen" doesn't apply in this case. And an unfortunate
> regression at that -- not because one guy writes a silly blog entry
> about it, but because it breaks real code -- undocumented be
> damned.
>
> Are we going to fix it, without allowing those crashes again?

I already said "yes" for 2.5. Brett, can you work up a patch?

Anthony
-- 
Anthony Baxter     <anthony at interlink.com.au>
It's never too late to have a happy childhood.

From brett at python.org  Thu Jul 13 07:06:25 2006
From: brett at python.org (Brett Cannon)
Date: Wed, 12 Jul 2006 22:06:25 -0700
Subject: [Python-Dev] User's complaints
In-Reply-To: <200607131456.16309.anthony@interlink.com.au>
References: <ee2a432c0607041649i7ecb6cc1w8ebdf47f7d02097a@mail.gmail.com>
	<bbaeab100607121842h49569499pdd4520437629bb89@mail.gmail.com>
	<ca471dc20607122146l1ee5d426ie216e17647db4a6b@mail.gmail.com>
	<200607131456.16309.anthony@interlink.com.au>
Message-ID: <bbaeab100607122206p60bc13d6hfb0a1df3e6651758@mail.gmail.com>

On 7/12/06, Anthony Baxter <anthony at interlink.com.au> wrote:
>
> On Thursday 13 July 2006 14:46, Guido van Rossum wrote:
> > Thanks for confirming  memory! So it was an intentional regression;
> > "bugs happen" doesn't apply in this case. And an unfortunate
> > regression at that -- not because one guy writes a silly blog entry
> > about it, but because it breaks real code -- undocumented be
> > damned.
> >
> > Are we going to fix it, without allowing those crashes again?
>
> I already said "yes" for 2.5. Brett, can you work up a patch?


Yep, it's on my todo list and will get done.  Might have to wait until next
week, though (if that is okay), since my girlfriend is leaving town Sunday
and thus I will have a ton more time after that.

-Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20060712/3aae1f34/attachment.html 

From ashemedai at gmail.com  Thu Jul 13 07:43:30 2006
From: ashemedai at gmail.com (Jeroen Ruigrok van der Werven)
Date: Thu, 13 Jul 2006 07:43:30 +0200
Subject: [Python-Dev] User's complaints
In-Reply-To: <ee2a432c0607041649i7ecb6cc1w8ebdf47f7d02097a@mail.gmail.com>
References: <ee2a432c0607041649i7ecb6cc1w8ebdf47f7d02097a@mail.gmail.com>
Message-ID: <3e1553560607122243q754dd71dq2e6d95cdf11e89dd@mail.gmail.com>

On 7/5/06, Neal Norwitz <nnorwitz at gmail.com> wrote:
> For example, we heard grumblings about the releases coming too often.
> Once we went to an 18 month release schedule, there was minimal
> complaining.  It should be fairly safe to assume this silence means
> people think we are doing a good job.  What are the things that could
> be fixed that would silence the most number of user's complaints?

I frequent a lot of IRC channels either devoted to Python or heaving
people use Python a lot next to my own use so I think I can,
hopefully, give some comments. Of course, things are highly
subjective.

When some of us first saw what PEP 3000 suggested we were thinking:
shit, there goes Python. It incredibly felt like Python's best
measures were being dumbed down.

The release cycle is not bothering people from what I gather,
especially not given the fact that the world is still changing when it
comes to things like XML. What is bothering people is how you have to
reinstall all your site-packages when you go from one major version to
another. I understand this might be a difficult problem to tackle, but
I can also understand the hassle if people have more than 10-15
modules installed.

Another point is how the module documentation is very terse in a lot
of areas. Especially when it concerns what kind of exceptions a
particular function can/will raise. As a professional, part-time,
technical writer I wonder how to best tackle the entirety of the
documentation we have. (Note that this is in no way any jibe towards
Fred Drake, since I bet it's quite a bulk of documentation to work
with/on every time.)
Also it can get quite confusing when you need the introductory manual,
when the language reference and when the module documentation.

Things that struck me as peculiar is the old:

if __name__ == "__main__":
    whatever()

This is so out of tune with the rest of python it becomes a nuisance.

> PS.  One thing I tend to talk to users about is stability of the
> interpreter.  When I talk about crashing the interpreter, the most
> common first reaction I get is "you can crash the interpreter? How do
> you do that?"  I take that answer as a good sign. :-)

I've come relatively late to the Python scene, but ever since 2.2 I
never once managed to crash the interpreter.

I'll ask around and see what people are reporting to me as their top 3
or 5 Python complaints though.

-- 
Jeroen Ruigrok van der Werven

From nnorwitz at gmail.com  Thu Jul 13 08:24:22 2006
From: nnorwitz at gmail.com (Neal Norwitz)
Date: Wed, 12 Jul 2006 23:24:22 -0700
Subject: [Python-Dev] User's complaints
In-Reply-To: <3e1553560607122243q754dd71dq2e6d95cdf11e89dd@mail.gmail.com>
References: <ee2a432c0607041649i7ecb6cc1w8ebdf47f7d02097a@mail.gmail.com>
	<3e1553560607122243q754dd71dq2e6d95cdf11e89dd@mail.gmail.com>
Message-ID: <ee2a432c0607122324k633ed966n6c55a4921e529099@mail.gmail.com>

On 7/12/06, Jeroen Ruigrok van der Werven <ashemedai at gmail.com> wrote:
> On 7/5/06, Neal Norwitz <nnorwitz at gmail.com> wrote:
> > For example, we heard grumblings about the releases coming too often.
> > Once we went to an 18 month release schedule, there was minimal
> > complaining.  It should be fairly safe to assume this silence means
> > people think we are doing a good job.  What are the things that could
> > be fixed that would silence the most number of user's complaints?
>
> I frequent a lot of IRC channels either devoted to Python or heaving
> people use Python a lot next to my own use so I think I can,
> hopefully, give some comments. Of course, things are highly
> subjective.

Jeroen,

Thank you very much for your feedback.  It helps.

> The release cycle is not bothering people from what I gather,
> especially not given the fact that the world is still changing when it
> comes to things like XML. What is bothering people is how you have to
> reinstall all your site-packages when you go from one major version to
> another. I understand this might be a difficult problem to tackle, but
> I can also understand the hassle if people have more than 10-15
> modules installed.

If it's pure python, why don't people just copy everything under
site-packages after installing?  They could/should run compileall
after that to recompile the .pyc files.  With 2.5 on 64-bit machines,
C extension modules *must* be recompiled due to lots of internal
changes.

One thing you didn't mention that I've heard from time to time is the
stdlib should be improved.  For example, cleaning up old modules.
Though I'm not really sure everyone has the same thing in mind when it
comes to improving the stdlib.

> Another point is how the module documentation is very terse in a lot
> of areas. Especially when it concerns what kind of exceptions a
> particular function can/will raise. As a professional, part-time,
> technical writer I wonder how to best tackle the entirety of the
> documentation we have. (Note that this is in no way any jibe towards
> Fred Drake, since I bet it's quite a bulk of documentation to work
> with/on every time.)
> Also it can get quite confusing when you need the introductory manual,
> when the language reference and when the module documentation.

Do you think you could help with the doc?  How can we get people,
especially tech writers, interested in improving the doc?  Most people
agree it's important, but few make time to really improve the doc.
We've talked about making it easier for people to contribute to the
docs, perhaps adding something like a wiki/comments.  Do you think
that would help?

> Things that struck me as peculiar is the old:
>
> if __name__ == "__main__":
>     whatever()
>
> This is so out of tune with the rest of python it becomes a nuisance.

I'm not sure I understand your point.  Can you provide more info about
what you dislike/expect instead?

> I'll ask around and see what people are reporting to me as their top 3
> or 5 Python complaints though.

Great.  Thanks!

n

From tds333+pydev at gmail.com  Thu Jul 13 09:37:45 2006
From: tds333+pydev at gmail.com (Wolfgang Langner)
Date: Thu, 13 Jul 2006 09:37:45 +0200
Subject: [Python-Dev] User's complaints
In-Reply-To: <3e1553560607122243q754dd71dq2e6d95cdf11e89dd@mail.gmail.com>
References: <ee2a432c0607041649i7ecb6cc1w8ebdf47f7d02097a@mail.gmail.com>
	<3e1553560607122243q754dd71dq2e6d95cdf11e89dd@mail.gmail.com>
Message-ID: <4c45c1530607130037r1c14e3f6u972232bdfa481534@mail.gmail.com>

On 7/13/06, Jeroen Ruigrok van der Werven <ashemedai at gmail.com> wrote:
> Things that struck me as peculiar is the old:
>
> if __name__ == "__main__":
>     whatever()
>
> This is so out of tune with the rest of python it becomes a nuisance.

It is not beautiful but very useful.
In Python 3000 we can replace it with:

@main
def whatever():
    ...

to mark this function as main function if module executed directly.

-- 
bye by Wolfgang

From python-dev at zesty.ca  Thu Jul 13 09:44:41 2006
From: python-dev at zesty.ca (Ka-Ping Yee)
Date: Thu, 13 Jul 2006 02:44:41 -0500 (CDT)
Subject: [Python-Dev] User's complaints
In-Reply-To: <4c45c1530607130037r1c14e3f6u972232bdfa481534@mail.gmail.com>
References: <ee2a432c0607041649i7ecb6cc1w8ebdf47f7d02097a@mail.gmail.com>
	<3e1553560607122243q754dd71dq2e6d95cdf11e89dd@mail.gmail.com>
	<4c45c1530607130037r1c14e3f6u972232bdfa481534@mail.gmail.com>
Message-ID: <Pine.LNX.4.58.0607130240480.31087@server1.LFW.org>

On Thu, 13 Jul 2006, Wolfgang Langner wrote:
> On 7/13/06, Jeroen Ruigrok van der Werven <ashemedai at gmail.com> wrote:
> > Things that struck me as peculiar is the old:
> >
> > if __name__ == "__main__":
> >     whatever()
> >
> > This is so out of tune with the rest of python it becomes a nuisance.
>
> It is not beautiful but very useful.
> In Python 3000 we can replace it with:
>
> @main
> def whatever():
>     ...
>
> to mark this function as main function if module executed directly.

Why not simply:

    def __main__():
        ...

or even pass in the command-line arguments:

    def __main__(*args):
        ...

Having to 'import sys' to get at the command-line arguments always
seemed awkward to me.  'import sys' feels like it should be a
privileged operation (access to interpreter internals), and getting
the command-line args isn't privileged.


-- ?!ng

From ashemedai at gmail.com  Thu Jul 13 09:52:21 2006
From: ashemedai at gmail.com (Jeroen Ruigrok van der Werven)
Date: Thu, 13 Jul 2006 09:52:21 +0200
Subject: [Python-Dev] User's complaints
In-Reply-To: <ee2a432c0607122324k633ed966n6c55a4921e529099@mail.gmail.com>
References: <ee2a432c0607041649i7ecb6cc1w8ebdf47f7d02097a@mail.gmail.com>
	<3e1553560607122243q754dd71dq2e6d95cdf11e89dd@mail.gmail.com>
	<ee2a432c0607122324k633ed966n6c55a4921e529099@mail.gmail.com>
Message-ID: <3e1553560607130052m66899441t50065786495928fe@mail.gmail.com>

On 7/13/06, Neal Norwitz <nnorwitz at gmail.com> wrote:
> On 7/12/06, Jeroen Ruigrok van der Werven <ashemedai at gmail.com> wrote:
> Thank you very much for your feedback.  It helps.

With apologies in advance if my own level of understanding is, of
course, lacking of advanced constructs.

> If it's pure python, why don't people just copy everything under
> site-packages after installing?  They could/should run compileall
> after that to recompile the .pyc files.  With 2.5 on 64-bit machines,
> C extension modules *must* be recompiled due to lots of internal
> changes.

I wasn't even aware of the compileall step, can you elaborate since
this is the first time I see it being mentioned.

> One thing you didn't mention that I've heard from time to time is the
> stdlib should be improved.  For example, cleaning up old modules.
> Though I'm not really sure everyone has the same thing in mind when it
> comes to improving the stdlib.

How do you envision cleaning up old modules?

> Do you think you could help with the doc?  How can we get people,
> especially tech writers, interested in improving the doc?  Most people
> agree it's important, but few make time to really improve the doc.
> We've talked about making it easier for people to contribute to the
> docs, perhaps adding something like a wiki/comments.  Do you think
> that would help?

Wiki/comments are useful, sure, the only problem is that if you have
to sift through every single page to spot a useful comment it is not
helping. I never liked PHP's comment system to take an example.
Personally, but that's just me of course, I prefer a periodic
evaluation of comments and reintegration of said comments into the
documentation itself.

I've been looking at Python's docstrings in the main codebase and that
use alone is inconsistent, some areas feature a lot of docstrings,
some have none. But with alone docstrings you will not cut it I think.

I hope to get around in the coming time to work up some documentation
changes/patches and submit these.


[if __name__ == "__main__":]
> I'm not sure I understand your point.  Can you provide more info about
> what you dislike/expect instead?

Not the above construct at least. :)
To me it just feels like a kludge, perhaps due to the fact that the
rest of Python just flows from your fingers when writing/programming.

Perhaps one could even get away with it in the form of:

def __main__():
    ...main program...

Some other complaints/wishes from a hard core pythonista:

- There's no support for real macros.
- Writable closures would be nice.
- Open classes would be nice.

-- 
Jeroen Ruigrok van der Werven

From bob at redivi.com  Thu Jul 13 09:58:08 2006
From: bob at redivi.com (Bob Ippolito)
Date: Thu, 13 Jul 2006 00:58:08 -0700
Subject: [Python-Dev] User's complaints
In-Reply-To: <4c45c1530607130037r1c14e3f6u972232bdfa481534@mail.gmail.com>
References: <ee2a432c0607041649i7ecb6cc1w8ebdf47f7d02097a@mail.gmail.com>
	<3e1553560607122243q754dd71dq2e6d95cdf11e89dd@mail.gmail.com>
	<4c45c1530607130037r1c14e3f6u972232bdfa481534@mail.gmail.com>
Message-ID: <6713F515-C684-486E-8FFF-EAC60C83E069@redivi.com>


On Jul 13, 2006, at 12:37 AM, Wolfgang Langner wrote:

> On 7/13/06, Jeroen Ruigrok van der Werven <ashemedai at gmail.com> wrote:
>> Things that struck me as peculiar is the old:
>>
>> if __name__ == "__main__":
>>     whatever()
>>
>> This is so out of tune with the rest of python it becomes a nuisance.
>
> It is not beautiful but very useful.
> In Python 3000 we can replace it with:
>
> @main
> def whatever():
>     ...
>
> to mark this function as main function if module executed directly.

It would probably need to be called something else, because main is  
often the name of the main function... but you could write such a  
decorator now if you really wanted to.

def mainfunc(fn):
     if fn.func_globals.get('__name__') == '__main__':
         # ensure the function is in globals
         fn.func_globals[fn.__name__] = fn
         fn()
     return fn

@mainfunc
def main():
     print 'this is in __main__'

-bob


From mike at skew.org  Thu Jul 13 10:26:43 2006
From: mike at skew.org (Mike Brown)
Date: Thu, 13 Jul 2006 02:26:43 -0600 (MDT)
Subject: [Python-Dev] urllib.quote and unicode bug resuscitation attempt
In-Reply-To: <44B4B274.4050602@ofai.at>
Message-ID: <200607130826.k6D8QhHA077212@chilled.skew.org>

Stefan Rank wrote:
> on 12.07.2006 07:53 Martin v. L?wis said the following:
> > Anthony Baxter wrote:
> >>> The right thing to do is IRIs. 
> >> For 2.5, should we at least detect that it's unicode and raise a 
> >> useful error?
> > 
> > That can certainly be done, sure.
> > 
> > Martin
> 
> That would be great.
> 
> And I agree that updating urllib.quote for unicode should be part of a 
> grand plan that updates all of urllib[2] and introduces an irilib / 
> urischemes / uriparse module in 2.6 as Martin and John J Lee suggested.
>   =)
> 
> cheers,
> stefan

Put me down as +1 on raising a useful error instead of a KeyError or whatever,
and +1 on having an irilib, but -1 on working toward accepting unicode in the
URI-oriented urllib.quote(), because (a.) user expectations for strings that
contain non-ASCII-range characters will vary, and (b.) percent-encoding is
supposed to only operate on a byte-encoded view of non-URI information, not
the information itself.

Longer explanation:

I, too, initially thought that quote() was outdated since it choked on unicode
input, but eventually I came to realize that it's wise to reject such input,
because to attempt to percent-encode characters, rather than bytes, reflects a
fundamental misunderstanding of the level at which percent-encoding is
intended to operate.

This is one of the hardest aspects of URI processing to grok, and I'm not
very good at explaining it, even though I've tried my best in the Wikipedia
articles. It's basically these 3 points:

1. A URI can only consist of 'unreserved' characters, as I'm sure you know. 
It's a specific set that has varied slightly over the years, and is a subset 
of printable ASCII.

2. A URI scheme is essentially a mapping of non-URI information to a sequence
of URI characters. That is, it is a method of producing a URI from non-URI
information within a particular information domain ...and vice-versa.

3. A URI scheme should (though may not do so very clearly, especially the
older it is!) tell you that the way to represent a particular bit of non-URI
information, 'info', in a URI is to convert_to_bytes(info), and then, as per
STD 66, make the bytes that correspond, in ASCII, to unreserved characters
manifest as those characters, and all others manifest as their percent-encoded
equivalents. In urllib parlance, this step is 'quoting' the bytes.

3.1. [This isn't crucial to my argument, but has to be mentioned to complete
the explanation of percent-encoding.] In addition, those bytes corresponding,
in ASCII, to some 'reserved' characters are exempt from needing to be
percent-encoded, so long as they're not being used for their reserved purpose
(if any) in whatever URI component they're going into -- Semantically, there's
no difference between such bytes when expressed in the URI as a literal
reserved character or as a percent-encoded byte. URI scheme specs vary greatly
in how they deal with this nuance. In any case, urllib.quote() has the 'safe' 
argument which can be used to specify the exempt reserved characters.



In the days when the specs that urllib was based on were relevant, 99% of the
time, the bytes being 'quoted' were ASCII-encoded strings representing ASCII
character-based non-URI information, so quite a few of us, including many URI
scheme authors, were tempted to think that what was being
'quoted'/percent-encoded *was* the original non-URI information, rather than a
bytewise view of it mandated by a URI scheme.  That's what I was doing when I
thought that quote(some_unicode_path) should 'work', especially in light of
Python's "treat all strings alike" guideline.  But if you accept all of the
above, which is what I believe the standard requires, then unicode input is a
very different situation from str input; it's unclear whether and how the
caller wants the input to be converted to bytes, if they even understand what
they're doing at all.

See, right now, quote('abc 123%') returns 'abc%20123%25', as you would expect. 
Similarly, everyone would probably expect u'abc 123%' to return
u'abc%20123%25', and if we were to implement that, there'd probably be no harm 
done.

But look at quote('\xb7'), which, assuming you accept everything I've said
above is correct, rightfully returns '%B7'.  What would someone expect
quote(u'\xb7') to return?  Some might want u'%B7' because they want the same
result type as the input they gave, with no other changes from how it would
normally be handled. Some might want u'%C2%B7' because they're conflating the
levels of abstraction and expect, say, UTF-8 conversion to be done on their
input.  Some (like me) might want a TypeError or ValueError because we
shouldn't be handing such ambiguous data to quote() in the first place. And 
then there's the u'\u0100'-and-up input to worry about; what does a user
expect to be done with that?

I would prefer to see quote() always reject unicode input with a TypeError. 
Alternatively, if it accepts unicode, it should produce unicode, and since it
can only reasonably assume what the user wants done with ASCII-range
characters, it should only accept input < u'\x80'.

In any case, quote() should be better documented to explain what it accepts 
( a byte sequence )
and why 
( it is intended to be used at the stage of URI production where non-URI 
info, such as a unicode filesystem path, has already been converted to bytes 
according to the requirements of a URI scheme, and now needs to be represented 
as a URI-safe character sequence )
and exactly what it produces
( a str representing URI character s).

Mike


From bingham at cenix-bioscience.com  Thu Jul 13 10:07:44 2006
From: bingham at cenix-bioscience.com (Aaron Bingham)
Date: Thu, 13 Jul 2006 10:07:44 +0200
Subject: [Python-Dev] User's complaints
In-Reply-To: <Pine.LNX.4.58.0607130240480.31087@server1.LFW.org>
References: <ee2a432c0607041649i7ecb6cc1w8ebdf47f7d02097a@mail.gmail.com>	<3e1553560607122243q754dd71dq2e6d95cdf11e89dd@mail.gmail.com>	<4c45c1530607130037r1c14e3f6u972232bdfa481534@mail.gmail.com>
	<Pine.LNX.4.58.0607130240480.31087@server1.LFW.org>
Message-ID: <44B5FF50.3060000@cenix-bioscience.com>

Ka-Ping Yee wrote:

>On Thu, 13 Jul 2006, Wolfgang Langner wrote:
>  
>
>>On 7/13/06, Jeroen Ruigrok van der Werven <ashemedai at gmail.com> wrote:
>>    
>>
>>>Things that struck me as peculiar is the old:
>>>
>>>if __name__ == "__main__":
>>>    whatever()
>>>
>>>This is so out of tune with the rest of python it becomes a nuisance.
>>>      
>>>
>>It is not beautiful but very useful.
>>In Python 3000 we can replace it with:
>>
>>@main
>>def whatever():
>>    ...
>>
>>to mark this function as main function if module executed directly.
>>    
>>
>
>Why not simply:
>
>    def __main__():
>        ...
>
>or even pass in the command-line arguments:
>
>    def __main__(*args):
>        ...
>
>Having to 'import sys' to get at the command-line arguments always
>seemed awkward to me.  'import sys' feels like it should be a
>privileged operation (access to interpreter internals), and getting
>the command-line args isn't privileged.
>  
>
+1, seems a lot more elegant than "if __name__ == '__main__'"

Regards,

-- 
--------------------------------------------------------------------
Aaron Bingham
Senior Software Engineer
Cenix BioScience GmbH
--------------------------------------------------------------------


From greg.ewing at canterbury.ac.nz  Thu Jul 13 10:49:37 2006
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Thu, 13 Jul 2006 20:49:37 +1200
Subject: [Python-Dev] User's complaints
In-Reply-To: <4c45c1530607130037r1c14e3f6u972232bdfa481534@mail.gmail.com>
References: <ee2a432c0607041649i7ecb6cc1w8ebdf47f7d02097a@mail.gmail.com>
	<3e1553560607122243q754dd71dq2e6d95cdf11e89dd@mail.gmail.com>
	<4c45c1530607130037r1c14e3f6u972232bdfa481534@mail.gmail.com>
Message-ID: <44B60921.9010300@canterbury.ac.nz>

Wolfgang Langner wrote:

> @main
> def whatever():
>     ...

This seems like replacing one unpythonic feature
with another. (I *still* can't get used to that
@ syntax -- it looks like an intruder from
Rubyland...)

--
Greg

From bingham at cenix-bioscience.com  Thu Jul 13 10:53:59 2006
From: bingham at cenix-bioscience.com (Aaron Bingham)
Date: Thu, 13 Jul 2006 10:53:59 +0200
Subject: [Python-Dev] User's complaints
In-Reply-To: <44B5FF50.3060000@cenix-bioscience.com>
References: <ee2a432c0607041649i7ecb6cc1w8ebdf47f7d02097a@mail.gmail.com>	<3e1553560607122243q754dd71dq2e6d95cdf11e89dd@mail.gmail.com>	<4c45c1530607130037r1c14e3f6u972232bdfa481534@mail.gmail.com>	<Pine.LNX.4.58.0607130240480.31087@server1.LFW.org>
	<44B5FF50.3060000@cenix-bioscience.com>
Message-ID: <44B60A27.2040303@cenix-bioscience.com>

Aaron Bingham wrote:

>Ka-Ping Yee wrote:
>
>  
>
>>Why not simply:
>>
>>   def __main__():
>>       ...
>>
>>or even pass in the command-line arguments:
>>
>>   def __main__(*args):
>>       ...
>>
>>Having to 'import sys' to get at the command-line arguments always
>>seemed awkward to me.  'import sys' feels like it should be a
>>privileged operation (access to interpreter internals), and getting
>>the command-line args isn't privileged.
>> 
>>
>>    
>>
>+1, seems a lot more elegant than "if __name__ == '__main__'"
>  
>
Just to clarify, I was writing in support of Ping's suggestion of def 
__main__() or def __main__(*args).  The decorator-based solution is ugly.

Regards,

-- 
--------------------------------------------------------------------
Aaron Bingham
Senior Software Engineer
Cenix BioScience GmbH
--------------------------------------------------------------------


From greg.ewing at canterbury.ac.nz  Thu Jul 13 10:59:06 2006
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Thu, 13 Jul 2006 20:59:06 +1200
Subject: [Python-Dev] Handling of sys.args (Re:  User's complaints)
In-Reply-To: <Pine.LNX.4.58.0607130240480.31087@server1.LFW.org>
References: <ee2a432c0607041649i7ecb6cc1w8ebdf47f7d02097a@mail.gmail.com>
	<3e1553560607122243q754dd71dq2e6d95cdf11e89dd@mail.gmail.com>
	<4c45c1530607130037r1c14e3f6u972232bdfa481534@mail.gmail.com>
	<Pine.LNX.4.58.0607130240480.31087@server1.LFW.org>
Message-ID: <44B60B5A.2080309@canterbury.ac.nz>

Ka-Ping Yee wrote:

> Having to 'import sys' to get at the command-line arguments always
> seemed awkward to me.  'import sys' feels like it should be a
> privileged operation (access to interpreter internals), and getting
> the command-line args isn't privileged.

Would it help if sys were pre-imported into the builtins?
Or do you think that args shouldn't live in sys at all?

Recently I've come to appreciate the ability to get at
the args from anywhere, instead of having to catch them
from a call to main() and pass them around. So I'd
like to still be able to import them from somewhere
if I want (doesn't have to be sys, though).

And while we're on the subject, anyone think it would
be a good idea to drop the silly feature of having
the program name as args[0]? You almost *never* want
to treat it the same way as the rest of the args,
so the first thing you always do is args[1:].

It's not so bad in C, where it's just as easy to
start indexing argv from 1 instead of 0. But it
makes no sense in Python, IMO. It would be much
more sensible to move it into a separate attribute
of whatever module we decide to put args in.

--
Greg

From greg.ewing at canterbury.ac.nz  Thu Jul 13 11:02:07 2006
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Thu, 13 Jul 2006 21:02:07 +1200
Subject: [Python-Dev] User's complaints
In-Reply-To: <3e1553560607130052m66899441t50065786495928fe@mail.gmail.com>
References: <ee2a432c0607041649i7ecb6cc1w8ebdf47f7d02097a@mail.gmail.com>
	<3e1553560607122243q754dd71dq2e6d95cdf11e89dd@mail.gmail.com>
	<ee2a432c0607122324k633ed966n6c55a4921e529099@mail.gmail.com>
	<3e1553560607130052m66899441t50065786495928fe@mail.gmail.com>
Message-ID: <44B60C0F.7060405@canterbury.ac.nz>

Jeroen Ruigrok van der Werven wrote:

> - Open classes would be nice.

What do you mean by "open classes"? Python
classes already seem pretty open to me, by
the standards of other languages!

--
Greg


From skip at pobox.com  Thu Jul 13 12:18:29 2006
From: skip at pobox.com (skip at pobox.com)
Date: Thu, 13 Jul 2006 05:18:29 -0500
Subject: [Python-Dev] User's complaints
In-Reply-To: <20060712202908.GA13303@code0.codespeak.net>
References: <ee2a432c0607041649i7ecb6cc1w8ebdf47f7d02097a@mail.gmail.com>
	<dc1166600607111730i44912cedqcea01d918f2b0a85@mail.gmail.com>
	<17588.18475.171696.506911@montanaro.dyndns.org>
	<bbaeab100607111805g5f6ca07cvf0ba9cb09b68bcfa@mail.gmail.com>
	<20060712202908.GA13303@code0.codespeak.net>
Message-ID: <17590.7669.73105.356443@montanaro.dyndns.org>


    Armin> On Tue, Jul 11, 2006 at 06:05:21PM -0700, Brett Cannon wrote:
    >> It is the last point in the first paragraph on time.strftime()
    >> discussing what changed in Python 2.4 as to what the change was.
    >> It's also in Misc/NEWS .  Basically the guy didn't read the release
    >> notes or the docs to see why that changed and that it was legitimate
    >> and needed for stability.

    Armin> Surely everybody should read and think carefully about each
    Armin> (longish) NEWS file for each software package whenever they
    Armin> update their machines or switch to one with newer software than
    Armin> they last used.

I'll add one further point here.  I exchanged a couple emails with Greg
Black.  What happened to him was that some of his customers upgraded their
operating systems and got Python 2.4 in the bargain.  His software started
throwing off errors.  At that point he read the NEWS file and saw the
problem.  He clearly wasn't fully master of the environment in which his
customers ran his software, so I think it's understandable that he was
caught by surprise by this change.

Skip

From fredrik at pythonware.com  Thu Jul 13 12:21:55 2006
From: fredrik at pythonware.com (Fredrik Lundh)
Date: Thu, 13 Jul 2006 12:21:55 +0200
Subject: [Python-Dev] Proposal: Add Sudoku Solver To The "this" Module
Message-ID: <e956rv$kc1$1@sea.gmane.org>

given that java has beaten us with some 60 bytes:

    http://programming.reddit.com/info/9xha/comments/c9y8b

and in order to further improve Python's Kolmogorov rating:

    http://en.wikipedia.org/wiki/Kolmogorov_complexity

how about adding Peter Norvig's constraint-based solver to the Python library:

    http://www.norvig.com/sudoku.html

?

Anthony ?

("import this;this.suduko()" is 25 characters, and we could always optimize this
a bit more in Python 2.6, if needed).

[\F] 




From skip at pobox.com  Thu Jul 13 12:23:44 2006
From: skip at pobox.com (skip at pobox.com)
Date: Thu, 13 Jul 2006 05:23:44 -0500
Subject: [Python-Dev] User's complaints
In-Reply-To: <ee2a432c0607121409h566f7ffbv54dc53ee08a1e026@mail.gmail.com>
References: <ee2a432c0607041649i7ecb6cc1w8ebdf47f7d02097a@mail.gmail.com>
	<dc1166600607111730i44912cedqcea01d918f2b0a85@mail.gmail.com>
	<17588.18475.171696.506911@montanaro.dyndns.org>
	<bbaeab100607111805g5f6ca07cvf0ba9cb09b68bcfa@mail.gmail.com>
	<20060712202908.GA13303@code0.codespeak.net>
	<ee2a432c0607121409h566f7ffbv54dc53ee08a1e026@mail.gmail.com>
Message-ID: <17590.7984.990599.47651@montanaro.dyndns.org>


    Neal> I agree, but some of this responsibility has to fall to users.
    Neal> Sometimes these breakages are bugs, pure and simple.  Our tests
    Neal> don't catch everything.  This is why it's really, really important
    Neal> to get as many alpha/beta testers as possible.  Had the issues
    Neal> been brought up before 2.4 was released, we could have addressed
    Neal> them.

You're of course preaching to the choir here.  Somehow we need to do a
better job convincing the Python-using public to pay attention to alphas and
betas, especially software consultants like Greg Black.

Another area where this particular process broke down was that there simply
wasn't already a test case for the fairly common case of calling strftime
with a six-zeroes pad to format a simple date.  Had it been there already,
it would have been discovered when Brett added the range checks.

Skip


From bob at redivi.com  Thu Jul 13 12:25:16 2006
From: bob at redivi.com (Bob Ippolito)
Date: Thu, 13 Jul 2006 03:25:16 -0700
Subject: [Python-Dev] User's complaints
In-Reply-To: <44B60C0F.7060405@canterbury.ac.nz>
References: <ee2a432c0607041649i7ecb6cc1w8ebdf47f7d02097a@mail.gmail.com>
	<3e1553560607122243q754dd71dq2e6d95cdf11e89dd@mail.gmail.com>
	<ee2a432c0607122324k633ed966n6c55a4921e529099@mail.gmail.com>
	<3e1553560607130052m66899441t50065786495928fe@mail.gmail.com>
	<44B60C0F.7060405@canterbury.ac.nz>
Message-ID: <928173CD-DD5D-4B10-9CC4-ED63CC90933A@redivi.com>


On Jul 13, 2006, at 2:02 AM, Greg Ewing wrote:

> Jeroen Ruigrok van der Werven wrote:
>
>> - Open classes would be nice.
>
> What do you mean by "open classes"? Python
> classes already seem pretty open to me, by
> the standards of other languages!

I'm guessing he's talking about being like Ruby or Objective-C where  
you can add methods to any other class in the runtime. Basically we'd  
have that if the built-in classes were mutable, but that just really  
encourages fragile code. The big problem you run into with open  
classes is that you end up depending on two libraries that have a  
different idea of what the "foo" method on string objects should do.

Adding open classes would make it easier to develop DSLs, but you'd  
only be able to reasonably do one per interpreter (unless you mangled  
the class in a "with" block or something).

-bob


From fredrik at pythonware.com  Thu Jul 13 12:25:22 2006
From: fredrik at pythonware.com (Fredrik Lundh)
Date: Thu, 13 Jul 2006 12:25:22 +0200
Subject: [Python-Dev] User's complaints
References: <ee2a432c0607041649i7ecb6cc1w8ebdf47f7d02097a@mail.gmail.com><dc1166600607111730i44912cedqcea01d918f2b0a85@mail.gmail.com><17588.18475.171696.506911@montanaro.dyndns.org><bbaeab100607111805g5f6ca07cvf0ba9cb09b68bcfa@mail.gmail.com><20060712202908.GA13303@code0.codespeak.net>
	<17590.7669.73105.356443@montanaro.dyndns.org>
Message-ID: <e9572d$l2c$1@sea.gmane.org>

skip at pobox.com wrote:

> He clearly wasn't fully master of the environment in which his
> customers ran his software, so I think it's understandable that he was
> caught by surprise by this change.

a programmer that's surprised that code that relies on undocumented behaviour
might behave differently if run on a different platform ?  *that's* surprising.

</F> 




From sreeram at tachyontech.net  Thu Jul 13 12:39:37 2006
From: sreeram at tachyontech.net (K.S.Sreeram)
Date: Thu, 13 Jul 2006 16:09:37 +0530
Subject: [Python-Dev] Proposal: Add Sudoku Solver To The "this" Module
In-Reply-To: <e956rv$kc1$1@sea.gmane.org>
References: <e956rv$kc1$1@sea.gmane.org>
Message-ID: <44B622E9.4000802@tachyontech.net>

Fredrik Lundh wrote:
> given that java has beaten us with some 60 bytes:
> and in order to further improve Python's Kolmogorov rating:
> how about adding Peter Norvig's constraint-based solver to the Python library:

lol!
(just waiting for somebody to give a serious explanation on why this is
a bad idea!)

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 252 bytes
Desc: OpenPGP digital signature
Url : http://mail.python.org/pipermail/python-dev/attachments/20060713/4e491363/attachment.pgp 

From fredrik at pythonware.com  Thu Jul 13 12:36:07 2006
From: fredrik at pythonware.com (Fredrik Lundh)
Date: Thu, 13 Jul 2006 12:36:07 +0200
Subject: [Python-Dev] User's complaints
References: <ee2a432c0607041649i7ecb6cc1w8ebdf47f7d02097a@mail.gmail.com><3e1553560607122243q754dd71dq2e6d95cdf11e89dd@mail.gmail.com><ee2a432c0607122324k633ed966n6c55a4921e529099@mail.gmail.com><3e1553560607130052m66899441t50065786495928fe@mail.gmail.com><44B60C0F.7060405@canterbury.ac.nz>
	<928173CD-DD5D-4B10-9CC4-ED63CC90933A@redivi.com>
Message-ID: <e957mk$n22$1@sea.gmane.org>

Bob Ippolito wrote:

>> What do you mean by "open classes"? Python
>> classes already seem pretty open to me, by
>> the standards of other languages!
>
> I'm guessing he's talking about being like Ruby or Objective-C where
> you can add methods to any other class in the runtime.

wouldn't a standard multiargument dispatch mechanism solve this, for
most (all?) practical use cases.

(and while we're at it, wouldn't a standard multiargument dispatch be
nice replacement for the instance-oriented lookup we're using today?
dispatching on a single value is so last century ;-)

</F> 




From fredrik at pythonware.com  Thu Jul 13 12:49:40 2006
From: fredrik at pythonware.com (Fredrik Lundh)
Date: Thu, 13 Jul 2006 12:49:40 +0200
Subject: [Python-Dev] Proposal: Add Sudoku Solver To The "this" Module
References: <e956rv$kc1$1@sea.gmane.org> <44B622E9.4000802@tachyontech.net>
Message-ID: <e958fs$pe9$1@sea.gmane.org>

"K.S.Sreeram" wrote:

> (just waiting for somebody to give a serious explanation on why this is
> a bad idea!)

\F might have to post this to comp.lang.python first...

</F> 




From skip at pobox.com  Thu Jul 13 12:53:28 2006
From: skip at pobox.com (skip at pobox.com)
Date: Thu, 13 Jul 2006 05:53:28 -0500
Subject: [Python-Dev] User's complaints
In-Reply-To: <e9572d$l2c$1@sea.gmane.org>
References: <ee2a432c0607041649i7ecb6cc1w8ebdf47f7d02097a@mail.gmail.com>
	<dc1166600607111730i44912cedqcea01d918f2b0a85@mail.gmail.com>
	<17588.18475.171696.506911@montanaro.dyndns.org>
	<bbaeab100607111805g5f6ca07cvf0ba9cb09b68bcfa@mail.gmail.com>
	<20060712202908.GA13303@code0.codespeak.net>
	<17590.7669.73105.356443@montanaro.dyndns.org>
	<e9572d$l2c$1@sea.gmane.org>
Message-ID: <17590.9768.952125.349646@montanaro.dyndns.org>


    >> He clearly wasn't fully master of the environment in which his
    >> customers ran his software, so I think it's understandable that he
    >> was caught by surprise by this change.

    Fredrik> a programmer that's surprised that code that relies on
    Fredrik> undocumented behaviour might behave differently if run on a
    Fredrik> different platform ?  *that's* surprising.

Not necessarily.  He may not have realized his customers were going to
upgrade their operating systems or that the upgrades would involve a new
version of Python.  Not everyone tracks every Python release.  In this case,
a) Python 2.4 had been out long enough to turn up in a new (presumably
Linux) OS release, and b) since Mr. Black appears to program in Python for a
living he should probably have been more attentive to new releases.

Maybe Mr. Black hadn't gotten around to testing his software with 2.4 yet.
Maybe the software was written on-site for a client and he didn't have
access to it to even test in his own environment.

Skip

From ashemedai at gmail.com  Thu Jul 13 14:02:22 2006
From: ashemedai at gmail.com (Jeroen Ruigrok van der Werven)
Date: Thu, 13 Jul 2006 14:02:22 +0200
Subject: [Python-Dev] User's complaints
In-Reply-To: <928173CD-DD5D-4B10-9CC4-ED63CC90933A@redivi.com>
References: <ee2a432c0607041649i7ecb6cc1w8ebdf47f7d02097a@mail.gmail.com>
	<3e1553560607122243q754dd71dq2e6d95cdf11e89dd@mail.gmail.com>
	<ee2a432c0607122324k633ed966n6c55a4921e529099@mail.gmail.com>
	<3e1553560607130052m66899441t50065786495928fe@mail.gmail.com>
	<44B60C0F.7060405@canterbury.ac.nz>
	<928173CD-DD5D-4B10-9CC4-ED63CC90933A@redivi.com>
Message-ID: <3e1553560607130502o253a8359mf327964ee5d8ece8@mail.gmail.com>

Hi Bob,

On 7/13/06, Bob Ippolito <bob at redivi.com> wrote:
> Adding open classes would make it easier to develop DSLs, but you'd
> only be able to reasonably do one per interpreter (unless you mangled
> the class in a "with" block or something).

The person whose 'complaints' I was stating says that DSLs (Domain
Specific Languages for those who, like me, were confused about the
acronym) are a big part of what he is after and one per interpreter is
fine by him. He also realises that the application(s) he needs them
for might be unusual. He doesn't specifically need the builtin types
to be extendable. It's just nice to be able to define a single class
in multiple modules. Even C++ allows this to some extent (but not as
much as he'd like).

He understands the implications of allowing open classes (import vs.
no import changes semantics, etc.). Personally, he doesn't care *too*
much about newbie safety since he's not a newbie. To quote verbatim:
"give me the big guns :-)"

And while we're at it, he also stated: "[...] add multiple dispatch to
your list of improvements for Python".

I hope this clarifies it a bit for other people.

-- 
Jeroen Ruigrok van der Werven

From ncoghlan at gmail.com  Thu Jul 13 14:57:30 2006
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 13 Jul 2006 22:57:30 +1000
Subject: [Python-Dev] User's complaints
In-Reply-To: <3e1553560607130502o253a8359mf327964ee5d8ece8@mail.gmail.com>
References: <ee2a432c0607041649i7ecb6cc1w8ebdf47f7d02097a@mail.gmail.com>	<3e1553560607122243q754dd71dq2e6d95cdf11e89dd@mail.gmail.com>	<ee2a432c0607122324k633ed966n6c55a4921e529099@mail.gmail.com>	<3e1553560607130052m66899441t50065786495928fe@mail.gmail.com>	<44B60C0F.7060405@canterbury.ac.nz>	<928173CD-DD5D-4B10-9CC4-ED63CC90933A@redivi.com>
	<3e1553560607130502o253a8359mf327964ee5d8ece8@mail.gmail.com>
Message-ID: <44B6433A.1080705@gmail.com>

Jeroen Ruigrok van der Werven wrote:
> Hi Bob,
> 
> On 7/13/06, Bob Ippolito <bob at redivi.com> wrote:
>> Adding open classes would make it easier to develop DSLs, but you'd
>> only be able to reasonably do one per interpreter (unless you mangled
>> the class in a "with" block or something).
> 
> The person whose 'complaints' I was stating says that DSLs (Domain
> Specific Languages for those who, like me, were confused about the
> acronym) are a big part of what he is after and one per interpreter is
> fine by him. He also realises that the application(s) he needs them
> for might be unusual. He doesn't specifically need the builtin types
> to be extendable. It's just nice to be able to define a single class
> in multiple modules. Even C++ allows this to some extent (but not as
> much as he'd like).

I'm somewhat confused as to how Python's classes aren't open. Sure, types like 
the builtin types that don't have a __dict__ aren't open because there isn't 
anywhere to put the extensions, but metaclassing lets you do whatever you want 
to any other class:


def extends(orig_cls):
     if not hasattr(orig_cls, "__dict__"):
         raise TypeError("Cannot extend %r" % cls)
     class ExtendMeta(type):
         def __new__(mcl, name, bases, ns):
             if len(bases) != 1:
                 raise TypeError("Can only extend single class")
             if bases[0] is object:
                 return type.__new__(mcl, name, bases, ns)
             for key, value in ns.iteritems():
                 if key not in ("__metaclass__", "__dict__"):
                     setattr(orig_cls, key, value)
             return orig_cls
     class ExtendCls(object):
         __metaclass__ = ExtendMeta
     return ExtendCls

 >>> class A1(object):
...     def method1(self):
...         print "Hi, I'm method 1!"
...
 >>> class A2(extends(A1)):
...     def method2(self):
...         print "Hi, I'm method 2!"
...
 >>> x = A1()
 >>> x.method1()
Hi, I'm method 1!
 >>> x.method2()
Hi, I'm method 2!
 >>> y = A2()
 >>> y.method1()
Hi, I'm method 1!
 >>> y.method2()
Hi, I'm method 2!
 >>> A1 is A2
True

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
             http://www.boredomandlaziness.org

From fredrik at pythonware.com  Thu Jul 13 15:05:49 2006
From: fredrik at pythonware.com (Fredrik Lundh)
Date: Thu, 13 Jul 2006 15:05:49 +0200
Subject: [Python-Dev] User's complaints
References: <ee2a432c0607041649i7ecb6cc1w8ebdf47f7d02097a@mail.gmail.com>	<3e1553560607122243q754dd71dq2e6d95cdf11e89dd@mail.gmail.com>	<ee2a432c0607122324k633ed966n6c55a4921e529099@mail.gmail.com>	<3e1553560607130052m66899441t50065786495928fe@mail.gmail.com>	<44B60C0F.7060405@canterbury.ac.nz>	<928173CD-DD5D-4B10-9CC4-ED63CC90933A@redivi.com><3e1553560607130502o253a8359mf327964ee5d8ece8@mail.gmail.com>
	<44B6433A.1080705@gmail.com>
Message-ID: <e95gfd$len$1@sea.gmane.org>

Nick Coghlan wrote:

>> The person whose 'complaints' I was stating says that DSLs (Domain
>> Specific Languages for those who, like me, were confused about the
>> acronym) are a big part of what he is after and one per interpreter is
>> fine by him. He also realises that the application(s) he needs them
>> for might be unusual. He doesn't specifically need the builtin types
>> to be extendable. It's just nice to be able to define a single class
>> in multiple modules. Even C++ allows this to some extent (but not as
>> much as he'd like).
>
> I'm somewhat confused as to how Python's classes aren't open. Sure, types like
> the builtin types that don't have a __dict__ aren't open because there isn't
> anywhere to put the extensions, but metaclassing lets you do whatever you want
> to any other class:

you don't even need metaclass trickery to deal with the "define a single
class in multiple modules" problem; just use multiple inheritance to bring
all the component classes together.

</F> 




From ncoghlan at gmail.com  Thu Jul 13 15:15:46 2006
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 13 Jul 2006 23:15:46 +1000
Subject: [Python-Dev] Support for PyGetSetDefs in pydoc
In-Reply-To: <EF6E6AF2-654F-4F53-91A8-A6C9B8A80745@python.org>
References: <A027CA35-EE77-4658-BA59-E849368C440B@python.org>
	<EF6E6AF2-654F-4F53-91A8-A6C9B8A80745@python.org>
Message-ID: <44B64782.9010706@gmail.com>

Barry Warsaw wrote:
> For example, I could change inspect locally so that it gets the type  
> of datetime.timedelta.days without adding a constant to types.py.  Or  
> I could patch pydoc.py directly and leave even inspect.py out of it.   
> Or I could create some stupid internal type in some stupid internal  
> module who's only purpose would be to have a handle on member  
> descriptors.  Or I could change datetime to be built-in.  (see what I  
> mean about levels of ickyness? :).
> 
> I'm up for suggestions.  I think this would be worthwhile to address  
> in Python 2.5 since I think it would be good to have an accurate  
> representation of Python's built-in types in types.py.  Ultimately, I  
> really care about teaching pydoc.help() about instances of these  
> types so that users can get better help when they encounter them  
> (such as might be the case in 3rd party extension modules).
> 
> Suggestions are welcome.

Could you include a "look up late-breaking types" function in types.py that 
site.py calls after it finishes setting up the standard library path?

Still a little hackish, I know, but it seems less icky than your suggestions 
above, as it means that fixing anything similar that comes up in the future 
will only require modification of types.py.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
             http://www.boredomandlaziness.org

From ncoghlan at gmail.com  Thu Jul 13 15:21:43 2006
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 13 Jul 2006 23:21:43 +1000
Subject: [Python-Dev] User's complaints
In-Reply-To: <e95gfd$len$1@sea.gmane.org>
References: <ee2a432c0607041649i7ecb6cc1w8ebdf47f7d02097a@mail.gmail.com>	<3e1553560607122243q754dd71dq2e6d95cdf11e89dd@mail.gmail.com>	<ee2a432c0607122324k633ed966n6c55a4921e529099@mail.gmail.com>	<3e1553560607130052m66899441t50065786495928fe@mail.gmail.com>	<44B60C0F.7060405@canterbury.ac.nz>	<928173CD-DD5D-4B10-9CC4-ED63CC90933A@redivi.com><3e1553560607130502o253a8359mf327964ee5d8ece8@mail.gmail.com>	<44B6433A.1080705@gmail.com>
	<e95gfd$len$1@sea.gmane.org>
Message-ID: <44B648E7.3050403@gmail.com>

Fredrik Lundh wrote:
> Nick Coghlan wrote:
> 
>>> The person whose 'complaints' I was stating says that DSLs (Domain
>>> Specific Languages for those who, like me, were confused about the
>>> acronym) are a big part of what he is after and one per interpreter is
>>> fine by him. He also realises that the application(s) he needs them
>>> for might be unusual. He doesn't specifically need the builtin types
>>> to be extendable. It's just nice to be able to define a single class
>>> in multiple modules. Even C++ allows this to some extent (but not as
>>> much as he'd like).
>> I'm somewhat confused as to how Python's classes aren't open. Sure, types like
>> the builtin types that don't have a __dict__ aren't open because there isn't
>> anywhere to put the extensions, but metaclassing lets you do whatever you want
>> to any other class:
> 
> you don't even need metaclass trickery to deal with the "define a single
> class in multiple modules" problem; just use multiple inheritance to bring
> all the component classes together.

I didn't mean to say that I thought what I posted was a sensible thing to do - 
I was mainly curious as to how close I could get to Ruby's class extension 
syntax by using metaclasses (with the answer being "very", IMO).

If someone really wants to extend a Python class, they can just define a 
function that does what they want and assign it to whatever method names on 
whatever mutable classes they feel like. No fancy syntax needed :)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
             http://www.boredomandlaziness.org

From guido at python.org  Thu Jul 13 16:20:33 2006
From: guido at python.org (Guido van Rossum)
Date: Thu, 13 Jul 2006 07:20:33 -0700
Subject: [Python-Dev] User's complaints
In-Reply-To: <3e1553560607122243q754dd71dq2e6d95cdf11e89dd@mail.gmail.com>
References: <ee2a432c0607041649i7ecb6cc1w8ebdf47f7d02097a@mail.gmail.com>
	<3e1553560607122243q754dd71dq2e6d95cdf11e89dd@mail.gmail.com>
Message-ID: <ca471dc20607130720q3fe1b363y533a7bff46579174@mail.gmail.com>

Somebody whose name doesn't matter (it's not about him) wrote:
> When some of us first saw what PEP 3000 suggested we were thinking:
> shit, there goes Python. [...]

And later in the same message the same person wrote:

> Things that struck me as peculiar is the old:
>
> if __name__ == "__main__":
>     whatever()
>
> This is so out of tune with the rest of python it becomes a nuisance.

This is an illustration of the dilemma of maintaining a popular
language: Everybody hates change (me too!) but everybody also has one
thing that's bothering them so much they absolutely want it to be
changed. If you were to implement all those personal pet peeves, you'd
get a language that's more different from Python than Python is from
Fortran.

So where's the middle ground? I believe it's established without a
doubt that in biological evolution, changes comes in spurts: A species
may change hardly at all for millions of years, and then suddenly, due
to not quite understood mechanisms, it starts to change rapidly until
a completely new species (or several) has evolved, which again remains
stable for a long time.

I don't want to adopt this time scale for Python (:-), but I do think
it's useful to think of language evolution as a kind of fractal
movement -- at every time scale, there are small jumps and the
occasional large jump. Python 2.2 was a fairly large jump (new-style
classes, iterators, generators, decorators). Python 3000 will be the
largest jump so far. There will be larger jumps yet in the distant
future. But in between, there will be long periods of (relative)
stability.

Will it hurt? You bet! But for many species, sooner or later it's
evolve or become extinct.

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From stefan.rank at ofai.at  Thu Jul 13 16:29:15 2006
From: stefan.rank at ofai.at (Stefan Rank)
Date: Thu, 13 Jul 2006 16:29:15 +0200
Subject: [Python-Dev] urllib.quote and unicode bug resuscitation attempt
In-Reply-To: <200607130826.k6D8QhHA077212@chilled.skew.org>
References: <44B4B274.4050602@ofai.at>
	<200607130826.k6D8QhHA077212@chilled.skew.org>
Message-ID: <44B658BB.5090308@ofai.at>

on 13.07.2006 10:26 Mike Brown said the following:
> Stefan Rank wrote:
>> on 12.07.2006 07:53 Martin v. L?wis said the following:
>>> Anthony Baxter wrote:
>>>>> The right thing to do is IRIs. 
>>>> For 2.5, should we at least detect that it's unicode and raise a 
>>>> useful error?
>>> That can certainly be done, sure.

<snip>

> Put me down as +1 on raising a useful error instead of a KeyError or whatever,
> and +1 on having an irilib, but -1 on working toward accepting unicode in the
> URI-oriented urllib.quote(), because

<snip convincing explanation>

> See, right now, quote('abc 123%') returns 'abc%20123%25', as you would expect. 
> Similarly, everyone would probably expect u'abc 123%' to return
> u'abc%20123%25', and if we were to implement that, there'd probably be no harm 
> done.

Well, originally, I would have expected it to return a byte str(ing),
BUT
I am now converted and think it is best to raise a TypeError for 
unicode, and leave the encoding decisions to higher level code.

So I'll repeat the "patch" #1, slightly modified::

  if isinstance(s, unicode):
      raise TypeError("quote expects an encoded byte string as argument")

Is it safe to assume that code that breaks because of this change was 
already broken?

stefan


From barry at python.org  Thu Jul 13 18:12:28 2006
From: barry at python.org (Barry Warsaw)
Date: Thu, 13 Jul 2006 12:12:28 -0400
Subject: [Python-Dev] Support for PyGetSetDefs in pydoc
In-Reply-To: <44B64782.9010706@gmail.com>
References: <A027CA35-EE77-4658-BA59-E849368C440B@python.org>
	<EF6E6AF2-654F-4F53-91A8-A6C9B8A80745@python.org>
	<44B64782.9010706@gmail.com>
Message-ID: <7FE62713-24B0-47CE-BA94-41150DB683F4@python.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Jul 13, 2006, at 9:15 AM, Nick Coghlan wrote:

> Could you include a "look up late-breaking types" function in  
> types.py that site.py calls after it finishes setting up the  
> standard library path?
>
> Still a little hackish, I know, but it seems less icky than your  
> suggestions above, as it means that fixing anything similar that  
> comes up in the future will only require modification of types.py.

I've come to the conclusion that it's much better to add a "helper"  
module that defines a few extension types solely for the purposes of  
exposing them in types.py, than it is to rely on accidents of  
implementation discovered in the existing interpreter.  To that end,  
I've implemented a _typesmodule.c that exposes a getset descriptor  
and a member descriptor on a type that can't be instantiated.  That  
module gets built into Python and provides a central place to add any  
future C extension types.  It also provides a handy place for Jython  
or IronPython to hang their own equivalents, if and when they ever  
get them.

Then in types.py, we import _types and pull the appropriate types  
from there.  types.py will not define GetSetType or  
MemberDescriptorType if that module can't be imported so that it's  
robust for other Python implementations.  I like not defining them  
better than setting them to None in that case.

Then inspect.py defines isgetset() and ismemberdescriptor() in terms  
of those types, but it always returns False if those types don't  
exist.  So inspect.py too is robust for other Python implementations.

I've updated SF patch #1520294 and assigned it back to Georg for  
another quick review.  Assuming he likes the updated patch and  
Anthony approves, please assign it back to me for final checkin.

In thinking about this more, re: back porting to Python 2.4, I don't  
think we can expose GetSetType, MemberDescriptorType, isgetset(), or  
ismemberdescriptor() in the appropriate modules.  That smells too  
much like the True/False thing.  However, I believe I can weedle a  
fix into pydoc for those types without exposing these new names, so  
I'd like to do that if there are no objections.

- -Barry

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.3 (Darwin)

iQCVAwUBRLZw8XEjvBPtnXfVAQLkIAP+O0WXh+xO902KNah/kIYy/GSPsO/CfRUO
+YchtF2HdbyEXORwhsLXJVTKVROYgohCQrZp7LUX5vNUah0J0ycIPXhzpqrB0Gmc
8AaRChkFO8EGofzEGOvtunGUGdl9733u2mNUUAhGk15YFs8GDyiuzkL8SitcJQ3q
OHnCsdNRuSU=
=pbA8
-----END PGP SIGNATURE-----

From jeremy at alum.mit.edu  Thu Jul 13 18:14:50 2006
From: jeremy at alum.mit.edu (Jeremy Hylton)
Date: Thu, 13 Jul 2006 12:14:50 -0400
Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
In-Reply-To: <e93s6t$5fu$1@sea.gmane.org>
References: <e8k58c$q7u$1@sea.gmane.org>
	<r01050400-1039-AEFC2D140DA011DBB8DF001124365170@10.0.0.24>
	<e8ldjg$eq7$1@sea.gmane.org> <e90j6o$fij$1@sea.gmane.org>
	<e93s6t$5fu$1@sea.gmane.org>
Message-ID: <e8bf7a530607130914w6c2abe95r78aaa7d79b7cf9d8@mail.gmail.com>

On 7/12/06, Fredrik Lundh <fredrik at pythonware.com> wrote:
> Boris Borcic wrote:
>
> >> note that most examples of this type already work, if the target type is
> >> mutable, and implement the right operations:
> >>
> >>       def counter(num):
> >>           num = mutable_int(num)
> >>           def inc():
> >>               num += 1
> >>               return num
> >>           return inc
> >
> > I agree with you (and argued it in "scopes vs augmented assignment vs sets"
> > recently) that mutating would be sufficient /if/ the compiler would view
> > augmented assignment as mutations operators
>
> feel free to replace that += with an .add(1) method call; the point
> wasn't the behaviour of augmented assigment, the point was that that the
> most common use pattern involves *mutation* of the target object.
>
> the syntax isn't that important, really.

Mutation is different from rebinding.  A tuple is immutable, but you
can rebind the variable that refers to the tuple.  I think we will
confuse users if we use the term mutation to refer to name binding.
Name binding is already a subtle issue, so I think the risk is
significant.

Jeremy

From bob at redivi.com  Thu Jul 13 18:49:31 2006
From: bob at redivi.com (Bob Ippolito)
Date: Thu, 13 Jul 2006 09:49:31 -0700
Subject: [Python-Dev] User's complaints
In-Reply-To: <3e1553560607130502o253a8359mf327964ee5d8ece8@mail.gmail.com>
References: <ee2a432c0607041649i7ecb6cc1w8ebdf47f7d02097a@mail.gmail.com>
	<3e1553560607122243q754dd71dq2e6d95cdf11e89dd@mail.gmail.com>
	<ee2a432c0607122324k633ed966n6c55a4921e529099@mail.gmail.com>
	<3e1553560607130052m66899441t50065786495928fe@mail.gmail.com>
	<44B60C0F.7060405@canterbury.ac.nz>
	<928173CD-DD5D-4B10-9CC4-ED63CC90933A@redivi.com>
	<3e1553560607130502o253a8359mf327964ee5d8ece8@mail.gmail.com>
Message-ID: <89F00269-C67E-474A-9AF0-282EE343FC18@redivi.com>


On Jul 13, 2006, at 5:02 AM, Jeroen Ruigrok van der Werven wrote:

> Hi Bob,
>
> On 7/13/06, Bob Ippolito <bob at redivi.com> wrote:
>> Adding open classes would make it easier to develop DSLs, but you'd
>> only be able to reasonably do one per interpreter (unless you mangled
>> the class in a "with" block or something).
>
> The person whose 'complaints' I was stating says that DSLs (Domain
> Specific Languages for those who, like me, were confused about the
> acronym) are a big part of what he is after and one per interpreter is
> fine by him. He also realises that the application(s) he needs them
> for might be unusual. He doesn't specifically need the builtin types
> to be extendable. It's just nice to be able to define a single class
> in multiple modules. Even C++ allows this to some extent (but not as
> much as he'd like).
>
> He understands the implications of allowing open classes (import vs.
> no import changes semantics, etc.). Personally, he doesn't care *too*
> much about newbie safety since he's not a newbie. To quote verbatim:
> "give me the big guns :-)"
>
> And while we're at it, he also stated: "[...] add multiple dispatch to
> your list of improvements for Python".
>
> I hope this clarifies it a bit for other people.

Well, if this person really weren't a newbie then of course they'd  
know how to define a metaclass that can be used to extend a (non- 
built-in) class from another module. They'd probably also know of two  
or three different implementations of multiple dispatch (or  
equivalent, such as generic functions) available, and could probably  
write their own if they had to ;)

The only valid complaint, really, is that built-in classes are read- 
only. I doubt anyone wants to change that. If they want to write  
things in the style of Ruby, why not just use it?

-bob


From theller at python.net  Thu Jul 13 18:52:33 2006
From: theller at python.net (Thomas Heller)
Date: Thu, 13 Jul 2006 18:52:33 +0200
Subject: [Python-Dev] The buffer() function
Message-ID: <e95toj$8uu$1@sea.gmane.org>

I just answered a question on comp.lang.python for someone
who was asking about how to convert the internal buffer of
a ctypes instance into base64 coding, without too much copying:

  "The conversion calls in the base64 module expect strings as input, so
  right now I'm converting the binary blocks to strings first, then
  converting the resulting string to base64.  This seems highly
  inefficient and I'd like to just go straight from binary to a base64
  string."

Naturally I tried to call base64.encodestring(buffer(ctypes_instance))
and it worked, so that was my answer.

As it turns out the guy (unusal!) tried to look up information
about the buffer function in the documentation.  What he found
is this:

  "Python programmers, trainers, students and bookwriters should feel
  free to bypass these functions without concerns about missing something
  important".

Not an encouragement to use this function.  So, what is the real state
of the buffer() function, and the buffer object?

IIUC, the buffer object was broken some time ago, but I think it has
been fixed.  Can the 'status' of the buffer function be changed?
To quote the next question from the OP:

  "Is buffer safe to use?  Is there an alternative?"

My thinking is that it *is* safe to use, and that there is
no alternative (but imo also no alternative is needed).

Thanks,
Thomas


From brett at python.org  Thu Jul 13 18:54:22 2006
From: brett at python.org (Brett Cannon)
Date: Thu, 13 Jul 2006 09:54:22 -0700
Subject: [Python-Dev] Handling of sys.args (Re: User's complaints)
In-Reply-To: <44B60B5A.2080309@canterbury.ac.nz>
References: <ee2a432c0607041649i7ecb6cc1w8ebdf47f7d02097a@mail.gmail.com>
	<3e1553560607122243q754dd71dq2e6d95cdf11e89dd@mail.gmail.com>
	<4c45c1530607130037r1c14e3f6u972232bdfa481534@mail.gmail.com>
	<Pine.LNX.4.58.0607130240480.31087@server1.LFW.org>
	<44B60B5A.2080309@canterbury.ac.nz>
Message-ID: <bbaeab100607130954g429f9fdbhb6b1159e5e97f2b0@mail.gmail.com>

On 7/13/06, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
>
> Ka-Ping Yee wrote:
>
> > Having to 'import sys' to get at the command-line arguments always
> > seemed awkward to me.  'import sys' feels like it should be a
> > privileged operation (access to interpreter internals), and getting
> > the command-line args isn't privileged.
>
> Would it help if sys were pre-imported into the builtins?
> Or do you think that args shouldn't live in sys at all?
>
> Recently I've come to appreciate the ability to get at
> the args from anywhere, instead of having to catch them
> from a call to main() and pass them around. So I'd
> like to still be able to import them from somewhere
> if I want (doesn't have to be sys, though).
>
> And while we're on the subject, anyone think it would
> be a good idea to drop the silly feature of having
> the program name as args[0]? You almost *never* want
> to treat it the same way as the rest of the args,
> so the first thing you always do is args[1:].
>
> It's not so bad in C, where it's just as easy to
> start indexing argv from 1 instead of 0. But it
> makes no sense in Python, IMO. It would be much
> more sensible to move it into a separate attribute
> of whatever module we decide to put args in.


Makes sense to me.  Ruby does this and it makes working with arguments a
little bit nicer.

-Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20060713/41f25f26/attachment.html 

From fredrik at pythonware.com  Thu Jul 13 18:57:46 2006
From: fredrik at pythonware.com (Fredrik Lundh)
Date: Thu, 13 Jul 2006 18:57:46 +0200
Subject: [Python-Dev] The buffer() function
In-Reply-To: <e95toj$8uu$1@sea.gmane.org>
References: <e95toj$8uu$1@sea.gmane.org>
Message-ID: <e95u28$a1p$1@sea.gmane.org>

Thomas Heller wrote:

> Naturally I tried to call base64.encodestring(buffer(ctypes_instance))
> and it worked, so that was my answer.

does ctypes_instance implement the buffer API ?  if it does, is the 
buffer() call even necessary ?

</F>


From theller at python.net  Thu Jul 13 19:03:22 2006
From: theller at python.net (Thomas Heller)
Date: Thu, 13 Jul 2006 19:03:22 +0200
Subject: [Python-Dev] The buffer() function
In-Reply-To: <e95u28$a1p$1@sea.gmane.org>
References: <e95toj$8uu$1@sea.gmane.org> <e95u28$a1p$1@sea.gmane.org>
Message-ID: <e95ucs$b6q$1@sea.gmane.org>

Fredrik Lundh schrieb:
> Thomas Heller wrote:
> 
>> Naturally I tried to call base64.encodestring(buffer(ctypes_instance))
>> and it worked, so that was my answer.
> 
> does ctypes_instance implement the buffer API ?  if it does, is the 
> buffer() call even necessary ?

Yes, in both cases.

Thomas


From fredrik at pythonware.com  Thu Jul 13 19:20:53 2006
From: fredrik at pythonware.com (Fredrik Lundh)
Date: Thu, 13 Jul 2006 19:20:53 +0200
Subject: [Python-Dev] The buffer() function
In-Reply-To: <e95ucs$b6q$1@sea.gmane.org>
References: <e95toj$8uu$1@sea.gmane.org> <e95u28$a1p$1@sea.gmane.org>
	<e95ucs$b6q$1@sea.gmane.org>
Message-ID: <e95vdi$fcu$1@sea.gmane.org>

Thomas Heller wrote:

>>> Naturally I tried to call base64.encodestring(buffer(ctypes_instance))
>>> and it worked, so that was my answer.
 >>
>> does ctypes_instance implement the buffer API ?  if it does, is the 
>> buffer() call even necessary ?
> 
> Yes, in both cases.

are you sure?  does it implement the bf_getreadbuffer slot?  afaik, 
afaicr, and from what I can tell by looking briefly at the sources, the 
binascii.b2a_base64 function uses the the s# marker, and that argument 
type accepts strings *and* read only buffers, right away.  no extra crud 
should be necessary.

</F>


From theller at python.net  Thu Jul 13 19:38:28 2006
From: theller at python.net (Thomas Heller)
Date: Thu, 13 Jul 2006 19:38:28 +0200
Subject: [Python-Dev] The buffer() function
In-Reply-To: <e95vdi$fcu$1@sea.gmane.org>
References: <e95toj$8uu$1@sea.gmane.org>
	<e95u28$a1p$1@sea.gmane.org>	<e95ucs$b6q$1@sea.gmane.org>
	<e95vdi$fcu$1@sea.gmane.org>
Message-ID: <e960em$jad$1@sea.gmane.org>

Fredrik Lundh schrieb:
> Thomas Heller wrote:
> 
>>>> Naturally I tried to call base64.encodestring(buffer(ctypes_instance))
>>>> and it worked, so that was my answer.
>  >>
>>> does ctypes_instance implement the buffer API ?  if it does, is the 
>>> buffer() call even necessary ?
>> 
>> Yes, in both cases.
> 
> are you sure?  does it implement the bf_getreadbuffer slot?  afaik, 
> afaicr, and from what I can tell by looking briefly at the sources, the 
> binascii.b2a_base64 function uses the the s# marker, and that argument 
> type accepts strings *and* read only buffers, right away.  no extra crud 
> should be necessary.

Yes, for binascii.b2a_base64 you are right: bf_getreadbuffer is sufficient.
For binascii.b2a_hex (for example), it is not sufficient.

But that was not the question.  What about the status of the buffer function?

All the above functions accept a buffer object.

Thomas


From glyph at divmod.com  Thu Jul 13 20:03:22 2006
From: glyph at divmod.com (glyph at divmod.com)
Date: Thu, 13 Jul 2006 14:03:22 -0400
Subject: [Python-Dev] Community buildbots (was Re: User's complaints)
In-Reply-To: 0
Message-ID: <20060713180322.29014.1145514133.divmod.quotient.35181@ohm>

On Wed, 12 Jul 2006 10:30:17 +1000, Michael Ellerman <michael at ellerman.id.au> wrote:

>Well here's one I stumbled across the other day. I don't know if it's
>legit, but it's still bad PR:
>
>http://www.gbch.net/gjb/blog/software/discuss/python-sucks.html

Having been exhorted (or maybe I mean "excoriated") by your friendly release 
manager earlier this week to post my comments and criticisms about Python here
rather than vent in random IRC chatter, I feel morally compelled to comment.

I see some responses to that post which indicate that the specific bug will be
fixed, and that's good, but there is definitely a pattern he's talking about
here, not just one issue.  I think there is a general pattern of small,
difficult to detect breakages in Python.  Twisted (and the various other
Divmod projects that I maintain) are thoroughly unit-tested, but there are
still significant issues in each Python release that require changes.

Unlike the jerk who posted that "python sucks" rant, I'm not leaving Python
because one function changed in a major release; I do expect to have to
maintain my projects myself, and major releases are "major" for a reason.  I
only wish that upgrading all my other dependencies were as easy as upgrading
Python!  I do see that the developers are working with some caution to avoid
breaking code, and attempting to consider the cost of each change.

However, although I've seen lots of discussions of what "average" code might
break when exposed to new versions of Python, these discussions tend to be
entirely hypothetical.  Do the core Python developers typically run the test
suites of major Python applications when considering major changes, such as 
Zope, Plone, TurboGears, and of course Twisted?  Not that breakages would be
considered unacceptable -- some gain is surely worth the cost -- but to
establish some empirical level of burden on the community?

I would like to propose, although I certainly don't have time to implement,
a program by which Python-using projects could contribute buildslaves which
would run their projects' tests with the latest Python trunk.  This would
provide two useful incentives: Python code would gain a reputation as
generally well-tested (since there is a direct incentive to write tests for
your project: get notified when core python changes might break it), and the
core developers would have instant feedback when a "small" change breaks more
code than it was expected to.

I can see that certain Python developers expect that some of this work is the
responsibility of the user community, and to some extent that's true, but at
least half of the work needs to be done _before_ the changes are made.  If
some Python change breaks half of Twisted, I would like to know about it in
time to complain about the implementation, rather than flailing around once
the Python feature-freeze has set in and hoping that it's nothing too serious.
For example, did anyone here know that the new-style exceptions stuff in 2.5
caused hundreds of unit-test failures in Twisted?  I am glad the change was
made, and one of our users did catch it, so the process isn't fatally broken,
but it is still worrying.

Another problem is simply that the Python developers don't have the same
very public voice that other language developers do.  It doesn't necessarily
have to be a blog, but python-dev is fast-paced and intimidating, and a
vehicle for discussion among those in the know, rather than dissimenation to
the community at large.  It's a constant source of anxiety to me that I might
miss some key feature addition to Python which breaks or distorts some key bit
of Twisted functionality (as the new-style exceptions, or recent ImportWarning
almost did) because I don't have enough time to follow all the threads here.
I really appreciate the work that Steve Bethard et. al. are doing on the
python-dev summaries, but they're still pretty dry and low level.

While the new python.org is very nice, I do note that there's no "blogs" entry
on the front page, something which has become a fixture on almost every other 
website I visit regularly.  The news page is not very personal, mainly a 
listing of releases and events.  There's no forum for getting the community 
_excited_ about new features (especially new features which are explicitly 
enabled by potential breakages), and selling them on the cool uses.  Who 
knows, maybe I'll even start using decorators syntax all over the place if I 
see them presented somewhere by someone who is very excited about the feature 
and thinks it's worthwhile, rather than as a defense on a mailing list 
against a criticism, or a simple announcement of the feature's existence.

I've seen the other side of this problem as well, so I can appreciate that it
is quite difficult to get this kind of thing right: lots of applications using
Twisted break when we change broken or deprecated APIs.  Twisted is lucky
though; it has numerous subprojects, and I maintain a half-dozen unrelated
projects in a different repository.  Twisted itself is a fairly
representative "application using Twisted", so when something breaks "average"
Twisted code, I know about it right away.  Python is less an application using
Python: the standard library is rather passive (there is no "end-to-end" 
application path to test, only individual functions and classes), and the test
coverage of included Python modules is rather spotty.

I hope someone finds some of these suggestions helpful.  I only wish I had the
time and energy to act on some of them.

From mwh at python.net  Thu Jul 13 20:19:08 2006
From: mwh at python.net (Michael Hudson)
Date: Thu, 13 Jul 2006 19:19:08 +0100
Subject: [Python-Dev] Community buildbots
In-Reply-To: <20060713180322.29014.1145514133.divmod.quotient.35181@ohm>
	(glyph@divmod.com's message of "Thu, 13 Jul 2006 14:03:22 -0400")
References: <20060713180322.29014.1145514133.divmod.quotient.35181@ohm>
Message-ID: <2mfyh52wrn.fsf@starship.python.net>

No real time to respond in detail here, but one small comment.

glyph at divmod.com writes:

> I see some responses to that post which indicate that the specific bug will be
> fixed, and that's good, but there is definitely a pattern he's talking about
> here, not just one issue.  I think there is a general pattern of small,
> difficult to detect breakages in Python.

Remember these words...

> For example, did anyone here know that the new-style exceptions stuff in 2.5
> caused hundreds of unit-test failures in Twisted?  I am glad the change was
> made, and one of our users did catch it, so the process isn't fatally broken,
> but it is still worrying.

When implementing this stuff, I could have merely made it possible for
exceptions to be new-style, and left the builtin exceptions as classic
classes.  This didn't seem to be an especially good idea, as it would
leave code working _most_ of the time, only to break horribly when
confronted with a rare new-style exception.  So I made the decision
(and I don't think I ever got around to explicitly discussing this
with anyone, nor if the people who actually updated my patch and
checked it in thought about it at all) to make the built in exceptions
new-style, precisely to make it a screamingly obvious change.  I
didn't _know_ when I was doing it that I'd break Twisted unit tests,
but I was hardly a surprise.

I think the idea of a buildbot running various projects tests suites
with svn HEAD is a great idea -- back when I was doing 2.2.1 I did run
a few third party test suites (zodb comes to mind) but as always with
these things, automation is a good idea.

Cheers,
mwh

-- 
  > Why are we talking about bricks and concrete in a lisp newsgroup?
  After long experiment it was found preferable to talking about why
  Lisp is slower than C++...
                        -- Duane Rettig & Tim Bradshaw, comp.lang.lisp

From jcarlson at uci.edu  Thu Jul 13 20:30:20 2006
From: jcarlson at uci.edu (Josiah Carlson)
Date: Thu, 13 Jul 2006 11:30:20 -0700
Subject: [Python-Dev] The buffer() function
In-Reply-To: <e960em$jad$1@sea.gmane.org>
References: <e95vdi$fcu$1@sea.gmane.org> <e960em$jad$1@sea.gmane.org>
Message-ID: <20060713112730.0181.JCARLSON@uci.edu>


Thomas Heller <theller at python.net> wrote:
> But that was not the question.  What about the status of the buffer function?

From what I understand, it is relatively safe as long as you don't
mutate an object while there is a buffer attached to it.

That is:

    import array
    a = array.array(...)
    b = buffer(a)
    for i in xrange(...):
        a.extend(a[:])
    print str(b)

... may cause you some problems (the a[:] bit was to pointer movement
movement on realloc).  Those problems will depend on your platform.

 - Josiah


From aahz at pythoncraft.com  Thu Jul 13 20:29:16 2006
From: aahz at pythoncraft.com (Aahz)
Date: Thu, 13 Jul 2006 11:29:16 -0700
Subject: [Python-Dev] Community buildbots (was Re: User's complaints)
In-Reply-To: <20060713180322.29014.1145514133.divmod.quotient.35181@ohm>
References: <20060713180322.29014.1145514133.divmod.quotient.35181@ohm>
Message-ID: <20060713182916.GA12832@panix.com>

On Thu, Jul 13, 2006, glyph at divmod.com wrote:
>
> I would like to propose, although I certainly don't have time to
> implement, a program by which Python-using projects could contribute
> buildslaves which would run their projects' tests with the latest
> Python trunk.  This would provide two useful incentives: Python code
> would gain a reputation as generally well-tested (since there is a
> direct incentive to write tests for your project: get notified when
> core python changes might break it), and the core developers would
> have instant feedback when a "small" change breaks more code than it
> was expected to.

There's been some recent discussion in the PSF wondering where it would
make sense to throw some money to remove grit in the wheels; do you think
this is a case where that would help?
-- 
Aahz (aahz at pythoncraft.com)           <*>         http://www.pythoncraft.com/

"I saw `cout' being shifted "Hello world" times to the left and stopped
right there."  --Steve Gonedes

From amk at amk.ca  Thu Jul 13 20:42:55 2006
From: amk at amk.ca (A.M. Kuchling)
Date: Thu, 13 Jul 2006 14:42:55 -0400
Subject: [Python-Dev] Community buildbots (was Re: User's complaints)
In-Reply-To: <20060713180322.29014.1145514133.divmod.quotient.35181@ohm>
References: <20060713180322.29014.1145514133.divmod.quotient.35181@ohm>
Message-ID: <20060713184255.GA3929@rogue.amk.ca>

On Thu, Jul 13, 2006 at 02:03:22PM -0400, glyph at divmod.com wrote:
> I would like to propose, although I certainly don't have time to implement,
> a program by which Python-using projects could contribute buildslaves which
> would run their projects' tests with the latest Python trunk.

An excellent idea!

> It's a constant source of anxiety to me
> that I might miss some key feature addition to Python which breaks
> or distorts some key bit of Twisted functionality (as the new-style
> exceptions, or recent ImportWarning almost did) because I don't have
> enough time to follow all the threads here.

So don't read threads -- just try the alphas and betas!  The "What's
New" documents have a 'porting' section that flags changes that may
require application changes, but items are added only if I think of
them or if someone suggests them (e.g. generator.gi_frame can be None
in Python 2.5 -- I would never have thought people would have code
that broke because of this).

> While the new python.org is very nice, I do note that there's no "blogs" 
> entry
> on the front page, something which has become a fixture on almost every 
> other website I visit regularly.  

A 'Blogs' link could be trivially added by linking to
planet.python.org, though the blogs collected there are not in any way
'the Python developers', but a jumble of developers and users.  I
don't think enough core developers have weblogs (or write about
Python) to make a 'python-dev only' planet very useful.

> mainly a listing of releases and events.  There's no forum for getting the 
> community _excited_ about new features (especially new features which are 
> explicitly enabled by potential breakages), and selling them on the cool 
> uses.  

I think the venue for this would be weblog entries or news sites such
as oreillynet.com.  I do worry that we don't have enough people
writing articles and things, but have no idea how to encourage people
to write more.

--amk

From foom at fuhm.net  Thu Jul 13 20:36:59 2006
From: foom at fuhm.net (James Y Knight)
Date: Thu, 13 Jul 2006 14:36:59 -0400
Subject: [Python-Dev] The buffer() function
In-Reply-To: <e95toj$8uu$1@sea.gmane.org>
References: <e95toj$8uu$1@sea.gmane.org>
Message-ID: <1AE30CC8-3CD9-421D-98AD-C8DF4145A6F0@fuhm.net>


On Jul 13, 2006, at 12:52 PM, Thomas Heller wrote:

> IIUC, the buffer object was broken some time ago, but I think it has
> been fixed.  Can the 'status' of the buffer function be changed?
> To quote the next question from the OP:
>
>   "Is buffer safe to use?  Is there an alternative?"
>
> My thinking is that it *is* safe to use, and that there is
> no alternative (but imo also no alternative is needed).

I believe it's safe, except when used on an array.array object.  
However, that's not buffer's fault, but rather a bug in the array class.

The buffer interface requires that, as long as a reference to a  
python object is alive, pointers into its buffer will not become  
invalidated. Array breaks that guarantee.

To fix this, array ought to make a sub-object that this guarantee  
_does_ hold for. And when it needs more storage, simply make a new  
sub-object with more storage. Then, the buffer's reference would be  
to the refcounted sub-object, and thus the associated memory wouldn't  
go away until the buffer was done with it.

James

From python-dev at zesty.ca  Thu Jul 13 20:41:10 2006
From: python-dev at zesty.ca (Ka-Ping Yee)
Date: Thu, 13 Jul 2006 13:41:10 -0500 (CDT)
Subject: [Python-Dev] Handling of sys.args (Re:  User's complaints)
In-Reply-To: <44B60B5A.2080309@canterbury.ac.nz>
References: <ee2a432c0607041649i7ecb6cc1w8ebdf47f7d02097a@mail.gmail.com>
	<3e1553560607122243q754dd71dq2e6d95cdf11e89dd@mail.gmail.com>
	<4c45c1530607130037r1c14e3f6u972232bdfa481534@mail.gmail.com>
	<Pine.LNX.4.58.0607130240480.31087@server1.LFW.org>
	<44B60B5A.2080309@canterbury.ac.nz>
Message-ID: <Pine.LNX.4.58.0607131333290.31087@server1.LFW.org>

On Thu, 13 Jul 2006, Greg Ewing wrote:
> Would it help if sys were pre-imported into the builtins?
> Or do you think that args shouldn't live in sys at all?

I feel like the command-line arguments don't really belong in sys,
and i'd rather not have 'sys' pre-imported into the builtins.

I think of 'sys' as the place for sensitive interpreter internals
and the builtins as the place for harmless functions.  They're at
opposite ends of the scale -- 'sys' is the most privileged;
builtins are the least privileged.  From a security point of view,
'import' is the main way to get dangerous new functionality.
I realize 'open' breaks this concept, but it's nice to be able to
say, to a rough approximation, that preventing 'import' gives you
an interpreter that can do computations but isn't a security risk.

These are just my feelings -- i'm not sure how consistent this is
with the original intent of 'sys'.  I think it would be a quite
useful distinction to have, though.

> And while we're on the subject, anyone think it would
> be a good idea to drop the silly feature of having
> the program name as args[0]? You almost *never* want
> to treat it the same way as the rest of the args,
> so the first thing you always do is args[1:].

Starting the args array with the first argument (and putting
the program name somewhere else) seems Pythonic to me.


-- ?!ng

From theller at python.net  Thu Jul 13 20:51:42 2006
From: theller at python.net (Thomas Heller)
Date: Thu, 13 Jul 2006 20:51:42 +0200
Subject: [Python-Dev] The buffer() function
In-Reply-To: <20060713112730.0181.JCARLSON@uci.edu>
References: <e95vdi$fcu$1@sea.gmane.org> <e960em$jad$1@sea.gmane.org>
	<20060713112730.0181.JCARLSON@uci.edu>
Message-ID: <e964o0$2kc$1@sea.gmane.org>

Josiah Carlson schrieb:
> Thomas Heller <theller at python.net> wrote:
>> But that was not the question.  What about the status of the buffer function?
> 
>>From what I understand, it is relatively safe as long as you don't
> mutate an object while there is a buffer attached to it.
> 
> That is:
> 
>     import array
>     a = array.array(...)
>     b = buffer(a)
>     for i in xrange(...):
>         a.extend(a[:])
>     print str(b)
> 
> ... may cause you some problems (the a[:] bit was to pointer movement
> movement on realloc).  Those problems will depend on your platform.

AFAIK, the buffer object now does not hold a pointer into the object
it has been constructed from, it only gets it when its needed.

IMO Objects/bufferobject.c, revision 35400 is considered safe.

The checkin comment (by nascheme) was, more than 2 years ago:
"Make buffer objects based on mutable objects (like array) safe."

Thomas


From guido at python.org  Thu Jul 13 21:25:50 2006
From: guido at python.org (Guido van Rossum)
Date: Thu, 13 Jul 2006 12:25:50 -0700
Subject: [Python-Dev] Handling of sys.args (Re: User's complaints)
In-Reply-To: <Pine.LNX.4.58.0607131333290.31087@server1.LFW.org>
References: <ee2a432c0607041649i7ecb6cc1w8ebdf47f7d02097a@mail.gmail.com>
	<3e1553560607122243q754dd71dq2e6d95cdf11e89dd@mail.gmail.com>
	<4c45c1530607130037r1c14e3f6u972232bdfa481534@mail.gmail.com>
	<Pine.LNX.4.58.0607130240480.31087@server1.LFW.org>
	<44B60B5A.2080309@canterbury.ac.nz>
	<Pine.LNX.4.58.0607131333290.31087@server1.LFW.org>
Message-ID: <ca471dc20607131225x63220367q83e402abf7420250@mail.gmail.com>

On 7/13/06, Ka-Ping Yee <python-dev at zesty.ca> wrote:
> On Thu, 13 Jul 2006, Greg Ewing wrote:
> > Would it help if sys were pre-imported into the builtins?
> > Or do you think that args shouldn't live in sys at all?
>
> I feel like the command-line arguments don't really belong in sys,
> and i'd rather not have 'sys' pre-imported into the builtins.
>
> I think of 'sys' as the place for sensitive interpreter internals
> and the builtins as the place for harmless functions.  They're at
> opposite ends of the scale -- 'sys' is the most privileged;
> builtins are the least privileged.  From a security point of view,
> 'import' is the main way to get dangerous new functionality.
> I realize 'open' breaks this concept, but it's nice to be able to
> say, to a rough approximation, that preventing 'import' gives you
> an interpreter that can do computations but isn't a security risk.
>
> These are just my feelings -- i'm not sure how consistent this is
> with the original intent of 'sys'.  I think it would be a quite
> useful distinction to have, though.

sys is a grab-bag. Note that it also has a good deal of harmless
read-only data: maxint, version, platform and many more. But it seems
to specialize in things that you may modify to change global behavior:
path, modules, stdout, ps1, displayhook and others. I'nm afraid if we
were to split it by functionality we'd have to split it 5-way or so...

> > And while we're on the subject, anyone think it would
> > be a good idea to drop the silly feature of having
> > the program name as args[0]? You almost *never* want
> > to treat it the same way as the rest of the args,
> > so the first thing you always do is args[1:].
>
> Starting the args array with the first argument (and putting
> the program name somewhere else) seems Pythonic to me.

As long as it's called args, not argv, I agree.

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From jcarlson at uci.edu  Thu Jul 13 21:33:23 2006
From: jcarlson at uci.edu (Josiah Carlson)
Date: Thu, 13 Jul 2006 12:33:23 -0700
Subject: [Python-Dev] The buffer() function
In-Reply-To: <e964o0$2kc$1@sea.gmane.org>
References: <20060713112730.0181.JCARLSON@uci.edu> <e964o0$2kc$1@sea.gmane.org>
Message-ID: <20060713123144.0184.JCARLSON@uci.edu>


Thomas Heller <theller at python.net> wrote:
> AFAIK, the buffer object now does not hold a pointer into the object
> it has been constructed from, it only gets it when its needed.
> 
> IMO Objects/bufferobject.c, revision 35400 is considered safe.
> 
> The checkin comment (by nascheme) was, more than 2 years ago:
> "Make buffer objects based on mutable objects (like array) safe."

It's good to hear that bug has been squished.  I believe buffer() is
generally safe then, according to my somewhat limited experience in its
use and knowledge about its implementation.

 - Josiah


From nmm1 at cus.cam.ac.uk  Thu Jul 13 21:55:37 2006
From: nmm1 at cus.cam.ac.uk (Nick Maclaren)
Date: Thu, 13 Jul 2006 20:55:37 +0100
Subject: [Python-Dev] Handling of sys.args (Re: User's complaints)
Message-ID: <E1G17Hd-0005nY-Iu@libra.cus.cam.ac.uk>

On systems that are not Unix-derived (which, nowadays, are rare),
there is commonly no such thing as a program name in the first place.
It is possible to get into that state on some Unices - i.e. ones which
have a form of exec that takes a file descriptor, inode number or
whatever.

This is another argument for separating off argv[0] and allowing the
program name to be None if there IS no program name.


Regards,
Nick Maclaren,
University of Cambridge Computing Service,
New Museums Site, Pembroke Street, Cambridge CB2 3QH, England.
Email:  nmm1 at cam.ac.uk
Tel.:  +44 1223 334761    Fax:  +44 1223 334679

From barry at python.org  Thu Jul 13 22:05:47 2006
From: barry at python.org (Barry Warsaw)
Date: Thu, 13 Jul 2006 16:05:47 -0400
Subject: [Python-Dev] Community buildbots (was Re: User's complaints)
In-Reply-To: <20060713180322.29014.1145514133.divmod.quotient.35181@ohm>
References: <20060713180322.29014.1145514133.divmod.quotient.35181@ohm>
Message-ID: <65ECC6D7-7E46-463C-92D9-37AF2BF15234@python.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Jul 13, 2006, at 2:03 PM, glyph at divmod.com wrote:

> Having been exhorted (or maybe I mean "excoriated") by your  
> friendly release
> manager earlier this week to post my comments and criticisms about  
> Python here
> rather than vent in random IRC chatter, I feel morally compelled to  
> comment.

And I'm glad you did because you made some excellent comments.

> However, although I've seen lots of discussions of what "average"  
> code might
> break when exposed to new versions of Python, these discussions  
> tend to be
> entirely hypothetical.  Do the core Python developers typically run  
> the test
> suites of major Python applications when considering major changes,  
> such as
> Zope, Plone, TurboGears, and of course Twisted?  Not that breakages  
> would be
> considered unacceptable -- some gain is surely worth the cost --  
> but to
> establish some empirical level of burden on the community?

I do plan on very soon running Mailman's meager test suite and  
running more manual functional tests on Python 2.5b2.  Compared to  
Twisted, Mailman is pretty simple and doesn't do that much fancy  
stuff so I suspect that it'll mostly work, but you never know until  
you try it.  I'd support adding Mailman to a community buildbot army,  
although I don't personally have the cycles to build out such a beast.

I also plan on testing my Real Job's embedded application against  
Python 2.5b2 sometime this week or next.  We're on Python 2.4.2 atm  
and that is much more complicated because of all the C API we use.   
I'm less sanguine about breakage there, but I'll post here if I find  
anything egregious (fortunately, we have an extensive test suite to  
rely upon).

> Twisted itself is a fairly
> representative "application using Twisted", so when something  
> breaks "average"
> Twisted code, I know about it right away.  Python is less an  
> application using
> Python: the standard library is rather passive (there is no "end-to- 
> end"
> application path to test, only individual functions and classes),  
> and the test
> coverage of included Python modules is rather spotty.

This really is an excellent point and makes me think that we may want  
to consider elaborating on the Python release cycle to include a  
gamma phase or a longer release candidate cycle.  OT1H I think there  
will always be people or projects that won't try anything until the  
gold release, and that if we've broken anything in their code we  
simply won't know it until after that, no matter how diligent we  
are.  OTOH, a more formal gamma phase would allow us to say  
"absolutely no changes are allowed now unless it's to fix backward  
compatibility".  No more sneaking in new sys functions or types  
module constants <wink> during the gamma phase.

- -Barry
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.3 (Darwin)

iQCVAwUBRLanoHEjvBPtnXfVAQJB6gP/SwVtejenN0/7tszePbJ4O20l98k2Z/7N
rs9dfF2+Jcy/OLzSCcW/OW4iVf+iPJMWUNqHEPFDQO/+nVifh8pjjWGTQJMc8ynm
7HNCk/ZciZyNqeGbmGvzHoywmrldbDPkx5Y6Fo13yel0slw2Qo6gC6W8aygi7giP
boJiovnaKH0=
=wRxy
-----END PGP SIGNATURE-----

From pierrebai at hotmail.com  Thu Jul 13 22:10:11 2006
From: pierrebai at hotmail.com (Pierre Baillargeon)
Date: Thu, 13 Jul 2006 20:10:11 +0000 (UTC)
Subject: [Python-Dev] Partial support for dlmodule.c in 64-bits OSes
Message-ID: <loom.20060713T215347-860@post.gmane.org>

Currently, many 64-bits Oses cannot uses the dlmodule due to the conflicts
between the sizes of int, long and char *. That is well. The check is made as
run-time, which is also very well.

The problem is that the Python configuration script (setup.py) also makes the
check and plainly excludes dlmodule.c from being built and deployed. That is not
so well.

The reason is that we use the dlmodule solely to get access to the various flags
(RTLD_NOW, RTLD_GLOBAL, etc), so that we can do some magic with loaded shared
libraries, such as over-ridding the import mechanism so that the default load
flags get changed (via sys.setdlopenflags()) to force some semantics.

Currently this doesn't work on most 64-bits OSes because the dl module doesn't
exists, so it cannot be imported and its RTLD_* symbols are not accessible.

So I ask if it would be possible that the test for sys.maxint == 0x7fffffff in
setup.py be dropped in future releases.

Here's a relevant diff from the current setup.py in SVN:

1025,1030c1025,1029
+         # This requires sizeof(int) == sizeof(long) == sizeof(char*)
+         # but it will be tested at run-time: give access to the dl module so
+         # that RTDL_* symbols can be accessed.
+         dl_inc = find_file('dlfcn.h', [], inc_dirs)
+         if (dl_inc is not None) and (platform not in ['atheos']):
+             exts.append( Extension('dl', ['dlmodule.c']) )
---
-         if sys.maxint == 0x7fffffff:
-             # This requires sizeof(int) == sizeof(long) == sizeof(char*)
-             dl_inc = find_file('dlfcn.h', [], inc_dirs)
-             if (dl_inc is not None) and (platform not in ['atheos']):
-                 exts.append( Extension('dl', ['dlmodule.c']) )



From glyph at divmod.com  Thu Jul 13 22:41:46 2006
From: glyph at divmod.com (glyph at divmod.com)
Date: Thu, 13 Jul 2006 16:41:46 -0400
Subject: [Python-Dev] Community buildbots
In-Reply-To: <2mfyh52wrn.fsf@starship.python.net>
Message-ID: <20060713204146.29014.283506497.divmod.quotient.35352@ohm>

On Thu, 13 Jul 2006 19:19:08 +0100, Michael Hudson <mwh at python.net> wrote:
>glyph at divmod.com writes:

>> For example, did anyone here know that the new-style exceptions stuff in 2.5
>> caused hundreds of unit-test failures in Twisted?  I am glad the change was
>> made, and one of our users did catch it, so the process isn't fatally broken,
>> but it is still worrying.
>
>When implementing this stuff, I could have (... snip ...)

To be clear, I agree with the decision you made in this particular case.  I
just would have appreciated the opportunity to participate in the
discussion before the betas were out and the featureset frozen.  (Of course I
*can* always do that, and some other Twisted devs watch python-dev a bit more
closely than I do, but the point is that the amount of effort required to do
this is prohibitive for the average Python hacker, whereas the time to set up an
individual buildbot might not be.)

(Aside: IMHO, the sooner we can drop old-style classes entirely, the better.
That is one bumpy Python upgrade process that I will be _very_ happy to do.
There's no way to have documentation that expresses the requirement that an 
implementation of an interface be new-style or old-style without reference to 
numerous historical accidents, which are bewildering and upsetting to people
reading documentation for the first time.)

From rasky at develer.com  Thu Jul 13 22:49:57 2006
From: rasky at develer.com (Giovanni Bajo)
Date: Thu, 13 Jul 2006 22:49:57 +0200
Subject: [Python-Dev] Community buildbots (was Re: User's complaints)
References: <20060713180322.29014.1145514133.divmod.quotient.35181@ohm>
	<65ECC6D7-7E46-463C-92D9-37AF2BF15234@python.org>
Message-ID: <058b01c6a6bd$e6b6b080$d503030a@trilan>

Barry Warsaw wrote:

> OTOH, a more formal gamma phase would allow us to say
> "absolutely no changes are allowed now unless it's to fix backward
> compatibility".  No more sneaking in new sys functions or types
> module constants <wink> during the gamma phase.

This is pretty common in project management. For instance, GCC has a rather
complex 4-stage release process, whose last phase (beginning at the point
the release is branched in SVN) is made of commits only for regressions.
-- 
Giovanni Bajo


From brett at python.org  Thu Jul 13 22:52:50 2006
From: brett at python.org (Brett Cannon)
Date: Thu, 13 Jul 2006 13:52:50 -0700
Subject: [Python-Dev] Community buildbots
In-Reply-To: <20060713204146.29014.283506497.divmod.quotient.35352@ohm>
References: <2mfyh52wrn.fsf@starship.python.net>
	<20060713204146.29014.283506497.divmod.quotient.35352@ohm>
Message-ID: <bbaeab100607131352p24a9e2cepfa000571871aebcc@mail.gmail.com>

On 7/13/06, glyph at divmod.com <glyph at divmod.com> wrote:
>
> On Thu, 13 Jul 2006 19:19:08 +0100, Michael Hudson <mwh at python.net> wrote:
> >glyph at divmod.com writes:
>
> >> For example, did anyone here know that the new-style exceptions stuff
> in 2.5
> >> caused hundreds of unit-test failures in Twisted?  I am glad the change
> was
> >> made, and one of our users did catch it, so the process isn't fatally
> broken,
> >> but it is still worrying.
> >
> >When implementing this stuff, I could have (... snip ...)
>
> To be clear, I agree with the decision you made in this particular
> case.  I
> just would have appreciated the opportunity to participate in the
> discussion before the betas were out and the featureset frozen.  (Of
> course I
> *can* always do that, and some other Twisted devs watch python-dev a bit
> more
> closely than I do, but the point is that the amount of effort required to
> do
> this is prohibitive for the average Python hacker, whereas the time to set
> up an
> individual buildbot might not be.)


One way to try to track python-dev is the python-dev Summaries.  While
Steven is only human and thus  cannot always get them out immediately, they
do happen frequently enough that major decisions will be covered before
betas are hit.

-Brett

(Aside: IMHO, the sooner we can drop old-style classes entirely, the better.
> That is one bumpy Python upgrade process that I will be _very_ happy to
> do.
> There's no way to have documentation that expresses the requirement that
> an
> implementation of an interface be new-style or old-style without reference
> to
> numerous historical accidents, which are bewildering and upsetting to
> people
> reading documentation for the first time.)
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/brett%40python.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20060713/9de6c971/attachment.html 

From rasky at develer.com  Thu Jul 13 22:53:02 2006
From: rasky at develer.com (Giovanni Bajo)
Date: Thu, 13 Jul 2006 22:53:02 +0200
Subject: [Python-Dev] Community buildbots
References: <2mfyh52wrn.fsf@starship.python.net>
	<20060713204146.29014.283506497.divmod.quotient.35352@ohm>
Message-ID: <05af01c6a6be$54b653b0$d503030a@trilan>

glyph at divmod.com wrote:

> (Aside: IMHO, the sooner we can drop old-style classes entirely, the
> better.
> That is one bumpy Python upgrade process that I will be _very_ happy
> to do.

I think python should have a couple more of future imports. "from __future__
import new_classes" and "from __future__ import unicode_literals" would be
really welcome, and would smooth the Py3k migration process
-- 
Giovanni Bajo


From radix at twistedmatrix.com  Thu Jul 13 23:06:04 2006
From: radix at twistedmatrix.com (Christopher Armstrong)
Date: Thu, 13 Jul 2006 17:06:04 -0400
Subject: [Python-Dev] Community buildbots
In-Reply-To: <05af01c6a6be$54b653b0$d503030a@trilan>
References: <2mfyh52wrn.fsf@starship.python.net>
	<20060713204146.29014.283506497.divmod.quotient.35352@ohm>
	<05af01c6a6be$54b653b0$d503030a@trilan>
Message-ID: <60ed19d40607131406u498b0901oc84296f4e631ae5e@mail.gmail.com>

On 7/13/06, Giovanni Bajo <rasky at develer.com> wrote:
> I think python should have a couple more of future imports. "from __future__
> import new_classes" and "from __future__ import unicode_literals" would be
> really welcome, and would smooth the Py3k migration process

Along similar lines as glyph, after complaining about this for a long
time "offline" with my friends, I decided it's probably a good idea to
get involved and vote that the __future__ import for unicode_literals
be implemented. python -U is a failure for obvious reasons and a
__future__ import is clearly better.

Does anyone want to pair on this?

-- 
Christopher Armstrong
International Man of Twistery
http://radix.twistedmatrix.com/
http://twistedmatrix.com/
http://canonical.com/

From bob at redivi.com  Thu Jul 13 23:07:07 2006
From: bob at redivi.com (Bob Ippolito)
Date: Thu, 13 Jul 2006 14:07:07 -0700
Subject: [Python-Dev] Community buildbots
In-Reply-To: <05af01c6a6be$54b653b0$d503030a@trilan>
References: <2mfyh52wrn.fsf@starship.python.net>
	<20060713204146.29014.283506497.divmod.quotient.35352@ohm>
	<05af01c6a6be$54b653b0$d503030a@trilan>
Message-ID: <B02BC5ED-9165-4C01-B79F-4BEDE9EC5C59@redivi.com>


On Jul 13, 2006, at 1:53 PM, Giovanni Bajo wrote:

> glyph at divmod.com wrote:
>
>> (Aside: IMHO, the sooner we can drop old-style classes entirely, the
>> better.
>> That is one bumpy Python upgrade process that I will be _very_ happy
>> to do.
>
> I think python should have a couple more of future imports. "from  
> __future__
> import new_classes" and "from __future__ import unicode_literals"  
> would be
> really welcome, and would smooth the Py3k migration process

"from __future__ import new_classes" exists, but the syntax is  
different:

__metaclass__ = type

-bob


From glyph at divmod.com  Fri Jul 14 00:30:24 2006
From: glyph at divmod.com (glyph at divmod.com)
Date: Thu, 13 Jul 2006 18:30:24 -0400
Subject: [Python-Dev] Community buildbots (was Re: User's complaints)
In-Reply-To: <20060713182916.GA12832@panix.com>
Message-ID: <20060713223024.29014.1648933869.divmod.quotient.35468@ohm>



On Thu, 13 Jul 2006 11:29:16 -0700, Aahz <aahz at pythoncraft.com> wrote:

>There's been some recent discussion in the PSF wondering where it would
>make sense to throw some money to remove grit in the wheels; do you think
>this is a case where that would help?

Most likely yes.  It's not a huge undertaking, and there are a lot of people out
there in the community with the knowledge of Buildbot to make this happen.

From radix at twistedmatrix.com  Fri Jul 14 00:32:45 2006
From: radix at twistedmatrix.com (Christopher Armstrong)
Date: Thu, 13 Jul 2006 18:32:45 -0400
Subject: [Python-Dev] Community buildbots (was Re: User's complaints)
In-Reply-To: <20060713223024.29014.1648933869.divmod.quotient.35468@ohm>
References: <20060713182916.GA12832@panix.com>
	<20060713223024.29014.1648933869.divmod.quotient.35468@ohm>
Message-ID: <60ed19d40607131532g3c0a5cc1n3177a22d90999dab@mail.gmail.com>

On 7/13/06, glyph at divmod.com <glyph at divmod.com> wrote:
> On Thu, 13 Jul 2006 11:29:16 -0700, Aahz <aahz at pythoncraft.com> wrote:
>
> >There's been some recent discussion in the PSF wondering where it would
> >make sense to throw some money to remove grit in the wheels; do you think
> >this is a case where that would help?
>
> Most likely yes.  It's not a huge undertaking, and there are a lot of people out
> there in the community with the knowledge of Buildbot to make this happen.

I'm at least willing to set up the buildbots for projects I care about
(Twisted, pydoctor, whatever), and perhaps help out with the setting
up the buildmaster.

-- 
Christopher Armstrong
International Man of Twistery
http://radix.twistedmatrix.com/
http://twistedmatrix.com/
http://canonical.com/

From greg.ewing at canterbury.ac.nz  Fri Jul 14 02:36:29 2006
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Fri, 14 Jul 2006 12:36:29 +1200
Subject: [Python-Dev] User's complaints
In-Reply-To: <e957mk$n22$1@sea.gmane.org>
References: <ee2a432c0607041649i7ecb6cc1w8ebdf47f7d02097a@mail.gmail.com>
	<3e1553560607122243q754dd71dq2e6d95cdf11e89dd@mail.gmail.com>
	<ee2a432c0607122324k633ed966n6c55a4921e529099@mail.gmail.com>
	<3e1553560607130052m66899441t50065786495928fe@mail.gmail.com>
	<44B60C0F.7060405@canterbury.ac.nz>
	<928173CD-DD5D-4B10-9CC4-ED63CC90933A@redivi.com>
	<e957mk$n22$1@sea.gmane.org>
Message-ID: <44B6E70D.9010508@canterbury.ac.nz>

Fredrik Lundh wrote:

> (and while we're at it, wouldn't a standard multiargument dispatch be
> nice replacement for the instance-oriented lookup we're using today?
> dispatching on a single value is so last century ;-)

That's highly debatable, and as I'm sure you
remember, has been highly debated. :-)

--
Greg

From greg.ewing at canterbury.ac.nz  Fri Jul 14 02:41:25 2006
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Fri, 14 Jul 2006 12:41:25 +1200
Subject: [Python-Dev] User's complaints
In-Reply-To: <3e1553560607130502o253a8359mf327964ee5d8ece8@mail.gmail.com>
References: <ee2a432c0607041649i7ecb6cc1w8ebdf47f7d02097a@mail.gmail.com>
	<3e1553560607122243q754dd71dq2e6d95cdf11e89dd@mail.gmail.com>
	<ee2a432c0607122324k633ed966n6c55a4921e529099@mail.gmail.com>
	<3e1553560607130052m66899441t50065786495928fe@mail.gmail.com>
	<44B60C0F.7060405@canterbury.ac.nz>
	<928173CD-DD5D-4B10-9CC4-ED63CC90933A@redivi.com>
	<3e1553560607130502o253a8359mf327964ee5d8ece8@mail.gmail.com>
Message-ID: <44B6E835.30208@canterbury.ac.nz>

Jeroen Ruigrok van der Werven wrote:
> It's just nice to be able to define a single class
> in multiple modules.

It *seems* nice until you want to track down which
source file the definition of some method comes
from.

Those used to the "one huge global namespace" of
C and C++ likely don't see this as a problem. But
since I've come to appreciate the benefits of
Python's module system, I don't want to go back
to that nightmare.

--
Greg

From terry at jon.es  Fri Jul 14 03:15:14 2006
From: terry at jon.es (Terry Jones)
Date: Fri, 14 Jul 2006 03:15:14 +0200
Subject: [Python-Dev] User's complaints
In-Reply-To: Your message at 12:41:25 on Friday, 14 July 2006
References: <ee2a432c0607041649i7ecb6cc1w8ebdf47f7d02097a@mail.gmail.com>
	<3e1553560607122243q754dd71dq2e6d95cdf11e89dd@mail.gmail.com>
	<ee2a432c0607122324k633ed966n6c55a4921e529099@mail.gmail.com>
	<3e1553560607130052m66899441t50065786495928fe@mail.gmail.com>
	<44B60C0F.7060405@canterbury.ac.nz>
	<928173CD-DD5D-4B10-9CC4-ED63CC90933A@redivi.com>
	<3e1553560607130502o253a8359mf327964ee5d8ece8@mail.gmail.com>
	<44B6E835.30208@canterbury.ac.nz>
Message-ID: <17590.61474.301502.95204@terry.jones.tc>

>>>>> "Greg" == Greg Ewing <greg.ewing at canterbury.ac.nz> writes:
Greg> Jeroen Ruigrok van der Werven wrote:
>> It's just nice to be able to define a single class
>> in multiple modules.

Greg> It *seems* nice until you want to track down which
Greg> source file the definition of some method comes
Greg> from.

Greg> Those used to the "one huge global namespace" of
Greg> C and C++ likely don't see this as a problem. But
Greg> since I've come to appreciate the benefits of
Greg> Python's module system, I don't want to go back
Greg> to that nightmare.

While I think there are good arguments both ways, I don't think that
finding source definitions of methods or classes counts as one - let alone
as a nightmare. Tools like tags (in vi and emacs and lisp environments)
work quickly and accurately, are easy to use (one keystroke in vi and
M-. or similar in emacs to go to a defn, and then a tags pop to come back),
work in an identical way on many source languages, and they have been
around for literally decades. That's to say nothing of IDE or CASE tools
that support finding definitions, callers, etc.

Regards,
Terry

From fdrake at acm.org  Fri Jul 14 03:21:27 2006
From: fdrake at acm.org (Fred L. Drake, Jr.)
Date: Thu, 13 Jul 2006 21:21:27 -0400
Subject: [Python-Dev] Community buildbots (was Re: User's complaints)
In-Reply-To: <65ECC6D7-7E46-463C-92D9-37AF2BF15234@python.org>
References: <20060713180322.29014.1145514133.divmod.quotient.35181@ohm>
	<65ECC6D7-7E46-463C-92D9-37AF2BF15234@python.org>
Message-ID: <200607132121.27521.fdrake@acm.org>

On Thursday 13 July 2006 16:05, Barry Warsaw wrote:
 > This really is an excellent point and makes me think that we may want
 > to consider elaborating on the Python release cycle to include a
 > gamma phase or a longer release candidate cycle.  OT1H I think there
...
 > "absolutely no changes are allowed now unless it's to fix backward
 > compatibility".  No more sneaking in new sys functions or types
 > module constants <wink> during the gamma phase.

+42

It feels like the release cycle from alpha1 to final has gotten increasingly 
rushed.  While I'm sure some of that is just me having my attention 
elsewhere, I suspect a longer tail on the cycle to do gammas (or release 
candidates, or whatever) would definately encourage more testing with 
applications and the larger frameworks.

No, it won't catch everything, but I think it would help.


  -Fred

-- 
Fred L. Drake, Jr.   <fdrake at acm.org>

From nnorwitz at gmail.com  Fri Jul 14 05:02:13 2006
From: nnorwitz at gmail.com (Neal Norwitz)
Date: Thu, 13 Jul 2006 20:02:13 -0700
Subject: [Python-Dev] User's complaints
In-Reply-To: <3e1553560607130052m66899441t50065786495928fe@mail.gmail.com>
References: <ee2a432c0607041649i7ecb6cc1w8ebdf47f7d02097a@mail.gmail.com>
	<3e1553560607122243q754dd71dq2e6d95cdf11e89dd@mail.gmail.com>
	<ee2a432c0607122324k633ed966n6c55a4921e529099@mail.gmail.com>
	<3e1553560607130052m66899441t50065786495928fe@mail.gmail.com>
Message-ID: <ee2a432c0607132002k901eff7geb472e7da6c919fd@mail.gmail.com>

On 7/13/06, Jeroen Ruigrok van der Werven <ashemedai at gmail.com> wrote:
>
> > If it's pure python, why don't people just copy everything under
> > site-packages after installing?  They could/should run compileall
> > after that to recompile the .pyc files.  With 2.5 on 64-bit machines,
> > C extension modules *must* be recompiled due to lots of internal
> > changes.
>
> I wasn't even aware of the compileall step, can you elaborate since
> this is the first time I see it being mentioned.

python -mcompileall -h

It's in the stdlib.  It is used by the build process to recursively
traverse directories and compile all the .py files into .pyc (or .pyo)
files.  So if you wanted to copy everything from site-package in 2.4
to 2.5, you could do something like:

cp -r /usr/lib/python2.{4,5}/site-packages
/usr/bin/python2.5 -mcompileall -d /usr/lib/python2.5/site-packages

That should really be all that's required. It of course doesn't verify
the packages are correct or that you still want to keep all the files.
 This generally works for .so's too.  However a warning will be
generated each time you import the .so since it was built on an old
version of Python..  On 2.5 in 64-bit systems, this will definitely
not work.

> > One thing you didn't mention that I've heard from time to time is the
> > stdlib should be improved.  For example, cleaning up old modules.
> > Though I'm not really sure everyone has the same thing in mind when it
> > comes to improving the stdlib.
>
> How do you envision cleaning up old modules?

For me, manually.  Do you still need moduleX?  There's no way for a
computer to tell you the answer, it depends on what you will use with
the new version of Python.

n

From anthony at interlink.com.au  Fri Jul 14 05:43:36 2006
From: anthony at interlink.com.au (Anthony Baxter)
Date: Fri, 14 Jul 2006 13:43:36 +1000
Subject: [Python-Dev] Community buildbots (was Re: User's complaints)
In-Reply-To: <20060713180322.29014.1145514133.divmod.quotient.35181@ohm>
References: <20060713180322.29014.1145514133.divmod.quotient.35181@ohm>
Message-ID: <200607141343.38424.anthony@interlink.com.au>

FWIW, I tend to run a few project(*) test suites when doing minor 
releases (2.x.y), just to make sure I don't break things. For major 
releases, it's a fair bit more work - something like Twisted or Zope3 
play at such a low level with the Python interfaces that there's 
nearly always breakages or massive warning dumps (from memory, 
zope.interface uses 'with' as an argument name, a lot). 

The "community buildbot" idea is a good one, although it should just 
be possible to write something for buildbot that checks out and 
builds the latest Python SVN, then installs it to a temporary 
location, then adds that location to the front of the path. Then each 
project could just add a "Python SVN buildbot" to their exists bbot 
install.


Anthony

(*) Typically includes Zope3 and Twisted, because I have these just 
lying around. 

From skip at pobox.com  Fri Jul 14 06:27:56 2006
From: skip at pobox.com (skip at pobox.com)
Date: Thu, 13 Jul 2006 23:27:56 -0500
Subject: [Python-Dev] Community buildbots
In-Reply-To: <20060713204146.29014.283506497.divmod.quotient.35352@ohm>
References: <2mfyh52wrn.fsf@starship.python.net>
	<20060713204146.29014.283506497.divmod.quotient.35352@ohm>
Message-ID: <17591.7500.168119.701223@montanaro.dyndns.org>


    glyph> *can* always do that, and some other Twisted devs watch
    glyph> python-dev a bit more closely than I do, but the point is that
    glyph> the amount of effort required to do this is prohibitive for the
    glyph> average Python hacker, whereas the time to set up an individual
    glyph> buildbot might not be.)

If you're concerned about noticing when a new release train is pulling out
of the station I think it would be sufficient to subscribe to the low-volume
python-announce mailing list.  You will see announcements about alphas and
betas there.  Even lower volume would be a subscription to the RSS feed of
Python announcements on the python.org front page (scroll to the bottom).

The buildbot idea sounds excellent.

Skip

From skip at pobox.com  Fri Jul 14 06:32:52 2006
From: skip at pobox.com (skip at pobox.com)
Date: Thu, 13 Jul 2006 23:32:52 -0500
Subject: [Python-Dev] Community buildbots (was Re: User's complaints)
In-Reply-To: <200607132121.27521.fdrake@acm.org>
References: <20060713180322.29014.1145514133.divmod.quotient.35181@ohm>
	<65ECC6D7-7E46-463C-92D9-37AF2BF15234@python.org>
	<200607132121.27521.fdrake@acm.org>
Message-ID: <17591.7796.553081.425160@montanaro.dyndns.org>


    Fred> It feels like the release cycle from alpha1 to final has gotten
    Fred> increasingly rushed.

Same here.  I believe there was some shortening of the 2.5 release cycle two
or three months ago.  I don't recall why or by how much, but I think the
acceleration has resulted in a lot of the "can't we please squeeze this one
little change in?" that's been happening.  Shortening a micro release a bit
seems reasonably easy to accommodate, but since minor releases occur so
infrequently, I think it would be better to stretch them out if necessary.

Skip

From greg.ewing at canterbury.ac.nz  Fri Jul 14 06:55:34 2006
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Fri, 14 Jul 2006 16:55:34 +1200
Subject: [Python-Dev] Handling of sys.args (Re:  User's complaints)
In-Reply-To: <Pine.LNX.4.58.0607131333290.31087@server1.LFW.org>
References: <ee2a432c0607041649i7ecb6cc1w8ebdf47f7d02097a@mail.gmail.com>
	<3e1553560607122243q754dd71dq2e6d95cdf11e89dd@mail.gmail.com>
	<4c45c1530607130037r1c14e3f6u972232bdfa481534@mail.gmail.com>
	<Pine.LNX.4.58.0607130240480.31087@server1.LFW.org>
	<44B60B5A.2080309@canterbury.ac.nz>
	<Pine.LNX.4.58.0607131333290.31087@server1.LFW.org>
Message-ID: <44B723C6.9010309@canterbury.ac.nz>

Ka-Ping Yee wrote:

> I think of 'sys' as the place for sensitive interpreter internals

Well, it seems to be rather a mixture at the moment.
I suppose you could regard sys.modules as fairly
sensitive, since messing with it can have big effects
on the behaviour of the whole program, and changing
sys.stdout or sys.stderr also has global effects.

But it's also how you do things like getting info
about the current exception, which doesn't seem
particularly sensitive to me.

Maybe sys needs to be split into two modules, with
the non-sensitive one pre-imported (so that the
importless interpreter you suggest wouldn't be
unnecessarily crippled).

--
Greg

From greg.ewing at canterbury.ac.nz  Fri Jul 14 07:04:00 2006
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Fri, 14 Jul 2006 17:04:00 +1200
Subject: [Python-Dev] Handling of sys.args (Re: User's complaints)
In-Reply-To: <ca471dc20607131225x63220367q83e402abf7420250@mail.gmail.com>
References: <ee2a432c0607041649i7ecb6cc1w8ebdf47f7d02097a@mail.gmail.com>
	<3e1553560607122243q754dd71dq2e6d95cdf11e89dd@mail.gmail.com>
	<4c45c1530607130037r1c14e3f6u972232bdfa481534@mail.gmail.com>
	<Pine.LNX.4.58.0607130240480.31087@server1.LFW.org>
	<44B60B5A.2080309@canterbury.ac.nz>
	<Pine.LNX.4.58.0607131333290.31087@server1.LFW.org>
	<ca471dc20607131225x63220367q83e402abf7420250@mail.gmail.com>
Message-ID: <44B725C0.5000202@canterbury.ac.nz>

Guido van Rossum wrote:
> I'nm afraid if we
> were to split it by functionality we'd have to split it 5-way or so...

What about just splitting it into "mutable" and
"immutable" parts? That would be a fairly clear
division, I think.

--
Greg

From greg.ewing at canterbury.ac.nz  Fri Jul 14 07:09:40 2006
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Fri, 14 Jul 2006 17:09:40 +1200
Subject: [Python-Dev] Handling of sys.args (Re: User's complaints)
In-Reply-To: <E1G17Hd-0005nY-Iu@libra.cus.cam.ac.uk>
References: <E1G17Hd-0005nY-Iu@libra.cus.cam.ac.uk>
Message-ID: <44B72714.6030502@canterbury.ac.nz>

Nick Maclaren wrote:
> On systems that are not Unix-derived (which, nowadays, are rare),
> there is commonly no such thing as a program name in the first place.
> It is possible to get into that state on some Unices - i.e. ones which
> have a form of exec that takes a file descriptor, inode number or
> whatever.

I don't think that applies to the Python args[] though,
since its args[0] isn't the path of the OS-level
executable, it's the path of the main Python script.

But you could still end up without one, if the main
script comes from somewhere other than a file.

--
Greg

From glyph at divmod.com  Fri Jul 14 07:11:57 2006
From: glyph at divmod.com (glyph at divmod.com)
Date: Fri, 14 Jul 2006 01:11:57 -0400
Subject: [Python-Dev] Community buildbots
In-Reply-To: <17591.7500.168119.701223@montanaro.dyndns.org>
Message-ID: <20060714051157.29014.803759518.divmod.quotient.35955@ohm>

On Thu, 13 Jul 2006 23:27:56 -0500, skip at pobox.com wrote:

>The buildbot idea sounds excellent.

Thanks.  If someone can set this up, it pretty much addresses my concerns.

>If you're concerned about noticing when a new release train is pulling out
>of the station I think it would be sufficient to subscribe to the low-volume
>python-announce mailing list.  You will see announcements about alphas and
>betas there.  Even lower volume would be a subscription to the RSS feed of
>Python announcements on the python.org front page (scroll to the bottom).

I am aware of when new releases come out :).  What I'm not aware of is what
features (may) have broken my code, and why.  As long as Python's trunk at HEAD
continues to run the test suites cleanly, I am mostly unconcerned.  When it
breaks, though, I want a chance to look at the cause of the breakage, *before*
there is an alpha or beta Python release out and people are starting to write
code that depends on its new features.  Most importantly, python-dev is
extremely context-sensitive.  I want to be able to participate in the discussion
when my concerns are relevant and still reasonable to act upon.

Additionally I would like to know about these changes so that I can modify code
to support Python releases and release a compatible version of Twisted _in
advance_ of the Python release that's going to break them, so assuming users are
keeping relatively current, there won't be a window where their most recent
Twisted release will not work with the most recent Python release.

From talin at acm.org  Fri Jul 14 07:15:27 2006
From: talin at acm.org (Talin)
Date: Thu, 13 Jul 2006 22:15:27 -0700
Subject: [Python-Dev] Community buildbots
In-Reply-To: <05af01c6a6be$54b653b0$d503030a@trilan>
References: <2mfyh52wrn.fsf@starship.python.net>	<20060713204146.29014.283506497.divmod.quotient.35352@ohm>
	<05af01c6a6be$54b653b0$d503030a@trilan>
Message-ID: <44B7286F.7070406@acm.org>

Giovanni Bajo wrote:
> glyph at divmod.com wrote:
> 
> I think python should have a couple more of future imports. "from __future__
> import new_classes" and "from __future__ import unicode_literals" would be
> really welcome, and would smooth the Py3k migration process

Actually - can we make new-style classes the default, but allow a way to 
switch to old-style classes if needed? Perhaps a command-line argument 
to set the default back to old-style?

-- Talin

From greg.ewing at canterbury.ac.nz  Fri Jul 14 07:20:41 2006
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Fri, 14 Jul 2006 17:20:41 +1200
Subject: [Python-Dev] Community buildbots (was Re: User's complaints)
In-Reply-To: <65ECC6D7-7E46-463C-92D9-37AF2BF15234@python.org>
References: <20060713180322.29014.1145514133.divmod.quotient.35181@ohm>
	<65ECC6D7-7E46-463C-92D9-37AF2BF15234@python.org>
Message-ID: <44B729A9.3060705@canterbury.ac.nz>

Barry Warsaw wrote:
> we may want  
> to consider elaborating on the Python release cycle to include a  
> gamma phase or a longer release candidate cycle.

Maybe there could be an "unstable" release phase that
lasts for a whole release cycle. So you'd first release
version 2.n as "unstable", and keep 2.(n-1) as the
current "stable" release. Then when 2.(n+1) is ready,
2.n would become "stable" and 2.(n+1) would become
the new "unstable".

(Feel free to substitute other terms for "stable"
and "unstable" if you don't like them.)

That would give people plenty of warning and time to
try things out with the new version.

This wouldn't actually be much different to what is
done now, except for the naming. But by not officially
blessing the latest release as current for a while,
it might give less of an impression that stuff is
being sprung on the community unawares.

> OT1H I think there  
> will always be people or projects that won't try anything until the  
> gold release,

Well, at least they've had a chance to try it. If they
don't take that chance, they don't have much ground for
complaint.

--
Greg

From fredrik at pythonware.com  Fri Jul 14 07:26:39 2006
From: fredrik at pythonware.com (Fredrik Lundh)
Date: Fri, 14 Jul 2006 07:26:39 +0200
Subject: [Python-Dev] Community buildbots
In-Reply-To: <44B7286F.7070406@acm.org>
References: <2mfyh52wrn.fsf@starship.python.net>	<20060713204146.29014.283506497.divmod.quotient.35352@ohm>	<05af01c6a6be$54b653b0$d503030a@trilan>
	<44B7286F.7070406@acm.org>
Message-ID: <e979ud$61l$1@sea.gmane.org>

Talin wrote:

>> I think python should have a couple more of future imports. "from __future__
>> import new_classes" and "from __future__ import unicode_literals" would be
>> really welcome, and would smooth the Py3k migration process
> 
> Actually - can we make new-style classes the default, but allow a way to 
> switch to old-style classes if needed? Perhaps a command-line argument 
> to set the default back to old-style?

so people complain about new releases breaking too much code, and your 
response to that is break even more code ?

</F>


From greg.ewing at canterbury.ac.nz  Fri Jul 14 07:27:57 2006
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Fri, 14 Jul 2006 17:27:57 +1200
Subject: [Python-Dev] Community buildbots
In-Reply-To: <B02BC5ED-9165-4C01-B79F-4BEDE9EC5C59@redivi.com>
References: <2mfyh52wrn.fsf@starship.python.net>
	<20060713204146.29014.283506497.divmod.quotient.35352@ohm>
	<05af01c6a6be$54b653b0$d503030a@trilan>
	<B02BC5ED-9165-4C01-B79F-4BEDE9EC5C59@redivi.com>
Message-ID: <44B72B5D.9050905@canterbury.ac.nz>

Bob Ippolito wrote:

> "from __future__ import new_classes" exists, but the syntax is  
> different:
> 
> __metaclass__ = type

Although it's not a very obvious spelling,
particularly to the casual reader who may not be
familiar with the intricacies of classes and
metaclasses. I don't think it would hurt to have
it available as a __future__ import as well.

There's also the advantage that all of a
module's future assumptions could then be
documented uniformly in one place, i.e. in
a __future__ import at the top.

--
Greg

From fdrake at acm.org  Fri Jul 14 07:39:38 2006
From: fdrake at acm.org (Fred L. Drake, Jr.)
Date: Fri, 14 Jul 2006 01:39:38 -0400
Subject: [Python-Dev] Community buildbots (was Re: User's complaints)
In-Reply-To: <17591.7796.553081.425160@montanaro.dyndns.org>
References: <20060713180322.29014.1145514133.divmod.quotient.35181@ohm>
	<200607132121.27521.fdrake@acm.org>
	<17591.7796.553081.425160@montanaro.dyndns.org>
Message-ID: <200607140139.38384.fdrake@acm.org>

On Friday 14 July 2006 00:32, skip at pobox.com wrote:
 > Same here.  I believe there was some shortening of the 2.5 release cycle
 > two or three months ago.  I don't recall why or by how much, but I think
 > the acceleration has resulted in a lot of the "can't we please squeeze
 > this one little change in?" that's been happening.  Shortening a micro
 > release a bit seems reasonably easy to accommodate, but since minor
 > releases occur so infrequently, I think it would be better to stretch them
 > out if necessary.

The squeezing of the releases isn't where the problem is, I think.  It's that, 
once squeezed, more releases aren't being added to compensate.  We really 
need to determine what time we need to go from beta1 to (gamma|rc)1, and then 
from (gamma|rc)1 to final.  Plenty of interim releases in the beta phase is 
good.


  -Fred

-- 
Fred L. Drake, Jr.   <fdrake at acm.org>

From fredrik at pythonware.com  Fri Jul 14 07:45:24 2006
From: fredrik at pythonware.com (Fredrik Lundh)
Date: Fri, 14 Jul 2006 07:45:24 +0200
Subject: [Python-Dev] Community buildbots (was Re: User's complaints)
In-Reply-To: <200607132121.27521.fdrake@acm.org>
References: <20060713180322.29014.1145514133.divmod.quotient.35181@ohm>	<65ECC6D7-7E46-463C-92D9-37AF2BF15234@python.org>
	<200607132121.27521.fdrake@acm.org>
Message-ID: <e97b1i$8nt$1@sea.gmane.org>

Fred L. Drake, Jr. wrote:

> It feels like the release cycle from alpha1 to final has gotten increasingly 
> rushed.

python 2.2: ~5 months
python 2.3: ~7 months
python 2.4: ~5 months
python 2.5: ~4 months

I think the biggest problem is the time between "basically stable beta" 
and "final"; it was ~90 days in 2.3 and is now ~50 days, most of which 
is vacation time in my part of the world.

we could extend the beta period by playing silly games with naming, but 
as long as the "big things" go in *before* the beta period starts, I 
think it's easier to simply extend the beta period by adding another 
month to it.

I'd prefer something like 90 days.

</F>


From fdrake at acm.org  Fri Jul 14 07:50:13 2006
From: fdrake at acm.org (Fred L. Drake, Jr.)
Date: Fri, 14 Jul 2006 01:50:13 -0400
Subject: [Python-Dev] Community buildbots (was Re: User's complaints)
In-Reply-To: <e97b1i$8nt$1@sea.gmane.org>
References: <20060713180322.29014.1145514133.divmod.quotient.35181@ohm>
	<200607132121.27521.fdrake@acm.org> <e97b1i$8nt$1@sea.gmane.org>
Message-ID: <200607140150.13563.fdrake@acm.org>

On Friday 14 July 2006 01:45, Fredrik Lundh wrote:
 > I'd prefer something like 90 days.

+1


  -Fred

-- 
Fred L. Drake, Jr.   <fdrake at acm.org>

From anthony at interlink.com.au  Fri Jul 14 07:51:00 2006
From: anthony at interlink.com.au (Anthony Baxter)
Date: Fri, 14 Jul 2006 15:51:00 +1000
Subject: [Python-Dev] Community buildbots (was Re: User's complaints)
In-Reply-To: <65ECC6D7-7E46-463C-92D9-37AF2BF15234@python.org>
References: <20060713180322.29014.1145514133.divmod.quotient.35181@ohm>
	<65ECC6D7-7E46-463C-92D9-37AF2BF15234@python.org>
Message-ID: <200607141551.06701.anthony@interlink.com.au>

On Friday 14 July 2006 06:05, Barry Warsaw wrote:
> This really is an excellent point and makes me think that we may
> want to consider elaborating on the Python release cycle to include
> a gamma phase or a longer release candidate cycle.  OT1H I think
> there will always be people or projects that won't try anything
> until the gold release, and that if we've broken anything in their
> code we simply won't know it until after that, no matter how
> diligent we are.  OTOH, a more formal gamma phase would allow us to
> say "absolutely no changes are allowed now unless it's to fix
> backward compatibility".  No more sneaking in new sys functions or
> types module constants <wink> during the gamma phase.

    alpha 1: April 5, 2006 [completed]
    alpha 2: April 27, 2006 [completed]
    beta 1:  June 20, 2006 [completed]
    beta 2:  July 11, 2006 [completed]
    rc 1:    August 1, 2006 [planned]
    final:   August 8, 2006 [planned]

Four months would seem to me to be quite long enough as a release 
cycle. Extending it means far more work for everyone - either we have 
special checkin rules for the trunk for a longer period of time 
(requiring extra monitoring by people like myself and Neal), or we 
branch earlier, requiring double commits to the trunk and the branch 
for bugfixes. 

I also strongly doubt that making a longer "release candidate" cycle 
would lead to any significant additional testing by end-users.

Anthony
-- 
Anthony Baxter     <anthony at interlink.com.au>
It's never too late to have a happy childhood.

From greg.ewing at canterbury.ac.nz  Fri Jul 14 07:52:05 2006
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Fri, 14 Jul 2006 17:52:05 +1200
Subject: [Python-Dev] Community buildbots
In-Reply-To: <44B7286F.7070406@acm.org>
References: <2mfyh52wrn.fsf@starship.python.net>
	<20060713204146.29014.283506497.divmod.quotient.35352@ohm>
	<05af01c6a6be$54b653b0$d503030a@trilan> <44B7286F.7070406@acm.org>
Message-ID: <44B73105.70707@canterbury.ac.nz>

Talin wrote:

> Actually - can we make new-style classes the default, but allow a way to 
> switch to old-style classes if needed?

That sounds dangerously like a "from __past__"
kind of feature, and Guido has said that there
will never be a __past__ module.

Also, this is probably not something that
should be a program-wide setting.

--
Greg


From nnorwitz at gmail.com  Fri Jul 14 07:55:04 2006
From: nnorwitz at gmail.com (Neal Norwitz)
Date: Thu, 13 Jul 2006 22:55:04 -0700
Subject: [Python-Dev] Community buildbots (was Re: User's complaints)
In-Reply-To: <17591.7796.553081.425160@montanaro.dyndns.org>
References: <20060713180322.29014.1145514133.divmod.quotient.35181@ohm>
	<65ECC6D7-7E46-463C-92D9-37AF2BF15234@python.org>
	<200607132121.27521.fdrake@acm.org>
	<17591.7796.553081.425160@montanaro.dyndns.org>
Message-ID: <ee2a432c0607132255l2a06ac3avaf200283239424ee@mail.gmail.com>

On 7/13/06, skip at pobox.com <skip at pobox.com> wrote:
>
>     Fred> It feels like the release cycle from alpha1 to final has gotten
>     Fred> increasingly rushed.

I think that's just because you are getting older and time goes by
faster the less time you've got left. :-)  It seems to be going quite
quick, but I'm not so sure it's really different.

> Same here.  I believe there was some shortening of the 2.5 release cycle two
> or three months ago.  I don't recall why or by how much, but I think the
> acceleration has resulted in a lot of the "can't we please squeeze this one
> little change in?" that's been happening.  Shortening a micro release a bit
> seems reasonably easy to accommodate, but since minor releases occur so
> infrequently, I think it would be better to stretch them out if necessary.

Not exactly.  The PEP was created Feb 7 (r42266) with a very relaxed
schedule to allow enough time for ssize_t branch to solidify.  Martin
was ready sooner, so on Feb 26 (r42577) sped up the schedule to
something more reasonable:

-    alpha 1: April 1, 2006 [planned]
-    alpha 2: April 29, 2006 [planned]
-    beta 1:  June 24, 2006 [planned]
-    beta 2:  July 15, 2006 [planned]
-    rc 1:    August 5, 2006 [planned]
-    final:   August 19, 2006 [planned]

The current schedule is:

+    alpha 1: April 5, 2006 [completed]
+    alpha 2: April 27, 2006 [completed]
+    beta 1:  June 20, 2006 [completed]
+    beta 2:  July 11, 2006 [completed]
+    rc 1:    August 1, 2006 [planned]
+    final:   August 8, 2006 [planned]

So beta1 was 4 days earlier than the date set at the end of Feb.  The
first beta was  about 19 months after 2.4 was released if I'm counting
correctly.  In 2.4 beta1 to release was 1.5 months.  For 2.5 it is
about the same.  The total 2.4 schedule (from a1 to release) was 4.5
months which included 3 alphas.  For 2.5 we only had 2 alphas and the
total schedule is just over 4 months.  So 2.5 is just a little faster.
 But 2.5 has had a ton more testing than any previous released due to
the buildbots, not to mention Coverity and now Klocwork.

I definitely pushed the schedule, but I don't think it was really all
that different from in the past.

Given that several people here think we should lengthen the schedule
in some way, I suspect we will do something.  I'm not really against
it, but I don't think it will provide much benefit either.  A few more
bugs will be fixed since we have more time.  What I think might really
make a difference if we could leverage the buildbots and copy the
builds (especially WIndows) up to python.org and make them available
for download.  That might make it easier for people to try stuff out.

n

From nnorwitz at gmail.com  Fri Jul 14 07:59:48 2006
From: nnorwitz at gmail.com (Neal Norwitz)
Date: Thu, 13 Jul 2006 22:59:48 -0700
Subject: [Python-Dev] Community buildbots (was Re: User's complaints)
In-Reply-To: <60ed19d40607131532g3c0a5cc1n3177a22d90999dab@mail.gmail.com>
References: <20060713182916.GA12832@panix.com>
	<20060713223024.29014.1648933869.divmod.quotient.35468@ohm>
	<60ed19d40607131532g3c0a5cc1n3177a22d90999dab@mail.gmail.com>
Message-ID: <ee2a432c0607132259n5a2fd52jd903cf37a24f0c3a@mail.gmail.com>

On 7/13/06, Christopher Armstrong <radix at twistedmatrix.com> wrote:
> On 7/13/06, glyph at divmod.com <glyph at divmod.com> wrote:
> > On Thu, 13 Jul 2006 11:29:16 -0700, Aahz <aahz at pythoncraft.com> wrote:
> >
> > >There's been some recent discussion in the PSF wondering where it would
> > >make sense to throw some money to remove grit in the wheels; do you think
> > >this is a case where that would help?
> >
> > Most likely yes.  It's not a huge undertaking, and there are a lot of people out
> > there in the community with the knowledge of Buildbot to make this happen.
>
> I'm at least willing to set up the buildbots for projects I care about
> (Twisted, pydoctor, whatever), and perhaps help out with the setting
> up the buildmaster.

It would be really great to have someone setup buildbots for other
important/large python projects to test python svn HEAD with.  I can't
even stay on top of the python buildbots, so I'm not volunteering for
more work.  I will greatly appreciate and thank whoever takes this on
though.  :-)

The harder problem is keeping it running.  Setting it up, in some ways
is a lot easier.  It's at least well defined.  Maintaining it for
several months is a different story.

n

From fredrik at pythonware.com  Fri Jul 14 08:15:31 2006
From: fredrik at pythonware.com (Fredrik Lundh)
Date: Fri, 14 Jul 2006 08:15:31 +0200
Subject: [Python-Dev] Community buildbots (was Re: User's complaints)
In-Reply-To: <ee2a432c0607132255l2a06ac3avaf200283239424ee@mail.gmail.com>
References: <20060713180322.29014.1145514133.divmod.quotient.35181@ohm>	<65ECC6D7-7E46-463C-92D9-37AF2BF15234@python.org>	<200607132121.27521.fdrake@acm.org>	<17591.7796.553081.425160@montanaro.dyndns.org>
	<ee2a432c0607132255l2a06ac3avaf200283239424ee@mail.gmail.com>
Message-ID: <e97cq1$cmm$1@sea.gmane.org>

Neal Norwitz wrote:

> Given that several people here think we should lengthen the schedule
> in some way, I suspect we will do something.  I'm not really against
> it, but I don't think it will provide much benefit either.  A few more
> bugs will be fixed since we have more time.

you're still missing the point of this thread: it isn't bugs in the core 
Python distribution that's the problem, it's compatibility with third-
party modules and applications.

I'm quite sure that we could ship the current beta2 as 2.5 final and 
have a Python release that, in itself, is more stable than all the ones 
before it, but since 2.5 introduces lots of new stuff, that stability 
doesn't necessarily translate to a better experience for people who 
wants to use their existing applications and libraries.

a longer beta period gives *external* developers more time to catch up, 
and results in less work for the end users.

</F>


From nnorwitz at gmail.com  Fri Jul 14 08:39:06 2006
From: nnorwitz at gmail.com (Neal Norwitz)
Date: Thu, 13 Jul 2006 23:39:06 -0700
Subject: [Python-Dev] Community buildbots (was Re: User's complaints)
In-Reply-To: <e97cq1$cmm$1@sea.gmane.org>
References: <20060713180322.29014.1145514133.divmod.quotient.35181@ohm>
	<65ECC6D7-7E46-463C-92D9-37AF2BF15234@python.org>
	<200607132121.27521.fdrake@acm.org>
	<17591.7796.553081.425160@montanaro.dyndns.org>
	<ee2a432c0607132255l2a06ac3avaf200283239424ee@mail.gmail.com>
	<e97cq1$cmm$1@sea.gmane.org>
Message-ID: <ee2a432c0607132339o1d35c402ib13884c85115856a@mail.gmail.com>

On 7/13/06, Fredrik Lundh <fredrik at pythonware.com> wrote:
> Neal Norwitz wrote:
>
> > Given that several people here think we should lengthen the schedule
> > in some way, I suspect we will do something.  I'm not really against
> > it, but I don't think it will provide much benefit either.  A few more
> > bugs will be fixed since we have more time.
>
> you're still missing the point of this thread: it isn't bugs in the core
> Python distribution that's the problem, it's compatibility with third-
> party modules and applications.

...

> a longer beta period gives *external* developers more time to catch up,
> and results in less work for the end users.

This is the part I don't get.  For the external developers, if they
care about compatibility, why aren't they testing periodically,
regardless of alpha/beta releases?  How often is the python build
broken or otherwise unusable?  On Unix there's no more or less work to
grab a tarball whether it's a nightly or a release.  For Windows it's
definitely easier to wait for a release.  If there was an installable
windows version made by the buildbots, would that mean there would be
more testing?

Is part of your point that these developers only care about something
called "release" and they won't start testing before then?  If that's
the case why don't we start making (semi-)automated alpha releases
every month?  That will give people lots more time.  Somehow I don't
think it will make a difference.  People mostly work on schedules or
should I say avoid schedules.  Give them 5 months and they will still
wait to the last minute.

Remember I also tried to push for more features to go in early?  That
would have given more time for external testing.  Still features are
coming in.  Python developers weren't happy about having to get things
in earlier.  I don't see a practical way to implement what you propose
(see Anthony's comments).

n

From fredrik at pythonware.com  Fri Jul 14 09:00:20 2006
From: fredrik at pythonware.com (Fredrik Lundh)
Date: Fri, 14 Jul 2006 09:00:20 +0200
Subject: [Python-Dev] Community buildbots (was Re: User's complaints)
In-Reply-To: <ee2a432c0607132339o1d35c402ib13884c85115856a@mail.gmail.com>
References: <20060713180322.29014.1145514133.divmod.quotient.35181@ohm>	<65ECC6D7-7E46-463C-92D9-37AF2BF15234@python.org>	<200607132121.27521.fdrake@acm.org>	<17591.7796.553081.425160@montanaro.dyndns.org>	<ee2a432c0607132255l2a06ac3avaf200283239424ee@mail.gmail.com>	<e97cq1$cmm$1@sea.gmane.org>
	<ee2a432c0607132339o1d35c402ib13884c85115856a@mail.gmail.com>
Message-ID: <e97fe2$jjc$1@sea.gmane.org>

Neal Norwitz wrote:

> This is the part I don't get.  For the external developers, if they
> care about compatibility, why aren't they testing periodically,
> regardless of alpha/beta releases?

because they don't want to waste time and effort on testing against 
releases that are not necessarily a close approximation of the full
release?  testing takes time, and time can be used for many different 
things.

</F>


From anthony at interlink.com.au  Fri Jul 14 09:10:56 2006
From: anthony at interlink.com.au (Anthony Baxter)
Date: Fri, 14 Jul 2006 17:10:56 +1000
Subject: [Python-Dev] Community buildbots (was Re: User's complaints)
In-Reply-To: <e97fe2$jjc$1@sea.gmane.org>
References: <20060713180322.29014.1145514133.divmod.quotient.35181@ohm>
	<ee2a432c0607132339o1d35c402ib13884c85115856a@mail.gmail.com>
	<e97fe2$jjc$1@sea.gmane.org>
Message-ID: <200607141711.03069.anthony@interlink.com.au>

On Friday 14 July 2006 17:00, Fredrik Lundh wrote:
> because they don't want to waste time and effort on testing against
> releases that are not necessarily a close approximation of the full
> release?  testing takes time, and time can be used for many
> different things.

Oddly enough, so does release management.

Anthony

-- 
Anthony Baxter     <anthony at interlink.com.au>
It's never too late to have a happy childhood.

From anthony at interlink.com.au  Fri Jul 14 09:14:06 2006
From: anthony at interlink.com.au (Anthony Baxter)
Date: Fri, 14 Jul 2006 17:14:06 +1000
Subject: [Python-Dev] Community buildbots (was Re: User's complaints)
In-Reply-To: <ee2a432c0607132339o1d35c402ib13884c85115856a@mail.gmail.com>
References: <20060713180322.29014.1145514133.divmod.quotient.35181@ohm>
	<e97cq1$cmm$1@sea.gmane.org>
	<ee2a432c0607132339o1d35c402ib13884c85115856a@mail.gmail.com>
Message-ID: <200607141714.09304.anthony@interlink.com.au>

On Friday 14 July 2006 16:39, Neal Norwitz wrote:
> Remember I also tried to push for more features to go in early? 
> That would have given more time for external testing.  Still
> features are coming in.  Python developers weren't happy about
> having to get things in earlier.  I don't see a practical way to
> implement what you propose (see Anthony's comments).

Following up on this point: 
Given the number of "just-this-one-more-thing-please" we've _already_ 
had since the b1 feature freeze, do you really except that 90 days of 
feature freeze is feasible? And if there's not going to be a feature 
freeze, there's hardly any point to people doing testing until there 
_is_ a feature freeze, is there? Oh, great, my code works with 2.5b1. 
Oops. 2.5b9 added a new feature that broke my code, but I didn't test 
with that. 

Anthony
-- 
Anthony Baxter     <anthony at interlink.com.au>
It's never too late to have a happy childhood.

From nmm1 at cus.cam.ac.uk  Fri Jul 14 10:42:45 2006
From: nmm1 at cus.cam.ac.uk (Nick Maclaren)
Date: Fri, 14 Jul 2006 09:42:45 +0100
Subject: [Python-Dev] Handling of sys.args (Re: User's complaints)
In-Reply-To: Your message of "Fri, 14 Jul 2006 17:09:40 +1200."
	<44B72714.6030502@canterbury.ac.nz> 
Message-ID: <E1G1JG1-0001uU-8s@libra.cus.cam.ac.uk>

Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
> 
> > On systems that are not Unix-derived (which, nowadays, are rare),
> > there is commonly no such thing as a program name in the first place.
> > It is possible to get into that state on some Unices - i.e. ones which
> > have a form of exec that takes a file descriptor, inode number or
> > whatever.
> 
> I don't think that applies to the Python args[] though,
> since its args[0] isn't the path of the OS-level
> executable, it's the path of the main Python script.

Oh, yes, it does!  The file descriptor or inode number could refer to
the script just as well as it could to the interpreter binary.

> But you could still end up without one, if the main
> script comes from somewhere other than a file.

I didn't want to include that, to avoid confusing people who haven't
used systems with such features.  Several systems have had the ability
to exec to a memory segment, for example.  But, yes.


Regards,
Nick Maclaren,
University of Cambridge Computing Service,
New Museums Site, Pembroke Street, Cambridge CB2 3QH, England.
Email:  nmm1 at cam.ac.uk
Tel.:  +44 1223 334761    Fax:  +44 1223 334679

From fredrik at pythonware.com  Fri Jul 14 10:55:32 2006
From: fredrik at pythonware.com (Fredrik Lundh)
Date: Fri, 14 Jul 2006 10:55:32 +0200
Subject: [Python-Dev] Handling of sys.args (Re: User's complaints)
In-Reply-To: <E1G1JG1-0001uU-8s@libra.cus.cam.ac.uk>
References: <44B72714.6030502@canterbury.ac.nz>
	<E1G1JG1-0001uU-8s@libra.cus.cam.ac.uk>
Message-ID: <e97m62$6v7$1@sea.gmane.org>

Nick Maclaren wrote:

>> I don't think that applies to the Python args[] though,
>> since its args[0] isn't the path of the OS-level
>> executable, it's the path of the main Python script.
> 
> Oh, yes, it does!  The file descriptor or inode number could refer to
> the script just as well as it could to the interpreter binary.

(insert sound of head hitting desk here)

</F>


From rasky at develer.com  Fri Jul 14 11:06:17 2006
From: rasky at develer.com (Giovanni Bajo)
Date: Fri, 14 Jul 2006 11:06:17 +0200
Subject: [Python-Dev] Community buildbots
References: <2mfyh52wrn.fsf@starship.python.net>
	<20060713204146.29014.283506497.divmod.quotient.35352@ohm>
	<05af01c6a6be$54b653b0$d503030a@trilan>
	<B02BC5ED-9165-4C01-B79F-4BEDE9EC5C59@redivi.com>
	<44B72B5D.9050905@canterbury.ac.nz>
Message-ID: <091801c6a724$c3ee1860$12472597@bagio>

Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:

>> "from __future__ import new_classes" exists, but the syntax is
>> different:
>> 
>> __metaclass__ = type
> 
> Although it's not a very obvious spelling,
> particularly to the casual reader who may not be
> familiar with the intricacies of classes and
> metaclasses. I don't think it would hurt to have
> it available as a __future__ import as well.
> 
> There's also the advantage that all of a
> module's future assumptions could then be
> documented uniformly in one place, i.e. in
> a __future__ import at the top.

+1.

Giovanni Bajo


From mwh at python.net  Fri Jul 14 11:15:11 2006
From: mwh at python.net (Michael Hudson)
Date: Fri, 14 Jul 2006 10:15:11 +0100
Subject: [Python-Dev] Community buildbots
In-Reply-To: <20060713204146.29014.283506497.divmod.quotient.35352@ohm>
	(glyph@divmod.com's message of "Thu, 13 Jul 2006 16:41:46 -0400")
References: <20060713204146.29014.283506497.divmod.quotient.35352@ohm>
Message-ID: <2mbqrs35uo.fsf@starship.python.net>

glyph at divmod.com writes:

> I just would have appreciated the opportunity to participate in the
> discussion before the betas were out and the featureset frozen.

I think something that has happened to some extent with this release
is that there was a long-ish period where stuff got discussed and
implemented (PEPs 343 and 344, new-style exceptions, the new compiler,
the ssize_t branch) and then suddenly we went "crap!  we have to
release this!" and then the whole alpha/beta/release part has happened
much more quickly.  Maybe we should have had a more explicit
discussion on what was going to be in 2.5 before beta1, but that kind
of thing is difficult to make happen (it's entirely possible that it
did happen and I just missed it by being snowed under, but I don't
think so).

Cheers,
mwh

-- 
 Have you considered downgrading your arrogance to a reasonable level?
        -- Erik Naggum, comp.lang.lisp, to yet another C++-using troll

From ncoghlan at gmail.com  Fri Jul 14 11:31:11 2006
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 14 Jul 2006 19:31:11 +1000
Subject: [Python-Dev] Handling of sys.args (Re: User's complaints)
In-Reply-To: <44B72714.6030502@canterbury.ac.nz>
References: <E1G17Hd-0005nY-Iu@libra.cus.cam.ac.uk>
	<44B72714.6030502@canterbury.ac.nz>
Message-ID: <44B7645F.6040900@gmail.com>

Greg Ewing wrote:
> Nick Maclaren wrote:
>> On systems that are not Unix-derived (which, nowadays, are rare),
>> there is commonly no such thing as a program name in the first place.
>> It is possible to get into that state on some Unices - i.e. ones which
>> have a form of exec that takes a file descriptor, inode number or
>> whatever.
> 
> I don't think that applies to the Python args[] though,
> since its args[0] isn't the path of the OS-level
> executable, it's the path of the main Python script.
> 
> But you could still end up without one, if the main
> script comes from somewhere other than a file.

sys.argv[0] can end up being None in Python 2.5  - zipimporter objects don't 
currently expose the info runpy.run_module needs to set __file__ and 
sys.argv[0] correctly, so they'll both end up being None if you use -m to run 
a module from a zipfile.

PEP 302 went a fairly long way towards decoupling imports from the filesystem 
lay out, but it isn't all the way there just yet.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
             http://www.boredomandlaziness.org

From glingl at aon.at  Fri Jul 14 11:33:13 2006
From: glingl at aon.at (Gregor Lingl)
Date: Fri, 14 Jul 2006 11:33:13 +0200
Subject: [Python-Dev] IDLE - firewall warning
Message-ID: <44B764D8.5000101@aon.at>

I have posted the following message to idle-dev,
but no reply yet. Just a suggestion:

The firewall warning message in the Pythonshell window
was introduced  in Python 2.3 (IDLE 1.0 or something similar?)

I remember well the problems, to which it was the answer. 
(If I remember correctly I was involved in the discussion 
thread which led to the introduction of that message.)

Since more than three years I've used IDLE on many different
systems and I didn't encounter a single Firewall warning since
then. (It seems that firewalls nowadays are not offended by
the use of 127.0.0.1)

Therefore, and because the message is long and ugly, I'd like to
suggest to delete it from the Python Shell window
and - if considered necessary - for instance to put
it into the IDLE Help - "About IDLE" submenu or
in a special IDLE-Firewall warning submenu of IDLE-Help.

Please observe that it pops up thousands of times and it's
read at most once.

Or perhaps a single-line message like
=== subprocesses use internal loopback interface ===
would do it, just to signal where you are as does the
respective line  in  -n mode.?


Regards,
Gregor Lingl



From rasky at develer.com  Fri Jul 14 12:00:07 2006
From: rasky at develer.com (Giovanni Bajo)
Date: Fri, 14 Jul 2006 12:00:07 +0200
Subject: [Python-Dev] Community buildbots (was Re: User's complaints)
References: <20060713180322.29014.1145514133.divmod.quotient.35181@ohm><65ECC6D7-7E46-463C-92D9-37AF2BF15234@python.org><200607132121.27521.fdrake@acm.org><17591.7796.553081.425160@montanaro.dyndns.org><ee2a432c0607132255l2a06ac3avaf200283239424ee@mail.gmail.com><e97cq1$cmm$1@sea.gmane.org>
	<ee2a432c0607132339o1d35c402ib13884c85115856a@mail.gmail.com>
Message-ID: <09f901c6a72c$495f2690$12472597@bagio>

Neal Norwitz <nnorwitz at gmail.com> wrote:

>> a longer beta period gives *external* developers more time to catch
>> up, and results in less work for the end users.
>
> This is the part I don't get.  For the external developers, if they
> care about compatibility, why aren't they testing periodically,
> regardless of alpha/beta releases?

Because it is a cost, and I don't want to waste my time if the trunk is
unstable or whatnot. There is no official statement of the kind "all the real
development is done in branches, the trunk is always very stable, feel free to
grab it". Thus, we (external developers) assume that it's better to wait for
the first alpha or beta before starting doing any testing. Personally, I won't
touch something before the first beta: there are already enough known bugs when
the alphas are shipped that I prefer to wait a little bit more.

In my case, the beta period is too short. It's already a great luck if, during
the beta cycle, I'm not deep into some milestone or release cycle for my own
software. It might well happen that I have barely time to notice a Python beta
is out, but I can't afford spending any time on it because of time constraint.
By the time I'm done with my own deadlines, Python final is already out. But
for the sake of the argument and the rest of this mail, let's assume I have
tons of spare time to dedicate to Python 2.5 beta testing.

My applications have several external dependencies (I wouldn't classify those
as "many", but still around 6-7 libraries in average). For each of those
libraries, there won't be Py2.5-ready RPM or Windows installer to grab and
build. Most of the time, I have to download the source package, and try to
build it. This also means hunting down and fixing Py2.5-related bugs in all
those libraries *before* I can even boot my own application (which is what I
would care about). Then I occasionally hit a core Python bug while doing so,
which is good, but I have to sit and wait. Some libraries have very complex
build process which are still distutils-based, but might require many other
external C/C++ libraries, which need to be fetched and compiled just to setup
the build environment.

Alternatively, I could cross my fingers and wait for all the maintainers of the
external libraries to have spare time, and dedicate to Python 2.5 upgrading. If
I'm lucky, by the time RC1 is out, most of them might have binary packages
available for download, or at least have their (unstable) CVS/SVN trunk fixed
for Python 2.5 (which means that I'll have to fetch that unstable version and
basically perform a forced upgrade of the library, which might trigger another
class of compatibility/feature/regression bugs in my application, not related
at all to Python 2.5 by itself, but still needed to be dealt).

So I think it's useless to ask people to rush testing beta releases: it takes
months to get the community synched, and will always take. It's also useless to
point the finger to external developers if they don't test Python and there is
a bug in a .0 release. Bugs happen. It's software that evolves. My own
suggestion is that it's useless to stall a release for months to give external
developers time to fix things. I think it's much better to "release early -
release often", and just get the damn release out of the door. Usually, it's at
that point that the whole community start working on it, and discover bugs. And
so what? For production usage, .0 releases of libraries (let alone the
interpreter of the language) are a no-go for all software, and I know that for
a long time already. I don't ship an application of mine with a Python .0
release no matter what, no matter even if the whole testsuite passes and
everything seems to work. I don't have enough benefits for the risks, so I'll
wait for .1 or .2 release anyway. It's *very* fine by me if .0 release is
actually the first "official" "beta" release for the community, when the core
developers say "we're done" and external developers really start get things
going.

If you really care about .0 stability from a purely commercial point-of-view
(it's bad advertisement if many people complain if a major release is broken,
etc. etc.), I might suggest you to coordinate with a small group of selected
external developers maintaing the larger external packages (Twisted and
others). You could put a list of those packages in the release PEP as
showstoppers for the release: you should not get a .0 out if those packages are
broken. I think this could help smooth out the process. If important external
libraries work, many applications relying on it will *mostly* work as well. I
personally don't think it's such a big problem if one has to fix a couple of
things in a 100K-line application to adjust it to the new .0 release, even if
it's actually because of a bug in Python itself.

Giovanni Bajo


From ncoghlan at gmail.com  Fri Jul 14 12:09:02 2006
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 14 Jul 2006 20:09:02 +1000
Subject: [Python-Dev] Handling of sys.args (Re:  User's complaints)
In-Reply-To: <44B723C6.9010309@canterbury.ac.nz>
References: <ee2a432c0607041649i7ecb6cc1w8ebdf47f7d02097a@mail.gmail.com>	<3e1553560607122243q754dd71dq2e6d95cdf11e89dd@mail.gmail.com>	<4c45c1530607130037r1c14e3f6u972232bdfa481534@mail.gmail.com>	<Pine.LNX.4.58.0607130240480.31087@server1.LFW.org>	<44B60B5A.2080309@canterbury.ac.nz>	<Pine.LNX.4.58.0607131333290.31087@server1.LFW.org>
	<44B723C6.9010309@canterbury.ac.nz>
Message-ID: <44B76D3E.90503@gmail.com>

Greg Ewing wrote:
> Maybe sys needs to be split into two modules, with
> the non-sensitive one pre-imported (so that the
> importless interpreter you suggest wouldn't be
> unnecessarily crippled).

Maybe not splitting it, but providing a read-only mechanism of getting at 
certain elements, with "import sys" still being the way of *modifying* any of 
these things.

Relatively safe items (could be provided as attributes and methods of a 
read-only class instance in builtins instead of as a module):

   argv (as a tuple instead of a list)
   byteorder
   maxint
   maxunicode
   builtin_module_names
   copyright
   exc_info()
   exec_prefix
   executable
   exit([arg])
   getdefaultencoding()
   getfilesystemencoding()
   getwindowsversion()
   hexversion
   platform
   prefix
   stdin
   stdout
   stderr
   version
   version_info
   winver


Arguably privileged information (no real reason for non-privileged code to 
know this stuff):
   subversion
   getcheckinterval()
   getdlopenflags()
   dllhandle
   _current_frames()
   getrefcount(object)
   getrecursionlimit()
   _getframe([depth])
   __displayhook__
   __excepthook__
   __stdin__
   __stdout__
   __stderr__
   api_version
   warnoptions
   tracebacklimit
   displayhook(value)
   excepthook(type, value, traceback)
   ps1
   ps2


Definitely privileged operations:
   Actually *setting* any of the above to something different
   modules
   path
   exc_clear()
   setcheckinterval(interval)
   setdefaultencoding(name)
   setdlopenflags(n)
   setprofile(profilefunc)
   setrecursionlimit(limit)
   settrace(tracefunc)
   settscdump(on_flag)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
             http://www.boredomandlaziness.org

From ncoghlan at gmail.com  Fri Jul 14 12:45:14 2006
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 14 Jul 2006 20:45:14 +1000
Subject: [Python-Dev] Community buildbots (was Re: User's complaints)
In-Reply-To: <200607141714.09304.anthony@interlink.com.au>
References: <20060713180322.29014.1145514133.divmod.quotient.35181@ohm>	<e97cq1$cmm$1@sea.gmane.org>	<ee2a432c0607132339o1d35c402ib13884c85115856a@mail.gmail.com>
	<200607141714.09304.anthony@interlink.com.au>
Message-ID: <44B775BA.1070905@gmail.com>

Anthony Baxter wrote:
> On Friday 14 July 2006 16:39, Neal Norwitz wrote:
>> Remember I also tried to push for more features to go in early? 
>> That would have given more time for external testing.  Still
>> features are coming in.  Python developers weren't happy about
>> having to get things in earlier.  I don't see a practical way to
>> implement what you propose (see Anthony's comments).
> 
> Following up on this point: 
> Given the number of "just-this-one-more-thing-please" we've _already_ 
> had since the b1 feature freeze, do you really except that 90 days of 
> feature freeze is feasible? And if there's not going to be a feature 
> freeze, there's hardly any point to people doing testing until there 
> _is_ a feature freeze, is there? Oh, great, my code works with 2.5b1. 
> Oops. 2.5b9 added a new feature that broke my code, but I didn't test 
> with that. 

I think this is where release candidates can come into their own - the beta's 
can flush out the "just-one-more-thing" pleases, the maintenance branch is 
forked for rc1, and then any changes on the branch are purely to fix regressions.

As far as the idea of a suite of buildbots running the unit tests of large 
Python applications against the current SVN trunk goes, one problem is that 
failures in those buildbots will come from two sources:
   - Python itself fails (broken checkin)
   - the application unit tests fail (backwards incompatibility)

To weed out the false alarms, the slaves will need to be set up to run the 
Python unit tests first and then the application unit tests only if the Python 
test run is successful.

When the slave suffers a real failure due to a backwards incompatibility, it 
will take a developer of the application to figure out what it was that broke 
the application's tests.

So while I think it's a great idea, I also think it will need significant 
support from the application developers in debugging any buildbot failures to 
really make it work.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
             http://www.boredomandlaziness.org

From ncoghlan at gmail.com  Fri Jul 14 12:56:22 2006
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 14 Jul 2006 20:56:22 +1000
Subject: [Python-Dev] IDLE - firewall warning
In-Reply-To: <44B764D8.5000101@aon.at>
References: <44B764D8.5000101@aon.at>
Message-ID: <44B77856.8080106@gmail.com>

Gregor Lingl wrote:
> I have posted the following message to idle-dev,
> but no reply yet. Just a suggestion:
> 
> The firewall warning message in the Pythonshell window
> was introduced  in Python 2.3 (IDLE 1.0 or something similar?)
> 
> I remember well the problems, to which it was the answer. 
> (If I remember correctly I was involved in the discussion 
> thread which led to the introduction of that message.)
> 
> Since more than three years I've used IDLE on many different
> systems and I didn't encounter a single Firewall warning since
> then. (It seems that firewalls nowadays are not offended by
> the use of 127.0.0.1)

Unless that firewall is ZoneAlarm on Windows XP :)

Something that can easily happen to suppress the firewall warnings for IDLE is 
that someone runs a Python application and grants it access to the internet. 
For an executable file signature based outbound firewall like ZoneAlarm, this 
actually grants passage through the firewall to *all* Python applications, 
since it is the interpreter binary that gets registered (ZA has a similar lack 
of granularity when it comes to Java applications).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
             http://www.boredomandlaziness.org

From amk at amk.ca  Fri Jul 14 13:21:37 2006
From: amk at amk.ca (A.M. Kuchling)
Date: Fri, 14 Jul 2006 07:21:37 -0400
Subject: [Python-Dev] Community buildbots (was Re: User's complaints)
In-Reply-To: <09f901c6a72c$495f2690$12472597@bagio>
References: <ee2a432c0607132339o1d35c402ib13884c85115856a@mail.gmail.com>
	<09f901c6a72c$495f2690$12472597@bagio>
Message-ID: <20060714112137.GA891@Andrew-iBook2.local>

On Fri, Jul 14, 2006 at 12:00:07PM +0200, Giovanni Bajo wrote:
> unstable or whatnot. There is no official statement of the kind "all
> the real development is done in branches, the trunk is always very
> stable, feel free to grab it". Thus, we (external developers) assume
> that it's better to wait for the first alpha or beta before starting
> doing any testing. 

Where could we put such a statement?
http://www.python.org/dev/tools/, in a discussion of checkin policies,
does say:  

	The Python source tree is managed for stability, meaning that
	if you make a checkout at a random point in time the tree will almost
	always compile and be quite stable. Large sets of changes, such as the
	2.2 type/class rewrite, are usually tested on a branch first and
	merged into the main stream of development once they're believed to be
	complete.

but this is buried pretty deeply.  Maybe some text should be added to
the release announcements about this.

> I'm lucky, by the time RC1 is out, most of them might have binary packages
> available for download, or at least have their (unstable) CVS/SVN trunk fixed
> for Python 2.5 (which means that I'll have to fetch that unstable version and

And many people won't make binary packages until 2.5 is final, so
you're probably not often lucky.

--amk

From skip at pobox.com  Fri Jul 14 13:28:31 2006
From: skip at pobox.com (skip at pobox.com)
Date: Fri, 14 Jul 2006 06:28:31 -0500
Subject: [Python-Dev] Community buildbots
In-Reply-To: <20060714051157.29014.803759518.divmod.quotient.35955@ohm>
References: <17591.7500.168119.701223@montanaro.dyndns.org>
	<20060714051157.29014.803759518.divmod.quotient.35955@ohm>
Message-ID: <17591.32735.270029.593594@montanaro.dyndns.org>


    >> If you're concerned about noticing when a new release train is
    >> pulling out of the station ...

    glyph> I am aware of when new releases come out :).  What I'm not aware
    glyph> of is what features (may) have broken my code, and why.  

Hmmm...  Well, you could subscribe to the python-checkins mailing list.
Most of the time you'd probably decide rather quickly to delete the
messages, but the stuff you care about should be in the checkin comments.

Skip

From skip at pobox.com  Fri Jul 14 13:30:36 2006
From: skip at pobox.com (skip at pobox.com)
Date: Fri, 14 Jul 2006 06:30:36 -0500
Subject: [Python-Dev] Community buildbots (was Re: User's complaints)
In-Reply-To: <44B729A9.3060705@canterbury.ac.nz>
References: <20060713180322.29014.1145514133.divmod.quotient.35181@ohm>
	<65ECC6D7-7E46-463C-92D9-37AF2BF15234@python.org>
	<44B729A9.3060705@canterbury.ac.nz>
Message-ID: <17591.32860.610966.442590@montanaro.dyndns.org>


    Greg> Maybe there could be an "unstable" release phase that lasts for a
    Greg> whole release cycle. So you'd first release version 2.n as
    Greg> "unstable", and keep 2.(n-1) as the current "stable" release. Then
    Greg> when 2.(n+1) is ready, 2.n would become "stable" and 2.(n+1) would
    Greg> become the new "unstable".

In GCC don't they do an odd (stable)/even (unstable) release schedule?  Same
for Linux kernels?  Would that help?

Skip

From skip at pobox.com  Fri Jul 14 13:32:48 2006
From: skip at pobox.com (skip at pobox.com)
Date: Fri, 14 Jul 2006 06:32:48 -0500
Subject: [Python-Dev] Community buildbots (was Re: User's complaints)
In-Reply-To: <200607140139.38384.fdrake@acm.org>
References: <20060713180322.29014.1145514133.divmod.quotient.35181@ohm>
	<200607132121.27521.fdrake@acm.org>
	<17591.7796.553081.425160@montanaro.dyndns.org>
	<200607140139.38384.fdrake@acm.org>
Message-ID: <17591.32992.90867.191048@montanaro.dyndns.org>


    >> I believe there was some shortening of the 2.5 release cycle two or
    >> three months ago.

    Fred> The squeezing of the releases isn't where the problem is, I think.

Had we been in alpha a bit longer (officially before feature freeze) I think
there'd have been less back-and-forth about a couple otherwise
noncontroversial checkins.

Skip

From fredrik at pythonware.com  Fri Jul 14 13:37:28 2006
From: fredrik at pythonware.com (Fredrik Lundh)
Date: Fri, 14 Jul 2006 13:37:28 +0200
Subject: [Python-Dev] Community buildbots (was Re: User's complaints)
References: <20060713180322.29014.1145514133.divmod.quotient.35181@ohm>
	<20060713184255.GA3929@rogue.amk.ca>
Message-ID: <e97vmg$4m3$1@sea.gmane.org>

"A.M. Kuchling" wrote:
>> While the new python.org is very nice, I do note that there's no "blogs"
>> entry on the front page, something which has become a fixture on almost
>> every other website I visit regularly.
>
> A 'Blogs' link could be trivially added by linking to
> planet.python.org, though the blogs collected there are not in any way
> 'the Python developers', but a jumble of developers and users.  I
> don't think enough core developers have weblogs (or write about
> Python) to make a 'python-dev only' planet very useful.

on the other hand, the material you're producing for the "what's new" document
would be a rather excellent development blog in itself.  just post new sections to
the blog when they're ready...

(add PEP announcements and python-dev summary items to the mix, and you
have a high-quality development blog generated entirely from existing content)

(hmm.  maybe this could be put together by a robot?  time to start hacking on
"The Daily URL 2.0 Beta", I suppose...)

</F> 




From skip at pobox.com  Fri Jul 14 13:46:55 2006
From: skip at pobox.com (skip at pobox.com)
Date: Fri, 14 Jul 2006 06:46:55 -0500
Subject: [Python-Dev] Community buildbots (was Re: User's complaints)
In-Reply-To: <ee2a432c0607132339o1d35c402ib13884c85115856a@mail.gmail.com>
References: <20060713180322.29014.1145514133.divmod.quotient.35181@ohm>
	<65ECC6D7-7E46-463C-92D9-37AF2BF15234@python.org>
	<200607132121.27521.fdrake@acm.org>
	<17591.7796.553081.425160@montanaro.dyndns.org>
	<ee2a432c0607132255l2a06ac3avaf200283239424ee@mail.gmail.com>
	<e97cq1$cmm$1@sea.gmane.org>
	<ee2a432c0607132339o1d35c402ib13884c85115856a@mail.gmail.com>
Message-ID: <17591.33839.641795.66854@montanaro.dyndns.org>


    Neal> How often is the python build broken or otherwise unusable?  

Not very often.  I use the head branch of the repository as the source of
the interpreter I run on my laptop.  It generally takes just a couple
minutes on my now-aging PowerBook to svn up and reinstall.  I can only
recall one time in the past where I had to temporarily fall back to 2.4
because of some change that broke an application.

Admittedly, I'm not as sophisticated a user as Fredrik or Glyph, but I
suspect that my usage of the language isn't all that different from most
Python developers out there.

    Neal> Is part of your point that these developers only care about
    Neal> something called "release" and they won't start testing before
    Neal> then?  If that's the case why don't we start making
    Neal> (semi-)automated alpha releases every month?  

How would that be any easier than a user setting up a read-only repository
and svn-up-ing it once a month then using that as the default interpreter on
that person's development machine?  I maintain interpreters for 2.3, 2.4 and
bleeding edge at the moment.  If I need to it's fairly trivial (a symlink
change) to fall back to the latest stable release.

Glyph, would that sort of scheme work for you?

Skip

From skip at pobox.com  Fri Jul 14 13:49:24 2006
From: skip at pobox.com (skip at pobox.com)
Date: Fri, 14 Jul 2006 06:49:24 -0500
Subject: [Python-Dev] Community buildbots (was Re: User's complaints)
In-Reply-To: <e97fe2$jjc$1@sea.gmane.org>
References: <20060713180322.29014.1145514133.divmod.quotient.35181@ohm>
	<65ECC6D7-7E46-463C-92D9-37AF2BF15234@python.org>
	<200607132121.27521.fdrake@acm.org>
	<17591.7796.553081.425160@montanaro.dyndns.org>
	<ee2a432c0607132255l2a06ac3avaf200283239424ee@mail.gmail.com>
	<e97cq1$cmm$1@sea.gmane.org>
	<ee2a432c0607132339o1d35c402ib13884c85115856a@mail.gmail.com>
	<e97fe2$jjc$1@sea.gmane.org>
Message-ID: <17591.33988.996995.204882@montanaro.dyndns.org>


    >> This is the part I don't get.  For the external developers, if they
    >> care about compatibility, why aren't they testing periodically,
    >> regardless of alpha/beta releases?

    Fredrik> because they don't want to waste time and effort on testing
    Fredrik> against releases that are not necessarily a close approximation
    Fredrik> of the full release?  testing takes time, and time can be used
    Fredrik> for many different things.

How is that a waste of time?  You've got the latest stable 2.4 installed I
presume.  Do the svn-up/make dance (at least on Unix-y systems - can it be
much more difficult on Windows?).  If it breaks something, either figure out
why or drop back to 2.4 for a week or two and try again.  If application
breakage remains, investigate.

Skip

From bborcic at gmail.com  Fri Jul 14 14:38:22 2006
From: bborcic at gmail.com (Boris Borcic)
Date: Fri, 14 Jul 2006 14:38:22 +0200
Subject: [Python-Dev] Python Style Sheets ? Re: User's complaints
In-Reply-To: <ca471dc20607130720q3fe1b363y533a7bff46579174@mail.gmail.com>
References: <ee2a432c0607041649i7ecb6cc1w8ebdf47f7d02097a@mail.gmail.com>	<3e1553560607122243q754dd71dq2e6d95cdf11e89dd@mail.gmail.com>
	<ca471dc20607130720q3fe1b363y533a7bff46579174@mail.gmail.com>
Message-ID: <e9838o$fpc$1@sea.gmane.org>

Guido van Rossum wrote:
...
> 
> This is an illustration of the dilemma of maintaining a popular
> language: Everybody hates change (me too!) but everybody also has one
> thing that's bothering them so much they absolutely want it to be
> changed. If you were to implement all those personal pet peeves, you'd
> get a language that's more different from Python than Python is from
> Fortran.
> 
> So where's the middle ground?

I feel some freedom could be reclaimed with a solution in the spirit of Turing 
equivalence. Or, to take a less grandiose comparison, web style sheets - 
separation of content and presentation.

Suppose the standard required a (possibly empty) style-defining file prefix that 
constrains the python source code in the file, and concurrently defined (mostly) 
reversible and transparent source-to-source transforms that would map any source 
code file to an equivalent source code file with an arbitrary chosen prefix. 
Then users could chose their style of Python and either transform all source 
files they install to their own style, or setup their editor to do it 
back-and-forth for them. The choice of python presentation style would then 
become a private choice.

To illustrate the idea, this already exists in very embryonic form with unicode 
encoding modelines. The current standard allows to imagine a Python editor that 
would permit to set a "local standard encoding modeline" and then present any 
source file as if it had been written while taking maximal profit from the 
chosen encoding. Which may also be simple ascii.

Cheers, BB
--
"C++ is a contradiction in terms" - Lorentz, Einstein, Poincar?


From jeremy at alum.mit.edu  Fri Jul 14 14:45:04 2006
From: jeremy at alum.mit.edu (Jeremy Hylton)
Date: Fri, 14 Jul 2006 08:45:04 -0400
Subject: [Python-Dev] Community buildbots (was Re: User's complaints)
In-Reply-To: <200607141714.09304.anthony@interlink.com.au>
References: <20060713180322.29014.1145514133.divmod.quotient.35181@ohm>
	<e97cq1$cmm$1@sea.gmane.org>
	<ee2a432c0607132339o1d35c402ib13884c85115856a@mail.gmail.com>
	<200607141714.09304.anthony@interlink.com.au>
Message-ID: <e8bf7a530607140545m6953c0c6r2a6969f716276549@mail.gmail.com>

On 7/14/06, Anthony Baxter <anthony at interlink.com.au> wrote:
> On Friday 14 July 2006 16:39, Neal Norwitz wrote:
> > Remember I also tried to push for more features to go in early?
> > That would have given more time for external testing.  Still
> > features are coming in.  Python developers weren't happy about
> > having to get things in earlier.  I don't see a practical way to
> > implement what you propose (see Anthony's comments).
>
> Following up on this point:
> Given the number of "just-this-one-more-thing-please" we've _already_
> had since the b1 feature freeze, do you really except that 90 days of
> feature freeze is feasible? And if there's not going to be a feature
> freeze, there's hardly any point to people doing testing until there
> _is_ a feature freeze, is there? Oh, great, my code works with 2.5b1.
> Oops. 2.5b9 added a new feature that broke my code, but I didn't test
> with that.

Maybe the basic question is right, but the emphasis needs to be
changed.  If we had a rule that said the final release was 90 days
after the last submission that wasn't to fix a regression, we'd ask
"Is this feature important enough to warrant delaying the release
until three months from now?"  I'm not sure what I think, but it
doesn't seem like an implausible policy.

Jeremy

>
> Anthony
> --
> Anthony Baxter     <anthony at interlink.com.au>
> It's never too late to have a happy childhood.
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/jeremy%40alum.mit.edu
>

From jeremy at alum.mit.edu  Fri Jul 14 14:48:07 2006
From: jeremy at alum.mit.edu (Jeremy Hylton)
Date: Fri, 14 Jul 2006 08:48:07 -0400
Subject: [Python-Dev] Community buildbots (was Re: User's complaints)
In-Reply-To: <44B775BA.1070905@gmail.com>
References: <20060713180322.29014.1145514133.divmod.quotient.35181@ohm>
	<e97cq1$cmm$1@sea.gmane.org>
	<ee2a432c0607132339o1d35c402ib13884c85115856a@mail.gmail.com>
	<200607141714.09304.anthony@interlink.com.au>
	<44B775BA.1070905@gmail.com>
Message-ID: <e8bf7a530607140548m5de298efs7a31f8948d462b38@mail.gmail.com>

On 7/14/06, Nick Coghlan <ncoghlan at gmail.com> wrote:
> Anthony Baxter wrote:
> > On Friday 14 July 2006 16:39, Neal Norwitz wrote:
> >> Remember I also tried to push for more features to go in early?
> >> That would have given more time for external testing.  Still
> >> features are coming in.  Python developers weren't happy about
> >> having to get things in earlier.  I don't see a practical way to
> >> implement what you propose (see Anthony's comments).
> >
> > Following up on this point:
> > Given the number of "just-this-one-more-thing-please" we've _already_
> > had since the b1 feature freeze, do you really except that 90 days of
> > feature freeze is feasible? And if there's not going to be a feature
> > freeze, there's hardly any point to people doing testing until there
> > _is_ a feature freeze, is there? Oh, great, my code works with 2.5b1.
> > Oops. 2.5b9 added a new feature that broke my code, but I didn't test
> > with that.
>
> I think this is where release candidates can come into their own - the beta's
> can flush out the "just-one-more-thing" pleases, the maintenance branch is
> forked for rc1, and then any changes on the branch are purely to fix regressions.
>
> As far as the idea of a suite of buildbots running the unit tests of large
> Python applications against the current SVN trunk goes, one problem is that
> failures in those buildbots will come from two sources:
>    - Python itself fails (broken checkin)
>    - the application unit tests fail (backwards incompatibility)
>
> To weed out the false alarms, the slaves will need to be set up to run the
> Python unit tests first and then the application unit tests only if the Python
> test run is successful.
>
> When the slave suffers a real failure due to a backwards incompatibility, it
> will take a developer of the application to figure out what it was that broke
> the application's tests.
>
> So while I think it's a great idea, I also think it will need significant
> support from the application developers in debugging any buildbot failures to
> really make it work.

These buildbots should run the tests from stable, released versions of
external packages, assuming that those packages don't ship releases
with failing tests.  If you ran the test suite for a Zope release and
had a test failure, I think there would be a reasonable expectation
that it was a Python bug.

I'd definitely avoiding mixing a development version of Python and a
development version of the test software in the automated build
environment.

Jeremy

>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
> ---------------------------------------------------------------
>              http://www.boredomandlaziness.org
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/jeremy%40alum.mit.edu
>

From rasky at develer.com  Fri Jul 14 15:06:06 2006
From: rasky at develer.com (Giovanni Bajo)
Date: Fri, 14 Jul 2006 15:06:06 +0200
Subject: [Python-Dev] Community buildbots (was Re: User's complaints)
References: <20060713180322.29014.1145514133.divmod.quotient.35181@ohm><65ECC6D7-7E46-463C-92D9-37AF2BF15234@python.org><44B729A9.3060705@canterbury.ac.nz>
	<17591.32860.610966.442590@montanaro.dyndns.org>
Message-ID: <015d01c6a746$44808320$d503030a@trilan>

skip at pobox.com wrote:

>     Greg> Maybe there could be an "unstable" release phase that lasts
>     for a Greg> whole release cycle. So you'd first release version
>     2.n as Greg> "unstable", and keep 2.(n-1) as the current "stable"
>     release. Then Greg> when 2.(n+1) is ready, 2.n would become
>     "stable" and 2.(n+1) would Greg> become the new "unstable".
>
> In GCC don't they do an odd (stable)/even (unstable) release
> schedule?  Same for Linux kernels?  Would that help?

No. Linux kernel releases are more aggressive because of the fact that all
the patches are mostly developed in different branch/repositories and get
merged when they are already well tested and incorporated. Linus can merge
literally hundreds of patches daily into his *stable* tree, and do releases
from it even weekly, because most destabilizing works are being done in
large branches carried on for months before they even are evaluated for
being merged; or because patches were settled in the -mm tree for months.
Linus' tree is kind-of a release branch, with the difference that he is the
BDFL and does what he wants with his tree :) To keep this into perspective,
remember also that they don't have *any* kind of testsuite (nor a debugger,
if I might say).

GCC has a more "old-fashioned" release process, where the trunk evolves
through 3 stages: Stage 1 is open for all kind of changes (eg: from simple
polishing/refactoring, to merging of large branches containing work of
several man-years). Stage 2 is still open for new features, but not for big
merges. Stage 3 is feature-freezed, bug-fixing only. Then, the trunk is
branched into the new release branch, and the trunk gets back to Stage 1.
Nominally, a stage lasts 3 months, but Stage 3 often stretches up to 6
months.

The release branches are open for *only* regression fixes (that is, fixes
that correct things that used to work in previous releases but do not work
anymore). Any regression fix (with a corresponding Bugzilla entry, where
it's marked and confirmed as "regression") is applied to trunk *and* the
open release branches where the regression exists. For convoluted or
dangerous regression fixes, usually maintainers prefer to wait 1 week for
the patch to settle down on the trunk before applying it to the release
branches. The release manager pulls dot releases from the release branch.
Usually, the first release (.0) happens 3-4 months *after* the release
branch was created, that is after several months of regression-only patches
being merged to it (while new work is being done on the trunk in parallel,
in its aggressive Stage 1).

The 3-Stage work in the trunk is streamlined by the release manager. At the
beginning of Stage 1, a detailed techinical list of on-going "projects" (new
features) is presented to the mailing list, explaining the current status of
the work and its ETA, and the release manager then publishes a work-plan for
Stage 1 and 2, telling which projects will be merged when. This avoids
multiple large projects to hit the trunk at the same time, causing may
headaches to all the other developers. The work-plan is managed and updated
in the GCC Wiki (which is off-line right now, but I'll post a link as
example when it's back).
-- 
Giovanni Bajo


From amk at amk.ca  Fri Jul 14 15:26:54 2006
From: amk at amk.ca (A.M. Kuchling)
Date: Fri, 14 Jul 2006 09:26:54 -0400
Subject: [Python-Dev] Community buildbots (was Re: User's complaints)
In-Reply-To: <e97vmg$4m3$1@sea.gmane.org>
References: <20060713180322.29014.1145514133.divmod.quotient.35181@ohm>
	<20060713184255.GA3929@rogue.amk.ca> <e97vmg$4m3$1@sea.gmane.org>
Message-ID: <20060714132654.GA4858@localhost.localdomain>

On Fri, Jul 14, 2006 at 01:37:28PM +0200, Fredrik Lundh wrote:
> (add PEP announcements and python-dev summary items to the mix, and you
> have a high-quality development blog generated entirely from existing content)
> (hmm.  maybe this could be put together by a robot?  time to start hacking on
> "The Daily URL 2.0 Beta", I suppose...)

There's already an RSS feed for the python-dev summaries, so that
could go into an aggregator.  There isn't a feed for the PEPs, but
those events are rarer and could be handled manually.

--amk

From bborcic at gmail.com  Fri Jul 14 15:46:26 2006
From: bborcic at gmail.com (Boris Borcic)
Date: Fri, 14 Jul 2006 15:46:26 +0200
Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?)
In-Reply-To: <e8bf7a530607130914w6c2abe95r78aaa7d79b7cf9d8@mail.gmail.com>
References: <e8k58c$q7u$1@sea.gmane.org>	<r01050400-1039-AEFC2D140DA011DBB8DF001124365170@10.0.0.24>	<e8ldjg$eq7$1@sea.gmane.org>
	<e90j6o$fij$1@sea.gmane.org>	<e93s6t$5fu$1@sea.gmane.org>
	<e8bf7a530607130914w6c2abe95r78aaa7d79b7cf9d8@mail.gmail.com>
Message-ID: <e9878p$t8r$1@sea.gmane.org>


[Fredrik Lundh]
>>>>
>>>>       def counter(num):
>>>>           num = mutable_int(num)
>>>>           def inc():
>>>>               num += 1
>>>>               return num
>>>>           return inc
>>
>> feel free to replace that += with an .add(1) method call; the point
>> wasn't the behaviour of augmented assigment, the point was that that the
>> most common use pattern involves *mutation* of the target object.
>>
>> the syntax isn't that important, really.

No it isn't, but I believe it deserves notice that it's easy enough, currently, 
to mock your mutable_int replacing the "+=" by a simple "+", while the "+=" 
itself is not possible even though it reflects the intention more precisely.

[Jeremy Hylton]
> 
> Mutation is different from rebinding.A tuple is immutable, but you
> can rebind the variable that refers to the tuple. 

It's more than just rebinding, it's rebinding to a function of the currently 
bound value. And /that/ has clear features in common with mutation (witness most 
of the current semantics and implementation of augmented assignments, ie, how 
much and far it manages to postpone the decision as to which is actually the case).

 > I think we will
> confuse users if we use the term mutation to refer to name binding.
> Name binding is already a subtle issue, so I think the risk is
> significant.

I'd tend to agree, but imho there is a complementary risk of propagating 
confusion by denying what augmented assignments clearly show : that rebinding to 
a function of the currently bound value has so much in common with mutation that 
augmented assignments allow to confuse both cases in source code. Except for the 
scoping issue under discussion, that is.

Cheers, BB
--
"C++ is a contradiction in terms" - Lorentz, Einstein, Poincar?


From theller at python.net  Fri Jul 14 17:09:37 2006
From: theller at python.net (Thomas Heller)
Date: Fri, 14 Jul 2006 17:09:37 +0200
Subject: [Python-Dev] Partial support for dlmodule.c in 64-bits OSes
In-Reply-To: <loom.20060713T215347-860@post.gmane.org>
References: <loom.20060713T215347-860@post.gmane.org>
Message-ID: <e98c3h$edl$1@sea.gmane.org>

Pierre Baillargeon schrieb:
> Currently, many 64-bits Oses cannot uses the dlmodule due to the conflicts
> between the sizes of int, long and char *. That is well. The check is made as
> run-time, which is also very well.
> 
> The problem is that the Python configuration script (setup.py) also makes the
> check and plainly excludes dlmodule.c from being built and deployed. That is not
> so well.
> 
> The reason is that we use the dlmodule solely to get access to the various flags
> (RTLD_NOW, RTLD_GLOBAL, etc), so that we can do some magic with loaded shared
> libraries, such as over-ridding the import mechanism so that the default load
> flags get changed (via sys.setdlopenflags()) to force some semantics.
> 
> Currently this doesn't work on most 64-bits OSes because the dl module doesn't
> exists, so it cannot be imported and its RTLD_* symbols are not accessible.
> 
> So I ask if it would be possible that the test for sys.maxint == 0x7fffffff in
> setup.py be dropped in future releases.

I don't know if your patch is acceptable or not, but if it is applied Lib/test/test_dl.py
crashes on 64-bit platforms, so this must be changed as well.  Further, patches should
be uploaded to the SF tracker so they don't get lost or forgotten.

OTOH, the RTLD_ constants are possibly available in the DLFCN module, as the documentation
explains.

Thomas


From ncoghlan at gmail.com  Fri Jul 14 17:12:29 2006
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 15 Jul 2006 01:12:29 +1000
Subject: [Python-Dev] Community buildbots (was Re: User's complaints)
In-Reply-To: <e8bf7a530607140548m5de298efs7a31f8948d462b38@mail.gmail.com>
References: <20060713180322.29014.1145514133.divmod.quotient.35181@ohm>	
	<e97cq1$cmm$1@sea.gmane.org>	
	<ee2a432c0607132339o1d35c402ib13884c85115856a@mail.gmail.com>	
	<200607141714.09304.anthony@interlink.com.au>	
	<44B775BA.1070905@gmail.com>
	<e8bf7a530607140548m5de298efs7a31f8948d462b38@mail.gmail.com>
Message-ID: <44B7B45D.1070709@gmail.com>

Jeremy Hylton wrote:
>> When the slave suffers a real failure due to a backwards 
>> incompatibility, it
>> will take a developer of the application to figure out what it was 
>> that broke
>> the application's tests.
>>
>> So while I think it's a great idea, I also think it will need significant
>> support from the application developers in debugging any buildbot 
>> failures to
>> really make it work.
> 
> These buildbots should run the tests from stable, released versions of
> external packages, assuming that those packages don't ship releases
> with failing tests.  If you ran the test suite for a Zope release and
> had a test failure, I think there would be a reasonable expectation
> that it was a Python bug.

Definitely, but there's a difference between "bug that broke Python's own unit 
tests" and "change to a behaviour that package X depended on".

It's the latter cases that the external buildbots would be looking for - and 
while some of those will be shallow enough that the regression is obvious from 
the unit test error message and the recent Python checkins, the non-obvious 
ones will require a collaborative resolution.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
             http://www.boredomandlaziness.org

From glyph at divmod.com  Fri Jul 14 17:26:36 2006
From: glyph at divmod.com (glyph at divmod.com)
Date: Fri, 14 Jul 2006 11:26:36 -0400
Subject: [Python-Dev] Community buildbots (was Re: User's complaints)
In-Reply-To: <17591.33839.641795.66854@montanaro.dyndns.org>
Message-ID: <20060714152636.29014.1693386533.divmod.quotient.36584@ohm>

On Fri, 14 Jul 2006 06:46:55 -0500, skip at pobox.com wrote:
>
>    Neal> How often is the python build broken or otherwise unusable?
>
>Not very often.

I have to agree.  The effort I'm talking about is not in fixing large numbers
of problems, but simply gearing up to properly test to see if there *are*
problems.  Keep in mind though: just because the problems are small or easy
to fix doesn't mean they're not severe.  One tiny bug can prevent a program
from even starting up.

>Admittedly, I'm not as sophisticated a user as Fredrik or Glyph, but I
>suspect that my usage of the language isn't all that different from most
>Python developers out there.

A huge percentage of Python developers are working with Zope, which means that
although *their* code might not be terribly "sophisticated", it is
nevertheless sitting on top of a rather large and intricate pile of implicit
dependencies on interpreter behavior.

>    Neal> Is part of your point that these developers only care about
>    Neal> something called "release" and they won't start testing before
>    Neal> then?  If that's the case why don't we start making
>    Neal> (semi-)automated alpha releases every month?

>How would that be any easier than a user setting up a read-only repository
>and svn-up-ing it once a month then using that as the default interpreter on
>that person's development machine?  I maintain interpreters for 2.3, 2.4 and
>bleeding edge at the moment.  If I need to it's fairly trivial (a symlink
>change) to fall back to the latest stable release.

>Glyph, would that sort of scheme work for you?

No.

I really think you're underestimating the sort of effort required to upgrade
Python.

First of all, I do have a job :) and it'd be very hard to make the case to an
investor that it was worth tracking down every potential bug I had to
determine whether it was a problem because I was working with an unreleased
version of Python.  This is the same reason I don't use beta versions of the
kernel or libc for development.

For that matter, I've had to avoid posting to this mailing list because even
*this* is stretching my already overburdened schedule :).

Secondly, I have to work with a few extension modules.  To name a few:
  * ctypes
  * PyPAM
  * pysqlite2
  * zope.interface (C optimizations)
  * xapian
  * pylucene
  * dspam
  * PyOpenSSL
  * PIL
  * PyCrypto
  * pyexpat
  * pygtk
  * pyvte

Recompiling all of these is a project that takes a day.  PyLucene, in fact,
I've never managed to build myself, and I can only run because there happen
to be debian packages which work (with some fiddling) on Ubuntu.  There's no
interactive warning during 'svn up' that it's time to recompile everything,
either, so I don't even know when the ABIs are going to have changed.

Even if everything works perfectly, and were perfectly automated, the compile 
itself would take a few hours.

I also test work on Windows on occasion and recompiling these things _there_
is the work of a week and a half, not to mention it requires that I be sitting
at the one machine where I have my Microsoft? Developer? Tools? installed.

I made the buildbot recommendation specifically because it would centralize
the not inconsiderable effort of integrating these numerous dependencies
(presuming that I could submit a Divmod buildbot as well as a Twisted one).

From glyph at divmod.com  Fri Jul 14 17:27:41 2006
From: glyph at divmod.com (glyph at divmod.com)
Date: Fri, 14 Jul 2006 11:27:41 -0400
Subject: [Python-Dev] Community buildbots (was Re: User's complaints)
In-Reply-To: <ee2a432c0607132339o1d35c402ib13884c85115856a@mail.gmail.com>
Message-ID: <20060714152741.29014.392313751.divmod.quotient.36585@ohm>

On Thu, 13 Jul 2006 23:39:06 -0700, Neal Norwitz <nnorwitz at gmail.com> wrote:
>On 7/13/06, Fredrik Lundh <fredrik at pythonware.com> wrote:

>> a longer beta period gives *external* developers more time to catch up,
>> and results in less work for the end users.

>This is the part I don't get.  For the external developers, if they
>care about compatibility, why aren't they testing periodically,
>regardless of alpha/beta releases?  How often is the python build
>broken or otherwise unusable?

How often do you test new builds of Python against the most recent alpha of,
e.g. the Linux kernel?  This isn't just a hypothetical question: Twisted has
broken because of changes to Linux as often as it has broken due to changes in
Python :).  In Linux's case we're all lucky because *any* regressions with
existing software are considered bugs, whereas in Python's case, some breakagaes
are considered acceptable since it's more feasible to have multiple builds of
Python installed more than multiple kernels for different applications.

From jcarlson at uci.edu  Fri Jul 14 18:21:47 2006
From: jcarlson at uci.edu (Josiah Carlson)
Date: Fri, 14 Jul 2006 09:21:47 -0700
Subject: [Python-Dev] IDLE - firewall warning
In-Reply-To: <44B764D8.5000101@aon.at>
References: <44B764D8.5000101@aon.at>
Message-ID: <20060714091004.0194.JCARLSON@uci.edu>


Gregor Lingl <glingl at aon.at> wrote:
> I have posted the following message to idle-dev,
> but no reply yet. Just a suggestion:
> 
> The firewall warning message in the Pythonshell window
> was introduced  in Python 2.3 (IDLE 1.0 or something similar?)

Speaking of 'features that would have been nice to get into 2.5'; had we
gotten the asynchronous subprocess support in, we could have done
away with the socket support, though it would have required a bit of
modification to the XML-RPC implementation.

 - Josiah


From steven.bethard at gmail.com  Fri Jul 14 18:24:56 2006
From: steven.bethard at gmail.com (Steven Bethard)
Date: Fri, 14 Jul 2006 10:24:56 -0600
Subject: [Python-Dev] Community buildbots (was Re: User's complaints)
In-Reply-To: <20060714132654.GA4858@localhost.localdomain>
References: <20060713180322.29014.1145514133.divmod.quotient.35181@ohm>
	<20060713184255.GA3929@rogue.amk.ca> <e97vmg$4m3$1@sea.gmane.org>
	<20060714132654.GA4858@localhost.localdomain>
Message-ID: <d11dcfba0607140924m4ada781agb66ff1466aef4a8d@mail.gmail.com>

On 7/14/06, A.M. Kuchling <amk at amk.ca> wrote:
> On Fri, Jul 14, 2006 at 01:37:28PM +0200, Fredrik Lundh wrote:
> > (add PEP announcements and python-dev summary items to the mix, and you
> > have a high-quality development blog generated entirely from existing content)
> > (hmm.  maybe this could be put together by a robot?  time to start hacking on
> > "The Daily URL 2.0 Beta", I suppose...)
>
> There's already an RSS feed for the python-dev summaries, so that
> could go into an aggregator.  There isn't a feed for the PEPs, but
> those events are rarer and could be handled manually.

I believe that's broken at the moment, at least in the sense that it's
no longer updated:
    http://psf.pollenation.net/cgi-bin/trac.cgi/ticket/366
I don't have enough time right now to figure out the new website and
how to fix it.  Maybe when I get back from Sydney in a couple of
weeks.

Steve
-- 
I'm not *in*-sane. Indeed, I am so far *out* of sane that you appear a
tiny blip on the distant coast of sanity.
        --- Bucky Katt, Get Fuzzy

From guido at python.org  Fri Jul 14 21:44:21 2006
From: guido at python.org (Guido van Rossum)
Date: Fri, 14 Jul 2006 12:44:21 -0700
Subject: [Python-Dev] Python Style Sheets ? Re: User's complaints
In-Reply-To: <e9838o$fpc$1@sea.gmane.org>
References: <ee2a432c0607041649i7ecb6cc1w8ebdf47f7d02097a@mail.gmail.com>
	<3e1553560607122243q754dd71dq2e6d95cdf11e89dd@mail.gmail.com>
	<ca471dc20607130720q3fe1b363y533a7bff46579174@mail.gmail.com>
	<e9838o$fpc$1@sea.gmane.org>
Message-ID: <ca471dc20607141244x5c075c6bnd795493101cf19dd@mail.gmail.com>

You must be misunderstanding. The root problem is that people
(rightly) complain that the language changes too much. And you want to
"fix" this by adding a deep and fundamental change to the language?
What planet are you from? It reminds me of Jini, which was presented
as a new standard to address the problem of too many conflicting
standard. Get it? :-)

--Guido

On 7/14/06, Boris Borcic <bborcic at gmail.com> wrote:
> Guido van Rossum wrote:
> ...
> >
> > This is an illustration of the dilemma of maintaining a popular
> > language: Everybody hates change (me too!) but everybody also has one
> > thing that's bothering them so much they absolutely want it to be
> > changed. If you were to implement all those personal pet peeves, you'd
> > get a language that's more different from Python than Python is from
> > Fortran.
> >
> > So where's the middle ground?
>
> I feel some freedom could be reclaimed with a solution in the spirit of Turing
> equivalence. Or, to take a less grandiose comparison, web style sheets -
> separation of content and presentation.
>
> Suppose the standard required a (possibly empty) style-defining file prefix that
> constrains the python source code in the file, and concurrently defined (mostly)
> reversible and transparent source-to-source transforms that would map any source
> code file to an equivalent source code file with an arbitrary chosen prefix.
> Then users could chose their style of Python and either transform all source
> files they install to their own style, or setup their editor to do it
> back-and-forth for them. The choice of python presentation style would then
> become a private choice.
>
> To illustrate the idea, this already exists in very embryonic form with unicode
> encoding modelines. The current standard allows to imagine a Python editor that
> would permit to set a "local standard encoding modeline" and then present any
> source file as if it had been written while taking maximal profit from the
> chosen encoding. Which may also be simple ascii.
>
> Cheers, BB
> --
> "C++ is a contradiction in terms" - Lorentz, Einstein, Poincar?
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org
>


-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From guido at python.org  Fri Jul 14 21:48:56 2006
From: guido at python.org (Guido van Rossum)
Date: Fri, 14 Jul 2006 12:48:56 -0700
Subject: [Python-Dev] Handling of sys.args (Re: User's complaints)
In-Reply-To: <44B76D3E.90503@gmail.com>
References: <ee2a432c0607041649i7ecb6cc1w8ebdf47f7d02097a@mail.gmail.com>
	<3e1553560607122243q754dd71dq2e6d95cdf11e89dd@mail.gmail.com>
	<4c45c1530607130037r1c14e3f6u972232bdfa481534@mail.gmail.com>
	<Pine.LNX.4.58.0607130240480.31087@server1.LFW.org>
	<44B60B5A.2080309@canterbury.ac.nz>
	<Pine.LNX.4.58.0607131333290.31087@server1.LFW.org>
	<44B723C6.9010309@canterbury.ac.nz> <44B76D3E.90503@gmail.com>
Message-ID: <ca471dc20607141248i5143b747lb8f3cbc9a97b703c@mail.gmail.com>

Whoa, whoa. What's the *problem* we're trying to solve here?

On 7/14/06, Nick Coghlan <ncoghlan at gmail.com> wrote:
> Greg Ewing wrote:
> > Maybe sys needs to be split into two modules, with
> > the non-sensitive one pre-imported (so that the
> > importless interpreter you suggest wouldn't be
> > unnecessarily crippled).
>
> Maybe not splitting it, but providing a read-only mechanism of getting at
> certain elements, with "import sys" still being the way of *modifying* any of
> these things.
>
> Relatively safe items (could be provided as attributes and methods of a
> read-only class instance in builtins instead of as a module):
>
>    argv (as a tuple instead of a list)
>    byteorder
>    maxint
>    maxunicode
>    builtin_module_names
>    copyright
>    exc_info()
>    exec_prefix
>    executable
>    exit([arg])
>    getdefaultencoding()
>    getfilesystemencoding()
>    getwindowsversion()
>    hexversion
>    platform
>    prefix
>    stdin
>    stdout
>    stderr
>    version
>    version_info
>    winver
>
>
> Arguably privileged information (no real reason for non-privileged code to
> know this stuff):
>    subversion
>    getcheckinterval()
>    getdlopenflags()
>    dllhandle
>    _current_frames()
>    getrefcount(object)
>    getrecursionlimit()
>    _getframe([depth])
>    __displayhook__
>    __excepthook__
>    __stdin__
>    __stdout__
>    __stderr__
>    api_version
>    warnoptions
>    tracebacklimit
>    displayhook(value)
>    excepthook(type, value, traceback)
>    ps1
>    ps2
>
>
> Definitely privileged operations:
>    Actually *setting* any of the above to something different
>    modules
>    path
>    exc_clear()
>    setcheckinterval(interval)
>    setdefaultencoding(name)
>    setdlopenflags(n)
>    setprofile(profilefunc)
>    setrecursionlimit(limit)
>    settrace(tracefunc)
>    settscdump(on_flag)
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
> ---------------------------------------------------------------
>              http://www.boredomandlaziness.org
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org
>


-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From brett at python.org  Fri Jul 14 23:30:17 2006
From: brett at python.org (Brett Cannon)
Date: Fri, 14 Jul 2006 14:30:17 -0700
Subject: [Python-Dev] Handling of sys.args (Re: User's complaints)
In-Reply-To: <ca471dc20607141248i5143b747lb8f3cbc9a97b703c@mail.gmail.com>
References: <ee2a432c0607041649i7ecb6cc1w8ebdf47f7d02097a@mail.gmail.com>
	<3e1553560607122243q754dd71dq2e6d95cdf11e89dd@mail.gmail.com>
	<4c45c1530607130037r1c14e3f6u972232bdfa481534@mail.gmail.com>
	<Pine.LNX.4.58.0607130240480.31087@server1.LFW.org>
	<44B60B5A.2080309@canterbury.ac.nz>
	<Pine.LNX.4.58.0607131333290.31087@server1.LFW.org>
	<44B723C6.9010309@canterbury.ac.nz> <44B76D3E.90503@gmail.com>
	<ca471dc20607141248i5143b747lb8f3cbc9a97b703c@mail.gmail.com>
Message-ID: <bbaeab100607141430q34a516b4s1d6a8495c482bfc@mail.gmail.com>

On 7/14/06, Guido van Rossum <guido at python.org> wrote:
>
> Whoa, whoa. What's the *problem* we're trying to solve here?


I have a use case for sandboxing.  I am already having to plan to have a
mini-sys module in a sandbox so that they cannot get access to dangerous
things.

-Brett

On 7/14/06, Nick Coghlan <ncoghlan at gmail.com> wrote:
> > Greg Ewing wrote:
> > > Maybe sys needs to be split into two modules, with
> > > the non-sensitive one pre-imported (so that the
> > > importless interpreter you suggest wouldn't be
> > > unnecessarily crippled).
> >
> > Maybe not splitting it, but providing a read-only mechanism of getting
> at
> > certain elements, with "import sys" still being the way of *modifying*
> any of
> > these things.
> >
> > Relatively safe items (could be provided as attributes and methods of a
> > read-only class instance in builtins instead of as a module):
> >
> >    argv (as a tuple instead of a list)
> >    byteorder
> >    maxint
> >    maxunicode
> >    builtin_module_names
> >    copyright
> >    exc_info()
> >    exec_prefix
> >    executable
> >    exit([arg])
> >    getdefaultencoding()
> >    getfilesystemencoding()
> >    getwindowsversion()
> >    hexversion
> >    platform
> >    prefix
> >    stdin
> >    stdout
> >    stderr
> >    version
> >    version_info
> >    winver
> >
> >
> > Arguably privileged information (no real reason for non-privileged code
> to
> > know this stuff):
> >    subversion
> >    getcheckinterval()
> >    getdlopenflags()
> >    dllhandle
> >    _current_frames()
> >    getrefcount(object)
> >    getrecursionlimit()
> >    _getframe([depth])
> >    __displayhook__
> >    __excepthook__
> >    __stdin__
> >    __stdout__
> >    __stderr__
> >    api_version
> >    warnoptions
> >    tracebacklimit
> >    displayhook(value)
> >    excepthook(type, value, traceback)
> >    ps1
> >    ps2
> >
> >
> > Definitely privileged operations:
> >    Actually *setting* any of the above to something different
> >    modules
> >    path
> >    exc_clear()
> >    setcheckinterval(interval)
> >    setdefaultencoding(name)
> >    setdlopenflags(n)
> >    setprofile(profilefunc)
> >    setrecursionlimit(limit)
> >    settrace(tracefunc)
> >    settscdump(on_flag)
> >
> > Cheers,
> > Nick.
> >
> > --
> > Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
> > ---------------------------------------------------------------
> >              http://www.boredomandlaziness.org
> > _______________________________________________
> > Python-Dev mailing list
> > Python-Dev at python.org
> > http://mail.python.org/mailman/listinfo/python-dev
> > Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/guido%40python.org
> >
>
>
> --
> --Guido van Rossum (home page: http://www.python.org/~guido/)
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/brett%40python.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20060714/42f411c8/attachment-0001.html 

From guido at python.org  Fri Jul 14 23:32:52 2006
From: guido at python.org (Guido van Rossum)
Date: Fri, 14 Jul 2006 14:32:52 -0700
Subject: [Python-Dev] Handling of sys.args (Re: User's complaints)
In-Reply-To: <bbaeab100607141430q34a516b4s1d6a8495c482bfc@mail.gmail.com>
References: <ee2a432c0607041649i7ecb6cc1w8ebdf47f7d02097a@mail.gmail.com>
	<3e1553560607122243q754dd71dq2e6d95cdf11e89dd@mail.gmail.com>
	<4c45c1530607130037r1c14e3f6u972232bdfa481534@mail.gmail.com>
	<Pine.LNX.4.58.0607130240480.31087@server1.LFW.org>
	<44B60B5A.2080309@canterbury.ac.nz>
	<Pine.LNX.4.58.0607131333290.31087@server1.LFW.org>
	<44B723C6.9010309@canterbury.ac.nz> <44B76D3E.90503@gmail.com>
	<ca471dc20607141248i5143b747lb8f3cbc9a97b703c@mail.gmail.com>
	<bbaeab100607141430q34a516b4s1d6a8495c482bfc@mail.gmail.com>
Message-ID: <ca471dc20607141432s19c7af2cweab062f768d3078b@mail.gmail.com>

On 7/14/06, Brett Cannon <brett at python.org> wrote:
> On 7/14/06, Guido van Rossum <guido at python.org> wrote:
> > Whoa, whoa. What's the *problem* we're trying to solve here?
>
> I have a use case for sandboxing.  I am already having to plan to have a
> mini-sys module in a sandbox so that they cannot get access to dangerous
> things.

OK, then I propose that we wait to see which things you end up having
to provide to sandboxed code, rather than trying to analyze it to
death in abstracto.

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From brett at python.org  Fri Jul 14 23:36:07 2006
From: brett at python.org (Brett Cannon)
Date: Fri, 14 Jul 2006 14:36:07 -0700
Subject: [Python-Dev] Handling of sys.args (Re: User's complaints)
In-Reply-To: <ca471dc20607141432s19c7af2cweab062f768d3078b@mail.gmail.com>
References: <ee2a432c0607041649i7ecb6cc1w8ebdf47f7d02097a@mail.gmail.com>
	<4c45c1530607130037r1c14e3f6u972232bdfa481534@mail.gmail.com>
	<Pine.LNX.4.58.0607130240480.31087@server1.LFW.org>
	<44B60B5A.2080309@canterbury.ac.nz>
	<Pine.LNX.4.58.0607131333290.31087@server1.LFW.org>
	<44B723C6.9010309@canterbury.ac.nz> <44B76D3E.90503@gmail.com>
	<ca471dc20607141248i5143b747lb8f3cbc9a97b703c@mail.gmail.com>
	<bbaeab100607141430q34a516b4s1d6a8495c482bfc@mail.gmail.com>
	<ca471dc20607141432s19c7af2cweab062f768d3078b@mail.gmail.com>
Message-ID: <bbaeab100607141436h1977ac34k179b5165593a967@mail.gmail.com>

On 7/14/06, Guido van Rossum <guido at python.org> wrote:
>
> On 7/14/06, Brett Cannon <brett at python.org> wrote:
> > On 7/14/06, Guido van Rossum <guido at python.org> wrote:
> > > Whoa, whoa. What's the *problem* we're trying to solve here?
> >
> > I have a use case for sandboxing.  I am already having to plan to have a
> > mini-sys module in a sandbox so that they cannot get access to dangerous
> > things.
>
> OK, then I propose that we wait to see which things you end up having
> to provide to sandboxed code, rather than trying to analyze it to
> death in abstracto.


Fine by me.  My design doc already has a preliminary list going  that people
can take a look at.  But it could still change if security analysis shows
possible problems with exposing some of them.

-Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20060714/0dbcccc5/attachment.html 

From martin at v.loewis.de  Fri Jul 14 23:38:52 2006
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Fri, 14 Jul 2006 23:38:52 +0200
Subject: [Python-Dev] Community buildbots
In-Reply-To: <60ed19d40607131406u498b0901oc84296f4e631ae5e@mail.gmail.com>
References: <2mfyh52wrn.fsf@starship.python.net>	<20060713204146.29014.283506497.divmod.quotient.35352@ohm>	<05af01c6a6be$54b653b0$d503030a@trilan>
	<60ed19d40607131406u498b0901oc84296f4e631ae5e@mail.gmail.com>
Message-ID: <44B80EEC.1030306@v.loewis.de>

Christopher Armstrong wrote:
> python -U is a failure for obvious reasons and a
> __future__ import is clearly better.

I disagree.

Regards,
Martin

From martin at v.loewis.de  Fri Jul 14 23:42:30 2006
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Fri, 14 Jul 2006 23:42:30 +0200
Subject: [Python-Dev] Community buildbots (was Re: User's complaints)
In-Reply-To: <60ed19d40607131532g3c0a5cc1n3177a22d90999dab@mail.gmail.com>
References: <20060713182916.GA12832@panix.com>	<20060713223024.29014.1648933869.divmod.quotient.35468@ohm>
	<60ed19d40607131532g3c0a5cc1n3177a22d90999dab@mail.gmail.com>
Message-ID: <44B80FC6.4090006@v.loewis.de>

Christopher Armstrong wrote:
> I'm at least willing to set up the buildbots for projects I care about
> (Twisted, pydoctor, whatever), and perhaps help out with the setting
> up the buildmaster.

If you need trigger calls from subversion's post-commit hooks, please
let me know, and I can set them up.

Regards,
Martin

From tjreedy at udel.edu  Sat Jul 15 04:12:44 2006
From: tjreedy at udel.edu (Terry Reedy)
Date: Fri, 14 Jul 2006 22:12:44 -0400
Subject: [Python-Dev] Community buildbots (was Re: User's complaints)
References: <ee2a432c0607132339o1d35c402ib13884c85115856a@mail.gmail.com><09f901c6a72c$495f2690$12472597@bagio>
	<20060714112137.GA891@Andrew-iBook2.local>
Message-ID: <e99iut$u6n$1@sea.gmane.org>


"A.M. Kuchling" <amk at amk.ca> wrote in message 
news:20060714112137.GA891 at Andrew-iBook2.local...
> http://www.python.org/dev/tools/, in a discussion of checkin policies,
> does say:
>
> The Python source tree is managed for stability, meaning that
> if you make a checkout at a random point in time the tree will almost
> always compile and be quite stable.

That is the goal, but when I watched the buildbot results last spring, the 
degree of stability (greenness) appeared to vary.  Is it possible to tag 
particular versions as a 'green' version, or the 'most recent green 
version' worth playing with?

tjr




From glyph at divmod.com  Sat Jul 15 04:47:26 2006
From: glyph at divmod.com (glyph at divmod.com)
Date: Fri, 14 Jul 2006 22:47:26 -0400
Subject: [Python-Dev] Community buildbots
In-Reply-To: <44B80EEC.1030306@v.loewis.de>
Message-ID: <20060715024726.29014.1894639716.divmod.quotient.37353@ohm>

On Fri, 14 Jul 2006 23:38:52 +0200, "\"Martin v. L?wis\"" <martin at v.loewis.de> wrote:
>Christopher Armstrong wrote:
>> python -U is a failure for obvious reasons and a
>> __future__ import is clearly better.
>
>I disagree.

I am surprised that you do, since I thought that Chris's conclusion was 
pretty obvious.  Python -U doesn't work, even on the standard library.

For example,

glyph at legion:~% python -S -U -c 'import pickletools'
Traceback (most recent call last):
  File "<string>", line 1, in ?
  File "/usr/lib/python2.4/pickletools.py", line 219, in ?
    doc="One-byte unsigned integer.")
  File "/usr/lib/python2.4/pickletools.py", line 187, in __init__
    assert isinstance(name, str)
AssertionError

This was just the first convenient example.  There are others.

A __future__ import would allow these behaviors to be upgraded module-by-module.
Right now, all -U provides is an option that can't be used on any realistically
sized program, so I don't see what the utility is.

From tjreedy at udel.edu  Sat Jul 15 06:13:35 2006
From: tjreedy at udel.edu (Terry Reedy)
Date: Sat, 15 Jul 2006 00:13:35 -0400
Subject: [Python-Dev] Community buildbots
References: <17591.7500.168119.701223@montanaro.dyndns.org>
	<20060714051157.29014.803759518.divmod.quotient.35955@ohm>
Message-ID: <e99q1g$cpg$1@sea.gmane.org>


<glyph at divmod.com> wrote in message 
news:20060714051157.29014.803759518.divmod.quotient.35955 at ohm...
> On Thu, 13 Jul 2006 23:27:56 -0500, skip at pobox.com wrote:
>
>>The buildbot idea sounds excellent.
>
> Thanks.  If someone can set this up, it pretty much addresses my 
> concerns.
> ...
> I am aware of when new releases come out :).  What I'm not aware of is 
> what
> features (may) have broken my code, and why.  As long as Python's 
> trunk at HEAD
> continues to run the test suites cleanly, I am mostly unconcerned.  When 
> it
> breaks, though, I want a chance to look at the cause of the breakage, 
> *before*
> there is an alpha or beta Python release out and people are starting to 
> write
> code that depends on its new features.

Is the following something like what you are suggesting?

A Python Application Testing (PAT) machine is set up with buildbot and any 
needed custom scripts.  Sometime after that and after 2.5 is released, when 
you have a version of, for instance, Twisted that passes its automated test 
suite when run on 2.5, you send it (or a URL) and an email address to PAT. 
Other developers do the same.  Periodically (once a week?), when PAT is 
free and a new green development version of either the 2.5.x or 2.6 
branches is available, PAT runs the test suites against that version.  An 
email is sent for any that fail, perhaps accompanied by the concatenation 
of the relevant checkin message.  Some possible options are to select just 
one of the branches for testing, to have more than one stable version being 
tested, and to receive pass emails.

Terry Jan Reedy




From ncoghlan at gmail.com  Sat Jul 15 06:35:08 2006
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 15 Jul 2006 14:35:08 +1000
Subject: [Python-Dev] Community buildbots
In-Reply-To: <20060715024726.29014.1894639716.divmod.quotient.37353@ohm>
References: <20060715024726.29014.1894639716.divmod.quotient.37353@ohm>
Message-ID: <44B8707C.3050207@gmail.com>

glyph at divmod.com wrote:
> A __future__ import would allow these behaviors to be upgraded module-by-module.

No it wouldn't.

__future__ works solely on the semantics of different pieces of syntax, 
because any syntax changes are purely local to the current module. This 
doesn't work for datatypes, because data types can cross module boundaries - 
other modules may still be using the old behaviour, and there's nothing the 
current module can do about it.

Changing all the literals in a module to be unicode instances instead of str 
instances is merely scratching the surface of the problem - such a module 
would still cause severe problems for any non-Unicode aware applications that 
expected it to return strings.

> Right now, all -U provides is an option that can't be used on any realistically
> sized program, so I don't see what the utility is.

There's never been a concerted effort to make even the standard library work 
under -U. Maybe that should be a goal for Python 2.6 - figure out what tools 
or changes are needed to make code -U safe, add them, and then update the 
standard library to use them.

PEP 349 (currently deferred, although I don't recall why) discusses some of 
the issues involved in making code unicode-safe, while still supporting 
non-unicode safe applications as clients.

Cheers,
Nick.

[1] http://www.python.org/dev/peps/pep-0349/

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
             http://www.boredomandlaziness.org

From nmm1 at cus.cam.ac.uk  Sat Jul 15 08:59:53 2006
From: nmm1 at cus.cam.ac.uk (Nick Maclaren)
Date: Sat, 15 Jul 2006 07:59:53 +0100
Subject: [Python-Dev] Handling of sys.args (Re: User's complaints)
Message-ID: <E1G1e81-0005qu-S5@draco.cus.cam.ac.uk>

"Guido van Rossum" <guido at python.org> wrote:
>
> OK, then I propose that we wait to see which things you end up having
> to provide to sandboxed code, rather than trying to analyze it to
> death in abstracto.

However, the ORIGINAL proposal in this thread (to split off argv[0]
and/or make that and the arguments read-only) is entirely different.
That is purely a matter of convenience, cleanliness of specification
or whatever you call it.  I can't imagine any good reason to separate
argv[0] from argv[1:] by a sandbox (either way).


Regards,
Nick Maclaren,
University of Cambridge Computing Service,
New Museums Site, Pembroke Street, Cambridge CB2 3QH, England.
Email:  nmm1 at cam.ac.uk
Tel.:  +44 1223 334761    Fax:  +44 1223 334679

From ashemedai at gmail.com  Sat Jul 15 09:43:00 2006
From: ashemedai at gmail.com (Jeroen Ruigrok van der Werven)
Date: Sat, 15 Jul 2006 09:43:00 +0200
Subject: [Python-Dev] Community buildbots (was Re: User's complaints)
In-Reply-To: <200607141343.38424.anthony@interlink.com.au>
References: <20060713180322.29014.1145514133.divmod.quotient.35181@ohm>
	<200607141343.38424.anthony@interlink.com.au>
Message-ID: <3e1553560607150043j603889fco95a993a2ca587419@mail.gmail.com>

On 7/14/06, Anthony Baxter <anthony at interlink.com.au> wrote:
> The "community buildbot" idea is a good one, although it should just
> be possible to write something for buildbot that checks out and
> builds the latest Python SVN, then installs it to a temporary
> location, then adds that location to the front of the path. Then each
> project could just add a "Python SVN buildbot" to their exists bbot
> install.

This is what Xenofarm for Pike has done for a long time now. See for
example: http://pike.ida.liu.se/development/pikefarm/7.7.xml

This is also what Bitten (http://bitten.cmlenz.net/) has implemented
for Trac (which is one of the bug/incident trackers for Python's call
for trackers).

-- 
Jeroen Ruigrok van der Werven

From martin at v.loewis.de  Sat Jul 15 10:36:28 2006
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sat, 15 Jul 2006 10:36:28 +0200
Subject: [Python-Dev] Community buildbots (was Re: User's complaints)
In-Reply-To: <e99iut$u6n$1@sea.gmane.org>
References: <ee2a432c0607132339o1d35c402ib13884c85115856a@mail.gmail.com><09f901c6a72c$495f2690$12472597@bagio>	<20060714112137.GA891@Andrew-iBook2.local>
	<e99iut$u6n$1@sea.gmane.org>
Message-ID: <44B8A90C.6070309@v.loewis.de>

Terry Reedy wrote:
> That is the goal, but when I watched the buildbot results last spring, the 
> degree of stability (greenness) appeared to vary.  Is it possible to tag 
> particular versions as a 'green' version, or the 'most recent green 
> version' worth playing with?

Don't get confused by these colors. The tree compiled nearly all of the
time, even if some tests were failing for an extended period of time on
some platforms.

Regards,
Martin

From martin at v.loewis.de  Sat Jul 15 10:40:26 2006
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sat, 15 Jul 2006 10:40:26 +0200
Subject: [Python-Dev] Community buildbots (was Re: User's complaints)
In-Reply-To: <3e1553560607150043j603889fco95a993a2ca587419@mail.gmail.com>
References: <20060713180322.29014.1145514133.divmod.quotient.35181@ohm>	<200607141343.38424.anthony@interlink.com.au>
	<3e1553560607150043j603889fco95a993a2ca587419@mail.gmail.com>
Message-ID: <44B8A9FA.90209@v.loewis.de>

Jeroen Ruigrok van der Werven wrote:
> This is what Xenofarm for Pike has done for a long time now. See for
> example: http://pike.ida.liu.se/development/pikefarm/7.7.xml
> 
> This is also what Bitten (http://bitten.cmlenz.net/) has implemented
> for Trac (which is one of the bug/incident trackers for Python's call
> for trackers).

Are you aware of http://www.python.org/dev/buildbot/ ?
We are not just talking about buildbots here (which the links you refer
to seem to be); we are talking about buildbots that don't test Python,
but test Python applications. We do know how to run a buildbot.

Regards,
Martin

From martin at v.loewis.de  Sat Jul 15 10:43:22 2006
From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=)
Date: Sat, 15 Jul 2006 10:43:22 +0200
Subject: [Python-Dev] Community buildbots
In-Reply-To: <20060715024726.29014.1894639716.divmod.quotient.37353@ohm>
References: <20060715024726.29014.1894639716.divmod.quotient.37353@ohm>
Message-ID: <44B8AAAA.70304@v.loewis.de>

glyph at divmod.com wrote:
>>> python -U is a failure for obvious reasons and a
>>> __future__ import is clearly better.
>> I disagree.
> 
> I am surprised that you do, since I thought that Chris's conclusion was 
> pretty obvious.  Python -U doesn't work, even on the standard library.

Sure, it doesn't work. That doesn't mean it's "a failure". It just
hasn't been completed yet.

> A __future__ import would allow these behaviors to be upgraded module-by-module.
> Right now, all -U provides is an option that can't be used on any realistically
> sized program, so I don't see what the utility is.

People can use it to improve the Unicode support in the Python standard
library. When they find that something doesn't work, they can study the
problem, and ponder possible solutions. Then, they can contribute
patches. -U has worked fine for me in the past, I contributed various
patches to make it work better. It hasn't failed for me at all.

Regards,
Martin

From stephen.thorne at gmail.com  Sat Jul 15 10:44:18 2006
From: stephen.thorne at gmail.com (Stephen Thorne)
Date: Sat, 15 Jul 2006 18:44:18 +1000
Subject: [Python-Dev] list.__init__() vs. dict.__init__() behaviour
Message-ID: <3e8ca5c80607150144r6b3990ccq7b61660ebfeb7858@mail.gmail.com>

Hi,

When testing some 'real world' code using pypy, an inconsistancy with
the way __init__ works between lists and dicts.

The assumption was made when implementing __init__ for pypy that
list.__init__ and dict.__init__ would both wipe the contents of the
objects, but it seems that in cpython, this isn't precisely the case.

>>> l = [2,3]
>>> list.__init__(l)
>>> l
[]

>>> d = {2: 3}
>>> dict.__init__(d)
>>> d
{2: 3}

dict.__init__(mydict) does not wipe the keys. list.__init__(mylist)
wipes the lists contents.

https://codespeak.net/issue/pypy-dev/issue240

Is there a good reason for this behaviour? It has broken my code (a
subclass of dict that populates a key before calling the superclasses
constructer, in the twisted codebase).

-- 
Stephen Thorne

"Give me enough bandwidth and a place to sit and I will move the world."
  --Jonathan Lange

From ashemedai at gmail.com  Sat Jul 15 10:57:30 2006
From: ashemedai at gmail.com (Jeroen Ruigrok van der Werven)
Date: Sat, 15 Jul 2006 10:57:30 +0200
Subject: [Python-Dev] Community buildbots (was Re: User's complaints)
In-Reply-To: <44B8A9FA.90209@v.loewis.de>
References: <20060713180322.29014.1145514133.divmod.quotient.35181@ohm>
	<200607141343.38424.anthony@interlink.com.au>
	<3e1553560607150043j603889fco95a993a2ca587419@mail.gmail.com>
	<44B8A9FA.90209@v.loewis.de>
Message-ID: <3e1553560607150157o2e521e64m677016161741ffc@mail.gmail.com>

On 7/15/06, "Martin v. L?wis" <martin at v.loewis.de> wrote:
> Are you aware of http://www.python.org/dev/buildbot/ ?

Yes. And it does not seem to be open for all, but then again, any
documentation with regard to it seems to be very sparse or hidden so I
can very well be wrong here. Ah, hidden away on the wiki:
http://wiki.python.org/moin/BuildBot

> We are not just talking about buildbots here (which the links you refer
> to seem to be); we are talking about buildbots that don't test Python,
> but test Python applications.

Then I would dare to say you haven't fully investigated the links
fully, Bitten, for example, also runs the unit-tests for any target
you configure, saves all the outputs and provides quick overviews as
well as coverage statistics. So that would fall perfectly into that
category. See http://bitten.cmlenz.net/build/0.5.x/762 for an example.

> We do know how to run a buildbot.

How relevant was this comment in the entire? I am merely trying to
help out here pointing out other similar projects when the community
participating building issue was raised.

-- 
Jeroen Ruigrok van der Werven

From martin at v.loewis.de  Sat Jul 15 12:25:19 2006
From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=)
Date: Sat, 15 Jul 2006 12:25:19 +0200
Subject: [Python-Dev] Community buildbots (was Re: User's complaints)
In-Reply-To: <3e1553560607150157o2e521e64m677016161741ffc@mail.gmail.com>
References: <20060713180322.29014.1145514133.divmod.quotient.35181@ohm>	
	<200607141343.38424.anthony@interlink.com.au>	
	<3e1553560607150043j603889fco95a993a2ca587419@mail.gmail.com>	
	<44B8A9FA.90209@v.loewis.de>
	<3e1553560607150157o2e521e64m677016161741ffc@mail.gmail.com>
Message-ID: <44B8C28F.1090207@v.loewis.de>

Jeroen Ruigrok van der Werven wrote:
>> Are you aware of http://www.python.org/dev/buildbot/ ?
> 
> Yes. And it does not seem to be open for all

Ah, ok. It indeed isn't open for anonymous participation; the test
results are open for all, though.

> 
>> We are not just talking about buildbots here (which the links you refer
>> to seem to be); we are talking about buildbots that don't test Python,
>> but test Python applications.
> 
> Then I would dare to say you haven't fully investigated the links
> fully, Bitten, for example, also runs the unit-tests for any target
> you configure

I don't understand that comment. Bitten seems to be a software package,
similar to buildbot. It doesn't do anything useful until I install and
configure it, unlike, say, SourceForge, which is not (just) a soft
package, but a running installation of it also. Right?

The effort is in installing and configuring it, given that many such
packages are already written. Distributed testing frameworks typically
support running arbitrary target comments, so Bitten is not really
special here (buildbot can also run about anything if you configure
it to).

>> We do know how to run a buildbot.
> 
> How relevant was this comment in the entire? I am merely trying to
> help out here pointing out other similar projects when the community
> participating building issue was raised.

It would have been helpful if you had stated *why* you think these
URLs are relevant in the context of the discussion. To me, it seemed
you are pointing to the existence of distributed testing frameworks.
I was pointing out that we (at least, I) am aware of the existence
of such frameworks, and that we were doing it for quite some time
now also, just like Pike.

Regards,
Martin


From guido at python.org  Sat Jul 15 16:45:37 2006
From: guido at python.org (Guido van Rossum)
Date: Sat, 15 Jul 2006 07:45:37 -0700
Subject: [Python-Dev] Handling of sys.args (Re: User's complaints)
In-Reply-To: <E1G1e81-0005qu-S5@draco.cus.cam.ac.uk>
References: <E1G1e81-0005qu-S5@draco.cus.cam.ac.uk>
Message-ID: <ca471dc20607150745q461c09ddx65b70a8b458fb73f@mail.gmail.com>

On 7/14/06, Nick Maclaren <nmm1 at cus.cam.ac.uk> wrote:
> "Guido van Rossum" <guido at python.org> wrote:
> >
> > OK, then I propose that we wait to see which things you end up having
> > to provide to sandboxed code, rather than trying to analyze it to
> > death in abstracto.
>
> However, the ORIGINAL proposal in this thread (to split off argv[0]
> and/or make that and the arguments read-only) is entirely different.
> That is purely a matter of convenience, cleanliness of specification
> or whatever you call it.  I can't imagine any good reason to separate
> argv[0] from argv[1:] by a sandbox (either way).

Sure. But that proposal has already been discussed (and found little
opposition). The thread then dispersed in a much less focused
discussion about the true purpose of the sys module..

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From guido at python.org  Sat Jul 15 16:50:31 2006
From: guido at python.org (Guido van Rossum)
Date: Sat, 15 Jul 2006 07:50:31 -0700
Subject: [Python-Dev] list.__init__() vs. dict.__init__() behaviour
In-Reply-To: <3e8ca5c80607150144r6b3990ccq7b61660ebfeb7858@mail.gmail.com>
References: <3e8ca5c80607150144r6b3990ccq7b61660ebfeb7858@mail.gmail.com>
Message-ID: <ca471dc20607150750vf63048apc8d1474cc850886f@mail.gmail.com>

On 7/15/06, Stephen Thorne <stephen.thorne at gmail.com> wrote:
> When testing some 'real world' code using pypy, an inconsistancy with
> the way __init__ works between lists and dicts.
>
> The assumption was made when implementing __init__ for pypy that
> list.__init__ and dict.__init__ would both wipe the contents of the
> objects, but it seems that in cpython, this isn't precisely the case.
>
> >>> l = [2,3]
> >>> list.__init__(l)
> >>> l
> []
>
> >>> d = {2: 3}
> >>> dict.__init__(d)
> >>> d
> {2: 3}
>
> dict.__init__(mydict) does not wipe the keys. list.__init__(mylist)
> wipes the lists contents.

I think it's an accident of implementation. I never thought about this
from the POV of a subclass. Apparently dict.__init__() shares some
code with dict.update(). Or does it do a merge without overwriting
existing keys?

> https://codespeak.net/issue/pypy-dev/issue240
>
> Is there a good reason for this behaviour? It has broken my code (a
> subclass of dict that populates a key before calling the superclasses
> constructer, in the twisted codebase).

I think your code was skating on awfully thin ice by making any
assumptions whatsoever about the base class constructor.

But I also think that this ought to be specified in the language spec.
I'm not sure that consistency between list and dict is important to
me, but they probably should both answer to some higher principle
which I cannot formulate ATM.

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From aahz at pythoncraft.com  Sat Jul 15 17:00:52 2006
From: aahz at pythoncraft.com (Aahz)
Date: Sat, 15 Jul 2006 08:00:52 -0700
Subject: [Python-Dev] Community buildbots
In-Reply-To: <44B7286F.7070406@acm.org>
References: <2mfyh52wrn.fsf@starship.python.net>
	<20060713204146.29014.283506497.divmod.quotient.35352@ohm>
	<05af01c6a6be$54b653b0$d503030a@trilan> <44B7286F.7070406@acm.org>
Message-ID: <20060715150052.GA5194@panix.com>

On Thu, Jul 13, 2006, Talin wrote:
>
> Actually - can we make new-style classes the default, but allow a
> way to switch to old-style classes if needed? Perhaps a command-line
> argument to set the default back to old-style?

Nope.  Python 3.0 will have new-style classes as the default, but there
will probably not be any way to use old-style classes.  As has been said
before, if you want to make new-style classes the default for any module,
just put

__metaclass__ = type

at the top.

Please, just drop this subject.  Guido has long since Pronounced.
-- 
Aahz (aahz at pythoncraft.com)           <*>         http://www.pythoncraft.com/

"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are, by
definition, not smart enough to debug it."  --Brian W. Kernighan

From glyph at divmod.com  Sat Jul 15 20:19:52 2006
From: glyph at divmod.com (glyph at divmod.com)
Date: Sat, 15 Jul 2006 14:19:52 -0400
Subject: [Python-Dev] Community buildbots
In-Reply-To: <e99q1g$cpg$1@sea.gmane.org>
Message-ID: <20060715181952.29014.1007814129.divmod.quotient.38416@ohm>



On Sat, 15 Jul 2006 00:13:35 -0400, Terry Reedy <tjreedy at udel.edu> wrote:

>Is the following something like what you are suggesting?

Something like it, but...

>A Python Application Testing (PAT) machine is set up with buildbot and any
>needed custom scripts.  Sometime after that and after 2.5 is released, when
>you have a version of, for instance, Twisted that passes its automated test
>suite when run on 2.5, you send it (or a URL) and an email address to PAT.
>Other developers do the same.  Periodically (once a week?), when PAT is
                                              ^
                                     "once per checkin to Python trunk"
>free and a new green development version of either the 2.5.x or 2.6
>branches is available, PAT runs the test suites against that version.  An
>email is sent for any that fail, perhaps accompanied by the concatenation
>of the relevant checkin message.  Some possible options are to select just
>one of the branches for testing, to have more than one stable version being
>tested, and to receive pass emails.

Sending email also isn't really necessary; I would just like a web page I can
look at (and draw the attention of the python core developers to).

From glyph at divmod.com  Sat Jul 15 20:37:34 2006
From: glyph at divmod.com (glyph at divmod.com)
Date: Sat, 15 Jul 2006 14:37:34 -0400
Subject: [Python-Dev] Community buildbots
In-Reply-To: <44B8707C.3050207@gmail.com>
Message-ID: <20060715183734.29014.989719061.divmod.quotient.38438@ohm>

On Sat, 15 Jul 2006 14:35:08 +1000, Nick Coghlan <ncoghlan at gmail.com> wrote:
>glyph at divmod.com wrote:
>>A __future__ import would allow these behaviors to be upgraded module-by- 
>>module.
>
>No it wouldn't.

Yes it would! :)

>__future__ works solely on the semantics of different pieces of syntax, 
>because any syntax changes are purely local to the current module.
...
>Changing all the literals in a module to be unicode instances instead of str 
>instances is merely scratching the surface of the problem - such a module 
>would still cause severe problems for any non-Unicode aware applications 
>that expected it to return strings.

A module with the given __future__ import could be written to expect that
literals are unicode instances instead of str, and encode them appropriately
when passing to modules that expect str.  This doesn't solve the problem, but
unlike -U, you can make fixes which will work persistently without having to
treat the type of every string literal as unknown.

The obvious way to write code that works under -U and still works in normal
Python is to .encode('charmap') every value intended to be an octet, and put
'u' in front of every string intended to be unicode.  That would seem to
defeat the purpose of changing the default literal type.


From jdahlin at async.com.br  Sat Jul 15 20:38:04 2006
From: jdahlin at async.com.br (Johan Dahlin)
Date: Sat, 15 Jul 2006 15:38:04 -0300
Subject: [Python-Dev] Dynamic module namspaces
Message-ID: <e9bcni$va0$1@sea.gmane.org>

In an effort to reduce the memory usage used by GTK+ applications 
written in python I've recently added a feature that allows attributes 
to be lazy loaded in a module namespace. The gtk python module contains 
quite a few attributes (around 850) of which many are classes or 
interfaces (150+)

The changes to PyGTK I had to make can not be considered anything but a 
hack; I had to put a subclass of a ModuleType in sys.modules and 
override __getattribute__ to be able to get old code which accessed 
gtk.__dict__ directly to still work (PyModule_GetDict requires that).
However, even if I didn't have to use __getattribute__ overriding 
sys.modules is rather unpleasent and I'm afraid it'll cause problems in 
the future.

My point is that I consider this to be a valid use case, the amount of 
saved memory is significan, and I could not find another way of doing it 
and still keep the gtk interface (import gtk; gtk.Button) to still be 
backwards compatible.

What I want to ask, is it possible to have a sanctioned way to implement 
a dynamic module/namespace in python?

For instance, it could be implemented to allow you to replace the 
__dict__ attribute in a module with a user provided object which 
implements the dictionary protocol.

Johan


From glyph at divmod.com  Sat Jul 15 20:40:45 2006
From: glyph at divmod.com (glyph at divmod.com)
Date: Sat, 15 Jul 2006 14:40:45 -0400
Subject: [Python-Dev] Community buildbots
In-Reply-To: <44B8AAAA.70304@v.loewis.de>
Message-ID: <20060715184045.29014.1300881616.divmod.quotient.38443@ohm>

On Sat, 15 Jul 2006 10:43:22 +0200, "\"Martin v. L?wis\"" <martin at v.loewis.de> wrote:

>People can use [-U] to improve the Unicode support in the Python standard
>library. When they find that something doesn't work, they can study the
>problem, and ponder possible solutions. Then, they can contribute
>patches. -U has worked fine for me in the past, I contributed various
>patches to make it work better. It hasn't failed for me at all.

I guess it makes more sense as a development tool to work on zero-dependency
tools like the standard library.  Still, -Q has both a __future__ import and
a command-line option, why not -U?

From rasky at develer.com  Sat Jul 15 20:48:20 2006
From: rasky at develer.com (Giovanni Bajo)
Date: Sat, 15 Jul 2006 20:48:20 +0200
Subject: [Python-Dev] Dynamic module namspaces
References: <e9bcni$va0$1@sea.gmane.org>
Message-ID: <07e001c6a83f$3dd96060$d503030a@trilan>

Johan Dahlin wrote:

> My point is that I consider this to be a valid use case, the amount of
> saved memory is significan, and I could not find another way of doing
> it and still keep the gtk interface (import gtk; gtk.Button) to still be
> backwards compatible.

You may want to have a look at SIP/PyQt. They implement the full Qt
interface which is rather large, but import time is blazingly fast and
memory occupation grows only of 4-5 Mb at import-time. The trick is that
methods are "generated" dynamically at their first usage somehow (but dir()
and introspection still works...).

SIP is free and generic btw, you may want to consider it as a tool.
-- 
Giovanni Bajo


From pje at telecommunity.com  Sat Jul 15 20:53:06 2006
From: pje at telecommunity.com (Phillip J. Eby)
Date: Sat, 15 Jul 2006 14:53:06 -0400
Subject: [Python-Dev] Dynamic module namspaces
In-Reply-To: <e9bcni$va0$1@sea.gmane.org>
Message-ID: <5.1.1.6.0.20060715144208.02864158@sparrow.telecommunity.com>

At 03:38 PM 7/15/2006 -0300, Johan Dahlin wrote:
>In an effort to reduce the memory usage used by GTK+ applications
>written in python I've recently added a feature that allows attributes
>to be lazy loaded in a module namespace. The gtk python module contains
>quite a few attributes (around 850) of which many are classes or
>interfaces (150+)
>
>The changes to PyGTK I had to make can not be considered anything but a
>hack; I had to put a subclass of a ModuleType in sys.modules and
>override __getattribute__ to be able to get old code which accessed
>gtk.__dict__ directly to still work (PyModule_GetDict requires that).

Just as a point of reference, the Importing package does something very 
similar, to support "weak" and "lazy" imports:

http://cheeseshop.python.org/pypi/Importing


>However, even if I didn't have to use __getattribute__ overriding
>sys.modules is rather unpleasent and I'm afraid it'll cause problems in
>the future.

The things most likely to be problems are tools like pydoc or other 
inspect-using code that expects modules to be exactly ModuleType and don't 
use isinstance().  Apart from that, I've been using the technique since the 
early days of Python 2.2 without encountering any problems until the PEP 
302 "reload()" bug came along, but that was fixed in 2.3.5.  I haven't seen 
any other problems since.

On the other hand, the Importing package takes a much more conservative 
approach than you are doing; it simply runs reload() on a module when 
__getattribute__ is called (after restoring the old version of 
__getattribute__).  So, as soon as you touch a lazily loaded module, it 
ceases to be particularly special, and it has a real __dict__.  It's 
possible that what you're doing could have more side-effects than what I'm 
doing.


>What I want to ask, is it possible to have a sanctioned way to implement
>a dynamic module/namespace in python?

That would be nice, but I think that what you and I are doing are probably 
the One Obvious Ways to do the respective things we're doing.



From mal at egenix.com  Sat Jul 15 21:15:04 2006
From: mal at egenix.com (M.-A. Lemburg)
Date: Sat, 15 Jul 2006 21:15:04 +0200
Subject: [Python-Dev] Community buildbots
In-Reply-To: <20060715183734.29014.989719061.divmod.quotient.38438@ohm>
References: <20060715183734.29014.989719061.divmod.quotient.38438@ohm>
Message-ID: <44B93EB8.5020908@egenix.com>

glyph at divmod.com wrote:
> On Sat, 15 Jul 2006 14:35:08 +1000, Nick Coghlan <ncoghlan at gmail.com> wrote:
>> glyph at divmod.com wrote:
>>> A __future__ import would allow these behaviors to be upgraded module-by- 
>>> module.
>> No it wouldn't.
> 
> Yes it would! :)
> 
>> __future__ works solely on the semantics of different pieces of syntax, 
>> because any syntax changes are purely local to the current module.
> ...
>> Changing all the literals in a module to be unicode instances instead of str 
>> instances is merely scratching the surface of the problem - such a module 
>> would still cause severe problems for any non-Unicode aware applications 
>> that expected it to return strings.
> 
> A module with the given __future__ import could be written to expect that
> literals are unicode instances instead of str, and encode them appropriately
> when passing to modules that expect str.  This doesn't solve the problem, but
> unlike -U, you can make fixes which will work persistently without having to
> treat the type of every string literal as unknown.
> 
> The obvious way to write code that works under -U and still works in normal
> Python is to .encode('charmap') every value intended to be an octet, and put
> 'u' in front of every string intended to be unicode.  That would seem to
> defeat the purpose of changing the default literal type.

Think of it this way: with the u'' literal prefix you can upgrade
the behavior on a per-literal basis ;-)

Seriously, it really does makes more sense to upgrade to
Unicode on a per-literal basis than on a per-module or per-
process basis. When upgrading to Unicode you have to carefully
consider the path each changed object will take through the
code.

Note that it also helps setting the default encoding
to 'unknown'. That way you disable the coercion of strings
to Unicode and all the places where this implicit conversion
takes place crop up, allowing you to take proper action (i.e.
explicit conversion or changing of the string to Unicode
as appropriate).

Later on, you can then simply remove the u'' prefix when
moving to Py3k. In fact, Py3k could simply ignore the prefix
for you (and maybe generate an optional warning) to make the
transition easier.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Jul 15 2006)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! ::::

From foom at fuhm.net  Sat Jul 15 22:00:35 2006
From: foom at fuhm.net (James Y Knight)
Date: Sat, 15 Jul 2006 16:00:35 -0400
Subject: [Python-Dev] Community buildbots
In-Reply-To: <44B93EB8.5020908@egenix.com>
References: <20060715183734.29014.989719061.divmod.quotient.38438@ohm>
	<44B93EB8.5020908@egenix.com>
Message-ID: <C0545DE4-6C3D-4B91-A902-4C8C70893D35@fuhm.net>

On Jul 15, 2006, at 3:15 PM, M.-A. Lemburg wrote:
> Note that it also helps setting the default encoding
> to 'unknown'. That way you disable the coercion of strings
> to Unicode and all the places where this implicit conversion
> takes place crop up, allowing you to take proper action (i.e.
> explicit conversion or changing of the string to Unicode
> as appropriate).

I've tried that before to verify no such conversion issues occurred  
in Twisted, but, as the python stdlib isn't usable like that, it's  
hard to use it to find bugs in any other libraries. (in particular,  
the re module is badly broken, some other stuff was too).

James

From mike at skew.org  Sat Jul 15 22:27:15 2006
From: mike at skew.org (Mike Brown)
Date: Sat, 15 Jul 2006 14:27:15 -0600 (MDT)
Subject: [Python-Dev] urllib.quote and unicode bug resuscitation attempt
In-Reply-To: <44B658BB.5090308@ofai.at>
Message-ID: <200607152027.k6FKRFGO053680@chilled.skew.org>

Stefan Rank wrote:
> Well, originally, I would have expected it to return a byte str(ing),

I'd assume unicode in, unicode out, and str in, str out, but since it's
always going to produce ASCII-range characters, it wouldn't matter.
Moot point anyway.

> BUT
> I am now converted and think it is best to raise a TypeError for 
> unicode, and leave the encoding decisions to higher level code.
> 
> So I'll repeat the "patch" #1, slightly modified::
> 
>   if isinstance(s, unicode):
>       raise TypeError("quote expects an encoded byte string as argument")
> 
> Is it safe to assume that code that breaks because of this change was 
> already broken?

Yes. The patch seems fine to me, although maybe

if not isinstance(s, str)

would be better?

From mal at egenix.com  Sat Jul 15 23:01:56 2006
From: mal at egenix.com (M.-A. Lemburg)
Date: Sat, 15 Jul 2006 23:01:56 +0200
Subject: [Python-Dev] Community buildbots
In-Reply-To: <C0545DE4-6C3D-4B91-A902-4C8C70893D35@fuhm.net>
References: <20060715183734.29014.989719061.divmod.quotient.38438@ohm>	<44B93EB8.5020908@egenix.com>
	<C0545DE4-6C3D-4B91-A902-4C8C70893D35@fuhm.net>
Message-ID: <44B957C4.2010307@egenix.com>

James Y Knight wrote:
> On Jul 15, 2006, at 3:15 PM, M.-A. Lemburg wrote:
>> Note that it also helps setting the default encoding
>> to 'unknown'. That way you disable the coercion of strings
>> to Unicode and all the places where this implicit conversion
>> takes place crop up, allowing you to take proper action (i.e.
>> explicit conversion or changing of the string to Unicode
>> as appropriate).
> 
> I've tried that before to verify no such conversion issues occurred in
> Twisted, but, as the python stdlib isn't usable like that, it's hard to
> use it to find bugs in any other libraries. (in particular, the re
> module is badly broken, some other stuff was too).

True: it breaks a lot of code that was written to work
with both strings and Unicode (or does so by accident ;-).

The stdlib isn't too well prepared for Unicode yet, so
if your code relies a lot on it, then the above may not
be the right strategy for you.

Perhaps a new ASCII codec that issues warnings for all these
cases would help ?! (one that still converts to Unicode
assuming ASCII, but issues a warning pointing to the location
in the code where the conversion happend)

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Jul 15 2006)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! ::::

From martin at v.loewis.de  Sun Jul 16 01:38:04 2006
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sun, 16 Jul 2006 01:38:04 +0200
Subject: [Python-Dev] Community buildbots
In-Reply-To: <20060715183734.29014.989719061.divmod.quotient.38438@ohm>
References: <20060715183734.29014.989719061.divmod.quotient.38438@ohm>
Message-ID: <44B97C5C.6070001@v.loewis.de>

glyph at divmod.com wrote:
> A module with the given __future__ import could be written to expect that
> literals are unicode instances instead of str, and encode them appropriately
> when passing to modules that expect str.

Such changes might have to be reverted in Python 3, since the module
might then expect character strings instead of byte strings, and then
might complain when it gets byte strings (which .encode will produce).
So declaring that all string literals are Unicode objects might not
help in the future, contrary to what the future import suggests.

> The obvious way to write code that works under -U and still works in normal
> Python is to .encode('charmap') every value intended to be an octet, and put
> 'u' in front of every string intended to be unicode.  That would seem to
> defeat the purpose of changing the default literal type.

The not so obvious way is to change the modules/methods receiving these
strings to work with either string type if that is reasonable.

Regards,
Martin

From nnorwitz at gmail.com  Sun Jul 16 03:35:24 2006
From: nnorwitz at gmail.com (Neal Norwitz)
Date: Sat, 15 Jul 2006 18:35:24 -0700
Subject: [Python-Dev] Community buildbots (was Re: User's complaints)
In-Reply-To: <44B8A90C.6070309@v.loewis.de>
References: <ee2a432c0607132339o1d35c402ib13884c85115856a@mail.gmail.com>
	<09f901c6a72c$495f2690$12472597@bagio>
	<20060714112137.GA891@Andrew-iBook2.local>
	<e99iut$u6n$1@sea.gmane.org> <44B8A90C.6070309@v.loewis.de>
Message-ID: <ee2a432c0607151835m741dc92dpa5cc3a7149842a1c@mail.gmail.com>

On 7/15/06, "Martin v. L?wis" <martin at v.loewis.de> wrote:
> Terry Reedy wrote:
> > That is the goal, but when I watched the buildbot results last spring, the
> > degree of stability (greenness) appeared to vary.  Is it possible to tag
> > particular versions as a 'green' version, or the 'most recent green
> > version' worth playing with?
>
> Don't get confused by these colors. The tree compiled nearly all of the
> time, even if some tests were failing for an extended period of time on
> some platforms.

Right.  It's very rare that the interpreter or stdlib is broken.
There are 19 buildbots currently.  7 are not green.  5 of those are
offline, most with known (to me and the person that donated the
machines) hardware issues ranging from lack of air conditioning, to a
bad router, to no power.  1 is cygwin which is running an old version.
 I just (an hour or so ago) got an account on a cygwin box to help
diagnose the status and figure out if anything within Python or the
tests are broken.  That leaves 1 unexplained failure on a Windows bot.
 This started happening recently after being down for a while.  I
haven't had time to investigate.

The reason why it was not very green in spring was because that's when
we were getting it set up!  The master looks like it was installed at
the end of 2005/beginning of 2006.  It took several months to get rid
of many testing issues.  Tests that couldn't be run in a particular
order, tests that couldn't be run simultaneously (socket), bad
compilation of sqlite/bsddb modules (not in python), misconfigured
systems, tests verifying something that was system dependent, signed
addresses, etc.

Of all the problems there were, I only remember a single problem in
Python.  (There were probably more, I just remember one.)  That was in
test code (xxmodule or something like that.  There were a bunch of
problems with ctypes and/or sqlite when they got integrated having to
deal with these new platforms.  That may be what you are recalling.
Right before alpha 2 was a particularly bad time.

What we mean by stable is that at any time/any stage of development,
if a change goes in that breaks the tests, it is likely to be fixed or
reverted within hours.  The amount of time the build or tests are
broken on the majority of platforms is quite small.  It took a while
to get to this point due to a bunch of flaky tests, but those have
mostly been fixed.  The only known problems are mentioned on the wiki.

When the buildbots fail, we get mail to python-checkins.
Unfortunately, that's mostly spam at this point.  I hope to fix that
at some point.  I also hope to change the main page to give more info
in less space.

n

From tim.peters at gmail.com  Sun Jul 16 05:47:50 2006
From: tim.peters at gmail.com (Tim Peters)
Date: Sat, 15 Jul 2006 23:47:50 -0400
Subject: [Python-Dev] Community buildbots (was Re: User's complaints)
In-Reply-To: <ee2a432c0607151835m741dc92dpa5cc3a7149842a1c@mail.gmail.com>
References: <ee2a432c0607132339o1d35c402ib13884c85115856a@mail.gmail.com>
	<09f901c6a72c$495f2690$12472597@bagio>
	<20060714112137.GA891@Andrew-iBook2.local>
	<e99iut$u6n$1@sea.gmane.org> <44B8A90C.6070309@v.loewis.de>
	<ee2a432c0607151835m741dc92dpa5cc3a7149842a1c@mail.gmail.com>
Message-ID: <1f7befae0607152047u43993a15ue5180b990f9a530f@mail.gmail.com>

[Neal Norwitz]
> ...
> That leaves 1 unexplained failure on a Windows bot.

It wasn't my Windows bot, but I believe test_profile has failed
(rarely) on several of the bots, and in the same (or very similar)
way.  Note that the failure went away on the Windows bot in question
the next time the tests were run on it.  That's typical of
test_profile failures.

Unfortunately, because test_profile works by comparing stdout against
a canned expected-output file under Lib/test/output/, when we re-run
the test in verbose mode at the end, that comparison isn't done, so
there's isn't a simple way to know whether the test passed or failed
during its second run.  I _bet_ it actually passed, but don't know.

This problem is obscured because when you run regrtest.py "by hand"
with -v, you get this message at the end:

"""
CAUTION:  stdout isn't compared in verbose mode:
a test that passes in verbose mode may fail without it.
"""

which is intended to alert you to the possible problem.

But when we re-run failing tests in verbose mode "by magic" (via
passing -w), that warning isn't produced.

BTW, the best solution to this is to convert all the tests away from
regrtest's original "compare stdout against a canned output/TEST_NAME
file" scheme.  That won't fix test_profile, but would make it less of
a puzzle to figure out what's wrong with it.

From tjreedy at udel.edu  Sun Jul 16 06:29:29 2006
From: tjreedy at udel.edu (Terry Reedy)
Date: Sun, 16 Jul 2006 00:29:29 -0400
Subject: [Python-Dev] Community buildbots
References: <e99q1g$cpg$1@sea.gmane.org>
	<20060715181952.29014.1007814129.divmod.quotient.38416@ohm>
Message-ID: <e9cfba$a5c$1@sea.gmane.org>


<glyph at divmod.com> wrote in message 
news:20060715181952.29014.1007814129.divmod.quotient.38416 at ohm...
>
>
> On Sat, 15 Jul 2006 00:13:35 -0400, Terry Reedy <tjreedy at udel.edu> wrote:
>>Other developers do the same.  Periodically (once a week?), when PAT is
>                                              ^
>                                     "once per checkin to Python trunk"

If the average rate of checkins is equal to or greater than the maximum 
rate of test runs then that is physically impossible.  (Indeed, queueing 
theory suggests that  keeping the queue size small requires that the 
checkin be more like half the test rate.)  This appears to be currently 
true for the Python trunk buildbot and I would expect running the test of 
numerous large applications would take even longer, making the test rate 
even lower.  If you want something like twisted run against multiple Python 
versions a day, I suspect you will have to do so on a system you control.

Terry Jan Reedy




From greg.ewing at canterbury.ac.nz  Sun Jul 16 10:12:11 2006
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sun, 16 Jul 2006 20:12:11 +1200
Subject: [Python-Dev] Problem with super() usage
Message-ID: <44B9F4DB.3030004@canterbury.ac.nz>

For about the third time in my life, I thought I might
have found a use for cooperative super calls, but I've
run into another problem with the concept.

Consider:

  class A(object):
   def m(self):
     print "A.m"

  class B(object):
   def m(self):
    print "B.m"
    super(B, self).m()

  class C(B, A):
   def m(self):
    print "C.m"
    super(C, self).m()

  >>> c = C()
  >>> c.m()
  C.m
  B.m
  A.m

Okay so far, but... what if I want to use class B on
its own, or in combination with other classes that don't
have an m() method?

 >>> b = B()
 >>> b.m()
B.m
Traceback (most recent call last):
   File "<stdin>", line 1, in ?
   File "<stdin>", line 4, in m
AttributeError: 'super' object has no attribute 'm'

In general, how is one supposed to use super() in a
class which may or may not be at the end of the mro
with respect to a particular method?

The only thing I can think of is to write all my
super()-using methods defensively like this:

   def m(self):
     ...
     s = super(C, self)
     if hasattr(s, 'm'):
       s.m()

which seems awfully tedious.

Does the Theory of Cooperative Method Calls have
anything to say about this?

--
Greg

From scott+python-dev at scottdial.com  Sun Jul 16 11:42:30 2006
From: scott+python-dev at scottdial.com (Scott Dial)
Date: Sun, 16 Jul 2006 05:42:30 -0400
Subject: [Python-Dev] Capabilities / Restricted Execution
In-Reply-To: <44B3E99A.4080000@acm.org>
References: <5.1.1.6.0.20060711105343.01eed878@sparrow.telecommunity.com>	<44B3E029.5070604@scottdial.com>
	<44B3E99A.4080000@acm.org>
Message-ID: <44BA0A06.40101@scottdial.com>

Talin wrote:
> Scott Dial wrote:
>> Phillip J. Eby wrote:
>>
>>> A function's func_closure contains cell objects that hold the 
>>> variables.  These are readable if you can set the func_closure of some 
>>> function of your own.  If the overall plan includes the ability to restrict 
>>> func_closure setting (or reading) in a restricted interpreter, then you 
>>> might be okay.
>>
>> Except this function (__getattribute__) has been trapped inside of a
>> class which does not expose it as an attribute. So, you shouldn't be
>> able to get to the func_closure attribute of the __getattribute__
>> function for an instance of the Guard class. I can't come up with a way
>> to defeat this protection, at least. If you have a way, then I'd be
>> interested to hear it.
> 
> I've thought of several ways to break it already. Some are repairable, 
> I'm not sure that they all are.
> 
> For example, neither of the following statements blows up:
> 
>     print t2.get_name.func_closure[0]
>     print object.__getattribute__( t2, '__dict__' )
> 
> Still, its perhaps a useful basis for experimentation.
> 
> -- Talin

I quickly poked around it in python and realized that in 2.5 (as opposed 
to the 2.4 python I was playing in) the cell object exposes 
cell_contents.. blargh. So, yes, you can defeat the protection because 
the wrapped instance is exposed.

     print t2.get_name()
     t2.get_name.func_closure[0].cell_contents.im_self.name = 'poop'
     print t2.get_name()

Although, your second example with using the object.__getattribute__ 
doesn't seem to really be an issue. You retrieved the __dict__ for the 
Guard class which is empty and is something we should not feel concerned 
about being leaked.

Only way I see this as viable is if in "restricted" mode cell_contents 
was removed from cell objects.

-- 
Scott Dial
scott at scottdial.com
scodial at indiana.edu

From foom at fuhm.net  Sun Jul 16 19:20:43 2006
From: foom at fuhm.net (James Y Knight)
Date: Sun, 16 Jul 2006 13:20:43 -0400
Subject: [Python-Dev] Capabilities / Restricted Execution
In-Reply-To: <44BA0A06.40101@scottdial.com>
References: <5.1.1.6.0.20060711105343.01eed878@sparrow.telecommunity.com>	<44B3E029.5070604@scottdial.com>
	<44B3E99A.4080000@acm.org> <44BA0A06.40101@scottdial.com>
Message-ID: <36ABCB42-2560-466B-A18B-E831036BCFCF@fuhm.net>


On Jul 16, 2006, at 5:42 AM, Scott Dial wrote:

> Talin wrote:
>> Scott Dial wrote:
>>> Phillip J. Eby wrote:
>>>
>>>> A function's func_closure contains cell objects that hold the
>>>> variables.  These are readable if you can set the func_closure  
>>>> of some
>>>> function of your own.  If the overall plan includes the ability  
>>>> to restrict
>>>> func_closure setting (or reading) in a restricted interpreter,  
>>>> then you
>>>> might be okay.
>>>
>>> Except this function (__getattribute__) has been trapped inside of a
>>> class which does not expose it as an attribute. So, you shouldn't be
>>> able to get to the func_closure attribute of the __getattribute__
>>> function for an instance of the Guard class. I can't come up with  
>>> a way
>>> to defeat this protection, at least. If you have a way, then I'd be
>>> interested to hear it.
>>
>> I've thought of several ways to break it already. Some are  
>> repairable,
>> I'm not sure that they all are.
>>
>> For example, neither of the following statements blows up:
>>
>>     print t2.get_name.func_closure[0]
>>     print object.__getattribute__( t2, '__dict__' )
>>
>> Still, its perhaps a useful basis for experimentation.
>>
>> -- Talin
>
> I quickly poked around it in python and realized that in 2.5 (as  
> opposed
> to the 2.4 python I was playing in) the cell object exposes
> cell_contents.. blargh. So, yes, you can defeat the protection because
> the wrapped instance is exposed.
>
>      print t2.get_name()
>      t2.get_name.func_closure[0].cell_contents.im_self.name = 'poop'
>      print t2.get_name()
>
> Although, your second example with using the object.__getattribute__
> doesn't seem to really be an issue. You retrieved the __dict__ for the
> Guard class which is empty and is something we should not feel  
> concerned
> about being leaked.
>
> Only way I see this as viable is if in "restricted" mode cell_contents
> was removed from cell objects.

Similarly to how function attributes aren't accessible in restricted  
mode. In older versions of python, it's always been possible to get  
the closure variables in non-restricted mode, via mutating func_code...

def get_closure_contents(fun):
   num = len(fun.func_closure)
   vars = ["x%d" % n for n in range(num)]
   defines = ' = '.join(vars) + " = None"
   returns = ', '.join(vars)+','
   exec """
def b():
   %s
   def bb():
     return %s
   return bb
""" % (defines, returns)
   old_code = fun.func_code
   fun.func_code = b().func_code
   result = fun()
   fun.func_code = old_code
   return dict(zip(old_code.co_freevars, result))

def make_secret(x,y):
   def g():
     return x*y
   return g


 >>> secret = f(5,7)
 >>> secret()
35
 >>> get_closure_contents(secret)
{'y': 7, 'x': 5}







From barry at python.org  Mon Jul 17 00:18:50 2006
From: barry at python.org (Barry Warsaw)
Date: Sun, 16 Jul 2006 18:18:50 -0400
Subject: [Python-Dev] Pronouncement on SF #1520294 sought
Message-ID: <E6EAC3D8-9839-4390-958E-8296C40FDE60@python.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Just as 2.5b2 was being release, I updated SF patch #1520294:

https://sourceforge.net/tracker/index.php? 
func=detail&aid=1520294&group_id=5470&atid=305470	

This fixes the pydoc, inspect, and types modules for built-in types  
like getset and member descriptors. I'm not sure if Georg has had a  
chance to look over the latest rev of the patch (which is currently  
assigned to him).

I'm basically just looking for an RM pronouncement one way or the  
other.  Can this be added to 2.5 or does it have to wait for 2.6?

Thanks!
- -Barry

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.3 (Darwin)

iQCVAwUBRLq7SnEjvBPtnXfVAQLCDAP/UWVSiX9zTnG4qMZJaHHbJDFqxZZnURkx
m7OXiCo0H9u4+mY59+FJRos1QA+CldEQ815n+AdxfJQFl0pQqOLaFrTBQtRTzbRU
wLV06bUT0OZLOW6mrLxCGNVjAZeua4bcNJBBet8uwj9tx+dftbUnQR921bDjrz9V
ZFUjUVXVirs=
=7utP
-----END PGP SIGNATURE-----

From qiuyingbo at gmail.com  Mon Jul 17 05:29:37 2006
From: qiuyingbo at gmail.com (Yingbo Qiu)
Date: Mon, 17 Jul 2006 11:29:37 +0800
Subject: [Python-Dev] I have submitted a patch that implement IrDA socket
	support .
Message-ID: <fd038ca40607162029i24b02ed4tcbd5bd6e626291c3@mail.gmail.com>

I have uploaded it at
https://sourceforge.net/tracker/download.php?group_id=5470&atid=305470&file_id=184896&aid=1522400.

socketmodule.c already have a bluetooth socket, so i think we could
put irda socket support in it. Now I can run a simple obex session
through IrDA socket layer.

my reference include:
 http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dnirda/html/irdawp.asp
 ftp://ftp.ravioli.pasta.cs.uit. no/pub/Ravioli/pyobex/patch-py152-irda3

Will anyone have interest in the patch?

From foom at fuhm.net  Mon Jul 17 06:33:23 2006
From: foom at fuhm.net (James Y Knight)
Date: Mon, 17 Jul 2006 00:33:23 -0400
Subject: [Python-Dev] Dynamic module namspaces
In-Reply-To: <e9bcni$va0$1@sea.gmane.org>
References: <e9bcni$va0$1@sea.gmane.org>
Message-ID: <D579349B-2C5B-4057-B6B5-A364387A659C@fuhm.net>


On Jul 15, 2006, at 2:38 PM, Johan Dahlin wrote:
> What I want to ask, is it possible to have a sanctioned way to  
> implement
> a dynamic module/namespace in python?
>
> For instance, it could be implemented to allow you to replace the
> __dict__ attribute in a module with a user provided object which
> implements the dictionary protocol.

I'd like this, as well, although my use case is different: I'd like  
to be able to deprecate attributes in a module. That is, if I have:

foo.py:
SOME_CONSTANT = 5

I'd like to be able to do something such that any time anyone  
accessed foo.SOME_CONSTANT, it'd emit a DeprecationWarning.

James

From andrew-pythondev at puzzling.org  Mon Jul 17 07:15:44 2006
From: andrew-pythondev at puzzling.org (Andrew Bennetts)
Date: Mon, 17 Jul 2006 15:15:44 +1000
Subject: [Python-Dev] Dynamic module namspaces
In-Reply-To: <e9bcni$va0$1@sea.gmane.org>
References: <e9bcni$va0$1@sea.gmane.org>
Message-ID: <20060717051544.GE21600@steerpike.home.puzzling.org>

On Sat, Jul 15, 2006 at 03:38:04PM -0300, Johan Dahlin wrote:
> In an effort to reduce the memory usage used by GTK+ applications 
> written in python I've recently added a feature that allows attributes 
> to be lazy loaded in a module namespace. The gtk python module contains 
> quite a few attributes (around 850) of which many are classes or 
> interfaces (150+)

Have you seen the "demandload" hack that Mercurial uses?  You can find it here:
    http://selenic.com/repo/hg?f=cb4715847a81;file=mercurial/demandload.py

You can see an example use of it here:
    http://selenic.com/repo/hg?f=d276571f2c4b;file=mercurial/commands.py

The advantage for an interactive command line tool isn't so much memory
consumption as speed.  Why waste hundreds of milliseconds importing code that
isn't used?  There's an experimental branch to use the same demandload code in
bzr, the reported results are "400ms for 'bzr rocks' down to 100ms, 'bzr root'
from 400ms => 200ms, etc." (according to
http://permalink.gmane.org/gmane.comp.version-control.bazaar-ng.general/13967)

Over half the runtime wasted on importing unused code!  There's a definite need
for a nice solution to this, and it should be included in the standard
batteries that come with Python.

If we can address related problems at the same time, like emitting deprecation
warnings for accessing certain module attributes, then even better!

-Andrew.


From jcarlson at uci.edu  Mon Jul 17 08:52:48 2006
From: jcarlson at uci.edu (Josiah Carlson)
Date: Sun, 16 Jul 2006 23:52:48 -0700
Subject: [Python-Dev] Dynamic module namspaces
In-Reply-To: <20060717051544.GE21600@steerpike.home.puzzling.org>
References: <e9bcni$va0$1@sea.gmane.org>
	<20060717051544.GE21600@steerpike.home.puzzling.org>
Message-ID: <20060716234046.0CA7.JCARLSON@uci.edu>


Andrew Bennetts <andrew-pythondev at puzzling.org> wrote:
> On Sat, Jul 15, 2006 at 03:38:04PM -0300, Johan Dahlin wrote:
> > In an effort to reduce the memory usage used by GTK+ applications 
> > written in python I've recently added a feature that allows attributes 
> > to be lazy loaded in a module namespace. The gtk python module contains 
> > quite a few attributes (around 850) of which many are classes or 
> > interfaces (150+)
> 
> Have you seen the "demandload" hack that Mercurial uses?  You can find it here:
>     http://selenic.com/repo/hg?f=cb4715847a81;file=mercurial/demandload.py
> 
> You can see an example use of it here:
>     http://selenic.com/repo/hg?f=d276571f2c4b;file=mercurial/commands.py

The problem with that particular method is that it requires that you use
a string to describe the set of modules you would like to import, rather
than a name.

In the py3k mailing list I recently described a mechanism (though not
implementation) for a somewhat related method to support automatic
reloading when a file has changed (for things like web frameworks), but
by removing that file change check, you get the equivalent to what you
describe, though you don't need to use strings, you can use names...

    from dynamicreload import DR

and used like...

    DR.os.path.join(...)

As long as you use the DR.-prefixed name, you get automatic reloading 
(if desired) on access.


> The advantage for an interactive command line tool isn't so much memory
> consumption as speed.  Why waste hundreds of milliseconds importing code that
> isn't used?  There's an experimental branch to use the same demandload code in
> bzr, the reported results are "400ms for 'bzr rocks' down to 100ms, 'bzr root'
> from 400ms => 200ms, etc." (according to
> http://permalink.gmane.org/gmane.comp.version-control.bazaar-ng.general/13967)
> 
> Over half the runtime wasted on importing unused code!  There's a definite need
> for a nice solution to this, and it should be included in the standard
> batteries that come with Python.

Well, just starting up Python without loading any modules can be time
consuming; perhaps even dwarfing the few hundred ms saved by dynamic
loading.


> If we can address related problems at the same time, like emitting deprecation
> warnings for accessing certain module attributes, then even better!

__deprecated__ = ['name1', ...]

Make your dynamic load/reload aware of the __deprecated__ attribute, and
you are done.


 - Josiah


From martin at v.loewis.de  Mon Jul 17 09:21:56 2006
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Mon, 17 Jul 2006 09:21:56 +0200
Subject: [Python-Dev] I have submitted a patch that implement IrDA
 socket support .
In-Reply-To: <fd038ca40607162029i24b02ed4tcbd5bd6e626291c3@mail.gmail.com>
References: <fd038ca40607162029i24b02ed4tcbd5bd6e626291c3@mail.gmail.com>
Message-ID: <44BB3A94.1040607@v.loewis.de>

Yingbo Qiu wrote:
> Will anyone have interest in the patch?

This patch comes to late for Python 2.5. It might be considered for 2.6,
if anybody has time to review it.

Regards,
Martin

From anthony at interlink.com.au  Mon Jul 17 10:20:16 2006
From: anthony at interlink.com.au (Anthony Baxter)
Date: Mon, 17 Jul 2006 18:20:16 +1000
Subject: [Python-Dev] Community buildbots (was Re: User's complaints)
In-Reply-To: <e8bf7a530607140545m6953c0c6r2a6969f716276549@mail.gmail.com>
References: <20060713180322.29014.1145514133.divmod.quotient.35181@ohm>
	<200607141714.09304.anthony@interlink.com.au>
	<e8bf7a530607140545m6953c0c6r2a6969f716276549@mail.gmail.com>
Message-ID: <200607171820.19936.anthony@interlink.com.au>

On Friday 14 July 2006 22:45, Jeremy Hylton wrote:
> Maybe the basic question is right, but the emphasis needs to be
> changed.  If we had a rule that said the final release was 90 days
> after the last submission that wasn't to fix a regression, we'd ask
> "Is this feature important enough to warrant delaying the release
> until three months from now?"  I'm not sure what I think, but it
> doesn't seem like an implausible policy.

I really really doubt that this would work. There's always pressure 
for "just one more feature" - and as the release drags on, this would 
increase, not decrease. Note that dragging the release process out 
has it's own costs, as mentioned previously - either the trunk is in 
some sort of near-frozen state for an extended period, or else we end 
up in the double-applying-bugfixes state by forking much earlier. 

This approach would also make it extremely difficult to plan for 
releases. I know that my free time varies across the course of the 
year, and I need _some_ sort of plan for when the release will happen 
so I can make sure I have the time free to spend on it. 

Anthony
-- 
Anthony Baxter     <anthony at interlink.com.au>
It's never too late to have a happy childhood.

From andrew-pythondev at puzzling.org  Mon Jul 17 10:57:15 2006
From: andrew-pythondev at puzzling.org (Andrew Bennetts)
Date: Mon, 17 Jul 2006 18:57:15 +1000
Subject: [Python-Dev] Dynamic module namspaces
In-Reply-To: <20060716234046.0CA7.JCARLSON@uci.edu>
References: <e9bcni$va0$1@sea.gmane.org>
	<20060717051544.GE21600@steerpike.home.puzzling.org>
	<20060716234046.0CA7.JCARLSON@uci.edu>
Message-ID: <20060717085715.GJ21600@steerpike.home.puzzling.org>

On Sun, Jul 16, 2006 at 11:52:48PM -0700, Josiah Carlson wrote:
> Andrew Bennetts <andrew-pythondev at puzzling.org> wrote:
[...]
> > 
> > Have you seen the "demandload" hack that Mercurial uses?  You can find it here:
> >     http://selenic.com/repo/hg?f=cb4715847a81;file=mercurial/demandload.py
> > 
> > You can see an example use of it here:
> >     http://selenic.com/repo/hg?f=d276571f2c4b;file=mercurial/commands.py
> 
> The problem with that particular method is that it requires that you use
> a string to describe the set of modules you would like to import, rather
> than a name.

I agree, it's ugly.  I'd like there to be a nicer solution.

> In the py3k mailing list I recently described a mechanism (though not
> implementation) for a somewhat related method to support automatic
> reloading when a file has changed (for things like web frameworks), but
> by removing that file change check, you get the equivalent to what you
> describe, though you don't need to use strings, you can use names...
> 
>     from dynamicreload import DR
> 
> and used like...
> 
>     DR.os.path.join(...)
> 
> As long as you use the DR.-prefixed name, you get automatic reloading 
> (if desired) on access.

Aside from also being ugly in its own way, a magic prefix like this would add a
small performance penalty to places that use it.  I believe demandload has the
nice feature that once a demandloaded object is accessed, it is replaced with
the actual object rather than the demandload gunk, so there's only a one-off
performance penalty.

> > The advantage for an interactive command line tool isn't so much memory
> > consumption as speed.  Why waste hundreds of milliseconds importing code that
> > isn't used?  There's an experimental branch to use the same demandload code in
> > bzr, the reported results are "400ms for 'bzr rocks' down to 100ms, 'bzr root'
> > from 400ms => 200ms, etc." (according to
> > http://permalink.gmane.org/gmane.comp.version-control.bazaar-ng.general/13967)
> > 
> > Over half the runtime wasted on importing unused code!  There's a definite need
> > for a nice solution to this, and it should be included in the standard
> > batteries that come with Python.
> 
> Well, just starting up Python without loading any modules can be time
> consuming; perhaps even dwarfing the few hundred ms saved by dynamic
> loading.

Well, it's only about 10ms on my laptop running Ubuntu (it varies up to 90ms,
but I expect that's just noise), and the "-S" switch makes no obvious difference
(tested with "python -c ''").  10ms overhead to start python I can live with.
It takes about that long run "svn --version".

> > If we can address related problems at the same time, like emitting deprecation
> > warnings for accessing certain module attributes, then even better!
> 
> __deprecated__ = ['name1', ...]
> 
> Make your dynamic load/reload aware of the __deprecated__ attribute, and
> you are done.

I'm fine with that syntax.  But regardless of how it's spelled, I'd really like
the standard library or interpreter to have support for it, rather than having a
non-standard 3rd-party system for it.  I want there to "be one-- and preferably
only one --obvious way to do it".

-Andrew.


From bborcic at gmail.com  Mon Jul 17 14:51:17 2006
From: bborcic at gmail.com (Boris Borcic)
Date: Mon, 17 Jul 2006 14:51:17 +0200
Subject: [Python-Dev] Python Style Sheets ? Re: User's complaints
In-Reply-To: <ca471dc20607141244x5c075c6bnd795493101cf19dd@mail.gmail.com>
References: <ee2a432c0607041649i7ecb6cc1w8ebdf47f7d02097a@mail.gmail.com>	<3e1553560607122243q754dd71dq2e6d95cdf11e89dd@mail.gmail.com>	<ca471dc20607130720q3fe1b363y533a7bff46579174@mail.gmail.com>	<e9838o$fpc$1@sea.gmane.org>
	<ca471dc20607141244x5c075c6bnd795493101cf19dd@mail.gmail.com>
Message-ID: <e9g156$3pd$1@sea.gmane.org>

Guido van Rossum wrote:
> You must be misunderstanding.

I don't think so. You appeared to say that the language changes too much because 
everyone wants different changes - that accumulate. I suggested a mechanism 
allowing people to see only the changes they want - or none at all - might be 
devised.

Regards, BB

> The root problem is that people
> (rightly) complain that the language changes too much. And you want to
> "fix" this by adding a deep and fundamental change to the language?
> What planet are you from? It reminds me of Jini, which was presented
> as a new standard to address the problem of too many conflicting
> standard. Get it? :-)
> 
> --Guido
> 
> On 7/14/06, Boris Borcic <bborcic at gmail.com> wrote:
>> Guido van Rossum wrote:
>> ...
>>> This is an illustration of the dilemma of maintaining a popular
>>> language: Everybody hates change (me too!) but everybody also has one
>>> thing that's bothering them so much they absolutely want it to be
>>> changed. If you were to implement all those personal pet peeves, you'd
>>> get a language that's more different from Python than Python is from
>>> Fortran.
>>>
>>> So where's the middle ground?
>> I feel some freedom could be reclaimed with a solution in the spirit of Turing
>> equivalence. Or, to take a less grandiose comparison, web style sheets -
>> separation of content and presentation.
>>
>> Suppose the standard required a (possibly empty) style-defining file prefix that
>> constrains the python source code in the file, and concurrently defined (mostly)
>> reversible and transparent source-to-source transforms that would map any source
>> code file to an equivalent source code file with an arbitrary chosen prefix.
>> Then users could chose their style of Python and either transform all source
>> files they install to their own style, or setup their editor to do it
>> back-and-forth for them. The choice of python presentation style would then
>> become a private choice.
>>
>> To illustrate the idea, this already exists in very embryonic form with unicode
>> encoding modelines. The current standard allows to imagine a Python editor that
>> would permit to set a "local standard encoding modeline" and then present any
>> source file as if it had been written while taking maximal profit from the
>> chosen encoding. Which may also be simple ascii.
>>
>> Cheers, BB
>> --
>> "C++ is a contradiction in terms" - Lorentz, Einstein, Poincar?
>>
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> http://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org
>>
> 
> 



From g.brandl at gmx.net  Mon Jul 17 15:11:36 2006
From: g.brandl at gmx.net (Georg Brandl)
Date: Mon, 17 Jul 2006 15:11:36 +0200
Subject: [Python-Dev] Pronouncement on SF #1520294 sought
In-Reply-To: <E6EAC3D8-9839-4390-958E-8296C40FDE60@python.org>
References: <E6EAC3D8-9839-4390-958E-8296C40FDE60@python.org>
Message-ID: <e9g2a8$80u$1@sea.gmane.org>

Barry Warsaw wrote:
> Just as 2.5b2 was being release, I updated SF patch #1520294:
> 
> https://sourceforge.net/tracker/index.php?
> func=detail&aid=1520294&group_id=5470&atid=305470
> 
> This fixes the pydoc, inspect, and types modules for built-in types
> like getset and member descriptors. I'm not sure if Georg has had a
> chance to look over the latest rev of the patch (which is currently
> assigned to him).

I have now reviewed it and added a comment to the patch.

Georg


From jdahlin at async.com.br  Mon Jul 17 15:41:30 2006
From: jdahlin at async.com.br (Johan Dahlin)
Date: Mon, 17 Jul 2006 10:41:30 -0300
Subject: [Python-Dev] Dynamic module namspaces
In-Reply-To: <20060717051544.GE21600@steerpike.home.puzzling.org>
References: <e9bcni$va0$1@sea.gmane.org>
	<20060717051544.GE21600@steerpike.home.puzzling.org>
Message-ID: <44BB938A.1040400@async.com.br>

Andrew Bennetts wrote:
> On Sat, Jul 15, 2006 at 03:38:04PM -0300, Johan Dahlin wrote:
>> In an effort to reduce the memory usage used by GTK+ applications 
>> written in python I've recently added a feature that allows attributes 
>> to be lazy loaded in a module namespace. The gtk python module contains 
>> quite a few attributes (around 850) of which many are classes or 
>> interfaces (150+)
> 
> Have you seen the "demandload" hack that Mercurial uses?  You can find it here:
>     http://selenic.com/repo/hg?f=cb4715847a81;file=mercurial/demandload.py

It seems quite similar to Philips Importer, it's not completely solving the
problem I'm having, since I do something like this:

class LazyNamespace(ModuleType):
    def __init__(self, realmodule, locals):
        attributes = {}
        for attr in realmodule._get_lazy_attribute_names():
            attributes[attr] = None

        def __getattribute__(_, name):
            if name in attributes:
                value = realmodule._construct_lazy_attribute(name)
	    ...
            return value

There are almost 1000 symbols in the gtk namespace, creating all of them at
import time wastes memory and speed, while I've been mainly looking at the
memory consumption speed will also benefit. Importing gtk at this point
quite fast actually, less than 200ms on my fairly old box.

GUI programs does not need to be as responsive as command line applications
as hg & bzr, users seems to accept that it takes a second or a few to start
up a GUI application.

-- 
Johan Dahlin <jdahlin at async.com.br>
Async Open Source

From andymac at bullseye.apana.org.au  Mon Jul 17 12:27:28 2006
From: andymac at bullseye.apana.org.au (Andrew MacIntyre)
Date: Mon, 17 Jul 2006 21:27:28 +1100
Subject: [Python-Dev] Community buildbots (was Re: User's complaints)
In-Reply-To: <200607171820.19936.anthony@interlink.com.au>
References: <20060713180322.29014.1145514133.divmod.quotient.35181@ohm>	<200607141714.09304.anthony@interlink.com.au>	<e8bf7a530607140545m6953c0c6r2a6969f716276549@mail.gmail.com>
	<200607171820.19936.anthony@interlink.com.au>
Message-ID: <44BB6610.7080307@bullseye.apana.org.au>

Anthony Baxter wrote:
> On Friday 14 July 2006 22:45, Jeremy Hylton wrote:
>> Maybe the basic question is right, but the emphasis needs to be
>> changed.  If we had a rule that said the final release was 90 days
>> after the last submission that wasn't to fix a regression, we'd ask
>> "Is this feature important enough to warrant delaying the release
>> until three months from now?"  I'm not sure what I think, but it
>> doesn't seem like an implausible policy.
> 
> I really really doubt that this would work. There's always pressure 
> for "just one more feature" - and as the release drags on, this would 
> increase, not decrease. Note that dragging the release process out 
> has it's own costs, as mentioned previously - either the trunk is in 
> some sort of near-frozen state for an extended period, or else we end 
> up in the double-applying-bugfixes state by forking much earlier. 
> 
> This approach would also make it extremely difficult to plan for 
> releases. I know that my free time varies across the course of the 
> year, and I need _some_ sort of plan for when the release will happen 
> so I can make sure I have the time free to spend on it. 

On the face of it, it seems to me that branching a new major release at
the 1st beta would be one way of managing this.  The trunk is not frozen
for an extended period, and any "features" and bug fixes could probably
be backported from the trunk on clearance from the RE with no more pain
than is currently endured.

One down side would be the possibility of a loss in emphasis in finishing
the job on the release to be.  I'm sure there are others...

-------------------------------------------------------------------------
Andrew I MacIntyre                     "These thoughts are mine alone..."
E-mail: andymac at bullseye.apana.org.au  (pref) | Snail: PO Box 370
        andymac at pcug.org.au             (alt) |        Belconnen ACT 2616
Web:    http://www.andymac.org/               |        Australia

From jdahlin at async.com.br  Mon Jul 17 15:31:44 2006
From: jdahlin at async.com.br (Johan Dahlin)
Date: Mon, 17 Jul 2006 10:31:44 -0300
Subject: [Python-Dev] Dynamic module namspaces
In-Reply-To: <D579349B-2C5B-4057-B6B5-A364387A659C@fuhm.net>
References: <e9bcni$va0$1@sea.gmane.org>
	<D579349B-2C5B-4057-B6B5-A364387A659C@fuhm.net>
Message-ID: <44BB9140.5050602@async.com.br>

James Y Knight wrote:
> 
> On Jul 15, 2006, at 2:38 PM, Johan Dahlin wrote:
>> What I want to ask, is it possible to have a sanctioned way to implement
>> a dynamic module/namespace in python?
>>
>> For instance, it could be implemented to allow you to replace the
>> __dict__ attribute in a module with a user provided object which
>> implements the dictionary protocol.
> 
> I'd like this, as well, although my use case is different: I'd like to
> be able to deprecate attributes in a module. That is, if I have:
> 
> foo.py:
> SOME_CONSTANT = 5
> 
> I'd like to be able to do something such that any time anyone accessed
> foo.SOME_CONSTANT, it'd emit a DeprecationWarning.

Agreed, this would be another nice feature to have.

I've did something similar a time ago for PyGTK aswell, while less elegant
than your proposed solution, it seems that it's working fairly well:

DeprecatedConstant can be found here:
http://cvs.gnome.org/viewcvs/pygtk/gtk/deprecation.py?view=markup

-- 
Johan Dahlin <jdahlin at async.com.br>
Async Open Source

From jdahlin at async.com.br  Mon Jul 17 15:29:22 2006
From: jdahlin at async.com.br (Johan Dahlin)
Date: Mon, 17 Jul 2006 10:29:22 -0300
Subject: [Python-Dev] Dynamic module namspaces
In-Reply-To: <5.1.1.6.0.20060715144208.02864158@sparrow.telecommunity.com>
References: <5.1.1.6.0.20060715144208.02864158@sparrow.telecommunity.com>
Message-ID: <44BB90B2.5010907@async.com.br>

Phillip J. Eby wrote:

> Just as a point of reference, the Importing package does something very
> similar, to support "weak" and "lazy" imports:
> 
> http://cheeseshop.python.org/pypi/Importing

Interesting, I was not aware of that, thanks for the pointer.
Another reason for including this feature in the standard library ;-)

> The things most likely to be problems are tools like pydoc or other
> inspect-using code that expects modules to be exactly ModuleType and
> don't use isinstance().  Apart from that, I've been using the technique
> since the early days of Python 2.2 without encountering any problems
> until the PEP 302 "reload()" bug came along, but that was fixed in
> 2.3.5.  I haven't seen any other problems since.

I'd argue that pydoc & friends are broken if they assume that a module will
always be of ModuleType and not a subclass of it.

> On the other hand, the Importing package takes a much more conservative
> approach than you are doing; it simply runs reload() on a module when
> __getattribute__ is called (after restoring the old version of
> __getattribute__).  So, as soon as you touch a lazily loaded module, it
> ceases to be particularly special, and it has a real __dict__.  It's
> possible that what you're doing could have more side-effects than what
> I'm doing.

This is an interesting approach, I thought of using that but I didn't quite
manage to find the time to implement it properly.

However, for the gtk namespace this won't be enough, since I want to avoid
creating all the types when the first one is accessed; when gtk.Button is
accessed it, gtk.Window will still not be created.

>> What I want to ask, is it possible to have a sanctioned way to implement
>> a dynamic module/namespace in python?
> 
> That would be nice, but I think that what you and I are doing are
> probably the One Obvious Ways to do the respective things we're doing.

I consider __getattribute__ a hack, being able to override __dict__ is less
hackish, IMHO.

-- 
Johan Dahlin <jdahlin at async.com.br>
Async Open Source

From anthony at interlink.com.au  Mon Jul 17 18:54:23 2006
From: anthony at interlink.com.au (Anthony Baxter)
Date: Tue, 18 Jul 2006 02:54:23 +1000
Subject: [Python-Dev] Community buildbots (was Re: User's complaints)
In-Reply-To: <44BB6610.7080307@bullseye.apana.org.au>
References: <20060713180322.29014.1145514133.divmod.quotient.35181@ohm>
	<200607171820.19936.anthony@interlink.com.au>
	<44BB6610.7080307@bullseye.apana.org.au>
Message-ID: <200607180254.26666.anthony@interlink.com.au>

On Monday 17 July 2006 20:27, Andrew MacIntyre wrote:
> On the face of it, it seems to me that branching a new major
> release at the 1st beta would be one way of managing this.  The
> trunk is not frozen for an extended period, and any "features" and
> bug fixes could probably be backported from the trunk on clearance
> from the RE with no more pain than is currently endured.
>
> One down side would be the possibility of a loss in emphasis in
> finishing the job on the release to be.  I'm sure there are
> others...

The major one is the doubling of the workload for bugfixes. SVN is 
better than CVS, but it's still not great when it comes to 
backporting fixes between branches.

Anthony
-- 
Anthony Baxter     <anthony at interlink.com.au>
It's never too late to have a happy childhood.

From glyph at divmod.com  Mon Jul 17 18:59:23 2006
From: glyph at divmod.com (glyph at divmod.com)
Date: Mon, 17 Jul 2006 12:59:23 -0400
Subject: [Python-Dev] Dynamic module namspaces
In-Reply-To: <44BB90B2.5010907@async.com.br>
Message-ID: <20060717165923.29014.1363902968.divmod.quotient.41335@ohm>

On Mon, 17 Jul 2006 10:29:22 -0300, Johan Dahlin <jdahlin at async.com.br> wrote:

>I consider __getattribute__ a hack, being able to override __dict__ is less
>hackish, IMHO.

Why do you feel one is more "hackish" than the other?  In my experience the
opposite is true: certain C APIs expect __dict__ to be a "real" dictionary,
and if you monkey with it they won't call the overridden functions you expect,
whereas things accessing attributes will generally call through all the
appropriate Python-level APIs.

This makes sense to me for efficiency reasons and for clarity as well; if you're
trawling around in a module's __dict__ then you'd better be ready for what
you're going to get - *especially* if the module is actually a package.  Even in
"normal" python code, packages can have names which would be bound if they were
imported, but aren't yet.

From jcarlson at uci.edu  Mon Jul 17 19:10:35 2006
From: jcarlson at uci.edu (Josiah Carlson)
Date: Mon, 17 Jul 2006 10:10:35 -0700
Subject: [Python-Dev] Python Style Sheets ? Re: User's complaints
In-Reply-To: <e9g156$3pd$1@sea.gmane.org>
References: <ca471dc20607141244x5c075c6bnd795493101cf19dd@mail.gmail.com>
	<e9g156$3pd$1@sea.gmane.org>
Message-ID: <20060717092746.0CB0.JCARLSON@uci.edu>


Boris Borcic <bborcic at gmail.com> wrote:
> Guido van Rossum wrote:
> > You must be misunderstanding.
> 
> I don't think so. You appeared to say that the language changes too much because 
> everyone wants different changes - that accumulate. I suggested a mechanism 
> allowing people to see only the changes they want - or none at all - might be 
> devised.

Let us imagine that such a system were devised and implemented.
Invariably user X and Y would have different sets of changes that they
want to use.  Presumably, if the features were nontrivial, then they
would no longer be able to exchange code because it would have been
directed at a different 'version' of Python, whose syntax or semantics
were different.

What you are suggesting is essentially built-in language fragmentation,
which is quite a bad idea.

 - Josiah


From guido at python.org  Mon Jul 17 19:25:09 2006
From: guido at python.org (Guido van Rossum)
Date: Mon, 17 Jul 2006 10:25:09 -0700
Subject: [Python-Dev] Problem with super() usage
In-Reply-To: <44B9F4DB.3030004@canterbury.ac.nz>
References: <44B9F4DB.3030004@canterbury.ac.nz>
Message-ID: <ca471dc20607171025k294bcac6q4585f360d04ff757@mail.gmail.com>

On 7/16/06, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
> For about the third time in my life, I thought I might
> have found a use for cooperative super calls, but I've
> run into another problem with the concept.
>
> Consider:
>
>   class A(object):
>    def m(self):
>      print "A.m"
>
>   class B(object):
>    def m(self):
>     print "B.m"
>     super(B, self).m()
>
>   class C(B, A):
>    def m(self):
>     print "C.m"
>     super(C, self).m()
>
>   >>> c = C()
>   >>> c.m()
>   C.m
>   B.m
>   A.m
>
> Okay so far, but... what if I want to use class B on
> its own, or in combination with other classes that don't
> have an m() method?
>
>  >>> b = B()
>  >>> b.m()
> B.m
> Traceback (most recent call last):
>    File "<stdin>", line 1, in ?
>    File "<stdin>", line 4, in m
> AttributeError: 'super' object has no attribute 'm'
>
> In general, how is one supposed to use super() in a
> class which may or may not be at the end of the mro
> with respect to a particular method?

One isn't. In the world where cooperative multiple inheritance
originated (C++), this would be a static error. You can only use super
when you're overriding a method, not when you're defining a new
method.

To do what you want to do in such a world, you'd have to create a base
class with a dummy implementation and inherit from that. You can do
that in Python, too.

> The only thing I can think of is to write all my
> super()-using methods defensively like this:
>
>    def m(self):
>      ...
>      s = super(C, self)
>      if hasattr(s, 'm'):
>        s.m()
>
> which seems awfully tedious.

Definitely don't that.

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From arigo at tunes.org  Mon Jul 17 20:22:35 2006
From: arigo at tunes.org (Armin Rigo)
Date: Mon, 17 Jul 2006 20:22:35 +0200
Subject: [Python-Dev] User's complaints
In-Reply-To: <3e1553560607130502o253a8359mf327964ee5d8ece8@mail.gmail.com>
References: <ee2a432c0607041649i7ecb6cc1w8ebdf47f7d02097a@mail.gmail.com>
	<3e1553560607122243q754dd71dq2e6d95cdf11e89dd@mail.gmail.com>
	<ee2a432c0607122324k633ed966n6c55a4921e529099@mail.gmail.com>
	<3e1553560607130052m66899441t50065786495928fe@mail.gmail.com>
	<44B60C0F.7060405@canterbury.ac.nz>
	<928173CD-DD5D-4B10-9CC4-ED63CC90933A@redivi.com>
	<3e1553560607130502o253a8359mf327964ee5d8ece8@mail.gmail.com>
Message-ID: <20060717182235.GA20517@code0.codespeak.net>

Hi Jeroen,

On Thu, Jul 13, 2006 at 02:02:22PM +0200, Jeroen Ruigrok van der Werven wrote:
> He doesn't specifically need the builtin types to be extendable. It's
> just nice to be able to define a single class in multiple modules.

There are various simple ways to do this; the one I'm using from time to
time is the "extendabletype" metaclass from:

   http://codespeak.net/svn/pypy/dist/pypy/annotation/pairtype.py

Example:

   class A:
       __metaclass__ = extendabletype
       def f(...): ...

Somewhere else:

   class __extend__(A):
       def g(...): ...

FWIW the above 30-lines file also contains a fast double-dispatch
multimethod implementation :-)


A bientot,

Armin

From arigo at tunes.org  Mon Jul 17 20:25:21 2006
From: arigo at tunes.org (Armin Rigo)
Date: Mon, 17 Jul 2006 20:25:21 +0200
Subject: [Python-Dev] User's complaints
In-Reply-To: <6713F515-C684-486E-8FFF-EAC60C83E069@redivi.com>
References: <ee2a432c0607041649i7ecb6cc1w8ebdf47f7d02097a@mail.gmail.com>
	<3e1553560607122243q754dd71dq2e6d95cdf11e89dd@mail.gmail.com>
	<4c45c1530607130037r1c14e3f6u972232bdfa481534@mail.gmail.com>
	<6713F515-C684-486E-8FFF-EAC60C83E069@redivi.com>
Message-ID: <20060717182521.GB20517@code0.codespeak.net>

Hi Bob,

On Thu, Jul 13, 2006 at 12:58:08AM -0700, Bob Ippolito wrote:
> > @main
> > def whatever():
> >     ...
> 
> It would probably need to be called something else, because main is  
> often the name of the main function...

Ah, but there is theoretically no name clash here :-)

    @main             # <- from the built-ins
    def main():       # <- and only then set the global
        ...


Just-making-a-stupid-point-and-not-endorsing-the-feature-ly yours,

Armin

From bob at redivi.com  Mon Jul 17 21:39:08 2006
From: bob at redivi.com (Bob Ippolito)
Date: Mon, 17 Jul 2006 12:39:08 -0700
Subject: [Python-Dev] User's complaints
In-Reply-To: <20060717182521.GB20517@code0.codespeak.net>
References: <ee2a432c0607041649i7ecb6cc1w8ebdf47f7d02097a@mail.gmail.com>
	<3e1553560607122243q754dd71dq2e6d95cdf11e89dd@mail.gmail.com>
	<4c45c1530607130037r1c14e3f6u972232bdfa481534@mail.gmail.com>
	<6713F515-C684-486E-8FFF-EAC60C83E069@redivi.com>
	<20060717182521.GB20517@code0.codespeak.net>
Message-ID: <C2984742-AD1A-4E50-80CE-7EFC85640CFA@redivi.com>


On Jul 17, 2006, at 11:25 AM, Armin Rigo wrote:

> Hi Bob,
>
> On Thu, Jul 13, 2006 at 12:58:08AM -0700, Bob Ippolito wrote:
>>> @main
>>> def whatever():
>>>     ...
>>
>> It would probably need to be called something else, because main is
>> often the name of the main function...
>
> Ah, but there is theoretically no name clash here :-)
>
>     @main             # <- from the built-ins
>     def main():       # <- and only then set the global
>         ...
>
>
> Just-making-a-stupid-point-and-not-endorsing-the-feature-ly yours,

Of course it *works*, but it's still overriding a built-in... Who  
knows when assignment to main will become a SyntaxError like None ;)

-bob


From misa at redhat.com  Mon Jul 17 21:39:55 2006
From: misa at redhat.com (Mihai Ibanescu)
Date: Mon, 17 Jul 2006 15:39:55 -0400
Subject: [Python-Dev] logging module broken because of locale
Message-ID: <20060717193955.GK3699@abulafia.devel.redhat.com>

Hi,

This is reported on sourceforge:

http://sourceforge.net/tracker/index.php?func=detail&aid=1524081&group_id=5470&atid=105470

I am willing to try and patch the problem, but I'd like to discuss my ideas
first.

The basic problem is that, in some locale, "INFO".lower() != "info". So,
initializing a dictionary with keys that appear to be lower-case in English
is unsafe for other locale.

Now, the other problem is, people can choose to initialize the locale any time
they want, and can choose to change it too. My initial approach was to create
the dictionary with strings to which lower() was explicitly called.

But, since priority_names is class-scoped, it gets evaluated at module import
time, which is way too early in the game (most likely the locale will be set
afterwards).

Any thoughts on the proper way to approach this?

Thanks,
Misa

From bborcic at gmail.com  Mon Jul 17 21:43:23 2006
From: bborcic at gmail.com (Boris Borcic)
Date: Mon, 17 Jul 2006 21:43:23 +0200
Subject: [Python-Dev] Python Style Sheets ? Re: User's complaints
In-Reply-To: <20060717092746.0CB0.JCARLSON@uci.edu>
References: <ca471dc20607141244x5c075c6bnd795493101cf19dd@mail.gmail.com>	<e9g156$3pd$1@sea.gmane.org>
	<20060717092746.0CB0.JCARLSON@uci.edu>
Message-ID: <e9gp9p$v2f$1@sea.gmane.org>

Josiah Carlson wrote:
> Boris Borcic <bborcic at gmail.com> wrote:
>> Guido van Rossum wrote:
>>> You must be misunderstanding.
>> I don't think so. You appeared to say that the language changes too much because 
>> everyone wants different changes - that accumulate. I suggested a mechanism 
>> allowing people to see only the changes they want - or none at all - might be 
>> devised.
> 
> Let us imagine that such a system were devised and implemented.

A minimal requirement to such usage of the first person plural is some effort to 
make sure "we" talk about the same thing.

> Invariably user X and Y would have different sets of changes that they
> want to use.  Presumably, if the features were nontrivial, then they
> would no longer be able to exchange code because it would have been
> directed at a different 'version' of Python, whose syntax or semantics
> were different.

Of course, and that's why in my initial post I was talking of transparent 
reversible transforms and central control of "styles" through the standard.
Means not to fall into the trap you describe. Or else I would have asked for 
macros ! Are you implying that /no/ measure of language variability can be dealt 
with by such means as standards-controlled reversible transforms ? I guess not.

I am just saying :

(1) be aware that "reversible source to source transforms" means that language 
differences can *EFFECTIVELY* be made invisible - turned to purely private 
choices (in the sense of "because there exists perfect automatic two-way 
translation between equivalent languages", *NOT* "because we don't exchange code 
written in incompatible languages").

(2) of course that frame limits language differences, but nevertheless it 
relaxes that limit relative to the current situation of "no differences", and 
please don't write it off without any effort to evaluate its possibilities.

Regards, BB
--
666 ? - 666 ~ .666 ~  1 - 1/3  ~  2/3  ~ tertium non datur ~ the excluded middle
             ~ "either with us, or against us"


From fuzzyman at voidspace.org.uk  Mon Jul 17 22:06:35 2006
From: fuzzyman at voidspace.org.uk (Michael Foord)
Date: Mon, 17 Jul 2006 21:06:35 +0100
Subject: [Python-Dev] Python Style Sheets ? Re: User's complaints
In-Reply-To: <e9gp9p$v2f$1@sea.gmane.org>
References: <ca471dc20607141244x5c075c6bnd795493101cf19dd@mail.gmail.com>	<e9g156$3pd$1@sea.gmane.org>	<20060717092746.0CB0.JCARLSON@uci.edu>
	<e9gp9p$v2f$1@sea.gmane.org>
Message-ID: <44BBEDCB.4070607@voidspace.org.uk>

Boris Borcic wrote:
>> Boris Borcic <bborcic at gmail.com> wrote:
>>     
>> [snip..]
> I am just saying :
>
> (1) be aware that "reversible source to source transforms" means that language 
> differences can *EFFECTIVELY* be made invisible - turned to purely private 
> choices (in the sense of "because there exists perfect automatic two-way 
> translation between equivalent languages", *NOT* "because we don't exchange code 
> written in incompatible languages").
>
> (2) of course that frame limits language differences, but nevertheless it 
> relaxes that limit relative to the current situation of "no differences", and 
> please don't write it off without any effort to evaluate its possibilities.
>
>   
Something similar to your suggestions are already possible through third 
part extensions like EasyExtend or Logix. (Effectively implementing a 
separate compiler.)

Readability (one way to do it...) are a central part of the Python 
philosophy. This philosophy means that it is important that code written 
by one developer should be readable by others.

Your suggestions have little to no chance of making it into the core 
Python distribution.

All the best,


Michael Foord
http://www.voidspace.org.uk/python/index.shtml

> Regards, BB
> --
> 666 ? - 666 ~ .666 ~  1 - 1/3  ~  2/3  ~ tertium non datur ~ the excluded middle
>              ~ "either with us, or against us"
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk
>
>   


From jcarlson at uci.edu  Mon Jul 17 22:10:48 2006
From: jcarlson at uci.edu (Josiah Carlson)
Date: Mon, 17 Jul 2006 13:10:48 -0700
Subject: [Python-Dev] Python Style Sheets ? Re: User's complaints
In-Reply-To: <e9gp9p$v2f$1@sea.gmane.org>
References: <20060717092746.0CB0.JCARLSON@uci.edu> <e9gp9p$v2f$1@sea.gmane.org>
Message-ID: <20060717125959.0CBF.JCARLSON@uci.edu>


Boris Borcic <bborcic at gmail.com> wrote:
> Josiah Carlson wrote:
> > Invariably user X and Y would have different sets of changes that they
> > want to use.  Presumably, if the features were nontrivial, then they
> > would no longer be able to exchange code because it would have been
> > directed at a different 'version' of Python, whose syntax or semantics
> > were different.
> 
> Of course, and that's why in my initial post I was talking of transparent 
> reversible transforms and central control of "styles" through the standard.
> Means not to fall into the trap you describe. Or else I would have asked for 
> macros ! Are you implying that /no/ measure of language variability can be dealt 
> with by such means as standards-controlled reversible transforms ? I guess not.

Regardless of the existance of reversable transforms, a user's ability
to understand and/or maintain code is dependant on the syntax and
semantics of the language.  In allowing different language variants, one
is changing the user-understood meaning of a block of code, which
necessarily increses the burden of programming and maintenance.

 - Josiah


From t-bruch at microsoft.com  Mon Jul 17 22:14:36 2006
From: t-bruch at microsoft.com (Bruce Christensen)
Date: Mon, 17 Jul 2006 13:14:36 -0700
Subject: [Python-Dev] Pickling objects that return string from reduce
Message-ID: <3581AA168D87A2479D88EA319BDF7D32E05276@RED-MSG-80.redmond.corp.microsoft.com>

I'm still plugging away on IronPython's cPickle, and I've stumbled
across another ambiguity in the pickle docs. From
http://docs.python.org/lib/node69.html:

"If a string is returned, it names a global variable whose contents are
pickled as normal. The string returned by __reduce__ should be the
object's local name relative to its module; the pickle module searches
the module namespace to determine the object's module."

What exactly does that last clause mean? Must the object have a
__module__ attribute? What if it doesn't? What if the module exists but
isn't currently imported? 

Is something like the following close?

    # Call copy_reg-registered func, obj.__reduce_ex__, or
obj.__reduce__
    result = call_reduce(obj)

    if type(result) == tuple:
        ... (do appropriate things here)
    elif isinstance(result, basestring):
        name = result
        module = "<unknown module>"
        try:
            module = obj.__module__
            found_obj = getattr(sys.modules[module], name)
        except AttributeError, KeyError:
            raise PicklingError(
                "Can't pickle %r: it's not found as %s.%s"
                % (obj, module, name)
                )

        if found_obj is not obj:
            raise PicklingError(
                "Can't pickle %r: it's not the same object as %s.%s"
                % (obj, module, name)
                )

        emit("c%s\n%s\n" % module, name)

Thanks,

--Bruce


From westymatt at gmail.com  Mon Jul 17 22:25:29 2006
From: westymatt at gmail.com (matt westerburg)
Date: Mon, 17 Jul 2006 16:25:29 -0400
Subject: [Python-Dev] new guy
Message-ID: <4b490f2f0607171325l4aec5bf5v66e44682f6ba2773@mail.gmail.com>

Hi my name is Matt Westerburg, I am a student and have only recently gotten
into Python.  But have fallen in love with the language thus far.  Fantastic
language and thank you very much for making it what it is today.  I am
looking into getting into working on Python.  Still need sometime working
with the language, but I am interested in both the standard library and the
interpreter.  Can anyone recommend a good starting place?

thank you
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20060717/9b4d9be5/attachment.htm 

From brett at python.org  Mon Jul 17 22:53:34 2006
From: brett at python.org (Brett Cannon)
Date: Mon, 17 Jul 2006 13:53:34 -0700
Subject: [Python-Dev] new guy
In-Reply-To: <4b490f2f0607171325l4aec5bf5v66e44682f6ba2773@mail.gmail.com>
References: <4b490f2f0607171325l4aec5bf5v66e44682f6ba2773@mail.gmail.com>
Message-ID: <bbaeab100607171353y2a7b61fak2f1af4df2210d424@mail.gmail.com>

On 7/17/06, matt westerburg <westymatt at gmail.com> wrote:
>
> Hi my name is Matt Westerburg, I am a student and have only recently
> gotten into Python.  But have fallen in love with the language thus far.
> Fantastic language and thank you very much for making it what it is today.
> I am looking into getting into working on Python.  Still need sometime
> working with the language, but I am interested in both the standard library
> and the interpreter.  Can anyone recommend a good starting place?
>


First, welcome!

Second, in terms of a good starting point, you can read
http://www.python.org/dev/intro for an introduction on how things work
around here (roughly) and what it takes to get something in.

Really the best way to get introduced is to read this mailing list and to
dive into some source (reviewing a patch can be a good way to go about
this).

-Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20060717/1b9467d2/attachment.html 

From martin at v.loewis.de  Mon Jul 17 23:04:27 2006
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Mon, 17 Jul 2006 23:04:27 +0200
Subject: [Python-Dev] Pickling objects that return string from reduce
In-Reply-To: <3581AA168D87A2479D88EA319BDF7D32E05276@RED-MSG-80.redmond.corp.microsoft.com>
References: <3581AA168D87A2479D88EA319BDF7D32E05276@RED-MSG-80.redmond.corp.microsoft.com>
Message-ID: <44BBFB5B.2090806@v.loewis.de>

Bruce Christensen wrote:
> Is something like the following close?

Close, yes.

>     if type(result) == tuple:
>         ... (do appropriate things here)
>     elif isinstance(result, basestring):
>         name = result
>         module = "<unknown module>"
>         try:
>             module = obj.__module__
>             found_obj = getattr(sys.modules[module], name)
>         except AttributeError, KeyError:
>             raise PicklingError(
>                 "Can't pickle %r: it's not found as %s.%s"
>                 % (obj, module, name)
>                 )

If  obj has no __module__ attribute (or if it is None), pickle
(didn't check cPickle) also does

for n, module in sys.module.items():
  if "module-ignored": continue
  if getattr(module, result, None) is obj:
     break # use n as module name

If obj does have a __module__ attribute, it uses __import__
to import the module, just to make sure it gets into sys.modules.

Otherwise, it looks correct.

Regards,
Martin

From martin at v.loewis.de  Mon Jul 17 23:10:00 2006
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Mon, 17 Jul 2006 23:10:00 +0200
Subject: [Python-Dev] new guy
In-Reply-To: <4b490f2f0607171325l4aec5bf5v66e44682f6ba2773@mail.gmail.com>
References: <4b490f2f0607171325l4aec5bf5v66e44682f6ba2773@mail.gmail.com>
Message-ID: <44BBFCA8.4080503@v.loewis.de>

matt westerburg wrote:
> Hi my name is Matt Westerburg, I am a student and have only recently
> gotten into Python.  But have fallen in love with the language thus
> far.  Fantastic language and thank you very much for making it what it
> is today.  I am looking into getting into working on Python.  Still need
> sometime working with the language, but I am interested in both the
> standard library and the interpreter.  Can anyone recommend a good
> starting place?

As Brett says, reviewing bugs and patches would be most appreciated.
Review a patch, and either tell the submitter how the patch should
be improved, or tell us that you think the patch is ready to be applied
(make sure the bug has documentation and/or test suite changes as
necessary).

Review a bug, and determine that it is
a) reproducable
b) constitutes a true problem (rather than a misunderstanding on
   the submitter's side);
if both hold, try to come up with a patch. If it is not reproducable,
tell the submitter and ask for more information. If it is not the
true problem, tell us that you think it deserves to be closed.

If you are uncertain which bug or patch to review, select one that
a) is currently unassigned
b) about a year old
c) in an area of your personal interests
Property a) and b) combined should guarantee that nobody beats you
in fixing the problem while you are researching it.

Regards,
Martin

From greg.ewing at canterbury.ac.nz  Tue Jul 18 09:46:40 2006
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Tue, 18 Jul 2006 19:46:40 +1200
Subject: [Python-Dev] Problem with super() usage
In-Reply-To: <ca471dc20607171025k294bcac6q4585f360d04ff757@mail.gmail.com>
References: <44B9F4DB.3030004@canterbury.ac.nz>
	<ca471dc20607171025k294bcac6q4585f360d04ff757@mail.gmail.com>
Message-ID: <44BC91E0.2040804@canterbury.ac.nz>

Guido van Rossum wrote:
> In the world where cooperative multiple inheritance
> originated (C++), this would be a static error.

I wasn't aware that C++ had anything resembling super().
Is it a recent addition to the language?

--
Greg

From bborcic at gmail.com  Tue Jul 18 13:36:41 2006
From: bborcic at gmail.com (Boris Borcic)
Date: Tue, 18 Jul 2006 13:36:41 +0200
Subject: [Python-Dev] Python Style Sheets ? Re: User's complaints
In-Reply-To: <20060717125959.0CBF.JCARLSON@uci.edu>
References: <20060717092746.0CB0.JCARLSON@uci.edu> <e9gp9p$v2f$1@sea.gmane.org>
	<20060717125959.0CBF.JCARLSON@uci.edu>
Message-ID: <e9ih56$qmu$1@sea.gmane.org>

Josiah Carlson wrote:
> Boris Borcic <bborcic at gmail.com> wrote:
>> Of course, and that's why in my initial post I was talking of transparent 
>> reversible transforms and central control of "styles" through the standard.
>> Means not to fall into the trap you describe. Or else I would have asked for 
>> macros ! Are you implying that /no/ measure of language variability can be dealt 
>> with by such means as standards-controlled reversible transforms ? I guess not.
> 
> Regardless of the existance of reversable transforms, a user's ability
> to understand and/or maintain code is dependant on the syntax and
> semantics of the language.

If you have an effective isomorphism, that's irrelevant. Everybody works with 
the language she understands. Ever tried a slide rule ?

> In allowing different language variants, one
> is changing the user-understood meaning of a block of code, which
> necessarily increses the burden of programming and maintenance.

Allowing different language variants connected by reversible transforms means 
one need not change any user's understood meaning of any block of code. The user 
stipulates the language variant she likes and the system translates 
back-and-forth from/to distinct variants other users might prefer.

- BB
--
666 ? - 666 ~ .666 ~ 2/3 ~ 1-1/3 ~ tertium non datur ~ the excluded middle
             ~ "either with us, or against us"


From misa at redhat.com  Tue Jul 18 14:40:19 2006
From: misa at redhat.com (Mihai Ibanescu)
Date: Tue, 18 Jul 2006 08:40:19 -0400
Subject: [Python-Dev] logging module broken because of locale
In-Reply-To: <20060717193955.GK3699@abulafia.devel.redhat.com>
References: <20060717193955.GK3699@abulafia.devel.redhat.com>
Message-ID: <20060718124019.GB16294@abulafia.devel.redhat.com>

On Mon, Jul 17, 2006 at 03:39:55PM -0400, Mihai Ibanescu wrote:
> Hi,
> 
> This is reported on sourceforge:
> 
> http://sourceforge.net/tracker/index.php?func=detail&aid=1524081&group_id=5470&atid=105470
> 
> I am willing to try and patch the problem, but I'd like to discuss my ideas
> first.
> 
> The basic problem is that, in some locale, "INFO".lower() != "info". So,
> initializing a dictionary with keys that appear to be lower-case in English
> is unsafe for other locale.
> 
> Now, the other problem is, people can choose to initialize the locale any time
> they want, and can choose to change it too. My initial approach was to create
> the dictionary with strings to which lower() was explicitly called.
> 
> But, since priority_names is class-scoped, it gets evaluated at module import
> time, which is way too early in the game (most likely the locale will be set
> afterwards).
> 
> Any thoughts on the proper way to approach this?

To follow up on my own email: it looks like, even though in some locale
"INFO".lower() != "info"

u"INFO".lower() == "info" (at least in the Turkish locale).

Is that guaranteed, at least for now (for the current versions of python)?

Thanks,
Misa

From aahz at pythoncraft.com  Tue Jul 18 14:50:45 2006
From: aahz at pythoncraft.com (Aahz)
Date: Tue, 18 Jul 2006 05:50:45 -0700
Subject: [Python-Dev] Python Style Sheets ? Re: User's complaints
In-Reply-To: <e9ih56$qmu$1@sea.gmane.org>
References: <20060717092746.0CB0.JCARLSON@uci.edu> <e9gp9p$v2f$1@sea.gmane.org>
	<20060717125959.0CBF.JCARLSON@uci.edu> <e9ih56$qmu$1@sea.gmane.org>
Message-ID: <20060718125045.GA17271@panix.com>

Please.  Just end this discussion.  It ain't gonna happen.
-- 
Aahz (aahz at pythoncraft.com)           <*>         http://www.pythoncraft.com/

"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are, by
definition, not smart enough to debug it."  --Brian W. Kernighan

From scott+python-dev at scottdial.com  Tue Jul 18 15:10:11 2006
From: scott+python-dev at scottdial.com (Scott Dial)
Date: Tue, 18 Jul 2006 09:10:11 -0400
Subject: [Python-Dev] Problem with super() usage
In-Reply-To: <44BC91E0.2040804@canterbury.ac.nz>
References: <44B9F4DB.3030004@canterbury.ac.nz>	<ca471dc20607171025k294bcac6q4585f360d04ff757@mail.gmail.com>
	<44BC91E0.2040804@canterbury.ac.nz>
Message-ID: <44BCDDB3.6090607@scottdial.com>

Greg Ewing wrote:
> Guido van Rossum wrote:
>> In the world where cooperative multiple inheritance
>> originated (C++), this would be a static error.
> 
> I wasn't aware that C++ had anything resembling super().
> Is it a recent addition to the language?
> 

It is much more explicit, but you call the function from the 
superclass's namespace:

class B : public A {
public:
   void m(void) {
     A::m(); // The call to my super
   };
};

C++ has no concept of MRO, so "super()" would be completely ambiguous. 
In fact, if you try to replicate your code in C++ using a generic M 
class (which defines a dummy m method), you'll get such an error from 
your compiler: "`M' is an ambiguous base of `C'"

This is very much a dynamic language quirk that you can call out to a 
function that may or may not exist, and we should avoid comparing it to 
other languages which don't allow it. I agree with Guido that in python, 
the reasonable fix is to have a superclass which defines an empty method.

-- 
Scott Dial
scott at scottdial.com
scodial at indiana.edu

From exarkun at divmod.com  Tue Jul 18 15:24:57 2006
From: exarkun at divmod.com (Jean-Paul Calderone)
Date: Tue, 18 Jul 2006 09:24:57 -0400
Subject: [Python-Dev] Problem with super() usage
In-Reply-To: <44BCDDB3.6090607@scottdial.com>
Message-ID: <20060718132457.29014.276758938.divmod.quotient.42664@ohm>

On Tue, 18 Jul 2006 09:10:11 -0400, Scott Dial <scott+python-dev at scottdial.com> wrote:
>Greg Ewing wrote:
>> Guido van Rossum wrote:
>>> In the world where cooperative multiple inheritance
>>> originated (C++), this would be a static error.
>>
>> I wasn't aware that C++ had anything resembling super().
>> Is it a recent addition to the language?
>>
>
>It is much more explicit, but you call the function from the
>superclass's namespace:
>
> [snip - something totally unlike super]
>
>C++ has no concept of MRO, so "super()" would be completely ambiguous.

I think this was Greg's point.  Talking about C++ and super() is
nonsensical.

Jean-Paul

From glyph at divmod.com  Tue Jul 18 17:59:39 2006
From: glyph at divmod.com (glyph at divmod.com)
Date: Tue, 18 Jul 2006 11:59:39 -0400
Subject: [Python-Dev] Problem with super() usage
In-Reply-To: <20060718132457.29014.276758938.divmod.quotient.42664@ohm>
Message-ID: <20060718155939.29014.1269904967.divmod.quotient.42845@ohm>



On Tue, 18 Jul 2006 09:24:57 -0400, Jean-Paul Calderone <exarkun at divmod.com> wrote:
>On Tue, 18 Jul 2006 09:10:11 -0400, Scott Dial <scott+python-dev at scottdial.com> wrote:
>>Greg Ewing wrote:
>>> Guido van Rossum wrote:
>>>> In the world where cooperative multiple inheritance
>>>> originated (C++), this would be a static error.

>>> I wasn't aware that C++ had anything resembling super().
>>> Is it a recent addition to the language?

>>C++ has no concept of MRO, so "super()" would be completely ambiguous.

>I think this was Greg's point.  Talking about C++ and super() is
>nonsensical.

C++ originally specified multiple inheritance, but it wasn't "cooperative" in
the sense that super is.  In Lisp, though, where cooperative method dispatch
originated, call-next-method does basically the same thing in the case where
there's no next method: it calls "no-next-method" which signals a generic
error.

    http://www.lisp.org/HyperSpec/Body/locfun_call-next-method.html

However, you can write methods for no-next-method, so you can override that
behavior as appropriate.  In Python you might achieve a similar effect using a
hack like the one Greg suggested, but in a slightly more systematic way; using
Python's regular inheritance-based method ordering, of course, not bothering 
with multimethods.  Stand-alone it looks like an awful hack, but with a bit
of scaffolding I think it looks nice enough; at least, it looks like the Lisp
solution, which while potentially ugly, is complete :).

This is just implemented as a function for brevity; you could obviously use a
proxy object with all the features of 'super', including optional self,
method getters, etc.  For cooperative classes you could implement noNextMethod
to always be OK, or to provide an appropriate "null" value for a type map of
a method's indicated return value ('' for str, 0 for int, None for object, 
etc).

# ---cut here---

def callNextMethod(cls, self, methodName, *a, **k):
    sup = super(cls, self)
    method = getattr(sup, methodName, None)
    if method is not None:
        return method(*a, **k)
    else:
        return self.noNextMethod(methodName, *a, **k)

class NextMethodHelper(object):
    def noNextMethod(self, methodName, *a, **k):
        return getattr(self, "noNext_"+methodName)(*a, **k)

class A(object):
    def m(self):
        print "A.m"

class B(NextMethodHelper):
    def m(self):
        print "B.m"
        return callNextMethod(B, self, "m")

    def noNext_m(self):
        # it's ok not to have an 'm'!
        print "No next M, but that's OK!"
        return None

class C(B, A):
    def m(self):
        print "C.m"
        return callNextMethod(C, self, "m")


#>>> c = C()
#>>> c.m()
#C.m
#B.m
#A.m
#>>> b = B()
#>>> b.m()
#B.m
#No next M, but that's OK!

From tjreedy at udel.edu  Tue Jul 18 18:19:08 2006
From: tjreedy at udel.edu (Terry Reedy)
Date: Tue, 18 Jul 2006 12:19:08 -0400
Subject: [Python-Dev] Python Style Sheets ? Re: User's complaints
References: <20060717092746.0CB0.JCARLSON@uci.edu>
	<e9gp9p$v2f$1@sea.gmane.org><20060717125959.0CBF.JCARLSON@uci.edu>
	<e9ih56$qmu$1@sea.gmane.org>
Message-ID: <e9j1lt$ncj$1@sea.gmane.org>


"Boris Borcic" <bborcic at gmail.com> wrote in message 
news:e9ih56$qmu$1 at sea.gmane.org...
> Allowing different language variants connected by reversible transforms 
> means
> one need not change any user's understood meaning of any block of code. 
> The user
> stipulates the language variant she likes and the system translates
> back-and-forth from/to distinct variants other users might prefer.

Editing systems like this are and should remain 3rd-party add-ons.  This 
discussion belongs on the general python list (clp) or on the discussion 
list of current projects in this area.

tjr




From metawilm at gmail.com  Tue Jul 18 18:22:11 2006
From: metawilm at gmail.com (Willem Broekema)
Date: Tue, 18 Jul 2006 18:22:11 +0200
Subject: [Python-Dev] Problem with super() usage
In-Reply-To: <20060718155939.29014.1269904967.divmod.quotient.42845@ohm>
References: <20060718132457.29014.276758938.divmod.quotient.42664@ohm>
	<20060718155939.29014.1269904967.divmod.quotient.42845@ohm>
Message-ID: <f6bc9b490607180922n3dd01e62ja9ce17464c0ded50@mail.gmail.com>

On 7/18/06, glyph at divmod.com <glyph at divmod.com> wrote:
> C++ originally specified multiple inheritance, but it wasn't "cooperative" in
> the sense that super is.  In Lisp, though, where cooperative method dispatch
> originated, call-next-method does basically the same thing in the case where
> there's no next method: it calls "no-next-method" which signals a generic
> error.

Don't forget Lisp's "next-method-p", which tests ("-p" for
"predicate") if there is any next method to call. Highly elegant, I'd
say.

> http://www.lisp.org/HyperSpec/Body/locfun_call-next-method.html

http://www.lisp.org/HyperSpec/Body/locfun_next-method-p.html

- Willem

From guido at python.org  Tue Jul 18 18:58:54 2006
From: guido at python.org (Guido van Rossum)
Date: Tue, 18 Jul 2006 09:58:54 -0700
Subject: [Python-Dev] Problem with super() usage
In-Reply-To: <44BC91E0.2040804@canterbury.ac.nz>
References: <44B9F4DB.3030004@canterbury.ac.nz>
	<ca471dc20607171025k294bcac6q4585f360d04ff757@mail.gmail.com>
	<44BC91E0.2040804@canterbury.ac.nz>
Message-ID: <ca471dc20607180958l52a2f3a0v2d6da68d7dfdc6b4@mail.gmail.com>

On 7/18/06, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
> Guido van Rossum wrote:
> > In the world where cooperative multiple inheritance
> > originated (C++), this would be a static error.
>
> I wasn't aware that C++ had anything resembling super().
> Is it a recent addition to the language?

I don't know about pure C++, but the C++ variant (or was it a
library?) that originated the cooperative multitasking ideas does have
a super call. And it is strongly typed as I explained.

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From guido at python.org  Tue Jul 18 19:02:20 2006
From: guido at python.org (Guido van Rossum)
Date: Tue, 18 Jul 2006 10:02:20 -0700
Subject: [Python-Dev] Problem with super() usage
In-Reply-To: <20060718132457.29014.276758938.divmod.quotient.42664@ohm>
References: <44BCDDB3.6090607@scottdial.com>
	<20060718132457.29014.276758938.divmod.quotient.42664@ohm>
Message-ID: <ca471dc20607181002h1e290fean1d8e909b9c124fd1@mail.gmail.com>

On 7/18/06, Jean-Paul Calderone <exarkun at divmod.com> wrote:
> I think this was Greg's point.  Talking about C++ and super() is
> nonsensical.

If you're talking pure C++, yes. But I was talking about programming
system built on top of C++ implementing cooperating multitasking.

As I am fond of repeating, if the ideas originally came from Lisp, I
don't know -- I got them from this book:

Putting Metaclasses to Work: A New Dimension in Object-Oriented
Programming, by Ira R. Forman and Scott H. Danforth. Addison-Wesley,
1999, ISBN 0-201-43305-2.

It uses C++ (or perhaps a language very much like it), not Lisp. And
it has super.

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From guido at python.org  Tue Jul 18 19:04:32 2006
From: guido at python.org (Guido van Rossum)
Date: Tue, 18 Jul 2006 10:04:32 -0700
Subject: [Python-Dev] Python Style Sheets ? Re: User's complaints
In-Reply-To: <e9g156$3pd$1@sea.gmane.org>
References: <ee2a432c0607041649i7ecb6cc1w8ebdf47f7d02097a@mail.gmail.com>
	<3e1553560607122243q754dd71dq2e6d95cdf11e89dd@mail.gmail.com>
	<ca471dc20607130720q3fe1b363y533a7bff46579174@mail.gmail.com>
	<e9838o$fpc$1@sea.gmane.org>
	<ca471dc20607141244x5c075c6bnd795493101cf19dd@mail.gmail.com>
	<e9g156$3pd$1@sea.gmane.org>
Message-ID: <ca471dc20607181004l46b02012x61e745b41a201f2@mail.gmail.com>

On 7/17/06, Boris Borcic <bborcic at gmail.com> wrote:
> Guido van Rossum wrote:
> > You must be misunderstanding.
>
> I don't think so. You appeared to say that the language changes too much because
> everyone wants different changes - that accumulate. I suggested a mechanism
> allowing people to see only the changes they want - or none at all - might be
> devised.

Oh, but don't fool yourself into thinking that that's not a language
change -- and an extremely drastic one at that. Nobody will be able to
read other people's source code any more without first applying the
conversion tool -- which is kind of painful when you're confronted
with a code snippet in email or a book, for example.

Let me rephrase that. I don't believe your proposal solves that
problem, *and* I don't think it's a good idea for other reasons. Am I
clear now? :-)

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From guido at python.org  Tue Jul 18 19:19:54 2006
From: guido at python.org (Guido van Rossum)
Date: Tue, 18 Jul 2006 10:19:54 -0700
Subject: [Python-Dev] logging module broken because of locale
In-Reply-To: <20060718124019.GB16294@abulafia.devel.redhat.com>
References: <20060717193955.GK3699@abulafia.devel.redhat.com>
	<20060718124019.GB16294@abulafia.devel.redhat.com>
Message-ID: <ca471dc20607181019y700f8f9u9e2e772e2a1f533e@mail.gmail.com>

Alternatively, does "info".upper() == "INFO" everywhere?

On 7/18/06, Mihai Ibanescu <misa at redhat.com> wrote:
> On Mon, Jul 17, 2006 at 03:39:55PM -0400, Mihai Ibanescu wrote:
> > Hi,
> >
> > This is reported on sourceforge:
> >
> > http://sourceforge.net/tracker/index.php?func=detail&aid=1524081&group_id=5470&atid=105470
> >
> > I am willing to try and patch the problem, but I'd like to discuss my ideas
> > first.
> >
> > The basic problem is that, in some locale, "INFO".lower() != "info". So,
> > initializing a dictionary with keys that appear to be lower-case in English
> > is unsafe for other locale.
> >
> > Now, the other problem is, people can choose to initialize the locale any time
> > they want, and can choose to change it too. My initial approach was to create
> > the dictionary with strings to which lower() was explicitly called.
> >
> > But, since priority_names is class-scoped, it gets evaluated at module import
> > time, which is way too early in the game (most likely the locale will be set
> > afterwards).
> >
> > Any thoughts on the proper way to approach this?
>
> To follow up on my own email: it looks like, even though in some locale
> "INFO".lower() != "info"
>
> u"INFO".lower() == "info" (at least in the Turkish locale).
>
> Is that guaranteed, at least for now (for the current versions of python)?
>
> Thanks,
> Misa
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org
>


-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From misa at redhat.com  Tue Jul 18 19:45:41 2006
From: misa at redhat.com (Mihai Ibanescu)
Date: Tue, 18 Jul 2006 13:45:41 -0400
Subject: [Python-Dev] logging module broken because of locale
In-Reply-To: <ca471dc20607181019y700f8f9u9e2e772e2a1f533e@mail.gmail.com>
References: <20060717193955.GK3699@abulafia.devel.redhat.com>
	<20060718124019.GB16294@abulafia.devel.redhat.com>
	<ca471dc20607181019y700f8f9u9e2e772e2a1f533e@mail.gmail.com>
Message-ID: <20060718174541.GI27181@abulafia.devel.redhat.com>

On Tue, Jul 18, 2006 at 10:19:54AM -0700, Guido van Rossum wrote:
> Alternatively, does "info".upper() == "INFO" everywhere?

Not in the Turkish locale :-(

# begin /tmp/foo.py
import locale

locale.setlocale(locale.LC_ALL, '')

print "info".upper()
print "info".upper() == "INFO"
# end /tmp/foo.py

LANG=tr_TR.UTF-8 python /tmp/foo.py
iNFO
False

Thanks,
Misa

From guido at python.org  Tue Jul 18 19:53:23 2006
From: guido at python.org (Guido van Rossum)
Date: Tue, 18 Jul 2006 10:53:23 -0700
Subject: [Python-Dev] logging module broken because of locale
In-Reply-To: <20060718174541.GI27181@abulafia.devel.redhat.com>
References: <20060717193955.GK3699@abulafia.devel.redhat.com>
	<20060718124019.GB16294@abulafia.devel.redhat.com>
	<ca471dc20607181019y700f8f9u9e2e772e2a1f533e@mail.gmail.com>
	<20060718174541.GI27181@abulafia.devel.redhat.com>
Message-ID: <ca471dc20607181053v1ac6412dw602a6ee0739e1e72@mail.gmail.com>

And u"info".upper()?

On 7/18/06, Mihai Ibanescu <misa at redhat.com> wrote:
> On Tue, Jul 18, 2006 at 10:19:54AM -0700, Guido van Rossum wrote:
> > Alternatively, does "info".upper() == "INFO" everywhere?
>
> Not in the Turkish locale :-(
>
> # begin /tmp/foo.py
> import locale
>
> locale.setlocale(locale.LC_ALL, '')
>
> print "info".upper()
> print "info".upper() == "INFO"
> # end /tmp/foo.py
>
> LANG=tr_TR.UTF-8 python /tmp/foo.py
> iNFO
> False
>
> Thanks,
> Misa
>


-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From martin at v.loewis.de  Tue Jul 18 19:54:28 2006
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Tue, 18 Jul 2006 19:54:28 +0200
Subject: [Python-Dev] logging module broken because of locale
In-Reply-To: <20060718124019.GB16294@abulafia.devel.redhat.com>
References: <20060717193955.GK3699@abulafia.devel.redhat.com>
	<20060718124019.GB16294@abulafia.devel.redhat.com>
Message-ID: <44BD2054.4020705@v.loewis.de>

Mihai Ibanescu wrote:
> To follow up on my own email: it looks like, even though in some locale
> "INFO".lower() != "info"
> 
> u"INFO".lower() == "info" (at least in the Turkish locale).
> 
> Is that guaranteed, at least for now (for the current versions of python)?

It's guaranteed for now; unicode.lower is not locale-aware.

Regards,
Martin

From foom at fuhm.net  Tue Jul 18 20:01:09 2006
From: foom at fuhm.net (James Y Knight)
Date: Tue, 18 Jul 2006 14:01:09 -0400
Subject: [Python-Dev] logging module broken because of locale
In-Reply-To: <44BD2054.4020705@v.loewis.de>
References: <20060717193955.GK3699@abulafia.devel.redhat.com>
	<20060718124019.GB16294@abulafia.devel.redhat.com>
	<44BD2054.4020705@v.loewis.de>
Message-ID: <9AC7F55A-4106-414F-9C20-64FFFD8C6131@fuhm.net>

On Jul 18, 2006, at 1:54 PM, Martin v. L?wis wrote:

> Mihai Ibanescu wrote:
>> To follow up on my own email: it looks like, even though in some  
>> locale
>> "INFO".lower() != "info"
>>
>> u"INFO".lower() == "info" (at least in the Turkish locale).
>>
>> Is that guaranteed, at least for now (for the current versions of  
>> python)?
>
> It's guaranteed for now; unicode.lower is not locale-aware.

That seems backwards of how it should be ideally: the byte-string  
upper and lower should always do ascii uppering-and-lowering, and the  
unicode ones should do it according to locale. Perhaps that can be  
cleaned up in py3k?

James

From mal at egenix.com  Tue Jul 18 20:38:16 2006
From: mal at egenix.com (M.-A. Lemburg)
Date: Tue, 18 Jul 2006 20:38:16 +0200
Subject: [Python-Dev] logging module broken because of locale
In-Reply-To: <9AC7F55A-4106-414F-9C20-64FFFD8C6131@fuhm.net>
References: <20060717193955.GK3699@abulafia.devel.redhat.com>	<20060718124019.GB16294@abulafia.devel.redhat.com>	<44BD2054.4020705@v.loewis.de>
	<9AC7F55A-4106-414F-9C20-64FFFD8C6131@fuhm.net>
Message-ID: <44BD2A98.7000705@egenix.com>

James Y Knight wrote:
> On Jul 18, 2006, at 1:54 PM, Martin v. L?wis wrote:
> 
>> Mihai Ibanescu wrote:
>>> To follow up on my own email: it looks like, even though in some  
>>> locale
>>> "INFO".lower() != "info"
>>>
>>> u"INFO".lower() == "info" (at least in the Turkish locale).
>>>
>>> Is that guaranteed, at least for now (for the current versions of  
>>> python)?
>> It's guaranteed for now; unicode.lower is not locale-aware.
> 
> That seems backwards of how it should be ideally: the byte-string  
> upper and lower should always do ascii uppering-and-lowering, and the  
> unicode ones should do it according to locale. Perhaps that can be  
> cleaned up in py3k?

Actually, you've got that backwards ;-) ...

There are no .lower()/.upper() methods for bytes.

The reason these methods are locale aware for 8-bit strings
lies in the fact that we're using the C lib functions, which
are locale setting dependent - with all the drawbacks that
go with it.

The Unicode database OTOH *defines* the upper/lower case mapping in
a locale independent way, so the mappings are guaranteed
to always produce the same results on all platforms.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Jul 18 2006)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! ::::

From guido at python.org  Tue Jul 18 20:40:26 2006
From: guido at python.org (Guido van Rossum)
Date: Tue, 18 Jul 2006 11:40:26 -0700
Subject: [Python-Dev] logging module broken because of locale
In-Reply-To: <9AC7F55A-4106-414F-9C20-64FFFD8C6131@fuhm.net>
References: <20060717193955.GK3699@abulafia.devel.redhat.com>
	<20060718124019.GB16294@abulafia.devel.redhat.com>
	<44BD2054.4020705@v.loewis.de>
	<9AC7F55A-4106-414F-9C20-64FFFD8C6131@fuhm.net>
Message-ID: <ca471dc20607181140l2639471br6913705f6e5bcc47@mail.gmail.com>

On 7/18/06, James Y Knight <foom at fuhm.net> wrote:
> On Jul 18, 2006, at 1:54 PM, Martin v. L?wis wrote:
>
> > Mihai Ibanescu wrote:
> >> To follow up on my own email: it looks like, even though in some
> >> locale
> >> "INFO".lower() != "info"
> >>
> >> u"INFO".lower() == "info" (at least in the Turkish locale).
> >>
> >> Is that guaranteed, at least for now (for the current versions of
> >> python)?
> >
> > It's guaranteed for now; unicode.lower is not locale-aware.
>
> That seems backwards of how it should be ideally: the byte-string
> upper and lower should always do ascii uppering-and-lowering, and the
> unicode ones should do it according to locale. Perhaps that can be
> cleaned up in py3k?

No, you've got it backwards. 8-bit strings are assumed to be encoded
using the current locale's default encoding so upper and lower behave
locale-dependent. Unicode strings don't need the locale as additional
input for upper and lower; in a different locale you simply use a
different code point. I believe the original issue was that in
Turkish, there are i's with and without dots in both lower and upper
case. I'm guessing that the ASCII code points are used for lowercase
dotted i and uppercase undotted I; code points with the high bit set
are used for the uppercase dotted i and lowercase undotted I (which I
can't easily type here).

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From misa at redhat.com  Tue Jul 18 20:46:04 2006
From: misa at redhat.com (Mihai Ibanescu)
Date: Tue, 18 Jul 2006 14:46:04 -0400
Subject: [Python-Dev] logging module broken because of locale
In-Reply-To: <ca471dc20607181053v1ac6412dw602a6ee0739e1e72@mail.gmail.com>
References: <20060717193955.GK3699@abulafia.devel.redhat.com>
	<20060718124019.GB16294@abulafia.devel.redhat.com>
	<ca471dc20607181019y700f8f9u9e2e772e2a1f533e@mail.gmail.com>
	<20060718174541.GI27181@abulafia.devel.redhat.com>
	<ca471dc20607181053v1ac6412dw602a6ee0739e1e72@mail.gmail.com>
Message-ID: <20060718184604.GA363@abulafia.devel.redhat.com>

On Tue, Jul 18, 2006 at 10:53:23AM -0700, Guido van Rossum wrote:
> And u"info".upper()?

Yepp, that shows the right thing (at least in the several locales I tested,
Turkish included).

It's along the lines of u"INFO".lower() I was proposing in my second post :-)

Misa

From misa at redhat.com  Tue Jul 18 20:52:38 2006
From: misa at redhat.com (Mihai Ibanescu)
Date: Tue, 18 Jul 2006 14:52:38 -0400
Subject: [Python-Dev] logging module broken because of locale
In-Reply-To: <44BD2054.4020705@v.loewis.de>
References: <20060717193955.GK3699@abulafia.devel.redhat.com>
	<20060718124019.GB16294@abulafia.devel.redhat.com>
	<44BD2054.4020705@v.loewis.de>
Message-ID: <20060718185238.GB363@abulafia.devel.redhat.com>

On Tue, Jul 18, 2006 at 07:54:28PM +0200, "Martin v. L?wis" wrote:
> Mihai Ibanescu wrote:
> > To follow up on my own email: it looks like, even though in some locale
> > "INFO".lower() != "info"
> > 
> > u"INFO".lower() == "info" (at least in the Turkish locale).
> > 
> > Is that guaranteed, at least for now (for the current versions of python)?
> 
> It's guaranteed for now; unicode.lower is not locale-aware.

OK, should I write a patch for the logging module to convert the string to
unicode before applying lower()? So far that seems like the way to go.

Maybe this could also be explained in the documentation:

http://docs.python.org/lib/node323.html

I don't think I've seen it in the locale documentation that locale settings do
not affect unicode strings, and that particular page says 

<quote>
If, when coding a module for general use, you need a locale independent version
of an operation that is affected by the locale (such as string.lower(), or
certain formats used with time.strftime()), you will have to find a way to do
it without using the standard library routine.
</quote>

Unicode might be a perfectly acceptable suggestion for others too.

Misa

From fdrake at acm.org  Tue Jul 18 20:55:56 2006
From: fdrake at acm.org (Fred L. Drake, Jr.)
Date: Tue, 18 Jul 2006 14:55:56 -0400
Subject: [Python-Dev] logging module broken because of locale
In-Reply-To: <20060718185238.GB363@abulafia.devel.redhat.com>
References: <20060717193955.GK3699@abulafia.devel.redhat.com>
	<44BD2054.4020705@v.loewis.de>
	<20060718185238.GB363@abulafia.devel.redhat.com>
Message-ID: <200607181455.56535.fdrake@acm.org>

On Tuesday 18 July 2006 14:52, Mihai Ibanescu wrote:
 > Unicode might be a perfectly acceptable suggestion for others too.

Are we still supporting builds that don't include Unicode?  If so, that needs 
to be considered in a patch as well.


  -Fred

-- 
Fred L. Drake, Jr.   <fdrake at acm.org>

From martin at v.loewis.de  Tue Jul 18 21:26:28 2006
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Tue, 18 Jul 2006 21:26:28 +0200
Subject: [Python-Dev] logging module broken because of locale
In-Reply-To: <9AC7F55A-4106-414F-9C20-64FFFD8C6131@fuhm.net>
References: <20060717193955.GK3699@abulafia.devel.redhat.com>
	<20060718124019.GB16294@abulafia.devel.redhat.com>
	<44BD2054.4020705@v.loewis.de>
	<9AC7F55A-4106-414F-9C20-64FFFD8C6131@fuhm.net>
Message-ID: <44BD35E4.3050402@v.loewis.de>

James Y Knight wrote:
> That seems backwards of how it should be ideally: the byte-string upper
> and lower should always do ascii uppering-and-lowering, and the unicode
> ones should do it according to locale. Perhaps that can be cleaned up in
> py3k?

Cleaned-up, yes. But it is currently not backwards.

For a byte string, you need an encoding, which comes from the locale.
So for byte strings, case-conversion *has* to be locale-aware (in
principle, making it encoding-aware only would almost suffice, but
there is no universal API for that).

OTOH, for Unicode, due to the unification, case-conversion mostly
does not need to be locale-aware. Nearly all case-conversions are
only script-dependent, not language-dependent. So it is nearly possible
to make case-conversion locale-independent, and that is what Python
provides.

The "nearly" above refers to *very* few exceptions, in *very*
few languages. Most of the details are collected in UAX#21, some
highlights are:
- case conversions are not always reversible
- sometimes, case conversion may convert a single
  character to multiple characters; the canonical
  example is German ? (considered lower-case) -> "SS"
  (historically, this is just typographical, since there
   is no upper case sharp s in our script)
- sometimes, conversion depends on the position of
  the letter in the word, see Final_Sigma
  in SpecialCasing.txt, or on the subsequent
  combining accents, see Lithuanian More_Above

I believe the unicode.lower behaviour is currently right
for most applications, so it should continue to be the
default. An additional locale-aware version should be added,
but that probably means to incorporate ICU into Python,
to get this and other locale properties right in a
platform-independent fashion.

Regards,
Martin


From martin at v.loewis.de  Tue Jul 18 21:32:39 2006
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Tue, 18 Jul 2006 21:32:39 +0200
Subject: [Python-Dev] logging module broken because of locale
In-Reply-To: <44BD2A98.7000705@egenix.com>
References: <20060717193955.GK3699@abulafia.devel.redhat.com>	<20060718124019.GB16294@abulafia.devel.redhat.com>	<44BD2054.4020705@v.loewis.de>
	<9AC7F55A-4106-414F-9C20-64FFFD8C6131@fuhm.net>
	<44BD2A98.7000705@egenix.com>
Message-ID: <44BD3757.7030101@v.loewis.de>

M.-A. Lemburg wrote:
> The Unicode database OTOH *defines* the upper/lower case mapping in
> a locale independent way, so the mappings are guaranteed
> to always produce the same results on all platforms.

Actually, that isn't the full truth; see UAX#21, which is now official
part of Unicode 4. It specifies two kinds of case conversion:
simple case conversion, and full case conversion. Python only supports
simple case conversion at the moment. Full case conversion is context
(locale) dependent, and must take into account SpecialCasing.txt.

Regards,
Martin

From rhettinger at ewtllc.com  Tue Jul 18 22:37:36 2006
From: rhettinger at ewtllc.com (Raymond Hettinger)
Date: Tue, 18 Jul 2006 13:37:36 -0700
Subject: [Python-Dev] Strategy for converting the decimal module to C
Message-ID: <44BD4690.3070402@ewtllc.com>

I briefly had a chance to look at some of the work being done on a C 
implementation of decimal, and it looks like the approach is following 
the Python version too literally.

Ideally, it should be written as if Python were not involved and 
afterwards add the appropriate method wrappers.  Contexts should be 
small structures that include the traps and flags as bitfields.  
Likewise, the internal representation of a decimal should be in a simple 
structure using a byte array for the decimal digits -- it should not be 
a Python object.  Internally, the implementation creates many temporary 
decimals for a scratchpad, it makes many copies of contexts, it 
frequently tests components of a context, and the functions frequently 
call each other.  Those operations need to be as cheap as possible -- 
that means no internal tuple objects, no ref counting, and no passing 
variables through tuples of arguments, etc.

I recommend writing most of the module to be independent of the Python C 
API.  After a working implementation is built, grafting on the wrappers 
should be a trivial step.  Unless we stay true to this course, the code 
will end-up being unnecessarily complex and the performance will be 
disappointing.


my-two-cents,


Raymond


P.S.  The dictionary approach to context objects should likely be 
abandoned for the C version.  If the API has to change a bit, then so be it.

From aahz at pythoncraft.com  Tue Jul 18 23:00:22 2006
From: aahz at pythoncraft.com (Aahz)
Date: Tue, 18 Jul 2006 14:00:22 -0700
Subject: [Python-Dev] Strategy for converting the decimal module to C
In-Reply-To: <44BD4690.3070402@ewtllc.com>
References: <44BD4690.3070402@ewtllc.com>
Message-ID: <20060718210017.GA7850@panix.com>

On Tue, Jul 18, 2006, Raymond Hettinger wrote:
>
> P.S.  The dictionary approach to context objects should likely be
> abandoned for the C version.  If the API has to change a bit, then so
> be it.

Why do you say that?  The rest I agree with; seems to me that making a
small wrapper for dict access works well, too.
-- 
Aahz (aahz at pythoncraft.com)           <*>         http://www.pythoncraft.com/

"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are, by
definition, not smart enough to debug it."  --Brian W. Kernighan

From mal at egenix.com  Tue Jul 18 23:03:54 2006
From: mal at egenix.com (M.-A. Lemburg)
Date: Tue, 18 Jul 2006 23:03:54 +0200
Subject: [Python-Dev] logging module broken because of locale
In-Reply-To: <44BD3757.7030101@v.loewis.de>
References: <20060717193955.GK3699@abulafia.devel.redhat.com>	<20060718124019.GB16294@abulafia.devel.redhat.com>	<44BD2054.4020705@v.loewis.de>	<9AC7F55A-4106-414F-9C20-64FFFD8C6131@fuhm.net>	<44BD2A98.7000705@egenix.com>
	<44BD3757.7030101@v.loewis.de>
Message-ID: <44BD4CBA.7040605@egenix.com>

Martin v. L?wis wrote:
> M.-A. Lemburg wrote:
>> The Unicode database OTOH *defines* the upper/lower case mapping in
>> a locale independent way, so the mappings are guaranteed
>> to always produce the same results on all platforms.
> 
> Actually, that isn't the full truth; see UAX#21, which is now official
> part of Unicode 4. It specifies two kinds of case conversion:
> simple case conversion, and full case conversion. Python only supports
> simple case conversion at the moment. Full case conversion is context
> (locale) dependent, and must take into account SpecialCasing.txt.

Right. In fact, some case mappings are not available in the Unicode
database, since that only contains mappings which don't increase or
decrease the length of the Unicode string. A typical example is the
German u'?'. u'?'.upper() would have to give u'SS', but instead
returns u'?'.

However, the point I wanted to make was that these mappings don't depend
on the locale setting of the C lib - you have to explicitly
access the mapping in the context of a locale and/or text.

As an example, here's the definition for the dotted/dotless i's in
Turkish taken from that file
(http://www.unicode.org/Public/UNIDATA/SpecialCasing.txt):

"""
# The entries in this file are in the following machine-readable format:
#
# <code>; <lower> ; <title> ; <upper> ; (<condition_list> ;)? # <comment>
#

...

# I and i-dotless; I-dot and i are case pairs in Turkish and Azeri
# The following rules handle those cases.

0130; 0069; 0130; 0130; tr; # LATIN CAPITAL LETTER I WITH DOT ABOVE
0130; 0069; 0130; 0130; az; # LATIN CAPITAL LETTER I WITH DOT ABOVE

# When lowercasing, remove dot_above in the sequence I + dot_above,
which will turn into i.
# This matches the behavior of the canonically equivalent I-dot_above

0307; ; 0307; 0307; tr After_I; # COMBINING DOT ABOVE
0307; ; 0307; 0307; az After_I; # COMBINING DOT ABOVE

# When lowercasing, unless an I is before a dot_above, it turns into a
dotless i.

0049; 0131; 0049; 0049; tr Not_Before_Dot; # LATIN CAPITAL LETTER I
0049; 0131; 0049; 0049; az Not_Before_Dot; # LATIN CAPITAL LETTER I

# When uppercasing, i turns into a dotted capital I

0069; 0069; 0130; 0130; tr; # LATIN SMALL LETTER I
0069; 0069; 0130; 0130; az; # LATIN SMALL LETTER I

# Note: the following case is already in the UnicodeData file.

# 0131; 0131; 0049; 0049; tr; # LATIN SMALL LETTER DOTLESS I
"""

Note how the context of the usage of the code points matters
when doing case-conversions.

To make things even more complicated, there are so called
language tags which can be embedded into the Unicode string,
so the language can also change within a Unicode string.

	http://www.unicode.org/reports/tr7/

To get a feeling of what it takes to do locale aware handling
of Unicode right, have a look at the Locale Data Markup
Language (LDML):

	http://www.unicode.org/reports/tr35/

(hey, perhaps Google could contribute support for this to Python ;-)

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Jul 18 2006)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! ::::

From martin at v.loewis.de  Tue Jul 18 23:15:17 2006
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Tue, 18 Jul 2006 23:15:17 +0200
Subject: [Python-Dev] logging module broken because of locale
In-Reply-To: <44BD4CBA.7040605@egenix.com>
References: <20060717193955.GK3699@abulafia.devel.redhat.com>	<20060718124019.GB16294@abulafia.devel.redhat.com>	<44BD2054.4020705@v.loewis.de>	<9AC7F55A-4106-414F-9C20-64FFFD8C6131@fuhm.net>	<44BD2A98.7000705@egenix.com>
	<44BD3757.7030101@v.loewis.de> <44BD4CBA.7040605@egenix.com>
Message-ID: <44BD4F65.9040307@v.loewis.de>

M.-A. Lemburg wrote:
> Right. In fact, some case mappings are not available in the Unicode
> database, since that only contains mappings which don't increase or
> decrease the length of the Unicode string. A typical example is the
> German u'?'. u'?'.upper() would have to give u'SS', but instead
> returns u'?'.

Actually, that is in the Unicode database (SpecialCasing.txt):

00DF; 00DF; 0053 0073; 0053 0053; # LATIN SMALL LETTER SHARP S

> However, the point I wanted to make was that these mappings don't depend
> on the locale setting of the C lib - you have to explicitly
> access the mapping in the context of a locale and/or text.

I don't get that point. SpecialCasing.txt is clearly intended to take
locale context into account. Whether this is the "C locale", or some
other locale mechanism, is out of scope of the Unicode specification.
It could be the C locale (and indeed, the C locale implementations
often take the Unicode casing procedure into account these days).

Regards,
Martin


From rhettinger at ewtllc.com  Wed Jul 19 00:39:20 2006
From: rhettinger at ewtllc.com (Raymond Hettinger)
Date: Tue, 18 Jul 2006 15:39:20 -0700
Subject: [Python-Dev] Strategy for converting the decimal module to C
In-Reply-To: <20060718210017.GA7850@panix.com>
References: <44BD4690.3070402@ewtllc.com> <20060718210017.GA7850@panix.com>
Message-ID: <44BD6318.6010806@ewtllc.com>

An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20060718/b66f8c5d/attachment.htm 

From tim.peters at gmail.com  Wed Jul 19 01:01:37 2006
From: tim.peters at gmail.com (Tim Peters)
Date: Tue, 18 Jul 2006 19:01:37 -0400
Subject: [Python-Dev] Strategy for converting the decimal module to C
In-Reply-To: <44BD6318.6010806@ewtllc.com>
References: <44BD4690.3070402@ewtllc.com> <20060718210017.GA7850@panix.com>
	<44BD6318.6010806@ewtllc.com>
Message-ID: <1f7befae0607181601g65bc3e1ax5f598c82b9de71fb@mail.gmail.com>

[Raymond Hettinger]
> ...
> If the current approach gets in their way, the C implementers should feel free to
> make an alternate design choice.

I expect they will, eventually.  Converting this to C is a big job,
and at the NFS sprint we settled on an "incremental" strategy allowing
most of the module to remain written in Python, converting methods to
C one at a time.  Changing the user-visible API is a hard egg to
swallow, and it's unfortunate that the Python code used a dict to hold
"flags" to begin with.  The dict doesn't just record whether an
exception has occurred, it also counts how many times the exception
occurred.  It's possible that someone, somewhere, has latched on to
that as "a feature".

From dalcinl at gmail.com  Wed Jul 19 01:15:18 2006
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Tue, 18 Jul 2006 20:15:18 -0300
Subject: [Python-Dev] Strategy for converting the decimal module to C
In-Reply-To: <1f7befae0607181601g65bc3e1ax5f598c82b9de71fb@mail.gmail.com>
References: <44BD4690.3070402@ewtllc.com> <20060718210017.GA7850@panix.com>
	<44BD6318.6010806@ewtllc.com>
	<1f7befae0607181601g65bc3e1ax5f598c82b9de71fb@mail.gmail.com>
Message-ID: <e7ba66e40607181615v72ac1b7eud8d5e3e094cc153a@mail.gmail.com>

On 7/18/06, Tim Peters <tim.peters at gmail.com> wrote:
> [Raymond Hettinger]
> > ...
> > If the current approach gets in their way, the C implementers should feel free to
> > make an alternate design choice.
>
> I expect they will, eventually.  Converting this to C is a big job,
> and at the NFS sprint we settled on an "incremental" strategy allowing
> most of the module to remain written in Python, converting methods to
> C one at a time.  Changing the user-visible API is a hard egg to
> swallow, and it's unfortunate that the Python code used a dict to hold
> "flags" to begin with.  The dict doesn't just record whether an
> exception has occurred, it also counts how many times the exception
> occurred.  It's possible that someone, somewhere, has latched on to
> that as "a feature".

Why not a 'cDecimal' module instead?


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From rhettinger at ewtllc.com  Wed Jul 19 02:06:06 2006
From: rhettinger at ewtllc.com (Raymond Hettinger)
Date: Tue, 18 Jul 2006 17:06:06 -0700
Subject: [Python-Dev] Strategy for converting the decimal module to C
In-Reply-To: <e7ba66e40607181615v72ac1b7eud8d5e3e094cc153a@mail.gmail.com>
References: <44BD4690.3070402@ewtllc.com>
	<20060718210017.GA7850@panix.com>	<44BD6318.6010806@ewtllc.com>	<1f7befae0607181601g65bc3e1ax5f598c82b9de71fb@mail.gmail.com>
	<e7ba66e40607181615v72ac1b7eud8d5e3e094cc153a@mail.gmail.com>
Message-ID: <44BD776E.1040404@ewtllc.com>

An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20060718/5d4775ac/attachment.html 

From rasky at develer.com  Wed Jul 19 02:34:28 2006
From: rasky at develer.com (Giovanni Bajo)
Date: Wed, 19 Jul 2006 02:34:28 +0200
Subject: [Python-Dev] Strategy for converting the decimal module to C
References: <44BD4690.3070402@ewtllc.com>
	<20060718210017.GA7850@panix.com><44BD6318.6010806@ewtllc.com>
	<1f7befae0607181601g65bc3e1ax5f598c82b9de71fb@mail.gmail.com>
Message-ID: <089a01c6aacb$184cd7d0$d503030a@trilan>

Tim Peters wrote:

> Changing the user-visible API is a hard egg to
> swallow, and it's unfortunate that the Python code used a dict to hold
> "flags" to begin with.  The dict doesn't just record whether an
> exception has occurred, it also counts how many times the exception
> occurred.  It's possible that someone, somewhere, has latched on to
> that as "a feature".

Especially since it was a documented one:

>>> import decimal
>>> help(decimal.Context)
Help on class Context in module decimal:

class Context(__builtin__.object)
 |  Contains the context for a Decimal instance.
[...]
 |  flags  - When an exception is caused, flags[exception] is incremented.
 |           (Whether or not the trap_enabler is set)
 |           Should be reset by user of Decimal instance.
[...]

-- 
Giovanni Bajo

From greg.ewing at canterbury.ac.nz  Wed Jul 19 03:17:20 2006
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Wed, 19 Jul 2006 13:17:20 +1200
Subject: [Python-Dev] logging module broken because of locale
In-Reply-To: <9AC7F55A-4106-414F-9C20-64FFFD8C6131@fuhm.net>
References: <20060717193955.GK3699@abulafia.devel.redhat.com>
	<20060718124019.GB16294@abulafia.devel.redhat.com>
	<44BD2054.4020705@v.loewis.de>
	<9AC7F55A-4106-414F-9C20-64FFFD8C6131@fuhm.net>
Message-ID: <44BD8820.9080602@canterbury.ac.nz>

James Y Knight wrote:

> That seems backwards of how it should be ideally: the byte-string  
> upper and lower should always do ascii uppering-and-lowering, and the  
> unicode ones should do it according to locale. Perhaps that can be  
> cleaned up in py3k?

I would expect bytes objects not to have upper() and lower()
methods at all in Py3k.

--
Greg

From g.brandl at gmx.net  Wed Jul 19 09:46:59 2006
From: g.brandl at gmx.net (Georg Brandl)
Date: Wed, 19 Jul 2006 09:46:59 +0200
Subject: [Python-Dev] Strategy for converting the decimal module to C
In-Reply-To: <44BD6318.6010806@ewtllc.com>
References: <44BD4690.3070402@ewtllc.com> <20060718210017.GA7850@panix.com>
	<44BD6318.6010806@ewtllc.com>
Message-ID: <e9ko1k$gv1$1@sea.gmane.org>

Raymond Hettinger wrote:
> Aahz wrote:
>> On Tue, Jul 18, 2006, Raymond Hettinger wrote:
>>   
>>> P.S.  The dictionary approach to context objects should likely be
>>> abandoned for the C version.  If the API has to change a bit, then so
>>> be it.
>>>     
>>
>> Why do you say that?  The rest I agree with; seems to me that making a
>> small wrapper for dict access works well, too.
>>   
> I think it was tripping-up the folks working on the C implementation. 
> Georg can speak to it more directly.  IIRC, the issue was that the
> context object exposed a dictionary which a user could update directly
> and there was no notification back to the surrounding object so it could
> update an underlying bitfield representation.

Yes, that's exactly what the problem was. Working with bitfields internally
is fine as long as Python code doesn't want to change the dicts exposed
as a wrapper.

>  If the current approach
> gets in their way, the C implementers should feel free to make an
> alternate design choice.

+1. (cDecimal is an ugly name, but a sound concept)

I don't know what progress Mateusz' work has made until now, but he wrote
at one time that he would "start optimizing as soon as he's certain that
it works".

Georg


From ncoghlan at gmail.com  Wed Jul 19 12:31:16 2006
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 19 Jul 2006 20:31:16 +1000
Subject: [Python-Dev] Strategy for converting the decimal module to C
In-Reply-To: <e9ko1k$gv1$1@sea.gmane.org>
References: <44BD4690.3070402@ewtllc.com>
	<20060718210017.GA7850@panix.com>	<44BD6318.6010806@ewtllc.com>
	<e9ko1k$gv1$1@sea.gmane.org>
Message-ID: <44BE09F4.6000009@gmail.com>

Georg Brandl wrote:
> Raymond Hettinger wrote:
>>  If the current approach
>> gets in their way, the C implementers should feel free to make an
>> alternate design choice.
> 
> +1. (cDecimal is an ugly name, but a sound concept)
> 
> I don't know what progress Mateusz' work has made until now, but he wrote
> at one time that he would "start optimizing as soon as he's certain that
> it works".

dmath (decimal-math, modelled on cmath for complex-math) could work.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
             http://www.boredomandlaziness.org

From mattjfleming at googlemail.com  Wed Jul 19 12:45:05 2006
From: mattjfleming at googlemail.com (Matt Fleming)
Date: Wed, 19 Jul 2006 11:45:05 +0100
Subject: [Python-Dev] [Python-checkins] r50708 - in python/trunk:
	Lib/test/test_sys.py Misc/NEWS Python/pystate.c
In-Reply-To: <5ff4a1e50607190334x62ae8f56r64dd71bd1c6de261@mail.gmail.com>
References: <20060719000321.36AF31E401E@bag.python.org>
	<20060719010840.GD2540@performancedrivers.com>
	<ee2a432c0607181936l6174016l6cd381678b722abe@mail.gmail.com>
	<e9kqrq$oto$1@sea.gmane.org>
	<5ff4a1e50607190334x62ae8f56r64dd71bd1c6de261@mail.gmail.com>
Message-ID: <5ff4a1e50607190345w21f32050w55cec402f9845972@mail.gmail.com>

On 19/07/06, Thomas Heller <theller at python.net> wrote:
> Neal Norwitz schrieb:
> > On 7/18/06, Jack Diederich <jackdied at jackdied.com> wrote:
> >>
> >> were pre-2003 and talking about mod_python.  HURD and FreeBSD came up a
> >> couple times.  Do we need to add more *BSD buildbots?
> >
> > Yes.  We only have OpenBSD now.  It would be nice to have {Free,Net}BSD too..
>
> Maybe some of the buildbots should (in addition to the normal build?)
> configure Python with --without-threads?
>

I have an AMD64 NetBSD machine that isn't doing much at the moment, I
can regurlary run tests (I submitted a patch not long back to make
regrtest netbsd-3 aware). However, I can't turn it into a buildbot,
sorry.

Matt

--
http://mattssanctuary.blogspot.com

From rhettinger at ewtllc.com  Wed Jul 19 21:24:18 2006
From: rhettinger at ewtllc.com (Raymond Hettinger)
Date: Wed, 19 Jul 2006 12:24:18 -0700
Subject: [Python-Dev] Strategy for converting the decimal module to C
In-Reply-To: <e9ko1k$gv1$1@sea.gmane.org>
References: <44BD4690.3070402@ewtllc.com>
	<20060718210017.GA7850@panix.com>	<44BD6318.6010806@ewtllc.com>
	<e9ko1k$gv1$1@sea.gmane.org>
Message-ID: <44BE86E2.6070802@ewtllc.com>

An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20060719/dc62ed02/attachment.html 

From g.brandl at gmx.net  Wed Jul 19 21:42:26 2006
From: g.brandl at gmx.net (Georg Brandl)
Date: Wed, 19 Jul 2006 21:42:26 +0200
Subject: [Python-Dev] Strategy for converting the decimal module to C
In-Reply-To: <44BE86E2.6070802@ewtllc.com>
References: <44BD4690.3070402@ewtllc.com>	<20060718210017.GA7850@panix.com>	<44BD6318.6010806@ewtllc.com>	<e9ko1k$gv1$1@sea.gmane.org>
	<44BE86E2.6070802@ewtllc.com>
Message-ID: <e9m1va$5pv$1@sea.gmane.org>

Raymond Hettinger wrote:
> 
>>> I think it was tripping-up the folks working on the C implementation. 
>>> Georg can speak to it more directly.  IIRC, the issue was that the
>>> context object exposed a dictionary which a user could update directly
>>> and there was no notification back to the surrounding object so it could
>>> update an underlying bitfield representation.
>>>     
>>
>> Yes, that's exactly what the problem was. Working with bitfields internally
>> is fine as long as Python code doesn't want to change the dicts exposed
>> as a wrapper.
>>   
> 
> If you want to stick with dictionary-like access to traps and flags,
> then use a dict subclass that overrides each of the mutating methods.

That may be easiest to do in Python code. I'll leave it to Mateusz to decide.

> Even then, we need to drop the concept of having the flags as counters
> rather than booleans.

Yes. Given that even Tim couldn't imagine a use case for counting the
exceptions, I think it's sensible.

Georg


From nmm1 at cus.cam.ac.uk  Wed Jul 19 22:45:00 2006
From: nmm1 at cus.cam.ac.uk (Nick Maclaren)
Date: Wed, 19 Jul 2006 21:45:00 +0100
Subject: [Python-Dev] Strategy for converting the decimal module to C
Message-ID: <E1G3Iui-0006Zk-CQ@draco.cus.cam.ac.uk>

Georg Brandl <g.brandl at gmx.net> wrote:
>
> > Even then, we need to drop the concept of having the flags as counters
> > rather than booleans.
>
> Yes. Given that even Tim couldn't imagine a use case for counting the
> exceptions, I think it's sensible.

Well, I can.  There is a traditional, important use - tuning.

When such arithmetic is implemented in hardware, it is normal for
exceptional cases to be handled by interrupt, and that is VERY
expensive - often 100-1,000 times the cost of a single operation,
occasionally 10,000 times.  It then becomes important to know how
many of the things you got, to know whether it is worth putting
code in to avoid them or even using a different algorithm.

Now, it is perfectly correct to say that this does not apply to
emulated arithmetic and that there is no justification for such
ghastly implementations.  But, regrettably, almost all exception
handling on modern systems IS ghastly - at least by the standards
of the 1960s.

Whether you regard the use of Python for tuning code that is to be
run using hardware, where the arithmetic will be performance-
critical as important, is a matter of taste.  I don't :-)


Regards,
Nick Maclaren,
University of Cambridge Computing Service,
New Museums Site, Pembroke Street, Cambridge CB2 3QH, England.
Email:  nmm1 at cam.ac.uk
Tel.:  +44 1223 334761    Fax:  +44 1223 334679

From misa at redhat.com  Wed Jul 19 23:29:19 2006
From: misa at redhat.com (Mihai Ibanescu)
Date: Wed, 19 Jul 2006 17:29:19 -0400
Subject: [Python-Dev] logging module broken because of locale
In-Reply-To: <200607181455.56535.fdrake@acm.org>
References: <20060717193955.GK3699@abulafia.devel.redhat.com>
	<44BD2054.4020705@v.loewis.de>
	<20060718185238.GB363@abulafia.devel.redhat.com>
	<200607181455.56535.fdrake@acm.org>
Message-ID: <20060719212919.GD15030@abulafia.devel.redhat.com>

On Tue, Jul 18, 2006 at 02:55:56PM -0400, Fred L. Drake, Jr. wrote:
> On Tuesday 18 July 2006 14:52, Mihai Ibanescu wrote:
>  > Unicode might be a perfectly acceptable suggestion for others too.
> 
> Are we still supporting builds that don't include Unicode?  If so, that needs 
> to be considered in a patch as well.

Good point. Does the attached patch look reasonable?

Thanks,
Misa
-------------- next part --------------
--- Python-2.4.3/Lib/logging/handlers.py.nolocale	2006-07-19 12:15:46.000000000 -0400
+++ Python-2.4.3/Lib/logging/handlers.py	2006-07-19 12:16:14.000000000 -0400
@@ -44,6 +44,12 @@
 DEFAULT_SOAP_LOGGING_PORT   = 9023
 SYSLOG_UDP_PORT             = 514
 
+# If python was not built with unicode support, use the str function instead
+# of the unicode type, and hope locale doesn't break things.
+
+if not hasattr(__builtins__, 'unicode'):
+    unicode = str
+
 class BaseRotatingHandler(logging.FileHandler):
     """
     Base class for handlers that rotate log files at a certain point.
@@ -162,7 +168,7 @@
     """
     def __init__(self, filename, when='h', interval=1, backupCount=0, encoding=None):
         BaseRotatingHandler.__init__(self, filename, 'a', encoding)
-        self.when = string.upper(when)
+        self.when = unicode(when).upper()
         self.backupCount = backupCount
         # Calculate the real rollover interval, which is just the number of
         # seconds between rollovers.  Also set the filename suffix used when
@@ -642,10 +648,12 @@
         """
         We need to convert record level to lowercase, maybe this will
         change in the future.
+        We convert it to unicode first, to avoid locale from changing the
+        meaning of lower() and upper()
         """
         msg = self.log_format_string % (
             self.encodePriority(self.facility,
-                                string.lower(record.levelname)),
+                                unicode(record.levelname).lower()),
             msg)
         try:
             if self.unixsocket:
@@ -854,7 +862,7 @@
         ("GET" or "POST")
         """
         logging.Handler.__init__(self)
-        method = string.upper(method)
+        method = unicode(method).upper()
         if method not in ["GET", "POST"]:
             raise ValueError, "method must be GET or POST"
         self.host = host

From jeremy at alum.mit.edu  Thu Jul 20 00:03:17 2006
From: jeremy at alum.mit.edu (Jeremy Hylton)
Date: Wed, 19 Jul 2006 18:03:17 -0400
Subject: [Python-Dev] Python sprint in NY and CA, Aug. 21-24
Message-ID: <e8bf7a530607191503p6038f538y6decf87bae1a808a@mail.gmail.com>

I'd like to repeat my invitation to spend a week at Google in
California or New York for a Python sprint.  We are hosting sprints at
our offices in Mountain View and New York City the week of Aug. 21,
Monday through Thursday.

We're planning to work broadly on Python 2.6 and Python 3000.  If
you're interested in the core implementation, the standard library,
the build process, or documentation, they're all in scope.  Working on
a test process using large community projects like Twisted and Zope
would be a great activity, too.

There's a wiki page with some more details:
 http://wiki.python.org/moin/GoogleSprint

Feel free to follow up with Neal or me if you have questions.

Jeremy

From brett at python.org  Thu Jul 20 00:35:45 2006
From: brett at python.org (Brett Cannon)
Date: Wed, 19 Jul 2006 15:35:45 -0700
Subject: [Python-Dev] new security doc using object-capabilities
Message-ID: <bbaeab100607191535p543cb0ddj66410cf985dd9b77@mail.gmail.com>

After various people suggesting object-capabilities, takling with Mark S.
Miller of the E programming language, and the people Mark works with at HP
Labs (who have been giving talks every week during this month here at Google
on object-capabilities), I have decided to go with object-capabilities for
securing interpreters.  I have rewritten my design doc from scratch and
deleted the old one.  The new doc is named securing_python.txt and can be
found through the svn web interface at
http://svn.python.org/view/python/branches/bcannon-sandboxing/securing_python.txt?rev=50717&view=log.
 I have pretty much ignored any concrete API and such and gone more
with a
conceptual doc to make sure the API does not get in the way of the core
security model.

Using object-capabilities should make the implementation much cleaner.
There is much less work directly on the interpreter and more of it gets
pushed up to extension modules.  I also have the okay of my supervisor to
use this approach in my dissertation so this will get done.

Two things do fall out of all of this which will make development much more
modular and easier.  First, the memory cap work just becomes a special build
on its own; no need to tie into the security work.  So I will be cleaning up
the bcannon-sandboxing branch code as it stands, and then either create a
separate branch for the object-capabilities work, or create another branch
for the memory cap stuff and shift the changes over there.  I will most
likely do the former so as to not lose the history on the checkins.

I also plan to rewrite the import machinery in pure Python.  This will make
the code much more maintainable and make creating proxies for the import
machinery much easier.  I will be doing that in a directory in the sandbox
initially since it needs to work from what Python has now (and possibly some
new extension module code) before it can be integrated into the interpreter
directly.  Anyone who wants to help with that can.  I already have some
perliminary notes on the whole thing and I think it will be reasonably
doable.

Anyway, there you go.  Here is to hoping I have thought this all through
properly.  =)

-Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20060719/09a113fc/attachment.html 

From python-dev at zesty.ca  Thu Jul 20 01:11:24 2006
From: python-dev at zesty.ca (Ka-Ping Yee)
Date: Wed, 19 Jul 2006 18:11:24 -0500 (CDT)
Subject: [Python-Dev] new security doc using object-capabilities
In-Reply-To: <bbaeab100607191535p543cb0ddj66410cf985dd9b77@mail.gmail.com>
References: <bbaeab100607191535p543cb0ddj66410cf985dd9b77@mail.gmail.com>
Message-ID: <Pine.LNX.4.58.0607191810460.31087@server1.LFW.org>

On Wed, 19 Jul 2006, Brett Cannon wrote:
> I have decided to go with object-capabilities for
> securing interpreters.  I have rewritten my design doc from scratch and
> deleted the old one.  The new doc is named securing_python.txt and can be
> found through the svn web interface at
> http://svn.python.org/view/python/branches/bcannon-sandboxing/securing_python.txt?rev=50717&view=log.

This is amazing news!!  I'm going off to read your document right now.


-- ?!ng

From fuzzyman at voidspace.org.uk  Thu Jul 20 01:25:26 2006
From: fuzzyman at voidspace.org.uk (Michael Foord)
Date: Thu, 20 Jul 2006 00:25:26 +0100
Subject: [Python-Dev] new security doc using object-capabilities
In-Reply-To: <bbaeab100607191535p543cb0ddj66410cf985dd9b77@mail.gmail.com>
References: <bbaeab100607191535p543cb0ddj66410cf985dd9b77@mail.gmail.com>
Message-ID: <44BEBF66.9020102@voidspace.org.uk>

Brett Cannon wrote:
> After various people suggesting object-capabilities, takling with Mark 
> S. Miller of the E programming language, and the people Mark works 
> with at HP Labs (who have been giving talks every week during this 
> month here at Google on object-capabilities), I have decided to go 
> with object-capabilities for securing interpreters.  I have rewritten 
> my design doc from scratch and deleted the old one.  The new doc is 
> named securing_python.txt and can be found through the svn web 
> interface at 
> http://svn.python.org/view/python/branches/bcannon-sandboxing/securing_python.txt?rev=50717&view=log 
> <http://svn.python.org/view/python/branches/bcannon-sandboxing/securing_python.txt?rev=50717&view=log> 
> .  I have pretty much ignored any concrete API and such and gone more 
> with a conceptual doc to make sure the API does not get in the way of 
> the core security model.
>

This may not be relevant or  possible, in which case I apologise, but 
the .NET model of creating application domains is extremely useful. It 
allows you to assign domains and run code within those domains. This 
means, for example, you can create a plugin system and run the plugins 
in a secure domain.

I realise that this was the intent of the original rexec module, and 
your proposed new design (which is very exciting) overcomes the 
difficulties in that approach. The only approach using the new system 
would be interprocess communication (?) with a trusted interpreter 
communicating with an un-trusted one. Would the communication layer need 
to be implemented as a C extension, or will a standard Python API be 
possible ? Hmmm.... maybe I should read your doc. :-)

Michael Foord
http://www.voidspace.org.uk/python/index.shtml

> Using object-capabilities should make the implementation much 
> cleaner.  There is much less work directly on the interpreter and more 
> of it gets pushed up to extension modules.  I also have the okay of my 
> supervisor to use this approach in my dissertation so this will get done.
>
> Two things do fall out of all of this which will make development much 
> more modular and easier.  First, the memory cap work just becomes a 
> special build on its own; no need to tie into the security work.  So I 
> will be cleaning up the bcannon-sandboxing branch code as it stands, 
> and then either create a separate branch for the object-capabilities 
> work, or create another branch for the memory cap stuff and shift the 
> changes over there.  I will most likely do the former so as to not 
> lose the history on the checkins.
>
> I also plan to rewrite the import machinery in pure Python.  This will 
> make the code much more maintainable and make creating proxies for the 
> import machinery much easier.  I will be doing that in a directory in 
> the sandbox initially since it needs to work from what Python has now 
> (and possibly some new extension module code) before it can be 
> integrated into the interpreter directly.  Anyone who wants to help 
> with that can.  I already have some perliminary notes on the whole 
> thing and I think it will be reasonably doable.
>
> Anyway, there you go.  Here is to hoping I have thought this all 
> through properly.  =)
>
> -Brett
> ------------------------------------------------------------------------
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk
>   


From fuzzyman at voidspace.org.uk  Thu Jul 20 01:50:33 2006
From: fuzzyman at voidspace.org.uk (Michael Foord)
Date: Thu, 20 Jul 2006 00:50:33 +0100
Subject: [Python-Dev] new security doc using object-capabilities
In-Reply-To: <44BEBF66.9020102@voidspace.org.uk>
References: <bbaeab100607191535p543cb0ddj66410cf985dd9b77@mail.gmail.com>
	<44BEBF66.9020102@voidspace.org.uk>
Message-ID: <44BEC549.5060305@voidspace.org.uk>

Michael Foord wrote:
> Brett Cannon wrote:
>   
>> After various people suggesting object-capabilities, takling with Mark 
>> S. Miller of the E programming language, and the people Mark works 
>> with at HP Labs (who have been giving talks every week during this 
>> month here at Google on object-capabilities), I have decided to go 
>> with object-capabilities for securing interpreters.  I have rewritten 
>> my design doc from scratch and deleted the old one.  The new doc is 
>> named securing_python.txt and can be found through the svn web 
>> interface at 
>> http://svn.python.org/view/python/branches/bcannon-sandboxing/securing_python.txt?rev=50717&view=log 
>> <http://svn.python.org/view/python/branches/bcannon-sandboxing/securing_python.txt?rev=50717&view=log> 
>> .  I have pretty much ignored any concrete API and such and gone more 
>> with a conceptual doc to make sure the API does not get in the way of 
>> the core security model.
>>
>>     
>
> This may not be relevant or  possible, in which case I apologise, but 
> the .NET model of creating application domains is extremely useful. It 
> allows you to assign domains and run code within those domains. This 
> means, for example, you can create a plugin system and run the plugins 
> in a secure domain.
>
> I realise that this was the intent of the original rexec module, and 
> your proposed new design (which is very exciting) overcomes the 
> difficulties in that approach. The only approach using the new system 
> would be interprocess communication (?) with a trusted interpreter 
> communicating with an un-trusted one. Would the communication layer need 
> to be implemented as a C extension, or will a standard Python API be 
> possible ? Hmmm.... maybe I should read your doc. :-)
>
>   
Ok, started to read the doc - and realise it specifically addresses 
these issues. My apologies :-)

Michael
http://www.voidspace.org.uk/python/index.shtml

> Michael Foord
> http://www.voidspace.org.uk/python/index.shtml
>
>   
>> Using object-capabilities should make the implementation much 
>> cleaner.  There is much less work directly on the interpreter and more 
>> of it gets pushed up to extension modules.  I also have the okay of my 
>> supervisor to use this approach in my dissertation so this will get done.
>>
>> Two things do fall out of all of this which will make development much 
>> more modular and easier.  First, the memory cap work just becomes a 
>> special build on its own; no need to tie into the security work.  So I 
>> will be cleaning up the bcannon-sandboxing branch code as it stands, 
>> and then either create a separate branch for the object-capabilities 
>> work, or create another branch for the memory cap stuff and shift the 
>> changes over there.  I will most likely do the former so as to not 
>> lose the history on the checkins.
>>
>> I also plan to rewrite the import machinery in pure Python.  This will 
>> make the code much more maintainable and make creating proxies for the 
>> import machinery much easier.  I will be doing that in a directory in 
>> the sandbox initially since it needs to work from what Python has now 
>> (and possibly some new extension module code) before it can be 
>> integrated into the interpreter directly.  Anyone who wants to help 
>> with that can.  I already have some perliminary notes on the whole 
>> thing and I think it will be reasonably doable.
>>
>> Anyway, there you go.  Here is to hoping I have thought this all 
>> through properly.  =)
>>
>> -Brett
>> ------------------------------------------------------------------------
>>
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> http://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk
>>   
>>     
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk
>
>   


From tim.peters at gmail.com  Thu Jul 20 02:52:11 2006
From: tim.peters at gmail.com (Tim Peters)
Date: Wed, 19 Jul 2006 20:52:11 -0400
Subject: [Python-Dev] Strategy for converting the decimal module to C
In-Reply-To: <e9m1va$5pv$1@sea.gmane.org>
References: <44BD4690.3070402@ewtllc.com> <20060718210017.GA7850@panix.com>
	<44BD6318.6010806@ewtllc.com> <e9ko1k$gv1$1@sea.gmane.org>
	<44BE86E2.6070802@ewtllc.com> <e9m1va$5pv$1@sea.gmane.org>
Message-ID: <1f7befae0607191752j7b4a828eu99ea6cc28be40f28@mail.gmail.com>

...

[Raymond]
>> Even then, we need to drop the concept of having the flags as counters
>> rather than booleans.

[Georg Brandl]
> Yes. Given that even Tim couldn't imagine a use case for counting the
> exceptions, I think it's sensible.

That's not it -- someone will "find a use" for anything.  It's
unfortunate that we used a dict with counts because the /standard/
we're trying to meet requires no such thing, and clearly had a "pile
of on/off bits" model in mind.  Extending a standard without strong
need creates problems (for example, this one <0.5 wink>).

From greg.ewing at canterbury.ac.nz  Thu Jul 20 03:30:50 2006
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Thu, 20 Jul 2006 13:30:50 +1200
Subject: [Python-Dev] Strategy for converting the decimal module to C
In-Reply-To: <E1G3Iui-0006Zk-CQ@draco.cus.cam.ac.uk>
References: <E1G3Iui-0006Zk-CQ@draco.cus.cam.ac.uk>
Message-ID: <44BEDCCA.1070305@canterbury.ac.nz>

Nick Maclaren wrote:

> When such arithmetic is implemented in hardware, it is normal for
> exceptional cases to be handled by interrupt, and that is VERY
> expensive ...  It then becomes important to know how
> many of the things you got, to know whether it is worth putting
> code in to avoid them or even using a different algorithm.

But couldn't you just put in an interrupt handler that
counts the interrupts, for the purpose of measurement?

--
Greg

From rhettinger at ewtllc.com  Thu Jul 20 03:45:51 2006
From: rhettinger at ewtllc.com (Raymond Hettinger)
Date: Wed, 19 Jul 2006 18:45:51 -0700
Subject: [Python-Dev] Strategy for converting the decimal module to C
In-Reply-To: <1f7befae0607191752j7b4a828eu99ea6cc28be40f28@mail.gmail.com>
References: <44BD4690.3070402@ewtllc.com>
	<20060718210017.GA7850@panix.com>	<44BD6318.6010806@ewtllc.com>
	<e9ko1k$gv1$1@sea.gmane.org>	<44BE86E2.6070802@ewtllc.com>
	<e9m1va$5pv$1@sea.gmane.org>
	<1f7befae0607191752j7b4a828eu99ea6cc28be40f28@mail.gmail.com>
Message-ID: <44BEE04F.9000209@ewtllc.com>

An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20060719/058ab3cb/attachment.html 

From t-bruch at microsoft.com  Thu Jul 20 04:29:09 2006
From: t-bruch at microsoft.com (Bruce Christensen)
Date: Wed, 19 Jul 2006 19:29:09 -0700
Subject: [Python-Dev] Pickling objects that return string from reduce
In-Reply-To: <44BBFB5B.2090806@v.loewis.de>
References: <3581AA168D87A2479D88EA319BDF7D32E05276@RED-MSG-80.redmond.corp.microsoft.com>
	<44BBFB5B.2090806@v.loewis.de>
Message-ID: <3581AA168D87A2479D88EA319BDF7D32EF2FC7@RED-MSG-80.redmond.corp.microsoft.com>

Martin v. L?wis" wrote:

> If  obj has no __module__ attribute (or if it is None), pickle
> (didn't check cPickle) also does
> 
> for n, module in sys.module.items():
>   if "module-ignored": continue
>   if getattr(module, result, None) is obj:
>      break # use n as module name
> 
> If obj does have a __module__ attribute, it uses __import__
> to import the module, just to make sure it gets into sys.modules.

What is "module-ignored" above? It's obviously not a literal string...

--Bruce

From kbk at shore.net  Thu Jul 20 06:55:00 2006
From: kbk at shore.net (Kurt B. Kaiser)
Date: Thu, 20 Jul 2006 00:55:00 -0400 (EDT)
Subject: [Python-Dev] Weekly Python Patch/Bug Summary
Message-ID: <200607200455.k6K4t0Vw018820@bayview.thirdcreek.com>

Patch / Bug Summary
___________________

Patches :  398 open ( +5) /  3334 closed (+19) /  3732 total (+24)
Bugs    :  904 open ( -4) /  6011 closed (+36) /  6915 total (+32)
RFE     :  222 open ( -1) /   231 closed ( +2) /   453 total ( +1)

New / Reopened Patches
______________________

Fix for #1513611 and #1511497; xml.sax imports  (2006-07-10)
       http://python.org/sf/1519796  opened by  ?iga Seilnacht

telnetlib.py change to ease option handling.  (2006-07-10)
       http://python.org/sf/1520081  opened by  Ernest ter Kuile

Support for PyGetSetDefs in pydoc, inspect, and types  (2006-07-10)
       http://python.org/sf/1520294  opened by  Barry A. Warsaw

Distutils bugfix: Read $AR from the environment/Makefile.  (2006-07-11)
       http://python.org/sf/1520877  opened by  Douglas Greiman

make install change: Allow $DESTDIR to be relative  (2006-07-11)
       http://python.org/sf/1520879  opened by  Douglas Greiman

Fix tests that assume they can write to Lib/test  (2006-07-12)
       http://python.org/sf/1520904  opened by  Douglas Greiman

Don't produce core file in test_subprocess.py  (2006-07-12)
       http://python.org/sf/1520905  opened by  Douglas Greiman

Extra configurability for doctest TestCases  (2006-07-12)
       http://python.org/sf/1521051  opened by  Russell Keith-Magee

--help and --version long options  (2006-07-12)
CLOSED http://python.org/sf/1521179  opened by  Georg Brandl

smtplib login fails with aol smtp server  (2006-07-12)
       http://python.org/sf/1521196  opened by  Peter

Reintroduce index checking in 1-element ctypes arrays  (2006-07-13)
CLOSED http://python.org/sf/1521817  opened by  Thomas Heller

Fix grammatical errors in Doc/howto/doanddont.tex  (2006-07-13)
CLOSED http://python.org/sf/1521874  opened by  Collin Winter

Give Cookie.py its own _idmap  (2006-07-13)
       http://python.org/sf/1521882  opened by  Collin Winter

Add some explication to PEP 3100  (2006-07-13)
       http://python.org/sf/1522038  opened by  Collin Winter

Remove operator.truth() and operator.abs()  (2006-07-13)
       http://python.org/sf/1522059  opened by  Collin Winter

Improve docs for filter()  (2006-07-13)
CLOSED http://python.org/sf/1522211  opened by  Collin Winter

irda socket support  (2006-07-14)
       http://python.org/sf/1522400  opened by  ??????

Tix.Grid patch  (2006-07-14)
       http://python.org/sf/1522587  opened by  klappnase

XML Test Runner for unittest module  (2006-07-14)
       http://python.org/sf/1522704  opened by  Sebastian Rittau

(partial?) fix for Misc/python-config.in  (2006-07-16)
       http://python.org/sf/1523356  opened by  M. Levinson

Var-default patch undid part of backtick to repr() patch  (2006-07-18)
CLOSED http://python.org/sf/1524429  opened by  Graham Horler

Fix Tkinter Tcl-commands memory-leaks  (2006-07-18)
       http://python.org/sf/1524639  opened by  Graham Horler

Fix --without-threads build error  (2006-07-18)
CLOSED http://python.org/sf/1524724  opened by  Matt Fleming

ConfigParser: accept leading whitespace on options+comments  (2006-07-18)
       http://python.org/sf/1524825  opened by  Ken Lalonde

Patches Closed
______________

Patch for bug 1441486: bad unary minus folding in compiler  (2006-03-10)
       http://python.org/sf/1446922  closed by  nascheme

OpenVMS patches Modules directory  (2006-07-04)
       http://python.org/sf/1516912  closed by  nnorwitz

Rough documentation for xml.etree.ElementTree  (2006-06-10)
       http://python.org/sf/1504046  closed by  nnorwitz

turtle.py: correcting begin_fill  (2006-07-09)
       http://python.org/sf/1519566  closed by  loewis

pdb: fix for #1472251('run/runeval' commands bug)  (2006-04-18)
       http://python.org/sf/1472257  closed by  jakamkon

urllib2 redirection fix  (2006-07-01)
       http://python.org/sf/1515745  closed by  nnorwitz

Python long option support  (2006-05-03)
       http://python.org/sf/1481112  closed by  gbrandl

--help and --version long options  (2006-07-12)
       http://python.org/sf/1521179  closed by  gbrandl

Reintroduce index checking in 1-element ctypes arrays  (2006-07-13)
       http://python.org/sf/1521817  closed by  theller

Fix grammatical errors in Doc/howto/doanddont.tex  (2006-07-13)
       http://python.org/sf/1521874  closed by  gbrandl

Improve docs for filter()  (2006-07-13)
       http://python.org/sf/1522211  closed by  rhettinger

python / pythonw replacement in C  (2004-10-29)
       http://python.org/sf/1056561  closed by  etrepum

OS X: Can't use #!/usr/bin/pythonw  (2004-10-02)
       http://python.org/sf/1038911  closed by  etrepum

update the binhex module for Mach-O  (2005-06-14)
       http://python.org/sf/1220874  closed by  etrepum

Remove dependencies on the sets module  (2006-06-04)
       http://python.org/sf/1500609  closed by  gbrandl

ColorDelegator - Several bug fixes  (2006-04-30)
       http://python.org/sf/1479219  closed by  kbk

Var-default patch undid part of backtick to repr() patch  (2006-07-18)
       http://python.org/sf/1524429  closed by  loewis

zipfile: support for ZIP64  (2006-03-09)
       http://python.org/sf/1446489  closed by  ronaldoussoren

Fix --without-threads build error  (2006-07-18)
       http://python.org/sf/1524724  closed by  tim_one

New / Reopened Bugs
___________________

Cannot use high-numbered sockets in 2.4.3  (2006-05-24)
CLOSED http://python.org/sf/1494314  reopened by  anthonybaxter

Incorrect lineno's in code objects  (2006-06-26)
CLOSED http://python.org/sf/1512814  reopened by  twouters

AttributeError in the shelve module  (2006-07-09)
CLOSED http://python.org/sf/1519786  opened by  Martin Kelly

Proxy does not work in 2.4.3  (2006-07-10)
       http://python.org/sf/1519816  opened by  Michal Niklas

non-uniform behavior in  'startswith' / 'endswith'  (2006-07-10)
       http://python.org/sf/1520176  opened by  Milind

__missing__ does not get called  (2006-07-11)
CLOSED http://python.org/sf/1520327  opened by  Milind

fcntl.ioctl fails to copy back exactly-1024 buffer  (2006-07-11)
       http://python.org/sf/1520818  opened by  Mark Eichin

urrlib2 max_redirections=0 disables redirects  (2006-07-11)
       http://python.org/sf/1520831  opened by  rus_r_orange

unpack list of singleton tuples not unpacking  (2006-07-11)
CLOSED http://python.org/sf/1520864  opened by  Anthony Tuininga

time.strftime breakage in 2.4/2.5  (2006-07-11)
CLOSED http://python.org/sf/1520914  opened by  Skip Montanaro

ctypes test overwrites /dev/null  (2006-07-12)
CLOSED http://python.org/sf/1521375  opened by  Scot Doyle

file.seek() influelce write() when opened with a+ mode  (2006-07-12)
       http://python.org/sf/1521491  opened by  Lior

isinstance failure in 2.6 Beta 2  (2006-07-13)
CLOSED http://python.org/sf/1521726  opened by  Nick Maclaren

possible bug in mystrtol.c with recent gcc  (2006-07-13)
       http://python.org/sf/1521947  opened by  Marien Zwart

shlex.split() does not tokenize like the shell  (2006-07-13)
       http://python.org/sf/1521950  opened by  Dan Christian

filter() doesn't use __len__ of str/unicode/tuple subclasses  (2006-07-13)
CLOSED http://python.org/sf/1522016  opened by  Collin Winter

RPM build fails for Py2.5b2  (2006-07-13)
       http://python.org/sf/1522046  opened by  Stefan Behnel

_threading_local.py logic error in _localbase __new__  (2006-07-13)
       http://python.org/sf/1522237  opened by  Tony Nelson

Patch #1388073 is not mentioned in NEWS  (2006-07-14)
CLOSED http://python.org/sf/1522771  opened by  Collin Winter

incorrect tcp checksum  (2006-07-15)
CLOSED http://python.org/sf/1523136  opened by  ecir hana

threading.Thread  Traceback  (2006-07-16)
CLOSED http://python.org/sf/1523465  opened by  roee88

PyArg_ParseTupleAndKeywords potential core dump  (2006-07-16)
       http://python.org/sf/1523610  opened by  Eric Huss

2.4.2 file.read caches EOF state  (2006-07-17)
       http://python.org/sf/1523853  opened by  Jarkko Torppa

logging using the SysLog handler fails if locale is set  (2006-07-17)
       http://python.org/sf/1524081  opened by  Mihai Ibanescu

os.listdir doesn't check error code from FindNextFile  (2006-07-18)
       http://python.org/sf/1524310  opened by  Roger Upole

configure --without-threads fails to build  (2006-07-18)
CLOSED http://python.org/sf/1524317  opened by  Ram Bhamidipaty

MemoryError with a lot of available memory - gc not called  (2006-07-19)
       http://python.org/sf/1524938  opened by  Mark Matusevich

Webserver TypeError: expected read buffer, NoneType found  (2006-07-19)
       http://python.org/sf/1525343  opened by  jbet

Build fails on OS X with case sensitive fs  (2006-07-19)
       http://python.org/sf/1525447  opened by  gideon may

SimpleXMLRpcServer still uses sys.exc_value and sys.exc_type  (2006-07-19)
       http://python.org/sf/1525469  opened by  Russell Warren

Document additions from PEP 302  (2006-07-19)
       http://python.org/sf/1525549  opened by  Brett Cannon

Malloc, memory error, failmalloc, low memory.  (2006-07-19)
       http://python.org/sf/1525589  opened by  Rene Dudfield

ldap get_option(0) causes segfault  (2006-07-19)
CLOSED http://python.org/sf/1525590  opened by  Sean Burford

exec and eval allocate lots of memory and do not free it  (2006-07-20)
       http://python.org/sf/1525678  opened by  Connelly

Bugs Closed
___________

yet another svn head compiler change  (2006-03-02)
       http://python.org/sf/1441486  closed by  nascheme

Cannot use high-numbered sockets in 2.4.3  (2006-05-24)
       http://python.org/sf/1494314  closed by  anthonybaxter

Incorrect lineno's in code objects  (2006-06-26)
       http://python.org/sf/1512814  closed by  nnorwitz

Incorrect lineno's in code objects  (2006-06-26)
       http://python.org/sf/1512814  closed by  nnorwitz

c_void_pointer should accept a long pointer > 0x7fffffff  (2006-07-06)
       http://python.org/sf/1518190  closed by  theller

ImportWarning should be removed  (2006-06-30)
       http://python.org/sf/1515169  closed by  nnorwitz

test_ctypes fails on OSX 10.3  (2006-04-10)
       http://python.org/sf/1467450  closed by  theller

AttributeError in the shelve module  (2006-07-10)
       http://python.org/sf/1519786  closed by  gbrandl

inspect.py: still infinite recursion inspecting frames  (2006-07-03)
       http://python.org/sf/1516184  closed by  pje

AttributeError in upload_file on interrupted connection  (2006-03-23)
       http://python.org/sf/1457312  closed by  pje

subprocess test cases fail with noexec-mounted /tmp  (2006-03-06)
       http://python.org/sf/1444408  closed by  astrand

subprocess.CalledProcessError uses errno incorrectly  (2005-12-08)
       http://python.org/sf/1376309  closed by  astrand

subprocess.py abuse of errno  (2005-06-20)
       http://python.org/sf/1223937  closed by  astrand

__missing__ does not get called  (2006-07-11)
       http://python.org/sf/1520327  closed by  gbrandl

unpack list of singleton tuples not unpacking  (2006-07-11)
       http://python.org/sf/1520864  closed by  nnorwitz

time.strftime breakage in 2.4/2.5  (2006-07-11)
       http://python.org/sf/1520914  closed by  bcannon

ctypes test overwrites /dev/null  (2006-07-12)
       http://python.org/sf/1521375  closed by  theller

isinstance failure in 2.5 Beta 2  (2006-07-13)
       http://python.org/sf/1521726  closed by  gbrandl

filter() doesn't use __len__ of str/unicode/tuple subclasses  (2006-07-13)
       http://python.org/sf/1522016  closed by  rhettinger

msvccompiler.py using VC6 with Python 2.5a2  (2006-06-18)
       http://python.org/sf/1508010  closed by  loewis

incorrect locale.strcoll() return in Windows  (2006-07-08)
       http://python.org/sf/1519069  closed by  loewis

Patch #1388073 is not mentioned in NEWS  (2006-07-14)
       http://python.org/sf/1522771  closed by  gbrandl

IDLE fails to launch after 46272 (2.4 with 2.5 IDLE)  (2006-06-27)
       http://python.org/sf/1513617  closed by  kbk

ext/win-cookbook.html has a broken link to distutils  (2006-07-07)
       http://python.org/sf/1518772  closed by  quiver

PIL binary package missing jpeg support (pimp)  (2003-07-23)
       http://python.org/sf/776600  closed by  etrepum

IDE needs easy access to builtin help()  (2003-03-29)
       http://python.org/sf/711991  closed by  etrepum

Raise IDE output window over splash screen on early crash  (2003-03-26)
       http://python.org/sf/710374  closed by  etrepum

IDE stdin doesn't have readlines  (2003-03-26)
       http://python.org/sf/710373  closed by  etrepum

Mac IDE behaviour (output to console)  (2002-06-24)
       http://python.org/sf/573174  closed by  etrepum

incorrect tcp checksum  (2006-07-15)
       http://python.org/sf/1523136  closed by  loewis

threading.Thread  Traceback  (2006-07-16)
       http://python.org/sf/1523465  closed by  tim_one

"as" keyword sometimes highlighted in strings  (2005-10-12)
       http://python.org/sf/1325071  closed by  kbk

email.Utils.py: "'" in RFC2231 header  (2005-06-10)
       http://python.org/sf/1218081  closed by  bwarsaw

Incorrect RFC 2231 decoding  (2005-01-15)
       http://python.org/sf/1102973  closed by  bwarsaw

link path probs on OSX re: Tcl, Tk & fink's /sw  (07/13/05)
       http://python.org/sf/1237697  closed by  sf-robot

termios.c in qnx4.25  (09/19/05)
       http://python.org/sf/1295179  closed by  sf-robot

IDLE bug - changing shortcuts - Shift-Tab broken!  (2005-04-08)
       http://python.org/sf/1179168  closed by  kbk

expat crash python  (2005-09-20)
       http://python.org/sf/1296433  closed by  bcannon

configure --without-threads fails to build  (2006-07-18)
       http://python.org/sf/1524317  closed by  tim_one

ldap get_option(0) causes segfault  (2006-07-19)
       http://python.org/sf/1525590  closed by  nnorwitz

Getting an error message import site failed -v traceback.  (07/05/06)
       http://python.org/sf/1517370  closed by  sf-robot

New / Reopened RFE
__________________

support all of strftime(3)  (2006-07-11)
       http://python.org/sf/1520662  opened by  Toni Mueller

RFE Closed
__________

Add syntax coloring to Mac IDE  (2002-08-07)
       http://python.org/sf/592047  closed by  etrepum

Enhance PackageManager functionality  (2003-07-28)
       http://python.org/sf/779160  closed by  etrepum


From martin at v.loewis.de  Thu Jul 20 08:47:40 2006
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Thu, 20 Jul 2006 08:47:40 +0200
Subject: [Python-Dev] Pickling objects that return string from reduce
In-Reply-To: <3581AA168D87A2479D88EA319BDF7D32EF2FC7@RED-MSG-80.redmond.corp.microsoft.com>
References: <3581AA168D87A2479D88EA319BDF7D32E05276@RED-MSG-80.redmond.corp.microsoft.com>
	<44BBFB5B.2090806@v.loewis.de>
	<3581AA168D87A2479D88EA319BDF7D32EF2FC7@RED-MSG-80.redmond.corp.microsoft.com>
Message-ID: <44BF270C.90202@v.loewis.de>

Bruce Christensen wrote:
>> If  obj has no __module__ attribute (or if it is None), pickle
>> (didn't check cPickle) also does
>>
>> for n, module in sys.module.items():
>>   if "module-ignored": continue
>>   if getattr(module, result, None) is obj:
>>      break # use n as module name
> 
> What is "module-ignored" above? It's obviously not a literal string...

It's skipped if module is None (skip dummy package entries)
or n=='__main__'.

Regards,
Martin

From l.oluyede at gmail.com  Thu Jul 20 10:17:32 2006
From: l.oluyede at gmail.com (Lawrence Oluyede)
Date: Thu, 20 Jul 2006 10:17:32 +0200
Subject: [Python-Dev] new security doc using object-capabilities
In-Reply-To: <bbaeab100607191535p543cb0ddj66410cf985dd9b77@mail.gmail.com>
References: <bbaeab100607191535p543cb0ddj66410cf985dd9b77@mail.gmail.com>
Message-ID: <9eebf5740607200117r4d4613e2i91665ea211bab46@mail.gmail.com>

That's great. I just read your draft but I have little comments to do
but before let me say that I liked the idea to borrow concepts from E.
I've crossed the E's path in the beginning of this year and I found it
a pot of really nice ideas (for promises and capabilities). Here are
my comments about the draft:

- it's not really clear to me what the "powerbox" is. I think I got
the concept of "super process" but maybe it's to be clarified, isn't
it? It become clear in the "threat model" paragraph

- I hope no Rubystas will read the "Problem of No Private Namespace"
section because they have private/protected keywords to enforce this
stuff :-) Writing proxies in C will slow down the dev process (altough
will speed up the performance maybe) but in a far future someone will
come up with an alternative closer to the Python level

- Can you write down a simple example of what you mean with "changing
something of the built-in objects"? (in "Problem of mutable shared
state")

- What about the performance issues of the capabilities model overall?

- I know what you meant to say but the paragraph about pythonicness
and the security model seems a little "fuzzy" to me. Which are the
boundaries of the allowed changes for the security stuff?

- You don't say anything about networking and networked resources in
the list of the standard sandboxed interpreter

- Suppose we have a .py module. Based on your security model we can
import it, right? When imported it generates a .pyc file. The second
time we import it what happens? .pyc is ignored? import is not allowed
at all? We can't rely on the name of the file.pyc because an attacker
who knows the file.py is secure and the second import is done against
file.pyc can replace the "secure" file.pyc with an implementation not
secure and can do some kind of harm to the sandbox

- About "Filesystem information". Does the sandboxed interpreter need
to know all that information about file paths, files and so on? Can't
we reset those attributes to something arbitrary?

- About sys module: I think the best way is to have a purged fake sys
module with only the stuff you need. pypy has the concept of faked
modules too (altough for a different reason)

- About networking: what do you think about the E's model of really
safe networking, protected remotable objects and safe RPC? Is that
model applicable to Python's in some way? We can't use the E's model
as a whole (ask people to generate a safe key and send it by email is
unfeasible)

- is the protected memory model a some kind of memory monitor system?

I think that's all for the draft. I wrote these comments during the
reading of the document.

Hope some of these help

-- 
Lawrence
http://www.oluyede.org/blog

From fuzzyman at voidspace.org.uk  Thu Jul 20 11:12:43 2006
From: fuzzyman at voidspace.org.uk (Michael Foord)
Date: Thu, 20 Jul 2006 10:12:43 +0100
Subject: [Python-Dev] os.utime and os.chmod failures (etc) Python 2.5b2
Message-ID: <44BF490B.1090101@voidspace.org.uk>

Hello all,

There may be a reasonable cause for this (i.e. it is likely to be my 
fault) - but it is consistent across two different machines I have tried 
it on.

With Python 2.5b2 (from the msi at Python.org), running on Windows XP 
Pro SP2,  ``os.utime`` and ``os.chmod`` fail with WindowsError. The same 
code runs fine on Python 2.3 and Python 2.4.

[err]     shutil.copytree(thisentry, targetdir)
[err]   File "C:\Python25\lib\shutil.py", line 130, in copytree
[err]     copystat(src, dst)
[err]   File "C:\Python25\lib\shutil.py", line 67, in copystat
[err]     os.utime(dst, (st.st_atime, st.st_mtime))
[err] WindowsError: [Error 13] Access is denied: 'lib\\Pmw'
[err]

The script uses ``shutil.copytree`` to copy a directory (using relative 
paths).

The source code snippet is :

if os.path.isdir(targetdir):
    shutil.rmtree(targetdir)
shutil.copytree(thisentry, targetdir)

The code in shutil.py is :

def copystat(src, dst):
    """Copy all stat info (mode bits, atime and mtime) from src to dst"""
    st = os.stat(src)
    mode = stat.S_IMODE(st.st_mode)
    if hasattr(os, 'utime'):
        os.utime(dst, (st.st_atime, st.st_mtime))
    if hasattr(os, 'chmod'):
        os.chmod(dst, mode)


Additionally, after running a couple of times I get core dumps when 
trying to run the interactive interpreter. Oddly IDLE and other programs 
still run. However this is after installing py2exe, wxPython and pywin32 
(all of which have Python 2.5 builds), so there is possibly some weird 
interaction.

My install directory has the following three dlls in it :

MSVCIRT.dll
MSVCP60.dll
MSVCRT.dll

It has none of the msvc7 dlls that I would expect.

I'm now at work and so don't have time to experiment, but tonight I will 
try uninstalling all my Python 2.5 stuff and re-installing just 2.5b2. I 
will see what dlls are present, run the interpreter a few times, and 
test a simple script with shutil.copytree.

All the best,


Michael Foord
http://www.voidspace.org.uk/python/index.shtml

From nmm1 at cus.cam.ac.uk  Thu Jul 20 12:03:41 2006
From: nmm1 at cus.cam.ac.uk (Nick Maclaren)
Date: Thu, 20 Jul 2006 11:03:41 +0100
Subject: [Python-Dev] Strategy for converting the decimal module to C
Message-ID: <E1G3VNd-0003SW-6E@draco.cus.cam.ac.uk>

Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
>
> But couldn't you just put in an interrupt handler that
> counts the interrupts, for the purpose of measurement?

No, but the reasons are very arcane.

The general reason is that taking an interrupt handler and returning
is not transparent, and is often not possible on modern systems.  If
that problem is at the hardware level (as on the Alpha and 3086/7),
you are stuffed.  But, more often, it is due the the fact that the
architecture means that such handling can only be done at maximally
privileged level.

Now, interrupting into that level has to be transparent, in order to
support TLB misses, clock interrupts, device interrupts, machine-check
interrupts and so on.  But the kernels rarely support transparent
callbacks from that state into user code (though they used to); it is
actually VERY hard to do, and even the mainframes had problems.  This
very commonly means that such counting breaks other facilities, unless
it is done IN the privileged code.

Of course, a GOOD hardware architecture wouldn't leave the process
state when it gets a floating-point interrupt, but would just invoke
an asynchronous routine call.  That used to be done.

As I said, none of this is directly relevant to emulated implementations,
such as the current Python ones, but it IS to the design of an arithmetic
specification.    It could become relevant if Python wants to start to use
a hardware implementation, because your proposal would mean that it would
have to try to ensure that such callbacks are transparent.

As one of the few people still working who has extensive experience
with doing that, I can assure you that it is an order of magnitude
fouler than you can imagine.  A decimal order of magnitude :-(

But note that I wasn't saying that such things should be put into the
API, merely that there is a very good reason to do so for hardware
implementations and ones used to tune code for such implementations.
Personally, I wouldn't bother.


Regards,
Nick Maclaren,
University of Cambridge Computing Service,
New Museums Site, Pembroke Street, Cambridge CB2 3QH, England.
Email:  nmm1 at cam.ac.uk
Tel.:  +44 1223 334761    Fax:  +44 1223 334679

From skip at pobox.com  Thu Jul 20 12:09:38 2006
From: skip at pobox.com (skip at pobox.com)
Date: Thu, 20 Jul 2006 05:09:38 -0500
Subject: [Python-Dev] logging module broken because of locale
In-Reply-To: <20060719212919.GD15030@abulafia.devel.redhat.com>
References: <20060717193955.GK3699@abulafia.devel.redhat.com>
	<44BD2054.4020705@v.loewis.de>
	<20060718185238.GB363@abulafia.devel.redhat.com>
	<200607181455.56535.fdrake@acm.org>
	<20060719212919.GD15030@abulafia.devel.redhat.com>
Message-ID: <17599.22114.395451.103228@montanaro.dyndns.org>


    Misa> Good point. Does the attached patch look reasonable?

    ...
    Misa> -        self.when = string.upper(when)
    Misa> +        self.when = unicode(when).upper()
    ...

The use of the string module instead of string methods suggests to me that
the logging package attempts to work with older versions of Python.  Looking
at PEP 291 it looks like 1.5.2 compatibility is desired (no string methods,
no unicode).  I think a conscious decision by someone (probably Vinay Sajip)
to give up that compatibility would be required.

Skip

From rasky at develer.com  Thu Jul 20 12:36:35 2006
From: rasky at develer.com (Giovanni Bajo)
Date: Thu, 20 Jul 2006 12:36:35 +0200
Subject: [Python-Dev] new security doc using object-capabilities
References: <bbaeab100607191535p543cb0ddj66410cf985dd9b77@mail.gmail.com>
Message-ID: <012801c6abe8$5f9524d0$d503030a@trilan>

Brett Cannon wrote:

>> The new doc is named securing_python.txt and
>> can be
>> found through the svn web interface at
>>
http://svn.python.org/view/python/branches/bcannon-sandboxing/securing_python.txt?rev=50717&view=log.

How do you plan to handle CPU-hogs? Stuff like execution of a gigantic
integer multiplication.

This recipe for safe_eval:
http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/496746
which is otherwise very cute, does not handle this case as well: it tries to
catch and interrupt long-running operations through a secondary thread, but
fails on a single long operation because the GIL is not released and the
alarm thread does not get its chance to run.
-- 
Giovanni Bajo


From nmm1 at cus.cam.ac.uk  Thu Jul 20 13:10:13 2006
From: nmm1 at cus.cam.ac.uk (Nick Maclaren)
Date: Thu, 20 Jul 2006 12:10:13 +0100
Subject: [Python-Dev] new security doc using object-capabilities
Message-ID: <E1G3WQ1-0007C7-Ov@libra.cus.cam.ac.uk>

"Giovanni Bajo" <rasky at develer.com> wrote:
> 
> This recipe for safe_eval:
> http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/496746
> which is otherwise very cute, does not handle this case as well: it tries to
> catch and interrupt long-running operations through a secondary thread, but
> fails on a single long operation because the GIL is not released and the
> alarm thread does not get its chance to run.

Grin :-)

You have put your finger on the Great Myth of such virtualisations,
which applies to the system-level ones and even to the hardware-level
ones.  In practice, there is always some request that a sandbox can
make to the hypervisor that can lock out or otherwise affect other
sandboxes.

The key is, of course, to admit that and to specify what is and is
not properly virtualised, so that the consequences can at least be
analysed.


Regards,
Nick Maclaren,
University of Cambridge Computing Service,
New Museums Site, Pembroke Street, Cambridge CB2 3QH, England.
Email:  nmm1 at cam.ac.uk
Tel.:  +44 1223 334761    Fax:  +44 1223 334679

From rasky at develer.com  Thu Jul 20 13:40:06 2006
From: rasky at develer.com (Giovanni Bajo)
Date: Thu, 20 Jul 2006 13:40:06 +0200
Subject: [Python-Dev] new security doc using object-capabilities
References: <E1G3WQ1-0007C7-Ov@libra.cus.cam.ac.uk>
Message-ID: <030001c6abf1$3f231690$d503030a@trilan>

Nick Maclaren wrote:

>> This recipe for safe_eval:
>> http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/496746
>> which is otherwise very cute, does not handle this case as well: it
>> tries to catch and interrupt long-running operations through a
>> secondary thread, but fails on a single long operation because the
>> GIL is not released and the alarm thread does not get its chance to
>> run.
>
> Grin :-)
>
> You have put your finger on the Great Myth of such virtualisations,
> which applies to the system-level ones and even to the hardware-level
> ones.  In practice, there is always some request that a sandbox can
> make to the hypervisor that can lock out or otherwise affect other
> sandboxes.
>
> The key is, of course, to admit that and to specify what is and is
> not properly virtualised, so that the consequences can at least be
> analysed.

I agree, and in fact Brett's work on a proper security model is greatly
welcome. It's just that us mere mortals need to use eval() *now*, and that
recipe is good enough for many practice uses. If you can't win, you can at
least lose with dignity :)
-- 
Giovanni Bajo


From ncoghlan at gmail.com  Thu Jul 20 13:43:21 2006
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 20 Jul 2006 21:43:21 +1000
Subject: [Python-Dev] new security doc using object-capabilities
In-Reply-To: <bbaeab100607191535p543cb0ddj66410cf985dd9b77@mail.gmail.com>
References: <bbaeab100607191535p543cb0ddj66410cf985dd9b77@mail.gmail.com>
Message-ID: <44BF6C59.6010707@gmail.com>

For code objects, their construction is already commonly written as 
"compile(source)".

For type objects, the constructor doesn't let you do anything you can't 
already do with a class statement. It doesn't need securing.

For rewriting import.c in Python, the PEP 302 compliant import system API in 
pkgutil would be a good starting point.

Your doc also asks about the imp.get_suffixes() list, and wonder where to set 
it from Python.

As far as I am aware, you can't. get_suffixes() is built from 
_PyImport_FileTab, which is a C array. A switch statement is used to get from 
the file table entries to the appropriate handler functions.

Quoting from the suggestions I put to the Py3k list:

Use smarter data structures
---------------------------
Currently, the individual handlers to load a fully identified module are
exposed to Python code in a way that reflects the C-style data structures used
in the current implementation.

Simply switching to more powerful data structures for the file type handlers
(i.e. use a PyTuple for filedescr values, a PyList for _PyImport_FileTab, and
a PyDict instead of a switch statement to go from filedescr values to module
loading/initialisation functions) and manipulating them all as normal Python
objects could make the code in import.c much easier to follow.

Extensible file type handling
-----------------------------
If the file type handlers are stored in normal Python data structures as
described above, it becomes feasible to make the import system extensible to
different file types as well as to different file locations.

This could be handled on a per-package basis, e.g. via a __file_types__
special attribute in packages.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
             http://www.boredomandlaziness.org

From theller at python.net  Thu Jul 20 14:26:44 2006
From: theller at python.net (Thomas Heller)
Date: Thu, 20 Jul 2006 14:26:44 +0200
Subject: [Python-Dev] os.utime and os.chmod failures (etc) Python 2.5b2
In-Reply-To: <44BF490B.1090101@voidspace.org.uk>
References: <44BF490B.1090101@voidspace.org.uk>
Message-ID: <e9nsq5$6m6$1@sea.gmane.org>

Michael Foord schrieb:
> Hello all,
> 
> There may be a reasonable cause for this (i.e. it is likely to be my 
> fault) - but it is consistent across two different machines I have tried 
> it on.
> 
> With Python 2.5b2 (from the msi at Python.org), running on Windows XP 
> Pro SP2,  ``os.utime`` and ``os.chmod`` fail with WindowsError. The same 
> code runs fine on Python 2.3 and Python 2.4.
> 
> [err]     shutil.copytree(thisentry, targetdir)
> [err]   File "C:\Python25\lib\shutil.py", line 130, in copytree
> [err]     copystat(src, dst)
> [err]   File "C:\Python25\lib\shutil.py", line 67, in copystat
> [err]     os.utime(dst, (st.st_atime, st.st_mtime))
> [err] WindowsError: [Error 13] Access is denied: 'lib\\Pmw'
> [err]
> 
> The script uses ``shutil.copytree`` to copy a directory (using relative 
> paths).

IMO this is a bug in Python 2.5, on Windows.
The problem is that the call to 'copystat(src, dst)' was added to
the shutil.copytree function, in svn r38363 probably.  It will fail
always on Windows, since os.utime does not work on directories (as the
docs correctly explain).

I guess that a patch similar to this one should fix it:

Index: shutil.py
===================================================================
--- shutil.py	(Revision 50710)
+++ shutil.py	(Arbeitskopie)
@@ -127,7 +127,12 @@
         # continue with other files
         except Error, err:
             errors.extend(err.args[0])
-    copystat(src, dst)
+    try:
+        copystat(src, dst)
+    except WindowsError:
+        pass
+    except OSError, err:
+        errors.extend(err.args[0])
     if errors:
         raise Error, errors
 
But you should report this to the bug tracker.

Thomas


From fuzzyman at voidspace.org.uk  Thu Jul 20 14:43:48 2006
From: fuzzyman at voidspace.org.uk (Michael Foord)
Date: Thu, 20 Jul 2006 13:43:48 +0100
Subject: [Python-Dev] os.utime and os.chmod failures (etc) Python 2.5b2
In-Reply-To: <e9nsq5$6m6$1@sea.gmane.org>
References: <44BF490B.1090101@voidspace.org.uk> <e9nsq5$6m6$1@sea.gmane.org>
Message-ID: <44BF7A84.2010908@voidspace.org.uk>

Thomas Heller wrote:
> Michael Foord schrieb:
>   
>> Hello all,
>>
>> There may be a reasonable cause for this (i.e. it is likely to be my 
>> fault) - but it is consistent across two different machines I have tried 
>> it on.
>>
>> With Python 2.5b2 (from the msi at Python.org), running on Windows XP 
>> Pro SP2,  ``os.utime`` and ``os.chmod`` fail with WindowsError. The same 
>> code runs fine on Python 2.3 and Python 2.4.
>>
>> [err]     shutil.copytree(thisentry, targetdir)
>> [err]   File "C:\Python25\lib\shutil.py", line 130, in copytree
>> [err]     copystat(src, dst)
>> [err]   File "C:\Python25\lib\shutil.py", line 67, in copystat
>> [err]     os.utime(dst, (st.st_atime, st.st_mtime))
>> [err] WindowsError: [Error 13] Access is denied: 'lib\\Pmw'
>> [err]
>>
>> The script uses ``shutil.copytree`` to copy a directory (using relative 
>> paths).
>>     
>
> IMO this is a bug in Python 2.5, on Windows.
> The problem is that the call to 'copystat(src, dst)' was added to
> the shutil.copytree function, in svn r38363 probably.  It will fail
> always on Windows, since os.utime does not work on directories (as the
> docs correctly explain).
>
> I guess that a patch similar to this one should fix it:
>
> Index: shutil.py
> ===================================================================
> --- shutil.py	(Revision 50710)
> +++ shutil.py	(Arbeitskopie)
> @@ -127,7 +127,12 @@
>          # continue with other files
>          except Error, err:
>              errors.extend(err.args[0])
> -    copystat(src, dst)
> +    try:
> +        copystat(src, dst)
> +    except WindowsError:
> +        pass
> +    except OSError, err:
> +        errors.extend(err.args[0])
>      if errors:
>          raise Error, errors
>  
> But you should report this to the bug tracker.
>   
Ok, thanks.

Michael
http://www.voidspace.org.uk/python/index.shtml

> Thomas
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk
>
>   


From mihaiibanescu at yahoo.com  Thu Jul 20 15:49:05 2006
From: mihaiibanescu at yahoo.com (Mihai Ibanescu)
Date: Thu, 20 Jul 2006 09:49:05 -0400
Subject: [Python-Dev] logging module broken because of locale
In-Reply-To: <17599.22114.395451.103228@montanaro.dyndns.org>
References: <20060717193955.GK3699@abulafia.devel.redhat.com>
	<44BD2054.4020705@v.loewis.de>
	<20060718185238.GB363@abulafia.devel.redhat.com>
	<200607181455.56535.fdrake@acm.org>
	<20060719212919.GD15030@abulafia.devel.redhat.com>
	<17599.22114.395451.103228@montanaro.dyndns.org>
Message-ID: <20060720134905.GA25083@abulafia.devel.redhat.com>

On Thu, Jul 20, 2006 at 05:09:38AM -0500, skip at pobox.com wrote:
> 
>     Misa> Good point. Does the attached patch look reasonable?
> 
>     ...
>     Misa> -        self.when = string.upper(when)
>     Misa> +        self.when = unicode(when).upper()
>     ...
> 
> The use of the string module instead of string methods suggests to me that
> the logging package attempts to work with older versions of Python.  Looking
> at PEP 291 it looks like 1.5.2 compatibility is desired (no string methods,
> no unicode).  I think a conscious decision by someone (probably Vinay Sajip)
> to give up that compatibility would be required.

Agreed. There is a note that the module should work with python >= 1.5.2 at
the top of the module.

It's up to Vinay to decide if we want to drop support for 1.5.2 in the module
included in newer pythons, or the attached patch would make it work for 1.5.2
as well (as in "it's not more broken than before"). 

I would like to redo the patch once more to get rid of the try-except and use
__builtins__ instead (but for some reason it kept jumping from being a module
to being a dictionary and I just wanted the proof of concept).

Misa
-------------- next part --------------
--- Python-2.4.3/Lib/logging/handlers.py.nolocale	2006-07-19 12:15:46.000000000 -0400
+++ Python-2.4.3/Lib/logging/handlers.py	2006-07-20 09:45:57.000000000 -0400
@@ -162,7 +162,7 @@
     """
     def __init__(self, filename, when='h', interval=1, backupCount=0, encoding=None):
         BaseRotatingHandler.__init__(self, filename, 'a', encoding)
-        self.when = string.upper(when)
+        self.when = _upper(when)
         self.backupCount = backupCount
         # Calculate the real rollover interval, which is just the number of
         # seconds between rollovers.  Also set the filename suffix used when
@@ -645,7 +645,7 @@
         """
         msg = self.log_format_string % (
             self.encodePriority(self.facility,
-                                string.lower(record.levelname)),
+                                _lower(record.levelname)),
             msg)
         try:
             if self.unixsocket:
@@ -854,7 +854,7 @@
         ("GET" or "POST")
         """
         logging.Handler.__init__(self)
-        method = string.upper(method)
+        method = _upper(method)
         if method not in ["GET", "POST"]:
             raise ValueError, "method must be GET or POST"
         self.host = host
@@ -1007,3 +1007,25 @@
         self.flush()
         self.target = None
         BufferingHandler.close(self)
+
+def _upper(s):
+    """A version of upper() that tries to be locale-independent by converting
+    the string to unicode (which is not subject to case conversion being
+    locale specific)
+    """
+    try:
+        ret = str(unicode(s).upper())
+    except NameError:
+        ret = string.upper(s)
+    return ret
+
+def _lower(s):
+    """A version of lower() that tries to be locale-independent by converting
+    the string to unicode (which is not subject to case conversion being
+    locale specific)
+    """
+    try:
+        ret = str(unicode(s).lower())
+    except NameError:
+        ret = string.lower(s)
+    return ret

From jimjjewett at gmail.com  Thu Jul 20 17:39:27 2006
From: jimjjewett at gmail.com (Jim Jewett)
Date: Thu, 20 Jul 2006 11:39:27 -0400
Subject: [Python-Dev] logging module broken because of locale
Message-ID: <fb6fbf560607200839y75e6414eqe0f79c985999db60@mail.gmail.com>

Mihai,

It does make sense to document this limit for people writing
subclasses, or using a Turkic codeset.  I'm not sure that logging is
the right place to document it, and I don't think changing the base
classes is a good idea.

TimedRotatingFileHandler and HTTPHandler restrict their input to a
tiny subset of ASCII; anything that would be handled differently after
these changes will already raise a ValueError unless you subclass.

Even SysLogHandler.emit doesn't actually print the string; it is only
used as a lookup key for a dictionary whose keys are all lower-case
ASCII.  In theory, you could monkey-patch that dictionary to add
additional values, but then you might as well subclass to do the right
thing with your new keys.  (It *might* make sense to change the base
function and dictionary to accept unicode, including undotted-i
synonyms.)

-jJ

From martin at v.loewis.de  Thu Jul 20 18:08:05 2006
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Thu, 20 Jul 2006 18:08:05 +0200
Subject: [Python-Dev] logging module broken because of locale
In-Reply-To: <20060720134905.GA25083@abulafia.devel.redhat.com>
References: <20060717193955.GK3699@abulafia.devel.redhat.com>
	<44BD2054.4020705@v.loewis.de>
	<20060718185238.GB363@abulafia.devel.redhat.com>
	<200607181455.56535.fdrake@acm.org>
	<20060719212919.GD15030@abulafia.devel.redhat.com>
	<17599.22114.395451.103228@montanaro.dyndns.org>
	<20060720134905.GA25083@abulafia.devel.redhat.com>
Message-ID: <44BFAA65.5040501@v.loewis.de>

Mihai Ibanescu wrote:
> It's up to Vinay to decide if we want to drop support for 1.5.2 in the module
> included in newer pythons, or the attached patch would make it work for 1.5.2
> as well (as in "it's not more broken than before"). 

That still wouldn't work with Python 1.5.2, as that version did not
support Unicode at all.

Regards,
Martin

From mihaiibanescu at yahoo.com  Thu Jul 20 18:13:14 2006
From: mihaiibanescu at yahoo.com (Mihai Ibanescu)
Date: Thu, 20 Jul 2006 12:13:14 -0400
Subject: [Python-Dev] logging module broken because of locale
In-Reply-To: <44BFAA65.5040501@v.loewis.de>
References: <20060717193955.GK3699@abulafia.devel.redhat.com>
	<44BD2054.4020705@v.loewis.de>
	<20060718185238.GB363@abulafia.devel.redhat.com>
	<200607181455.56535.fdrake@acm.org>
	<20060719212919.GD15030@abulafia.devel.redhat.com>
	<17599.22114.395451.103228@montanaro.dyndns.org>
	<20060720134905.GA25083@abulafia.devel.redhat.com>
	<44BFAA65.5040501@v.loewis.de>
Message-ID: <20060720161314.GI25083@abulafia.devel.redhat.com>

On Thu, Jul 20, 2006 at 06:08:05PM +0200, "Martin v. L?wis" wrote:
> Mihai Ibanescu wrote:
> > It's up to Vinay to decide if we want to drop support for 1.5.2 in the module
> > included in newer pythons, or the attached patch would make it work for 1.5.2
> > as well (as in "it's not more broken than before"). 
> 
> That still wouldn't work with Python 1.5.2, as that version did not
> support Unicode at all.

Yes, as I said, it won't be more broken than before applying the patch (my
first patch was breaking 1.5.2 completely).

If people choose to compile python 2.4.3 without unicode support, it won't
work either. But if you choose to disable unicode you probably live in a very
constrained environment and you may not be affected by the locale bug at all.

I agree the patch is not perfect :-) but I think it solves the problem for
most people.

Misa

From mihaiibanescu at yahoo.com  Thu Jul 20 18:19:02 2006
From: mihaiibanescu at yahoo.com (mihaiibanescu at yahoo.com)
Date: Thu, 20 Jul 2006 12:19:02 -0400
Subject: [Python-Dev] logging module broken because of locale
In-Reply-To: <fb6fbf560607200839y75e6414eqe0f79c985999db60@mail.gmail.com>
References: <fb6fbf560607200839y75e6414eqe0f79c985999db60@mail.gmail.com>
Message-ID: <20060720161902.GJ25083@abulafia.devel.redhat.com>

On Thu, Jul 20, 2006 at 11:39:27AM -0400, Jim Jewett wrote:
> Mihai,
> 
> It does make sense to document this limit for people writing
> subclasses, or using a Turkic codeset.  I'm not sure that logging is
> the right place to document it, and I don't think changing the base
> classes is a good idea.
> 
> TimedRotatingFileHandler and HTTPHandler restrict their input to a
> tiny subset of ASCII; anything that would be handled differently after
> these changes will already raise a ValueError unless you subclass.
> 
> Even SysLogHandler.emit doesn't actually print the string; it is only
> used as a lookup key for a dictionary whose keys are all lower-case
> ASCII.  In theory, you could monkey-patch that dictionary to add
> additional values, but then you might as well subclass to do the right
> thing with your new keys.  (It *might* make sense to change the base
> function and dictionary to accept unicode, including undotted-i
> synonyms.)

Hi Jim,

I am afraid you lost me.

My initial approach was to compute the values of the dictionary based on the
current locale - and that doesn't work.

The issue is not that the dictionary doesn't accept unicode, it's that there's
no reliable way to do lookups in it. If you could sketch an example of what
you had in mind with the monkey-patching, please do so, since I don't think I
grasped the idea.

Misa

From martin at v.loewis.de  Thu Jul 20 18:41:04 2006
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Thu, 20 Jul 2006 18:41:04 +0200
Subject: [Python-Dev] logging module broken because of locale
In-Reply-To: <20060720161314.GI25083@abulafia.devel.redhat.com>
References: <20060717193955.GK3699@abulafia.devel.redhat.com>
	<44BD2054.4020705@v.loewis.de>
	<20060718185238.GB363@abulafia.devel.redhat.com>
	<200607181455.56535.fdrake@acm.org>
	<20060719212919.GD15030@abulafia.devel.redhat.com>
	<17599.22114.395451.103228@montanaro.dyndns.org>
	<20060720134905.GA25083@abulafia.devel.redhat.com>
	<44BFAA65.5040501@v.loewis.de>
	<20060720161314.GI25083@abulafia.devel.redhat.com>
Message-ID: <44BFB220.8060802@v.loewis.de>

Mihai Ibanescu wrote:
> Yes, as I said, it won't be more broken than before applying the patch (my
> first patch was breaking 1.5.2 completely).

Ah, I didn't notice that it deals with unicode() not being a builtin.
That's fine then.

Regards,
Martin

From brett at python.org  Thu Jul 20 19:04:43 2006
From: brett at python.org (Brett Cannon)
Date: Thu, 20 Jul 2006 10:04:43 -0700
Subject: [Python-Dev] new security doc using object-capabilities
In-Reply-To: <012801c6abe8$5f9524d0$d503030a@trilan>
References: <bbaeab100607191535p543cb0ddj66410cf985dd9b77@mail.gmail.com>
	<012801c6abe8$5f9524d0$d503030a@trilan>
Message-ID: <bbaeab100607201004j324b3856jb43b3cc3c3ec22c8@mail.gmail.com>

On 7/20/06, Giovanni Bajo <rasky at develer.com> wrote:
>
> Brett Cannon wrote:
>
> >> The new doc is named securing_python.txt and
> >> can be
> >> found through the svn web interface at
> >>
>
> http://svn.python.org/view/python/branches/bcannon-sandboxing/securing_python.txt?rev=50717&view=log
> .
>
> How do you plan to handle CPU-hogs? Stuff like execution of a gigantic
> integer multiplication.


I don't.  =)  Protecting the CPU is damn hard to do in any form of portable
fashion.  And even getting it to work on an OS you do know the details of
leads to probably an interrupt  implementation and that doesn't sound fun.

-Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20060720/4b9938f9/attachment.html 

From jimjjewett at gmail.com  Thu Jul 20 19:06:32 2006
From: jimjjewett at gmail.com (Jim Jewett)
Date: Thu, 20 Jul 2006 13:06:32 -0400
Subject: [Python-Dev] logging module broken because of locale
In-Reply-To: <20060720161902.GJ25083@abulafia.devel.redhat.com>
References: <fb6fbf560607200839y75e6414eqe0f79c985999db60@mail.gmail.com>
	<20060720161902.GJ25083@abulafia.devel.redhat.com>
Message-ID: <fb6fbf560607201006k171f810am91b1425b02dfddfd@mail.gmail.com>

On 7/20/06, mihaiibanescu at yahoo.com <mihaiibanescu at yahoo.com> wrote:
> On Thu, Jul 20, 2006 at 11:39:27AM -0400, Jim Jewett wrote:

> > Even SysLogHandler.emit doesn't actually print the string; it is only
> > used as a lookup key for a dictionary whose keys are all lower-case
> > ASCII.  In theory, you could monkey-patch that dictionary to add
> > additional values, but then you might as well subclass to do the right
> > thing with your new keys.  (It *might* make sense to change the base
> > function and dictionary to accept unicode, including undotted-i
> > synonyms.)

> My initial approach was to compute the values of the dictionary based on the
> current locale - and that doesn't work.

Correct, it would need to be done by hand.  As part of the current
class construction,

    SysLogHandler.priority_names["info"] = SysLogHandler.LOG_INFO

You could add other entries, so that

    SysLogHandler.priority_names["my_info"] = SysLogHandler.LOG_INFO

If you also changed SysLogHandler.encode_priority to accept unicode
keys, you could even add u"info" and the equivalent with an
undotted-i.  I'm not sure this is worth doing, though, since they
supposedly represent symbolic constants.

-jJ


> The issue is not that the dictionary doesn't accept unicode, it's that there's
> no reliable way to do lookups in it. If you could sketch an example of what
> you had in mind with the monkey-patching, please do so, since I don't think I
> grasped the idea.
>
> Misa
>

From brett at python.org  Thu Jul 20 19:09:07 2006
From: brett at python.org (Brett Cannon)
Date: Thu, 20 Jul 2006 10:09:07 -0700
Subject: [Python-Dev] new security doc using object-capabilities
In-Reply-To: <44BF6C59.6010707@gmail.com>
References: <bbaeab100607191535p543cb0ddj66410cf985dd9b77@mail.gmail.com>
	<44BF6C59.6010707@gmail.com>
Message-ID: <bbaeab100607201009x28a7354elde374c6651b1a322@mail.gmail.com>

On 7/20/06, Nick Coghlan <ncoghlan at gmail.com> wrote:
>
> For code objects, their construction is already commonly written as
> "compile(source)".


Right, but some people like to construct directly from bytecode.

For type objects, the constructor doesn't let you do anything you can't
> already do with a class statement. It doesn't need securing.


I figured as much, but when I was making the list I was not sure and didn't
want to stop my writing momentum to check.

For rewriting import.c in Python, the PEP 302 compliant import system API in
> pkgutil would be a good starting point.


Yep.  Plan on looking at all of the various modules in the stdlib that
assist with importing, package PEP (I think there is one), and PEP 302.

Your doc also asks about the imp.get_suffixes() list, and wonder where to
> set
> it from Python.
>
> As far as I am aware, you can't. get_suffixes() is built from
> _PyImport_FileTab, which is a C array. A switch statement is used to get
> from
> the file table entries to the appropriate handler functions.


Ah, OK.

Quoting from the suggestions I put to the Py3k list:
>
> Use smarter data structures
> ---------------------------
> Currently, the individual handlers to load a fully identified module are
> exposed to Python code in a way that reflects the C-style data structures
> used
> in the current implementation.
>
> Simply switching to more powerful data structures for the file type
> handlers
> (i.e. use a PyTuple for filedescr values, a PyList for _PyImport_FileTab,
> and
> a PyDict instead of a switch statement to go from filedescr values to
> module
> loading/initialisation functions) and manipulating them all as normal
> Python
> objects could make the code in import.c much easier to follow.


Yep.  I just kind of glanced at the rest of your suggestions, Nick, since I
assumed a lot of it would change (or could be changed) if import was redone
in as much Python as possible.

Extensible file type handling
> -----------------------------
> If the file type handlers are stored in normal Python data structures as
> described above, it becomes feasible to make the import system extensible
> to
> different file types as well as to different file locations.


Yep.  Although I am more interested in restricting than broadening the file
types.

This could be handled on a per-package basis, e.g. via a __file_types__
> special attribute in packages.


Maybe.  I don't want to get into introducing new abilities to start, though.


-Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20060720/32151d26/attachment-0001.htm 

From brett at python.org  Thu Jul 20 19:30:03 2006
From: brett at python.org (Brett Cannon)
Date: Thu, 20 Jul 2006 10:30:03 -0700
Subject: [Python-Dev] new security doc using object-capabilities
In-Reply-To: <9eebf5740607200117r4d4613e2i91665ea211bab46@mail.gmail.com>
References: <bbaeab100607191535p543cb0ddj66410cf985dd9b77@mail.gmail.com>
	<9eebf5740607200117r4d4613e2i91665ea211bab46@mail.gmail.com>
Message-ID: <bbaeab100607201030l2842b416wcfd8007c86c10e4f@mail.gmail.com>

On 7/20/06, Lawrence Oluyede <l.oluyede at gmail.com> wrote:
>
> That's great. I just read your draft but I have little comments to do
> but before let me say that I liked the idea to borrow concepts from E.
> I've crossed the E's path in the beginning of this year and I found it
> a pot of really nice ideas (for promises and capabilities). Here are
> my comments about the draft:
>
> - it's not really clear to me what the "powerbox" is. I think I got
> the concept of "super process" but maybe it's to be clarified, isn't
> it? It become clear in the "threat model" paragraph


The powerbox is the thing that gives your security domains their initial
abilities.  The OS gives the process its abilities, but it does not directly
work with the interpreter.  Since the process does, though, it is considered
the powerbox and farms out abilities that it has been given by the OS.

I have tried to clarify the definition at the start of the doc.

- I hope no Rubystas will read the "Problem of No Private Namespace"
> section because they have private/protected keywords to enforce this
> stuff :-) Writing proxies in C will slow down the dev process (altough
> will speed up the performance maybe) but in a far future someone will
> come up with an alternative closer to the Python level


Maybe.  As I said in the doc, any changes must be Pythonic and adding
private namespaces right now wouldn't be without much more thought and work.

And if Ruby ends up with this security model but more thoroughly, more power
to them.  Their language is different in the right ways to support it.

As for coding in C, thems the breaks.  I plan in adding stuff to the stdlib
for the common case.  I might eventually think of a good, generic proxy
object that could be used, but as of right now I am not worrying about that
since it would be icing on the cake.

- Can you write down a simple example of what you mean with "changing
> something of the built-in objects"? (in "Problem of mutable shared
> state")


Done.

- What about the performance issues of the capabilities model overall?


Should be faster than an IBAC model since certain calls will not need to
check the identity of the caller every time.

But I am not worrying about performance, I am worrying about correctness, so
I did not try to make any performance claims.

- I know what you meant to say but the paragraph about pythonicness
> and the security model seems a little "fuzzy" to me. Which are the
> boundaries of the allowed changes for the security stuff?


Being "pythonic" is a fuzzy term in itself and Guido is the only person who
can make definitive claims over what is and is not Pythonic.  As I have
said, this doc was mostly written with python-dev in mind since they are the
ones I have to convince to let this into the core and they all know the
term.

But I have tacked in a sentence on what the term means.

- You don't say anything about networking and networked resources in
> the list of the standard sandboxed interpreter


Nope.  Have not started worrying about that yet.  Just trying to get the
basic model laid out.

- Suppose we have a .py module. Based on your security model we can
> import it, right? When imported it generates a .pyc file. The second
> time we import it what happens? .pyc is ignored? import is not allowed
> at all? We can't rely on the name of the file.pyc because an attacker
> who knows the file.py is secure and the second import is done against
> file.pyc can replace the "secure" file.pyc with an implementation not
> secure and can do some kind of harm to the sandbox


It will be ignored.  But I am hoping that through rewriting the import
machinery more control over generating .pyc files can be had (see Skip
Montanaro's PEP on this; forget the number).  This is why exact details were
left out of the implementation details.  I just wanted people understand the
approach to everything, not the concrete details of how it will be coded up.

- About "Filesystem information". Does the sandboxed interpreter need
> to know all that information about file paths, files and so on? Can't
> we reset those attributes to something arbitrary?


That is the point.  It is not that the sandbox needs to know it, its that it
needs to be hidden from the sandbox.

- About sys module: I think the best way is to have a purged fake sys
> module with only the stuff you need. pypy has the concept of faked
> modules too (altough for a different reason)


OK.

- About networking: what do you think about the E's model of really
> safe networking, protected remotable objects and safe RPC? Is that
> model applicable to Python's in some way? We can't use the E's model
> as a whole (ask people to generate a safe key and send it by email is
> unfeasible)


I have not looked at it.  I am also not trying to build an RPC system *and*
a security model for Python.  That is just too much work right now.

- is the protected memory model a some kind of memory monitor system?


Basically.  It just keeps a size_t on the memory cap and another on memory
usage, and when memory is requested it makes sure that it won't go over the
cap.  And when memory is freed the usage goes down.  It's very rough (hard
to account for padding bits, etc. in C structs), but it should be good
enough to prevent a program from hitting 800 MB when you really just wanted
it to have 5 MB.

I think that's all for the draft. I wrote these comments during the
> reading of the document.
>
> Hope some of these help


Thanks, Lawrence.

-Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20060720/77ae9261/attachment.html 

From rasky at develer.com  Thu Jul 20 19:40:46 2006
From: rasky at develer.com (Giovanni Bajo)
Date: Thu, 20 Jul 2006 19:40:46 +0200
Subject: [Python-Dev] new security doc using object-capabilities
References: <bbaeab100607191535p543cb0ddj66410cf985dd9b77@mail.gmail.com>
	<012801c6abe8$5f9524d0$d503030a@trilan>
	<bbaeab100607201004j324b3856jb43b3cc3c3ec22c8@mail.gmail.com>
Message-ID: <084e01c6ac23$a1930ec0$d503030a@trilan>

Brett Cannon wrote:

>>
http://svn.python.org/view/python/branches/bcannon-sandboxing/securing_python.txt?rev=50717&view=log
>> .
>>
>> How do you plan to handle CPU-hogs? Stuff like execution of a
>> gigantic integer multiplication.
>
>
> I don't.  =)  Protecting the CPU is damn hard to do in any form of
> portable fashion.  And even getting it to work on an OS you do know
> the details of leads to probably an interrupt  implementation and
> that doesn't sound fun.

I think the trick used by the safe_eval recipe (a separate thread which
interrupts the script through thread.interrupt_main()) shows that, in most
cases, it's possible to make sure that an embedded script does not take too
long to execute. Do you agree that this usage case ("allow me to timeout an
embedded script") is something which would be a very good start in the right
direction?

Now, I wonder, in a restricted execution environment such as that depicted
in your document, how many different ways are there to make the Python
interpreter enter a long calcolation loop which does not release the GIL? I
can think of bignum*bignum, bignum**bignum or similar mathematical
operations, but there are really a few. If we could make those release the
GIL (or poll some kind of watchdog used to abort them, pretty much like they
normally poll CTRL+C), then the same trick used by the recipe could be used.
-- 
Giovanni Bajo


From brett at python.org  Thu Jul 20 20:10:29 2006
From: brett at python.org (Brett Cannon)
Date: Thu, 20 Jul 2006 11:10:29 -0700
Subject: [Python-Dev] new security doc using object-capabilities
In-Reply-To: <084e01c6ac23$a1930ec0$d503030a@trilan>
References: <bbaeab100607191535p543cb0ddj66410cf985dd9b77@mail.gmail.com>
	<012801c6abe8$5f9524d0$d503030a@trilan>
	<bbaeab100607201004j324b3856jb43b3cc3c3ec22c8@mail.gmail.com>
	<084e01c6ac23$a1930ec0$d503030a@trilan>
Message-ID: <bbaeab100607201110i4e4c924fvbc57425a7e4b3a36@mail.gmail.com>

On 7/20/06, Giovanni Bajo <rasky at develer.com> wrote:
>
> Brett Cannon wrote:
>
> >>
>
> http://svn.python.org/view/python/branches/bcannon-sandboxing/securing_python.txt?rev=50717&view=log
> >> .
> >>
> >> How do you plan to handle CPU-hogs? Stuff like execution of a
> >> gigantic integer multiplication.
> >
> >
> > I don't.  =)  Protecting the CPU is damn hard to do in any form of
> > portable fashion.  And even getting it to work on an OS you do know
> > the details of leads to probably an interrupt  implementation and
> > that doesn't sound fun.
>
> I think the trick used by the safe_eval recipe (a separate thread which
> interrupts the script through thread.interrupt_main()) shows that, in most
> cases, it's possible to make sure that an embedded script does not take
> too
> long to execute. Do you agree that this usage case ("allow me to timeout
> an
> embedded script") is something which would be a very good start in the
> right
> direction?


Probably.  I just don't feel like worrying about it right now.  =)

Now, I wonder, in a restricted execution environment such as that depicted
> in your document, how many different ways are there to make the Python
> interpreter enter a long calcolation loop which does not release the GIL?
> I
> can think of bignum*bignum, bignum**bignum or similar mathematical
> operations, but there are really a few. If we could make those release the
> GIL (or poll some kind of watchdog used to abort them, pretty much like
> they
> normally poll CTRL+C), then the same trick used by the recipe could be
> used.


Well, any work that does most of its calculation within C code and that does
not touch base with the interpreter on a semi-regular basis would need to
relesae the GIL.

-Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20060720/d60aea40/attachment.htm 

From pje at telecommunity.com  Thu Jul 20 20:57:07 2006
From: pje at telecommunity.com (Phillip J. Eby)
Date: Thu, 20 Jul 2006 14:57:07 -0400
Subject: [Python-Dev] Undocumented PEP 302 protocol change by need-for-speed
	sprint
Message-ID: <5.1.1.6.0.20060720144252.02622318@sparrow.telecommunity.com>

While investigating the need to apply http://python.org/sf/1525766 I found 
that there was a modification to pkgutil during the need-for-speed sprint 
that affects the PEP 302 protocol in a backwards incompatible way.

Specifically, PEP 302 documents that path_importer_cache always contains 
either importer objects or None.  Any code written to obtain importer 
objects is therefore now broken, because import.c is slapping False in for 
non-existent filesystem paths.

The pkgutil module was then hacked to work around this problem, thereby 
hiding the breakage from at least the standard library, but not any 
external libraries that follow the PEP 302 protocol to find importers.

There are several options as to how to proceed:

1. Revert the change
2. Document the breakage, update PEP 302, and make everybody update their code
3. Make it not break existing code, by using a NonexistentPathImporter or 
NullImporter type in place of "False" in sys.path_importer_cache.

Any thoughts?

Personally, the only code I know of that implements the PEP 302 protocol 
besides the pkgutil module that would be affected is pkg_resources in 
setuptools, so it's not like I can't fix it for 2.5.

However, I don't know if anybody else is using the protocol, and if so, how 
bad the breakage would be.  This should really only affect code that is 
walking sys.path, because paths with "False" in sys.path_importer_cache by 
definition cannot have any importable modules associated with them.  So, 
although I don't like option 2 on general principles, it may be an 
acceptable solution.


From tjreedy at udel.edu  Thu Jul 20 21:00:45 2006
From: tjreedy at udel.edu (Terry Reedy)
Date: Thu, 20 Jul 2006 15:00:45 -0400
Subject: [Python-Dev] new security doc using object-capabilities
References: <bbaeab100607191535p543cb0ddj66410cf985dd9b77@mail.gmail.com>
	<9eebf5740607200117r4d4613e2i91665ea211bab46@mail.gmail.com>
Message-ID: <e9ojsu$ui1$1@sea.gmane.org>


"Lawrence Oluyede" <l.oluyede at gmail.com> wrote in message 
news:9eebf5740607200117r4d4613e2i91665ea211bab46 at mail.gmail.com...
> - I know what you meant to say but the paragraph about pythonicness
> and the security model seems a little "fuzzy" to me.

I agree that this paragraph is weak and recommend that it be rewritten.  In 
particular, I think the 'pythonic*' words should go, especially if you 
expect this document to be read by anyone other than dedicated pythonistas. 
I would start with something like

"It is my goal that my thesis work be incorporated in some future version 
of the Python distribution.  This has two constraints.  First, changes to 
the core must not slow down normal operation.  Second, visible changes must 
not violate the spirit and style of Python that make it a distinctive 
language."

This alludes to the fact that your proposal discusses two highly 
overlapping yet separate projects: write a thesis that gains you a PhD 
degree; and produce an accepted patch set that give Python a useful 
security capability it does not now have.  They have to be thought of as 
somewhat separate because you have two sets of 'overseers' and approvers: 
your thesis advisor and committee for the first; and Guido and other Python 
developers for the second.

I think your thesis should currently be your first priority.  Your current 
paragraph implied to me that you would not follow a promising line of 
research if you could not see how to make it 'pythonic'.  If I were on your 
thesis committee, I think that would bother me ;-).

In any case, I wish you the best with a double project that is obviously 
not a 'gimme'.

Terry Jan Reedy (PhD, though not on any thesis committees)




From l.oluyede at gmail.com  Thu Jul 20 21:20:17 2006
From: l.oluyede at gmail.com (Lawrence Oluyede)
Date: Thu, 20 Jul 2006 21:20:17 +0200
Subject: [Python-Dev] new security doc using object-capabilities
In-Reply-To: <bbaeab100607201030l2842b416wcfd8007c86c10e4f@mail.gmail.com>
References: <bbaeab100607191535p543cb0ddj66410cf985dd9b77@mail.gmail.com>
	<9eebf5740607200117r4d4613e2i91665ea211bab46@mail.gmail.com>
	<bbaeab100607201030l2842b416wcfd8007c86c10e4f@mail.gmail.com>
Message-ID: <9eebf5740607201220t3edcb31k4b08d6b27f1546d6@mail.gmail.com>

> Should be faster than an IBAC model since certain calls will not need to
> check the identity of the caller every time.
>
> But I am not worrying about performance, I am worrying about correctness, so
> I did not try to make any performance claims.

Got that.

> Nope.  Have not started worrying about that yet.  Just trying to get the
> basic model laid out.

Ok sorry to have bothered

> That is the point.  It is not that the sandbox needs to know it, its that it
> needs to be hidden from the sandbox.

So I think that's a "simple" step during the importing step.

> I have not looked at it.  I am also not trying to build an RPC system *and*
> a security model for Python.  That is just too much work right now.

Ok sorry :-)

> Thanks, Lawrence.

Thank you!

-- 
Lawrence
http://www.oluyede.org/blog

From barry at python.org  Thu Jul 20 21:27:39 2006
From: barry at python.org (Barry Warsaw)
Date: Thu, 20 Jul 2006 15:27:39 -0400
Subject: [Python-Dev] Support for PyGetSetDefs in pydoc
In-Reply-To: <7FE62713-24B0-47CE-BA94-41150DB683F4@python.org>
References: <A027CA35-EE77-4658-BA59-E849368C440B@python.org>
	<EF6E6AF2-654F-4F53-91A8-A6C9B8A80745@python.org>
	<44B64782.9010706@gmail.com>
	<7FE62713-24B0-47CE-BA94-41150DB683F4@python.org>
Message-ID: <AB1C2209-329E-4F3D-8B7A-2548A40F892A@python.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


On Jul 13, 2006, at 12:12 PM, Barry Warsaw wrote:

> I've updated SF patch #1520294 and assigned it back to Georg for  
> another quick review.

Neal commented in the patch that it might help to explain the  
implementation a bit.  I'd like to do that and also explain why I  
think this patch should go in 2.5.  OTOH, I suspect most people just  
don't care, which is why I've gotten almost no comments on the patch  
(other than one or two mild nods of approval).

As a reminder, this patch is really about providing useful help() at  
the interactive prompt for two specific types of C implemented  
descriptors, getset and member descriptors.  Both objects can be  
given docstrings in the C code, but currently nothing in the tool  
chain pulls out and displays those docstrings, so help() on such  
objects is not very useful.  While certainly a minor point, I think  
it improves the user experience to be able to display any existing  
docstrings for these types of objects.  Of course, I'm scratching my  
own itch here because our app is deeply embedded and we have a lot of  
getset and member descriptors with docstrings.  I'd like for our  
users to be able to type "help(someobj.member)" at our interactive  
prompt and get our docstring printed back at them.

The implementation is fairly straightforward  I add two new constants  
to types.py, a couple of is*() methods to inspect.py, and a bit of  
extra support to pydoc.py.  Where it gets complicated is that I added  
a _types builtin module to provide a getset and member descriptor  
that could be turned into *Type constants in the types.py module.

Why did I do this instead of trying to hunt down some existing getset  
or member descriptor?  For one thing, there really aren't very good  
candidates for such objects in the built-in modules.  You can't use  
objects like datetime.timedelta.days in types.py because datetime is  
not importable early enough for types.py to use it. Even if there  
were likely candidates, they would be accidents of implementation and  
it doesn't seem like a good idea to force force some future datetime  
maintainer to have to fix types.py when she decides that  
datetime.timedelta.days should be implemented in some other way.

A 3rd party extension module doesn't work either because you really  
need the tie-in from types.py, inspect.py, and pydoc.py.  You  
certainly don't want to go poking new values into types.py from the  
outside, and besides inspect.py and pydoc.py also need to learn about  
these fundamental built-in Python types.

ISTM the most straightforward approach is to provide these types in a  
built-in C module, linked into the interpreter so types.py can get  
access to it as early as it needs to.  Also, because the type that  
these descriptors are attached to cannot be instantiated from Python,  
they should be quite benign, existing only in order to give type()  
something to query.

Neal also questions the reliance on the types.py module and whether  
we're moving away from it.  My own feeling is that certainly for  
common types available as type constructors in the built-in namespace  
(e.g. list, set, dict, etc.), no, we certainly don't need types.py.   
But there are enough other tools in Pythonland that need to  
introspect about types that aren't readily available, and I see no  
reason why types.py shouldn't continue to fulfill those needs.  How  
else would you find the type of frames or generators in a simple,  
documented, consistent way?

Neal also asks about other implementations.  I believe that I've  
patched types.py, inspect.py, and pydoc.py in such a way that if the  
implementation did not have these types available, those modules  
would not break.  For example, GetSetDescriptorType and  
MemberDescriptorType are not defined if _types can't be imported.  If  
those types aren't defined, the appropriate inspect.py is() methods  
will always return False.  Thus, other implementations can do hasattr 
() tests on types.py or better yet, inspect.isgetsetdescriptor() and  
get an answer it can deal with.

Further, by providing a _types built-in module, other  
implementations /could/ expose those types, if they were meaningful.   
I believe that an _types module provides a good place for those  
alternative implementations to hide types of their own should the  
need arise (e.g. if Jython had some implementation-specific  
JavaSpecialUnicornType, the object would live in Jython's _types.java  
and be exposed via a type(_type.javaspecialunicorn) in types.py).

Could this wait until Python 2.6?  Well, it /could/ but IMO it  
shouldn't.  Improving the user experience by providing useful help()  
shouldn't be postponed when we have the opportunity to do so now,  
with low risk of breakage.  Probably the biggest risk is whether I  
can modify the Windows build to DTRT with _types.c.  I've been  
building enough Windows s/w lately that I think I can handle it, and  
if not, I'll just buy the Timbot lunch as payment for his help.  I  
know what he likes to eat.

  So anyway, that's probably more text than the patch combined. :)  I  
hope it helps.
- -Barry

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.3 (Darwin)

iQCVAwUBRL/ZL3EjvBPtnXfVAQKEvgQAktmdjKnq5uf1+m5Wc5F1ZFY/LhdM1vQQ
A2v6zgXTxdQteqTNds15BktGAUwRNKKB2pU/U2wTcfLx/4rfjvkFG3cRbhEY8jI0
wa8wzPrGs17VTp8Z/LI65F9DC8aMEa9RBr2fXeuhewXlAheiZT9doelds8b5y0T/
Ed9Yl4MJ0es=
=Fh1P
-----END PGP SIGNATURE-----

From brett at python.org  Thu Jul 20 21:28:46 2006
From: brett at python.org (Brett Cannon)
Date: Thu, 20 Jul 2006 12:28:46 -0700
Subject: [Python-Dev] Undocumented PEP 302 protocol change by
	need-for-speed sprint
In-Reply-To: <5.1.1.6.0.20060720144252.02622318@sparrow.telecommunity.com>
References: <5.1.1.6.0.20060720144252.02622318@sparrow.telecommunity.com>
Message-ID: <bbaeab100607201228y797b601alac230a13c00c9b59@mail.gmail.com>

On 7/20/06, Phillip J. Eby <pje at telecommunity.com> wrote:
>
> While investigating the need to apply http://python.org/sf/1525766 I found
> that there was a modification to pkgutil during the need-for-speed sprint
> that affects the PEP 302 protocol in a backwards incompatible way.
>
> Specifically, PEP 302 documents that path_importer_cache always contains
> either importer objects or None.  Any code written to obtain importer
> objects is therefore now broken, because import.c is slapping False in for
> non-existent filesystem paths.
>
> The pkgutil module was then hacked to work around this problem, thereby
> hiding the breakage from at least the standard library, but not any
> external libraries that follow the PEP 302 protocol to find importers.
>
> There are several options as to how to proceed:
>
> 1. Revert the change
> 2. Document the breakage, update PEP 302, and make everybody update their
> code
> 3. Make it not break existing code, by using a NonexistentPathImporter or
> NullImporter type in place of "False" in sys.path_importer_cache.
>
> Any thoughts?


Revert it.  Is it really that much of a bonus to use False over None?  Both
evaluate to false and both are already singleton so you can use 'is' for
testing.

-Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20060720/3e21b965/attachment.htm 

From g.brandl at gmx.net  Thu Jul 20 21:32:58 2006
From: g.brandl at gmx.net (Georg Brandl)
Date: Thu, 20 Jul 2006 21:32:58 +0200
Subject: [Python-Dev] Support for PyGetSetDefs in pydoc
In-Reply-To: <AB1C2209-329E-4F3D-8B7A-2548A40F892A@python.org>
References: <A027CA35-EE77-4658-BA59-E849368C440B@python.org>	<EF6E6AF2-654F-4F53-91A8-A6C9B8A80745@python.org>	<44B64782.9010706@gmail.com>	<7FE62713-24B0-47CE-BA94-41150DB683F4@python.org>
	<AB1C2209-329E-4F3D-8B7A-2548A40F892A@python.org>
Message-ID: <e9olpa$4ql$1@sea.gmane.org>

Barry Warsaw wrote:

> Why did I do this instead of trying to hunt down some existing getset  
> or member descriptor?  For one thing, there really aren't very good  
> candidates for such objects in the built-in modules.  You can't use  
> objects like datetime.timedelta.days in types.py because datetime is  
> not importable early enough for types.py to use it. Even if there  
> were likely candidates, they would be accidents of implementation and  
> it doesn't seem like a good idea to force force some future datetime  
> maintainer to have to fix types.py when she decides that  
> datetime.timedelta.days should be implemented in some other way.
> 
> A 3rd party extension module doesn't work either because you really  
> need the tie-in from types.py, inspect.py, and pydoc.py.  You  
> certainly don't want to go poking new values into types.py from the  
> outside, and besides inspect.py and pydoc.py also need to learn about  
> these fundamental built-in Python types.
> 
> ISTM the most straightforward approach is to provide these types in a  
> built-in C module, linked into the interpreter so types.py can get  
> access to it as early as it needs to.  Also, because the type that  
> these descriptors are attached to cannot be instantiated from Python,  
> they should be quite benign, existing only in order to give type()  
> something to query.


Perhaps you could put the objects into _testcapi. That way no new module
has to be deployed (is _testcapi installed on every system?)

Georg


From pje at telecommunity.com  Thu Jul 20 21:37:25 2006
From: pje at telecommunity.com (Phillip J. Eby)
Date: Thu, 20 Jul 2006 15:37:25 -0400
Subject: [Python-Dev] Undocumented PEP 302 protocol change by
 need-for-speed sprint
In-Reply-To: <bbaeab100607201228y797b601alac230a13c00c9b59@mail.gmail.co
 m>
References: <5.1.1.6.0.20060720144252.02622318@sparrow.telecommunity.com>
	<5.1.1.6.0.20060720144252.02622318@sparrow.telecommunity.com>
Message-ID: <5.1.1.6.0.20060720153329.0202d570@sparrow.telecommunity.com>

At 12:28 PM 7/20/2006 -0700, Brett Cannon wrote:
>On 7/20/06, Phillip J. Eby 
><<mailto:pje at telecommunity.com>pje at telecommunity.com> wrote:
>>While investigating the need to apply 
>><http://python.org/sf/1525766>http://python.org/sf/1525766 I found
>>that there was a modification to pkgutil during the need-for-speed sprint
>>that affects the PEP 302 protocol in a backwards incompatible way.
>>
>>Specifically, PEP 302 documents that path_importer_cache always contains
>>either importer objects or None.  Any code written to obtain importer
>>objects is therefore now broken, because import.c is slapping False in for
>>non-existent filesystem paths.
>>
>>The pkgutil module was then hacked to work around this problem, thereby
>>hiding the breakage from at least the standard library, but not any
>>external libraries that follow the PEP 302 protocol to find importers.
>>
>>There are several options as to how to proceed:
>>
>>1. Revert the change
>>2. Document the breakage, update PEP 302, and make everybody update their 
>>code
>>3. Make it not break existing code, by using a NonexistentPathImporter or
>>NullImporter type in place of "False" in sys.path_importer_cache.
>>
>>Any thoughts?
>
>Revert it.  Is it really that much of a bonus to use False over 
>None?  Both evaluate to false and both are already singleton so you can 
>use 'is' for testing.

The changed code still uses None.  PEP 302 defines None as meaning that a 
sys.path entry does not have an importer.  It's just that the 
need-for-speed patch *adds* the use of True and False.  None still means 
"no importer", but True now means "no importer, path exists" and False now 
means "no importer, path does not exist".

The idea is that import.c can then skip checking the existence of the path 
when it sees True or False, but it then means that code that gets data from 
path_importer_cache needs to know about these new special values, or else 
it will get an attribute error when it tries to call True.find_module().


From brett at python.org  Thu Jul 20 21:58:42 2006
From: brett at python.org (Brett Cannon)
Date: Thu, 20 Jul 2006 12:58:42 -0700
Subject: [Python-Dev] Undocumented PEP 302 protocol change by
	need-for-speed sprint
In-Reply-To: <5.1.1.6.0.20060720153329.0202d570@sparrow.telecommunity.com>
References: <5.1.1.6.0.20060720144252.02622318@sparrow.telecommunity.com>
	<5.1.1.6.0.20060720153329.0202d570@sparrow.telecommunity.com>
Message-ID: <bbaeab100607201258sba22a52u4d713c07f92c4775@mail.gmail.com>

On 7/20/06, Phillip J. Eby <pje at telecommunity.com> wrote:
>
> At 12:28 PM 7/20/2006 -0700, Brett Cannon wrote:
> >On 7/20/06, Phillip J. Eby
> ><<mailto:pje at telecommunity.com>pje at telecommunity.com> wrote:
> >>While investigating the need to apply
> >><http://python.org/sf/1525766>http://python.org/sf/1525766 I found
> >>that there was a modification to pkgutil during the need-for-speed
> sprint
> >>that affects the PEP 302 protocol in a backwards incompatible way.
> >>
> >>Specifically, PEP 302 documents that path_importer_cache always contains
> >>either importer objects or None.  Any code written to obtain importer
> >>objects is therefore now broken, because import.c is slapping False in
> for
> >>non-existent filesystem paths.
> >>
> >>The pkgutil module was then hacked to work around this problem, thereby
> >>hiding the breakage from at least the standard library, but not any
> >>external libraries that follow the PEP 302 protocol to find importers.
> >>
> >>There are several options as to how to proceed:
> >>
> >>1. Revert the change
> >>2. Document the breakage, update PEP 302, and make everybody update
> their
> >>code
> >>3. Make it not break existing code, by using a NonexistentPathImporter
> or
> >>NullImporter type in place of "False" in sys.path_importer_cache.
> >>
> >>Any thoughts?
> >
> >Revert it.  Is it really that much of a bonus to use False over
> >None?  Both evaluate to false and both are already singleton so you can
> >use 'is' for testing.
>
> The changed code still uses None.  PEP 302 defines None as meaning that a
> sys.path entry does not have an importer.  It's just that the
> need-for-speed patch *adds* the use of True and False.  None still means
> "no importer", but True now means "no importer, path exists" and False now
> means "no importer, path does not exist".


Ah.  Sounds like None is not really even needed  with the change (although I
am not suggesting the removal of None).

The idea is that import.c can then skip checking the existence of the path
> when it sees True or False, but it then means that code that gets data
> from
> path_importer_cache needs to know about these new special values, or else
> it will get an attribute error when it tries to call True.find_module().
>
>
Well, I have not played with the PEP 302 stuff so I don't know how helpful
they are to have around.  But it is definitely a semantic change that either
needs to be reverted or documented.

-Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20060720/61ed8b42/attachment.html 

From glyph at divmod.com  Thu Jul 20 22:17:46 2006
From: glyph at divmod.com (glyph at divmod.com)
Date: Thu, 20 Jul 2006 16:17:46 -0400
Subject: [Python-Dev] Undocumented PEP 302 protocol change by
 need-for-speed sprint
In-Reply-To: <5.1.1.6.0.20060720144252.02622318@sparrow.telecommunity.com>
Message-ID: <20060720201746.29014.684853232.divmod.quotient.46224@ohm>

On Thu, 20 Jul 2006 14:57:07 -0400, "Phillip J. Eby" <pje at telecommunity.com> wrote:
>While investigating the need to apply http://python.org/sf/1525766 I found
>that there was a modification to pkgutil during the need-for-speed sprint
>that affects the PEP 302 protocol in a backwards incompatible way.

It just so happens that the bug that is reported was probably reported because
I'm working on some controversial new functionality in Twisted - controversial
because it replicates the functionality that bug is about in pkgutil.  This
functionality does make some use of PEP 302 functionality :).

    See <http://twistedmatrix.com/trac/ticket/1940>

>Specifically, PEP 302 documents that path_importer_cache always contains
>either importer objects or None.  Any code written to obtain importer
>objects is therefore now broken, because import.c is slapping False in for
>non-existent filesystem paths.

Oddly, for once I'm going to say I don't care about this change.  The code
I've written so far doesn't depend on this, and I was pretty careful to be
conservative about depending too much on the stuff described in PEP 302.  It
documents several features which don't exist (get_data, and methods in the 
"imp" module which don't exist in python2.3 or python2.4, where it was 
nominally accepted).

>There are several options as to how to proceed:

>2. Document the breakage, update PEP 302, and make everybody update their code

Personally I'd prefer it if PEP 302 were updated for a variety of reasons.
It's very hard to use as a reference for writing actual code because so many
features are "optional" or "open issues", and there's no description in the 
PEP of what their status is.

Better yet, this breakage (and other things) should be documented in the
Python reference, and the PEP should link to the documentation for different
versions, which can each describe the PEP's implementation status.  The
"importing modules" section of the library reference seems like a natural
place to put it.


From barry at python.org  Thu Jul 20 22:45:47 2006
From: barry at python.org (Barry Warsaw)
Date: Thu, 20 Jul 2006 16:45:47 -0400
Subject: [Python-Dev] Support for PyGetSetDefs in pydoc
In-Reply-To: <e9olpa$4ql$1@sea.gmane.org>
References: <A027CA35-EE77-4658-BA59-E849368C440B@python.org>	<EF6E6AF2-654F-4F53-91A8-A6C9B8A80745@python.org>	<44B64782.9010706@gmail.com>	<7FE62713-24B0-47CE-BA94-41150DB683F4@python.org>
	<AB1C2209-329E-4F3D-8B7A-2548A40F892A@python.org>
	<e9olpa$4ql$1@sea.gmane.org>
Message-ID: <1A2C07C0-58C4-45F0-90C1-FD0C26B7D9C5@python.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Jul 20, 2006, at 3:32 PM, Georg Brandl wrote:

> Perhaps you could put the objects into _testcapi. That way no new  
> module
> has to be deployed (is _testcapi installed on every system?)

That doesn't seem importable in types.py either.  You /could/ add  
that to Modules/config.c.in but I'm not sure it''s a good idea.

- -Barry

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.3 (Darwin)

iQCVAwUBRL/rgHEjvBPtnXfVAQLnHwP/Tzy1Iy8nbByuVaYaGrhel9zsdlbRXXlj
8wvLLlmzctmnzBrPBpZKV7aJMdiBd6261SKqPCrDZCj2LEplZWRa7za0zm3O2iZu
F1kbWCKTwWeI2/kVNpP1lTkm53u9yYp8grjxe1Q6fniB5nqQH11j79JehAYpBwdj
/RsVYoeQxng=
=lIEO
-----END PGP SIGNATURE-----

From tdelaney at avaya.com  Fri Jul 21 01:22:25 2006
From: tdelaney at avaya.com (Delaney, Timothy (Tim))
Date: Fri, 21 Jul 2006 09:22:25 +1000
Subject: [Python-Dev] FW: Bug? Certainly a new *behavior* from subprocess in
	2.5 on Win32
Message-ID: <2773CAC687FD5F4689F526998C7E4E5FF1E843@au3010avexu1.global.avaya.com>

Larry Hastings wrote:

> I run the following script:
> --
> from subprocess import *
> Popen("ls -l")
> --
> (yeah, I have ls.exe on Windows)
> 
> Under Python 2.4.2, this simply dumped the results of ls.exe to the
> terminal--sorry, to the "command shell".
> 
> Under Python 2.5, both beta 1 and beta 2, it dumps the results to the
> command shell, but *also* prints this:
> 
> Exception exceptions.AttributeError: "'NoneType' object has no
> attribute 'append'" in <bound method Popen.__del__ of
> <subprocess.Popen object at 0x00C04EB0>> ignored
> 
> Calling Popen() with a stdout = subprocess.PIPE does not throw this
> exception.

I've asked Larry to raise this on SourceForge, but with the SF email
problems I thought I'd better forward it here.

Looks like there's a bug in Popen.__del__ in 2.5. I'm not in a position
to have a look right now.

Tim Delaney

From bioinformed at gmail.com  Fri Jul 21 01:37:49 2006
From: bioinformed at gmail.com (Kevin Jacobs <jacobs@bioinformed.com>)
Date: Thu, 20 Jul 2006 19:37:49 -0400
Subject: [Python-Dev] FW: Bug? Certainly a new *behavior* from
	subprocess in 2.5 on Win32
In-Reply-To: <2773CAC687FD5F4689F526998C7E4E5FF1E843@au3010avexu1.global.avaya.com>
References: <2773CAC687FD5F4689F526998C7E4E5FF1E843@au3010avexu1.global.avaya.com>
Message-ID: <2e1434c10607201637l238635c7kc63cff228c5def82@mail.gmail.com>

Reported to the list about a week ago, with analysis.  Didn't get a
response.  Won't use sourceforge.  Sorry about the top post.

-Kevin

On 7/20/06, Delaney, Timothy (Tim) <tdelaney at avaya.com> wrote:
>
> Larry Hastings wrote:
>
> > I run the following script:
> > --
> > from subprocess import *
> > Popen("ls -l")
> > --
> > (yeah, I have ls.exe on Windows)
> >
> > Under Python 2.4.2, this simply dumped the results of ls.exe to the
> > terminal--sorry, to the "command shell".
> >
> > Under Python 2.5, both beta 1 and beta 2, it dumps the results to the
> > command shell, but *also* prints this:
> >
> > Exception exceptions.AttributeError: "'NoneType' object has no
> > attribute 'append'" in <bound method Popen.__del__ of
> > <subprocess.Popen object at 0x00C04EB0>> ignored
> >
> > Calling Popen() with a stdout = subprocess.PIPE does not throw this
> > exception.
>
> I've asked Larry to raise this on SourceForge, but with the SF email
> problems I thought I'd better forward it here.
>
> Looks like there's a bug in Popen.__del__ in 2.5. I'm not in a position
> to have a look right now.
>
> Tim Delaney
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/jacobs%40bioinformed.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20060720/a4ae87b3/attachment.htm 

From tdelaney at avaya.com  Fri Jul 21 02:44:39 2006
From: tdelaney at avaya.com (Delaney, Timothy (Tim))
Date: Fri, 21 Jul 2006 10:44:39 +1000
Subject: [Python-Dev] Behavior change in subprocess.py
Message-ID: <2773CAC687FD5F4689F526998C7E4E5FF1E847@au3010avexu1.global.avaya.com>

Hah - just found it. I even remember reading it...
 
I'll update the SF tracker (1526203) with your analysis.
 
Tim Delaney

	-----Original Message-----
	From: python-dev-bounces+tdelaney=avaya.com at python.org
[mailto:python-dev-bounces+tdelaney=avaya.com at python.org] On Behalf Of
Kevin Jacobs <jacobs at bioinformed.com>
	Sent: Thursday, 13 July 2006 12:33 AM
	To: python-dev at python.org
	Subject: [Python-Dev] Behavior change in subprocess.py
	
	
	During my testing of Python 2.5b2, I've found something that may
be worthy of discussion.  I suspect that recent GC and finalization
changes have altered the behavior of the Popen object in subprocess.py.
I am now getting many many many finalization warnings in my code like: 
	
	Exception exceptions.AttributeError: "'NoneType' object has no
attribute 'append'" in <bound method Popen.__del__ of <subprocess.Popen
object at 0x2aaaab910950>> ignored
	
	Is this a bug or a feature?  Personally, I'd like to see these
messages silenced, since it is being generated during interpreter
shutdown.  The following patch does the trick for me: 
	
	--- /usr/local/lib/python2.5/subprocess.py      2006-07-11
14:11:59.000000000 -0400
	+++ subprocess.py       2006-07-12 10:17:09.000000000 -0400
	@@ -613,7 +613,7 @@
	             return
	         # In case the child hasn't been waited on, check if
it's done. 
	         self.poll(_deadstate=sys.maxint)
	-        if self.returncode is None:
	+        if self.returncode is None and _active is not None:
	             # Child is still running, keep us alive until we
can wait on it. 
	             _active.append(self)
	
	
	Note that popen.py does something similar, though I am not
convinced that the test is right or if it is doing something more
subtle:
	
	    def __del__(self):
	        # In case the child hasn't been waited on, check if it's
done. 
	        self.poll(_deadstate=sys.maxint)
	        if self.sts < 0:
	            if _active:
	                # Child is still running, keep us alive until we
can wait on it.
	                _active.append(self) 
	
	Regards,
	-Kevin
	
	

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20060721/c21df170/attachment.html 

From greg.ewing at canterbury.ac.nz  Fri Jul 21 03:17:43 2006
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Fri, 21 Jul 2006 13:17:43 +1200
Subject: [Python-Dev] Strategy for converting the decimal module to C
In-Reply-To: <E1G3VNd-0003SW-6E@draco.cus.cam.ac.uk>
References: <E1G3VNd-0003SW-6E@draco.cus.cam.ac.uk>
Message-ID: <44C02B37.5080106@canterbury.ac.nz>

Nick Maclaren wrote:

> Now, interrupting into that level has to be transparent, in order to
> support TLB misses, clock interrupts, device interrupts, machine-check
> interrupts and so on.

I thought we were just talking about counting the number
of floating point exceptions that a particular piece of
code generates. Surely that's deterministic, and isn't
affected by any of that stuff?

--
Greg

From brett at python.org  Fri Jul 21 05:52:34 2006
From: brett at python.org (Brett Cannon)
Date: Thu, 20 Jul 2006 20:52:34 -0700
Subject: [Python-Dev] first draft of bug guidelines for www.python.org/dev/
Message-ID: <bbaeab100607202052t68ac220cs70b32a7c6a3190a2@mail.gmail.com>

Here is a first stab at writing up guidelines for people to follow
when reporting bug.  If this goes well I will also do ones for
patches, committing, and PEPs.

-Brett

---------------------------------------------------

These sets of guidelines are to help you file a bug report for the Python
 programming language on SourceForge_.  If your bug is not for the
language but for a third-party application, please report the bug to
that third-party.

*Please make sure to follow every step as it will make the lives
of the Python developers much easier!!!*

.. contents::

Get a SourceForge account
=======================================

In order to file a bug report, you must have an account_ on SourceForge_.  We
realize some people would like to have anonymous bug reporting for various
reasons (anonymity, ease or reporting, etc.).  But SourceForge does not support
anonymous reporting.  Plus, by registering, you are notified by email when any
action is been taken on your report.  This can be very important if a Python
developer needs more information from you about the bug.


Start a new bug
=======================================

You must be logged into SourceForge to file a bug!  See `Get a SourceForge
account`_ if you do not have one.

Go to the `SourceForge bug page`_ to start a new bug report.  There you will
find a link called `Submit New`_.  Click on it and it will allow you
to fill out a new
bug report.

Once you click on the link, you are presented with a page that has several
fields for you to fill in.  Here is what to do for each field:

* Category
    Set this to the area that the bug is related to (e.g.,
documentation, build, etc.).
* Group
    Usually this is set the major.minor version of Python that you
found the bug  in.
* Assigned To
    Leave this alone
* Priority
    Leave this alone
* Summary
    A one-line describing the problem so as to make it easy for
developers to spot whether they have the expertise needed to work on
the bug.
* Detailed Description
   Following sections of this document discuss what should go in here.
* Check to Upload and Attach a File
   If you are going to upload a file, you *must* check this box.
* <File Location Field>
   Click the Browse button to upload any file to accompany your bug
report (usually a succinct way to reproduce the bug).
* File Description
   A one-line describing the file; no date info is needed since the
upload is timestamped.


Specify Python version
=======================================

It is important that we have the most accurate version number of the
interpreter you are using in order to best diagnose the issue.  There are two
ways to get us the version information.

If you can run your Python interpreter, execute the following lines at an
interpreter and paste the result into the ``Detailed Description``
field of the bug report::

  >>> import sys
  >>> print sys.version

If you are running a version of Python newer than 2.4 and are working from a
source checkout of Python, the please also report the Subversion revision
number for the root of your checkout::

  python/trunk$ svnversion .

If your bug is preventing you from running the interpreter, execute Python with
teh ``-V`` command-line flag and paste the output::

  python/trunk$ python -V


Special settings for your Python interpreter
============================================

Sometimes your environment influences a bug and thus needs to be
reported to help find the problem.  This means we need to have
reported:

* Operating System
* Environment Variables
    + PYTHONSTARTUP
        If this is set and might be causing the issue, please either
upload the file or state what it does.
    + PYTHONCASEOK
        If your bug is on Windows and involves importing, please
report if this environment variable is set or not.
* site-packages
    If you have andy third-party packages installed that may be
contributing to the bug, please report those.
* Custom Patches
    Any differences between your code and the code the Python
developers work off of needs to be reported.


Sample code to reproduce bug
========================================

If you can, please upload a file the demonstrates the bug.  The more
succinct the better!  And please do not forget to check the upload
checkbox in the bug report.


Submit!
========================================

At this point you should have a detailed bug report for developers to
work off of.  Click the ``Submit`` button and read on to see what you
should do after the bug is reported.


Respond to requests from developers
========================================

No matter how detailed the bug report, there is always the off-chance
that a developer will need more information to fix a bug.
Please be prompt in replying to requests for information by submitting
a reply on the bug report.

You may be asked to test out a patch.  It is very important that you
help with this, especially if the bug is not reproducible by the
developer working on it.


Write a patch!
========================================

If you are so inclined, patches for your own bug reports are always
helpful!  Please make sure to reference the tracker item # in the
patch details.



.. _SourceForge: http://www.sourceforge.net/
.. _SourceForge bug page:
http://sourceforge.net/tracker/?group_id=5470&atid=105470
.. _account: http://sourceforge.net/account/newuser_emailverify.php
.. _Submit New:
http://sourceforge.net/tracker/?func=add&group_id=5470&atid=105470

From nyamatongwe at gmail.com  Fri Jul 21 06:10:27 2006
From: nyamatongwe at gmail.com (Neil Hodgson)
Date: Fri, 21 Jul 2006 14:10:27 +1000
Subject: [Python-Dev] first draft of bug guidelines for
	www.python.org/dev/
In-Reply-To: <bbaeab100607202052t68ac220cs70b32a7c6a3190a2@mail.gmail.com>
References: <bbaeab100607202052t68ac220cs70b32a7c6a3190a2@mail.gmail.com>
Message-ID: <50862ebd0607202110m661a3fc8sfd99b4a78b4db1b8@mail.gmail.com>

Brett Cannon:

> But SourceForge does not support anonymous reporting.

   SourceForge does support anonymous reporting. A large proportion of
the fault reports I receive for Scintilla are anonymous as indicated
by "nobody" in the "Submitted By" column.
https://sourceforge.net/tracker/?group_id=2439&atid=102439

   Neil

From fdrake at acm.org  Fri Jul 21 06:25:17 2006
From: fdrake at acm.org (Fred L. Drake, Jr.)
Date: Fri, 21 Jul 2006 00:25:17 -0400
Subject: [Python-Dev] first draft of bug guidelines for
	www.python.org/dev/
In-Reply-To: <50862ebd0607202110m661a3fc8sfd99b4a78b4db1b8@mail.gmail.com>
References: <bbaeab100607202052t68ac220cs70b32a7c6a3190a2@mail.gmail.com>
	<50862ebd0607202110m661a3fc8sfd99b4a78b4db1b8@mail.gmail.com>
Message-ID: <200607210025.17429.fdrake@acm.org>

On Friday 21 July 2006 00:10, Neil Hodgson wrote:
 > Brett Cannon:
 > > But SourceForge does not support anonymous reporting.
 >
 >    SourceForge does support anonymous reporting. A large proportion of
 > the fault reports I receive for Scintilla are anonymous as indicated
 > by "nobody" in the "Submitted By" column.

SourceForge supports anonymous reporting, but the Python project determined 
that the management cost of anonymous reports was higher than the value they 
provided.

It might be time to reconsider that decision (though my position hasn't 
changed).


  -Fred

-- 
Fred L. Drake, Jr.   <fdrake at acm.org>

From tjreedy at udel.edu  Fri Jul 21 06:42:12 2006
From: tjreedy at udel.edu (Terry Reedy)
Date: Fri, 21 Jul 2006 00:42:12 -0400
Subject: [Python-Dev] Support for PyGetSetDefs in pydoc
References: <A027CA35-EE77-4658-BA59-E849368C440B@python.org><EF6E6AF2-654F-4F53-91A8-A6C9B8A80745@python.org><44B64782.9010706@gmail.com><7FE62713-24B0-47CE-BA94-41150DB683F4@python.org>
	<AB1C2209-329E-4F3D-8B7A-2548A40F892A@python.org>
Message-ID: <e9plv5$rfe$1@sea.gmane.org>


"Barry Warsaw" <barry at python.org> wrote in message 
news:AB1C2209-329E-4F3D-8B7A-2548A40F892A at python.org...
> think this patch should go in 2.5.  OTOH, I suspect most people just
> don't care, which is why I've gotten almost no comments on the patch
> (other than one or two mild nods of approval).

I use help(ob) quite a bit, have noticed that somethings don't give 
anything useful, and definitely approve of making help more useful.

tjr




From oliphant.travis at ieee.org  Fri Jul 21 09:42:51 2006
From: oliphant.travis at ieee.org (Travis E. Oliphant)
Date: Fri, 21 Jul 2006 01:42:51 -0600
Subject: [Python-Dev] segfault when using PyGILState_Ensure/Release in
	Python2.3.4
Message-ID: <e9q0hs$jo4$1@sea.gmane.org>


I'm hoping somebody here can help me with an error I'm getting in Python 
  2.3.4 but not in Python 2.4.2 when I use PyGILState_Ensure in NumPy on 
Linux.

Perhaps somebody can point out what I'm doing wrong because while I've 
tried to understand the threading API it can be a bit confusing and 
maybe I'm doing it wrong.

Right now, I've had to disable threading support in NumPy for Python 2.3 
which is a bit annoying.

The problem shows up when I've released the GIL using 
PyEval_SaveThread() in one section of code.  Then the code calls 
functions that don't involve the Python C-API.

Then another function sometimes requires use of the C-API to set a 
Python Error or issue a warning.   So I call:

_save = PyGILState_Ensure();

<Use Python C-API to issue an error or warning>

Finally, before exiting this function

PyGILState_Release(_save);

is called.   Later when control returns to the original caller that 
released the GIL, PyEval_RestoreThread() is called.  But the segfault 
seems to be happening on the call to PyGILState_Release(_save);

All of this works fine when it runs under Python 2.4.2, but under Python 
2.3.4 I get a segfault.

Does anybody have any ideas?   Thanks very much.


From ncoghlan at gmail.com  Fri Jul 21 12:08:36 2006
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 21 Jul 2006 20:08:36 +1000
Subject: [Python-Dev] new security doc using object-capabilities
In-Reply-To: <bbaeab100607201009x28a7354elde374c6651b1a322@mail.gmail.com>
References: <bbaeab100607191535p543cb0ddj66410cf985dd9b77@mail.gmail.com>	
	<44BF6C59.6010707@gmail.com>
	<bbaeab100607201009x28a7354elde374c6651b1a322@mail.gmail.com>
Message-ID: <44C0A7A4.2030209@gmail.com>

Brett Cannon wrote:
>     Extensible file type handling
>     -----------------------------
>     If the file type handlers are stored in normal Python data
>     structures as
>     described above, it becomes feasible to make the import system
>     extensible to
>     different file types as well as to different file locations.
> 
> 
> Yep.  Although I am more interested in restricting than broadening the 
> file types.

Either way you'd be mutating the list of recognised file types :)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
             http://www.boredomandlaziness.org

From ncoghlan at gmail.com  Fri Jul 21 12:14:40 2006
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 21 Jul 2006 20:14:40 +1000
Subject: [Python-Dev] FW: Bug? Certainly a new *behavior* from
 subprocess in	2.5 on Win32
In-Reply-To: <2773CAC687FD5F4689F526998C7E4E5FF1E843@au3010avexu1.global.avaya.com>
References: <2773CAC687FD5F4689F526998C7E4E5FF1E843@au3010avexu1.global.avaya.com>
Message-ID: <44C0A910.4070002@gmail.com>

Delaney, Timothy (Tim) wrote:
> Looks like there's a bug in Popen.__del__ in 2.5. I'm not in a position
> to have a look right now.

For those not watching python-checkins, a check for "is not None" has been 
added before the offending line in Popen.__del__. (by Georg, IIRC)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
             http://www.boredomandlaziness.org

From nmm1 at cus.cam.ac.uk  Fri Jul 21 12:18:04 2006
From: nmm1 at cus.cam.ac.uk (Nick Maclaren)
Date: Fri, 21 Jul 2006 11:18:04 +0100
Subject: [Python-Dev] Strategy for converting the decimal module to C
Message-ID: <E1G3s56-0000pe-M5@draco.cus.cam.ac.uk>

Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
> 
> > Now, interrupting into that level has to be transparent, in order to
> > support TLB misses, clock interrupts, device interrupts, machine-check
> > interrupts and so on.
> 
> I thought we were just talking about counting the number
> of floating point exceptions that a particular piece of
> code generates. Surely that's deterministic, and isn't

Er, no.  Rather fundamentally, on two grounds.  Please bear with me, as
this IS relevant to Python.  See the summary at the end if you like :-)

The first is that such things are NOT deterministic, not even on simple
CPUs - take a look at the Alpha architecture for an example, and then
follow it up with the IA64 one if you have the stomach for it.  But
that wasn't my main point.

It is that modern CPUs have a SINGLE interrupt mechanism (a mistake in
itself, but they do), so a CPU may be interrupted when it is running
a device driver, other kernel thread or within a system call as much as
when running an application.  In fact, to some extent, interrupt handlers
can themselves be interrupted (let's skip the details).

Now, in order to allow the application to run its handler, the state
has to be saved, sanitised and converted back to application context;
and conversely on return.  That is hairy, and is why it is not possible
to handle interrupts generated within system calls on many systems.
But that is not directly Python's problem.

What is, is that the code gets interrupted at an unpredictable place,
and the registers and other state may not be consistent as the language
run-time system and Python are concerned.  It is critical (a) that a sane
state is restored before calling the handler and (b) that calling the
handler neither relies on nor disturbs any of the "in flight" actions
in the interrupted code.

To cut a long story short, it is impractical for a language run-time
system to call user-defined handlers with any degree of reliability
unless the compiled code and run-time interoperate carefully - I have
been there and done that many times, but few people still working have.
On architectures with out-of-order execution (and interrupts), you
have to assume that an interrupt may occur anywhere, even when the
code does not use the relevant facility.  Floating-point overflow
in the middle of a list insertion?  That's to be expected.

It becomes considerably easier if the (run-time system) interrupt
handler merely needs to flag or count interrupts, as it can use a
minimal handler which is defensive and non-intrusive.  Even that is
a pretty fair nightmare, as many systems temporarily corrupt critical
registers when they think that it is safe.  And few think of interrupts
when deciding that ....

So, in summary, please DON'T produce a design that relies on trapping
floating-point exceptions and passing control to a Python function.
This is several times harder than implementing fpectl.


Regards,
Nick Maclaren,
University of Cambridge Computing Service,
New Museums Site, Pembroke Street, Cambridge CB2 3QH, England.
Email:  nmm1 at cam.ac.uk
Tel.:  +44 1223 334761    Fax:  +44 1223 334679

From ndbecker2 at gmail.com  Fri Jul 21 12:56:33 2006
From: ndbecker2 at gmail.com (Neal Becker)
Date: Fri, 21 Jul 2006 06:56:33 -0400
Subject: [Python-Dev] Document performance requirements?
Message-ID: <e9qbtf$kk6$1@sea.gmane.org>

For a recent project I needed to select a container.  There are plenty of
python data structures to choose from.  It seems that information on
performance is missing (or not easy to find).

I think Python should include performance in the documentation of common
data structures to help users select the appropriate types.  Something in
the style of c++ STL.


From mwh at python.net  Fri Jul 21 13:25:12 2006
From: mwh at python.net (Michael Hudson)
Date: Fri, 21 Jul 2006 12:25:12 +0100
Subject: [Python-Dev] Ireland PyPy sprint 21th-27th August 2006
Message-ID: <2mirlryzc7.fsf@starship.python.net>

The next PyPy sprint will happen in the nice city of 
Limerick in Ireland from 21st till 27th August.  
(Most people intend to arrive 20th August). 

The main focus of the sprint will be on JIT compiler works, 
various optimization works, porting extension modules, 
infrastructure works like a build tool for PyPy, or 
extended (distributed) testing. 

It's also open to new topics.  If you are a student
consider to participate in `Summer of PyPy`_ in order 
get funding for your travels and accomodation. 

The sprint is being hosted by University of Limerick
(http://www.ul.ie/) - and is arranged in co-operation
with people from our sister project Calibre (www.calibre.ie).
Our contact at the University is P?r ?gerfalk and Eoin
Oconchuir. 

.. _`Summer of PyPy`: http://codespeak.net/pypy/dist/pypy/doc/summer-of-pypy

First day: introduction and workshop (possible to attend only this day)
------------------------------------------------------------------------

During the first day (21st of August) there will be talks on various subjects
related to PyPy:

* A tutorial and technical introduction to the PyPy codebase  
  (suited for people interested in getting an overview of PyPy?s architecture
  and/or contributing to PyPy)

* a workshop covering more in-depth technical aspects of PyPy and what PyPy
  can do for you. The workshop will also cover methodology, aiming at explaining
  the pros and cons of sprint-driven development.
  (suited for sprint attendants, students, staff and other 
  interested parties from/around the University and the local area)

The tutorial will be part of the sprint introduction - the workshop will take place if 
there is enough interest raised before the 21st of August from people planning to 
attend. You are of course welcome to attend just for this first day of the sprint.


If you want to come ...
----------------------------

If you'd like to come, please subscribe to the `pypy-sprint mailing list`_
and drop a note about your interests and post any questions.  More 
organisational information will be send to that list.  We'll keep
a list of `people`_ which we'll update (which you can do so yourself
if you have codespeak commit rights). 

.. _`Calibre`: http://www.calibre.ie 

A small disclaimer:
There might be people visiting the sprint in order to do research on 
how open source communities work, organize and communicate. This 
research might be done via filming, observing or interviewing. 
But of course you will be able to opt-out of being filmed at the sprint. 

Logistics 
--------------------------------------

NOTE: you need a UK style of power adapter (220V). 

The sprint will be held in the Computer Science Building, room CSG-025, 
University of Limerick (no 7 on http://www.ul.ie/main/places/campus.shtml). 

Bus 308 from Limerick city will take you to no 30 (approx.).
See http://www.ul.ie/main/places/travel.shtml for more on how to get to UL.

We will have access to the sprint facilities from 09:00-19:00 every day (it might
be even later than 19:00). Monday-Wednesday, Friday-Sunday are sprint days,
Thursday is likely a break day.

Food on campus varies in price and quality ;-) : from ca 4 EUR to 7-8 EUR for 
a lunch. There are of course a lot more food alternatives in down town Limerick.

Next Airports
------------------

Shannon Airport (SNN) is the nearest airport (Ryanair flies
there) - you may check out more information about flights
to/from the airport at

    http://www.shannonairport.com/index.html 

There are busses from there to downtown Limerick, and busses
from Limerick to the UL campus. Taxis are about 35 EUR.

Accomodation
-----------------

There is a website address for campus accomodation at 

    http://www.ul.ie/conference/accommodation.htm.  

The rate should be 49 euro for Bed and Breakfast.  If you are interested 
in booking campus accommodation, please contact deborah.tudge at ul ie
and make reference to the PyPy workshop and sprint.  Please try 
to book as soon as possible. 

As an off-campus accommodation alternative you can also try:

Castletroy Lodge and Castletroy Inn (Bed and Breakfast)
Dublin Road (15 to 20 mins walk to UL)
Tel: +353 61 338385 / +353 61 331167

.. _`pypy-sprint mailing list`: http://codespeak.net/mailman/listinfo/pypy-sprint
.. _`people`: people.html 

-- 
  surely, somewhere, somehow, in the history of computing, at least
  one manual has been written that you could at least remotely
  attempt to consider possibly glancing at.              -- Adam Rixey

From ncoghlan at gmail.com  Fri Jul 21 13:49:32 2006
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 21 Jul 2006 21:49:32 +1000
Subject: [Python-Dev] Document performance requirements?
In-Reply-To: <e9qbtf$kk6$1@sea.gmane.org>
References: <e9qbtf$kk6$1@sea.gmane.org>
Message-ID: <44C0BF4C.1010408@gmail.com>

Neal Becker wrote:
> For a recent project I needed to select a container.  There are plenty of
> python data structures to choose from.  It seems that information on
> performance is missing (or not easy to find).
> 
> I think Python should include performance in the documentation of common
> data structures to help users select the appropriate types.  Something in
> the style of c++ STL.

Do you mean absolute performance, or do you mean algorithmic order guarantees? 
I thought the latter were already documented. . .

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
             http://www.boredomandlaziness.org

From ncoghlan at gmail.com  Fri Jul 21 13:56:20 2006
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 21 Jul 2006 21:56:20 +1000
Subject: [Python-Dev] Support for PyGetSetDefs in pydoc
In-Reply-To: <e9olpa$4ql$1@sea.gmane.org>
References: <A027CA35-EE77-4658-BA59-E849368C440B@python.org>	<EF6E6AF2-654F-4F53-91A8-A6C9B8A80745@python.org>	<44B64782.9010706@gmail.com>	<7FE62713-24B0-47CE-BA94-41150DB683F4@python.org>	<AB1C2209-329E-4F3D-8B7A-2548A40F892A@python.org>
	<e9olpa$4ql$1@sea.gmane.org>
Message-ID: <44C0C0E4.6040403@gmail.com>

Georg Brandl wrote:
> Perhaps you could put the objects into _testcapi. That way no new module
> has to be deployed (is _testcapi installed on every system?)

Let's not corrupt _testcapi with something that has an entirely differently 
purpose.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
             http://www.boredomandlaziness.org

From ndbecker2 at gmail.com  Fri Jul 21 14:03:26 2006
From: ndbecker2 at gmail.com (Neal Becker)
Date: Fri, 21 Jul 2006 08:03:26 -0400
Subject: [Python-Dev] Document performance requirements?
References: <e9qbtf$kk6$1@sea.gmane.org> <44C0BF4C.1010408@gmail.com>
Message-ID: <e9qfqt$1t3$1@sea.gmane.org>

Nick Coghlan wrote:

> Neal Becker wrote:
>> For a recent project I needed to select a container.  There are plenty of
>> python data structures to choose from.  It seems that information on
>> performance is missing (or not easy to find).
>> 
>> I think Python should include performance in the documentation of common
>> data structures to help users select the appropriate types.  Something in
>> the style of c++ STL.
> 
> Do you mean absolute performance, or do you mean algorithmic order
> guarantees? I thought the latter were already documented. . .
> 

The latter.  Where is it documented?


From ncoghlan at gmail.com  Fri Jul 21 14:25:02 2006
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 21 Jul 2006 22:25:02 +1000
Subject: [Python-Dev] Document performance requirements?
In-Reply-To: <200607210759.56595.ndbecker2@gmail.com>
References: <e9qbtf$kk6$1@sea.gmane.org> <44C0BF4C.1010408@gmail.com>
	<200607210759.56595.ndbecker2@gmail.com>
Message-ID: <44C0C79E.3020309@gmail.com>

Neal Becker wrote:
> On Friday 21 July 2006 7:49 am, Nick Coghlan wrote:
>> Neal Becker wrote:
>>> For a recent project I needed to select a container.  There are plenty of
>>> python data structures to choose from.  It seems that information on
>>> performance is missing (or not easy to find).
>>>
>>> I think Python should include performance in the documentation of common
>>> data structures to help users select the appropriate types.  Something in
>>> the style of c++ STL.
>> Do you mean absolute performance, or do you mean algorithmic order
>> guarantees? I thought the latter were already documented. . .
>>
> 
> The latter.  Where are they documented?

Just because I think something, it doesn't mean it's true :)

The only reference I can actually find is the one in the collections module 
docs pointing out that collections.deque permits O(1) insertions and removals 
at the beginning of the sequence, as well as at the end (whereas lists are 
O(n) for operations at the beginning due to the resulting memory copying).

However, I'm also struggling to think of a case other than list vs deque where 
the choice of a builtin or standard library data structure would be dictated 
by big-O() concerns.

list vs array.array is based on memory efficiency
list vs deque is based on whether or not you need O(1) push/pop at both ends
list vs set is based on whether or not ordering matters
set vs dict is based on whether or not you need to map keys to values

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
             http://www.boredomandlaziness.org

From jason.orendorff at gmail.com  Fri Jul 21 16:14:20 2006
From: jason.orendorff at gmail.com (Jason Orendorff)
Date: Fri, 21 Jul 2006 10:14:20 -0400
Subject: [Python-Dev]  Document performance requirements?
In-Reply-To: <bb8868b90607210713p4599e8c3ka5fc8c61f8679486@mail.gmail.com>
References: <e9qbtf$kk6$1@sea.gmane.org> <44C0BF4C.1010408@gmail.com>
	<200607210759.56595.ndbecker2@gmail.com> <44C0C79E.3020309@gmail.com>
	<bb8868b90607210713p4599e8c3ka5fc8c61f8679486@mail.gmail.com>
Message-ID: <bb8868b90607210714g2b7dd8e1he8fa00ba842fe3ee@mail.gmail.com>

On 7/21/06, Nick Coghlan <ncoghlan at gmail.com> wrote:
> However, I'm also struggling to think of a case other than list vs deque where
> the choice of a builtin or standard library data structure would be dictated
> by big-O() concerns.

OK, but that doesn't mean the information is unimportant.  +1 on
making this something of a priority.  People looking for this info
should find it in the obvious place.  Some are unobvious. (How fast is
dict.__eq__ on average? Worst case?)

-j

From brett at python.org  Fri Jul 21 16:40:07 2006
From: brett at python.org (Brett Cannon)
Date: Fri, 21 Jul 2006 07:40:07 -0700
Subject: [Python-Dev] first draft of bug guidelines for
	www.python.org/dev/
In-Reply-To: <200607210025.17429.fdrake@acm.org>
References: <bbaeab100607202052t68ac220cs70b32a7c6a3190a2@mail.gmail.com>
	<50862ebd0607202110m661a3fc8sfd99b4a78b4db1b8@mail.gmail.com>
	<200607210025.17429.fdrake@acm.org>
Message-ID: <bbaeab100607210740pf9e2db5r9f427c8f3184a9f7@mail.gmail.com>

On 7/20/06, Fred L. Drake, Jr. <fdrake at acm.org> wrote:
>
> On Friday 21 July 2006 00:10, Neil Hodgson wrote:
> > Brett Cannon:
> > > But SourceForge does not support anonymous reporting.
> >
> >    SourceForge does support anonymous reporting. A large proportion of
> > the fault reports I receive for Scintilla are anonymous as indicated
> > by "nobody" in the "Submitted By" column.
>
> SourceForge supports anonymous reporting, but the Python project
> determined
> that the management cost of anonymous reports was higher than the value
> they
> provided.


OK.  Wording  has been changed in my copy.

It might be time to reconsider that decision (though my position hasn't
> changed).


Sure.  It can also wait until we begin discussing the transition to our next
bug tracker.

-Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20060721/1f9d3087/attachment.html 

From foom at fuhm.net  Fri Jul 21 17:05:51 2006
From: foom at fuhm.net (James Y Knight)
Date: Fri, 21 Jul 2006 11:05:51 -0400
Subject: [Python-Dev] Strategy for converting the decimal module to C
In-Reply-To: <E1G3s56-0000pe-M5@draco.cus.cam.ac.uk>
References: <E1G3s56-0000pe-M5@draco.cus.cam.ac.uk>
Message-ID: <8611AEB2-5D94-4C28-96D6-8626BA5B6DFC@fuhm.net>


On Jul 21, 2006, at 6:18 AM, Nick Maclaren wrote:
> To cut a long story short, it is impractical for a language run-time
> system to call user-defined handlers with any degree of reliability
> unless the compiled code and run-time interoperate carefully - I have
> been there and done that many times, but few people still working  
> have.
> On architectures with out-of-order execution (and interrupts), you
> have to assume that an interrupt may occur anywhere, even when the
> code does not use the relevant facility.  Floating-point overflow
> in the middle of a list insertion?  That's to be expected.

While this _is_ a real problem, is it _not_ a general problem as you  
are describing it. Processors are perfectly capable of generating  
precise interrupts, and the inability to do so has nothing to do with  
the out-of-order execution, etc. Almost all interrupts are precise.  
The only interesting one which is not, on x86 processors, is the x87  
floating point exception, which is basically for historical reasons.  
It has never been precise, ever since the actual 8087 coprocessor  
chip for the 8086. However, all is not lost: the exception cannot  
occur randomly. It can only occur on *some* floating point  
instruction, even if the instruction is not the one the error  
actually occurred in. So, unless your list insertion code uses  
floating point instructions, you should not get a floating point  
exception during your list insertion.

Also, looking forward, the "simd" floating point instructions (ie mmx/ 
sse/sse2/sse3) _do_ generate precise interrupts. And on x86-64, x87  
instructions are deprecated and everyone is recommended to use the  
simd ones, instead (so, for example, gcc defaults to using them).

James

From bioinformed at gmail.com  Fri Jul 21 18:43:44 2006
From: bioinformed at gmail.com (Kevin Jacobs <jacobs@bioinformed.com>)
Date: Fri, 21 Jul 2006 12:43:44 -0400
Subject: [Python-Dev] FW: Bug? Certainly a new *behavior* from
	subprocess in 2.5 on Win32
In-Reply-To: <44C0A910.4070002@gmail.com>
References: <2773CAC687FD5F4689F526998C7E4E5FF1E843@au3010avexu1.global.avaya.com>
	<44C0A910.4070002@gmail.com>
Message-ID: <2e1434c10607210943p10171eb9n607c5a5821768c7c@mail.gmail.com>

On 7/21/06, Nick Coghlan <ncoghlan at gmail.com> wrote:
>
> Delaney, Timothy (Tim) wrote:
> > Looks like there's a bug in Popen.__del__ in 2.5. I'm not in a position
> > to have a look right now.
>
> For those not watching python-checkins, a check for "is not None" has been
> added before the offending line in Popen.__del__. (by Georg, IIRC)
>

Many thanks for accepting my patch.  There remains a potentially related
problem in popen2.py, but it may be a lower priority, since most folks
should be using subprocess.

    def __del__(self):
        # In case the child hasn't been waited on, check if it's done.
        self.poll(_deadstate=sys.maxint)
        if self.sts < 0:
            if _active:
                # Child is still running, keep us alive until we can wait on
it.
                _active.append(self)


The is _active check, unless it intendeds to check for either empty or None,
should probably be revised to:

    def __del__(self):
        # In case the child hasn't been waited on, check if it's done.
        self.poll(_deadstate=sys.maxint)
        if self.sts < 0:
            if _active is None:
                # Child is still running, keep us alive until we can wait on
it.
                _active.append(self)

However, there may be a clever reason for doing what is doing that I do not
see.

Thanks again,
-Kevin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20060721/f57de211/attachment.html 

From rasky at develer.com  Fri Jul 21 18:45:12 2006
From: rasky at develer.com (Giovanni Bajo)
Date: Fri, 21 Jul 2006 18:45:12 +0200
Subject: [Python-Dev] Document performance requirements?
References: <e9qbtf$kk6$1@sea.gmane.org>
	<44C0BF4C.1010408@gmail.com><200607210759.56595.ndbecker2@gmail.com>
	<44C0C79E.3020309@gmail.com><bb8868b90607210713p4599e8c3ka5fc8c61f8679486@mail.gmail.com>
	<bb8868b90607210714g2b7dd8e1he8fa00ba842fe3ee@mail.gmail.com>
Message-ID: <054801c6ace5$09478c50$d503030a@trilan>

Jason Orendorff wrote:

>> However, I'm also struggling to think of a case other than list vs
>> deque where the choice of a builtin or standard library data
>> structure would be dictated by big-O() concerns.
>
> OK, but that doesn't mean the information is unimportant.  +1 on
> making this something of a priority.  People looking for this info
> should find it in the obvious place.  Some are unobvious. (How fast is
> dict.__eq__ on average? Worst case?)

I also found out that most people tend to think of Python's lists as a
magical data structure optimized for many operations (like a "rope" or
something complex like that). Documenting that it's just a bare vector
(std::vector in C++) would be of great help.
-- 
Giovanni Bajo


From grig.gheorghiu at gmail.com  Fri Jul 21 19:04:38 2006
From: grig.gheorghiu at gmail.com (Grig Gheorghiu)
Date: Fri, 21 Jul 2006 10:04:38 -0700
Subject: [Python-Dev] Community buildbots -- reprise
Message-ID: <3f09d5a00607211004kb45ffe2kcedcf418136e7102@mail.gmail.com>

Hi,

This message is in response to Glyph's plea
(<http://mail.python.org/pipermail/python-dev/2006-July/067366.html>
<http://mail.python.org/pipermail/python-dev/2006-July/067366.html%3E>).

Here's what Glyph said:

"I would like to propose, although I certainly don't have time to
implement, a program by which Python-using projects could contribute
buildslaves which would run their projects' tests with the latest
Python trunk.  This would provide two useful incentives: Python code
would gain a reputation as generally well-tested (since there is a
direct incentive to write tests for your project: get notified when
core python changes might break it), and the core developers would have
instant feedback when a "small" change breaks more code than it was
expected to."


I'm volunteering to organize this effort, is there is enough interest
on this list. In fact, I've done some prep work already:

 * got a domain name: pybots.org
 * got a $47/month Ubuntu-based VPS from JohnCompanies.com (root access
and everything); it's available at master.pybots.org, and it's ready to
be configured as a buildmaster for the pybots
 * got a mailing list: pybots at lists2.idyll.org
<http://us.f545.mail.yahoo.com/ym/Compose?To=pybots at lists2.idyll.org&YY=8414&order=down&sort=date&pos=0&view=a&head=b>

I can start configuring the Ubuntu machine as a buildmaster, and I can
also add a buildslave on the same machine that will check out the
latest Python trunk code, build it, then run the automated tests for a
sample project -- let's say for Twisted, since Glyph was the one
requesting this. This will also serve as a sample buildslave for other
people who will be interested in running buildslaves for their own
projects.

Apart from the goals stated by Glyph, I see this as a very valuable
effort in convincing people of the value of automated tests,
Python-related or not. A secondary effect I'd like to see would be for
these suites of tests to be invoked in a standard fashion -- maybe
'python setup.py test'.

If PSF can contribute some $$$ towards the hosting of the master
server, that would be appreciated, but not required. All that's
required is enough interest from the community.

Please let me know if you're interested.

Grig

 <http://agiletesting.blogspot.com/>

-- 
http://agiletesting.blogspot.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20060721/6ede65ff/attachment.htm 

From skip at pobox.com  Fri Jul 21 19:22:09 2006
From: skip at pobox.com (skip at pobox.com)
Date: Fri, 21 Jul 2006 12:22:09 -0500
Subject: [Python-Dev] first draft of bug guidelines for
	www.python.org/dev/
In-Reply-To: <bbaeab100607210740pf9e2db5r9f427c8f3184a9f7@mail.gmail.com>
References: <bbaeab100607202052t68ac220cs70b32a7c6a3190a2@mail.gmail.com>
	<50862ebd0607202110m661a3fc8sfd99b4a78b4db1b8@mail.gmail.com>
	<200607210025.17429.fdrake@acm.org>
	<bbaeab100607210740pf9e2db5r9f427c8f3184a9f7@mail.gmail.com>
Message-ID: <17601.3393.544378.871413@montanaro.dyndns.org>


    Brett> Sure.  It can also wait until we begin discussing the transition
    Brett> to our next bug tracker.

Would be kinda nice if the new bug tracker allowed submitters to enter a
followup email address without formally logging in.  (Of course, email-based
submissions would go a long way to minimizing the problem.)

Skip

From exarkun at divmod.com  Fri Jul 21 19:27:41 2006
From: exarkun at divmod.com (Jean-Paul Calderone)
Date: Fri, 21 Jul 2006 13:27:41 -0400
Subject: [Python-Dev] Community buildbots -- reprise
In-Reply-To: <3f09d5a00607211004kb45ffe2kcedcf418136e7102@mail.gmail.com>
Message-ID: <20060721172741.29014.1781472149.divmod.quotient.47688@ohm>

On Fri, 21 Jul 2006 10:04:38 -0700, Grig Gheorghiu <grig.gheorghiu at gmail.com> wrote:
>Hi,
>
>Apart from the goals stated by Glyph, I see this as a very valuable
>effort in convincing people of the value of automated tests,
>Python-related or not. A secondary effect I'd like to see would be for
>these suites of tests to be invoked in a standard fashion -- maybe
>'python setup.py test'.
>
>If PSF can contribute some $$$ towards the hosting of the master
>server, that would be appreciated, but not required. All that's
>required is enough interest from the community.
>
>Please let me know if you're interested.
>

This is certainly interesting to me.  If you need any help setting up
the Twisted buildslave, please let me know.

Jean-Paul

From g.brandl at gmx.net  Fri Jul 21 19:39:40 2006
From: g.brandl at gmx.net (Georg Brandl)
Date: Fri, 21 Jul 2006 19:39:40 +0200
Subject: [Python-Dev] FW: Bug? Certainly a new *behavior* from
 subprocess in 2.5 on Win32
In-Reply-To: <2e1434c10607210943p10171eb9n607c5a5821768c7c@mail.gmail.com>
References: <2773CAC687FD5F4689F526998C7E4E5FF1E843@au3010avexu1.global.avaya.com>	<44C0A910.4070002@gmail.com>
	<2e1434c10607210943p10171eb9n607c5a5821768c7c@mail.gmail.com>
Message-ID: <e9r3ee$4on$1@sea.gmane.org>

Kevin Jacobs <jacobs at bioinformed.com> wrote:
> On 7/21/06, *Nick Coghlan* <ncoghlan at gmail.com 
> <mailto:ncoghlan at gmail.com>> wrote:
> 
>     Delaney, Timothy (Tim) wrote:
>      > Looks like there's a bug in Popen.__del__ in 2.5. I'm not in a
>     position
>      > to have a look right now.
> 
>     For those not watching python-checkins, a check for "is not None"
>     has been
>     added before the offending line in Popen.__del__. (by Georg, IIRC)
> 
> 
> Many thanks for accepting my patch.  There remains a potentially related 
> problem in popen2.py, but it may be a lower priority, since most folks 
> should be using subprocess.
> 
>     def __del__(self):
>         # In case the child hasn't been waited on, check if it's done.
>         self.poll(_deadstate=sys.maxint)
>         if self.sts < 0:
>             if _active:
>                 # Child is still running, keep us alive until we can 
> wait on it.
>                 _active.append(self)
> 
> 
> The is _active check, unless it intendeds to check for either empty or 
> None, should probably be revised to:
> 
>     def __del__(self):
>         # In case the child hasn't been waited on, check if it's done.
>         self.poll(_deadstate=sys.maxint)
>         if self.sts < 0:
>             if _active is None:
>                 # Child is still running, keep us alive until we can 
> wait on it.
>                 _active.append(self)
> 
> However, there may be a clever reason for doing what is doing that I do 
> not see.

There's no reason (I know since I added the check myself ;). Thanks
for pointing out that obvious bug. (fixed in rev 50759).

Georg


From john at ewtllc.com  Fri Jul 21 19:44:02 2006
From: john at ewtllc.com (John Benediktsson)
Date: Fri, 21 Jul 2006 10:44:02 -0700
Subject: [Python-Dev] FW: Bug? Certainly a new *behavior* from
	subprocess in 2.5 on Win32
Message-ID: <B6FAC926EFE7B348B12F29CF7E4A93D401328DC8@hammer.office.bhtrader.com>


> The is _active check, unless it intendeds to check for either empty or

> None, should probably be revised to:
> 
>     def __del__(self):
>         # In case the child hasn't been waited on, check if it's done.
>         self.poll(_deadstate=sys.maxint)
>         if self.sts < 0:
>             if _active is None:
>                 # Child is still running, keep us alive until we can 
> wait on it.
>                 _active.append(self)
> 

It probably is obvious, but I think you mean:

    if _active is not None:
        ...

Thanks,
John.

From foom at fuhm.net  Fri Jul 21 19:48:28 2006
From: foom at fuhm.net (James Y Knight)
Date: Fri, 21 Jul 2006 13:48:28 -0400
Subject: [Python-Dev] Document performance requirements?
In-Reply-To: <054801c6ace5$09478c50$d503030a@trilan>
References: <e9qbtf$kk6$1@sea.gmane.org>
	<44C0BF4C.1010408@gmail.com><200607210759.56595.ndbecker2@gmail.com>
	<44C0C79E.3020309@gmail.com><bb8868b90607210713p4599e8c3ka5fc8c61f8679486@mail.gmail.com>
	<bb8868b90607210714g2b7dd8e1he8fa00ba842fe3ee@mail.gmail.com>
	<054801c6ace5$09478c50$d503030a@trilan>
Message-ID: <7458FE96-1A1D-40F3-B585-8E1DA55C4B14@fuhm.net>

On Jul 21, 2006, at 12:45 PM, Giovanni Bajo wrote:
> Jason Orendorff wrote:
>
>>> However, I'm also struggling to think of a case other than list vs
>>> deque where the choice of a builtin or standard library data
>>> structure would be dictated by big-O() concerns.
>>
>> OK, but that doesn't mean the information is unimportant.  +1 on
>> making this something of a priority.  People looking for this info
>> should find it in the obvious place.  Some are unobvious. (How  
>> fast is
>> dict.__eq__ on average? Worst case?)
>
> I also found out that most people tend to think of Python's lists as a
> magical data structure optimized for many operations (like a "rope" or
> something complex like that). Documenting that it's just a bare vector
> (std::vector in C++) would be of great help.

Indeed, I was talking to someone a while back who thought that lists  
were magically hashed, in that he did something like:
dictionary = open("/usr/share/dict/words").readlines()

and then expected:
"word" in dictionary

would be fast. And was very surprised when it turned out to be slow a  
linear search of the list. :)

James


From nnorwitz at gmail.com  Fri Jul 21 20:31:08 2006
From: nnorwitz at gmail.com (Neal Norwitz)
Date: Fri, 21 Jul 2006 11:31:08 -0700
Subject: [Python-Dev] Community buildbots -- reprise
In-Reply-To: <20060721172741.29014.1781472149.divmod.quotient.47688@ohm>
References: <3f09d5a00607211004kb45ffe2kcedcf418136e7102@mail.gmail.com>
	<20060721172741.29014.1781472149.divmod.quotient.47688@ohm>
Message-ID: <ee2a432c0607211131n67af50a5sa2518d5410a9c8e4@mail.gmail.com>

I have a server up and running.  I still need to polish some stuff
off.  I will mail more info when I get a chance.

n
--

On 7/21/06, Jean-Paul Calderone <exarkun at divmod.com> wrote:
> On Fri, 21 Jul 2006 10:04:38 -0700, Grig Gheorghiu <grig.gheorghiu at gmail.com> wrote:
> >Hi,
> >
> >Apart from the goals stated by Glyph, I see this as a very valuable
> >effort in convincing people of the value of automated tests,
> >Python-related or not. A secondary effect I'd like to see would be for
> >these suites of tests to be invoked in a standard fashion -- maybe
> >'python setup.py test'.
> >
> >If PSF can contribute some $$$ towards the hosting of the master
> >server, that would be appreciated, but not required. All that's
> >required is enough interest from the community.
> >
> >Please let me know if you're interested.
> >
>
> This is certainly interesting to me.  If you need any help setting up
> the Twisted buildslave, please let me know.
>
> Jean-Paul
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/nnorwitz%40gmail.com
>

From bioinformed at gmail.com  Fri Jul 21 20:53:56 2006
From: bioinformed at gmail.com (Kevin Jacobs <jacobs@bioinformed.com>)
Date: Fri, 21 Jul 2006 14:53:56 -0400
Subject: [Python-Dev] FW: Bug? Certainly a new *behavior* from
	subprocess in 2.5 on Win32
In-Reply-To: <B6FAC926EFE7B348B12F29CF7E4A93D401328DC8@hammer.office.bhtrader.com>
References: <B6FAC926EFE7B348B12F29CF7E4A93D401328DC8@hammer.office.bhtrader.com>
Message-ID: <2e1434c10607211153g392d122bj83d705ae82f5fa49@mail.gmail.com>

That'll teach me to fire off emails while running out the door.   Thanks.

-Kevin


On 7/21/06, John Benediktsson <john at ewtllc.com> wrote:
>
>
> > The is _active check, unless it intendeds to check for either empty or
>
> > None, should probably be revised to:
> >
> >     def __del__(self):
> >         # In case the child hasn't been waited on, check if it's done.
> >         self.poll(_deadstate=sys.maxint)
> >         if self.sts < 0:
> >             if _active is None:
> >                 # Child is still running, keep us alive until we can
> > wait on it.
> >                 _active.append(self)
> >
>
> It probably is obvious, but I think you mean:
>
>     if _active is not None:
>         ...
>
> Thanks,
> John.
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/jacobs%40bioinformed.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20060721/73f4b83d/attachment.html 

From bioinformed at gmail.com  Fri Jul 21 20:56:29 2006
From: bioinformed at gmail.com (Kevin Jacobs <jacobs@bioinformed.com>)
Date: Fri, 21 Jul 2006 14:56:29 -0400
Subject: [Python-Dev] first draft of bug guidelines for
	www.python.org/dev/
In-Reply-To: <17601.3393.544378.871413@montanaro.dyndns.org>
References: <bbaeab100607202052t68ac220cs70b32a7c6a3190a2@mail.gmail.com>
	<50862ebd0607202110m661a3fc8sfd99b4a78b4db1b8@mail.gmail.com>
	<200607210025.17429.fdrake@acm.org>
	<bbaeab100607210740pf9e2db5r9f427c8f3184a9f7@mail.gmail.com>
	<17601.3393.544378.871413@montanaro.dyndns.org>
Message-ID: <2e1434c10607211156s582441b1o753f25d2424e94b3@mail.gmail.com>

On 7/21/06, skip at pobox.com <skip at pobox.com> wrote:
>
>
>     Brett> Sure.  It can also wait until we begin discussing the
> transition
>     Brett> to our next bug tracker.
>
> Would be kinda nice if the new bug tracker allowed submitters to enter a
> followup email address without formally logging in.  (Of course,
> email-based
> submissions would go a long way to minimizing the problem.)
>


It may just be bad karma, but SourceForge tends to lock or go off into lala
land whenever I log in.  Thus, I would file many bug reports, with a
reply-to address, if non-login bug submissions where allowed.  My long term
hope is that you toss out SF and get something better.

Thanks,
-Kevin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20060721/bd31a869/attachment.htm 

From ncoghlan at gmail.com  Sat Jul 22 04:33:45 2006
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 22 Jul 2006 12:33:45 +1000
Subject: [Python-Dev] Document performance requirements?
In-Reply-To: <bb8868b90607210713p4599e8c3ka5fc8c61f8679486@mail.gmail.com>
References: <e9qbtf$kk6$1@sea.gmane.org> <44C0BF4C.1010408@gmail.com>	
	<200607210759.56595.ndbecker2@gmail.com>
	<44C0C79E.3020309@gmail.com>
	<bb8868b90607210713p4599e8c3ka5fc8c61f8679486@mail.gmail.com>
Message-ID: <44C18E89.5090603@gmail.com>

Jason Orendorff wrote:
> On 7/21/06, Nick Coghlan <ncoghlan at gmail.com> wrote:
>> However, I'm also struggling to think of a case other than list vs 
>> deque where
>> the choice of a builtin or standard library data structure would be 
>> dictated
>> by big-O() concerns.
> 
> OK, but that doesn't mean the information is unimportant.  +1 on
> making this something of a priority.  People looking for this info
> should find it in the obvious place.  Some are unobvious. (How fast is
> dict.__eq__ on average? Worst case?)

Agreed, but there's more to doing that than just writing down the O() implied 
by the current CPython implementation - it's up to Guido to decide which of 
the constraints are part of the language definition, and which are 
implementation accidents (e.g. CPython's list.sort() operation was stable for 
at least one release before GvR made stability part of the definition of the 
method at the language level).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
             http://www.boredomandlaziness.org

From ncoghlan at gmail.com  Sat Jul 22 04:37:37 2006
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 22 Jul 2006 12:37:37 +1000
Subject: [Python-Dev] first draft of bug guidelines
	for	www.python.org/dev/
In-Reply-To: <2e1434c10607211156s582441b1o753f25d2424e94b3@mail.gmail.com>
References: <bbaeab100607202052t68ac220cs70b32a7c6a3190a2@mail.gmail.com>	<50862ebd0607202110m661a3fc8sfd99b4a78b4db1b8@mail.gmail.com>	<200607210025.17429.fdrake@acm.org>	<bbaeab100607210740pf9e2db5r9f427c8f3184a9f7@mail.gmail.com>	<17601.3393.544378.871413@montanaro.dyndns.org>
	<2e1434c10607211156s582441b1o753f25d2424e94b3@mail.gmail.com>
Message-ID: <44C18F71.6020106@gmail.com>

Kevin Jacobs <jacobs at bioinformed.com> wrote:
> It may just be bad karma, but SourceForge tends to lock or go off into 
> lala land whenever I log in.  Thus, I would file many bug reports, with 
> a reply-to address, if non-login bug submissions where allowed.   My long
> term hope is that you toss out SF and get something better.

You're not the only one with that hope. With at least Trac, Jira and Roundup 
to choose from, the PSF's current tracker shootout should find us that 
replacement :)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
             http://www.boredomandlaziness.org

From arigo at tunes.org  Sat Jul 22 09:07:10 2006
From: arigo at tunes.org (Armin Rigo)
Date: Sat, 22 Jul 2006 09:07:10 +0200
Subject: [Python-Dev] new security doc using object-capabilities
In-Reply-To: <bbaeab100607191535p543cb0ddj66410cf985dd9b77@mail.gmail.com>
References: <bbaeab100607191535p543cb0ddj66410cf985dd9b77@mail.gmail.com>
Message-ID: <20060722070710.GA22334@code0.codespeak.net>

Hi Brett,

On Wed, Jul 19, 2006 at 03:35:45PM -0700, Brett Cannon wrote:
> I also plan to rewrite the import machinery in pure Python.

http://codespeak.net/svn/pypy/dist/pypy/module/__builtin__/importing.py


A bientot,

Armin

From martin at v.loewis.de  Sat Jul 22 10:34:08 2006
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sat, 22 Jul 2006 10:34:08 +0200
Subject: [Python-Dev] Document performance requirements?
In-Reply-To: <bb8868b90607210714g2b7dd8e1he8fa00ba842fe3ee@mail.gmail.com>
References: <e9qbtf$kk6$1@sea.gmane.org>
	<44C0BF4C.1010408@gmail.com>	<200607210759.56595.ndbecker2@gmail.com>
	<44C0C79E.3020309@gmail.com>	<bb8868b90607210713p4599e8c3ka5fc8c61f8679486@mail.gmail.com>
	<bb8868b90607210714g2b7dd8e1he8fa00ba842fe3ee@mail.gmail.com>
Message-ID: <44C1E300.8050803@v.loewis.de>

Jason Orendorff wrote:
> On 7/21/06, Nick Coghlan <ncoghlan at gmail.com> wrote:
>> However, I'm also struggling to think of a case other than list vs deque where
>> the choice of a builtin or standard library data structure would be dictated
>> by big-O() concerns.
> 
> OK, but that doesn't mean the information is unimportant.  +1 on
> making this something of a priority.  People looking for this info
> should find it in the obvious place.  Some are unobvious. (How fast is
> dict.__eq__ on average? Worst case?)

Contributions are welcome.

Regards,
Martin

From martin at v.loewis.de  Sat Jul 22 10:39:35 2006
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sat, 22 Jul 2006 10:39:35 +0200
Subject: [Python-Dev] Community buildbots -- reprise
In-Reply-To: <3f09d5a00607211004kb45ffe2kcedcf418136e7102@mail.gmail.com>
References: <3f09d5a00607211004kb45ffe2kcedcf418136e7102@mail.gmail.com>
Message-ID: <44C1E447.2020404@v.loewis.de>

Grig Gheorghiu wrote:
> Please let me know if you're interested.

As I said earlier: If you need some kind of post-commit
trigger on the python repository to trigger a build, just
let me know. We currently use a more-or-less plain
svn_buildbot.py to trigger our own builds.

Regards,
Martin

From arigo at tunes.org  Sat Jul 22 13:19:07 2006
From: arigo at tunes.org (Armin Rigo)
Date: Sat, 22 Jul 2006 13:19:07 +0200
Subject: [Python-Dev] Document performance requirements?
In-Reply-To: <44C18E89.5090603@gmail.com>
References: <e9qbtf$kk6$1@sea.gmane.org> <44C0BF4C.1010408@gmail.com>
	<200607210759.56595.ndbecker2@gmail.com>
	<44C0C79E.3020309@gmail.com>
	<bb8868b90607210713p4599e8c3ka5fc8c61f8679486@mail.gmail.com>
	<44C18E89.5090603@gmail.com>
Message-ID: <20060722111907.GA13782@code0.codespeak.net>

Hi,

On Sat, Jul 22, 2006 at 12:33:45PM +1000, Nick Coghlan wrote:
> Agreed, but there's more to doing that than just writing down the O() implied 
> by the current CPython implementation - it's up to Guido to decide which of 
> the constraints are part of the language definition, and which are 
> implementation accidents

I think that O-wise the current CPython situation should be documented
as a "minimal requirement" for implementations of the language, with
just one exception: the well-documented "don't rely on this" hack in 2.4
to make repeated 'str += str' amortized linear, for which the 2.3
quadratic behavior is considered compliant enough.

I suppose that allowing implementations to provide better algorithmic
complexities than required is fine, although I can think of some
problems with that (e.g. nice and efficient user code that would perform
horribly badly on CPython).


Armin

From arigo at tunes.org  Sat Jul 22 13:46:15 2006
From: arigo at tunes.org (Armin Rigo)
Date: Sat, 22 Jul 2006 13:46:15 +0200
Subject: [Python-Dev] new security doc using object-capabilities
In-Reply-To: <bbaeab100607191535p543cb0ddj66410cf985dd9b77@mail.gmail.com>
References: <bbaeab100607191535p543cb0ddj66410cf985dd9b77@mail.gmail.com>
Message-ID: <20060722114615.GB13782@code0.codespeak.net>

Re-hi,

On Wed, Jul 19, 2006 at 03:35:45PM -0700, Brett Cannon wrote:
> http://svn.python.org/view/python/branches/bcannon-sandboxing/securing_python.txt?rev=50717&view=log.

I'm not sure I understand what you propose to fix holes like
constructors and __subclasses__: it seems that you want to remove them
altogether (and e.g. make factory functions instead).  That would
completely break all programs, right?  I mean, there is no way such
changes would go into mainstream CPython.  Or do you propose to maintain
a CPython branch manually for the foreseeable future?  (From experience
this is a bad idea...)


A bientot,

Armin

From grig.gheorghiu at gmail.com  Sat Jul 22 17:32:07 2006
From: grig.gheorghiu at gmail.com (Grig Gheorghiu)
Date: Sat, 22 Jul 2006 08:32:07 -0700
Subject: [Python-Dev] Community buildbots -- reprise
In-Reply-To: <44C1E447.2020404@v.loewis.de>
References: <3f09d5a00607211004kb45ffe2kcedcf418136e7102@mail.gmail.com>
	<44C1E447.2020404@v.loewis.de>
Message-ID: <3f09d5a00607220832n57f0c5a6m8a193eb8e405846a@mail.gmail.com>

On 7/22/06, "Martin v. L?wis" <martin at v.loewis.de> wrote:
>
> Grig Gheorghiu wrote:
> > Please let me know if you're interested.
>
> As I said earlier: If you need some kind of post-commit
> trigger on the python repository to trigger a build, just
> let me know. We currently use a more-or-less plain
> svn_buildbot.py to trigger our own builds.



Wouldn't that put too much of a burden on the python core build system? It
would have to be aware of all the buildslaves running specific projects.

I was thinking about having a dedicated buildmaster machine, such as the one
Neal says he already has, and configure that machine to coordinate a small
army of buildslaves which will be contributed for people interested in this
effort.

Grig
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20060722/f68aec48/attachment.htm 

From martin at v.loewis.de  Sat Jul 22 17:53:23 2006
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sat, 22 Jul 2006 17:53:23 +0200
Subject: [Python-Dev] Community buildbots -- reprise
In-Reply-To: <3f09d5a00607220832n57f0c5a6m8a193eb8e405846a@mail.gmail.com>
References: <3f09d5a00607211004kb45ffe2kcedcf418136e7102@mail.gmail.com>	
	<44C1E447.2020404@v.loewis.de>
	<3f09d5a00607220832n57f0c5a6m8a193eb8e405846a@mail.gmail.com>
Message-ID: <44C249F3.8000908@v.loewis.de>

Grig Gheorghiu wrote:
>     As I said earlier: If you need some kind of post-commit
>     trigger on the python repository to trigger a build, just
>     let me know. We currently use a more-or-less plain
>     svn_buildbot.py to trigger our own builds.
> 
> Wouldn't that put too much of a burden on the python core build system?
> It would have to be aware of all the buildslaves running specific projects.

If there is a single "community buildbot", then no. In any case, it's
primarily administrative overhead, not so much cycles. python.org does
so many things simultaneously, making it trigger an additional build
remotely doesn't hurt.

> I was thinking about having a dedicated buildmaster machine, such as the
> one Neal says he already has, and configure that machine to coordinate a
> small army of buildslaves which will be contributed for people
> interested in this effort.

Right. You still need to find out when to rebuild, and getting triggers
from the source repositories is likely the easiest solution.

Regards,
Martin

From brett at python.org  Sat Jul 22 19:30:23 2006
From: brett at python.org (Brett Cannon)
Date: Sat, 22 Jul 2006 10:30:23 -0700
Subject: [Python-Dev] new security doc using object-capabilities
In-Reply-To: <20060722114615.GB13782@code0.codespeak.net>
References: <bbaeab100607191535p543cb0ddj66410cf985dd9b77@mail.gmail.com>
	<20060722114615.GB13782@code0.codespeak.net>
Message-ID: <bbaeab100607221030v5df16602ya72f58a43e82006b@mail.gmail.com>

On 7/22/06, Armin Rigo <arigo at tunes.org> wrote:
> Re-hi,
>
> On Wed, Jul 19, 2006 at 03:35:45PM -0700, Brett Cannon wrote:
> >
http://svn.python.org/view/python/branches/bcannon-sandboxing/securing_python.txt?rev=50717&view=log
.
>
> I'm not sure I understand what you propose to fix holes like
> constructors and __subclasses__: it seems that you want to remove them
> altogether (and e.g. make factory functions instead).  That would
> completely break all programs, right?

Not altogether, just constructors on select types who are considered
dangerous from a security standpoint.  The breakage won't be horrible, but
it will be there for advanced Python code.

I will try to make the wording more clear when I get back to work on
Tuesday.

>  I mean, there is no way such
> changes would go into mainstream CPython.

If this has to wait until Py3k then so be it.

>  Or do you propose to maintain
> a CPython branch manually for the foreseeable future?  (From experience
> this is a bad idea...)
>

Yeah, not my idea of fun either, but since this is a long term project, I
will at least need to for the foreseeable future.

-Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20060722/3f1b39e2/attachment.htm 

From brett at python.org  Sat Jul 22 19:33:19 2006
From: brett at python.org (Brett Cannon)
Date: Sat, 22 Jul 2006 10:33:19 -0700
Subject: [Python-Dev] new security doc using object-capabilities
In-Reply-To: <20060722070710.GA22334@code0.codespeak.net>
References: <bbaeab100607191535p543cb0ddj66410cf985dd9b77@mail.gmail.com>
	<20060722070710.GA22334@code0.codespeak.net>
Message-ID: <bbaeab100607221033t229d80c3h35b669009787cc87@mail.gmail.com>

On 7/22/06, Armin Rigo <arigo at tunes.org> wrote:
>
> Hi Brett,
>
> On Wed, Jul 19, 2006 at 03:35:45PM -0700, Brett Cannon wrote:
> > I also plan to rewrite the import machinery in pure Python.
>
> http://codespeak.net/svn/pypy/dist/pypy/module/__builtin__/importing.py


Thanks for the link, Armin.  Since you guys don't have the import
restrictions the CPython version would have and just have different coding
needs for RPython obviously I can't just do a blind copy.  But I will
definitely take a look as I develop.  Maybe you guys can even help to lower
the duplication if it makes sense for you.

BTW, do you guys happen to have extra tests from import?

-Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20060722/208e6338/attachment.html 

From grig.gheorghiu at gmail.com  Sat Jul 22 20:16:00 2006
From: grig.gheorghiu at gmail.com (Grig Gheorghiu)
Date: Sat, 22 Jul 2006 11:16:00 -0700
Subject: [Python-Dev] Community buildbots -- reprise
In-Reply-To: <44C249F3.8000908@v.loewis.de>
References: <3f09d5a00607211004kb45ffe2kcedcf418136e7102@mail.gmail.com>
	<44C1E447.2020404@v.loewis.de>
	<3f09d5a00607220832n57f0c5a6m8a193eb8e405846a@mail.gmail.com>
	<44C249F3.8000908@v.loewis.de>
Message-ID: <3f09d5a00607221116m37854df2kafaa381c9e2f81dc@mail.gmail.com>

On 7/22/06, "Martin v. L?wis" <martin at v.loewis.de> wrote:
>
> Grig Gheorghiu wrote:
> >     As I said earlier: If you need some kind of post-commit
> >     trigger on the python repository to trigger a build, just
> >     let me know. We currently use a more-or-less plain
> >     svn_buildbot.py to trigger our own builds.
> >
> > Wouldn't that put too much of a burden on the python core build system?
> > It would have to be aware of all the buildslaves running specific
> projects.
>
> If there is a single "community buildbot", then no. In any case, it's
> primarily administrative overhead, not so much cycles. python.org does
> so many things simultaneously, making it trigger an additional build
> remotely doesn't hurt.
>
> > I was thinking about having a dedicated buildmaster machine, such as the
> > one Neal says he already has, and configure that machine to coordinate a
> > small army of buildslaves which will be contributed for people
> > interested in this effort.
>
> Right. You still need to find out when to rebuild, and getting triggers
> from the source repositories is likely the easiest solution.



I see....I guess I was thinking about building periodically (every X hours
or at time Y) as opposed to getting svn triggers on each check-in. But if,
as you're saying, the overhead on python.org is not too great, we can do
what you suggested.

Grig


-- 
http://agiletesting.blogspot.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20060722/b7d1a491/attachment.htm 

From greg.ewing at canterbury.ac.nz  Sun Jul 23 03:24:43 2006
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sun, 23 Jul 2006 13:24:43 +1200
Subject: [Python-Dev] new security doc using object-capabilities
In-Reply-To: <20060722114615.GB13782@code0.codespeak.net>
References: <bbaeab100607191535p543cb0ddj66410cf985dd9b77@mail.gmail.com>
	<20060722114615.GB13782@code0.codespeak.net>
Message-ID: <44C2CFDB.4060700@canterbury.ac.nz>

Armin Rigo wrote:

> I'm not sure I understand what you propose to fix holes like
> constructors and __subclasses__: it seems that you want to remove them
> altogether (and e.g. make factory functions instead).  That would
> completely break all programs, right?  I mean, there is no way such
> changes would go into mainstream CPython.

How much code is actually out there that uses __subclasses__?
It seems like a fairly esoteric corner of the language to me.

In any case, I think this approach should certainly be tried,
and if it works out, considered for Py3k.

--
Greg

From david.nospam.hopwood at blueyonder.co.uk  Sun Jul 23 03:18:48 2006
From: david.nospam.hopwood at blueyonder.co.uk (David Hopwood)
Date: Sun, 23 Jul 2006 02:18:48 +0100
Subject: [Python-Dev] new security doc using object-capabilities
In-Reply-To: <20060722114615.GB13782@code0.codespeak.net>
References: <bbaeab100607191535p543cb0ddj66410cf985dd9b77@mail.gmail.com>
	<20060722114615.GB13782@code0.codespeak.net>
Message-ID: <44C2CE78.9020802@blueyonder.co.uk>

Armin Rigo wrote:
> Re-hi,
> 
> On Wed, Jul 19, 2006 at 03:35:45PM -0700, Brett Cannon wrote:
> 
>>http://svn.python.org/view/python/branches/bcannon-sandboxing/securing_python.txt?rev=50717&view=log.
> 
> I'm not sure I understand what you propose to fix holes like
> constructors and __subclasses__: it seems that you want to remove them
> altogether (and e.g. make factory functions instead).  That would
> completely break all programs, right?  I mean, there is no way such
> changes would go into mainstream CPython.

If I understand correctly, the proposal is that any incompatible changes
to the language would apply only in "sandboxed" interpreters. So there is
no reason why support for these couldn't go into the main branch.

Of course we want to minimize the changes that will need to be made to
programs and libraries to make them work in a sandboxed interpreter, but
not at the expense of security. Some incompatible changes will be necessary.

-- 
David Hopwood <david.nospam.hopwood at blueyonder.co.uk>



From arigo at tunes.org  Sun Jul 23 08:54:10 2006
From: arigo at tunes.org (Armin Rigo)
Date: Sun, 23 Jul 2006 08:54:10 +0200
Subject: [Python-Dev] new security doc using object-capabilities
In-Reply-To: <bbaeab100607221033t229d80c3h35b669009787cc87@mail.gmail.com>
References: <bbaeab100607191535p543cb0ddj66410cf985dd9b77@mail.gmail.com>
	<20060722070710.GA22334@code0.codespeak.net>
	<bbaeab100607221033t229d80c3h35b669009787cc87@mail.gmail.com>
Message-ID: <20060723065409.GC13782@code0.codespeak.net>

Hi Brett,

On Sat, Jul 22, 2006 at 10:33:19AM -0700, Brett Cannon wrote:
> Thanks for the link, Armin.  Since you guys don't have the import
> restrictions the CPython version would have and just have different coding
> needs for RPython obviously I can't just do a blind copy.  But I will
> definitely take a look as I develop.  Maybe you guys can even help to lower
> the duplication if it makes sense for you.

Yes, it should be possible to abstract the common logic in some way,
using some kind of interface for all OS inspection and 'sys.modules'
manipulations.

> BTW, do you guys happen to have extra tests from import?

Yes, there is
http://codespeak.net/svn/pypy/dist/pypy/module/__builtin__/test/test_import.py

which will also need a bit of rewriting, but that should be
straightforward.


A bientot,

Armin

From arigo at tunes.org  Sun Jul 23 09:08:12 2006
From: arigo at tunes.org (Armin Rigo)
Date: Sun, 23 Jul 2006 09:08:12 +0200
Subject: [Python-Dev] new security doc using object-capabilities
In-Reply-To: <44C2CE78.9020802@blueyonder.co.uk>
References: <bbaeab100607191535p543cb0ddj66410cf985dd9b77@mail.gmail.com>
	<20060722114615.GB13782@code0.codespeak.net>
	<44C2CE78.9020802@blueyonder.co.uk>
Message-ID: <20060723070811.GD13782@code0.codespeak.net>

Hi David, hi Brett,

On Sun, Jul 23, 2006 at 02:18:48AM +0100, David Hopwood wrote:
> If I understand correctly, the proposal is that any incompatible changes
> to the language would apply only in "sandboxed" interpreters. So there is
> no reason why support for these couldn't go into the main branch.

That's what I originally thought too, but Brett writes:

    Implementation Details
    ========================

    An important point to keep in mind when reading about the
    implementation details for the security model is that these are
    general changes and are not special to any type of interpreter,
    sandboxed or otherwise.  That means if a change to a built-in type is
    suggested and it does not involve a proxy, that change is meant
    Python-wide for *all* interpreters.

So that's why I'm starting to worry that Brett is proposing to change
the regular Python language too.  However, Brett, you also say somewhere
else that backward compatibility is not an issue.  So I'm a bit confused
actually...

Also, I hate to sound self-centered, but I should point out somewhere
that PyPy was started by people who no longer wanted to maintain a fork
of CPython, and preferred to work on building CPython-like variants
automatically.  Many of the security features you list would be quite
easier to implement and maintain in PyPy than CPython -- also from a
security perspective: it is easier to be sure that some protection is
complete, and remains complete over time, if it is systematically
generated instead of hand-patched in a dozen places.


A bientot,

Armin

From rasky at develer.com  Sun Jul 23 15:30:50 2006
From: rasky at develer.com (Giovanni Bajo)
Date: Sun, 23 Jul 2006 15:30:50 +0200
Subject: [Python-Dev] Document performance requirements?
References: <e9qbtf$kk6$1@sea.gmane.org>
	<44C0BF4C.1010408@gmail.com><200607210759.56595.ndbecker2@gmail.com><44C0C79E.3020309@gmail.com><bb8868b90607210713p4599e8c3ka5fc8c61f8679486@mail.gmail.com><44C18E89.5090603@gmail.com>
	<20060722111907.GA13782@code0.codespeak.net>
Message-ID: <008201c6ae5c$36b4ca40$d503030a@trilan>

Armin Rigo wrote:

> I think that O-wise the current CPython situation should be documented
> as a "minimal requirement" for implementations of the language, with
> just one exception: the well-documented "don't rely on this" hack in
> 2.4 to make repeated 'str += str' amortized linear, for which the 2.3
> quadratic behavior is considered compliant enough.
>
> I suppose that allowing implementations to provide better algorithmic
> complexities than required is fine, although I can think of some
> problems with that (e.g. nice and efficient user code that would
> perform horribly badly on CPython).

I'm not sure big-O tells the whole truth. For instance, do we want to allow
an implementation to use a hash table as underlying type for a list? It
would match big-O requirements, but would still be slower than a plain array
because of higher overhead of implementation (higher constant factor).

And if this is allowed, I would like to find in CPython tutorials and
documentations a simple statement like: "to implement the list and match its
requirements, CPython choose a simple array as underlying data structure".
-- 
Giovanni Bajo


From tnelson at onresolve.com  Sun Jul 23 19:42:59 2006
From: tnelson at onresolve.com (Trent Nelson)
Date: Sun, 23 Jul 2006 19:42:59 +0200
Subject: [Python-Dev] Python 2.4, VS 2005 & Profile Guided Optmization
Message-ID: <AB1BB4CF81B9214682079EB55A7125EDEFA707@mapibe05.exchange.xchg>

Hi,

Has anyone else built Python with Visual Studio 2005 and played around
with Profile Guided Optimization?  I had to build Python from source w/
VS 2005 as I had a few .pyd's built with VS 2005 that I wanted to load;
I ended up playing around with Profile Guided Optimization, running
``python.exe pystones.py'' to collect call-graph data after
python.exe/Python24.dll had been instrumented, then recompiling with the
optimizations fed back in.  

Results were interesting, an average speedup of around 33% was
noticeable:

ActiveState 2.4.3 python.exe:

    C:\Python24>python.exe Lib\test\pystone.py
    Pystone(1.1) time for 50000 passes = 0.980119 This machine
benchmarks at
    51014.2 pystones/second

The python compiled from branches/release24-maint with VS 2005 + profile
guided optimization:

    C:\Python24>python.exe Lib\test\pystone.py
    Pystone(1.1) time for 50000 passes = 0.73261 This machine benchmarks
at
    68249.2 pystones/second

Is there any motivation in the Win32 Python dev camp to switch from VC6
to VS 2005?

FWIW, although there were a shed-load of warnings when compiling python
and pythoncore (and a lot more errors when compiling other modules), I
only had to apply one patch to get it working well enough to run
pystone.py.  Without this patch, the VC8 CRT aborts at runtime as soon
as an invalid signal is passed to signal(); which is inevitable given
the current code in the initsignal() method:

	for (i = 1; i < NSIG; i++) {
		void (*t)(int);
		t = PyOS_getsig(i);


Regards,

	Trent.

--
http://www.onresolve.com


Index: signalmodule.c
===================================================================
--- signalmodule.c	(revision 47196)
+++ signalmodule.c	(working copy)
@@ -280,7 +280,21 @@
 	{NULL,			NULL}		/* sentinel */
 };
 
+#define WIN32VS2005HACK
+#ifdef WIN32VS2005HACK
+#include <stdio.h>
+#include <stdlib.h>
+#include <crtdbg.h> 
+void dummy_handler(const wchar_t *exp,
+                   const wchar_t *fn,
+                   const wchar_t *file,
+                   unsigned int line,
+                   uintptr_t reserved)
+{
 
+}
+#endif
+
 PyDoc_STRVAR(module_doc,
 "This module provides mechanisms to use signal handlers in Python.\n\
 \n\
@@ -339,6 +353,12 @@
                 goto finally;
 	Py_INCREF(IntHandler);
 
+#ifdef WIN32VS2005HACK
+        (void)_set_invalid_parameter_handler(&dummy_handler);
+        _CrtSetReportMode(_CRT_ASSERT, 0);
+#endif
+
+
 	Handlers[0].tripped = 0;
 	for (i = 1; i < NSIG; i++) {
 		void (*t)(int);

From tonynelson at georgeanelson.com  Sun Jul 23 19:26:59 2006
From: tonynelson at georgeanelson.com (Tony Nelson)
Date: Sun, 23 Jul 2006 13:26:59 -0400
Subject: [Python-Dev] Socket Timeouts patch 1519025
Message-ID: <v04020a00c0e95c95d9bc@[192.168.123.162]>

I request a review of my patch (1519025) to get socket timeouts to work
properly with errors and signals.  I don't expect this patch would make it
into 2.5, but perhaps it could be in 2.5.1, as it fixes a long-standing
bug.  I know that people are busy with getting 2.5 out the door, but it
would be helpful for me to know if my current patch is OK before I start on
another patch to make socket timeouts more useful.  There is also a version
of the patch for 2.4, which would make yum nicer in Fedora 4 and 5, and I
think that passing a review would make the patch more acceptable to
Fedora's maintainers.

My next patch will, if it works, make socket timeouts easier to use
per-thread, allow for the timing of entire operations rather than just
timing transaction phases, allow for setting an acceptable rate for file
transfers, and should be completely backward compatible, in that old code
would be unaffected and new code would work as well as possible now on
older unpatched versions.  That's my plan, anyway.  It would build on my
current patch, at least in its principles.
____________________________________________________________________
TonyN.:'                       <mailto:tonynelson at georgeanelson.com>
      '                              <http://www.georgeanelson.com/>

From brett at python.org  Sun Jul 23 22:00:06 2006
From: brett at python.org (Brett Cannon)
Date: Sun, 23 Jul 2006 13:00:06 -0700
Subject: [Python-Dev] new security doc using object-capabilities
In-Reply-To: <20060723070811.GD13782@code0.codespeak.net>
References: <bbaeab100607191535p543cb0ddj66410cf985dd9b77@mail.gmail.com>
	<20060722114615.GB13782@code0.codespeak.net>
	<44C2CE78.9020802@blueyonder.co.uk>
	<20060723070811.GD13782@code0.codespeak.net>
Message-ID: <bbaeab100607231300w187dfa8dx87ef0a94935e56ce@mail.gmail.com>

On 7/23/06, Armin Rigo <arigo at tunes.org> wrote:
>
> Hi David, hi Brett,
>
> On Sun, Jul 23, 2006 at 02:18:48AM +0100, David Hopwood wrote:
> > If I understand correctly, the proposal is that any incompatible changes
> > to the language would apply only in "sandboxed" interpreters. So there
> is
> > no reason why support for these couldn't go into the main branch.
>
> That's what I originally thought too, but Brett writes:
>
>     Implementation Details
>     ========================
>
>     An important point to keep in mind when reading about the
>     implementation details for the security model is that these are
>     general changes and are not special to any type of interpreter,
>     sandboxed or otherwise.  That means if a change to a built-in type is
>     suggested and it does not involve a proxy, that change is meant
>     Python-wide for *all* interpreters.
>
> So that's why I'm starting to worry that Brett is proposing to change
> the regular Python language too.


Yes, I am proposing changing some constructors and methods on some built-in
types for the regular languages.  That's it.  No new keywords or major
semantic changes and such.  If I make changes just for sandboxed
interpreters it changes the general approach of the security model by then
requiring an identity check to see if the interpreter is sandboxed or not.

  However, Brett, you also say somewhere
> else that backward compatibility is not an issue.  So I'm a bit confused
> actually...


Since this is my Ph.D. dissertation first and foremost, I am not going to
tie my hands in such a way that I have to make too much of a compromise in
order for this to work.  I obviously don't want to change the feel of
Python, but if I have to remove the constructor for code objects to prevent
evil bytecode or __subclasses__() from object to prevent poking around
stuff, then so be it.  For this project, security is trumpeting
backwards-compatibility when the latter is impossible in order to have the
former.  I will obviously try to minimize it, but something that works at
such a basic level of the language is just going to require some changes for
it to work.

Also, I hate to sound self-centered, but I should point out somewhere
> that PyPy was started by people who no longer wanted to maintain a fork
> of CPython, and preferred to work on building CPython-like variants
> automatically.  Many of the security features you list would be quite
> easier to implement and maintain in PyPy than CPython -- also from a
> security perspective: it is easier to be sure that some protection is
> complete, and remains complete over time, if it is systematically
> generated instead of hand-patched in a dozen places.


It doesn't sound self-centered.  =)  Problem is that my knowledge base is
obviously all in CPython so my startup costs are much lower than if I tried
this in PyPy.  Plus there is the point of embedding this into Firefox
(possibly) eventually.  Does PyPy support embedding yet at the C level?

-Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20060723/229c8fb8/attachment.htm 

From rasky at develer.com  Sun Jul 23 22:41:46 2006
From: rasky at develer.com (Giovanni Bajo)
Date: Sun, 23 Jul 2006 22:41:46 +0200
Subject: [Python-Dev] Python 2.4, VS 2005 & Profile Guided Optmization
References: <AB1BB4CF81B9214682079EB55A7125EDEFA707@mapibe05.exchange.xchg>
Message-ID: <034401c6ae98$6a0da920$d503030a@trilan>

Trent Nelson wrote:

> Has anyone else built Python with Visual Studio 2005 and played around
> with Profile Guided Optimization?

Yes, there was some work at the recent Need for Speed sprint. Python 2.5 has
a PCBuild8 directory (for VS 2005) with a specific project for PGO.

> Results were interesting, an average speedup of around 33% was
> noticeable:

Yes, they are.

> Is there any motivation in the Win32 Python dev camp to switch from
> VC6 to VS 2005?

I think Martin decided to keep VC71 (Visual Studio .NET 2003) for another
release cycle. Given the impressive results of VC8 with PGO, and the fact
that Visual Studio Express 2005 is free forever, I would hope as well for
the decision to be reconsidered.
-- 
Giovanni Bajo


From pje at telecommunity.com  Sun Jul 23 22:45:54 2006
From: pje at telecommunity.com (Phillip J. Eby)
Date: Sun, 23 Jul 2006 16:45:54 -0400
Subject: [Python-Dev] new security doc using object-capabilities
In-Reply-To: <bbaeab100607231300w187dfa8dx87ef0a94935e56ce@mail.gmail.co
 m>
References: <20060723070811.GD13782@code0.codespeak.net>
	<bbaeab100607191535p543cb0ddj66410cf985dd9b77@mail.gmail.com>
	<20060722114615.GB13782@code0.codespeak.net>
	<44C2CE78.9020802@blueyonder.co.uk>
	<20060723070811.GD13782@code0.codespeak.net>
Message-ID: <5.1.1.6.0.20060723164246.04144298@sparrow.telecommunity.com>

At 01:00 PM 7/23/2006 -0700, Brett Cannon wrote:
>I obviously don't want to change the feel of Python, but if I have to 
>remove the constructor for code objects to prevent evil bytecode or 
>__subclasses__() from object to prevent poking around stuff, then so be 
>it.  For this project, security is trumpeting backwards-compatibility when 
>the latter is impossible in order to have the former.  I will obviously 
>try to minimize it, but something that works at such a basic level of the 
>language is just going to require some changes for it to work.

Zope 3's sandboxing machinery manages to handle securing these things 
without any language changes.  So, declaring it "impossible" to manage 
without backward compatibility seems inappropriate, or at least 
incorrect.  But perhaps there is something I'm missing?


From david.nospam.hopwood at blueyonder.co.uk  Mon Jul 24 00:07:25 2006
From: david.nospam.hopwood at blueyonder.co.uk (David Hopwood)
Date: Sun, 23 Jul 2006 23:07:25 +0100
Subject: [Python-Dev] new security doc using object-capabilities
In-Reply-To: <5.1.1.6.0.20060723164246.04144298@sparrow.telecommunity.com>
References: <20060723070811.GD13782@code0.codespeak.net>	<bbaeab100607191535p543cb0ddj66410cf985dd9b77@mail.gmail.com>	<20060722114615.GB13782@code0.codespeak.net>	<44C2CE78.9020802@blueyonder.co.uk>	<20060723070811.GD13782@code0.codespeak.net>
	<5.1.1.6.0.20060723164246.04144298@sparrow.telecommunity.com>
Message-ID: <44C3F31D.3070201@blueyonder.co.uk>

Phillip J. Eby wrote:
> At 01:00 PM 7/23/2006 -0700, Brett Cannon wrote:
> 
>>I obviously don't want to change the feel of Python, but if I have to 
>>remove the constructor for code objects to prevent evil bytecode or 
>>__subclasses__() from object to prevent poking around stuff, then so be 
>>it.  For this project, security is [trumping] backwards-compatibility when 
>>the latter is impossible in order to have the former.  I will obviously 
>>try to minimize it, but something that works at such a basic level of the 
>>language is just going to require some changes for it to work.
> 
> Zope 3's sandboxing machinery manages to handle securing these things 
> without any language changes.  So, declaring it "impossible" to manage 
> without backward compatibility seems inappropriate, or at least 
> incorrect.

... if Zope's sandboxing is secure. I haven't done a security review of it,
but your argument assumes that it is.

In any case, Zope's sandboxing is not capability-based.

-- 
David Hopwood <david.nospam.hopwood at blueyonder.co.uk>



From foom at fuhm.net  Mon Jul 24 01:59:08 2006
From: foom at fuhm.net (James Y Knight)
Date: Sun, 23 Jul 2006 19:59:08 -0400
Subject: [Python-Dev] Python 2.4, VS 2005 & Profile Guided Optmization
In-Reply-To: <034401c6ae98$6a0da920$d503030a@trilan>
References: <AB1BB4CF81B9214682079EB55A7125EDEFA707@mapibe05.exchange.xchg>
	<034401c6ae98$6a0da920$d503030a@trilan>
Message-ID: <646666C6-A1DB-4D2C-8054-FA9F7E2129E4@fuhm.net>

On Jul 23, 2006, at 4:41 PM, Giovanni Bajo wrote:
> I think Martin decided to keep VC71 (Visual Studio .NET 2003) for  
> another
> release cycle. Given the impressive results of VC8 with PGO, and  
> the fact
> that Visual Studio Express 2005 is free forever, I would hope as  
> well for
> the decision to be reconsidered.

Wasn't there a "Free Forever" 2003 edition too, which has since  
completely disappeared? Why do you think that MS won't stop  
distributing the Free Forever VS 2005 once VS 2005+1 comes out, the  
same way they did the 2003 one?

James


From david.nospam.hopwood at blueyonder.co.uk  Mon Jul 24 01:32:47 2006
From: david.nospam.hopwood at blueyonder.co.uk (David Hopwood)
Date: Mon, 24 Jul 2006 00:32:47 +0100
Subject: [Python-Dev] new security doc using object-capabilities
In-Reply-To: <bbaeab100607231300w187dfa8dx87ef0a94935e56ce@mail.gmail.com>
References: <bbaeab100607191535p543cb0ddj66410cf985dd9b77@mail.gmail.com>	
	<20060722114615.GB13782@code0.codespeak.net>	
	<44C2CE78.9020802@blueyonder.co.uk>	
	<20060723070811.GD13782@code0.codespeak.net>
	<bbaeab100607231300w187dfa8dx87ef0a94935e56ce@mail.gmail.com>
Message-ID: <44C4071F.5050003@blueyonder.co.uk>

Brett Cannon wrote:
> On 7/23/06, Armin Rigo <arigo at tunes.org> wrote:
>
>> Hi David, hi Brett,
>>
>> On Sun, Jul 23, 2006 at 02:18:48AM +0100, David Hopwood wrote:
>> > If I understand correctly, the proposal is that any incompatible
>> > changes to the language would apply only in "sandboxed" interpreters.
>> > So there is no reason why support for these couldn't go into the
>> > main branch.
>>
>> That's what I originally thought too, but Brett writes:
>>
>>     Implementation Details
>>     ========================
>>
>>     An important point to keep in mind when reading about the
>>     implementation details for the security model is that these are
>>     general changes and are not special to any type of interpreter,
>>     sandboxed or otherwise.  That means if a change to a built-in type is
>>     suggested and it does not involve a proxy, that change is meant
>>     Python-wide for *all* interpreters.
>>
>> So that's why I'm starting to worry that Brett is proposing to change
>> the regular Python language too.
> 
> Yes, I am proposing changing some constructors and methods on some built-in
> types for the regular languages.  That's it.  No new keywords or major
> semantic changes and such.  If I make changes just for sandboxed
> interpreters it changes the general approach of the security model by then
> requiring an identity check to see if the interpreter is sandboxed or not.

I assume that the extent of incompatible changes would be limited as much as
possible. So the only checks would be in operations that are directly affected
by whatever incompatible changes are made. The performance and complexity
costs of this are likely to be small -- or at least should not be assumed to
be large before having hammered out a more detailed design.

Suppose, for the sake of argument, that we introduced private methods and
attributes. If an attribute in an existing standard library class was changed
to be private, then code depending on it would break. But if there were a
notion of a "compatibility private" attribute that acts as private only in a
sandboxed interpreter, then no code running in an unprotected interpreter
would break.

-- 
David Hopwood <david.nospam.hopwood at blueyonder.co.uk>



From tjreedy at udel.edu  Mon Jul 24 04:45:27 2006
From: tjreedy at udel.edu (Terry Reedy)
Date: Sun, 23 Jul 2006 22:45:27 -0400
Subject: [Python-Dev] Python 2.4, VS 2005 & Profile Guided Optmization
References: <AB1BB4CF81B9214682079EB55A7125EDEFA707@mapibe05.exchange.xchg>
	<034401c6ae98$6a0da920$d503030a@trilan>
Message-ID: <ea1c88$aa4$1@sea.gmane.org>


"Giovanni Bajo" <rasky at develer.com> wrote in message 
news:034401c6ae98$6a0da920$d503030a at trilan...
> that Visual Studio Express 2005 is free forever, I would hope as well for
> the decision to be reconsidered.

But is it freely redistributable forever?  Or even now?  I have the 2003 
toolkit sitting on my disk, but I am apparently not supposed to send it to 
anyone else.

tjr




From nyamatongwe at gmail.com  Mon Jul 24 05:03:22 2006
From: nyamatongwe at gmail.com (Neil Hodgson)
Date: Mon, 24 Jul 2006 13:03:22 +1000
Subject: [Python-Dev] Python 2.4, VS 2005 & Profile Guided Optmization
In-Reply-To: <AB1BB4CF81B9214682079EB55A7125EDEFA707@mapibe05.exchange.xchg>
References: <AB1BB4CF81B9214682079EB55A7125EDEFA707@mapibe05.exchange.xchg>
Message-ID: <50862ebd0607232003gc800a48r3853bd773d3a7f21@mail.gmail.com>

Trent Nelson:

> I ended up playing around with Profile Guided Optimization, running
> ``python.exe pystones.py'' to collect call-graph data after
> python.exe/Python24.dll had been instrumented, then recompiling with the
> optimizations fed back in.

   It'd be an idea to build a larger body of Python code to run the
profiling pass on so it doesn't just optimize the sort of code in
pystone which is not very representative. Could run the test suite as
it would have good coverage but would hit exceptional cases too
heavily. Other compilers (Intel?) support profile directed
optimization so would also benefit from such a body of code.

   Neil

From pje at telecommunity.com  Mon Jul 24 05:30:24 2006
From: pje at telecommunity.com (Phillip J. Eby)
Date: Sun, 23 Jul 2006 23:30:24 -0400
Subject: [Python-Dev] new security doc using object-capabilities
In-Reply-To: <44C3F31D.3070201@blueyonder.co.uk>
References: <5.1.1.6.0.20060723164246.04144298@sparrow.telecommunity.com>
	<20060723070811.GD13782@code0.codespeak.net>
	<bbaeab100607191535p543cb0ddj66410cf985dd9b77@mail.gmail.com>
	<20060722114615.GB13782@code0.codespeak.net>
	<44C2CE78.9020802@blueyonder.co.uk>
	<20060723070811.GD13782@code0.codespeak.net>
	<5.1.1.6.0.20060723164246.04144298@sparrow.telecommunity.com>
Message-ID: <5.1.1.6.0.20060723194448.0205fa50@sparrow.telecommunity.com>

At 11:07 PM 7/23/2006 +0100, David Hopwood wrote:
>Phillip J. Eby wrote:
> > At 01:00 PM 7/23/2006 -0700, Brett Cannon wrote:
> >
> >>I obviously don't want to change the feel of Python, but if I have to
> >>remove the constructor for code objects to prevent evil bytecode or
> >>__subclasses__() from object to prevent poking around stuff, then so be
> >>it.  For this project, security is [trumping] backwards-compatibility when
> >>the latter is impossible in order to have the former.  I will obviously
> >>try to minimize it, but something that works at such a basic level of the
> >>language is just going to require some changes for it to work.
> >
> > Zope 3's sandboxing machinery manages to handle securing these things
> > without any language changes.  So, declaring it "impossible" to manage
> > without backward compatibility seems inappropriate, or at least
> > incorrect.
>
>... if Zope's sandboxing is secure. I haven't done a security review of it,
>but your argument assumes that it is.

What argument is that?  I'm merely suggesting that coming up with a 
completely new way to secure Python without a serious consideration of 
existing practical prior art (with many years' deployment experience on the 
public internet!) seems ill-advised with respect to achieving practical goals.

Brett's securing_python.txt don't refer to or cite Zope in any way, but 
rather relies on broad and unsupported assertions about what can or can't 
be done with Python.  I hope he isn't doing the same in his thesis, as this 
is rather like writing about one's new theory of how to have a worldwide 
ball-kicking contest without making any reference as to how one's theory 
compares with the World Cup.

I'm not saying Zope is better or worse.  I'm simply saying that in a 
business context, a failure to compare and contrast a proposed "build" 
solution to show how it would be better than a well-established available 
"buy" solution would be called something like "lack of due diligence".  I 
think in the academic context it might be called something like "failure to 
cite", but the general idea is the same, i.e., not doing your homework.  :)

In other words, if the solution being proposed is better than what Zope 
does, the appropriate thing in business is to show the reasons why, and the 
appropriate thing in science is to state a hypothesis regarding the 
differences, and then perform an experiment to either prove or disprove it.


>In any case, Zope's sandboxing is not capability-based.

You're right: you haven't done a review of it.  :)  If you had, you'd know 
that one proxy plus one namechecker equals one capability.  In other words, 
you could take the restricted interpreter, the proxy mechanism, and the 
namechecker and leave most of the rest alone, and you'd have your 
capability system.  Then you could focus more time and attention on the 
parts of the problem that Zope *doesn't* solve, instead of reinventing the 
ones that it already does.

Now, if Brett believes that changing the Python language is a *better* way 
to implement capabilities than using proxies to implement them, then 
great.  His paper should explain why, and (presumably) include experimental 
results to show that they're either better or worse than Zope's approach 
based on some criteria.  The same information is relevant to Python-Dev as 
to what is an appropriate approach to support sandboxing in CPython.  What 
are the advantages of a built-in approach versus an add-on approach?  Are 
there interpreter facilities that could be added to shore up any awkward 
aspects of Zope's approach?  (Whatever those might be.)

For example, one part of Zope's approach uses a custom compiler and custom 
builtins in order to redefine how attribute access works in certain 
cases.  Could these customizations be replaced with options built into the 
Python compiler and interpreter?  What improvements would that result in?

Simply handwaving all of these questions away, however, with broad 
assertions of superiority and without even attempting to compare the new 
work to Zope's existing work is really not acceptable for academia OR 
Python development.

For the record: I have no personal interest in Zope's security system.  I 
didn't develop it and haven't had the need to use it, myself.  I once 
reviewed some of the code and offered some minor suggestions, mainly 
regarding performance improvement.  My only axe to grind in this matter is 
what I've already stated: I think it would be crazy (in the "monumental 
waste of resources" sense) to consider putting *any* sandboxing system into 
CPython without tapping the Zope team's experiences.  For example: having 
implemented such a system, what compiler or interpreter changes would've 
made the job easier?

Meanwhile, what Brett does or doesn't put in his thesis is between him and 
his advisor, but what gets put into Python shouldn't be based on ignoring 
the existing field experience and state of the art.


From unknown_kev_cat at hotmail.com  Mon Jul 24 06:13:50 2006
From: unknown_kev_cat at hotmail.com (Joe Smith)
Date: Mon, 24 Jul 2006 00:13:50 -0400
Subject: [Python-Dev] Python 2.4, VS 2005 & Profile Guided Optmization
References: <AB1BB4CF81B9214682079EB55A7125EDEFA707@mapibe05.exchange.xchg><034401c6ae98$6a0da920$d503030a@trilan>
	<646666C6-A1DB-4D2C-8054-FA9F7E2129E4@fuhm.net>
Message-ID: <ea1hdl$k0v$1@sea.gmane.org>


"James Y Knight" <foom at fuhm.net> wrote in message 
news:646666C6-A1DB-4D2C-8054-FA9F7E2129E4 at fuhm.net...
> On Jul 23, 2006, at 4:41 PM, Giovanni Bajo wrote:
>> I think Martin decided to keep VC71 (Visual Studio .NET 2003) for
>> another
>> release cycle. Given the impressive results of VC8 with PGO, and
>> the fact
>> that Visual Studio Express 2005 is free forever, I would hope as
>> well for
>> the decision to be reconsidered.
>
> Wasn't there a "Free Forever" 2003 edition too, which has since
> completely disappeared? Why do you think that MS won't stop
> distributing the Free Forever VS 2005 once VS 2005+1 comes out, the
> same way they did the 2003 one?

I am not aware of any full free ofeering based on 2003. There were ways
to get the command line tools, but the GUI's were not available.
Most of the freely available command line utilities are still available from
microsoft if you find the correct page.

I will note that according to the FAQ,  Profile Guided Optimizations will 
not be available
with Express. Because the compiler is the same, this means the tools to 
generate a profile
are all that is missing. Express should still be able to build using the 
optimized profile.

Microsoft as a general rule, does not go after people distributing products 
that Microsoft has labeled
free, even after Microsoft no longer distributes that product. So the 
express editions will continue to
be available long into the future if 2005+1 does not have a free version.

(The logic behind this is that Microsoft would have a hard time explaining 
to a jury how somebody can
"pirate" software that is available at no cost.)





From unknown_kev_cat at hotmail.com  Mon Jul 24 06:24:23 2006
From: unknown_kev_cat at hotmail.com (Joe Smith)
Date: Mon, 24 Jul 2006 00:24:23 -0400
Subject: [Python-Dev] Python 2.4, VS 2005 & Profile Guided Optmization
References: <AB1BB4CF81B9214682079EB55A7125EDEFA707@mapibe05.exchange.xchg>
	<50862ebd0607232003gc800a48r3853bd773d3a7f21@mail.gmail.com>
Message-ID: <ea1i1e$l5c$1@sea.gmane.org>


"Neil Hodgson" <nyamatongwe at gmail.com> wrote in message 
news:50862ebd0607232003gc800a48r3853bd773d3a7f21 at mail.gmail.com...
> Trent Nelson:
>
>> I ended up playing around with Profile Guided Optimization, running
>> ``python.exe pystones.py'' to collect call-graph data after
>> python.exe/Python24.dll had been instrumented, then recompiling with the
>> optimizations fed back in.
>
>   It'd be an idea to build a larger body of Python code to run the
> profiling pass on so it doesn't just optimize the sort of code in
> pystone which is not very representative. Could run the test suite as
> it would have good coverage but would hit exceptional cases too
> heavily. Other compilers (Intel?) support profile directed
> optimization so would also benefit from such a body of code.
>

GCC suppost profiling optimized code. One caveat is that the profile-enabled 
builds
ause a GPL'd library. Not a problem for Python right now as it is 
GPL-compatible,
but a caveat non-the-less.

That does not apply to the final optimized app. Anyway the flags used are: 
"-fprofile-generate"
and "-fprofile-use". 



From martin at v.loewis.de  Mon Jul 24 06:25:46 2006
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Mon, 24 Jul 2006 06:25:46 +0200
Subject: [Python-Dev] Python 2.4, VS 2005 & Profile Guided Optmization
In-Reply-To: <ea1hdl$k0v$1@sea.gmane.org>
References: <AB1BB4CF81B9214682079EB55A7125EDEFA707@mapibe05.exchange.xchg><034401c6ae98$6a0da920$d503030a@trilan>	<646666C6-A1DB-4D2C-8054-FA9F7E2129E4@fuhm.net>
	<ea1hdl$k0v$1@sea.gmane.org>
Message-ID: <44C44BCA.30404@v.loewis.de>

Joe Smith wrote:

> Microsoft as a general rule, does not go after people distributing
> products that Microsoft has labeled free, even after Microsoft no
> longer distributes that product. So the express editions will
> continue to be available long into the future if 2005+1 does not have
> a free version.

Interesting. So people can do the same with the free 2003 version.

Regards,
Martin


From brett at python.org  Mon Jul 24 06:27:50 2006
From: brett at python.org (Brett Cannon)
Date: Sun, 23 Jul 2006 21:27:50 -0700
Subject: [Python-Dev] new security doc using object-capabilities
In-Reply-To: <5.1.1.6.0.20060723194448.0205fa50@sparrow.telecommunity.com>
References: <bbaeab100607191535p543cb0ddj66410cf985dd9b77@mail.gmail.com>
	<20060722114615.GB13782@code0.codespeak.net>
	<44C2CE78.9020802@blueyonder.co.uk>
	<20060723070811.GD13782@code0.codespeak.net>
	<5.1.1.6.0.20060723164246.04144298@sparrow.telecommunity.com>
	<44C3F31D.3070201@blueyonder.co.uk>
	<5.1.1.6.0.20060723194448.0205fa50@sparrow.telecommunity.com>
Message-ID: <bbaeab100607232127p1fdb401ct12d471925453cb9b@mail.gmail.com>

On 7/23/06, Phillip J. Eby <pje at telecommunity.com> wrote:
>
> At 11:07 PM 7/23/2006 +0100, David Hopwood wrote:
> >Phillip J. Eby wrote:


[snip]

Brett's securing_python.txt don't refer to or cite Zope in any way, but
> rather relies on broad and unsupported assertions about what can or can't
> be done with Python.  I hope he isn't doing the same in his thesis, as
> this
> is rather like writing about one's new theory of how to have a worldwide
> ball-kicking contest without making any reference as to how one's theory
> compares with the World Cup.


The design doc is not meant to be taken as any sort of draft of my thesis.
I did read that link you sent me, Philip, but it was hard to follow.  So I
used Google to find another reference that explained it to me much more
clearly.  securing_python.txt is meant to explain what I am planning to
python-dev so that if someone sees some fatal flaw they can speak up and let
me know, not as a thorough comparison of why my approach is better than
anyone other one.

I'm not saying Zope is better or worse.  I'm simply saying that in a
> business context, a failure to compare and contrast a proposed "build"
> solution to show how it would be better than a well-established available
> "buy" solution would be called something like "lack of due diligence".  I
> think in the academic context it might be called something like "failure
> to
> cite", but the general idea is the same, i.e., not doing your
> homework.  :)
>
> In other words, if the solution being proposed is better than what Zope
> does, the appropriate thing in business is to show the reasons why, and
> the
> appropriate thing in science is to state a hypothesis regarding the
> differences, and then perform an experiment to either prove or disprove
> it.


I am not going to write out a blow-by-blow comparison right now.  It will
come with the thesis.  And I am not expecting my approach or code to be
checked in blindly anyway.

>In any case, Zope's sandboxing is not capability-based.
>
> You're right: you haven't done a review of it.  :)  If you had, you'd know
> that one proxy plus one namechecker equals one capability.  In other
> words,
> you could take the restricted interpreter, the proxy mechanism, and the
> namechecker and leave most of the rest alone, and you'd have your
> capability system.  Then you could focus more time and attention on the
> parts of the problem that Zope *doesn't* solve, instead of reinventing the
> ones that it already does.


Right, but I am trying to remove the need for a namechecker which makes it
an object-capabilities system.

Now, if Brett believes that changing the Python language is a *better* way
> to implement capabilities than using proxies to implement them, then
> great.  His paper should explain why, and (presumably) include
> experimental
> results to show that they're either better or worse than Zope's approach
> based on some criteria.  The same information is relevant to Python-Dev as
> to what is an appropriate approach to support sandboxing in CPython.  What
> are the advantages of a built-in approach versus an add-on approach?  Are
> there interpreter facilities that could be added to shore up any awkward
> aspects of Zope's approach?  (Whatever those might be.)


I think people are starting to lose sight of the purpose of the doc I
wrote.  It was to explain what I was doing for people to see if there was
any fatal flaw and to keep people updated on what I am planning on doing.
It is not meant to convince anyone that my way is the best way yet.  I am
not even going to attempt that until I have working code.

For example, one part of Zope's approach uses a custom compiler and custom
> builtins in order to redefine how attribute access works in certain
> cases.  Could these customizations be replaced with options built into the
> Python compiler and interpreter?  What improvements would that result in?


Part of my point is to help alleviate the need for custom anything.

Simply handwaving all of these questions away, however, with broad
> assertions of superiority and without even attempting to compare the new
> work to Zope's existing work is really not acceptable for academia OR
> Python development.
>
> For the record: I have no personal interest in Zope's security system.  I
> didn't develop it and haven't had the need to use it, myself.  I once
> reviewed some of the code and offered some minor suggestions, mainly
> regarding performance improvement.  My only axe to grind in this matter is
> what I've already stated: I think it would be crazy (in the "monumental
> waste of resources" sense) to consider putting *any* sandboxing system
> into
> CPython without tapping the Zope team's experiences.  For example: having
> implemented such a system, what compiler or interpreter changes would've
> made the job easier?
>
> Meanwhile, what Brett does or doesn't put in his thesis is between him and
> his advisor, but what gets put into Python shouldn't be based on ignoring
> the existing field experience and state of the art.


There is no ignoring of anything.  I understand their basic approach and I
want to try another one.  I like the fundemental design difference of
object-capabilities and so I am going to give it a shot and see how it works
out.  If Zope's proxies are better, then fine, get them to contribute to the
core so that we have security again.  No one is proposing blind acceptance
of my approach.

But as it stands now, Guido likes my initial ideas (including the possible
changes to the language), I like my approach, and my supervisor is willing
to let me give it a shot, so that is good enough for me to move forward.
There will be a thorough discussion I am sure once the code is finished and
ready to be proposed for possible inclusion in CPython.  Until then, though,
just consider it my little experiment that I am just keeping people abreast
of.

-Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20060723/bd0f74ea/attachment.htm 

From greg.ewing at canterbury.ac.nz  Mon Jul 24 06:51:23 2006
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Mon, 24 Jul 2006 16:51:23 +1200
Subject: [Python-Dev] Python 2.4, VS 2005 & Profile Guided Optmization
In-Reply-To: <ea1hdl$k0v$1@sea.gmane.org>
References: <AB1BB4CF81B9214682079EB55A7125EDEFA707@mapibe05.exchange.xchg>
	<034401c6ae98$6a0da920$d503030a@trilan>
	<646666C6-A1DB-4D2C-8054-FA9F7E2129E4@fuhm.net>
	<ea1hdl$k0v$1@sea.gmane.org>
Message-ID: <44C451CB.3060607@canterbury.ac.nz>

Joe Smith wrote:

> Microsoft as a general rule, does not go after people distributing products 
> that Microsoft has labeled
> free, even after Microsoft no longer distributes that product.

But if the licence agreement technically forbids
redistribution, it doesn't seem like a good idea
to rely on Microsoft turning a blind eye to that.

--
Greg


From pje at telecommunity.com  Mon Jul 24 08:10:44 2006
From: pje at telecommunity.com (Phillip J. Eby)
Date: Mon, 24 Jul 2006 02:10:44 -0400
Subject: [Python-Dev] new security doc using object-capabilities
In-Reply-To: <bbaeab100607232127p1fdb401ct12d471925453cb9b@mail.gmail.co
 m>
References: <5.1.1.6.0.20060723194448.0205fa50@sparrow.telecommunity.com>
	<bbaeab100607191535p543cb0ddj66410cf985dd9b77@mail.gmail.com>
	<20060722114615.GB13782@code0.codespeak.net>
	<44C2CE78.9020802@blueyonder.co.uk>
	<20060723070811.GD13782@code0.codespeak.net>
	<5.1.1.6.0.20060723164246.04144298@sparrow.telecommunity.com>
	<44C3F31D.3070201@blueyonder.co.uk>
	<5.1.1.6.0.20060723194448.0205fa50@sparrow.telecommunity.com>
Message-ID: <5.1.1.6.0.20060724015521.0261d160@sparrow.telecommunity.com>

At 09:27 PM 7/23/2006 -0700, Brett Cannon wrote:
>On 7/23/06, Phillip J. Eby 
><<mailto:pje at telecommunity.com>pje at telecommunity.com> wrote:
>>one proxy plus one namechecker equals one capability.  In other words,
>>you could take the restricted interpreter, the proxy mechanism, and the
>>namechecker and leave most of the rest alone, and you'd have your
>>capability system.  Then you could focus more time and attention on the
>>parts of the problem that Zope *doesn't* solve, instead of reinventing the
>>ones that it already does.
>
>Right, but I am trying to remove the need for a namechecker which makes it 
>an object-capabilities system.

As I said above: a namechecker plus a proxy *equals* an object capability.

When I say "name checker" I mean the Zope type that allows you to specify a 
list of names that are allowed for a given object.  This allowing is not 
based on identity or code signing or anything like that.  It's just a list 
of attribute names: i.e. a capability mask over an existing object.

When you create a proxy using this name mask, that proxy becomes a 
capability that allows access to the given names on the underlying object.


>I like the fundemental design difference of object-capabilities

It's not a difference at all, let alone a fundamental one.  Zope just 
happens to allow other kinds of security checking *in addition to* 
capabilities, if you want them.  However, most of its more basic 
encapsulation features are 100% capability based.

Meanwhile, if you want to implement an object-capability system, you will 
need something that is basically a mask, to allow one piece of code to 
create capabilities that can be given to another.  What you end up with for 
doing that is going to look almost exactly like a Zope proxy plus a Zope 
name checker.

I hate to harp on this point, but there seems to be a trend that when 
people have capabilities on their mind, they tend to look at Zope and 
dismiss it as not being capability-based, when in fact Zope's approach is 
capabilities *plus* other things.

(Of course, most of those "other things" have to do with closing holes like 
__subclasses__, while improving performance by still allowing lots of 
common objects not to be proxied.)


From brett at python.org  Mon Jul 24 09:50:09 2006
From: brett at python.org (Brett Cannon)
Date: Mon, 24 Jul 2006 00:50:09 -0700
Subject: [Python-Dev] new security doc using object-capabilities
In-Reply-To: <5.1.1.6.0.20060724015521.0261d160@sparrow.telecommunity.com>
References: <bbaeab100607191535p543cb0ddj66410cf985dd9b77@mail.gmail.com>
	<20060722114615.GB13782@code0.codespeak.net>
	<44C2CE78.9020802@blueyonder.co.uk>
	<20060723070811.GD13782@code0.codespeak.net>
	<5.1.1.6.0.20060723164246.04144298@sparrow.telecommunity.com>
	<44C3F31D.3070201@blueyonder.co.uk>
	<5.1.1.6.0.20060723194448.0205fa50@sparrow.telecommunity.com>
	<5.1.1.6.0.20060724015521.0261d160@sparrow.telecommunity.com>
Message-ID: <bbaeab100607240050h142e6bbcle137ff393f454684@mail.gmail.com>

On 7/23/06, Phillip J. Eby <pje at telecommunity.com> wrote:
>
> At 09:27 PM 7/23/2006 -0700, Brett Cannon wrote:
>
> When I say "name checker" I mean the Zope type that allows you to specify
> a
> list of names that are allowed for a given object.  This allowing is not
> based on identity or code signing or anything like that.  It's just a list
> of attribute names: i.e. a capability mask over an existing object.
>
> When you create a proxy using this name mask, that proxy becomes a
> capability that allows access to the given names on the underlying object.


OK, then using the term "namechecker" through me off.

>I like the fundemental design difference of object-capabilities
>
> It's not a difference at all, let alone a fundamental one.  Zope just
> happens to allow other kinds of security checking *in addition to*
> capabilities, if you want them.  However, most of its more basic
> encapsulation features are 100% capability based.



> Meanwhile, if you want to implement an object-capability system, you will
> need something that is basically a mask, to allow one piece of code to
> create capabilities that can be given to another.  What you end up with
> for
> doing that is going to look almost exactly like a Zope proxy plus a Zope
> name checker.



I hate to harp on this point, but there seems to be a trend that when
> people have capabilities on their mind, they tend to look at Zope and
> dismiss it as not being capability-based, when in fact Zope's approach is
> capabilities *plus* other things.


Well, Jim said that Zope proxies didn't conform to the strict definition of
object-capabilities the last time this all came about:
http://mail.python.org/pipermail/python-dev/2003-March/033884.html and
http://mail.python.org/pipermail/python-dev/2003-March/033915.html .  He
said they *could* be made to be what object-capabilities is defined as, but
they were not currently structured that way.  Those comments are one of the
reasons I never considered thinking of Zope proxies as a object-capabilities
system.

(Of course, most of those "other things" have to do with closing holes like
> __subclasses__, while improving performance by still allowing lots of
> common objects not to be proxied.)
>
>

OK, then I need something clarified.  If you read
http://www.zope.org/Wikis/DevSite/Projects/ComponentArchitecture/TransitionToSecurityProxies,
it talks about creating the proxies.  I get they restrict attribute
access
and wrap all returned objects in proxies themselves (unless they are
considered safe).  But to judge whether an attribute should be returned, it
checks the security context.  It also mentions how access to the security
policy must be available so that proper security checks can be done to
either grant or deny access.

So what I want to know is if this security context is this global thing that
proxies access every time to check whether something is allowed or not.  Or
is it a per-object specification?  And what is the security domain for Zope
proxies; objects, interpreter, running Python program, what?

-Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20060724/133649ce/attachment-0001.html 

From nmm1 at cus.cam.ac.uk  Mon Jul 24 11:46:52 2006
From: nmm1 at cus.cam.ac.uk (Nick Maclaren)
Date: Mon, 24 Jul 2006 10:46:52 +0100
Subject: [Python-Dev] Strategy for converting the decimal module to C
Message-ID: <E1G4x1Y-0005Vy-Js@virgo.cus.cam.ac.uk>

James Y Knight <foom at fuhm.net> wrote:
> 
> > To cut a long story short, it is impractical for a language run-time
> > system to call user-defined handlers with any degree of reliability
> > unless the compiled code and run-time interoperate carefully - I have
> > been there and done that many times, but few people still working  
> > have.
> > On architectures with out-of-order execution (and interrupts), you
> > have to assume that an interrupt may occur anywhere, even when the
> > code does not use the relevant facility.  Floating-point overflow
> > in the middle of a list insertion?  That's to be expected.
> 
> While this _is_ a real problem, is it _not_ a general problem as you  
> are describing it. Processors are perfectly capable of generating  
> precise interrupts, and the inability to do so has nothing to do with  
> the out-of-order execution, etc. Almost all interrupts are precise.  

I am sorry, but this is almost totally wrong, though I agree that you
will get that impression upon reading the architecture books unless
you are very deeply into that area.

Let's skip the hardware issues, as they aren't what I am talking about
(though see [*]).  I am referring to the interaction between the
compiled code, deep library functions and run-time interrupt handler.

It is almost universal for some deep library functions and common for
compiled code to leave data structures inconsistent in a short window
that "cannot possibly fail" - indeed, most system interfaces do this
around system calls.  If an interrupt occurs then, the run-time system
will receive control with those data structures in a state where they
must not be accessed.  And it is fairly common for such data structures
to include ones critical to the functioning of the run-time system.

Now, it IS possible to write run-time systems that are safe against
this, and still allow asynchronous interrupts, but I am one of three
people in the world that I know have done it in the past two decades.
There may be as many as six, but I doubt more, and I know of no such
implementation on any Unix or Microsoft system.  It is even possible
to do this for compiled code, but that is where the coordination between
the compiler and run-time system comes in.

> The only interesting one which is not, on x86 processors, is the x87  
> floating point exception, ...

Er, no.  Try a machine-check in a TLB miss handler.  But it is all
pretty irrelevant, as the problem arises with asychronous exceptions
(e.g. timer interrupts, signals from other processes), anyway.

> Also, looking forward, the "simd" floating point instructions (ie mmx/ 
> sse/sse2/sse3) _do_ ...

The critical problems with the x87 floating-point exception were
resolved in the 80386.


[*]  Whether or not it is a fundamental problem, it is very much
a general problem at present, and it will become more so as more CPUs
implement micro-threading.  For why it is tied up with out-of-order
execution etc., consider a system with 100 operations flying, of which
10 are memory accesses, and then consider what happens when you have
combinations of floating-point exceptions, TLB misses, machine-checks
(e.g. ECC problems on memory) and device/timer interrupts.  Once you
add user-defined handlers into that mix, you either start exposing
that mess to the program or have to implement them by stopping the
CPU, unwinding the pipeline, and rerunning in very, very serial mode
until the handler is called.  Look at IA64 ....


Regards,
Nick Maclaren,
University of Cambridge Computing Service,
New Museums Site, Pembroke Street, Cambridge CB2 3QH, England.
Email:  nmm1 at cam.ac.uk
Tel.:  +44 1223 334761    Fax:  +44 1223 334679

From ncoghlan at gmail.com  Mon Jul 24 13:21:22 2006
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 24 Jul 2006 21:21:22 +1000
Subject: [Python-Dev] new security doc using object-capabilities
In-Reply-To: <bbaeab100607231300w187dfa8dx87ef0a94935e56ce@mail.gmail.com>
References: <bbaeab100607191535p543cb0ddj66410cf985dd9b77@mail.gmail.com>	<20060722114615.GB13782@code0.codespeak.net>	<44C2CE78.9020802@blueyonder.co.uk>	<20060723070811.GD13782@code0.codespeak.net>
	<bbaeab100607231300w187dfa8dx87ef0a94935e56ce@mail.gmail.com>
Message-ID: <44C4AD32.20309@gmail.com>

Brett Cannon wrote:
> On 7/23/06, *Armin Rigo* <arigo at tunes.org <mailto:arigo at tunes.org>> wrote:
>     Also, I hate to sound self-centered, but I should point out somewhere
>     that PyPy was started by people who no longer wanted to maintain a fork
>     of CPython, and preferred to work on building CPython-like variants
>     automatically.  Many of the security features you list would be quite
>     easier to implement and maintain in PyPy than CPython -- also from a
>     security perspective: it is easier to be sure that some protection is
>     complete, and remains complete over time, if it is systematically
>     generated instead of hand-patched in a dozen places.
> 
> 
> It doesn't sound self-centered.  =)  Problem is that my knowledge base 
> is obviously all in CPython so my startup costs are much lower than if I 
> tried this in PyPy.  Plus there is the point of embedding this into 
> Firefox (possibly) eventually.  Does PyPy support embedding yet at the C 
> level?

Another rationale for basing the work on CPython is that it should be possible 
to implement the resulting security model regardless of the implementation 
language used for the interpreter core (C/Python, Java/Python, C#/Python, 
RPython/Python). If you can figure out how to do it in C, it should be 
feasible to do it in the others.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
             http://www.boredomandlaziness.org

From arigo at tunes.org  Mon Jul 24 14:56:40 2006
From: arigo at tunes.org (Armin Rigo)
Date: Mon, 24 Jul 2006 14:56:40 +0200
Subject: [Python-Dev] Document performance requirements?
In-Reply-To: <008201c6ae5c$36b4ca40$d503030a@trilan>
References: <e9qbtf$kk6$1@sea.gmane.org>
	<20060722111907.GA13782@code0.codespeak.net>
	<008201c6ae5c$36b4ca40$d503030a@trilan>
Message-ID: <20060724125640.GA10183@code0.codespeak.net>

Hi Giovanni,

On Sun, Jul 23, 2006 at 03:30:50PM +0200, Giovanni Bajo wrote:
> I'm not sure big-O tells the whole truth. For instance, do we want to allow
> an implementation to use a hash table as underlying type for a list? It
> would match big-O requirements, but would still be slower than a plain array
> because of higher overhead of implementation (higher constant factor).

A big-O difference can make the difference between a program that takes
0.5 seconds or 2 hours to run.  This is more important than a constant
factor difference, which different implementations are bound to exhibit
anyway.

> And if this is allowed, I would like to find in CPython tutorials and
> documentations a simple statement like: "to implement the list and match its
> requirements, CPython choose a simple array as underlying data structure".

Yes, the big-O notes don't have to be too technical: the docs should
tell people to think about Python lists as simple arrays, and the O
requirements follow naturally.


A bientot,

Armin

From amk at amk.ca  Mon Jul 24 16:08:19 2006
From: amk at amk.ca (A.M. Kuchling)
Date: Mon, 24 Jul 2006 10:08:19 -0400
Subject: [Python-Dev] Python sprint in Arlington July 29/30
Message-ID: <20060724140819.GA9507@rogue.amk.ca>

The CanDo developers are sprinting for three days starting on this
coming Friday, so there's space available for a Python sprint.  I'll
try to attend at least on Saturday (Sunday may not be possible for
me).  Does anyone want to come and work on Python stuff?  If yes,
please add your name to <http://wiki.python.org/moin/ArlingtonSprint>,
which also has directions.

Given that Python will still be in a code freeze, we'll have to be
careful about code changes.  I'm probably going to work on
documentation issues.

--amk

From pje at telecommunity.com  Mon Jul 24 18:32:32 2006
From: pje at telecommunity.com (Phillip J. Eby)
Date: Mon, 24 Jul 2006 12:32:32 -0400
Subject: [Python-Dev] new security doc using object-capabilities
In-Reply-To: <bbaeab100607240050h142e6bbcle137ff393f454684@mail.gmail.co
 m>
References: <5.1.1.6.0.20060724015521.0261d160@sparrow.telecommunity.com>
	<bbaeab100607191535p543cb0ddj66410cf985dd9b77@mail.gmail.com>
	<20060722114615.GB13782@code0.codespeak.net>
	<44C2CE78.9020802@blueyonder.co.uk>
	<20060723070811.GD13782@code0.codespeak.net>
	<5.1.1.6.0.20060723164246.04144298@sparrow.telecommunity.com>
	<44C3F31D.3070201@blueyonder.co.uk>
	<5.1.1.6.0.20060723194448.0205fa50@sparrow.telecommunity.com>
	<5.1.1.6.0.20060724015521.0261d160@sparrow.telecommunity.com>
Message-ID: <5.1.1.6.0.20060724110644.02024180@sparrow.telecommunity.com>

At 12:50 AM 7/24/2006 -0700, Brett Cannon wrote:
>OK, then I need something clarified.  If you read 
><http://www.zope.org/Wikis/DevSite/Projects/ComponentArchitecture/TransitionToSecurityProxies>http://www.zope.org/Wikis/DevSite/Projects/ComponentArchitecture/TransitionToSecurityProxies 
>, it talks about creating the proxies.  I get they restrict attribute 
>access and wrap all returned objects in proxies themselves (unless they 
>are considered safe).  But to judge whether an attribute should be 
>returned, it checks the security context.

That depends on the checker.  The proxy implementation delegates all access 
decisions to a "checker" object.  Some checkers check permissions, but a 
NamesChecker just checks a statically-defined list of names.


>   It also mentions how access to the security policy must be available so 
> that proper security checks can be done to either grant or deny access.
>
>So what I want to know is if this security context is this global thing 
>that proxies access every time to check whether something is allowed or not.

Proxies don't do that; checkers do.  The default Checker implementation 
doesn't even look at a security context if a name is declared public (i.e., 
it's a NamesChecker).  Look at the zope.security.checker module for details.

IOW, to make it a pure capabilities system, you would only *delete* code, 
not add any, as far as I can tell.


>   Or is it a per-object specification?

Each proxy can have its own checker, but an individual checker instance can 
be shared between proxies.


>   And what is the security domain for Zope proxies; objects, interpreter, 
> running Python program, what?

There are restricted eval and exec operations to run restricted code.

The primary language limitations imposed are the lack of eval/exec by the 
restricted code, and lack of support for raise and 
try/except.  Implementing these would require additional compiler hacking 
to add code to ensure that e.g. tracebacks get wrapped.


From eswierk at arastra.com  Mon Jul 24 19:20:34 2006
From: eswierk at arastra.com (Ed Swierk)
Date: Mon, 24 Jul 2006 17:20:34 +0000 (UTC)
Subject: [Python-Dev] setup.py and cross-compiling
Message-ID: <loom.20060724T190947-141@post.gmane.org>

I'm cross-compiling Python using uClibc buildroot, which installs target include
files and libraries to a directory like ~/toolchain rather than /usr.  I
couldn't figure out any way to convincing the top-level python/setup.py to look
in ~/toolchain instead of /usr when detecting what modules to build.  

I decided to hack up setup.py so that an optional root directory (passed via an
environment variable) is prepended to all the hardcoded paths like
"/usr/include", "/lib", "/lib64", and so on.

Am I missing an easier solution to this problem?  I'm happy to post my patch to
setup.py if anyone is interested.

--Ed



From unknown_kev_cat at hotmail.com  Mon Jul 24 19:35:36 2006
From: unknown_kev_cat at hotmail.com (Joe Smith)
Date: Mon, 24 Jul 2006 13:35:36 -0400
Subject: [Python-Dev] Python 2.4, VS 2005 & Profile Guided Optmization
References: <AB1BB4CF81B9214682079EB55A7125EDEFA707@mapibe05.exchange.xchg><034401c6ae98$6a0da920$d503030a@trilan><646666C6-A1DB-4D2C-8054-FA9F7E2129E4@fuhm.net><ea1hdl$k0v$1@sea.gmane.org>
	<44C451CB.3060607@canterbury.ac.nz>
Message-ID: <ea30d8$64o$1@sea.gmane.org>


"Greg Ewing" <greg.ewing at canterbury.ac.nz> wrote in message 
news:44C451CB.3060607 at canterbury.ac.nz...
> Joe Smith wrote:
>
>> Microsoft as a general rule, does not go after people distributing 
>> products
>> that Microsoft has labeled
>> free, even after Microsoft no longer distributes that product.
>
> But if the licence agreement technically forbids
> redistribution, it doesn't seem like a good idea
> to rely on Microsoft turning a blind eye to that.

I agree, although it seems very unlikely Microsoft would
bother as they are busy enough the the pirating of the for-cost software.

I was a little surprised that they did not allow verbatim duplicates of that 
software,
expecially as it has the feel of freeware, or even shareware (the commecial 
versions
have additional features). I would actually not be surprised if somebody 
could
convince Microsoft to allow them to distribute these versions even after 
Microsoft
no longer supports them. That would be the ideal situation.



From david.nospam.hopwood at blueyonder.co.uk  Mon Jul 24 18:22:39 2006
From: david.nospam.hopwood at blueyonder.co.uk (David Hopwood)
Date: Mon, 24 Jul 2006 17:22:39 +0100
Subject: [Python-Dev] new security doc using object-capabilities
In-Reply-To: <5.1.1.6.0.20060723194448.0205fa50@sparrow.telecommunity.com>
References: <5.1.1.6.0.20060723164246.04144298@sparrow.telecommunity.com>
	<20060723070811.GD13782@code0.codespeak.net>
	<bbaeab100607191535p543cb0ddj66410cf985dd9b77@mail.gmail.com>
	<20060722114615.GB13782@code0.codespeak.net>
	<44C2CE78.9020802@blueyonder.co.uk>
	<20060723070811.GD13782@code0.codespeak.net>
	<5.1.1.6.0.20060723164246.04144298@sparrow.telecommunity.com>
	<5.1.1.6.0.20060723194448.0205fa50@sparrow.telecommunity.com>
Message-ID: <44C4F3CF.1060104@blueyonder.co.uk>

Phillip J. Eby wrote:
> At 11:07 PM 7/23/2006 +0100, David Hopwood wrote:
>> Phillip J. Eby wrote:
>> > At 01:00 PM 7/23/2006 -0700, Brett Cannon wrote:
>> >
>> >> I obviously don't want to change the feel of Python, but if I have to
>> >> remove the constructor for code objects to prevent evil bytecode or
>> >> __subclasses__() from object to prevent poking around stuff, then so be
>> >> it.  For this project, security is [trumping] backwards-compatibility when
>> >> the latter is impossible in order to have the former.  I will obviously
>> >> try to minimize it, but something that works at such a basic level of the
>> >> language is just going to require some changes for it to work.
>> >
>> > Zope 3's sandboxing machinery manages to handle securing these things
>> > without any language changes.  So, declaring it "impossible" to manage
>> > without backward compatibility seems inappropriate, or at least
>> > incorrect.
>>
>> ... if Zope's sandboxing is secure. I haven't done a security review
>> of it, but your argument assumes that it is.
> 
> What argument is that?

You said "Zope 3's sandboxing machinery manages to handle securing these
things without any language changes." This assertion assumes that Zope 3's
sandboxing machinery is secure.

> I'm merely suggesting that coming up with a
> completely new way to secure Python without a serious consideration of
> existing practical prior art (with many years' deployment experience on
> the public internet!) seems ill-advised with respect to achieving
> practical goals.
> 
> Brett's securing_python.txt don't refer to or cite Zope in any way,

This is indeed an omission that should be corrected, in order to explain
why this project is not using Zope or following Zope's approach, and what
the differences are. (I've explained some of them below.)

> but rather relies on broad and unsupported assertions about what can or
> can't be done with Python.  I hope he isn't doing the same in his
> thesis, as this is rather like writing about one's new theory of how to
> have a worldwide ball-kicking contest without making any reference as to
> how one's theory compares with the World Cup.
> 
> I'm not saying Zope is better or worse.  I'm simply saying that in a
> business context, a failure to compare and contrast a proposed "build"
> solution to show how it would be better than a well-established
> available "buy" solution would be called something like "lack of due
> diligence".  I think in the academic context it might be called
> something like "failure to cite", but the general idea is the same,
> i.e., not doing your homework.  :)
> 
> In other words, if the solution being proposed is better than what Zope
> does, the appropriate thing in business is to show the reasons why, and
> the appropriate thing in science is to state a hypothesis regarding the
> differences, and then perform an experiment to either prove or disprove it.

I completely agree with this.

>> In any case, Zope's sandboxing is not capability-based.
> 
> You're right: you haven't done a review of it.  :)

I haven't done a detailed security review. However, I wouldn't have commented
on it without knowing what its basic approach is.

From
<http://www.zope.org/Wikis/DevSite/Projects/ComponentArchitecture/TransitionToSecurityProxies>:

# When an object is passed to untrusted code, it is wrapped in a security
# proxy unless it is already wrapped. Security proxies mediate all accesses
# to the wrapped object. Operations on security proxies return security
# proxies as well. Security proxies passed from untrusted code to trusted
# code remain wrapped, so untrusted code can't trick trusted code into
# performing operations that the untrusted code could not perform.

Restricting the actions of "trusted" code when called by "untrusted" code
is precisely what a capability system does *not* do. Indeed, capability
systems have no binary distinction between "trusted" and "untrusted" code
(outside the system TCB, which is as small as possible); the only security
distinction between protection domains is in what capabilities they hold.

The main reason why capability systems do not have any such restriction is
that it reduces the system's ability to support fine-grained delegation of
authority. We want to be able to grant an object just the authority it needs
for any particular task. For that to work, code that is otherwise untrusted
must be able to use any capability it is given, even if it is a very powerful
capability, and therefore must be able to call into more-trusted code without
restriction.

To characterise this as "tricking" the more-trusted code is a misconception:
if it is not intended that a particular protection domain should have some
authority, then it should not be given a capability for that authority in
the first place. Capability-based security design emphasises preventing
protection domains from gaining unintended capabilities; it is not about
restricting their use once granted (except to the extent needed to support
revocation, which is typically done via higher-level patterns rather than in
the basic access control mechanisms).

The kind of restrictions that Zope enforces would be a significant impediment
to this approach:

# With care, trusted code can explicitly unwrap security proxies and gain
# additional access. In particular, security proxies cannot be stored in the
# Zope object database. If an object wrapped by a security proxy is to be
# stored in another object, the security proxy must be removed.

This is a severe limitation to expressiveness; it effectively means that
references to secure objects are not first-class. In a capability system,
they must be first-class.

> If you had, you'd know that one proxy plus one namechecker equals one
> capability.

No, because Zope enforces some restrictions that are not desired in a
capability system.

(It is absolutely mistaken to think that a system that enforces more restrictions
must be more secure, in case anyone is under that impression.)

I do not know whether the easiest way to get to a capability-secure version
of Python would be by changing the restrictions enforced by Zope, or by adding
restrictions to CPython (or another Python implementation). My point is simply
that the security model of the Zope sandboxing system as it stands, is quite
different from that of a capability system.

[...]
> Now, if Brett believes that changing the Python language is a *better*
> way to implement capabilities than using proxies to implement them, then
> great.  His paper should explain why, and (presumably) include
> experimental results to show that they're either better or worse than
> Zope's approach based on some criteria.

Requiring experimental results is not appropriate for a proposal that has
not been implemented. It is also not a paper; it's an informal, early-stage
design outline.

-- 
David Hopwood <david.nospam.hopwood at blueyonder.co.uk>



From unknown_kev_cat at hotmail.com  Mon Jul 24 22:08:43 2006
From: unknown_kev_cat at hotmail.com (Joe Smith)
Date: Mon, 24 Jul 2006 16:08:43 -0400
Subject: [Python-Dev] Python 2.4, VS 2005 & Profile Guided Optmization
References: <AB1BB4CF81B9214682079EB55A7125EDEFA707@mapibe05.exchange.xchg><50862ebd0607232003gc800a48r3853bd773d3a7f21@mail.gmail.com>
	<ea1i1e$l5c$1@sea.gmane.org>
Message-ID: <ea39ca$7af$1@sea.gmane.org>


"Joe Smith" <unknown_kev_cat at hotmail.com> wrote in message
news:ea1i1e$l5c$1 at sea.gmane.org...
>
> "Neil Hodgson" <nyamatongwe at gmail.com> wrote in message
> news:50862ebd0607232003gc800a48r3853bd773d3a7f21 at mail.gmail.com...
>> Trent Nelson:
>>
>>> I ended up playing around with Profile Guided Optimization, running
>>> ``python.exe pystones.py'' to collect call-graph data after
>>> python.exe/Python24.dll had been instrumented, then recompiling with the
>>> optimizations fed back in.
>>
>>   It'd be an idea to build a larger body of Python code to run the
>> profiling pass on so it doesn't just optimize the sort of code in
>> pystone which is not very representative. Could run the test suite as
>> it would have good coverage but would hit exceptional cases too
>> heavily. Other compilers (Intel?) support profile directed
>> optimization so would also benefit from such a body of code.
>>
>
> GCC suppost profiling optimized code.
Err... That was supposed to say "supports profile-optimized code".
The rest of the message was indeed about Profile Guided Optimization in gcc.



From martin at v.loewis.de  Mon Jul 24 23:25:58 2006
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Mon, 24 Jul 2006 23:25:58 +0200
Subject: [Python-Dev] setup.py and cross-compiling
In-Reply-To: <loom.20060724T190947-141@post.gmane.org>
References: <loom.20060724T190947-141@post.gmane.org>
Message-ID: <44C53AE6.7040105@v.loewis.de>

Ed Swierk wrote:
> I decided to hack up setup.py so that an optional root directory (passed via an
> environment variable) is prepended to all the hardcoded paths like
> "/usr/include", "/lib", "/lib64", and so on.

I doubt this solves the problem. Distutils just doesn't support
cross-compilation, period.

The main problem is that it is the host python that runs setup.py,
not the target python. Various parts of distutils assume that the
information the interpreter provides is correct, yet it is not
in a cross-compilation case.

I'm uncertain how to solve this; using a target Makefile and pyconfig.h
instead of the installed one might be a starting point. One should then
review distutils to find out where it accesses host information when
target information is required, and then find a way to come up with
that information. If no information is available, then distutils should
fail instead of returning incorrect information.

Regards,
Martin

From scott+python-dev at scottdial.com  Tue Jul 25 00:43:57 2006
From: scott+python-dev at scottdial.com (Scott Dial)
Date: Mon, 24 Jul 2006 18:43:57 -0400
Subject: [Python-Dev] Document performance requirements?
In-Reply-To: <20060724125640.GA10183@code0.codespeak.net>
References: <e9qbtf$kk6$1@sea.gmane.org>	<20060722111907.GA13782@code0.codespeak.net>	<008201c6ae5c$36b4ca40$d503030a@trilan>
	<20060724125640.GA10183@code0.codespeak.net>
Message-ID: <44C54D2D.3070005@scottdial.com>

Between the two of you, I think you have made the case that the language 
specification is better to not include such details. As you both note, 
it is difficult to capture the essence of what is desired from the 
performance of the implementation. To tag on other version, what about 
Big-O space concerns with things like list.sort. I'm sure there are 
other things to add as well.

It seems reasonable to me that everyone has the same interests in mind 
when they write a program. Make it good, make it fast, make it small, 
etc. These sort of details should work themselves out if they are 
actually important. All of these algorithms should be treated as 
implementation accidents.

Having the information about CPython's implementation in the docs would 
be good. And go most of the way towards having everyone on the same page.

-- 
Scott Dial
scott at scottdial.com
scodial at indiana.edu

From martin at v.loewis.de  Tue Jul 25 01:19:25 2006
From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=)
Date: Tue, 25 Jul 2006 01:19:25 +0200
Subject: [Python-Dev] setup.py and cross-compiling
In-Reply-To: <c1bf1cf0607241519j157a7050l29998cecc4953485@mail.gmail.com>
References: <loom.20060724T190947-141@post.gmane.org>	
	<44C53AE6.7040105@v.loewis.de>
	<c1bf1cf0607241519j157a7050l29998cecc4953485@mail.gmail.com>
Message-ID: <44C5557D.4010004@v.loewis.de>

Ed Swierk wrote:
> Well, it seems buildroot solves this main problem by building another
> version of python and pygen that run on the build machine, and hacks
> the Makefile to run setup.py with these instead of whatever happens to
> be sitting in /usr/bin.

If you think its useful, please submit a patch to sf.net/projects/python
It would be good if you could get other people that attempt to
cross-build Python to comment. A detailed howto might help to get them
started with your code.

Regards,
Martin

From greg.ewing at canterbury.ac.nz  Tue Jul 25 02:04:44 2006
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Tue, 25 Jul 2006 12:04:44 +1200
Subject: [Python-Dev] new security doc using object-capabilities
In-Reply-To: <5.1.1.6.0.20060724015521.0261d160@sparrow.telecommunity.com>
References: <5.1.1.6.0.20060723194448.0205fa50@sparrow.telecommunity.com>
	<bbaeab100607191535p543cb0ddj66410cf985dd9b77@mail.gmail.com>
	<20060722114615.GB13782@code0.codespeak.net>
	<44C2CE78.9020802@blueyonder.co.uk>
	<20060723070811.GD13782@code0.codespeak.net>
	<5.1.1.6.0.20060723164246.04144298@sparrow.telecommunity.com>
	<44C3F31D.3070201@blueyonder.co.uk>
	<5.1.1.6.0.20060723194448.0205fa50@sparrow.telecommunity.com>
	<5.1.1.6.0.20060724015521.0261d160@sparrow.telecommunity.com>
Message-ID: <44C5601C.4070605@canterbury.ac.nz>

Phillip J. Eby wrote:

> When I say "name checker" I mean the Zope type that allows you to specify a 
> list of names that are allowed for a given object.  This allowing is not 
> based on identity or code signing or anything like that.  It's just a list 
> of attribute names: i.e. a capability mask over an existing object.

But this is backwards from what a true object-capability
system should be like if it's properly designed. Instead
of starting with too-powerful objects and trying to
hide some of their powers, the different powers should
be separated into different objects in the first place.

It sounds to me like Zope is using the approach it's
using because it's having to work with Python as it
currently is, not because its approach is the best one.

--
Greg

From greg.ewing at canterbury.ac.nz  Tue Jul 25 02:14:34 2006
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Tue, 25 Jul 2006 12:14:34 +1200
Subject: [Python-Dev] Strategy for converting the decimal module to C
In-Reply-To: <E1G4x1Y-0005Vy-Js@virgo.cus.cam.ac.uk>
References: <E1G4x1Y-0005Vy-Js@virgo.cus.cam.ac.uk>
Message-ID: <44C5626A.8080306@canterbury.ac.nz>

Nick Maclaren wrote:

> Er, no.  Try a machine-check in a TLB miss handler.  But it is all
> pretty irrelevant, as the problem arises with asychronous exceptions
> (e.g. timer interrupts, signals from other processes), anyway.

But we weren't talking about asynchronous exceptions,
we were talking about floating point exceptions. Unless
your TLB miss handler uses floating point arithmethic,
there's no way it can get interrupted by one. (And if
it does use floating point arithmetic in a way that
can cause an exception, you'd better write it to deal
with that!)

--
Greg

From pje at telecommunity.com  Tue Jul 25 03:19:33 2006
From: pje at telecommunity.com (Phillip J. Eby)
Date: Mon, 24 Jul 2006 21:19:33 -0400
Subject: [Python-Dev] new security doc using object-capabilities
In-Reply-To: <44C5601C.4070605@canterbury.ac.nz>
References: <5.1.1.6.0.20060724015521.0261d160@sparrow.telecommunity.com>
	<5.1.1.6.0.20060723194448.0205fa50@sparrow.telecommunity.com>
	<bbaeab100607191535p543cb0ddj66410cf985dd9b77@mail.gmail.com>
	<20060722114615.GB13782@code0.codespeak.net>
	<44C2CE78.9020802@blueyonder.co.uk>
	<20060723070811.GD13782@code0.codespeak.net>
	<5.1.1.6.0.20060723164246.04144298@sparrow.telecommunity.com>
	<44C3F31D.3070201@blueyonder.co.uk>
	<5.1.1.6.0.20060723194448.0205fa50@sparrow.telecommunity.com>
	<5.1.1.6.0.20060724015521.0261d160@sparrow.telecommunity.com>
Message-ID: <5.1.1.6.0.20060724211442.0262c008@sparrow.telecommunity.com>

At 12:04 PM 7/25/2006 +1200, Greg Ewing wrote:
>Phillip J. Eby wrote:
>
> > When I say "name checker" I mean the Zope type that allows you to 
> specify a
> > list of names that are allowed for a given object.  This allowing is not
> > based on identity or code signing or anything like that.  It's just a list
> > of attribute names: i.e. a capability mask over an existing object.
>
>But this is backwards from what a true object-capability
>system should be like if it's properly designed. Instead
>of starting with too-powerful objects and trying to
>hide some of their powers, the different powers should
>be separated into different objects in the first place.

And what about code that needs to pass on a subset of a capability?  You 
need the ability to create such capability-restricted subsets anyway, no 
matter how "pure" a system you start with.

And being able to create capability masks for existing objects means you 
don't have to redesign every piece of code ever written for Python to make 
it secure.


>It sounds to me like Zope is using the approach it's
>using because it's having to work with Python as it
>currently is, not because its approach is the best one.


Well, that depends a lot on how you define "best".  Practicality beats 
purity, doesn't it?  ;)


From david.nospam.hopwood at blueyonder.co.uk  Tue Jul 25 02:36:48 2006
From: david.nospam.hopwood at blueyonder.co.uk (David Hopwood)
Date: Tue, 25 Jul 2006 01:36:48 +0100
Subject: [Python-Dev] new security doc using object-capabilities
In-Reply-To: <5.1.1.6.0.20060724211442.0262c008@sparrow.telecommunity.com>
References: <5.1.1.6.0.20060724015521.0261d160@sparrow.telecommunity.com>	<5.1.1.6.0.20060723194448.0205fa50@sparrow.telecommunity.com>	<bbaeab100607191535p543cb0ddj66410cf985dd9b77@mail.gmail.com>	<20060722114615.GB13782@code0.codespeak.net>	<44C2CE78.9020802@blueyonder.co.uk>	<20060723070811.GD13782@code0.codespeak.net>	<5.1.1.6.0.20060723164246.04144298@sparrow.telecommunity.com>	<44C3F31D.3070201@blueyonder.co.uk>	<5.1.1.6.0.20060723194448.0205fa50@sparrow.telecommunity.com>	<5.1.1.6.0.20060724015521.0261d160@sparrow.telecommunity.com>
	<5.1.1.6.0.20060724211442.0262c008@sparrow.telecommunity.com>
Message-ID: <44C567A0.2080906@blueyonder.co.uk>

Phillip J. Eby wrote:
> At 12:04 PM 7/25/2006 +1200, Greg Ewing wrote:
>>Phillip J. Eby wrote:
>>
>>>When I say "name checker" I mean the Zope type that allows you to specify
>>>a list of names that are allowed for a given object.  This allowing is not
>>>based on identity or code signing or anything like that.  It's just a list
>>>of attribute names: i.e. a capability mask over an existing object.
>>
>>But this is backwards from what a true object-capability
>>system should be like if it's properly designed. Instead
>>of starting with too-powerful objects and trying to
>>hide some of their powers, the different powers should
>>be separated into different objects in the first place.
> 
> And what about code that needs to pass on a subset of a capability?  You 
> need the ability to create such capability-restricted subsets anyway, no 
> matter how "pure" a system you start with.

That is true, but doing so for every secure object has significant costs.
A simple wrapper is not sufficient, because it would not prevent a wrapped
object from returning a reference to itself, bypassing the wrapper. To solve
this problem you need to use the more complex Membrane pattern, which also
wraps the results of method calls on a wrapped object, for example.

In fact Zope's approach does implement a membrane, but this does not really
dent the argument that Greg Ewing was making. A pure capability system incurs
the complexity and performance costs of wrappers or membranes only in cases
where they are needed, not for every object, and the complexity is only in
user code, not in the system's security kernel.

-- 
David Hopwood <david.nospam.hopwood at blueyonder.co.uk>



From nnorwitz at gmail.com  Tue Jul 25 06:20:40 2006
From: nnorwitz at gmail.com (Neal Norwitz)
Date: Mon, 24 Jul 2006 21:20:40 -0700
Subject: [Python-Dev] outstanding bugs to fix for 2.5
Message-ID: <ee2a432c0607242120o5365d6b4w2a8b200253686ea6@mail.gmail.com>

There are still a bunch of outstanding bugs.  rc1 is about a week away
and it would be great to fix these.  Many of these are also present in
2.4, but it would be nice to squash them in 2.5.  Here's the list from
PEP 356:

        http://python.org/sf/1526585 - SystemError concat long strings (2.4)
        http://python.org/sf/1523610 - PyArg_ParseTupleAndKeywords potential
                                        core dump (2.4)
        http://python.org/sf/1521947 - mystrtol.c fails with gcc 4.1 (2.4?)
                                        test_compile.test_unary_minus
        http://python.org/sf/1519025 - socket timeout crash when receive
                                        signal (2.4)
        http://python.org/sf/1517042 - Fix crashers/gc_inspection.py (2.4)
        http://python.org/sf/1515471 - stringobject (char buffers)
        http://python.org/sf/1513611 - XML: xml.sax.expatreader missing
        http://python.org/sf/1511497 - XML: xml.sax.ParseException issue
        http://python.org/sf/1475523 - gettext.py bug
        http://python.org/sf/1467929 - %-formatting and dicts (2.4)
        http://python.org/sf/1333982 - AST
        http://python.org/sf/1191458 - AST (test_trace issue mentioned below)

It would be great to fix *all* of these.  In this list, at least 3
(4?) can cause segfaults, and #1521947 can cause incorrect results.

n

From nnorwitz at gmail.com  Tue Jul 25 07:50:46 2006
From: nnorwitz at gmail.com (Neal Norwitz)
Date: Mon, 24 Jul 2006 22:50:46 -0700
Subject: [Python-Dev] remaining issues from Klocwork static analysis
Message-ID: <ee2a432c0607242250v5a7ac021p464b68c5c47ec538@mail.gmail.com>

I've fixed most of the problems (or determined they weren't problems)
from all the warnings issued by Klocwork's static analysis tool.  The
following are outstanding issues.

This first group looks like real problems to me:

# 74 Object/funcobject.c:143    Suspicious deref of ptr before NULL check
#169 Modules/threadmodule.c:497 Memory Leak

# 28 Modules/_sre.c:987   Array Index Out of Bounds

    Buffer overflow, array index of 'mark' may be outside the
    bounds. Array 'mark' of size 200 declared at sre.h:77 may use
    index values 0..536870911. Also there are 3 similar errors on
    lines 1006, 1225, 1237.  (Try limiting mark on line 589?)

#174 Modules/unicodedata.c:432   Array Index Out of Bounds

    Buffer overflow, array index of 'decomp_prefix' may be outside the
    bounds. Array 'decomp_prefix' of size 18 declared at
    unicodedata_db.h:529 may use index values 18..255. Also there is one
    similar error on line 433.

# 36 Modules/cPickle.c:3404   Memory Leak

    Memory leak. Dynamic memory stored in 's' allocated through
    function 'pystrndup' at line 3384 is lost at line 3404.

    s should not be freed on line 3407, but earlier.
    PDATA_PUSH can return on error and s will not be freed.

# 61 Modules/_sqlite/cursor.c:599  Null pointer may be dereferenced

    Null pointer 'self->statement' that comes from line 674 may be
    dereferenced by passing argument 1 to function
    'statement_mark_dirty' at line 599.

Most of these seem suspect.  I'm not so sure about them, but I haven't
looked into some at all.  Let me know if you want the details for any
of these or if you can provide an analysis to demonstrate they are
incorrect.

Null pointer may be dereferenced	Python/ast.c:641
Null pointer may be dereferenced	Python/ast.c:656
Ptr will be derefed after it was positively checked for
NULL	Python/compile.c:3020
Null pointer may be passed to function that may dereference
it	Python/compile.c:4459
Array Index Out of Bounds	Modules/_sre.c:987
Array Index Out of Bounds	Object/longobject.c:1787
Array Index Out of Bounds	Object/longobject.c:2475
Array Index Out of Bounds	Python/sysmodule.c:1016

Array Index Out of Bounds	Python/getpath.c:285
Buffer Overflow - Non-null Terminated String	Python/getpath.c:432
Unvalidated User Input Buffer Overflow-Non-Null Terminated
String	Python/getpath.c:431
Unvalidated User Input Buffer Overflow-Non-Null Terminated
String	Python/getpath.c:496
Unvalidated User Input Buffer Overflow-Non-Null Terminated
String	Python/getpath.c:497

Let me know if you want more info about any particular report.  It
would be great to have some help and fix these.

n

From martin at v.loewis.de  Tue Jul 25 07:54:24 2006
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Tue, 25 Jul 2006 07:54:24 +0200
Subject: [Python-Dev] outstanding bugs to fix for 2.5
In-Reply-To: <ee2a432c0607242120o5365d6b4w2a8b200253686ea6@mail.gmail.com>
References: <ee2a432c0607242120o5365d6b4w2a8b200253686ea6@mail.gmail.com>
Message-ID: <44C5B210.6020902@v.loewis.de>

Neal Norwitz wrote:
>         http://python.org/sf/1513611 - XML: xml.sax.expatreader missing
> 
> It would be great to fix *all* of these.  In this list, at least 3
> (4?) can cause segfaults, and #1521947 can cause incorrect results.

IMO, 1513611 should block the release, since it's a regression from 2.4
that may break many SAX applications, and perhaps also DOM applications.

Regards,
Martin

From martin at v.loewis.de  Tue Jul 25 09:40:44 2006
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Tue, 25 Jul 2006 09:40:44 +0200
Subject: [Python-Dev] remaining issues from Klocwork static analysis
In-Reply-To: <ee2a432c0607242250v5a7ac021p464b68c5c47ec538@mail.gmail.com>
References: <ee2a432c0607242250v5a7ac021p464b68c5c47ec538@mail.gmail.com>
Message-ID: <44C5CAFC.80004@v.loewis.de>

Neal Norwitz wrote:
> # 74 Object/funcobject.c:143    Suspicious deref of ptr before NULL check

Not quite sure what it is complaining about, but

        else if (PyTuple_Check(closure)) {
                Py_XINCREF(closure);
        }

looks indeed suspicious: Why do we check for NULL (XINCREF) when
we know closure can't be NULL (Tuple_Check). Drop the X, and see
if the warning goes away

> #169 Modules/threadmodule.c:497 Memory Leak

Does it say what memory is leaking? Perhaps it complains about
boot not being released if ident is not -1, however, in that case,
t_bootstrap will release the memory.

> # 28 Modules/_sre.c:987   Array Index Out of Bounds
> 
>     Buffer overflow, array index of 'mark' may be outside the
>     bounds. Array 'mark' of size 200 declared at sre.h:77 may use
>     index values 0..536870911. Also there are 3 similar errors on
>     lines 1006, 1225, 1237.  (Try limiting mark on line 589?)

ISTM that SRE has a limit of 100 MARK opcodes, meaning a maximum
of 100 groups per expression (so you need 200 mark pointers).
This can't overrun as sre_compile refuses to compile expressions
with more groups.

Of course, a malicious application could craft the opcodes itself
(bypassing sre_compile), in which case you could get a buffer
overrun.

The right solution is to have a dynamic marks array.

> #174 Modules/unicodedata.c:432   Array Index Out of Bounds
> 
>     Buffer overflow, array index of 'decomp_prefix' may be outside the
>     bounds. Array 'decomp_prefix' of size 18 declared at
>     unicodedata_db.h:529 may use index values 18..255. Also there is one
>     similar error on line 433.

This limit is enforced by Tools/unicode/makeunicodedata.py. There are
only 18 decomposition prefixes at the moment, yet we use 8 bits for
the decomposition prefix (makeunicodedata checks that prefix < 256)

Looking at the code, I now wonder why decomp_data can't be
"unsigned short", instead of "unsigned int" (the upper byte is the
decomposition length, and it can't be larger than 256, either).

> # 36 Modules/cPickle.c:3404   Memory Leak
> 
>     Memory leak. Dynamic memory stored in 's' allocated through
>     function 'pystrndup' at line 3384 is lost at line 3404.
> 
>     s should not be freed on line 3407, but earlier.
>     PDATA_PUSH can return on error and s will not be freed.

Correct. We should not use macros with embedded return statements.

> # 61 Modules/_sqlite/cursor.c:599  Null pointer may be dereferenced
> 
>     Null pointer 'self->statement' that comes from line 674 may be
>     dereferenced by passing argument 1 to function
>     'statement_mark_dirty' at line 599.

Looks like a problem. Maybe a break is missing after line 674?

Regards,
Martin

From nmm1 at cus.cam.ac.uk  Tue Jul 25 11:06:08 2006
From: nmm1 at cus.cam.ac.uk (Nick Maclaren)
Date: Tue, 25 Jul 2006 10:06:08 +0100
Subject: [Python-Dev] Strategy for converting the decimal module to C
Message-ID: <E1G5Irg-0003Zi-9b@libra.cus.cam.ac.uk>

Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
> 
> But we weren't talking about asynchronous exceptions,
> we were talking about floating point exceptions. Unless
> your TLB miss handler uses floating point arithmethic,
> there's no way it can get interrupted by one. (And if
> it does use floating point arithmetic in a way that
> can cause an exception, you'd better write it to deal
> with that!)

I am really not getting my message across, am I?

Yes, that is true - as far as it goes.  The trouble is that designing
systems based on assuming that IS true as far as it goes means that they
don't work when it goes further.  And it does.  Here are a FEW of the
many examples of where the simplistic model is likely to fail in an
x86 context:

The compiled code has made a data structure temporarily inconsistent
because the operation is safe (say, list insertion), and then gets an
asynchronous interrupt (e.g. SIGINT).  The SIGINT handler does some
operation (e.g. I/O) that implicitly uses floating-point, which then
interrupts.

The x86 architecture is extended to include out-of-order floating-point
as it had in the past, many systems have today, and is very likely to
happen in the future.  It is one of the standard ways to get better
performance, after all, and is on the increase.

The x86 architecture is extended to support micro-threading.  I have
not been told by Intel or AMD that either have such plans, but I have
very good reason to believe that both have such projects.  IBM and Sun
certainly do, though I don't know if IBM's is/are relevant.


Regards,
Nick Maclaren,
University of Cambridge Computing Service,
New Museums Site, Pembroke Street, Cambridge CB2 3QH, England.
Email:  nmm1 at cam.ac.uk
Tel.:  +44 1223 334761    Fax:  +44 1223 334679

From g.brandl at gmx.net  Tue Jul 25 11:57:23 2006
From: g.brandl at gmx.net (Georg Brandl)
Date: Tue, 25 Jul 2006 11:57:23 +0200
Subject: [Python-Dev] remaining issues from Klocwork static analysis
In-Reply-To: <44C5CAFC.80004@v.loewis.de>
References: <ee2a432c0607242250v5a7ac021p464b68c5c47ec538@mail.gmail.com>
	<44C5CAFC.80004@v.loewis.de>
Message-ID: <ea4pro$ere$1@sea.gmane.org>

Martin v. L?wis wrote:
> Neal Norwitz wrote:
>> # 74 Object/funcobject.c:143    Suspicious deref of ptr before NULL check
> 
> Not quite sure what it is complaining about, but
> 
>         else if (PyTuple_Check(closure)) {
>                 Py_XINCREF(closure);
>         }
> 
> looks indeed suspicious: Why do we check for NULL (XINCREF) when
> we know closure can't be NULL (Tuple_Check). Drop the X, and see
> if the warning goes away

In comparison, the PyFunction_SetDefaults function does check for
NULL, and raises an error in this case. However, since it is a C API function
only, passing NULL is an error anyway.

Georg


From martin at v.loewis.de  Tue Jul 25 21:44:12 2006
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Tue, 25 Jul 2006 21:44:12 +0200
Subject: [Python-Dev] More tracker demos online
Message-ID: <44C6748C.4060701@v.loewis.de>

Currently, we have two running tracker demos online:

Roundup:
http://efod.se/python-tracker/

Jira:
http://jira.python.atlassian.com/secure/Dashboard.jspa

These installation are in various forms of demo mode and
"pre-release" (meaning that the configuration is still not
complete). They both use the sample data that Fredrik Lundh
produced at some point, so don't be surprised that they
are behind SF wrt. content.

While these might not be in the final form of operation,
I think users should already try to use them, to find
out which one they like best.

Discussions/Comments can be sent to infrastructure at python.org,
however, for reports/reviews, please use the Wiki at

http://wiki.python.org/moin/CallForTrackers

You'll notice that it also lists Trac and Malone, however,
it seems that there is no progress on importing SF data
into these.

Regards,
Martin



From kiko at async.com.br  Tue Jul 25 22:38:36 2006
From: kiko at async.com.br (Christian Robottom Reis)
Date: Tue, 25 Jul 2006 17:38:36 -0300
Subject: [Python-Dev] More tracker demos online
In-Reply-To: <44C6748C.4060701@v.loewis.de>
References: <44C6748C.4060701@v.loewis.de>
Message-ID: <20060725203836.GA1542@anthem.async.com.br>

On Tue, Jul 25, 2006 at 09:44:12PM +0200, "Martin v. L?wis" wrote:
> You'll notice that it also lists Trac and Malone, however,
> it seems that there is no progress on importing SF data
> into these.

Actually, James Henstridge has been working on an import into Launchpad
(Malone is the codename for the bugtracker component of it) over last
week. We have a demo site up at:

    https://demo.launchpad.net/products/python/+bugs

Note that we're still ironing out some of the kinks with the import and
the code running there, so there will be changes before the official
announcement.
-- 
Christian Robottom Reis | http://async.com.br/~kiko/ | [+55 16] 3376 0125

From grig at gheorghiu.net  Fri Jul 21 02:20:58 2006
From: grig at gheorghiu.net (Grig Gheorghiu)
Date: Thu, 20 Jul 2006 17:20:58 -0700 (PDT)
Subject: [Python-Dev] Community buildbots -- reprise
Message-ID: <20060721002058.66596.qmail@web54515.mail.yahoo.com>

Hi,

This message is in response to Glyph's plea
(<http://mail.python.org/pipermail/python-dev/2006-July/067366.html>).

Here's what Glyph said:

"I would like to propose, although I certainly don't have time to
implement, a program by which Python-using projects could contribute
buildslaves which would run their projects' tests with the latest
Python trunk.  This would provide two useful incentives: Python code
would gain a reputation as generally well-tested (since there is a
direct incentive to write tests for your project: get notified when
core python changes might break it), and the core developers would have
instant feedback when a "small" change breaks more code than it was
expected to."


I'm volunteering to organize this effort, is there is enough interest
on this list. In fact, I've done some prep work already:

 * got a domain name: pybots.org
 * got a $47/month Ubuntu-based VPS from JohnCompanies.com (root access
and everything); it's available at master.pybots.org, and it's ready to
be configured as a buildmaster for the pybots
 * got a mailing list: pybots at lists2.idyll.org

I can start configuring the Ubuntu machine as a buildmaster, and I can
also add a buildslave on the same machine that will check out the
latest Python trunk code, build it, then run the automated tests for a
sample project -- let's say for Twisted, since Glyph was the one
requesting this. This will also serve as a sample buildslave for other
people who will be interested in running buildslaves for their own
projects.

Apart from the goals stated by Glyph, I see this as a very valuable
effort in convincing people of the value of automated tests,
Python-related or not. A secondary effect I'd like to see would be for
these suites of tests to be invoked in a standard fashion -- maybe
'python setup.py test'.

If PSF can contribute some $$$ towards the hosting of the master
server, that would be appreciated, but not required. All that's
required is enough interest from the community.

Please let me know if you're interested.

Grig

----
http://agiletesting.blogspot.com

From ndbecker2 at gmail.com  Fri Jul 21 13:59:56 2006
From: ndbecker2 at gmail.com (Neal Becker)
Date: Fri, 21 Jul 2006 07:59:56 -0400
Subject: [Python-Dev] Document performance requirements?
In-Reply-To: <44C0BF4C.1010408@gmail.com>
References: <e9qbtf$kk6$1@sea.gmane.org> <44C0BF4C.1010408@gmail.com>
Message-ID: <200607210759.56595.ndbecker2@gmail.com>

On Friday 21 July 2006 7:49 am, Nick Coghlan wrote:
> Neal Becker wrote:
> > For a recent project I needed to select a container.  There are plenty of
> > python data structures to choose from.  It seems that information on
> > performance is missing (or not easy to find).
> >
> > I think Python should include performance in the documentation of common
> > data structures to help users select the appropriate types.  Something in
> > the style of c++ STL.
>
> Do you mean absolute performance, or do you mean algorithmic order
> guarantees? I thought the latter were already documented. . .
>

The latter.  Where are they documented?

From eswierk at arastra.com  Tue Jul 25 00:19:46 2006
From: eswierk at arastra.com (Ed Swierk)
Date: Mon, 24 Jul 2006 15:19:46 -0700
Subject: [Python-Dev] setup.py and cross-compiling
In-Reply-To: <44C53AE6.7040105@v.loewis.de>
References: <loom.20060724T190947-141@post.gmane.org>
	<44C53AE6.7040105@v.loewis.de>
Message-ID: <c1bf1cf0607241519j157a7050l29998cecc4953485@mail.gmail.com>

On 7/24/06, "Martin v. L?wis" <martin at v.loewis.de> wrote:
> The main problem is that it is the host python that runs setup.py,
> not the target python. Various parts of distutils assume that the
> information the interpreter provides is correct, yet it is not
> in a cross-compilation case.

Well, it seems buildroot solves this main problem by building another
version of python and pygen that run on the build machine, and hacks
the Makefile to run setup.py with these instead of whatever happens to
be sitting in /usr/bin.

Thus the modules that do get built seem to work just fine, but
setup.py itself ignores all this careful hackery when determining what
modules to build and configuring distutils to search for system
libraries in various dynamically-discovered paths.

--Ed

From nyamatongwe at gmail.com  Wed Jul 26 02:10:07 2006
From: nyamatongwe at gmail.com (Neil Hodgson)
Date: Wed, 26 Jul 2006 10:10:07 +1000
Subject: [Python-Dev] More tracker demos online
In-Reply-To: <44C6748C.4060701@v.loewis.de>
References: <44C6748C.4060701@v.loewis.de>
Message-ID: <50862ebd0607251710p454d3cc1s84622aee45774c77@mail.gmail.com>

Martin v. L?wis:

> Currently, we have two running tracker demos online:

   After playing with them for 30 minutes, Jira seems to have too busy
an interface and finicky behaviour: not liking the back button
sometimes (similar to SF) and clicking on diffs wants to download them
rather than view them. Its disappointing that Jira and Launchpad use
different bug IDs as continuity should be maintained with the SF bug
IDs which will be referred to in other areas such as commit messages.
They do include the SF bug ID (as a field in Jira and a nickname in
Launchpad) but this makes it harder to navigate between related bugs.
I mostly looked at "os.startfile() still doesn't work with Unicode
filenames" and I would have tagged the patch on SF with a "looks OK to
me" if SF was working.

   The text in Launchpad was a bit sparsely formatted for me so would
like to see if indvidual users can choose a different style. The
others are OK although Roundup is clearer.

   Neil

From brett at python.org  Wed Jul 26 02:32:22 2006
From: brett at python.org (Brett Cannon)
Date: Tue, 25 Jul 2006 17:32:22 -0700
Subject: [Python-Dev] More tracker demos online
In-Reply-To: <50862ebd0607251710p454d3cc1s84622aee45774c77@mail.gmail.com>
References: <44C6748C.4060701@v.loewis.de>
	<50862ebd0607251710p454d3cc1s84622aee45774c77@mail.gmail.com>
Message-ID: <bbaeab100607251732u5771b6fet4b6e5a949297f8f9@mail.gmail.com>

On 7/25/06, Neil Hodgson <nyamatongwe at gmail.com> wrote:
>
> Martin v. L?wis:
>
> > Currently, we have two running tracker demos online:
>
>    After playing with them for 30 minutes, Jira seems to have too busy
> an interface and finicky behaviour: not liking the back button
> sometimes (similar to SF) and clicking on diffs wants to download them
> rather than view them. Its disappointing that Jira and Launchpad use
> different bug IDs as continuity should be maintained with the SF bug
> IDs which will be referred to in other areas such as commit messages.


Stuff like continuity in bug numbers and such can be fixed in the official
tracker upon launch.  More important is interface and general usage.

And as Martin said, if you wish to discuss, please do so on the
infrastructue mailing list.

-Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20060725/2af3211d/attachment.htm 

From nnorwitz at gmail.com  Wed Jul 26 05:46:13 2006
From: nnorwitz at gmail.com (Neal Norwitz)
Date: Tue, 25 Jul 2006 20:46:13 -0700
Subject: [Python-Dev] Community buildbots -- reprise
In-Reply-To: <20060721002058.66596.qmail@web54515.mail.yahoo.com>
References: <20060721002058.66596.qmail@web54515.mail.yahoo.com>
Message-ID: <ee2a432c0607252046p7a6c91a3gf1451fd1f2f3ed55@mail.gmail.com>

If you want I can send you the build master cfg I setup on python.org
and some simple instructions for how to connect to it.  I don't have
time to focus on this at the moment and probably won't until 2.5 is
out.

n
--

On 7/20/06, Grig Gheorghiu <grig at gheorghiu.net> wrote:
> Hi,
>
> This message is in response to Glyph's plea
> (<http://mail.python.org/pipermail/python-dev/2006-July/067366.html>).
>
> Here's what Glyph said:
>
> "I would like to propose, although I certainly don't have time to
> implement, a program by which Python-using projects could contribute
> buildslaves which would run their projects' tests with the latest
> Python trunk.  This would provide two useful incentives: Python code
> would gain a reputation as generally well-tested (since there is a
> direct incentive to write tests for your project: get notified when
> core python changes might break it), and the core developers would have
> instant feedback when a "small" change breaks more code than it was
> expected to."
>
>
> I'm volunteering to organize this effort, is there is enough interest
> on this list. In fact, I've done some prep work already:
>
>  * got a domain name: pybots.org
>  * got a $47/month Ubuntu-based VPS from JohnCompanies.com (root access
> and everything); it's available at master.pybots.org, and it's ready to
> be configured as a buildmaster for the pybots
>  * got a mailing list: pybots at lists2.idyll.org
>
> I can start configuring the Ubuntu machine as a buildmaster, and I can
> also add a buildslave on the same machine that will check out the
> latest Python trunk code, build it, then run the automated tests for a
> sample project -- let's say for Twisted, since Glyph was the one
> requesting this. This will also serve as a sample buildslave for other
> people who will be interested in running buildslaves for their own
> projects.
>
> Apart from the goals stated by Glyph, I see this as a very valuable
> effort in convincing people of the value of automated tests,
> Python-related or not. A secondary effect I'd like to see would be for
> these suites of tests to be invoked in a standard fashion -- maybe
> 'python setup.py test'.
>
> If PSF can contribute some $$$ towards the hosting of the master
> server, that would be appreciated, but not required. All that's
> required is enough interest from the community.
>
> Please let me know if you're interested.
>
> Grig
>
> ----
> http://agiletesting.blogspot.com
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/nnorwitz%40gmail.com
>

From nnorwitz at gmail.com  Wed Jul 26 06:47:08 2006
From: nnorwitz at gmail.com (Neal Norwitz)
Date: Tue, 25 Jul 2006 21:47:08 -0700
Subject: [Python-Dev] remaining issues from Klocwork static analysis
In-Reply-To: <44C5CAFC.80004@v.loewis.de>
References: <ee2a432c0607242250v5a7ac021p464b68c5c47ec538@mail.gmail.com>
	<44C5CAFC.80004@v.loewis.de>
Message-ID: <ee2a432c0607252147u6e3e4b37i1a77eb8114ac7349@mail.gmail.com>

On 7/25/06, "Martin v. L?wis" <martin at v.loewis.de> wrote:
> Neal Norwitz wrote:
> > # 74 Object/funcobject.c:143    Suspicious deref of ptr before NULL check
>
> Not quite sure what it is complaining about, but
>
>         else if (PyTuple_Check(closure)) {
>                 Py_XINCREF(closure);
>         }
>
> looks indeed suspicious: Why do we check for NULL (XINCREF) when
> we know closure can't be NULL (Tuple_Check). Drop the X, and see
> if the warning goes away

Yes, I definitely think dropping the X would make the warning go away.
 Do we want to check for a NULL pointer and raise an exception?  The
docs don't address the issue, so I think if we added a check, ie:  if
(closure && PyTuple_Check(closure)) and got rid of the X that would be
fine as well.

> > #169 Modules/threadmodule.c:497 Memory Leak
>
> Does it say what memory is leaking? Perhaps it complains about
> boot not being released if ident is not -1, however, in that case,
> t_bootstrap will release the memory.

I believe you are right, I never traced through t_bootstrap.   I think
this is a false positive.  There is some memory being leaked on thread
creation as reported by valgrind IIRC.  This doesn't seem to be it
though.

> > #174 Modules/unicodedata.c:432   Array Index Out of Bounds
> >
> >     Buffer overflow, array index of 'decomp_prefix' may be outside the
> >     bounds. Array 'decomp_prefix' of size 18 declared at
> >     unicodedata_db.h:529 may use index values 18..255. Also there is one
> >     similar error on line 433.
>
> This limit is enforced by Tools/unicode/makeunicodedata.py. There are
> only 18 decomposition prefixes at the moment, yet we use 8 bits for
> the decomposition prefix (makeunicodedata checks that prefix < 256)

Just to make sure I understand.  The code in question is accessing
decomp_prefix like this:
    decomp_prefix[decomp_data[index] & 255]

So decomp_prefix will be accessed with the result of:
    decomp_data[index] & 255

The first line of data is (fro unicodedata_db.h) is:

static unsigned int decomp_data[] = {
    0, 257, 32, 514, 32, 776, 259, 97, 514, 32, 772, 259, 50, 259, 51, 514,

If index == 2 (or 3-5, 7-10, etc), we have:
    decomp_prefix[decomp_data[2] & 255]
    decomp_prefix[32 & 255]
    decomp_prefix[32]

which is larger than the max size of decomp_prefix (18).  But from
what I think you stated above, index can't equal those values and the
code that prevents it is calculated a few lines above:

        index = decomp_index1[(code>>DECOMP_SHIFT)];
        index = decomp_index2[(index<<DECOMP_SHIFT)+
                             (code&((1<<DECOMP_SHIFT)-1))];

Is that correct?  If so, would it be correct to add:

    unsigned short prefix_index = decomp_data[index] & 255;
    assert(prefix_index < (sizeof(decomp_prefix)/sizeof(*decomp_prefix)));

n

From nnorwitz at gmail.com  Wed Jul 26 06:49:56 2006
From: nnorwitz at gmail.com (Neal Norwitz)
Date: Tue, 25 Jul 2006 21:49:56 -0700
Subject: [Python-Dev] remaining issues from Klocwork static analysis
In-Reply-To: <ea4pro$ere$1@sea.gmane.org>
References: <ee2a432c0607242250v5a7ac021p464b68c5c47ec538@mail.gmail.com>
	<44C5CAFC.80004@v.loewis.de> <ea4pro$ere$1@sea.gmane.org>
Message-ID: <ee2a432c0607252149t6136b679w1630fc0fb3dcba63@mail.gmail.com>

On 7/25/06, Georg Brandl <g.brandl at gmx.net> wrote:
> Martin v. L?wis wrote:
> > Neal Norwitz wrote:
> >> # 74 Object/funcobject.c:143    Suspicious deref of ptr before NULL check
> >
> > Not quite sure what it is complaining about, but
> >
> >         else if (PyTuple_Check(closure)) {
> >                 Py_XINCREF(closure);
> >         }
> >
> > looks indeed suspicious: Why do we check for NULL (XINCREF) when
> > we know closure can't be NULL (Tuple_Check). Drop the X, and see
> > if the warning goes away
>
> In comparison, the PyFunction_SetDefaults function does check for
> NULL, and raises an error in this case. However, since it is a C API function
> only, passing NULL is an error anyway.

Heh, that was me that added it 10 days ago. :-)
Might as well do the same here.

n

From grig.gheorghiu at gmail.com  Wed Jul 26 06:57:03 2006
From: grig.gheorghiu at gmail.com (Grig Gheorghiu)
Date: Tue, 25 Jul 2006 21:57:03 -0700
Subject: [Python-Dev] Community buildbots -- reprise
In-Reply-To: <ee2a432c0607252046p7a6c91a3gf1451fd1f2f3ed55@mail.gmail.com>
References: <20060721002058.66596.qmail@web54515.mail.yahoo.com>
	<ee2a432c0607252046p7a6c91a3gf1451fd1f2f3ed55@mail.gmail.com>
Message-ID: <3f09d5a00607252157q117c2e7dxd6c6f75ba7e4fd57@mail.gmail.com>

On 7/25/06, Neal Norwitz <nnorwitz at gmail.com> wrote:
>
> If you want I can send you the build master cfg I setup on python.org
> and some simple instructions for how to connect to it.  I don't have
> time to focus on this at the moment and probably won't until 2.5 is
> out.
>
> n
> --


Sure. I'm still a bit unclear on whether you want me to coordinate this by
adding buildslaves to the build master cfg, adding build steps etc. I'll
gladly do it if you need help. I'll need access to the server and proper
permissions of course.

Grig
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20060725/dac00dfd/attachment.html 

From martin at v.loewis.de  Wed Jul 26 07:05:44 2006
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Wed, 26 Jul 2006 07:05:44 +0200
Subject: [Python-Dev] remaining issues from Klocwork static analysis
In-Reply-To: <ee2a432c0607252147u6e3e4b37i1a77eb8114ac7349@mail.gmail.com>
References: <ee2a432c0607242250v5a7ac021p464b68c5c47ec538@mail.gmail.com>	
	<44C5CAFC.80004@v.loewis.de>
	<ee2a432c0607252147u6e3e4b37i1a77eb8114ac7349@mail.gmail.com>
Message-ID: <44C6F828.7020005@v.loewis.de>

Neal Norwitz wrote:
>> Not quite sure what it is complaining about, but
>>
>>         else if (PyTuple_Check(closure)) {
>>                 Py_XINCREF(closure);
>>         }
>>
>> looks indeed suspicious: Why do we check for NULL (XINCREF) when
>> we know closure can't be NULL (Tuple_Check). Drop the X, and see
>> if the warning goes away
> 
> Yes, I definitely think dropping the X would make the warning go away.
> Do we want to check for a NULL pointer and raise an exception?  The
> docs don't address the issue, so I think if we added a check, ie:  if
> (closure && PyTuple_Check(closure)) and got rid of the X that would be
> fine as well.

The docs do address the issue:

\var{closure} must be \var{Py_None} or a tuple of cell objects.

It doesn't allow for NULL, and None indicates that the closure
should become NULL. The only caller of it in the core will never
pass NULL.

If you want to check that this is not NULL on the grounds that
somebody may call it incorrectly, then you should also check that
op is not NULL, because somebody may call it incorrectly.

> The first line of data is (fro unicodedata_db.h) is:
> 
> static unsigned int decomp_data[] = {
>    0, 257, 32, 514, 32, 776, 259, 97, 514, 32, 772, 259, 50, 259, 51, 514,

Read this as
0: sentinel
257 = 256 | 1: length 1, prefix 1
32: U+0020
514 = 512 | 2: length 2, prefix 2
32: U+0020
776: U+308
...

> Is that correct?  If so, would it be correct to add:
> 
>    unsigned short prefix_index = decomp_data[index] & 255;
>    assert(prefix_index < (sizeof(decomp_prefix)/sizeof(*decomp_prefix)));

Yes, that would be correct.

Regards,
Martin

From martin at v.loewis.de  Wed Jul 26 07:09:47 2006
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Wed, 26 Jul 2006 07:09:47 +0200
Subject: [Python-Dev] More tracker demos online
In-Reply-To: <50862ebd0607251710p454d3cc1s84622aee45774c77@mail.gmail.com>
References: <44C6748C.4060701@v.loewis.de>
	<50862ebd0607251710p454d3cc1s84622aee45774c77@mail.gmail.com>
Message-ID: <44C6F91B.1010105@v.loewis.de>

Neil Hodgson wrote:
> Its disappointing that Jira and Launchpad use
> different bug IDs as continuity should be maintained with the SF bug
> IDs which will be referred to in other areas such as commit messages.

My plan is to keep the SF redirector alive, so python.org/sf/<bugid>
should continue to direct you to the right item in the new tracker.
This can only be done when we actually make the switch, since currently
the redirector still needs to direct to SF.

Regards,
Martin

From nnorwitz at gmail.com  Wed Jul 26 07:12:42 2006
From: nnorwitz at gmail.com (Neal Norwitz)
Date: Tue, 25 Jul 2006 22:12:42 -0700
Subject: [Python-Dev] remaining issues from Klocwork static analysis
In-Reply-To: <44C6F828.7020005@v.loewis.de>
References: <ee2a432c0607242250v5a7ac021p464b68c5c47ec538@mail.gmail.com>
	<44C5CAFC.80004@v.loewis.de>
	<ee2a432c0607252147u6e3e4b37i1a77eb8114ac7349@mail.gmail.com>
	<44C6F828.7020005@v.loewis.de>
Message-ID: <ee2a432c0607252212k5276852dlacb0d73dbd984434@mail.gmail.com>

On 7/25/06, "Martin v. L?wis" <martin at v.loewis.de> wrote:
> >
> > Yes, I definitely think dropping the X would make the warning go away.
> > Do we want to check for a NULL pointer and raise an exception?  The
> > docs don't address the issue, so I think if we added a check, ie:  if
> > (closure && PyTuple_Check(closure)) and got rid of the X that would be
> > fine as well.
>
> The docs do address the issue:
>
> \var{closure} must be \var{Py_None} or a tuple of cell objects.
>
> It doesn't allow for NULL, and None indicates that the closure
> should become NULL. The only caller of it in the core will never
> pass NULL.
>
> If you want to check that this is not NULL on the grounds that
> somebody may call it incorrectly, then you should also check that
> op is not NULL, because somebody may call it incorrectly.

We never really did address this issue did?  A while back we talked
about whether to assert vs check and do PyErr_BadInternalCall().  I
don't remember a clear resolution (though my memory).  I vaguely
remember a preference towards asserting, but I don't know if that was
in all cases or maybe it was just my preference. :-)

I'm happy to assert here too.  But it's really a broader question.  I
guess I'm even happy to just remove the X.  It would be nice to handle
this consistently going forward.

n

From martin at v.loewis.de  Wed Jul 26 07:21:16 2006
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Wed, 26 Jul 2006 07:21:16 +0200
Subject: [Python-Dev] More tracker demos online
In-Reply-To: <ea6msv$hgf$1@sea.gmane.org>
References: <44C6748C.4060701@v.loewis.de> <ea6msv$hgf$1@sea.gmane.org>
Message-ID: <44C6FBCC.5080801@v.loewis.de>

Terry Reedy wrote:
> ""Martin v. L?wis"" <martin at v.loewis.de> wrote in message 
> news:44C6748C.4060701 at v.loewis.de...
>> Currently, we have two running tracker demos online:
>>
>> Roundup:
>> http://efod.se/python-tracker/
>>
>> Jira:
>> http://jira.python.atlassian.com/secure/Dashboard.jspa
> 
> What user name and passwords will they accept, if any?

The roundup installation accepts SF user names, no
password; you can then set the password if you want to.

The Jira installation accepts SF user names; you have make
it send you a password reminder. The registered email
address is the SF one.

If you have problems getting in, please let me know.
I (and the other infrastruture people) have admin privs
on these installations, so I can learn how to administrate
them if something goes wrong :-)

Regards,
Martin

From martin at v.loewis.de  Wed Jul 26 07:32:01 2006
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Wed, 26 Jul 2006 07:32:01 +0200
Subject: [Python-Dev] remaining issues from Klocwork static analysis
In-Reply-To: <ee2a432c0607252212k5276852dlacb0d73dbd984434@mail.gmail.com>
References: <ee2a432c0607242250v5a7ac021p464b68c5c47ec538@mail.gmail.com>	<44C5CAFC.80004@v.loewis.de>	<ee2a432c0607252147u6e3e4b37i1a77eb8114ac7349@mail.gmail.com>	<44C6F828.7020005@v.loewis.de>
	<ee2a432c0607252212k5276852dlacb0d73dbd984434@mail.gmail.com>
Message-ID: <44C6FE51.7060509@v.loewis.de>

Neal Norwitz wrote:
> We never really did address this issue did?  A while back we talked
> about whether to assert vs check and do PyErr_BadInternalCall().  I
> don't remember a clear resolution (though my memory).  I vaguely
> remember a preference towards asserting, but I don't know if that was
> in all cases or maybe it was just my preference. :-)
> 
> I'm happy to assert here too.  But it's really a broader question.  I
> guess I'm even happy to just remove the X.  It would be nice to handle
> this consistently going forward.

I would just remove the X.

If we want to handle it consistently, we would have to check all pointer
parameters in all functions; this would be a huge task (and for little
value, IMO).

In any case, "closure && PyTuple_Check(closure)" would be wrong, since
it then goes into

                PyErr_Format(PyExc_SystemError,
                             "expected tuple for closure, got '%.100s'",
                             closure->ob_type->tp_name);

which crashes just the same.

Regards,
Martin

From nnorwitz at gmail.com  Wed Jul 26 07:41:14 2006
From: nnorwitz at gmail.com (Neal Norwitz)
Date: Tue, 25 Jul 2006 22:41:14 -0700
Subject: [Python-Dev] remaining issues from Klocwork static analysis
In-Reply-To: <44C6FE51.7060509@v.loewis.de>
References: <ee2a432c0607242250v5a7ac021p464b68c5c47ec538@mail.gmail.com>
	<44C5CAFC.80004@v.loewis.de>
	<ee2a432c0607252147u6e3e4b37i1a77eb8114ac7349@mail.gmail.com>
	<44C6F828.7020005@v.loewis.de>
	<ee2a432c0607252212k5276852dlacb0d73dbd984434@mail.gmail.com>
	<44C6FE51.7060509@v.loewis.de>
Message-ID: <ee2a432c0607252241i681435e2n7c846b2d46355936@mail.gmail.com>

On 7/25/06, "Martin v. L?wis" <martin at v.loewis.de> wrote:
> Neal Norwitz wrote:
> > We never really did address this issue did?  A while back we talked
> > about whether to assert vs check and do PyErr_BadInternalCall().  I
> > don't remember a clear resolution (though my memory).  I vaguely
> > remember a preference towards asserting, but I don't know if that was
> > in all cases or maybe it was just my preference. :-)
> >
> > I'm happy to assert here too.  But it's really a broader question.  I
> > guess I'm even happy to just remove the X.  It would be nice to handle
> > this consistently going forward.
>
> I would just remove the X.

I'll do that here since it's the easiest.

> If we want to handle it consistently, we would have to check all pointer
> parameters in all functions; this would be a huge task (and for little
> value, IMO).

I'm not suggesting changing existing code, unless we find issues.  I
agree that it would be a huge task and of little value.  I was
thinking about for future code.  I guess we aren't writing a lot of
new C APIs in 2.x, so it really doesn't much there matter.  Though for
3k, it would be nice to make it consistent as new APIs are written or
old APIs are cleaned up.

n

From greg.ewing at canterbury.ac.nz  Wed Jul 26 08:13:16 2006
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Wed, 26 Jul 2006 18:13:16 +1200
Subject: [Python-Dev] new security doc using object-capabilities
In-Reply-To: <44C567A0.2080906@blueyonder.co.uk>
References: <5.1.1.6.0.20060724015521.0261d160@sparrow.telecommunity.com>
	<5.1.1.6.0.20060723194448.0205fa50@sparrow.telecommunity.com>
	<bbaeab100607191535p543cb0ddj66410cf985dd9b77@mail.gmail.com>
	<20060722114615.GB13782@code0.codespeak.net>
	<44C2CE78.9020802@blueyonder.co.uk>
	<20060723070811.GD13782@code0.codespeak.net>
	<5.1.1.6.0.20060723164246.04144298@sparrow.telecommunity.com>
	<44C3F31D.3070201@blueyonder.co.uk>
	<5.1.1.6.0.20060723194448.0205fa50@sparrow.telecommunity.com>
	<5.1.1.6.0.20060724015521.0261d160@sparrow.telecommunity.com>
	<5.1.1.6.0.20060724211442.0262c008@sparrow.telecommunity.com>
	<44C567A0.2080906@blueyonder.co.uk>
Message-ID: <44C707FC.6@canterbury.ac.nz>

> Phillip J. Eby wrote:
> 
> > And what about code that needs to pass on a subset of a capability?

With one object == one capability, there is no such
thing as a subset of a capability -- the capabilities
are the atomic units at which you control access. So
you need to make them fine-grained enough to begin
with.

--
Greg

From g.brandl at gmx.net  Wed Jul 26 09:16:25 2006
From: g.brandl at gmx.net (Georg Brandl)
Date: Wed, 26 Jul 2006 09:16:25 +0200
Subject: [Python-Dev] remaining issues from Klocwork static analysis
In-Reply-To: <ee2a432c0607252147u6e3e4b37i1a77eb8114ac7349@mail.gmail.com>
References: <ee2a432c0607242250v5a7ac021p464b68c5c47ec538@mail.gmail.com>	<44C5CAFC.80004@v.loewis.de>
	<ee2a432c0607252147u6e3e4b37i1a77eb8114ac7349@mail.gmail.com>
Message-ID: <ea74pt$ign$1@sea.gmane.org>

Neal Norwitz wrote:
> On 7/25/06, "Martin v. L?wis" <martin at v.loewis.de> wrote:
>> Neal Norwitz wrote:
>> > # 74 Object/funcobject.c:143    Suspicious deref of ptr before NULL check
>>
>> Not quite sure what it is complaining about, but
>>
>>         else if (PyTuple_Check(closure)) {
>>                 Py_XINCREF(closure);
>>         }
>>
>> looks indeed suspicious: Why do we check for NULL (XINCREF) when
>> we know closure can't be NULL (Tuple_Check). Drop the X, and see
>> if the warning goes away
> 
> Yes, I definitely think dropping the X would make the warning go away.
>  Do we want to check for a NULL pointer and raise an exception?  The
> docs don't address the issue, so I think if we added a check, ie:  if
> (closure && PyTuple_Check(closure)) and got rid of the X that would be
> fine as well.

You'll have to do something about the error message, then, since it
uses closure->ob_type.

Georg


From greg.ewing at canterbury.ac.nz  Wed Jul 26 09:39:40 2006
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Wed, 26 Jul 2006 19:39:40 +1200
Subject: [Python-Dev] Strategy for converting the decimal module to C
In-Reply-To: <E1G5Irg-0003Zi-9b@libra.cus.cam.ac.uk>
References: <E1G5Irg-0003Zi-9b@libra.cus.cam.ac.uk>
Message-ID: <44C71C3C.4030604@canterbury.ac.nz>

Nick Maclaren wrote:

> The compiled code has made a data structure temporarily inconsistent
> because the operation is safe (say, list insertion), and then gets an
> asynchronous interrupt (e.g. SIGINT).  The SIGINT handler does some
> operation (e.g. I/O) that implicitly uses floating-point, which then
> interrupts.

Well, of course anything can be made to happen asynchronously
by calling it from something asynchronous, such as a SIGINT
handler. That doesn't change the fact that the floating
point operation itself is deterministic, including whether
it causes an exception.

Well-written programs don't do any more in a signal handler
than is absolutely necessary, for reasons which apply equally
well whether floating point is involved or not. I'd say
the mistake was made right at the beginning by assuming
that the data structure in question was safe while allowing
a SIGINT to occur to a handler that's not careful enough
about what it does.

BTW, it seems to me you could get exactly the same problem
if FP exceptions were handled entirely in user mode, as
you suggest. Not that I don't agree that would be a good
idea -- I do -- but it wouldn't prevent this particular
kind of mistake.

And all of this is getting rather far away from where we
started, which was simply instrumenting a piece of code
to count floating point exceptions. Such a program isn't
going to be doing I/O in SIGINT handlers or installing
FP exception handlers that mess with unrelated critical
data structures.

--
Greg

From gh at ghaering.de  Wed Jul 26 10:17:04 2006
From: gh at ghaering.de (=?ISO-8859-1?Q?Gerhard_H=E4ring?=)
Date: Wed, 26 Jul 2006 10:17:04 +0200
Subject: [Python-Dev] remaining issues from Klocwork static analysis
In-Reply-To: <44C5CAFC.80004@v.loewis.de>
References: <ee2a432c0607242250v5a7ac021p464b68c5c47ec538@mail.gmail.com>
	<44C5CAFC.80004@v.loewis.de>
Message-ID: <44C72500.9080301@ghaering.de>

Martin v. L?wis wrote:
> Neal Norwitz wrote:
>> # 61 Modules/_sqlite/cursor.c:599  Null pointer may be dereferenced
>>
>>     Null pointer 'self->statement' that comes from line 674 may be
>>     dereferenced by passing argument 1 to function
>>     'statement_mark_dirty' at line 599.
> 
> Looks like a problem. Maybe a break is missing after line 674?

The code is a bit complicated here, and admittedly not the nicest one, 
but I verified it and there is no problem here.

_query_execute() is designed to handle both executemany() and execute().

multiple is a local variable that is 1 if there is a set of tuples of 
SQL parameters (executemany) instead of a single tuple of SQL parameters 
(execute), in which case it's 0.

Before the while loop, the code makes sure that parameters_iter is an 
iterator that returns SQL parameter tuples. So if there is only a single 
parameter tuple, a temporary list is created an parameters_iter is an 
iterator over it.

So, if !multiple (referenced code in line 674), the while-loop will only 
be executed once, and in the second loop, the while-loop will exit 
because then the following code will break:

         parameters = PyIter_Next(parameters_iter);
         if (!parameters) {
             break;
         }

Code in line 599 is thus not executed with self->statement = 0.

-- Gerhard

From mwh at python.net  Wed Jul 26 10:53:53 2006
From: mwh at python.net (Michael Hudson)
Date: Wed, 26 Jul 2006 09:53:53 +0100
Subject: [Python-Dev] remaining issues from Klocwork static analysis
In-Reply-To: <ee2a432c0607252212k5276852dlacb0d73dbd984434@mail.gmail.com>
	(Neal Norwitz's message of "Tue, 25 Jul 2006 22:12:42 -0700")
References: <ee2a432c0607242250v5a7ac021p464b68c5c47ec538@mail.gmail.com>
	<44C5CAFC.80004@v.loewis.de>
	<ee2a432c0607252147u6e3e4b37i1a77eb8114ac7349@mail.gmail.com>
	<44C6F828.7020005@v.loewis.de>
	<ee2a432c0607252212k5276852dlacb0d73dbd984434@mail.gmail.com>
Message-ID: <2mwta0ycf2.fsf@starship.python.net>

"Neal Norwitz" <nnorwitz at gmail.com> writes:

> On 7/25/06, "Martin v. L?wis" <martin at v.loewis.de> wrote:
>> >
>> > Yes, I definitely think dropping the X would make the warning go away.
>> > Do we want to check for a NULL pointer and raise an exception?  The
>> > docs don't address the issue, so I think if we added a check, ie:  if
>> > (closure && PyTuple_Check(closure)) and got rid of the X that would be
>> > fine as well.
>>
>> The docs do address the issue:
>>
>> \var{closure} must be \var{Py_None} or a tuple of cell objects.
>>
>> It doesn't allow for NULL, and None indicates that the closure
>> should become NULL. The only caller of it in the core will never
>> pass NULL.
>>
>> If you want to check that this is not NULL on the grounds that
>> somebody may call it incorrectly, then you should also check that
>> op is not NULL, because somebody may call it incorrectly.
>
> We never really did address this issue did?  A while back we talked
> about whether to assert vs check and do PyErr_BadInternalCall().  I
> don't remember a clear resolution (though my memory).  I vaguely
> remember a preference towards asserting, but I don't know if that was
> in all cases or maybe it was just my preference. :-)
>
> I'm happy to assert here too.  But it's really a broader question.  I
> guess I'm even happy to just remove the X.  It would be nice to handle
> this consistently going forward.

I think I'm rather in favour of assert()ing this sort of thing.  If
you're programming in C, you can cause crashes any which way and
removing one doesn't seem worth making correct usage pay any kind of
(admittedly miniscule) performance penalty.  It would be nice if API
docs explicitly stated which pointer arguments could be NULL, and then
it would be a programming error to pass a NULL pointer argument in any
other place.  I have no idea how far away from this we are already :-)

Cheers,
mwh

-- 
  Gullible editorial staff continues to post links to any and all
  articles that vaguely criticize Linux in any way.
         -- Reason #4 for quitting slashdot today, from
            http://www.cs.washington.edu/homes/klee/misc/slashdot.html

From python at rcn.com  Wed Jul 26 11:18:24 2006
From: python at rcn.com (Raymond Hettinger)
Date: Wed, 26 Jul 2006 02:18:24 -0700
Subject: [Python-Dev] Strategy for converting the decimal module to C
References: <E1G5Irg-0003Zi-9b@libra.cus.cam.ac.uk>
	<44C71C3C.4030604@canterbury.ac.nz>
Message-ID: <003c01c6b094$72de56a0$0f10000a@RaymondLaptop1>

Greg Ewing
> And all of this is getting rather far away from where we
> started, which was simply instrumenting a piece of code
> to count floating point exceptions.

I'm thinking of adding a note to the Py2.5 docs that the counting feature is not 
part of the standard and should not be expected to work on other implementations 
of the standard (including a planned CPython extension module).


Raymond


From facundobatista at gmail.com  Wed Jul 26 17:52:48 2006
From: facundobatista at gmail.com (Facundo Batista)
Date: Wed, 26 Jul 2006 12:52:48 -0300
Subject: [Python-Dev] Strategy for converting the decimal module to C
In-Reply-To: <003c01c6b094$72de56a0$0f10000a@RaymondLaptop1>
References: <E1G5Irg-0003Zi-9b@libra.cus.cam.ac.uk>
	<44C71C3C.4030604@canterbury.ac.nz>
	<003c01c6b094$72de56a0$0f10000a@RaymondLaptop1>
Message-ID: <e04bdf310607260852qeee5239t97bad5a5f1424017@mail.gmail.com>

2006/7/26, Raymond Hettinger <python at rcn.com>:

> Greg Ewing
> > And all of this is getting rather far away from where we
> > started, which was simply instrumenting a piece of code
> > to count floating point exceptions.
>
> I'm thinking of adding a note to the Py2.5 docs that the counting feature is not
> part of the standard and should not be expected to work on other implementations
> of the standard (including a planned CPython extension module).

+1

-- 
.    Facundo

Blog: http://www.taniquetil.com.ar/plog/
PyAr: http://www.python.org/ar/

From syfou at users.sourceforge.net  Wed Jul 26 20:11:53 2006
From: syfou at users.sourceforge.net (Sylvain Fourmanoit)
Date: Wed, 26 Jul 2006 14:11:53 -0400 (EDT)
Subject: [Python-Dev] New miniconf module
Message-ID: <Pine.LNX.4.64.0607261402410.5956@sylvain>

I wrote a data persistence module called miniconf, aimed at making 
easy to create and safely retrieve configuration info from external, 
human-readable sources using Python syntax. I feel it would eventually 
make a nice addition to the standard library.

The code was only newly refactored in this form, but it as been 
broadly distributed and used as a part of the adesklets project for over a 
year by a significant user base on multiple platforms.

Here it is, as a patch against Python 2.5 SVN tree[1], or as a 
stand-alone module hosted on the Python Cheese Shop[2]; any feedback is 
welcomed.

-- 
Sylvain <syfou at users.sourceforge.net>

Hackers are just a migratory lifeform with a tropism for computers.

[1]http://sourceforge.net/tracker/index.php?func=detail&aid=1527597&group_id=5470&atid=355470
[2]http://cheeseshop.python.org/pypi?:action=display&name=miniconf&version=1.0.1

From amk at amk.ca  Wed Jul 26 19:58:24 2006
From: amk at amk.ca (A.M. Kuchling)
Date: Wed, 26 Jul 2006 13:58:24 -0400
Subject: [Python-Dev] 2.5: uses of sys.exc_type, exc_value
Message-ID: <20060726175824.GA17956@localhost.localdomain>

http://www.python.org/sf/1525469 reports that SimpleXMLRPCServer.py
still uses sys.exc_type and sys.exc_value when handling exceptions.
These variables aren't thread-safe and sys.exc_info() is the better
way.  I have a patch attached to the bug that fixes the problem.

Question 1: is this worth fixing for 2.5?  (It's not really a bugfix,
more of a style cleanup.)

Question 2: I searched for uses of the old variables and found these:

Lib/idlelib/WindowList.py:                      sys.exc_type, ":", sys.exc_value
Lib/logging/__init__.py:        return sys.exc_traceback.tb_frame.f_back
Lib/lib-tk/Tkinter.py:        exc, val, tb = sys.exc_type, sys.exc_value, sys.exc_traceback
Lib/plat-mac/cfmfile.py:                    raise Res.Error, "no 'cfrg' resource found", sys.exc_traceback
Lib/SocketServer.py:            sys.exc_traceback = None    # Help garbage collection

Plus some references in the test suite, the demos, and faqwizard.py.

SocketServer should use sys.exc_clear() instead.  Tkinter.py could
just call exc_info(), but I wonder if the usage of the variables is
intentional here.  sys.exc_info() was introduced in Python 1.5, so
logging/__init__.py could be fixed without affecting 1.5.2
compatibility.

Should the above uses be fixed, too?  

--amk

From guido at python.org  Wed Jul 26 20:21:50 2006
From: guido at python.org (Guido van Rossum)
Date: Wed, 26 Jul 2006 11:21:50 -0700
Subject: [Python-Dev] 2.5: uses of sys.exc_type, exc_value
In-Reply-To: <20060726175824.GA17956@localhost.localdomain>
References: <20060726175824.GA17956@localhost.localdomain>
Message-ID: <ca471dc20607261121i7bfe7115oa217c36170bddc92@mail.gmail.com>

Clearly they should be fixed. Whether in 2.5 or 2.6 I'll leave up to
Neal and Anthony.

On 7/26/06, A.M. Kuchling <amk at amk.ca> wrote:
> http://www.python.org/sf/1525469 reports that SimpleXMLRPCServer.py
> still uses sys.exc_type and sys.exc_value when handling exceptions.
> These variables aren't thread-safe and sys.exc_info() is the better
> way.  I have a patch attached to the bug that fixes the problem.
>
> Question 1: is this worth fixing for 2.5?  (It's not really a bugfix,
> more of a style cleanup.)
>
> Question 2: I searched for uses of the old variables and found these:
>
> Lib/idlelib/WindowList.py:                      sys.exc_type, ":", sys.exc_value
> Lib/logging/__init__.py:        return sys.exc_traceback.tb_frame.f_back
> Lib/lib-tk/Tkinter.py:        exc, val, tb = sys.exc_type, sys.exc_value, sys.exc_traceback
> Lib/plat-mac/cfmfile.py:                    raise Res.Error, "no 'cfrg' resource found", sys.exc_traceback
> Lib/SocketServer.py:            sys.exc_traceback = None    # Help garbage collection
>
> Plus some references in the test suite, the demos, and faqwizard.py.
>
> SocketServer should use sys.exc_clear() instead.  Tkinter.py could
> just call exc_info(), but I wonder if the usage of the variables is
> intentional here.  sys.exc_info() was introduced in Python 1.5, so
> logging/__init__.py could be fixed without affecting 1.5.2
> compatibility.
>
> Should the above uses be fixed, too?
>
> --amk
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org
>


-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From pje at telecommunity.com  Wed Jul 26 20:40:27 2006
From: pje at telecommunity.com (Phillip J. Eby)
Date: Wed, 26 Jul 2006 14:40:27 -0400
Subject: [Python-Dev] Release manager pronouncement needed: PEP 302 Fix
Message-ID: <5.1.1.6.0.20060726142456.036d9008@sparrow.telecommunity.com>

I posted last week about a need-for-speed patch that broke PEP 302 
compliance, and asked if it should be fixed or reverted.  I got exactly one 
response which said "yes, it should be fixed or reverted", which 
unfortunately didn't answer my question as to which one we should do.  :)

If we don't revert it, there are two ways to fix it.  One is to just change 
PEP 302 so that the behavior is unbroken by definition.  :)  The other is 
to actually go ahead and fix it by adding PathImporter and NullImporter 
types to import.c, along with a factory function on sys.path_hooks to 
create them.  (This would've been the PEP-compliant way to implement the 
need-for-speed patch.)

So, "fix" by documentation, fix by fixing, or fix by reverting?  Which 
should it be?


From david.nospam.hopwood at blueyonder.co.uk  Wed Jul 26 20:47:30 2006
From: david.nospam.hopwood at blueyonder.co.uk (David Hopwood)
Date: Wed, 26 Jul 2006 19:47:30 +0100
Subject: [Python-Dev] New miniconf module
In-Reply-To: <Pine.LNX.4.64.0607261402410.5956@sylvain>
References: <Pine.LNX.4.64.0607261402410.5956@sylvain>
Message-ID: <44C7B8C2.1060904@blueyonder.co.uk>

Sylvain Fourmanoit wrote:
> I wrote a data persistence module called miniconf, aimed at making 
> easy to create and safely retrieve configuration info from external, 
> human-readable sources using Python syntax. I feel it would eventually 
> make a nice addition to the standard library.

>From a security point of view, this is a great improvement on the existing
pickle, marshal, and shelve modules. Those modules could not be safely
imported from restricted code.

miniconf, OTOH, appears to have an interface compatible with capability
security. (I have not checked that the compiler.ast module used in its
implementation is safe.) However:

+Limitations
+===========
+
+miniconf has a few limitations one should be aware of:
[...]
+- It is not preemptiple: concurrent calls to dump() or load() will
+  have unpredictable results and must be avoided.

This limitation should be fixed before the module is added to the standard
library, IMHO.

-- 
David Hopwood <david.nospam.hopwood at blueyonder.co.uk>



From pje at telecommunity.com  Wed Jul 26 22:35:43 2006
From: pje at telecommunity.com (Phillip J. Eby)
Date: Wed, 26 Jul 2006 16:35:43 -0400
Subject: [Python-Dev] New miniconf module
In-Reply-To: <44C7B8C2.1060904@blueyonder.co.uk>
References: <Pine.LNX.4.64.0607261402410.5956@sylvain>
	<Pine.LNX.4.64.0607261402410.5956@sylvain>
Message-ID: <5.1.1.6.0.20060726162457.02d5a7c0@sparrow.telecommunity.com>

At 07:47 PM 7/26/2006 +0100, David Hopwood wrote:
>Sylvain Fourmanoit wrote:
> > I wrote a data persistence module called miniconf, aimed at making
> > easy to create and safely retrieve configuration info from external,
> > human-readable sources using Python syntax. I feel it would eventually
> > make a nice addition to the standard library.
>
> >From a security point of view, this is a great improvement on the existing
>pickle, marshal, and shelve modules. Those modules could not be safely
>imported from restricted code.
>
>miniconf, OTOH, appears to have an interface compatible with capability
>security. (I have not checked that the compiler.ast module used in its
>implementation is safe.) However:
>
>+Limitations
>+===========
>+
>+miniconf has a few limitations one should be aware of:
>[...]
>+- It is not preemptiple: concurrent calls to dump() or load() will
>+  have unpredictable results and must be avoided.
>
>This limitation should be fixed before the module is added to the standard
>library, IMHO.

It looks like it's trivial to fix; the code uses a strange and unnecessary 
complication of creating nested classes and nested singleton instances 
thereof.  Getting rid of the singletons to create a new instance for each 
dump/load call would suffice to make the implementation re-entrant, 
although de-nesting the classes would also be a good idea.  :)

The loading code could also be made a lot faster by using a dictionary 
mapping AST node types to functions, instead of doing string manipulation 
for each node.  Each function could take 'pedantic' as a parameter, which 
would eliminate the need to have an object at all, let alone a singleton.

Finally, there is an interesting characteristic of the code's 
interpretation of names: any name other than 'True' is interpreted as 'False'!

On the whole, though, I don't see a lot of difference between this format 
and say, JavaScript Object Notation (JSON), which can be parsed and 
generated by many other languages as well as multiple Python libraries already.


From syfou at users.sourceforge.net  Wed Jul 26 22:45:35 2006
From: syfou at users.sourceforge.net (Sylvain Fourmanoit)
Date: Wed, 26 Jul 2006 16:45:35 -0400 (EDT)
Subject: [Python-Dev] New miniconf module
In-Reply-To: <44C7B8C2.1060904@blueyonder.co.uk>
References: <Pine.LNX.4.64.0607261402410.5956@sylvain>
	<44C7B8C2.1060904@blueyonder.co.uk>
Message-ID: <Pine.LNX.4.64.0607261613130.6037@sylvain>

> miniconf, OTOH, appears to have an interface compatible with capability
> security (I have not checked that the compiler.ast module used in its
> implementation is safe.)

I woudn't be 100% sure either (obviously, I didn't write this nice piece 
of code, let alone the underlying parser), but I read it and tried to 
abuse it without success (I haven't found obvious buffer overflow and 
such)... As far as I know, the abstract  syntax tree generation exposed 
via compiler.ast is a safe operation, in the sense that it doesn't allow 
execution of code when feeded from  arbitrary strings via 
compiler.parse(); in the worst case scenario, it raises a SyntaxError or 
similar exceptions, as documented... If anybody know more on this issue, I 
will be happy to hear about it.

> miniconf has a few limitations one should be aware of:
>
> - It is not preemptiple: concurrent calls to dump() or load() will
>   have unpredictable results and must be avoided.
>
> This limitation should be fixed before the module is added to the 
> standard library, IMHO.

If this is the general opinion, I will be glad to change this... 
The only reason miniconf is not thread-safe for the moment is that I 
chose to re-use over and over a single instance of each of my two 
processing classes to reduce resources usage, but this seems pretty 
pointless (and overly complicated) now that I look at it. Yours,

-- 
Sylvain <syfou at users.sourceforge.net>

Your files are now being encrypted and thrown into the bit bucket.
EOF

From syfou at users.sourceforge.net  Wed Jul 26 23:43:21 2006
From: syfou at users.sourceforge.net (Sylvain Fourmanoit)
Date: Wed, 26 Jul 2006 17:43:21 -0400 (EDT)
Subject: [Python-Dev] New miniconf module
In-Reply-To: <Pine.LNX.4.64.0607261613130.6037@sylvain>
References: <Pine.LNX.4.64.0607261402410.5956@sylvain>
	<44C7B8C2.1060904@blueyonder.co.uk>
	<Pine.LNX.4.64.0607261613130.6037@sylvain>
Message-ID: <Pine.LNX.4.64.0607261742160.6037@sylvain>

> It looks like it's trivial to fix; the code uses a strange and 
> unnecessary complication of creating nested classes and nested 
> singleton instances thereof.  Getting rid of the singletons to create a 
> new instance for each dump/load call would suffice to make the 
> implementation re-entrant, although de-nesting the classes would also be 
> a good idea.  :)

OK then, I will change this.

> The loading code could also be made a lot faster by using a dictionary 
> mapping AST node types to functions, instead of doing string 
> manipulation for each node.  Each function could take 'pedantic' as a 
> parameter, which would eliminate the need to have an object at all, let 
> alone a singleton.
> 
I am not convinced the current string manipulation for mapping the nodes 
types to the methods of the _Load class has such a significant impact on 
performance, but I will test your suggestion... The only difference with 
current code is that we use a dynamically computed string as the 
dictionary key to locate the function instead of the node type themselves 
as keys.

>  Finally, there is an interesting characteristic of the code's
>  interpretation of names: any name other than 'True' is interpreted as
>  'False'!

;-) It will be corrected in the next release.

>  On the whole, though, I don't see a lot of difference between this format
>  and say, JavaScript Object Notation (JSON), which can be parsed and
>  generated by many other languages as well as multiple Python libraries
>  already.

The difference is that this is Python code, already familiar to all Python 
coders... Besides, it sits directly on top of the real Python parser, 
mitigating the need of a new one, and keeping the added code complexity 
to a strict minimum.

But I agree this looks a lot like JSON, since ecmascript syntax for 
literals looks a lot like the one of Python... For the same reasons there 
is a need for JSON, I think having something like miniconf in the 
standard lib would benefit the users.

--
Sylvain <syfou at users.sourceforge.net>

If you think the system is working, ask someone who's waiting for a 
prompt.


From pje at telecommunity.com  Thu Jul 27 00:03:50 2006
From: pje at telecommunity.com (Phillip J. Eby)
Date: Wed, 26 Jul 2006 18:03:50 -0400
Subject: [Python-Dev] New miniconf module
In-Reply-To: <Pine.LNX.4.64.0607261742160.6037@sylvain>
References: <Pine.LNX.4.64.0607261613130.6037@sylvain>
	<Pine.LNX.4.64.0607261402410.5956@sylvain>
	<44C7B8C2.1060904@blueyonder.co.uk>
	<Pine.LNX.4.64.0607261613130.6037@sylvain>
Message-ID: <5.1.1.6.0.20060726180017.0262c990@sparrow.telecommunity.com>

At 05:43 PM 7/26/2006 -0400, Sylvain Fourmanoit wrote:
> > The loading code could also be made a lot faster by using a dictionary
> > mapping AST node types to functions, instead of doing string
> > manipulation for each node.  Each function could take 'pedantic' as a
> > parameter, which would eliminate the need to have an object at all, let
> > alone a singleton.
> >
>I am not convinced the current string manipulation for mapping the nodes
>types to the methods of the _Load class has such a significant impact on
>performance, but I will test your suggestion...

I haven't tested this with your code specifically, but I know that in the 
past I have nearly tripled the speed of AST-visiting code by doing this; 
string manipulation plus attribute lookup is a lot more expensive than 
direct dictionary lookups.


>  The only difference with
>current code is that we use a dynamically computed string as the
>dictionary key to locate the function instead of the node type themselves
>as keys.

Actually you're doing string manipulation plus an *attribute* lookup, and 
attribute lookups can involve multiple dictionary lookups.  But anyway, 
test and see what you get.  :)


>But I agree this looks a lot like JSON, since ecmascript syntax for
>literals looks a lot like the one of Python... For the same reasons there
>is a need for JSON, I think having something like miniconf in the
>standard lib would benefit the users.

Actually, I would see more reason to include JSON in the standard library, 
since it's at least something approaching an internet protocol these days.


From jjl at pobox.com  Thu Jul 27 00:18:30 2006
From: jjl at pobox.com (John J Lee)
Date: Wed, 26 Jul 2006 22:18:30 +0000 (UTC)
Subject: [Python-Dev] New miniconf module
In-Reply-To: <5.1.1.6.0.20060726180017.0262c990@sparrow.telecommunity.com>
References: <Pine.LNX.4.64.0607261613130.6037@sylvain>
	<Pine.LNX.4.64.0607261402410.5956@sylvain>
	<44C7B8C2.1060904@blueyonder.co.uk>
	<Pine.LNX.4.64.0607261613130.6037@sylvain>
	<5.1.1.6.0.20060726180017.0262c990@sparrow.telecommunity.com>
Message-ID: <Pine.LNX.4.64.0607262218060.8397@localhost>

On Wed, 26 Jul 2006, Phillip J. Eby wrote:
[...]
> Actually, I would see more reason to include JSON in the standard library,
> since it's at least something approaching an internet protocol these days.

+1


John


From bob at redivi.com  Thu Jul 27 00:41:41 2006
From: bob at redivi.com (Bob Ippolito)
Date: Wed, 26 Jul 2006 15:41:41 -0700
Subject: [Python-Dev] JSON implementation in Python 2.6
In-Reply-To: <Pine.LNX.4.64.0607262218060.8397@localhost>
References: <Pine.LNX.4.64.0607261613130.6037@sylvain>
	<Pine.LNX.4.64.0607261402410.5956@sylvain>
	<44C7B8C2.1060904@blueyonder.co.uk>
	<Pine.LNX.4.64.0607261613130.6037@sylvain>
	<5.1.1.6.0.20060726180017.0262c990@sparrow.telecommunity.com>
	<Pine.LNX.4.64.0607262218060.8397@localhost>
Message-ID: <661F2C7D-E446-4BB3-B5A7-8237F0850E91@redivi.com>


On Jul 26, 2006, at 3:18 PM, John J Lee wrote:

> On Wed, 26 Jul 2006, Phillip J. Eby wrote:
> [...]
>> Actually, I would see more reason to include JSON in the standard  
>> library,
>> since it's at least something approaching an internet protocol  
>> these days.
>
> +1

If there's a consensus on that, my simplejson [1] implementation  
could migrate to the stdlib for 2.6.

The API is modeled after marshal and pickle, the code should be PEP 8  
compliant, its test suite has pretty good coverage, it's already used  
by (at least) TurboGears and Django, and it's the implementation  
currently "endorsed by" json.org.

The work that would be required would be:

- LaTeX docs (currently reST in docstrings)
- Move the tests around and make them run from the suite rather than  
via nose
- Possible module rename (jsonlib?)

[1] http://undefined.org/python/#simplejson

-bob


From steven.bethard at gmail.com  Thu Jul 27 01:42:58 2006
From: steven.bethard at gmail.com (Steven Bethard)
Date: Wed, 26 Jul 2006 17:42:58 -0600
Subject: [Python-Dev] DRAFT: python-dev summary for 2006-06-16 to 2006-06-30
Message-ID: <d11dcfba0607261642t73095955lc807b4d9a29464a0@mail.gmail.com>

Here's the draft for the second half of June.  As always, comments and
corrections are greatly appreciated.


=============
Announcements
=============

-------------------
Python 2.5 schedule
-------------------

A number of bugs are being squashed as Python 2.5 moves towards its
next release.  See `PEP 356`_ for more details and the full schedule.

.. _PEP 356: http://www.python.org/dev/peps/pep-0356/

Contributing threads:

- `Beta 1 schedule ? (Bug in stringobject?)
<http://mail.python.org/pipermail/python-dev/2006-June/066113.html>`__
- `Adding winerror module (Beta 1 schedule ?)
<http://mail.python.org/pipermail/python-dev/2006-June/066159.html>`__
- `current 2.5 issues
<http://mail.python.org/pipermail/python-dev/2006-June/066204.html>`__
- `TRUNK FREEZE IMMINENT FOR 2.5 BETA 1 - 00:00 UTC, 20-JUNE-2006
<http://mail.python.org/pipermail/python-dev/2006-June/066206.html>`__
- `beta1 coming real soon
<http://mail.python.org/pipermail/python-dev/2006-June/066253.html>`__
- `RELEASED Python 2.5 (beta 1)
<http://mail.python.org/pipermail/python-dev/2006-June/066318.html>`__
- `TRUNK is UNFROZEN, but in FEATURE FREEZE
<http://mail.python.org/pipermail/python-dev/2006-June/066322.html>`__
- `2.5 and beyond
<http://mail.python.org/pipermail/python-dev/2006-June/066807.html>`__

-----------------------------------------
Checkins for betas and release candidates
-----------------------------------------

Anthony Baxter announced some guidelines for checkins for the beta and
release candidate releases. For all beta releases:

* All checkins must have an entry for Misc/NEWS, a test and docs
* All checkins that add features must have approval from a release manager
* All checkins must not break any of the buildbots

For all release candidates:

* All checkins must have approval from a release manager

Approval from a release manager (Anthony or Neal) should preferably be
obtained in public (e.g. the python-dev list) and should be noted in
the commit message.

Contributing threads:

- `When to branch release25-maint?
<http://mail.python.org/pipermail/python-dev/2006-June/066200.html>`__
- `RFC: trunk checkins between now and 2.5 final
<http://mail.python.org/pipermail/python-dev/2006-June/066718.html>`__

-------------------------------------------
FishEye on the Python Subversion Repository
-------------------------------------------

FishEye is once again `available for the Python repository`_.

.. _available for the Python repository:
http://fisheye3.cenqua.com/browse/python

Contributing thread:

- `FishEye on Python CVS Repository
<http://mail.python.org/pipermail/python-dev/2006-June/066188.html>`__


=========
Summaries
=========

---------------------------------
PEP 3103: A Switch/Case Statement
---------------------------------

After Thomas Lee provided a `simple patch implementing a switch
statement`_ for Python, there was a massive discussion about it and
how `PEP 275`_ should best be implemented. After much discussion,
basically three camps arose:

* School I: The switch statement should just be syntactic sugar for
the corresponding if/elif chain.
* School II: The switch statement should dispatch on a precomputed
dict of values.
* School III: The switch statement should correspond to an if/elif
chain but require all expressions to be hashable (to allow for better
optimizations).

School I was primarily concerned with the repetition of the ``x ==``
in something like::

    if x == ...:
       ...
    elif x == ...:
       ...
    elif x == ...:
       ...
    else:
       ...

School II seemed to feel that just aiding DRY was not enough to
introduce a new construct, and that the switch statement should also
be able to avoid the function definitions in dispatching code like::

    def f(...):
        ...
    def g(...):
        ...
    def h(...):
        ...
    dispatch_dict = {x:f, y:g, z:h}
    dispatch_dict[value](*args)

In order to optimize this kind of code, School II wanted to be able to
compute the dispatch dict ahead of time, so that it wouldn't have be
recomputed each time the switch statement was executed. There was a
lot of discussion as to exactly when this freezing should occur, with
some voting for module compilation time (allowing only constants in
the cases), some voting for function definition time (allowing only
constants and non-local names in the cases) and some voting for the
first time the switch statement is executed (allowing only constants
and both local and non-local names). Guido put together a thorough
summary of the options in `PEP 3103`_.

There was some discussion of introducing a ``static`` keyword which
would cause an expression to be evaluated at function definition time,
so that, for example, the following code would create a list of
functions returning each of 0, 1, ... 9::

    funcs = [lambda: (static i) for i in xrange(10)]

The intention was that switch statement cases would then allow only
constants or static expressions. Guido requested a separate PEP on the
idea, and `Fredrik Lundh posted a proto-PEP`_, but at the time of this
summary, no official PEP had been submitted.

In the end, it looked like Guido was leaning towards the switch
statement as syntactic sugar for a dispatching dict, with the dict
frozen at function definition time (which would mean compile-time for
module-level switch statements). However, the introduction of the
statement seemed likely to be postponed at least until Python 3.0.

.. _simple patch implementing a switch statement: http://bugs.python.org/1504199
.. _PEP 275: http://www.python.org/dev/peps/pep-0275/
.. _PEP 3103: http://www.python.org/dev/peps/pep-3103/
.. _Fredrik Lundh posted a proto-PEP:
http://online.effbot.org/2006_06_01_archive.htm#pep-static

Contributing threads:

- `Switch statement
<http://mail.python.org/pipermail/python-dev/2006-June/066086.html>`__
- `An obscene computed goto bytecode hack for "switch" :)
<http://mail.python.org/pipermail/python-dev/2006-June/066115.html>`__
- `Simple Switch statement
<http://mail.python.org/pipermail/python-dev/2006-June/066499.html>`__
- `Alternatives to switch?
<http://mail.python.org/pipermail/python-dev/2006-June/066508.html>`__
- `Temporary Constantification
<http://mail.python.org/pipermail/python-dev/2006-June/066531.html>`__
- `Simple Switch statementZ
<http://mail.python.org/pipermail/python-dev/2006-June/066537.html>`__
- `PEP 3103: A Switch/Case Statement
<http://mail.python.org/pipermail/python-dev/2006-June/066570.html>`__
- `School IIb? <http://mail.python.org/pipermail/python-dev/2006-June/066585.html>`__
- `Switch statement - handling errors
<http://mail.python.org/pipermail/python-dev/2006-June/066650.html>`__
- `Split switch statement
<http://mail.python.org/pipermail/python-dev/2006-June/066652.html>`__
- `once [was: Simple Switch statementZ]
<http://mail.python.org/pipermail/python-dev/2006-June/066692.html>`__

------------------------------
Restricted execution in Python
------------------------------

For his Ph.D. thesis, Brett Cannon is looking into adding facilities
for restricted execution to Python, partly with the goal of getting
Python into Firefox alongside Javascript. His restricted execution
specifications aimed to take advantage of the C-to-Python language
barrier to enforce security restrictions. Though there's no real way
to get private attributes in pure Python, objects coded in C and
exposed to Python can select which attributes are exposed, thus making
the non-exposed attributes truly private to Python-level code.

His initial draft aimed to hide as many "dangerous" objects as
possible, and then cripple objects like ``file`` that would be
difficult to hide. A number of people seemed to prefer a hiding-only
approach, but comments from Armin Rigo seemed to suggest that plugging
all the introspection holes that give access to file objects might be
quite difficult.

The discussion continued on into the next fortnight.

Contributing threads:

- `doc for new restricted execution design for Python
<http://mail.python.org/pipermail/python-dev/2006-June/066344.html>`__
- `Is Lib/test/crashers/recursive_call.py really a crasher?
<http://mail.python.org/pipermail/python-dev/2006-June/066627.html>`__
- `For sandboxing: alternative to crippling file()
<http://mail.python.org/pipermail/python-dev/2006-June/066792.html>`__

-----------------------------------------------
NaN and infinities in Python float calculations
-----------------------------------------------

Nick Maclaren asked about trying to get more platform-independent
behavior in Python's floats, so that IEEE 754 values as in `PEP 754`_
would be produced more consistently. Currently, different OSes produce
different results when these values are involved::

    Python 2.4.2 (#1, May  2 2006, 08:28:01)
    [GCC 4.1.0 (SUSE Linux)] on linux2
    Type "help", "copyright", "credits" or "license" for more information.
    >>> a = "NaN"
    >>> b = float(a)
    >>> c = int(b)
    >>> d = (b == b)
    >>> print a, b, c, d
    NaN nan 0 False

    Python 2.3.3 (#1, Feb 18 2004, 11:58:04)
    [GCC 2.8.1] on sunos5
    Type "help", "copyright", "credits" or "license" for more information.
    >>> a = "NaN"
    >>> b = float(a)
    >>> c = int(b)
    >>> d = (b == b)
    >>> print a, b, c, d
    NaN NaN 0 True

Nick Maclaren suggested either raising an exception for all ambiguous
or invalid operations, or returning NaN or infinity as appropriate and
then raising exceptions whenever an operation that would lose the
error indication was performed. Nick Coghlan explained that the
decimal module already does most of this::

    >>> from decimal import Decimal as d
    >>> nan = d('NaN')
    >>> int(nan)
    Traceback (most recent call last):
      ...
    decimal.InvalidOperation
    >>>
    >>> from decimal import getcontext, Overflow
    >>> ctx = getcontext()
    >>> ctx.traps[Overflow] = False
    >>> d('1e999999999') * 10
    Decimal("Infinity")

Nick Maclaren seemed to suggest that he would be working on a PEP and
an implementation that would bring some of the decimal module
consistencies to Python's floats as well.

.. _PEP 754: http://www.python.org/dev/peps/pep-0754/

Contributing threads:

- `Numerical robustness, IEEE etc.
<http://mail.python.org/pipermail/python-dev/2006-June/066186.html>`__
- `Numerical robustness, IEEE etc.
<http://mail.python.org/pipermail/python-dev/2006-June/066192.html>`__
- `Python memory model (low level)
<http://mail.python.org/pipermail/python-dev/2006-June/066834.html>`__

-----------------------------------------------
ImportWarnings for directories without __init__
-----------------------------------------------

Ralf W. Grosse-Kunstleve complained that with Python 2.5 he started
getting tons of "ImportWarning: Not importing directory" messages.
James Y Knight pointed out that running Python in your home directory
is quite likely to issue such warnings if you have *any* directories
in your home directory that have the same name as a python module
(e.g. ``readline``). A number of options for silencing the errors were
discussed, including invoking Python like ``python -W'ignore:Not
importing directory'`` and including
``warnings.filterwarnings('ignore', 'Not importing directory',
ImportWarning)`` in site.py or .pythonrc.py. Two patches were provided
that introduce the warning only if the import fails, `one by Shane
Hathaway`_ and `one by Sergey A. Lipnevich`_. No final decision had
been made at the time of this summary.

.. _one by Shane Hathaway: http://bugs.python.org/1515361
.. _one by Sergey A. Lipnevich: http://bugs.python.org/1515609

Contributing threads:

- `Dropping __init__.py requirement for subpackages
<http://mail.python.org/pipermail/python-dev/2006-June/066280.html>`__
- `ImportWarning flood
<http://mail.python.org/pipermail/python-dev/2006-June/066345.html>`__

------------------
Updating turtle.py
------------------

Gregor Lingl proposed replacing turtle.py in the Python standard
library with his `new xturtle.py module`_. The xturtle module is
backwards compatible with the turtle module and adds a number of
enhancements. However, Gregor's request came after Python 2.5's
feature freeze, so he was told to propose it again in Python 2.6.
There was some discussion about this -- as the stdlib turtle module is
poorly tested, some contended that introducing the new APIs of xturtle
would not make things any worse. A couple of compromises were offered:
mentioning xturtle in the turtle module docs, and putting xturtle in
the Tools directory.

.. _new xturtle.py module: http://ada.rg16.asn-wien.ac.at/~python/xturtle/

Contributing threads:

- `xturtle.py a replacement for turtle.py(!?)
<http://mail.python.org/pipermail/python-dev/2006-June/066676.html>`__
- `xturtle.py - a replacement for turtle.py
<http://mail.python.org/pipermail/python-dev/2006-June/066677.html>`__
- `xturtle.py a replacement for turtle.py(!?) ATTENTION PLEASE!
<http://mail.python.org/pipermail/python-dev/2006-June/066734.html>`__
- `xturtle.py <http://mail.python.org/pipermail/python-dev/2006-June/066742.html>`__

----------------------------------------------------------
Relative imports and PEP 338: Executing Modules as Scripts
----------------------------------------------------------

Relative imports, as described in `PEP 328`_, introduced problems for
`PEP 338`_ which allows modules within packages and zipfiles to be run
with the -m command-line switch. The -m switch sets the __name__ of
the module to '__main__' so that ``if __name__ == '__main__'`` blocks
will get executed. However, relative imports use __name__ to determine
the parent package, so if a module that has a relative import is
executed using the -m switch, the relative import will fail.  Nick
Coghlan suggested adding a __module_name__ attribute that would not be
clobbered by the -m switch, but people generally seemed to think that
it would be simpler to just require absolute imports in main modules.

.. _PEP 328: http://www.python.org/dev/peps/pep-0328/
.. _PEP 338: http://www.python.org/dev/peps/pep-0338/

Contributing threads:

- `PEP 338 vs PEP 328 - a limitation of the -m switch
<http://mail.python.org/pipermail/python-dev/2006-June/066161.html>`__
- `PEP 328 and PEP 338, redux
<http://mail.python.org/pipermail/python-dev/2006-June/066609.html>`__
- `[Python-checkins] r47142 - in python/trunk: Doc/lib/librunpy.tex
Lib/runpy.py Lib/test/test_runpy.py
<http://mail.python.org/pipermail/python-dev/2006-June/066690.html>`__

--------------------------------------------
Importing modules within unicode directories
--------------------------------------------

Kristj?n V. J?nsson pointed out that currently, Python on Windows
cannot import modules from directories with unicode names, even if the
module names themselves are plain ASCII. Nick Coghlan suggested that
this was likely because import.c was doing something like
``u'c:/tmp/\u814c'.encode('mbcs')``, getting back ``'c:/tmp/?'`` and
being unable to do anything useful with that. Martin v. L?wis
suggested using the 8.3 simplified filename used by DOS, at least
until the import machinery gets reworked to better handle encodings,
hopefully for Python 2.6. `Thomas Heller had provided a patch`_ for
reworking import.c in this manner a while back, but it was large
enough that no one had reviewed it.

.. _Thomas Heller had provided a patch: http://bugs.python.org/1093253

Contributing thread:

- `unicode imports
<http://mail.python.org/pipermail/python-dev/2006-June/066103.html>`__

----------------------------------------
MS VC++ 2003 toolkit no longer available
----------------------------------------

Bill Janssen pointed out that Python 2.4 on Windows expects to be
compiled with the MS Visual C++ compiler version 7.1, and that the
corresponding MS VC++ 2003 toolkit is no longer available. Fredrik
Lundh explained that the compiler is still available in the .net SDK
as well as being available to MSDN subscribers. There was again some
discussion about moving to the VS 2005 toolkit for compiling Python.
It would have made compiling for 64bit architectures somewhat easier,
but would have meant that extension writers would have to install
three different compilers just to compile extensions for Python 2.3,
2.4 and 2.5, and would also have given problems for MinGW users as
MinGW does not yet easily support linking to the msvcr80 runtime
library.

Contributing threads:

- `Python 2.4 extensions require VC 7.1?
<http://mail.python.org/pipermail/python-dev/2006-June/066110.html>`__
- `Documentation enhancement: "MS free compiler"?
<http://mail.python.org/pipermail/python-dev/2006-June/066182.html>`__
- `Documentation enhancement: "MS free compiler"?
<http://mail.python.org/pipermail/python-dev/2006-June/066257.html>`__

---------------------------------
Keeping interned strings in a set
---------------------------------

Alexander Belopolsky tried out the new set C API by `replacing the
dict of interned strings with a set`_ instead. He had to make two
changes to get this to work: there's currently no way to retrieve a
single object from a set, and Py_Finalize() needed to be changed to
finalize sets after strings (instead of the other way around as it
used to be). There was some discussion about trying to get rid of
PySet_Fini() so the latter problem wouldn't be an issue at all, but
with all the other Py*Fini() functions already existing, it didn't
seem worth it.

The patch had no slowdown and reduced the memory consumption of the
interning structure slightly.

.. _replacing the dict of interned strings with a set:
http://bugs.python.org/1507011

Contributing threads:

- `Keeping interned strings in a set
<http://mail.python.org/pipermail/python-dev/2006-June/066084.html>`__
- `Keeping interned strings in a set
<http://mail.python.org/pipermail/python-dev/2006-June/066088.html>`__
- `setobject code
<http://mail.python.org/pipermail/python-dev/2006-June/066116.html>`__
- `Proposal to eliminate PySet_Fini
<http://mail.python.org/pipermail/python-dev/2006-June/066645.html>`__

-------------------------
Allowing empty subscripts
-------------------------

Guido finally vetoed the proposal to allow ``x[()]`` to be written as
``x[]``. The use-cases were weak, and in most cases the functionality
seemed better expressed as attribute access.

Contributing threads:

- `Pre-PEP: Allow Empty Subscript List Without Parentheses
<http://mail.python.org/pipermail/python-dev/2006-June/066099.html>`__
- `Empty Subscript PEP on Wiki - keep or toss?
<http://mail.python.org/pipermail/python-dev/2006-June/066848.html>`__

-------------------------------------
Creating range objects at the C level
-------------------------------------

Ralf W. Grosse-Kunstleve asked about the removal of the C function
``PyRange_New()`` which had been deprecated in Python 2.4. The right
way to create ranges is to call PyRange_Type with the appropriate
parameters, e.g. something like ``PyObject_CallFunction((PyObject*)
&PyRange_Type, "lll", start, stop, step)``. Ralf was nervous about
this alternative because it also appeared to be undocumented, and
requested that something like the above be at least put into the
What's New document.

Contributing threads:

- `PyRange_New() alternative?
<http://mail.python.org/pipermail/python-dev/2006-June/066343.html>`__
- `PyObject* aliasing (Was: PyRange_New() alternative?)
<http://mail.python.org/pipermail/python-dev/2006-June/066477.html>`__

----------------------------------
type(), __class__ and isinstance()
----------------------------------

Martin Maly pointed out that you can't fool isinstance() into thinking
your object is not a subclass of its true base class::

    >>> class C(object):
    ...     pass
    ...
    >>> class D(object):
    ...     __class__ = property(lambda self: C)
    ...
    >>> isinstance(D(), D)
    True
    >>> isinstance(D(), C)
    True

Phillip J. Eby explained that isinstance() checks both the type() of
the object and the __class__ attribute.  In essence, you can lie about
your __class__ to make isinstance() return True, but you can't lie to
make it return False. Guido suggested that these issues, as well as
lying about an object's __bases__, should be revisited for Python
3000.

Contributing thread:

- `Semantic of isinstance
<http://mail.python.org/pipermail/python-dev/2006-June/066591.html>`__

----------------------------------------------------------------
Requiring backward compatibility in the standard library modules
----------------------------------------------------------------

Ka-Ping Yee's uuid module, newly added for Python 2.5, contained a
comment "This module works with Python 2.3 or higher". George Yoshida
asked if that comment should be interpreted as requiring Python 2.3
compatibility. People generally felt like the list of backwards
compatible modules in `PEP 291`_ should be as small as possible so as
to keep maintenance as simple as possible. Ka-Ping removed the
comment, and submitted the module to PyPI for Python 2.3 and 2.4
users.

.. _PEP 291: http://www.python.org/dev/peps/pep-0291/

Contributing thread:

- `uuid backward compatibility
<http://mail.python.org/pipermail/python-dev/2006-June/066153.html>`__

---------------------
Figleaf code coverage
---------------------

`Titus Brown offered some reports`_ from his `figleaf code coverage`_
utility. People seemed particularly interested in trying to get
coverage across multiple platforms, perhaps using a BuildBot
extension, and Titus said he'd try to look into it. Walter D?rwald
also pointed to `his own code coverage module`_.

.. _Titus Brown offered some reports:
http://vallista.idyll.org/~t/temp/python2.4-svn/
.. _figleaf code coverage:
http://darcs.idyll.org/~t/projects/figleaf-latest.tar.gz
.. _his own code coverage module:
http://styx.livinglogic.de/~walter/python/coverage/PythonCodeCoverage.py

Contributing threads:

- `Code coverage reporting.
<http://mail.python.org/pipermail/python-dev/2006-June/066184.html>`__
- `Code coverage reporting.
<http://mail.python.org/pipermail/python-dev/2006-June/066193.html>`__

------------------------
Improving error messages
------------------------

Georg Brandl proposed going through abstract.c and modifying error
messages like "object does not support item assignment" to also
include the type of the object. He got little feedback, mainly because
everyone seemed to think it was such an obviously good idea that there
was no need for any.  Python 2.5 now incorporates `Georg's better
error messages`_.

.. _Georg's better error messages: http://bugs.python.org/1507676

Contributing threads:

- `Improve error msgs?
<http://mail.python.org/pipermail/python-dev/2006-June/066048.html>`__
- `Improve error msgs?
<http://mail.python.org/pipermail/python-dev/2006-June/066128.html>`__

-----------------------------------------
Allowing assignments in global statements
-----------------------------------------

Talin proposed allowing a global statement to be combined with an
assignment statement, e.g.::

    global badger = 42

Guido suggested that such a desire was a sure indicator of overuse of
``global``.

Contributing thread:

- `Allow assignments in 'global' statements?
<http://mail.python.org/pipermail/python-dev/2006-June/066347.html>`__

-----------------------------------------
Splitting Python tests from CPython tests
-----------------------------------------

Frank Wierzbicki volunteered some time into splitting out CPython
specific test from Python-the-language tests. Armin Rigo pointed him
to PyPy's `tests modified to be more implementation independent`_.

.. _tests modified to be more implementation independent:
http://codespeak.net/svn/pypy/dist/lib-python/modified-2.4.1/test

Contributing thread:

- `Cleanup of test harness for Python
<http://mail.python.org/pipermail/python-dev/2006-June/066817.html>`__

-----------------------------------------
A multi-dimensional array type for Python
-----------------------------------------

For `Google's Summer of Code`_, Karol Langner will be working on
`implementing a basic multi-dimensional array type`_ for Python core,
based on the numpy_ array struct. He asked for any comments or
suggestions that people had for the project.

.. _Google's Summer of Code: http://code.google.com/summerofcode.html
.. _numpy: http://www.numpy.org/
.. _implementing a basic multi-dimensional array type:
http://scipy.org/BaseArray

Contributing thread:

- `basearray <http://mail.python.org/pipermail/python-dev/2006-June/066516.html>`__


==================
Previous Summaries
==================

- `Source control tools
<http://mail.python.org/pipermail/python-dev/2006-June/066187.html>`__
- `Dropping externally maintained packages (Was: Please stop changing
wsgiref on the trunk)
<http://mail.python.org/pipermail/python-dev/2006-June/066195.html>`__


===============
Skipped Threads
===============

- `Last-minute curses patch
<http://mail.python.org/pipermail/python-dev/2006-June/066095.html>`__
- `Bug in stringobject?
<http://mail.python.org/pipermail/python-dev/2006-June/066100.html>`__
- `Fwd: subprocess.Popen(.... stdout=IGNORE, ...)
<http://mail.python.org/pipermail/python-dev/2006-June/066111.html>`__
- `About dynamic module loading
<http://mail.python.org/pipermail/python-dev/2006-June/066190.html>`__
- `PyString_FromFormat
<http://mail.python.org/pipermail/python-dev/2006-June/066213.html>`__
- `Misleading error message from PyObject_GenericSetAttr
<http://mail.python.org/pipermail/python-dev/2006-June/066227.html>`__
- `Bug: xml.dom.pulldom never gives you END_DOCUMENT events with an
Expat parser <http://mail.python.org/pipermail/python-dev/2006-June/066228.html>`__
- `os.getmtime now returns a float?
<http://mail.python.org/pipermail/python-dev/2006-June/066252.html>`__
- `XP build failing
<http://mail.python.org/pipermail/python-dev/2006-June/066258.html>`__
- `ETree: xml vs xmlcore
<http://mail.python.org/pipermail/python-dev/2006-June/066268.html>`__
- `test_ctypes failure on Mac OS X/PowerPC 10.3.9 (Panther)
<http://mail.python.org/pipermail/python-dev/2006-June/066282.html>`__
- `Small sqlite3 test suite fix (Python 2.5b1 candidate)
<http://mail.python.org/pipermail/python-dev/2006-June/066291.html>`__
- `Weekly Python Patch/Bug Summary
<http://mail.python.org/pipermail/python-dev/2006-June/066350.html>`__
- `Things to remember when adding *packages* to stdlib
<http://mail.python.org/pipermail/python-dev/2006-June/066353.html>`__
- `Moving the ctypes repository to python.org
<http://mail.python.org/pipermail/python-dev/2006-June/066417.html>`__
- `PyObject_CallFunction and 'N' format char
<http://mail.python.org/pipermail/python-dev/2006-June/066501.html>`__
- `pypy-0.9.0: stackless, new extension compiler
<http://mail.python.org/pipermail/python-dev/2006-June/066512.html>`__
- `[Python-checkins] Things to remember when adding *packages* to
stdlib <http://mail.python.org/pipermail/python-dev/2006-June/066515.html>`__
- `Import semantics
<http://mail.python.org/pipermail/python-dev/2006-June/066523.html>`__
- `2.5b1 Windows install
<http://mail.python.org/pipermail/python-dev/2006-June/066542.html>`__
- `Python-Dev Digest, Vol 35, Issue 143
<http://mail.python.org/pipermail/python-dev/2006-June/066577.html>`__
- `Problems building Python on OSX 10.4.6?
<http://mail.python.org/pipermail/python-dev/2006-June/066579.html>`__
- `enhancements for uuid module
<http://mail.python.org/pipermail/python-dev/2006-June/066583.html>`__
- `Do we need a bug triage day?
<http://mail.python.org/pipermail/python-dev/2006-June/066647.html>`__
- `Oh-why that?? Please ignore one of the two
<http://mail.python.org/pipermail/python-dev/2006-June/066678.html>`__
- `msvccompiler.py: some remarks
<http://mail.python.org/pipermail/python-dev/2006-June/066768.html>`__
- `Joke: Rush Limbaugh (a joke in and of himself)
<http://mail.python.org/pipermail/python-dev/2006-June/066780.html>`__
- `PyGIL_ and --without-threads
<http://mail.python.org/pipermail/python-dev/2006-June/066784.html>`__
- `document @property?
<http://mail.python.org/pipermail/python-dev/2006-June/066787.html>`__
- `Pickle implementation questions
<http://mail.python.org/pipermail/python-dev/2006-June/066803.html>`__
- `sys.settrace() in Python 2.3 vs. 2.4
<http://mail.python.org/pipermail/python-dev/2006-June/066820.html>`__
- `how long to wait for expat to incorporate a fix to prevent a
crasher? <http://mail.python.org/pipermail/python-dev/2006-June/066829.html>`__
- `LOAD_CONST POP_TOP
<http://mail.python.org/pipermail/python-dev/2006-June/066832.html>`__

From tim.peters at gmail.com  Thu Jul 27 02:02:37 2006
From: tim.peters at gmail.com (Tim Peters)
Date: Wed, 26 Jul 2006 20:02:37 -0400
Subject: [Python-Dev] [Windows, buildbot] kill_python.c mystery
Message-ID: <1f7befae0607261702o6d943f52x701be0d402345f65@mail.gmail.com>

Rarely I'll be running the Python tests in my sandbox from a DOS box,
and the test run will just end.  Like so:

C:\Code\python\PCbuild>python  -E -tt ../lib/test/regrtest.py -uall -rw
test_softspace
test_codecmaps_kr
...
test_float
test_userdict

C:\Code\python\PCbuild>

No indication of success or failure -- the process just vanishes mid-stream.

Today I noticed this happened when the buildbot started to run tests,
and I'm 100% sure it's due to this code in
Tools/buildbot/kill_python.c (the buildbot log files showed that
kill_python.c killed /some/ Python process, and the Python running
release-build tests in my sandbox was the only plausible candidate):

	if ((strstr(path, "build\\pcbuild\\python_d.exe") != NULL) ||
	    (strstr(path, "build\\python.exe") != NULL)) {
		printf("Terminating %s (pid %d)\n", path, pids[i]);
		if (!TerminateProcess(hProcess, 1)) {

The second clause in the first `if` looks for a substring match on:

    build\python.exe

and that just happens to match a suffix of:

    C:\Code\python\PCbuild\python.exe

which is the release-build Python I happen to be running in my sandbox.

Why is the second clause there?  That is, are we /trying/ to kill a
release-build Python running from the user's sandbox, and if so why?
Introducing the second clause was the sole change in rev 46817, and
the checkin comment doesn't really explain it:

    Port cygwin kill_python changes from 2.4 branch.

Since I don't know what it's trying to accomplish, I hesitate to
change it.  It's quite clear what the first clause is trying to
accomplish, and that one hasn't caused any problems.

From mattjfleming at googlemail.com  Thu Jul 27 02:20:19 2006
From: mattjfleming at googlemail.com (Matt Fleming)
Date: Thu, 27 Jul 2006 00:20:19 +0000
Subject: [Python-Dev] Improving unit tests for the standard library
Message-ID: <5ff4a1e50607261720o2a98572ft4cab275ed91ad344@mail.gmail.com>

Hi, after speaking with Neal off-list about writing tests for the
pkgutil module, we agreed it would be a good idea to start a page on
http://wiki.python.org/moin/ stating any tests for the standard
library that either,

a) need to be written
b) can be improved

I've started the page http://wiki.python.org/moin/ImprovingLibTests
that lists all the test files I could think of that need to be
written. Ive also included a table for improvements to existing tests,
along with a column that allows you to specify exactly what needs
improving.

I hope this will be of use to people, and I hope people will find time
to modify the page approriately. When I get some spare time from my
SoC project, I'll be working my way through the list.

Thanks, Matt

-- 
http://mattssanctuary.blogspot.com

From ocean at m2.ccsnet.ne.jp  Thu Jul 27 03:15:23 2006
From: ocean at m2.ccsnet.ne.jp (H.Yamamoto)
Date: Thu, 27 Jul 2006 10:15:23 +0900
Subject: [Python-Dev] patch for mbcs codec (again)
Message-ID: <002601c6b11a$22e3fa70$0400a8c0@whiterabc2znlh>

Hello.

I noticed mbcs codec still has problem when calls StreamReader.
Can anyone commit the patch "fix.patch version2" on
http://python.org/sf/1455898 ?

# Very sorry about this... I thought I checked this, but I definitely looked
at something
# wrong.



From kbk at shore.net  Thu Jul 27 04:18:12 2006
From: kbk at shore.net (Kurt B. Kaiser)
Date: Wed, 26 Jul 2006 22:18:12 -0400 (EDT)
Subject: [Python-Dev] Weekly Python Patch/Bug Summary
Message-ID: <200607270218.k6R2ICQ9021704@bayview.thirdcreek.com>

Patch / Bug Summary
___________________

Patches :  401 open ( +3) /  3342 closed ( +8) /  3743 total (+11)
Bugs    :  896 open ( -8) /  6035 closed (+24) /  6931 total (+16)
RFE     :  224 open ( +2) /   233 closed ( +2) /   457 total ( +4)

New / Reopened Patches
______________________

pkgutil.walk_packages ignores onerror arg  (2006-07-20)
CLOSED http://python.org/sf/1525766  opened by  James Y Knight

Tkdnd mouse cursor handling patch  (2006-07-20)
       http://python.org/sf/1525806  opened by  klappnase

calltip awkwardly shortened  (2006-07-20)
CLOSED http://python.org/sf/1525817  opened by  Lingl

str.__iter__ and unicode.__iter__  (2006-07-21)
       http://python.org/sf/1526367  opened by  Walter D??rwald

Fix socketmodule compile on NetBSD  (2006-07-21)
       http://python.org/sf/1526460  opened by  Matt Fleming

New module: miniconf  (2006-07-24)
       http://python.org/sf/1527597  opened by  S.Fourmanoit

winsound - probably win9x port is not working  (2006-07-24)
CLOSED http://python.org/sf/1527744  opened by  Hirokazu Yamamoto

Expose case-insensitivity of string.Template  (2006-07-25)
       http://python.org/sf/1528167  opened by  Chad Whitacre

PyShell.recall - fix indentation logic  (2006-07-25)
       http://python.org/sf/1528468  opened by  Tal Einat

patch for mbcs codecs  (2006-03-22)
       http://python.org/sf/1455898  reopened by  ocean-city

Patches Closed
______________

pkgutil.walk_packages ignores onerror arg  (2006-07-20)
       http://python.org/sf/1525766  closed by  gbrandl

calltip awkwardly shortened  (2006-07-20)
       http://python.org/sf/1525817  closed by  loewis

ParenMatch: workaround for misinterpreting of closing parens  (2006-01-16)
       http://python.org/sf/1407280  closed by  kbk

Syntax-related improvements to IDLE  (2004-02-28)
       http://python.org/sf/906702  closed by  kbk

(partial?) fix for Misc/python-config.in  (2006-07-16)
       http://python.org/sf/1523356  closed by  gbrandl

ConnectRegistry blocks all threads  (2006-03-12)
       http://python.org/sf/1448199  closed by  loewis

MS Windows - module search path fix  (2005-07-04)
       http://python.org/sf/1232023  closed by  loewis

winsound - probably win9x port is not working  (2006-07-24)
       http://python.org/sf/1527744  closed by  gbrandl

1515163 fix - traceback and str exc  (2006-06-30)
       http://python.org/sf/1515343  closed by  gbrandl

New / Reopened Bugs
___________________

inspect.py: still infinite recursion inspecting frames  (2006-07-03)
CLOSED http://python.org/sf/1516184  reopened by  nnorwitz

Bug in shutil.copytree on Windows  (2006-07-20)
       http://python.org/sf/1525866  opened by  Mike Foord

email package quoted printable behaviour changed  (2006-07-20)
       http://python.org/sf/1525919  opened by  Thomas Arendsen Hein

Win32: subprocess.Popen() w/o "pipe" throws an exception  (2006-07-21)
CLOSED http://python.org/sf/1526203  opened by  Larry Hastings

Concatenation on a long string breaks  (2006-07-21)
       http://python.org/sf/1526585  opened by  Jp Calderone

current directory added to sys.path on win32  (2006-07-22)
CLOSED http://python.org/sf/1526785  opened by  John Ehresman

unbalanced parentheses  from command line crash pdb  (2006-07-21)
       http://python.org/sf/1526834  opened by  Ilya Sandler

PythonLauncher uses incorrect working directory  (2006-07-23)
       http://python.org/sf/1527397  opened by  Bob Ippolito

optparse should support arbitrary number of arguments  (2006-07-24)
       http://python.org/sf/1527705  reopened by  riteshsarraf

optparse should support arbitrary number of arguments  (2006-07-24)
       http://python.org/sf/1527705  opened by  Ritesh Raj Sarraf

tarfile chokes on ipython archive on Windows  (2006-07-24)
       http://python.org/sf/1527974  opened by  Arve Knudsen

difflib.SequenceMatcher.find_longest_match()  wrong result  (2006-07-25)
       http://python.org/sf/1528074  opened by  John Machin

urllib2 data argument  (2006-07-25)
       http://python.org/sf/1528258  opened by  paul rubin

forward in turtle module may cause incorrect display  (2006-07-25)
       http://python.org/sf/1528363  opened by  NatureMage

IDLE: printing issue on OSX  (2006-07-25)
       http://python.org/sf/1528593  opened by  Ronald Oussoren

Python 2.5b2 fails to build on Solaris 10  (2006-07-25)
       http://python.org/sf/1528620  opened by  Guido Ostkamp

Python 2.5b2 fails to build (GCC) on Solaris 10  (2006-07-26)
       http://python.org/sf/1529269  opened by  Guido Ostkamp

Bugs Closed
___________

inspect.py: still infinite recursion inspecting frames  (2006-07-03)
       http://python.org/sf/1516184  closed by  pje

logging using the SysLog handler fails if locale is set  (2006-07-17)
       http://python.org/sf/1524081  closed by  vsajip

Win32: subprocess.Popen() w/o "pipe" throws an exception  (2006-07-21)
       http://python.org/sf/1526203  closed by  gbrandl

Malloc, memory error, failmalloc, low memory.  (2006-07-19)
       http://python.org/sf/1525589  closed by  nnorwitz

current directory added to sys.path on win32  (2006-07-22)
       http://python.org/sf/1526785  closed by  loewis

IDLE (macosx): Class and Path browsers show Tk menu  (2006-07-06)
       http://python.org/sf/1517996  closed by  ronaldoussoren

Column Number is not visible in MacOSX  (2003-07-09)
       http://python.org/sf/768481  closed by  kbk

Tooltip-window doesn't vanish if...  (2003-07-22)
       http://python.org/sf/775535  closed by  kbk

failure of test_ossaudiodev; elapsed time .1 sec faster  (2006-06-05)
       http://python.org/sf/1501330  closed by  gward

Argument missing from calltip for new-style class init  (2004-09-13)
       http://python.org/sf/1027566  closed by  kbk

length of unicode string changes print behaviour  (2006-02-22)
       http://python.org/sf/1436532  closed by  loewis

exec and eval allocate lots of memory and do not free it  (2006-07-20)
       http://python.org/sf/1525678  closed by  loewis

os.listdir doesn't check error code from FindNextFile  (2006-07-18)
       http://python.org/sf/1524310  closed by  loewis

os.path.abspath() / os.chdir() buggy with unicode paths  (2005-09-07)
       http://python.org/sf/1283895  closed by  loewis

traceback now masks some string exceptions  (2006-06-30)
       http://python.org/sf/1515163  closed by  gbrandl

sys.ps1 not protected in EditorWindow.py  (2004-08-16)
       http://python.org/sf/1010370  closed by  kbk

EditorWindow demo causes attr-error  (2006-01-28)
       http://python.org/sf/1417598  closed by  kbk

Building 2.5a1 requires python  (2006-04-06)
       http://python.org/sf/1465408  closed by  anthonybaxter

New / Reopened RFE
__________________

New sequences for Unicode classes needed  (2006-07-25)
       http://python.org/sf/1528154  opened by  gmarketer

RFE Closed
__________

Add Windows 9x/ME (lack of) support information to README.TX  (2006-06-22)
       http://python.org/sf/1510853  closed by  loewis


From collinw at gmail.com  Thu Jul 27 04:24:37 2006
From: collinw at gmail.com (Collin Winter)
Date: Wed, 26 Jul 2006 22:24:37 -0400
Subject: [Python-Dev] Which version of distutils to ship with Python 2.5?
Message-ID: <43aa6ff70607261924w65e01325o854f87d4c109026a@mail.gmail.com>

Is it intentional that Python 2.5 is (currently) shipping with
distutils 2.4.0, while Python 2.4 (at least 2.4.1, 2.4.2 and 2.4.3)
shipped with distutils 2.4.1? Judging from my own tests, distutils
2.4.1 fixed several bugs that some of my test suites depend on (the
fixes, not the bugs ; ).

Thanks,
Collin Winter

From david.nospam.hopwood at blueyonder.co.uk  Thu Jul 27 04:19:40 2006
From: david.nospam.hopwood at blueyonder.co.uk (David Hopwood)
Date: Thu, 27 Jul 2006 03:19:40 +0100
Subject: [Python-Dev] Internal namespace proposal
In-Reply-To: <bbaeab100607191650j4584de2cge554274e100f125e@mail.gmail.com>
References: <Pine.LNX.4.58.0607191811400.31087@server1.LFW.org>
	<bbaeab100607191650j4584de2cge554274e100f125e@mail.gmail.com>
Message-ID: <44C822BC.1070204@blueyonder.co.uk>

[This message is cc:d to the e-lang list, but please take any replies to
python-dev at python.org.]

Brett Cannon wrote:
> On 7/19/06, Ka-Ping Yee <cap-talk at zesty.ca> wrote:
> 
>> OMG!!!  Is all i can say at the moment.  Very excited.

This is very encouraging. Thanks to ?!ng, Michael Chermside and others for making
the case for capabilities.

> Also realize that I am using object-capabilities to secure the interpreter,
> not objects.  That will be enough of a challenge to do for now.  Who knows,
> maybe some day Python can support object-capabilities at the object level,
> but for now I am just trying to isolate and protect individual interpreters
> in the same process.

I think that the alternative of providing object-granularity protection domains
straight away is more practical than you suggest, and I'd like to at least make
sure that this possibility has been thoroughly explored.

Below is a first-cut proposal for enforcing namespace restrictions, i.e. support
for non-public attributes and methods, on Python objects and modules. It is not
sufficient by itself to provide capability security, but it could be the
basis for doing that at object granularity.

(Note that this proposal would only affect sandboxed/restricted interpreters,
at least for the time being. The encapsulation it provides is also useful
for reasons other than security, and I think there is nothing about it that
would be unreasonable to apply to an unrestricted interpreter, but for
compatibility, that would have to be enabled by a __future__ option or similar.)


Internal namespace proposal
===========================

Existing Python code tends to use a convention where the names of attributes
and methods intended only for internal use are prefixed by '_'. This convention
comes from PEP 8 <http://www.python.org/dev/peps/pep-0008/>, which says:

#  In addition, the following special forms using leading or trailing
#  underscores are recognized (these can generally be combined with any case
#  convention):
#
#   - _single_leading_underscore: weak "internal use" indicator.  E.g. "from M
#     import *" does not import objects whose name starts with an underscore.
#
#   - single_trailing_underscore_: used by convention to avoid conflicts with
#     Python keyword, e.g.
#
#     Tkinter.Toplevel(master, class_='ClassName')
#
#   - __double_leading_underscore: when naming a class attribute, invokes name
#     mangling (inside class FooBar, __boo becomes _FooBar__boo; see below).
#
#   - __double_leading_and_trailing_underscore__: "magic" objects or
#     attributes that live in user-controlled namespaces.  E.g. __init__,
#     __import__ or __file__.  Never invent such names; only use them
#     as documented.

I propose that the "internal" status of names beginning with _ (including
those beginning with __) should be enforced in restricted interpreters. This
is better than introducing a new annotation, because it will do the right
thing for existing code that follows this part of PEP 8.

More precisely:

  A restricted interpreter refuses access to any object attribute or method
  with a name beginning with '_' (by throwing a new exception type
  'InternalAccessException'), unless the access is from a method and its
  static target is that method's first argument variable.

  Also, a restricted interpreter refuses access to any module-global
  variable or module-global function with a name beginning with '_' (by
  throwing 'InternalAccessException'), unless the access is statically from
  the same module.


(A method's first argument is usually called 'self', but that's just a convention.
By "static target", I mean that to access an internal attribute _foo in a
method with first argument 'self', you must write "self._foo"; attempting to
access "x._foo" will fail even if 'x' happens to be the same object as 'self'.
This allows such accesses to be reported at compile-time, rather than only at
run-time.)

I am using the term "internal" rather than "private" or "protected", because
these semantics are not the same as either "private" or "protected" in C++ or
Java. In Python with this change, an object can only access its own internal
methods and attributes. In C++ and Java, an object can access private and protected
members of other objects of the same class. The rationale for this difference is
explained below.

The use of _single vs __double underscores encodes a useful distinction that
would not change. Ignoring the point in the previous paragraph, a _single
underscore is similar to "protected" in languages like C++ and Java, while a
__double underscore is similar to "private". This is purely a consequence of
the name mangling: if a class X and its subclass Y both name an attribute
__foo, then we will end up with two attributes _X__foo and _Y__foo in instances
of Y, which is the desired behaviour for private attributes. In the case of an
attribute called _foo, OTOH, there can be only one such attribute per object,
which is the desired behaviour for protected attributes. The name mangling
also ensures that an object will not *accidentally* access a private attribute
inherited from a superclass.

However, in the same example, an instance of Y can still deliberately access
the copy of the attribute inherited from X by specifying _X__foo. There is no
security problem here, because Y cannot do anything as a result that it could
not have done by copying X's code, rather than inheriting from it. Notice that
this is only true because we restrict an object to only accessing its own
internal attributes and methods; if we followed C++'s semantics where an object
can access protected members of any superclass, this would break security.

(Java solves this problem by applying a more complicated access rule for
protected members, which I considered to be unintuitive. More details on request.)

__dict__ is an internal attribute. This means that an object can only directly
reflect on itself. I know that there are other means of reflection (e.g. using
the 'inspect' module); blocking these or making them safe is a separate issue.

If desired, it would be safe to add a 'publicdict' attribute to each object, or
a 'publicdict(object)' built-in. This would return a *read-only* dict, probably
created lazily if needed, giving access only to public (non-internal) attributes
and methods.

__init__ is an internal method. This is as it should be, because it should not
be possible to call __init__ on an existing object; only to have __init__
implicitly called when a new object is constructed.

__repr__ and __str__ are internal under these rules, and probably shouldn't be.
Existing classes may expose private state in the strings returned by __repr__
or __str__, but in principle, there is nothing unsafe about being able to
convert the public state of an object to a string. OTOH, this functionality
is usually accessed via the built-ins 'repr' and 'str', which we could perhaps
allow to access '__repr__' and '__str__' as a special case.

-- 
David Hopwood <david.nospam.hopwood at blueyonder.co.uk>




From nnorwitz at gmail.com  Thu Jul 27 05:43:38 2006
From: nnorwitz at gmail.com (Neal Norwitz)
Date: Wed, 26 Jul 2006 20:43:38 -0700
Subject: [Python-Dev] [Windows, buildbot] kill_python.c mystery
In-Reply-To: <1f7befae0607261702o6d943f52x701be0d402345f65@mail.gmail.com>
References: <1f7befae0607261702o6d943f52x701be0d402345f65@mail.gmail.com>
Message-ID: <ee2a432c0607262043i323ae8d5k8fdc81ca0b7e6b39@mail.gmail.com>

On 7/26/06, Tim Peters <tim.peters at gmail.com> wrote:
>
> Today I noticed this happened when the buildbot started to run tests,
> and I'm 100% sure it's due to this code in
> Tools/buildbot/kill_python.c (the buildbot log files showed that
> kill_python.c killed /some/ Python process, and the Python running
> release-build tests in my sandbox was the only plausible candidate):
>
>         if ((strstr(path, "build\\pcbuild\\python_d.exe") != NULL) ||
>             (strstr(path, "build\\python.exe") != NULL)) {
>                 printf("Terminating %s (pid %d)\n", path, pids[i]);
>                 if (!TerminateProcess(hProcess, 1)) {
>
> The second clause in the first `if` looks for a substring match on:
>
>     build\python.exe
>
> and that just happens to match a suffix of:
>
>     C:\Code\python\PCbuild\python.exe
>
> which is the release-build Python I happen to be running in my sandbox.
>
> Why is the second clause there?  That is, are we /trying/ to kill a
> release-build Python running from the user's sandbox, and if so why?

No, I don't believe that was the intent.  The exe on cygwin uses the
unix convention, not the Windows convention for the filename.  ie,
either a debug
or release build on cygwin are both called python.exe.

So the second clause is there to kill the process when it's running
under cygwin.
It's interesting that the process appears to be running as
./python.exe, but build shows up in filename.  From that I deduce that
it must contain the complete path.  Assuming that is true, we can
change the code to ensure that build is a directory since that's what
buildbot does (add the leading \\):

>             (strstr(path, "\\build\\python.exe") != NULL)) {

I tested this change with a different path, so I believe it will work
fine and not catch PCbuild.  I'll check in this change and add some
comments.

n

From nnorwitz at gmail.com  Thu Jul 27 06:08:40 2006
From: nnorwitz at gmail.com (Neal Norwitz)
Date: Wed, 26 Jul 2006 21:08:40 -0700
Subject: [Python-Dev] Release manager pronouncement needed: PEP 302 Fix
In-Reply-To: <5.1.1.6.0.20060726142456.036d9008@sparrow.telecommunity.com>
References: <5.1.1.6.0.20060726142456.036d9008@sparrow.telecommunity.com>
Message-ID: <ee2a432c0607262108n2599f9aere07bae83535179b1@mail.gmail.com>

What is the behaviour that was added which broke compliance?  What is
the benefit of the behaviour?

>From your description of fixing the problem, it seems there's some
risk invovled as it's modiyfing import.c, plus adding new features.
What is your recommendation?

n
--

On 7/26/06, Phillip J. Eby <pje at telecommunity.com> wrote:
> I posted last week about a need-for-speed patch that broke PEP 302
> compliance, and asked if it should be fixed or reverted.  I got exactly one
> response which said "yes, it should be fixed or reverted", which
> unfortunately didn't answer my question as to which one we should do.  :)
>
> If we don't revert it, there are two ways to fix it.  One is to just change
> PEP 302 so that the behavior is unbroken by definition.  :)  The other is
> to actually go ahead and fix it by adding PathImporter and NullImporter
> types to import.c, along with a factory function on sys.path_hooks to
> create them.  (This would've been the PEP-compliant way to implement the
> need-for-speed patch.)
>
> So, "fix" by documentation, fix by fixing, or fix by reverting?  Which
> should it be?
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/nnorwitz%40gmail.com
>

From martin at v.loewis.de  Thu Jul 27 08:32:39 2006
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Thu, 27 Jul 2006 08:32:39 +0200
Subject: [Python-Dev] [Windows, buildbot] kill_python.c mystery
In-Reply-To: <1f7befae0607261702o6d943f52x701be0d402345f65@mail.gmail.com>
References: <1f7befae0607261702o6d943f52x701be0d402345f65@mail.gmail.com>
Message-ID: <44C85E07.3060707@v.loewis.de>

Tim Peters wrote:
> Today I noticed this happened when the buildbot started to run tests,
> and I'm 100% sure it's due to this code in
> Tools/buildbot/kill_python.c 

Didn't you know that you signed in to run arbitrary viruses, worms, and
trojan horses when you added your machine to the buildbot infrastructure
:-? You just haven't seen buildbot erasing your hard disk and filling
your coffee machine with tea, yet.

> 	    (strstr(path, "build\\python.exe") != NULL)) {
> Why is the second clause there?

That's for Cygwin (i.e. Anthony Baxter's machine). As Neal suggests,
preceding the executable path with another backslash should solve
this problem.

As a related note, this entire approach will also manage to kill
python.exe from an unrelated buildbot installation, e.g. a 2.4
build job might kill python.exe from the trunk. This actually helped
when I tried to get the Cygwin slave to get unstuck, and shouldn't
do harm since we currently don't run to builds on the same slave
simultaneously, but could be surprising when parallel builds
are activated some day.

Sorry for messing with your machine,

Martin

From martin at v.loewis.de  Thu Jul 27 08:40:50 2006
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Thu, 27 Jul 2006 08:40:50 +0200
Subject: [Python-Dev] Which version of distutils to ship with Python 2.5?
In-Reply-To: <43aa6ff70607261924w65e01325o854f87d4c109026a@mail.gmail.com>
References: <43aa6ff70607261924w65e01325o854f87d4c109026a@mail.gmail.com>
Message-ID: <44C85FF2.3030701@v.loewis.de>

Collin Winter wrote:
> Is it intentional that Python 2.5 is (currently) shipping with
> distutils 2.4.0, while Python 2.4 (at least 2.4.1, 2.4.2 and 2.4.3)
> shipped with distutils 2.4.1? Judging from my own tests, distutils
> 2.4.1 fixed several bugs that some of my test suites depend on (the
> fixes, not the bugs ; ).

Are these bugs not fixed in the distutils that shipped with Python 2.5b2?

In any case, I bumped the version number to 2.5, according to the policy
discussed in

http://mail.python.org/pipermail/distutils-sig/2005-January/004368.html

Thanks for pointing this out.

Regards,
Martin

From anthony at interlink.com.au  Thu Jul 27 09:01:53 2006
From: anthony at interlink.com.au (Anthony Baxter)
Date: Thu, 27 Jul 2006 17:01:53 +1000
Subject: [Python-Dev] Which version of distutils to ship with Python 2.5?
In-Reply-To: <44C85FF2.3030701@v.loewis.de>
References: <43aa6ff70607261924w65e01325o854f87d4c109026a@mail.gmail.com>
	<44C85FF2.3030701@v.loewis.de>
Message-ID: <200607271701.58223.anthony@interlink.com.au>

On Thursday 27 July 2006 16:40, Martin v. L?wis wrote:
> Collin Winter wrote:
> > Is it intentional that Python 2.5 is (currently) shipping with
> > distutils 2.4.0, while Python 2.4 (at least 2.4.1, 2.4.2 and
> > 2.4.3) shipped with distutils 2.4.1? Judging from my own tests,
> > distutils 2.4.1 fixed several bugs that some of my test suites
> > depend on (the fixes, not the bugs ; ).
>
> Are these bugs not fixed in the distutils that shipped with Python
> 2.5b2?
>
> In any case, I bumped the version number to 2.5, according to the
> policy discussed in
>

Could this not simply use the Python version number directly, instead?
Separate version numbers only make sense if the package is separately 
distributed - and even then, something like Barry's setup for the 
email package could keep that version number out of the Python trunk.

Fiddly little version numbers scattered throughout the standard 
library == pain.

Anthony

From greg.ewing at canterbury.ac.nz  Thu Jul 27 09:14:27 2006
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Thu, 27 Jul 2006 19:14:27 +1200
Subject: [Python-Dev] Internal namespace proposal
In-Reply-To: <44C822BC.1070204@blueyonder.co.uk>
References: <Pine.LNX.4.58.0607191811400.31087@server1.LFW.org>
	<bbaeab100607191650j4584de2cge554274e100f125e@mail.gmail.com>
	<44C822BC.1070204@blueyonder.co.uk>
Message-ID: <44C867D3.90306@canterbury.ac.nz>

David Hopwood wrote:

>   A restricted interpreter refuses access to any object attribute or method
>   with a name beginning with '_' (by throwing a new exception type
>   'InternalAccessException'), unless the access is from a method and its
>   static target is that method's first argument variable.

What's to stop

   def my_naughty_method(self):
     self = some_protected_object
     self._a_special_attribute = some_naughty_value

> __init__ is an internal method. This is as it should be, because it should not
> be possible to call __init__ on an existing object; only to have __init__
> implicitly called when a new object is constructed.

What about calling an inherited __init__ method?
Your proposed rule would seem to disallow

   BaseClass.__init__(self, ...)

--
Greg

From syfou at users.sourceforge.net  Thu Jul 27 09:39:39 2006
From: syfou at users.sourceforge.net (Sylvain Fourmanoit)
Date: Thu, 27 Jul 2006 03:39:39 -0400 (EDT)
Subject: [Python-Dev] New miniconf module
In-Reply-To: <5.1.1.6.0.20060726180017.0262c990@sparrow.telecommunity.com>
References: <Pine.LNX.4.64.0607261613130.6037@sylvain>
	<Pine.LNX.4.64.0607261402410.5956@sylvain>
	<44C7B8C2.1060904@blueyonder.co.uk>
	<Pine.LNX.4.64.0607261613130.6037@sylvain>
	<5.1.1.6.0.20060726180017.0262c990@sparrow.telecommunity.com>
Message-ID: <Pine.LNX.4.64.0607270200300.6000@sylvain>

An updated version is now available, based to the feedback of Phillip J. 
Eby and David Hopwood (stand-alone module[1], patch[2]):

- the module is now reentrant
- the sloppy case with Name nodes is now covered properly
- the node lookup procedure was optimized, leading to a 20% speed 
increase on the average case... Phillip, I was wrong to doubt you. ;-) 
There is undoubtedly still room from improvement, but that's a good start.

>> But I agree this looks a lot like JSON, since ecmascript syntax for
>> literals looks a lot like the one of Python... For the same reasons there
>> is a need for JSON, I think having something like miniconf in the
>> standard lib would benefit the users.
>
> Actually, I would see more reason to include JSON in the standard library, 
> since it's at least something approaching an internet protocol these days.

Having JSON there would indeed be nice: In fact, I recall being initially 
surprised it was not supported by the standard library.

But is there a need to choose? Why not have both? The miniconf approach 
has its advantages and differences:

- The code is short and simple. Since all the real work is performed by 
the Python parser, very little has to be done on top of that: it should be 
easy to maintain, and will benefit of all the future work (patches, etc.) 
that will be integrated to it in the future.

- The source it works on is valid Python source, which seems to be a plus 
for a dynamic, reflexive language such as Python... Things such as this 
will work:

>>> from miniconf import dump
>>> file('test.py','w').write(dump({'spam': 1}))
>>> import test

I know this in not the best example, but you get the idea...

- Unlike JSON, miniconf is not introducing any new notation or syntax at 
all: it uses a strict, well defined subset of the Python grammar that 
every Python user is already familiar with; it is in no way a 
data-interchange format, but it feels pretty natural in a all-python 
environment... In that sense, it is well documented and standardized.

- Am I missing something, or is JSON not supporting comments inside the 
parse tree? That's not really convenient for storage of configuration 
information.

Anyway, if I had to choose between the two, I would definitively want 
simplejson part of the standard library well before miniconf, since it can 
be used in so many different situations, but I wouldn't 
choose JSON as a configuration format given the choice to use the Python 
notation employed by miniconf either.

Yours,

-- 
Sylvain <syfou at users.sourceforge.net>

Nobody said computers were going to be polite.

[1]http://cheeseshop.python.org/pypi?:action=display&name=miniconf&version=1.1.0
[2]http://sourceforge.net/tracker/index.php?func=detail&aid=1527597&group_id=5470&atid=355470

From theller at python.net  Thu Jul 27 09:39:59 2006
From: theller at python.net (Thomas Heller)
Date: Thu, 27 Jul 2006 09:39:59 +0200
Subject: [Python-Dev] Patch for building ctypes on more OpenBSD target
	platforms
Message-ID: <ea9qke$nnu$1@sea.gmane.org>

I've uploaded a patch sent to me in private email by Damien Miller, who
is packaging ctypes for the OpenBSD port tree.

I'm requesting permission to commit this for Python 2.5.

http://python.org/sf/1529514

Thanks,
Thomas


From g.brandl at gmx.net  Thu Jul 27 10:47:30 2006
From: g.brandl at gmx.net (Georg Brandl)
Date: Thu, 27 Jul 2006 10:47:30 +0200
Subject: [Python-Dev] Release manager pronouncement needed: PEP 302 Fix
In-Reply-To: <ee2a432c0607262108n2599f9aere07bae83535179b1@mail.gmail.com>
References: <5.1.1.6.0.20060726142456.036d9008@sparrow.telecommunity.com>
	<ee2a432c0607262108n2599f9aere07bae83535179b1@mail.gmail.com>
Message-ID: <ea9ugp$3nu$1@sea.gmane.org>

Neal Norwitz wrote:
> What is the behaviour that was added which broke compliance?  What is
> the benefit of the behaviour?

sys.path_importer_cache is now used to cache if a real directory
exists on the filesystem. Previously, a value of None for a given
sys.path entry told find_module that no import hook exist, so it should
look for a filesystem directory. Now, the entry is set to True if that
directory really exists and to False if it doesn't exist, thus saving
quite a few open() calls to files in these not existing dirs.

>>From your description of fixing the problem, it seems there's some
> risk invovled as it's modiyfing import.c, plus adding new features.
> What is your recommendation?

I would prefer fixing the docs. Importing from filesystem directories
is common enough to be special cased.

Georg


From arigo at tunes.org  Thu Jul 27 12:33:31 2006
From: arigo at tunes.org (Armin Rigo)
Date: Thu, 27 Jul 2006 12:33:31 +0200
Subject: [Python-Dev] New miniconf module
In-Reply-To: <Pine.LNX.4.64.0607270200300.6000@sylvain>
References: <Pine.LNX.4.64.0607261613130.6037@sylvain>
	<Pine.LNX.4.64.0607261402410.5956@sylvain>
	<44C7B8C2.1060904@blueyonder.co.uk>
	<Pine.LNX.4.64.0607261613130.6037@sylvain>
	<5.1.1.6.0.20060726180017.0262c990@sparrow.telecommunity.com>
	<Pine.LNX.4.64.0607270200300.6000@sylvain>
Message-ID: <20060727103331.GA31912@code0.codespeak.net>

Hi,

On Thu, Jul 27, 2006 at 03:39:39AM -0400, Sylvain Fourmanoit wrote:
> Having JSON there would indeed be nice: In fact, I recall being initially 
> surprised it was not supported by the standard library.
> 
> But is there a need to choose? Why not have both? The miniconf approach 
> has its advantages and differences:

I support this point of view: miniconf fills the hole that the stdlib
leaves for a safe and cross-version dumper/loader for simple objects
using the Python syntax.  In the same spirit, maybe it could be slightly
re-oriented towards a dumper/loader for more than config files; for
example, it could provide a safe inverse of repr() for common built-in
types.  Such a functionality has been discussed here a few times if I
remember correctly, but the code in miniconf is very close to providing
it.


A bientot,

Armin

From arigo at tunes.org  Thu Jul 27 12:39:20 2006
From: arigo at tunes.org (Armin Rigo)
Date: Thu, 27 Jul 2006 12:39:20 +0200
Subject: [Python-Dev] Release manager pronouncement needed: PEP 302 Fix
In-Reply-To: <5.1.1.6.0.20060726142456.036d9008@sparrow.telecommunity.com>
References: <5.1.1.6.0.20060726142456.036d9008@sparrow.telecommunity.com>
Message-ID: <20060727103920.GB31912@code0.codespeak.net>

Hi Phillip,

On Wed, Jul 26, 2006 at 02:40:27PM -0400, Phillip J. Eby wrote:
> If we don't revert it, there are two ways to fix it.  One is to just change 
> PEP 302 so that the behavior is unbroken by definition.  :)  The other is 
> to actually go ahead and fix it by adding PathImporter and NullImporter 
> types to import.c, along with a factory function on sys.path_hooks to 
> create them.  (This would've been the PEP-compliant way to implement the 
> need-for-speed patch.)
> 
> So, "fix" by documentation, fix by fixing, or fix by reverting?  Which 
> should it be?

"fix" by changing the definition looks like a bad idea to me.  The
import logic is already extremely complicated and delicate, any change
to it is bound to break *some* code somewhere.

So although import.c is already by far the longest piece of code around,
I think that we need a patch doing the "right" thing, or else revert.


A bientot,

Armin

From g.brandl at gmx.net  Thu Jul 27 12:52:33 2006
From: g.brandl at gmx.net (Georg Brandl)
Date: Thu, 27 Jul 2006 12:52:33 +0200
Subject: [Python-Dev] Release manager pronouncement needed: PEP 302 Fix
In-Reply-To: <20060727103920.GB31912@code0.codespeak.net>
References: <5.1.1.6.0.20060726142456.036d9008@sparrow.telecommunity.com>
	<20060727103920.GB31912@code0.codespeak.net>
Message-ID: <eaa5rd$pe2$1@sea.gmane.org>

Armin Rigo wrote:
> Hi Phillip,
> 
> On Wed, Jul 26, 2006 at 02:40:27PM -0400, Phillip J. Eby wrote:
>> If we don't revert it, there are two ways to fix it.  One is to just change 
>> PEP 302 so that the behavior is unbroken by definition.  :)  The other is 
>> to actually go ahead and fix it by adding PathImporter and NullImporter 
>> types to import.c, along with a factory function on sys.path_hooks to 
>> create them.  (This would've been the PEP-compliant way to implement the 
>> need-for-speed patch.)
>> 
>> So, "fix" by documentation, fix by fixing, or fix by reverting?  Which 
>> should it be?
> 
> "fix" by changing the definition looks like a bad idea to me.  The
> import logic is already extremely complicated and delicate, any change
> to it is bound to break *some* code somewhere.

Though beta1 and beta2 shipped with this change nobody reported any bug that
could be linked to it. sys.path_importer_cache is quite an internal thing and
most code, even import hooks, shouldn't have to deal with it.

Georg


From arigo at tunes.org  Thu Jul 27 13:30:17 2006
From: arigo at tunes.org (Armin Rigo)
Date: Thu, 27 Jul 2006 13:30:17 +0200
Subject: [Python-Dev] Internal namespace proposal
In-Reply-To: <44C822BC.1070204@blueyonder.co.uk>
References: <Pine.LNX.4.58.0607191811400.31087@server1.LFW.org>
	<bbaeab100607191650j4584de2cge554274e100f125e@mail.gmail.com>
	<44C822BC.1070204@blueyonder.co.uk>
Message-ID: <20060727113016.GC31912@code0.codespeak.net>

Hi David,

Your proposal is too vague to be useful.  In Python I would not feel
that any compiler-enforced restrictions are going to be too restrictive,
and so I believe that your approach is not viable, but I cannot give you
many concrete examples of why before you come up with a more concrete
specification.

More importantly, this is going to depend critically on a restricted
interpreter in the first place, in the sense of Brett, but the safety of
your proposal depends on the restricted interpreter forbidding many
operations that it would not otherwise forbid for its original goal.
For example, in Brett's use case there is no need to prevent reading the
'func_globals' attribute of function objects, but if that's allowed,
then accessing any _attribute of any module is easy.

About special methods: how do built-in functions like str(), int(), and
so on, know in which context they are called?  Surely you don't propose
that '2+3' should be invalid because it accesses the hidden attribute '2
.__add__' ?

How would you formulate a rule in term on Python's attribute look-up
algorithm to prevent the following trivial attack? :

x.py:
    
    # supposedly secure?

    _hidden = [1,2,3]

    class A:
        def __init__(self):
            self._authorized = ...
        def read(self):
            if not self._authorized:
                raise Forbidden
            return _hidden

attack.py:

    import x
    class B(x.A):
        def __init__(self):
            self._authorized = True

    b = B()
    print b.read()   # => [1,2,3]

On any real-life example I'm sure that hacks like overriding selected
methods on the instance itself would allow an attacker to confuse the
remaining methods enough to leak hidden information.

Here is a metaclass attack against the rule "self._attr is only allowed
if syntactically inside the class definition of the exact class of
self":

    class SupposedlySecure(object):
        _hidden = [1,2,3]


    class MetaAttack(type):
        def read(self):
            return self._hidden   # seen as an instance attribute

    class Attack(SupposedlySecure):
        __metaclass__ = MetaAttack

    print Attack.read()


A bientot,

Armin.

From amk at amk.ca  Thu Jul 27 14:41:37 2006
From: amk at amk.ca (A.M. Kuchling)
Date: Thu, 27 Jul 2006 08:41:37 -0400
Subject: [Python-Dev] Release manager: pdb bugfix incompatibility
Message-ID: <20060727124137.GA4071@rogue.amk.ca>

Bug #1526834: if you do 'b f(' in pdb, the debugger crashes.  This bug
stems from pdb just sticking the string in a regex and compiling it.

 cre = re.compile(r'def\s+%s\s*[(]' % funcname)

A side effect of this is that 'b f()' works to match the function 'f',
because the empty parens are legal regex syntax declaring an empty
group that matches the null string.

It's easy to fix the crash by doing re.escape(funcname).  But this
means that 'b f()' will no longer find the function 'f' -- it will
look for 'f(' followed by another paren, and won't find it.

Should this be fixed by the re.escape(), or should the fix attempt to
keep 'b f()' working?  The pdb documentation doesn't claim that 'b
f()' should work.

--amk

From fuzzyman at voidspace.org.uk  Thu Jul 27 13:46:04 2006
From: fuzzyman at voidspace.org.uk (Michael Foord)
Date: Thu, 27 Jul 2006 12:46:04 +0100
Subject: [Python-Dev] New miniconf module
In-Reply-To: <20060727103331.GA31912@code0.codespeak.net>
References: <Pine.LNX.4.64.0607261613130.6037@sylvain>	<Pine.LNX.4.64.0607261402410.5956@sylvain>	<44C7B8C2.1060904@blueyonder.co.uk>	<Pine.LNX.4.64.0607261613130.6037@sylvain>	<5.1.1.6.0.20060726180017.0262c990@sparrow.telecommunity.com>	<Pine.LNX.4.64.0607270200300.6000@sylvain>
	<20060727103331.GA31912@code0.codespeak.net>
Message-ID: <44C8A77C.6060407@voidspace.org.uk>

Armin Rigo wrote:
> Hi,
>
> On Thu, Jul 27, 2006 at 03:39:39AM -0400, Sylvain Fourmanoit wrote:
>   
>> Having JSON there would indeed be nice: In fact, I recall being initially 
>> surprised it was not supported by the standard library.
>>
>> But is there a need to choose? Why not have both? The miniconf approach 
>> has its advantages and differences:
>>     
>
> I support this point of view: miniconf fills the hole that the stdlib
> leaves for a safe and cross-version dumper/loader for simple objects
> using the Python syntax.  In the same spirit, maybe it could be slightly
> re-oriented towards a dumper/loader for more than config files; for
> example, it could provide a safe inverse of repr() for common built-in
> types.  Such a functionality has been discussed here a few times if I
> remember correctly, but the code in miniconf is very close to providing
> it.
>   
ConfigObj [1] gained an 'unrepr' mode a while back. The code is simple, 
and originally came from CherryPy.

It can (safely) unrepr basic datatypes.

import compiler

def getObj(s):
    s = "a=" + s
    p = compiler.parse(s)
    return p.getChildren()[1].getChildren()[0].getChildren()[1]

class UnknownType(Exception):
    pass

class Builder:
   
    def build(self, o):
        m = getattr(self, 'build_' + o.__class__.__name__, None)
        if m is None:
            raise UnknownType(o.__class__.__name__)
        return m(o)
   
    def build_List(self, o):
        return map(self.build, o.getChildren())
   
    def build_Const(self, o):
        return o.value
   
    def build_Dict(self, o):
        d = {}
        i = iter(map(self.build, o.getChildren()))
        for el in i:
            d[el] = i.next()
        return d
   
    def build_Tuple(self, o):
        return tuple(self.build_List(o))
   
    def build_Name(self, o):
        if o.name == 'None':
            return None
        if o.name == 'True':
            return True
        if o.name == 'False':
            return False
       
        # An undefinted Name
        raise UnknownType('Undefined Name')
   
    def build_Add(self, o):
        real, imag = map(self.build_Const, o.getChildren())
        try:
            real = float(real)
        except TypeError:
            raise UnknownType('Add')
        if not isinstance(imag, complex) or imag.real != 0.0:
            raise UnknownType('Add')
        return real+imag
   
    def build_Getattr(self, o):
        parent = self.build(o.expr)
        return getattr(parent, o.attrname)
   
    def build_UnarySub(self, o):
        return -self.build_Const(o.getChildren()[0])
   
    def build_UnaryAdd(self, o):
        return self.build_Const(o.getChildren()[0])

def unrepr(s):
    if not s:
        return s
    return Builder().build(getObj(s))


HTH

Michael Foord

[1] http://www.voidspace.org.uk/python/configobj.html


>
> A bientot,
>
> Armin
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk
>
>   


From arigo at tunes.org  Thu Jul 27 15:54:17 2006
From: arigo at tunes.org (Armin Rigo)
Date: Thu, 27 Jul 2006 15:54:17 +0200
Subject: [Python-Dev] New miniconf module
In-Reply-To: <44C8A77C.6060407@voidspace.org.uk>
References: <Pine.LNX.4.64.0607261613130.6037@sylvain>
	<Pine.LNX.4.64.0607261402410.5956@sylvain>
	<44C7B8C2.1060904@blueyonder.co.uk>
	<Pine.LNX.4.64.0607261613130.6037@sylvain>
	<5.1.1.6.0.20060726180017.0262c990@sparrow.telecommunity.com>
	<Pine.LNX.4.64.0607270200300.6000@sylvain>
	<20060727103331.GA31912@code0.codespeak.net>
	<44C8A77C.6060407@voidspace.org.uk>
Message-ID: <20060727135417.GA21563@code0.codespeak.net>

Hi Michael,

On Thu, Jul 27, 2006 at 12:46:04PM +0100, Michael Foord wrote:
> > leaves for a safe and cross-version dumper/loader for simple objects
> > using the Python syntax.  In the same spirit, maybe it could be slightly
> > re-oriented towards a dumper/loader for more than config files; for
> > example, it could provide a safe inverse of repr() for common built-in
> > types.  Such a functionality has been discussed here a few times if I
> > remember correctly, but the code in miniconf is very close to providing
> > it.
> >   
> ConfigObj [1] gained an 'unrepr' mode a while back. The code is simple, 
> and originally came from CherryPy.

I'm sure, but my point was that the discussed miniconf already contains
mostly the same code already, so I suggested that it would be a
worthwhile addition to it, in its stdlib-hole-filler role.

If it goes in that direction, I'd suggest to rename the module to give
it a name closer to existing persistence-related modules already in the
stdlib.


Armin

From g.brandl at gmx.net  Thu Jul 27 17:40:57 2006
From: g.brandl at gmx.net (Georg Brandl)
Date: Thu, 27 Jul 2006 17:40:57 +0200
Subject: [Python-Dev] uuid test suite failing
Message-ID: <eaamnu$mrq$1@sea.gmane.org>

The UUID test suite, which wasn't run by regrtest.py until now, is
now failing on some buildbots (and my machine). This should be fixed
before releasing something.

Georg


From pje at telecommunity.com  Thu Jul 27 17:58:15 2006
From: pje at telecommunity.com (Phillip J. Eby)
Date: Thu, 27 Jul 2006 11:58:15 -0400
Subject: [Python-Dev] Release manager pronouncement needed: PEP 302 Fix
In-Reply-To: <eaa5rd$pe2$1@sea.gmane.org>
References: <20060727103920.GB31912@code0.codespeak.net>
	<5.1.1.6.0.20060726142456.036d9008@sparrow.telecommunity.com>
	<20060727103920.GB31912@code0.codespeak.net>
Message-ID: <5.1.1.6.0.20060727114924.0262c980@sparrow.telecommunity.com>

At 12:52 PM 7/27/2006 +0200, Georg Brandl wrote:
>Armin Rigo wrote:
> > Hi Phillip,
> >
> > On Wed, Jul 26, 2006 at 02:40:27PM -0400, Phillip J. Eby wrote:
> >> If we don't revert it, there are two ways to fix it.  One is to just 
> change
> >> PEP 302 so that the behavior is unbroken by definition.  :)  The other is
> >> to actually go ahead and fix it by adding PathImporter and NullImporter
> >> types to import.c, along with a factory function on sys.path_hooks to
> >> create them.  (This would've been the PEP-compliant way to implement the
> >> need-for-speed patch.)
> >>
> >> So, "fix" by documentation, fix by fixing, or fix by reverting?  Which
> >> should it be?
> >
> > "fix" by changing the definition looks like a bad idea to me.  The
> > import logic is already extremely complicated and delicate, any change
> > to it is bound to break *some* code somewhere.
>
>Though beta1 and beta2 shipped with this change nobody reported any bug that
>could be linked to it.

Because in at least setuptools' case, you have to be using unzipped 
namespace packages under the right set of circumstances to trigger a propblem.


>sys.path_importer_cache is quite an internal thing

Whose behavior is documented in a PEP.


>  and
>most code, even import hooks, shouldn't have to deal with it.

That doesn't make it unimportant.  It's a visible change in specified 
behavior between Python versions -- precisely the sort of thing that makes 
people mad at us renegade cowboy Python-dev hackers changing their language 
for no apparent reason.  The strftime thing that recently got hashed to 
death here was also an "internal thing" which "most code shouldn't have to 
deal with".

This is precisely how these kinds of problems happen.

So, this needs to either be documented in the What's New document and PEP 
302 at a minimum, or it needs to be reverted, unless somebody wants to 
bless the feature addition to fix it.

I'm willing to write code that makes it PEP 302 compliant, if the release 
manager will bless such an addition.  But if that's not acceptable, then 
somebody needs to produce the necessary documentation updates or revert the 
patch.  It absolutely should not be allowed to remain in *and* undocumented 
because it is a backwards-incompatible change to documented behavior of 
Python for two major releases (2.3 and 2.4).




From g.brandl at gmx.net  Thu Jul 27 18:20:39 2006
From: g.brandl at gmx.net (Georg Brandl)
Date: Thu, 27 Jul 2006 18:20:39 +0200
Subject: [Python-Dev] uuid test suite failing
In-Reply-To: <eaamnu$mrq$1@sea.gmane.org>
References: <eaamnu$mrq$1@sea.gmane.org>
Message-ID: <eaap2c$al$1@sea.gmane.org>

Georg Brandl wrote:
> The UUID test suite, which wasn't run by regrtest.py until now, is
> now failing on some buildbots (and my machine). This should be fixed
> before releasing something.

Okay, after fixing the test on my machine (locale issue) it looks like
some ifconfigs don't like to be called without arguments. "-a" seems to
be supported everywhere though, so I guess it's reasonable to use that
flag on every platform. Any objections?

Georg


From g.brandl at gmx.net  Thu Jul 27 18:24:15 2006
From: g.brandl at gmx.net (Georg Brandl)
Date: Thu, 27 Jul 2006 18:24:15 +0200
Subject: [Python-Dev] Release manager pronouncement needed: PEP 302 Fix
In-Reply-To: <5.1.1.6.0.20060727114924.0262c980@sparrow.telecommunity.com>
References: <20060727103920.GB31912@code0.codespeak.net>	<5.1.1.6.0.20060726142456.036d9008@sparrow.telecommunity.com>	<20060727103920.GB31912@code0.codespeak.net>
	<eaa5rd$pe2$1@sea.gmane.org>
	<5.1.1.6.0.20060727114924.0262c980@sparrow.telecommunity.com>
Message-ID: <eaap95$1bq$1@sea.gmane.org>

Phillip J. Eby wrote:

>>sys.path_importer_cache is quite an internal thing
> 
> Whose behavior is documented in a PEP.

Correct.

>>  and
>>most code, even import hooks, shouldn't have to deal with it.
> 
> That doesn't make it unimportant.  It's a visible change in specified 
> behavior between Python versions -- precisely the sort of thing that makes 
> people mad at us renegade cowboy Python-dev hackers changing their language 
> for no apparent reason.  The strftime thing that recently got hashed to 
> death here was also an "internal thing" which "most code shouldn't have to 
> deal with".
> 
> This is precisely how these kinds of problems happen.
> 
> So, this needs to either be documented in the What's New document and PEP 
> 302 at a minimum, or it needs to be reverted, unless somebody wants to 
> bless the feature addition to fix it.

I agree with you (now). ;)

> I'm willing to write code that makes it PEP 302 compliant, if the release 
> manager will bless such an addition.  But if that's not acceptable, then 
> somebody needs to produce the necessary documentation updates or revert the 
> patch.

A possible third option would be to store the information "this is an invalid
path" somewhere else, that is, an internal dictionary only available to
import.c.

I will write up docs and update the PEP in any case, if the release manager
agrees.

Georg


From amk at amk.ca  Thu Jul 27 18:36:47 2006
From: amk at amk.ca (A.M. Kuchling)
Date: Thu, 27 Jul 2006 12:36:47 -0400
Subject: [Python-Dev] uuid test suite failing
In-Reply-To: <eaamnu$mrq$1@sea.gmane.org>
References: <eaamnu$mrq$1@sea.gmane.org>
Message-ID: <20060727163647.GA4175@rogue.amk.ca>

On Thu, Jul 27, 2006 at 05:40:57PM +0200, Georg Brandl wrote:
> The UUID test suite, which wasn't run by regrtest.py until now, is
> now failing on some buildbots (and my machine). This should be fixed
> before releasing something.

Looking at the failures, there seem to be two problems on Unix variants:
 1) on some, '/sbin/ifconfig' prints a help message; you need 'ifconfig -a'
    to print information about all interfaces.
 2) on Solaris 9 (the only version in the SF compile farm), I can't 
    figure out how to make ifconfig print MAC addresses at all.
    Searching online finds the incantation 'arp <hostname>' to print the 
    MAC.

The XP build fails because it seems to be getting different node IDs 
from different calls.

The cygwin build is very unhappy, for reasons that don't look
connected to the newly-enabled tests.

--amk

From david.nospam.hopwood at blueyonder.co.uk  Thu Jul 27 17:18:01 2006
From: david.nospam.hopwood at blueyonder.co.uk (David Hopwood)
Date: Thu, 27 Jul 2006 16:18:01 +0100
Subject: [Python-Dev] Internal namespace proposal
In-Reply-To: <20060727113016.GC31912@code0.codespeak.net>
References: <Pine.LNX.4.58.0607191811400.31087@server1.LFW.org>
	<bbaeab100607191650j4584de2cge554274e100f125e@mail.gmail.com>
	<44C822BC.1070204@blueyonder.co.uk>
	<20060727113016.GC31912@code0.codespeak.net>
Message-ID: <44C8D929.2010508@blueyonder.co.uk>

Armin Rigo wrote:
> Hi David,
> 
> Your proposal is too vague to be useful.  In Python I would not feel
> that any compiler-enforced restrictions are going to be too restrictive,
> and so I believe that your approach is not viable, but I cannot give you
> many concrete examples of why before you come up with a more concrete
> specification.

The intention was not to require the restrictions to be compiler-enforced;
only to *allow* them to be compiler-enforced.

Code like this, for example:

  def someMethod(self, x):
      if self == x:
          foo(x._internal)

should not have to work.

> More importantly, this is going to depend critically on a restricted
> interpreter in the first place, in the sense of Brett, but the safety of
> your proposal depends on the restricted interpreter forbidding many
> operations that it would not otherwise forbid for its original goal.
> For example, in Brett's use case there is no need to prevent reading the
> 'func_globals' attribute of function objects, but if that's allowed,
> then accessing any _attribute of any module is easy.

I disagree that there is no need to prevent reading func_globals.
func_globals is clearly incompatible with capability security (as are
func_dict and func_closure; also the other function attributes should be
read-only). Functions in a capability language should be opaque.

I don't see that there is any problem with the proposal depending on a
restricted interpreter to prevent access via loopholes such as func_globals,
since that is the main intended context of its use. Remember that Brett's
document stated that protection could only be obtained at interpreter
granularity, rather than object granularity, primarily because objects
have no way to prevent access to their private state.

My intention in describing the basic idea of enforcing the PEP 8 convention
for internal attributes/methods, was to get precisely this kind of feedback
on potential problems. I have already obtained useful feedback (including
yours), and will prepare a more concrete proposal based on it.

> About special methods: how do built-in functions like str(), int(), and
> so on, know in which context they are called? Surely you don't propose
> that '2+3' should be invalid because it accesses the hidden attribute '2
> .__add__' ?

This and other examples have convinced me that names starting and ending with
double underscores should not automatically be considered internal. There
are a few such names that should be internal (e.g. __dict__), but it is
reasonable to treat those as special cases.

> How would you formulate a rule in term on Python's attribute look-up
> algorithm to prevent the following trivial attack? :
> 
> x.py:
>     
>     # supposedly secure?
> 
>     _hidden = [1,2,3]
> 
>     class A:
>         def __init__(self):
>             self._authorized = ...
>         def read(self):
>             if not self._authorized:
>                 raise Forbidden
>             return _hidden
> 
> attack.py:
> 
>     import x
>     class B(x.A):
>         def __init__(self):
>             self._authorized = True
> 
>     b = B()
>     print b.read()   # => [1,2,3]

Inheritance should be defined as though the code of inherited methods and
attributes were copied into the subclass (with global accesses updated to
point to the original module). IOW, B acts as though it is defined like this:

attack.py:

  class B(x.A):
      def __init__(self):
          self._authorized = True
      def read(self):
          if not self._authorized:
              raise Forbidden
          return x._hidden

Since x._hidden is not accessible from attack.py, the attack fails.

> On any real-life example I'm sure that hacks like overriding selected
> methods on the instance itself would allow an attacker to confuse the
> remaining methods enough to leak hidden information.

Yes, Java was subject to many attacks of this type. However, a code-copying
semantics for inheritance prevents all of them, by ensuring that a class
cannot do anything by inheritance that it could not do without it.

> Here is a metaclass attack against the rule "self._attr is only allowed
> if syntactically inside the class definition of the exact class of
> self":
> 
>     class SupposedlySecure(object):
>         _hidden = [1,2,3]
> 
>     class MetaAttack(type):
>         def read(self):
>             return self._hidden   # seen as an instance attribute
> 
>     class Attack(SupposedlySecure):
>         __metaclass__ = MetaAttack
> 
>     print Attack.read()

Metaclasses are a reflective feature; almost all such features would have
to be limited in restricted interpreters.

-- 
David Hopwood <david.nospam.hopwood at blueyonder.co.uk>



From david.nospam.hopwood at blueyonder.co.uk  Thu Jul 27 17:24:13 2006
From: david.nospam.hopwood at blueyonder.co.uk (David Hopwood)
Date: Thu, 27 Jul 2006 16:24:13 +0100
Subject: [Python-Dev] Internal namespace proposal
In-Reply-To: <44C8D929.2010508@blueyonder.co.uk>
References: <Pine.LNX.4.58.0607191811400.31087@server1.LFW.org>	<bbaeab100607191650j4584de2cge554274e100f125e@mail.gmail.com>	<44C822BC.1070204@blueyonder.co.uk>	<20060727113016.GC31912@code0.codespeak.net>
	<44C8D929.2010508@blueyonder.co.uk>
Message-ID: <44C8DA9D.9060709@blueyonder.co.uk>

David Hopwood wrote:
> The intention was not to require the restrictions to be compiler-enforced;
> only to *allow* them to be compiler-enforced.
> 
> Code like this, for example:
> 
>   def someMethod(self, x):
>       if self == x:

"if self is x:", I meant.

>           foo(x._internal)
> 
> should not have to work.

-- 
David Hopwood <david.nospam.hopwood at blueyonder.co.uk>



From bob at redivi.com  Thu Jul 27 19:04:35 2006
From: bob at redivi.com (Bob Ippolito)
Date: Thu, 27 Jul 2006 10:04:35 -0700
Subject: [Python-Dev] Release manager pronouncement needed: PEP 302 Fix
In-Reply-To: <eaa5rd$pe2$1@sea.gmane.org>
References: <5.1.1.6.0.20060726142456.036d9008@sparrow.telecommunity.com>
	<20060727103920.GB31912@code0.codespeak.net>
	<eaa5rd$pe2$1@sea.gmane.org>
Message-ID: <1BFAA40C-5B6C-4376-B1C9-D2477EE689B9@redivi.com>


On Jul 27, 2006, at 3:52 AM, Georg Brandl wrote:

> Armin Rigo wrote:
>> Hi Phillip,
>>
>> On Wed, Jul 26, 2006 at 02:40:27PM -0400, Phillip J. Eby wrote:
>>> If we don't revert it, there are two ways to fix it.  One is to  
>>> just change
>>> PEP 302 so that the behavior is unbroken by definition.  :)  The  
>>> other is
>>> to actually go ahead and fix it by adding PathImporter and  
>>> NullImporter
>>> types to import.c, along with a factory function on  
>>> sys.path_hooks to
>>> create them.  (This would've been the PEP-compliant way to  
>>> implement the
>>> need-for-speed patch.)
>>>
>>> So, "fix" by documentation, fix by fixing, or fix by reverting?   
>>> Which
>>> should it be?
>>
>> "fix" by changing the definition looks like a bad idea to me.  The
>> import logic is already extremely complicated and delicate, any  
>> change
>> to it is bound to break *some* code somewhere.
>
> Though beta1 and beta2 shipped with this change nobody reported any  
> bug that
> could be linked to it. sys.path_importer_cache is quite an internal  
> thing and
> most code, even import hooks, shouldn't have to deal with it.

Anyone trying to emulate what imp.find_module does in a PEP 302  
compliant way will need to introspect sys.path_importer_cache. I have  
some unreleased code based on the PEP 302 spec that does this and the  
way it was originally written would have broke in 2.5 if I had tested  
it there.

Just because it's obscure doesn't mean we should go change how things  
work in a way that's not consistent with the documentation. The  
documentation should change to match the code or vice versa, though I  
really don't have any strong feelings one way or the other.

-bob


From g.brandl at gmx.net  Thu Jul 27 19:13:43 2006
From: g.brandl at gmx.net (Georg Brandl)
Date: Thu, 27 Jul 2006 19:13:43 +0200
Subject: [Python-Dev] Release manager pronouncement needed: PEP 302 Fix
In-Reply-To: <1BFAA40C-5B6C-4376-B1C9-D2477EE689B9@redivi.com>
References: <5.1.1.6.0.20060726142456.036d9008@sparrow.telecommunity.com>	<20060727103920.GB31912@code0.codespeak.net>	<eaa5rd$pe2$1@sea.gmane.org>
	<1BFAA40C-5B6C-4376-B1C9-D2477EE689B9@redivi.com>
Message-ID: <eaas5s$br4$1@sea.gmane.org>

Bob Ippolito wrote:

> Just because it's obscure doesn't mean we should go change how things  
> work in a way that's not consistent with the documentation. The  
> documentation should change to match the code or vice versa, though I  
> really don't have any strong feelings one way or the other.

I never said it shouldn't be documented if the current code stays.

Georg


From ronaldoussoren at mac.com  Thu Jul 27 19:14:13 2006
From: ronaldoussoren at mac.com (Ronald Oussoren)
Date: Thu, 27 Jul 2006 19:14:13 +0200
Subject: [Python-Dev] uuid test suite failing
In-Reply-To: <eaap2c$al$1@sea.gmane.org>
References: <eaamnu$mrq$1@sea.gmane.org> <eaap2c$al$1@sea.gmane.org>
Message-ID: <D59E6AA9-C7D1-4439-9E45-12D11B6296ED@mac.com>


On Jul 27, 2006, at 6:20 PM, Georg Brandl wrote:

> Georg Brandl wrote:
>> The UUID test suite, which wasn't run by regrtest.py until now, is
>> now failing on some buildbots (and my machine). This should be fixed
>> before releasing something.
>
> Okay, after fixing the test on my machine (locale issue) it looks like
> some ifconfigs don't like to be called without arguments. "-a"  
> seems to
> be supported everywhere though, so I guess it's reasonable to use that
> flag on every platform. Any objections?

IIRC at least some versions of HP-UX do not support the -a flag for  
ifconfig, I'll check this tomorrow.

Ronald


From collinw at gmail.com  Thu Jul 27 19:51:51 2006
From: collinw at gmail.com (Collin Winter)
Date: Thu, 27 Jul 2006 13:51:51 -0400
Subject: [Python-Dev] Which version of distutils to ship with Python 2.5?
In-Reply-To: <44C85FF2.3030701@v.loewis.de>
References: <43aa6ff70607261924w65e01325o854f87d4c109026a@mail.gmail.com>
	<44C85FF2.3030701@v.loewis.de>
Message-ID: <43aa6ff70607271051h3d87b807w5fad3195949d051e@mail.gmail.com>

On 7/27/06, "Martin v. L?wis" <martin at v.loewis.de> wrote:
> Collin Winter wrote:
> > Is it intentional that Python 2.5 is (currently) shipping with
> > distutils 2.4.0, while Python 2.4 (at least 2.4.1, 2.4.2 and 2.4.3)
> > shipped with distutils 2.4.1? Judging from my own tests, distutils
> > 2.4.1 fixed several bugs that some of my test suites depend on (the
> > fixes, not the bugs ; ).
>
> Are these bugs not fixed in the distutils that shipped with Python 2.5b2?

I now believe this to be a new regression that I had confused with an
earlier bug report. I've filed a new report,
http://python.org/sf/1529871. I'd appreciate it if anyone could shed
some light on this.

Thanks,
Collin Winter

From david.nospam.hopwood at blueyonder.co.uk  Thu Jul 27 19:02:05 2006
From: david.nospam.hopwood at blueyonder.co.uk (David Hopwood)
Date: Thu, 27 Jul 2006 18:02:05 +0100
Subject: [Python-Dev] Internal namespace proposal
In-Reply-To: <44C867D3.90306@canterbury.ac.nz>
References: <Pine.LNX.4.58.0607191811400.31087@server1.LFW.org>
	<bbaeab100607191650j4584de2cge554274e100f125e@mail.gmail.com>
	<44C822BC.1070204@blueyonder.co.uk>
	<44C867D3.90306@canterbury.ac.nz>
Message-ID: <44C8F18D.5020503@blueyonder.co.uk>

Greg Ewing wrote:
> David Hopwood wrote:
> 
>>   A restricted interpreter refuses access to any object attribute or
>>   method with a name beginning with '_' (by throwing a new exception type
>>   'InternalAccessException'), unless the access is from a method and its
>>   static target is that method's first argument variable.
> 
> What's to stop
> 
>   def my_naughty_method(self):
>     self = some_protected_object
>     self._a_special_attribute = some_naughty_value

That's a good point -- I didn't describe the intended restriction correctly.

The reason for not just saying "... and its target is the object that was
passed as that method's first argument", was that I want it to be possible to
reject programs with internal accesses that cannot be statically recognized
as legal.

(This does not mean that a particular implementation would have to rely on
static detection of non-internal accesses, or on a trusted compiler.)

How about this:

     A restricted interpreter shall refuse access to any object attribute or
     method with a name beginning with '_', unless the access is from a method,
     and its target is the object that was passed as that method's first argument.
     If such an access is detected at run-time, then it shall be reported by
     throwing a new exception type 'InternalAccessException'.

     In addition, a program containing an access to an object attribute or
     method with a name beginning with '_', where the access is not from a
     method, or the target of the access is not the method's first argument
     variable, or there is an assignment to that variable in the method, is
     an illegal program.

>> __init__ is an internal method. This is as it should be, because it
>> should not be possible to call __init__ on an existing object

... from outside the object, that is ...

>> ; only to have __init__ implicitly called when a new object is constructed.
> 
> What about calling an inherited __init__ method?
> Your proposed rule would seem to disallow
> 
>   BaseClass.__init__(self, ...)

No, this call would be allowed because it is to self.

-- 
David Hopwood <david.nospam.hopwood at blueyonder.co.uk>



From mattjfleming at googlemail.com  Thu Jul 27 23:21:52 2006
From: mattjfleming at googlemail.com (Matt Fleming)
Date: Thu, 27 Jul 2006 21:21:52 +0000
Subject: [Python-Dev] uuid test suite failing
In-Reply-To: <5ff4a1e50607271022x35175b0coda2148d9d0d1db4b@mail.gmail.com>
References: <eaamnu$mrq$1@sea.gmane.org> <eaap2c$al$1@sea.gmane.org>
	<D59E6AA9-C7D1-4439-9E45-12D11B6296ED@mac.com>
	<5ff4a1e50607271022x35175b0coda2148d9d0d1db4b@mail.gmail.com>
Message-ID: <5ff4a1e50607271421v71cdb8fbt569b9cc6a339ed0f@mail.gmail.com>

> On 27/07/06, Ronald Oussoren <ronaldoussoren at mac.com> wrote:
> > IIRC at least some versions of HP-UX do not support the -a flag for
> > ifconfig, I'll check this tomorrow.
> >
> > Ronald
> >

  td192> /usr/sbin/ifconfig
  usage: ifconfig interface
         [ af [ address [ dest_addr ] ] [ up ] [ down ][ netmask mask ] ]
         [ metric n ]
         [ arp | -arp ]
         [ plumb | unplumb ]
  td192> /usr/sbin/ifconfig -a
  ifconfig: no such interface
  td192> uname -a
  HP-UX td192 B.11.11 U 9000/800 1839940656 unlimited-user license


Also fixed this test on my NetBSD machine by using 'ifconfig -a' and
checking for 'address:' in the output. But as Ronald said, not all
platforms support the '-a' flag. Not sure if this will fix the OpenBSD
buildbot, I don't have access to an OpenBSD machine.

Matt


-- 
http://mattssanctuary.blogspot.com

From pje at telecommunity.com  Thu Jul 27 23:23:00 2006
From: pje at telecommunity.com (Phillip J. Eby)
Date: Thu, 27 Jul 2006 17:23:00 -0400
Subject: [Python-Dev] Release manager pronouncement needed: PEP 302 Fix
In-Reply-To: <eaa5rd$pe2$1@sea.gmane.org>
References: <20060727103920.GB31912@code0.codespeak.net>
	<5.1.1.6.0.20060726142456.036d9008@sparrow.telecommunity.com>
	<20060727103920.GB31912@code0.codespeak.net>
Message-ID: <5.1.1.6.0.20060727171324.0262c318@sparrow.telecommunity.com>

At 12:52 PM 7/27/2006 +0200, Georg Brandl wrote:
>Though beta1 and beta2 shipped with this change nobody reported any bug that
>could be linked to it

Actually, that's no longer true.  See:  http://python.org/sf/1529871

This is a user-reported bug against 2.5b2, and I have just confirmed that 
it is caused by the need-for-speed PEP 302 breakage.  Reverting r46372 
fixes the reported problem.

Of course, since I'm the author of the affected package, I can certainly 
fix the problem by updating my code.  (And in fact, setuptools 0.7a1 works 
because it uses 2.5's pkgutil when available, and r46372 included a fix for 
pkgutil.)

At this point, I'm going to wait another day for somebody to step up to fix 
the documentation, or the release manager to authorize inclusion of a 
proper fix (which is unlikely to be allowed, since it will mean adding two 
new types to import.c, and restructuring a bit of pkgutil to subclass from 
them).

(It's a nice performance improvement you've got here, I would hate to see 
anything happen to it...)

Personally, I would prefer to see it properly fixed in 2.5 rather than 
having to rip it out.  It's more work for me to create the proper fix than 
it is to just work around it in my code, but it seems a more righteous 
labor, if you know what I mean.  It also means that already-shipped and 
distributed versions of my code would work with the 2.5 release.


From pje at telecommunity.com  Thu Jul 27 23:28:23 2006
From: pje at telecommunity.com (Phillip J. Eby)
Date: Thu, 27 Jul 2006 17:28:23 -0400
Subject: [Python-Dev] Which version of distutils to ship with Python 2.5?
In-Reply-To: <43aa6ff70607271051h3d87b807w5fad3195949d051e@mail.gmail.co
 m>
References: <44C85FF2.3030701@v.loewis.de>
	<43aa6ff70607261924w65e01325o854f87d4c109026a@mail.gmail.com>
	<44C85FF2.3030701@v.loewis.de>
Message-ID: <5.1.1.6.0.20060727172329.028f1178@sparrow.telecommunity.com>

At 01:51 PM 7/27/2006 -0400, Collin Winter wrote:
>On 7/27/06, "Martin v. L?wis" <martin at v.loewis.de> wrote:
> > Collin Winter wrote:
> > > Is it intentional that Python 2.5 is (currently) shipping with
> > > distutils 2.4.0, while Python 2.4 (at least 2.4.1, 2.4.2 and 2.4.3)
> > > shipped with distutils 2.4.1? Judging from my own tests, distutils
> > > 2.4.1 fixed several bugs that some of my test suites depend on (the
> > > fixes, not the bugs ; ).
> >
> > Are these bugs not fixed in the distutils that shipped with Python 2.5b2?
>
>I now believe this to be a new regression that I had confused with an
>earlier bug report. I've filed a new report,
>http://python.org/sf/1529871. I'd appreciate it if anyone could shed
>some light on this.

As I noted on the bug itself, the problem is due to r46372, a patch 
introduced by the need-for-speed sprint and which broke CPython's 
compliance with PEP 302, by introducing non-None, non-importer values into 
sys.path_importer_cache.

At the present time, I have not received an unequivocal response regarding 
how the problem should be corrected:

1. Update PEP 302 and the "What's New" documentation to reflect this 
backwards-incompatible change and require authors to update their code to 
work with Python 2.5

2. Fix the patch so it implements the speedup in a way that is conformant 
with PEP 302 as it currently stands  (I have volunteered to do this, if the 
release manager(s) will authorize it)

3. Revert the patch.

No one has volunteered for any of the work required for #1.  I belive that 
#3 must be done if #2 is not approved and no volunteers surface for #1 in a 
timely manner.


From barry at python.org  Fri Jul 28 01:54:57 2006
From: barry at python.org (Barry Warsaw)
Date: Thu, 27 Jul 2006 19:54:57 -0400
Subject: [Python-Dev] Support for PyGetSetDefs in pydoc
In-Reply-To: <e9plv5$rfe$1@sea.gmane.org>
References: <A027CA35-EE77-4658-BA59-E849368C440B@python.org><EF6E6AF2-654F-4F53-91A8-A6C9B8A80745@python.org><44B64782.9010706@gmail.com><7FE62713-24B0-47CE-BA94-41150DB683F4@python.org>
	<AB1C2209-329E-4F3D-8B7A-2548A40F892A@python.org>
	<e9plv5$rfe$1@sea.gmane.org>
Message-ID: <E666C3A3-9C45-41C7-B25A-C9CB244485B5@python.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Since Anthony didn't speak up, I took his silence as assent and went  
ahead and committed the changes.  r50881 and r50885 for *nix and  
Windows, just in case the deafening silence turns into a howl of  
derision :).

- -Barry

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.3 (Darwin)

iQCVAwUBRMlSV3EjvBPtnXfVAQL3WQP9H2RBIDG3FCEkzHjzmwyRWl4HU467yWMQ
bse0/XhUEAQHivwP2nLvAqn+Qrb8XaXIT3n5i9++saMFtxjTdfMJX2ZNBK+0JmVl
N+XvhTIXIu9XJy47c4FsZ6tbfHVSKQ3KRaE81sfMYuKQsPCnB9cNskKEJEpaS0Cy
F7GmpdE96sM=
=T3Ia
-----END PGP SIGNATURE-----

From tim.peters at gmail.com  Fri Jul 28 02:46:30 2006
From: tim.peters at gmail.com (Tim Peters)
Date: Thu, 27 Jul 2006 20:46:30 -0400
Subject: [Python-Dev] [Windows, buildbot] kill_python.c mystery
In-Reply-To: <44C85E07.3060707@v.loewis.de>
References: <1f7befae0607261702o6d943f52x701be0d402345f65@mail.gmail.com>
	<44C85E07.3060707@v.loewis.de>
Message-ID: <1f7befae0607271746y7bca18f2s439e8b5f996f132d@mail.gmail.com>

[Martin v. L?wis]
> Didn't you know that you signed in to run arbitrary viruses, worms, and
> trojan horses when you added your machine to the buildbot infrastructure
> :-?

Hey, I signed up for that when I bought a Windows box :-)

> You just haven't seen buildbot erasing your hard disk and filling
> your coffee machine with tea, yet.

Not the buildbot, no, but visiting random web pages does that routinely.

>>           (strstr(path, "build\\python.exe") != NULL)) {
>> Why is the second clause there?

> That's for Cygwin (i.e. Anthony Baxter's machine). As Neal suggests,
> preceding the executable path with another backslash should solve
> this problem.

And he checked that in, and I haven't noticed another similar problem
since (but then it was rare to begin with).

> As a related note, this entire approach will also manage to kill
> python.exe from an unrelated buildbot installation, e.g. a 2.4
> build job might kill python.exe from the trunk. This actually helped
> when I tried to get the Cygwin slave to get unstuck, and shouldn't
> do harm since we currently don't run to builds on the same slave
> simultaneously, but could be surprising when parallel builds
> are activated some day.

I don't think we /can/ yet -- I believe some tests T exist that
implicitly assume only one instance of T is running.  I don't recall
details, although I'm sure we'll bump into them sporadically the
instant parallel builds are enabled.  As a purely pragmatic matter, I
expect my hard drive would quickly be reduced to dust if two instances
of test_largefile ran simultaneously (Windows XP writes physical
zeroes in the entire multi-GB file, and that takes finite time only if
the disk head doesn't have to keep seeking).

> Sorry for messing with your machine,

No problem!  That's what it's here for :-)

From greg.ewing at canterbury.ac.nz  Fri Jul 28 02:51:43 2006
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Fri, 28 Jul 2006 12:51:43 +1200
Subject: [Python-Dev] Internal namespace proposal
In-Reply-To: <44C8D929.2010508@blueyonder.co.uk>
References: <Pine.LNX.4.58.0607191811400.31087@server1.LFW.org>
	<bbaeab100607191650j4584de2cge554274e100f125e@mail.gmail.com>
	<44C822BC.1070204@blueyonder.co.uk>
	<20060727113016.GC31912@code0.codespeak.net>
	<44C8D929.2010508@blueyonder.co.uk>
Message-ID: <44C95F9F.2090307@canterbury.ac.nz>

David Hopwood wrote:

> Inheritance should be defined as though the code of inherited methods and
> attributes were copied into the subclass (with global accesses updated to
> point to the original module).

You'll have to propose an implementation strategy for that
which works without actually copying all the code, though.

> Since x._hidden is not accessible from attack.py, the attack fails.

But if _hidden were an attribute of the A instance that
you were trying to protect, it would succeed. So you
can't actually protect any direct attribute of a class
that can be subclassed. Which means we're back to the
situation of having to prevent access to class objects.

--
Greg

From david.nospam.hopwood at blueyonder.co.uk  Fri Jul 28 03:40:53 2006
From: david.nospam.hopwood at blueyonder.co.uk (David Hopwood)
Date: Fri, 28 Jul 2006 02:40:53 +0100
Subject: [Python-Dev] Internal namespace proposal
In-Reply-To: <44C95F9F.2090307@canterbury.ac.nz>
References: <Pine.LNX.4.58.0607191811400.31087@server1.LFW.org>
	<bbaeab100607191650j4584de2cge554274e100f125e@mail.gmail.com>
	<44C822BC.1070204@blueyonder.co.uk>
	<20060727113016.GC31912@code0.codespeak.net>
	<44C8D929.2010508@blueyonder.co.uk>
	<44C95F9F.2090307@canterbury.ac.nz>
Message-ID: <44C96B25.5000709@blueyonder.co.uk>

Greg Ewing wrote:
> David Hopwood wrote:
> 
>> Inheritance should be defined as though the code of inherited methods and
>> attributes were copied into the subclass (with global accesses updated to
>> point to the original module).
> 
> You'll have to propose an implementation strategy for that
> which works without actually copying all the code, though.

The only difference between the copying semantics and the current semantics,
is in the visibility of module-global internal variables and functions. It's
sufficient to keep track of whether each class could access a variable or function
that is internal to its module. If it does, then it cannot be subclassed from
a different module. (It must not be possible to access internal variables/
functions reflectively.)

The effect of this is that if a programmer intends a class to be subclassable
from outside the module, they must make sure that all of the variables/functions
it depends on are public. Anyone performing a security review of the module then
does not have to consider inheritance from a different module when deciding
which variables/functions might be accessible.


There is a slightly more flexible version of this approach that is just as
secure, provided that functions are restricted to be stateless. If it is
possible to prove statically that a class does not access any internal
*variables* of a module (regardless of whether it accesses internal functions),
then it is safe to allow the class to be subclassed from another module. This
is because the subclassing module could have copied all of the code of the
original module (assuming it is written purely in Python); the only possible
sources of authority in a capability language are from access to variables or
primitives, not code.


If a class is not written in Python, then we cannot analyse whether it accesses
internal variables. In that case the class will be part of the TCB, and we have
to trust the class writer to mark whether it can be safely subclassed from
another module.

>> Since x._hidden is not accessible from attack.py, the attack fails.
> 
> But if _hidden were an attribute of the A instance that
> you were trying to protect, it would succeed.

No, attack.py could only access a _hidden attribute in an instance of B.
This is harmless, because it could just as well define the _hidden attribute
of B itself, rather than by subclassing.

-- 
David Hopwood <david.nospam.hopwood at blueyonder.co.uk>



From david.nospam.hopwood at blueyonder.co.uk  Fri Jul 28 03:55:58 2006
From: david.nospam.hopwood at blueyonder.co.uk (David Hopwood)
Date: Fri, 28 Jul 2006 02:55:58 +0100
Subject: [Python-Dev] Internal namespace proposal
In-Reply-To: <4BE3C1EC-8DE0-4518-A8B6-68F98477433B@commonground.com.au>
References: <Pine.LNX.4.58.0607191811400.31087@server1.LFW.org>
	<bbaeab100607191650j4584de2cge554274e100f125e@mail.gmail.com>
	<44C822BC.1070204@blueyonder.co.uk>
	<4BE3C1EC-8DE0-4518-A8B6-68F98477433B@commonground.com.au>
Message-ID: <44C96EAE.6040907@blueyonder.co.uk>

Richard Jones wrote:
> On 27/07/2006, at 12:19 PM, David Hopwood wrote:
> 
>>   A restricted interpreter refuses access to any object attribute or
>>   method with a name beginning with '_' (by throwing a new exception type
>>   'InternalAccessException'), unless the access is from a method and its
>>   static target is that method's first argument variable.
>>
>>   Also, a restricted interpreter refuses access to any module-global
>>   variable or module-global function with a name beginning with '_' (by
>>   throwing 'InternalAccessException'), unless the access is statically
>>   from the same module.
> 
> Note that this is a rule that Zope enforces in its restricted environment.

Is that documented anywhere?

-- 
David Hopwood <david.nospam.hopwood at blueyonder.co.uk>



From tim.peters at gmail.com  Fri Jul 28 05:21:15 2006
From: tim.peters at gmail.com (Tim Peters)
Date: Thu, 27 Jul 2006 23:21:15 -0400
Subject: [Python-Dev] how about adding ping's uuid module to the
	standard lib ?
In-Reply-To: <dui1u8$ge1$1@sea.gmane.org>
References: <dui1u8$ge1$1@sea.gmane.org>
Message-ID: <1f7befae0607272021k17b65a88g356c781c34967cc7@mail.gmail.com>

Georg discovered that test_uuid didn't run any tests, and fixed that
on Thursday.  A number of buildbots have failed that test since then.

My XP box appears unique among the Windows buildbots in failing.  It
always fails like so:

AssertionError: different sources disagree on node:
    from source 'getnode1', node was 00038a000015
    from source 'getnode2', node was 00038a000015
    from source 'ipconfig', node was 001111b2b7bf

0x00038a000015 /is/ the last 6 bytes returned by the Windows
UuidCreateSequential() on my box.  I confirmed that by writing a C
program calling it directly.  However, it doesn't appear to correspond
to any MAC address of any HW on my box.

All documented ways of determining the MAC address of my Ethernet card
(using UuidCreateSequential for this appears to be folklore rather
than documented behavior) agree that 0x001111b2b7bf is correct on this
box; e.g.,

$ getmac /fo list /v
Connection Name:  Local Area Connection
Network Adapter:  Marvell Yukon 88E8050 PCI-E ASF Gigabit Ethernet Controller
Physical Address: 00-11-11-B2-B7-BF
Transport Name:   \Device\Tcpip_...

Connection Name:  1394 Connection
Network Adapter:  1394 Net Adapter
Physical Address: 62-A1-AC-6C-FD-BE
Transport Name:   \Device\Tcpip_...

Connection Name:  1394 Connection 2
Network Adapter:  1394 Net Adapter
Physical Address: E2-1F-01-C6-5D-88
Transport Name:   \Device\Tcpip_...

The last two are for firewire interfaces, and don't match the
purported MAC address extracted from UuidCreateSequential's output
anyway.

So, at least on my box, this comment in uuid.py is incorrect
(UuidCreateSequential does not behave as it says):

    # On Windows prior to 2000, UuidCreate gives a UUID containing the
    # hardware address.  On Windows 2000 and later, UuidCreate makes a
    # random UUID and UuidCreateSequential gives a UUID containing the
    # hardware address. ...

Unfortunately, uuid.getnode() tries things in this order on Windows:

        getters = [_windll_getnode, _netbios_getnode, _ipconfig_getnode]

It's only the first one that returns the bogus 0x00038a000015; both of
the latter return 0x001111B2B7BF.

However, there's nothing I can do to that list to make test_uuid pass
on this box.  It wants to insist that all three ways of
getting/guessing the MAC address return the same thing, and that's
never going to happen here.

Given that _windll_getnode's actual behavior appears to have nothing
in common with what was expected for it here, best suggestion I can
make is to throw its code away.

From tim.peters at gmail.com  Fri Jul 28 06:27:51 2006
From: tim.peters at gmail.com (Tim Peters)
Date: Fri, 28 Jul 2006 00:27:51 -0400
Subject: [Python-Dev] how about adding ping's uuid module to the
	standard lib ?
In-Reply-To: <1f7befae0607272021k17b65a88g356c781c34967cc7@mail.gmail.com>
References: <dui1u8$ge1$1@sea.gmane.org>
	<1f7befae0607272021k17b65a88g356c781c34967cc7@mail.gmail.com>
Message-ID: <1f7befae0607272127u544fac8eq9608e95938ca2c55@mail.gmail.com>

[Tim]
> ... uuid.getnode() tries things in this order on Windows:
>
>         getters = [_windll_getnode, _netbios_getnode, _ipconfig_getnode]
>
> It's only the first one that returns the bogus 0x00038a000015; both of
> the latter return 0x001111B2B7BF [the correct MAC address for my
> network card].

That was on my desktop XP Pro SP2 box.  On my similar laptop box, it's
quite different:  _windll_getnode and _ipconfig_getnode return the MAC
address of my wireless Ethernet adapter, while _netbios_getnode
returns the MAC address of my LAN Ethernet card.

> However, there's nothing I can do to that list to make test_uuid pass
> on this box.  It wants to insist that all three ways of
> getting/guessing the MAC address return the same thing, and that's
> never going to happen here.

Or on my laptop, but for different reasons there.

> Given that _windll_getnode's actual behavior appears to have nothing
> in common with what was expected for it here, best suggestion I can
> make is to throw its code away.

Which wouldn't improve things on my laptop.  Best next ;-) suggestion
is to change test_uuid to stop believing that uuid.py knows multiple
ways to find a well-defined MAC address.  I'm going to make that
change -- someone who hates that can revert it after they buy me two
new computers that work they way they think computers should work ;-)

From nnorwitz at gmail.com  Fri Jul 28 06:49:48 2006
From: nnorwitz at gmail.com (Neal Norwitz)
Date: Thu, 27 Jul 2006 21:49:48 -0700
Subject: [Python-Dev] Release manager pronouncement needed: PEP 302 Fix
In-Reply-To: <5.1.1.6.0.20060727171324.0262c318@sparrow.telecommunity.com>
References: <5.1.1.6.0.20060726142456.036d9008@sparrow.telecommunity.com>
	<20060727103920.GB31912@code0.codespeak.net>
	<eaa5rd$pe2$1@sea.gmane.org>
	<5.1.1.6.0.20060727171324.0262c318@sparrow.telecommunity.com>
Message-ID: <ee2a432c0607272149xc22baf4u2945da575237a83@mail.gmail.com>

On 7/27/06, Phillip J. Eby <pje at telecommunity.com> wrote:
>
> Personally, I would prefer to see it properly fixed in 2.5 rather than
> having to rip it out.  It's more work for me to create the proper fix than
> it is to just work around it in my code, but it seems a more righteous
> labor, if you know what I mean.  It also means that already-shipped and
> distributed versions of my code would work with the 2.5 release.

Based on this comment, is it really acceptable to just document a
behaviour change?  ISTM there should really only be 2 choices:  fix
2.5 properly or revert the change.  This seemed to be Armin's
position.

n

From pje at telecommunity.com  Fri Jul 28 06:56:41 2006
From: pje at telecommunity.com (Phillip J. Eby)
Date: Fri, 28 Jul 2006 00:56:41 -0400
Subject: [Python-Dev] Release manager pronouncement needed: PEP 302 Fix
In-Reply-To: <ee2a432c0607272149xc22baf4u2945da575237a83@mail.gmail.com>
References: <5.1.1.6.0.20060727171324.0262c318@sparrow.telecommunity.com>
	<5.1.1.6.0.20060726142456.036d9008@sparrow.telecommunity.com>
	<20060727103920.GB31912@code0.codespeak.net>
	<eaa5rd$pe2$1@sea.gmane.org>
	<5.1.1.6.0.20060727171324.0262c318@sparrow.telecommunity.com>
Message-ID: <5.1.1.6.0.20060728005338.02a25860@sparrow.telecommunity.com>

At 09:49 PM 7/27/2006 -0700, Neal Norwitz wrote:
>On 7/27/06, Phillip J. Eby <pje at telecommunity.com> wrote:
>>
>>Personally, I would prefer to see it properly fixed in 2.5 rather than
>>having to rip it out.  It's more work for me to create the proper fix than
>>it is to just work around it in my code, but it seems a more righteous
>>labor, if you know what I mean.  It also means that already-shipped and
>>distributed versions of my code would work with the 2.5 release.
>
>Based on this comment, is it really acceptable to just document a
>behaviour change?  ISTM there should really only be 2 choices:  fix
>2.5 properly or revert the change.  This seemed to be Armin's
>position.

Well, it's a moot question since nobody has volunteered to update the 
docs.  Fixing it and reverting it are the only options unless somebody 
steps up to do the doc work.  I'll happily fix it or revert it, just tell 
me which one is the approved course of action and I'll get started.  :)


From rhettinger at ewtllc.com  Fri Jul 28 07:32:52 2006
From: rhettinger at ewtllc.com (Raymond Hettinger)
Date: Thu, 27 Jul 2006 22:32:52 -0700
Subject: [Python-Dev] Py2.5 release schedule
Message-ID: <44C9A184.7030701@ewtllc.com>

I suggest that there be a third beta release and that we then wait just 
a bit before going final.

The bugs that were found and fixed in the first two beta releases 
suggest that Py2.5 is not yet as stable as we would like.  Over the next 
few days, I'll try to run it on as much third-party code as possible.  
That would have detected the recently surfaced grammar error a little 
bit earlier (the one where "for x, in listOfTuples" would not unpack).

The release process itself is going well but I don't think the pervasive 
AST changes have been fully shaken-out yet.



Raymond

From martin at v.loewis.de  Fri Jul 28 07:33:37 2006
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Fri, 28 Jul 2006 07:33:37 +0200
Subject: [Python-Dev] Release manager pronouncement needed: PEP 302 Fix
In-Reply-To: <5.1.1.6.0.20060727114924.0262c980@sparrow.telecommunity.com>
References: <20060727103920.GB31912@code0.codespeak.net>	<5.1.1.6.0.20060726142456.036d9008@sparrow.telecommunity.com>	<20060727103920.GB31912@code0.codespeak.net>
	<5.1.1.6.0.20060727114924.0262c980@sparrow.telecommunity.com>
Message-ID: <44C9A1B1.3040102@v.loewis.de>

Phillip J. Eby wrote:
> I'm willing to write code that makes it PEP 302 compliant, if the release 
> manager will bless such an addition.  But if that's not acceptable, then 
> somebody needs to produce the necessary documentation updates or revert the 
> patch.  It absolutely should not be allowed to remain in *and* undocumented 
> because it is a backwards-incompatible change to documented behavior of 
> Python for two major releases (2.3 and 2.4).

You don't need a release manager pronouncement for that. It's a bug,
changing it is a bug fix, you don't need RM permission to fix a bug.

Do you have a patch ready that restores path_importer_cache behavior,
yet  preserves the property that it caches existence of a directory?
If not, I will have to produce one.

Regards,
Martin

From anthony at interlink.com.au  Fri Jul 28 07:39:48 2006
From: anthony at interlink.com.au (Anthony Baxter)
Date: Fri, 28 Jul 2006 15:39:48 +1000
Subject: [Python-Dev] Py2.5 release schedule
In-Reply-To: <44C9A184.7030701@ewtllc.com>
References: <44C9A184.7030701@ewtllc.com>
Message-ID: <200607281539.50794.anthony@interlink.com.au>

On Friday 28 July 2006 15:32, Raymond Hettinger wrote:
> I suggest that there be a third beta release and that we then wait
> just a bit before going final.
>
> The bugs that were found and fixed in the first two beta releases
> suggest that Py2.5 is not yet as stable as we would like.  Over the
> next few days, I'll try to run it on as much third-party code as
> possible. That would have detected the recently surfaced grammar
> error a little bit earlier (the one where "for x, in listOfTuples"
> would not unpack).
>
> The release process itself is going well but I don't think the
> pervasive AST changes have been fully shaken-out yet.

I've been thinking the same thing, too. A quick chat to Neal says that 
he also agrees.

There's still a lot more bugs popping up than I'm really comfortable 
with. I guess this is inevitable - there's a lot of new stuff in 2.5. 

Does anyone disagree with making the next release beta3?

Anthony
-- 
Anthony Baxter     <anthony at interlink.com.au>
It's never too late to have a happy childhood.

From greg at electricrain.com  Fri Jul 28 07:54:39 2006
From: greg at electricrain.com (Gregory P. Smith)
Date: Thu, 27 Jul 2006 22:54:39 -0700
Subject: [Python-Dev] [Windows, buildbot] kill_python.c mystery
In-Reply-To: <44C85E07.3060707@v.loewis.de>
References: <1f7befae0607261702o6d943f52x701be0d402345f65@mail.gmail.com>
	<44C85E07.3060707@v.loewis.de>
Message-ID: <20060728055439.GD22921@zot.electricrain.com>

> Didn't you know that you signed in to run arbitrary viruses, worms, and
> trojan horses when you added your machine to the buildbot infrastructure
> :-? You just haven't seen buildbot erasing your hard disk and filling
> your coffee machine with tea, yet.

VMware Server is free.  Run buildbots in a VM.  (but don't assume a VM
protects you from trojans that are designed to break out of it) 

-g


From nnorwitz at gmail.com  Fri Jul 28 09:24:52 2006
From: nnorwitz at gmail.com (Neal Norwitz)
Date: Fri, 28 Jul 2006 00:24:52 -0700
Subject: [Python-Dev] uuid test suite failing
In-Reply-To: <20060727163647.GA4175@rogue.amk.ca>
References: <eaamnu$mrq$1@sea.gmane.org> <20060727163647.GA4175@rogue.amk.ca>
Message-ID: <ee2a432c0607280024j56bf7facmd9ebdd26aafc013f@mail.gmail.com>

On 7/27/06, A.M. Kuchling <amk at amk.ca> wrote:
> On Thu, Jul 27, 2006 at 05:40:57PM +0200, Georg Brandl wrote:
> > The UUID test suite, which wasn't run by regrtest.py until now, is
> > now failing on some buildbots (and my machine). This should be fixed
> > before releasing something.
>
> Looking at the failures, there seem to be two problems on Unix variants:
>  1) on some, '/sbin/ifconfig' prints a help message; you need 'ifconfig -a'
>     to print information about all interfaces.
>  2) on Solaris 9 (the only version in the SF compile farm), I can't
>     figure out how to make ifconfig print MAC addresses at all.
>     Searching online finds the incantation 'arp <hostname>' to print the
>     MAC.

This is such a mess.  There are so many different ways of determining
the MAC addr on each flavour of Unix it seems hopeless to try.  I
fixed _ifconfig_getnode so it should work on at least:  Linux, Tru64,
Solaris, and HP-UX.  Who knows how many more variations there are.

This only fixes 1 of the 2 failures in test_uuid.  The other one is
due to _unixdll_getnode() failing.  This is because
_uuid_generate_time is None because we couldn't find it in the uuid
library.  This is just broken, not sure if it's the code or the test
though.  We should handle the case if _uuid_generate_time and the
others are None better.  I don't know what to do in this case.

Since getnode ignores exceptions, maybe it's the test that is broken?

n

From mwh at python.net  Fri Jul 28 11:53:47 2006
From: mwh at python.net (Michael Hudson)
Date: Fri, 28 Jul 2006 10:53:47 +0100
Subject: [Python-Dev] Py2.5 release schedule
In-Reply-To: <200607281539.50794.anthony@interlink.com.au> (Anthony Baxter's
	message of "Fri, 28 Jul 2006 15:39:48 +1000")
References: <44C9A184.7030701@ewtllc.com>
	<200607281539.50794.anthony@interlink.com.au>
Message-ID: <2mirlixdg4.fsf@starship.python.net>

Anthony Baxter <anthony at interlink.com.au> writes:

> On Friday 28 July 2006 15:32, Raymond Hettinger wrote:
>> I suggest that there be a third beta release and that we then wait
>> just a bit before going final.
>>
>> The bugs that were found and fixed in the first two beta releases
>> suggest that Py2.5 is not yet as stable as we would like.  Over the
>> next few days, I'll try to run it on as much third-party code as
>> possible. That would have detected the recently surfaced grammar
>> error a little bit earlier (the one where "for x, in listOfTuples"
>> would not unpack).
>>
>> The release process itself is going well but I don't think the
>> pervasive AST changes have been fully shaken-out yet.
>
> I've been thinking the same thing, too. A quick chat to Neal says that 
> he also agrees.
>
> There's still a lot more bugs popping up than I'm really comfortable 
> with. I guess this is inevitable - there's a lot of new stuff in 2.5. 
>
> Does anyone disagree with making the next release beta3?

It seems like a good idea to me.  I guess this will mean the final
release will be pushed back a bit?

Cheers,
mwh

-- 
  Gevalia is undrinkable low-octane see-through only slightly
  roasted bilge water. Compared to .us coffee it is quite
  drinkable.                                      -- M?ns Nilsson, asr

From arigo at tunes.org  Fri Jul 28 12:11:33 2006
From: arigo at tunes.org (Armin Rigo)
Date: Fri, 28 Jul 2006 12:11:33 +0200
Subject: [Python-Dev] Bad interaction of __index__ and sequence repeat
Message-ID: <20060728101133.GA339@code0.codespeak.net>

Hi,

There is an oversight in the design of __index__() that only just
surfaced :-(  It is responsible for the following behavior, on a 32-bit
machine with >= 2GB of RAM:

    >>> s = 'x' * (2**100)       # works!
    >>> len(s)
    2147483647

This is because PySequence_Repeat(v, w) works by applying w.__index__ in
order to call v->sq_repeat.  However, __index__ is defined to clip the
result to fit in a Py_ssize_t.  This means that the above problem exists
with all sequences, not just strings, given enough RAM to create such
sequences with 2147483647 items.

For reference, in 2.4 we correctly get an OverflowError.

Argh!  What should be done about it?


A bientot,

Armin.

From barry at python.org  Fri Jul 28 14:57:06 2006
From: barry at python.org (Barry Warsaw)
Date: Fri, 28 Jul 2006 08:57:06 -0400
Subject: [Python-Dev] Py2.5 release schedule
In-Reply-To: <200607281539.50794.anthony@interlink.com.au>
References: <44C9A184.7030701@ewtllc.com>
	<200607281539.50794.anthony@interlink.com.au>
Message-ID: <BBBDED8E-ACDD-4176-985A-144188B52F1E@python.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Jul 28, 2006, at 1:39 AM, Anthony Baxter wrote:

> I've been thinking the same thing, too. A quick chat to Neal says that
> he also agrees.
>
> There's still a lot more bugs popping up than I'm really comfortable
> with. I guess this is inevitable - there's a lot of new stuff in 2.5.
>
> Does anyone disagree with making the next release beta3?

+1.  It would give me more type to port and test a few of my  
applications to the new version.

FWIW, our commercial app went pretty smoothly, mostly dealing with  
Py_ssize_t conversions and adopting the new PySet C API.  I'm still  
working on Mailman but the most painful thing so far has been the  
conversion of exceptions to new-style classes, and even that wasn't / 
too/ painful.   I've only done limited testing of both, but I'm  
encouraged that the porting effort will be minor (probably no more  
than a day's worth of work for both apps combined).

- -Barry

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.3 (Darwin)

iQCVAwUBRMoJonEjvBPtnXfVAQJOjAP9HRbakdb39IOtFxGX/wP4QhiXAcNAbXXM
hGFIJ6vC0Gp/SSlTVMYPF5oJMIzuDCIDzs4Nrbgysgfj6Ehyphei/ed8W94PHLat
nh54Y0N5pvwLelHW6ChJBcIxulU8Fuj0Z9kIZCLiryTOAyXTh+t3+gZPEzWRY/tY
v2hd9ERXDl8=
=vK7s
-----END PGP SIGNATURE-----

From tomerfiliba at gmail.com  Fri Jul 28 15:35:27 2006
From: tomerfiliba at gmail.com (tomer filiba)
Date: Fri, 28 Jul 2006 15:35:27 +0200
Subject: [Python-Dev] patching pydoc?
Message-ID: <1d85506f0607280635q3a693682l230c7821dc6f408f@mail.gmail.com>

i have a problem with pydoc in rpyc. i wanted help(obj), where obj
is a NetProxy object, to work as if it were local.

i followed the code starting from site.help to pydoc.doc, which is the
ultimate function that generates and prints the text. i expected there
would be some function in the middle that prepares the text, and
another that writes it to the pager, but to my disappointment pydoc.doc
does both.

this means i can't transfer the document to my local machine (it's
printed directly to the remote console).

therefore, i would like to split this behavior into two parts:
* render_doc - a function that returns the document text
* doc - a function that calls render_doc and sends it to the pager

this way no existing code breaks (no existing function signatures
are changed) and i gain help on remote objects.
i hope people would be in favor, as it's not such a big change anyway.


is it possible to add to 2.5?

-tomer



this is the code of pydoc, starting at line 1457

<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
def doc(thing, title='Python Library Documentation: %s', forceload=0):
    """Display text documentation, given an object or a path to an object."""
    try:
        object, name = resolve(thing, forceload)
        desc = describe(object)
        module = inspect.getmodule(object)
        if name and '.' in name:
            desc += ' in ' + name[:name.rfind('.')]
        elif module and module is not object:
            desc += ' in module ' + module.__name__
        if not (inspect.ismodule(object) or
                inspect.isclass(object) or
                inspect.isroutine(object) or
                isinstance(object, property)):
            # If the passed object is a piece of data or an instance,
            # document its available methods instead of its value.
            object = type(object)
            desc += ' object'
        pager(title % desc + '\n\n' + text.document(object, name))
    except (ImportError, ErrorDuringImport), value:
        print value
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

this is the suggested code

<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
 def render_doc(thing, title='Python Library Documentation: %s', forceload=0):
    """generate the text"""
    object, name = resolve(thing, forceload)
    desc = describe(object)
    module = inspect.getmodule(object)
    if name and '.' in name:
        desc += ' in ' + name[:name.rfind('.')]
    elif module and module is not object:
        desc += ' in module ' + module.__name__
    if not (inspect.ismodule(object) or
            inspect.isclass(object) or
            inspect.isroutine(object) or
            isinstance(object, property)):
        # If the passed object is a piece of data or an instance,
        # document its available methods instead of its value.
        object = type(object)
        desc += ' object'
    return title % desc + '\n\n' + text.document(object, name)

def doc(*args, **kwargs):
    """Display text documentation, given an object or a path to an object."""
    try:
        text = render_doc(*args, **kwargs)
        pager(text)
    except (ImportError, ErrorDuringImport), value:
        print value
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

From skip at pobox.com  Fri Jul 28 16:02:10 2006
From: skip at pobox.com (skip at pobox.com)
Date: Fri, 28 Jul 2006 09:02:10 -0500
Subject: [Python-Dev] Another uuid problem
Message-ID: <17610.6370.422365.622901@montanaro.dyndns.org>


I just tried building and testing 2.5 on a Solaris 10 box at work.  The uuid
test failed with

    sh: ifconfig: not found
    sh: ifconfig: not found
    sh: ifconfig: not found
    sh: arp: not found
    sh: ifconfig: not found
    sh: ifconfig: not found
    sh: ifconfig: not found
    sh: arp: not found

In our environment at least it's uncommon for /usr/sbin to be in the PATH of
non-privileged users:

    piggy:% type -a ifconfig
    -bash: type: ifconfig: not found
    piggy:% PATH=$PATH:/usr/sbin type -a ifconfig
    ifconfig is /usr/sbin/ifconfig

Perhaps test_uuid needs to do a little investigation to find ifconfig and
arp.

Skip


From david.nospam.hopwood at blueyonder.co.uk  Fri Jul 28 14:50:51 2006
From: david.nospam.hopwood at blueyonder.co.uk (David Hopwood)
Date: Fri, 28 Jul 2006 13:50:51 +0100
Subject: [Python-Dev] Bad interaction of __index__ and sequence repeat
In-Reply-To: <20060728101133.GA339@code0.codespeak.net>
References: <20060728101133.GA339@code0.codespeak.net>
Message-ID: <44CA082B.6030908@blueyonder.co.uk>

Armin Rigo wrote:
> Hi,
> 
> There is an oversight in the design of __index__() that only just
> surfaced :-(  It is responsible for the following behavior, on a 32-bit
> machine with >= 2GB of RAM:
> 
>     >>> s = 'x' * (2**100)       # works!
>     >>> len(s)
>     2147483647
> 
> This is because PySequence_Repeat(v, w) works by applying w.__index__ in
> order to call v->sq_repeat.  However, __index__ is defined to clip the
> result to fit in a Py_ssize_t.

Clipping the result sounds like it would *never* be a good idea. What was
the rationale for that? It should throw an exception.

-- 
David Hopwood <david.nospam.hopwood at blueyonder.co.uk>



From grig.gheorghiu at gmail.com  Fri Jul 28 16:46:55 2006
From: grig.gheorghiu at gmail.com (Grig Gheorghiu)
Date: Fri, 28 Jul 2006 07:46:55 -0700
Subject: [Python-Dev] uuid test suite failing
In-Reply-To: <ee2a432c0607280024j56bf7facmd9ebdd26aafc013f@mail.gmail.com>
References: <eaamnu$mrq$1@sea.gmane.org> <20060727163647.GA4175@rogue.amk.ca>
	<ee2a432c0607280024j56bf7facmd9ebdd26aafc013f@mail.gmail.com>
Message-ID: <3f09d5a00607280746t57fc51b6m95cc8e1d1c927035@mail.gmail.com>

On 7/28/06, Neal Norwitz <nnorwitz at gmail.com> wrote:
>
> On 7/27/06, A.M. Kuchling <amk at amk.ca> wrote:
> > On Thu, Jul 27, 2006 at 05:40:57PM +0200, Georg Brandl wrote:
> > > The UUID test suite, which wasn't run by regrtest.py until now, is
> > > now failing on some buildbots (and my machine). This should be fixed
> > > before releasing something.
> >
> > Looking at the failures, there seem to be two problems on Unix variants:
> >  1) on some, '/sbin/ifconfig' prints a help message; you need 'ifconfig
> -a'
> >     to print information about all interfaces.
> >  2) on Solaris 9 (the only version in the SF compile farm), I can't
> >     figure out how to make ifconfig print MAC addresses at all.
> >     Searching online finds the incantation 'arp <hostname>' to print the
> >     MAC.
>
> This is such a mess.  There are so many different ways of determining
> the MAC addr on each flavour of Unix it seems hopeless to try.  I
> fixed _ifconfig_getnode so it should work on at least:  Linux, Tru64,
> Solaris, and HP-UX.  Who knows how many more variations there are.
>
> This only fixes 1 of the 2 failures in test_uuid.  The other one is
> due to _unixdll_getnode() failing.  This is because
> _uuid_generate_time is None because we couldn't find it in the uuid
> library.  This is just broken, not sure if it's the code or the test
> though.  We should handle the case if _uuid_generate_time and the
> others are None better.  I don't know what to do in this case.
>
> Since getnode ignores exceptions, maybe it's the test that is broken?



My 2 cents:  since there is no POSIX standard for getting a list of network
interfaces, trying to account for all the platform variations is one central
location is hopeless. Instead, I think the onus should be on whomever is
testing this on a particular platform -- in short, on the buildbot
maintainer on that platform.

There could be another regrtest.py-type suite, something like
platform_regrtest.py for example, which could be composed of highly
platform-dependent tests such as test_uuid.py. These tests would have empty
methods such as _ifconfig_getnode, which would then be defined on a
per-platform basis by the buildbot maintainer on that platform. The test
would obviously fail by default, unless those methods are properly defined.
Or these methods could account for just one platform, as an example of what
to do on other platforms.

Grig
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20060728/aaf51821/attachment.html 

From mwh at python.net  Fri Jul 28 17:26:37 2006
From: mwh at python.net (Michael Hudson)
Date: Fri, 28 Jul 2006 16:26:37 +0100
Subject: [Python-Dev] Bad interaction of __index__ and sequence repeat
In-Reply-To: <44CA082B.6030908@blueyonder.co.uk> (David Hopwood's message of
	"Fri, 28 Jul 2006 13:50:51 +0100")
References: <20060728101133.GA339@code0.codespeak.net>
	<44CA082B.6030908@blueyonder.co.uk>
Message-ID: <2mac6tyclu.fsf@starship.python.net>

David Hopwood <david.nospam.hopwood at blueyonder.co.uk> writes:

> Armin Rigo wrote:
>> Hi,
>> 
>> There is an oversight in the design of __index__() that only just
>> surfaced :-(  It is responsible for the following behavior, on a 32-bit
>> machine with >= 2GB of RAM:
>> 
>>     >>> s = 'x' * (2**100)       # works!
>>     >>> len(s)
>>     2147483647
>> 
>> This is because PySequence_Repeat(v, w) works by applying w.__index__ in
>> order to call v->sq_repeat.  However, __index__ is defined to clip the
>> result to fit in a Py_ssize_t.
>
> Clipping the result sounds like it would *never* be a good idea. What was
> the rationale for that? It should throw an exception.

Why would you expect range(10)[:2**32-1] and range(10)[:2**32] to do
different things?

Cheers,
mwh

-- 
  This makes it possible to pass complex object hierarchies to
  a C coder who thinks computer science has made no worthwhile
  advancements since the invention of the pointer.
                                       -- Gordon McMillan, 30 Jul 1998

From ncoghlan at gmail.com  Fri Jul 28 17:29:19 2006
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 29 Jul 2006 01:29:19 +1000
Subject: [Python-Dev] Bad interaction of __index__ and sequence repeat
In-Reply-To: <44CA082B.6030908@blueyonder.co.uk>
References: <20060728101133.GA339@code0.codespeak.net>
	<44CA082B.6030908@blueyonder.co.uk>
Message-ID: <44CA2D4F.1020601@gmail.com>

David Hopwood wrote:
> Armin Rigo wrote:
>> Hi,
>>
>> There is an oversight in the design of __index__() that only just
>> surfaced :-(  It is responsible for the following behavior, on a 32-bit
>> machine with >= 2GB of RAM:
>>
>>     >>> s = 'x' * (2**100)       # works!
>>     >>> len(s)
>>     2147483647
>>
>> This is because PySequence_Repeat(v, w) works by applying w.__index__ in
>> order to call v->sq_repeat.  However, __index__ is defined to clip the
>> result to fit in a Py_ssize_t.
> 
> Clipping the result sounds like it would *never* be a good idea. What was
> the rationale for that? It should throw an exception.

A simple demonstration of the clipping behaviour that works on machines with 
limited memory:

 >>> (2**100).__index__()
2147483647
 >>> (-2**100).__index__()
-2147483648

PEP 357 doesn't even mention the issue, and the comment on long_index in the 
code doesn't give a rationale - it just notes that the function clips the result.

Neither the PyNumber_AsIndex nor the __index__ documentation mention anything 
about the possibility of clipping, and there's no test case to verify this 
behaviour.

I'm inclined to call it a bug, too, but I've cc'ed Travis to see if he can 
shed some light on the question - the implementation of long_index explicitly 
suppresses the overflow error generated by _long_as_ssize_t, so the current 
behaviour appears to be deliberate.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
             http://www.boredomandlaziness.org

From pje at telecommunity.com  Fri Jul 28 17:30:31 2006
From: pje at telecommunity.com (Phillip J. Eby)
Date: Fri, 28 Jul 2006 11:30:31 -0400
Subject: [Python-Dev] Release manager pronouncement needed: PEP 302 Fix
In-Reply-To: <44C9A1B1.3040102@v.loewis.de>
References: <5.1.1.6.0.20060727114924.0262c980@sparrow.telecommunity.com>
	<20060727103920.GB31912@code0.codespeak.net>
	<5.1.1.6.0.20060726142456.036d9008@sparrow.telecommunity.com>
	<20060727103920.GB31912@code0.codespeak.net>
	<5.1.1.6.0.20060727114924.0262c980@sparrow.telecommunity.com>
Message-ID: <5.1.1.6.0.20060728112800.03a7fb48@sparrow.telecommunity.com>

At 07:33 AM 7/28/2006 +0200, Martin v. L?wis wrote:
>Phillip J. Eby wrote:
> > I'm willing to write code that makes it PEP 302 compliant, if the release
> > manager will bless such an addition.  But if that's not acceptable, then
> > somebody needs to produce the necessary documentation updates or revert 
> the
> > patch.  It absolutely should not be allowed to remain in *and* 
> undocumented
> > because it is a backwards-incompatible change to documented behavior of
> > Python for two major releases (2.3 and 2.4).
>
>You don't need a release manager pronouncement for that. It's a bug,
>changing it is a bug fix, you don't need RM permission to fix a bug.
>
>Do you have a patch ready that restores path_importer_cache behavior,
>yet  preserves the property that it caches existence of a directory?
>If not, I will have to produce one.

The issue is that a proper fix that caches existence requires adding new 
types to import.c and thus might appear to be more of a feature.  I was 
therefore reluctant to embark upon the work without some assurance that it 
wouldn't be rejected as adding a last-minute feature.


From nnorwitz at gmail.com  Fri Jul 28 17:53:07 2006
From: nnorwitz at gmail.com (Neal Norwitz)
Date: Fri, 28 Jul 2006 08:53:07 -0700
Subject: [Python-Dev] Another uuid problem
In-Reply-To: <17610.6370.422365.622901@montanaro.dyndns.org>
References: <17610.6370.422365.622901@montanaro.dyndns.org>
Message-ID: <ee2a432c0607280853w1d5a26d7k47264bb1430be879@mail.gmail.com>

It checks for ifconfig, /sbin/ifconfig, and /usr/sbin/ifconfig (same
for arp).  The problem is the os.pipe command doesn't hide these
issues.  It doesn't cause the test to fail, but is still broken.  The
test is presumably failing for the other reason I mentioned
(unixdll_getnode).  Let me know if you see that when running with -v.

n
--

On 7/28/06, skip at pobox.com <skip at pobox.com> wrote:
>
> I just tried building and testing 2.5 on a Solaris 10 box at work.  The uuid
> test failed with
>
>     sh: ifconfig: not found
>     sh: ifconfig: not found
>     sh: ifconfig: not found
>     sh: arp: not found
>     sh: ifconfig: not found
>     sh: ifconfig: not found
>     sh: ifconfig: not found
>     sh: arp: not found
>
> In our environment at least it's uncommon for /usr/sbin to be in the PATH of
> non-privileged users:
>
>     piggy:% type -a ifconfig
>     -bash: type: ifconfig: not found
>     piggy:% PATH=$PATH:/usr/sbin type -a ifconfig
>     ifconfig is /usr/sbin/ifconfig
>
> Perhaps test_uuid needs to do a little investigation to find ifconfig and
> arp.
>
> Skip
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/nnorwitz%40gmail.com
>

From ncoghlan at gmail.com  Fri Jul 28 17:54:30 2006
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 29 Jul 2006 01:54:30 +1000
Subject: [Python-Dev] Bad interaction of __index__ and sequence repeat
In-Reply-To: <2mac6tyclu.fsf@starship.python.net>
References: <20060728101133.GA339@code0.codespeak.net>	<44CA082B.6030908@blueyonder.co.uk>
	<2mac6tyclu.fsf@starship.python.net>
Message-ID: <44CA3336.6030807@gmail.com>

Michael Hudson wrote:
> David Hopwood <david.nospam.hopwood at blueyonder.co.uk> writes:
> 
>> Armin Rigo wrote:
>>> Hi,
>>>
>>> There is an oversight in the design of __index__() that only just
>>> surfaced :-(  It is responsible for the following behavior, on a 32-bit
>>> machine with >= 2GB of RAM:
>>>
>>>     >>> s = 'x' * (2**100)       # works!
>>>     >>> len(s)
>>>     2147483647
>>>
>>> This is because PySequence_Repeat(v, w) works by applying w.__index__ in
>>> order to call v->sq_repeat.  However, __index__ is defined to clip the
>>> result to fit in a Py_ssize_t.
>> Clipping the result sounds like it would *never* be a good idea. What was
>> the rationale for that? It should throw an exception.
> 
> Why would you expect range(10)[:2**32-1] and range(10)[:2**32] to do
> different things?

In that case, I believe it is the slice object that should be suppressing the 
overflow error (via PyErr_Occurred and PyErr_Matches) when calculating the 
indices for a given length, rather than having silent clipping be part of the 
basic implementation of long.__index__().

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
             http://www.boredomandlaziness.org

From tim.peters at gmail.com  Fri Jul 28 17:55:47 2006
From: tim.peters at gmail.com (Tim Peters)
Date: Fri, 28 Jul 2006 11:55:47 -0400
Subject: [Python-Dev] Bad interaction of __index__ and sequence repeat
In-Reply-To: <20060728101133.GA339@code0.codespeak.net>
References: <20060728101133.GA339@code0.codespeak.net>
Message-ID: <1f7befae0607280855l3f71febbo95ff5751fe31d99c@mail.gmail.com>

[Armin Rigo]
> There is an oversight in the design of __index__() that only just
> surfaced :-(  It is responsible for the following behavior, on a 32-bit
> machine with >= 2GB of RAM:
>
>     >>> s = 'x' * (2**100)       # works!
>     >>> len(s)
>     2147483647
>
> This is because PySequence_Repeat(v, w) works by applying w.__index__ in
> order to call v->sq_repeat.

?  I don't see an invocation of __index__ or nb_index in
PySequence_Repeat.  To the contrary, its /incoming/ `count` argument
is constrained to Py_ssize_t from the start:

    PyObject * PySequence_Repeat(PyObject *o, Py_ssize_t count)

... OK, I think you mean sequence_repeat() in abstract.c.  That does
invoke nb_index.  But, as below, I don't think it should in this case.

> However, __index__ is defined to clip the result to fit in a Py_ssize_t.
>  This means that the above problem exists
> with all sequences, not just strings, given enough RAM to create such
> sequences with 2147483647 items.
>
> For reference, in 2.4 we correctly get an OverflowError.
>
> Argh!  What should be done about it?

IMO, this is plain wrong.  PEP 357 isn't entirely clear, but it is
clear the author only had /slicing/ in mind (where clipping makes
sense -- and which makes `__index__` a misleading name).  Guido
pointed out the ambiguity here:

    http://mail.python.org/pipermail/python-dev/2006-February/060624.html

    There's also an ambiguity when using simple indexing. When writing
    x[i] where x is a sequence and i an object that isn't int or long but
    implements __index__, I think i.__index__() should be used rather than
    bailing out.  I suspect that you didn't think of this because you've
    already special-cased this in your code -- when a non-integer is
    passed, the mapping API is used (mp_subscript). This is done to
    suppose extended slicing. The built-in sequences (list, str, unicode,
    tuple for sure, probably more) that implement mp_subscript should
    probe for nb_index before giving up. The generic code in
    PyObject_GetItem should also check for nb_index before giving up.

So, e.g., plain a[i] shouldn't use __index__ either if i is already
int or long.  I don't see any justification for invoking nb_index in
sequence_repeat(), although if someone thinks it should, then as for
plain indexing it certainly shouldn't invoke nb_index if the incoming
count is an int or long to begin with.

Ah, fudge.  Contrary to Guido's advice above, I see that
PyObject_GetItem() /also/ unconditionally invokes nb_index (even when
the incoming key is already int or long).  It shouldn't do that either
(according to me).

OTOH, in the long discussion about PEP 357, I'm not sure anyone except
Travis was clear on whether nb_index was meant to apply only to
sequence /slicing/ or was meant to apply "everywhere an object gets
used in an index-like context".  Clipping makes sense only for the
former, but it looks like the implementation treats it more like the
latter.  This was probably exacerbated by:

    http://mail.python.org/pipermail/python-dev/2006-February/060663.html

    [Travis]
    There are other places in Python that check specifically for int objects
    and long integer objects and fail with anything else.  Perhaps all of
    these should aslo call the __index__ slot.

    [Guido]
    Right, absolutely.

This is a mess :-)

From nnorwitz at gmail.com  Fri Jul 28 17:59:51 2006
From: nnorwitz at gmail.com (Neal Norwitz)
Date: Fri, 28 Jul 2006 08:59:51 -0700
Subject: [Python-Dev] Py2.5 release schedule
In-Reply-To: <2mirlixdg4.fsf@starship.python.net>
References: <44C9A184.7030701@ewtllc.com>
	<200607281539.50794.anthony@interlink.com.au>
	<2mirlixdg4.fsf@starship.python.net>
Message-ID: <ee2a432c0607280859r6c293fa5l78c66011829ff466@mail.gmail.com>

On 7/28/06, Michael Hudson <mwh at python.net> wrote:
> Anthony Baxter <anthony at interlink.com.au> writes:
>
> > Does anyone disagree with making the next release beta3?
>
> It seems like a good idea to me.  I guess this will mean the final
> release will be pushed back a bit?

Anthony and I talked about still having b3 on Aug 1.  rc1 around Aug
17-18 (just before the Google sprint which Martin, Jeremy and I will
be attending).  Final around 24-29.  We didn't discuss with Martin
yet, so these dates are quite tentative.

n

From guido at python.org  Fri Jul 28 18:05:57 2006
From: guido at python.org (Guido van Rossum)
Date: Fri, 28 Jul 2006 09:05:57 -0700
Subject: [Python-Dev] Bad interaction of __index__ and sequence repeat
In-Reply-To: <44CA2D4F.1020601@gmail.com>
References: <20060728101133.GA339@code0.codespeak.net>
	<44CA082B.6030908@blueyonder.co.uk> <44CA2D4F.1020601@gmail.com>
Message-ID: <ca471dc20607280905y7447d005r2ab0ec664a69948@mail.gmail.com>

Argh.

I also find it a bug.

I also feel responsible because I reviewed the patch. :-(

In my recollection I tried to avoid this exact behavior. I wanted
__index__() to just return the unclipped int or long value, but have a
C API that clipped it for use in slice operations. It looks like I
failed (the patch went through so many revisions that at some point I
must've stopped caring).

--Guido

On 7/28/06, Nick Coghlan <ncoghlan at gmail.com> wrote:
> David Hopwood wrote:
> > Armin Rigo wrote:
> >> Hi,
> >>
> >> There is an oversight in the design of __index__() that only just
> >> surfaced :-(  It is responsible for the following behavior, on a 32-bit
> >> machine with >= 2GB of RAM:
> >>
> >>     >>> s = 'x' * (2**100)       # works!
> >>     >>> len(s)
> >>     2147483647
> >>
> >> This is because PySequence_Repeat(v, w) works by applying w.__index__ in
> >> order to call v->sq_repeat.  However, __index__ is defined to clip the
> >> result to fit in a Py_ssize_t.
> >
> > Clipping the result sounds like it would *never* be a good idea. What was
> > the rationale for that? It should throw an exception.
>
> A simple demonstration of the clipping behaviour that works on machines with
> limited memory:
>
>  >>> (2**100).__index__()
> 2147483647
>  >>> (-2**100).__index__()
> -2147483648
>
> PEP 357 doesn't even mention the issue, and the comment on long_index in the
> code doesn't give a rationale - it just notes that the function clips the result.
>
> Neither the PyNumber_AsIndex nor the __index__ documentation mention anything
> about the possibility of clipping, and there's no test case to verify this
> behaviour.
>
> I'm inclined to call it a bug, too, but I've cc'ed Travis to see if he can
> shed some light on the question - the implementation of long_index explicitly
> suppresses the overflow error generated by _long_as_ssize_t, so the current
> behaviour appears to be deliberate.
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
> ---------------------------------------------------------------
>              http://www.boredomandlaziness.org
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org
>


-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From tomerfiliba at gmail.com  Fri Jul 28 18:07:30 2006
From: tomerfiliba at gmail.com (tomer filiba)
Date: Fri, 28 Jul 2006 18:07:30 +0200
Subject: [Python-Dev] Fwd: patching pydoc?
In-Reply-To: <1d85506f0607280635q3a693682l230c7821dc6f408f@mail.gmail.com>
References: <1d85506f0607280635q3a693682l230c7821dc6f408f@mail.gmail.com>
Message-ID: <1d85506f0607280907x22589f98od79e925baba24ac1@mail.gmail.com>

submitted patch:
https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1530482&group_id=5470

-tomer

---------- Forwarded message ----------
From: tomer filiba <tomerfiliba at gmail.com>
Date: Jul 28, 2006 3:35 PM
Subject: patching pydoc?
To: python-dev at python.org


i have a problem with pydoc in rpyc. i wanted help(obj), where obj
is a NetProxy object, to work as if it were local.

i followed the code starting from site.help to pydoc.doc, which is the
ultimate function that generates and prints the text. i expected there
would be some function in the middle that prepares the text, and
another that writes it to the pager, but to my disappointment pydoc.doc
does both.

this means i can't transfer the document to my local machine (it's
printed directly to the remote console).

therefore, i would like to split this behavior into two parts:
* render_doc - a function that returns the document text
* doc - a function that calls render_doc and sends it to the pager

this way no existing code breaks (no existing function signatures
are changed) and i gain help on remote objects.
i hope people would be in favor, as it's not such a big change anyway.


is it possible to add to 2.5?

-tomer



this is the code of pydoc, starting at line 1457

<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
def doc(thing, title='Python Library Documentation: %s', forceload=0):
    """Display text documentation, given an object or a path to an object."""
    try:
        object, name = resolve(thing, forceload)
        desc = describe(object)
        module = inspect.getmodule(object)
        if name and '.' in name:
            desc += ' in ' + name[:name.rfind('.')]
        elif module and module is not object:
            desc += ' in module ' + module.__name__
        if not (inspect.ismodule(object) or
                inspect.isclass(object) or
                inspect.isroutine(object) or
                isinstance(object, property)):
            # If the passed object is a piece of data or an instance,
            # document its available methods instead of its value.
            object = type(object)
            desc += ' object'
        pager(title % desc + '\n\n' + text.document(object, name))
    except (ImportError, ErrorDuringImport), value:
        print value
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

this is the suggested code

<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
 def render_doc(thing, title='Python Library Documentation: %s', forceload=0):
    """generate the text"""
    object, name = resolve(thing, forceload)
    desc = describe(object)
    module = inspect.getmodule(object)
    if name and '.' in name:
        desc += ' in ' + name[:name.rfind('.')]
    elif module and module is not object:
        desc += ' in module ' + module.__name__
    if not (inspect.ismodule(object) or
            inspect.isclass(object) or
            inspect.isroutine(object) or
            isinstance(object, property)):
        # If the passed object is a piece of data or an instance,
        # document its available methods instead of its value.
        object = type(object)
        desc += ' object'
    return title % desc + '\n\n' + text.document(object, name)

def doc(*args, **kwargs):
    """Display text documentation, given an object or a path to an object."""
    try:
        text = render_doc(*args, **kwargs)
        pager(text)
    except (ImportError, ErrorDuringImport), value:
        print value
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

From ncoghlan at gmail.com  Fri Jul 28 18:33:01 2006
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 29 Jul 2006 02:33:01 +1000
Subject: [Python-Dev] Bad interaction of __index__ and sequence repeat
In-Reply-To: <1f7befae0607280855l3f71febbo95ff5751fe31d99c@mail.gmail.com>
References: <20060728101133.GA339@code0.codespeak.net>
	<1f7befae0607280855l3f71febbo95ff5751fe31d99c@mail.gmail.com>
Message-ID: <44CA3C3D.90805@gmail.com>

Tim Peters wrote:
> OTOH, in the long discussion about PEP 357, I'm not sure anyone except
> Travis was clear on whether nb_index was meant to apply only to
> sequence /slicing/ or was meant to apply "everywhere an object gets
> used in an index-like context".  Clipping makes sense only for the
> former, but it looks like the implementation treats it more like the
> latter.  This was probably exacerbated by:
> 
>     http://mail.python.org/pipermail/python-dev/2006-February/060663.html
> 
>     [Travis]
>     There are other places in Python that check specifically for int objects
>     and long integer objects and fail with anything else.  Perhaps all of
>     these should aslo call the __index__ slot.
> 
>     [Guido]
>     Right, absolutely.
> 
> This is a mess :-)

I've been trawling through the code a bit, and I don't think it's as bad as 
all that.

All I believe is really needed is to:
  - remove the PyErr_Occurred() check and its body from long_index in longobject.c
  - add a PyErr_Occurred() check to force a -1 return from PyNumber_Index in 
abstract.c
  - add a PyErr_Occurred() and PyErr_ExceptionMatches(PyOverflowError) check 
to invoke PyErr_Clear() in _PyEval_SliceIndex in ceval.c.

Add test cases to test_index.py to check that:

   (2**100).__index__() == 2**100
   (-2**100).__index__() == -2**100
   slice(-2**100, 2**100).indices(sys.maxint) == (0, sys.maxint, 1)
   "a" * 2**100 raises OverflowError

Add test cases to test_operator.py to check that:

   operator.index(2**100) == 2**100
   operator.index(-2**100) == -2**100

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
             http://www.boredomandlaziness.org

From ncoghlan at gmail.com  Fri Jul 28 20:19:24 2006
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 29 Jul 2006 04:19:24 +1000
Subject: [Python-Dev] Bad interaction of __index__ and sequence repeat
In-Reply-To: <44CA3C3D.90805@gmail.com>
References: <20060728101133.GA339@code0.codespeak.net>	<1f7befae0607280855l3f71febbo95ff5751fe31d99c@mail.gmail.com>
	<44CA3C3D.90805@gmail.com>
Message-ID: <44CA552C.9020805@gmail.com>

Nick Coghlan wrote:
> Tim Peters wrote:
>> OTOH, in the long discussion about PEP 357, I'm not sure anyone except
>> Travis was clear on whether nb_index was meant to apply only to
>> sequence /slicing/ or was meant to apply "everywhere an object gets
>> used in an index-like context".  Clipping makes sense only for the
>> former, but it looks like the implementation treats it more like the
>> latter.  This was probably exacerbated by:
>>
>>     http://mail.python.org/pipermail/python-dev/2006-February/060663.html
>>
>>     [Travis]
>>     There are other places in Python that check specifically for int objects
>>     and long integer objects and fail with anything else.  Perhaps all of
>>     these should aslo call the __index__ slot.
>>
>>     [Guido]
>>     Right, absolutely.
>>
>> This is a mess :-)
> 
> I've been trawling through the code a bit, and I don't think it's as bad as 
> all that.

Damn, it really is a mess. . . nb_index returns the Pyssize_t directly, and a 
whole heap of the code expects errors to be signalled via returning -1 before 
checking PyErr_Occurred().

To get it to work without clipping everywhere, wrap_lenfunc (typeobject.c), 
_PyEval_SliceIndex (ceval.c), PyNumber_Index (abstract.c) and sequence_repeat 
(abstract.c) all had to be modified to recognize PY_SSIZE_T_MIN and 
PY_SSIZE_T_MAX as potential error flags (in order to clear the overflow error 
for _PyEval_SliceIndex, and in order to propagate the exception for the other 
three).

And using this approach still means that (2**100).__index__() raises an 
OverflowError.

It would probably be cleaner to change the signature of nb_index to return a 
PyObject *, and let the code that uses it worry about how (or even whether!) 
to convert PyLong results to a Py_ssize_t.

Cheers,
Nick.


-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
             http://www.boredomandlaziness.org

From tjreedy at udel.edu  Fri Jul 28 20:29:50 2006
From: tjreedy at udel.edu (Terry Reedy)
Date: Fri, 28 Jul 2006 14:29:50 -0400
Subject: [Python-Dev] patching pydoc?
References: <1d85506f0607280635q3a693682l230c7821dc6f408f@mail.gmail.com>
Message-ID: <eadl2v$51q$1@sea.gmane.org>


"tomer filiba" <tomerfiliba at gmail.com> wrote in message 
news:1d85506f0607280635q3a693682l230c7821dc6f408f at mail.gmail.com...
...
> therefore, i would like to split this behavior into two parts:
> * render_doc - a function that returns the document text
> * doc - a function that calls render_doc and sends it to the pager
>
> this way no existing code breaks (no existing function signatures
> are changed) and i gain help on remote objects.
> i hope people would be in favor, as it's not such a big change anyway.
> is it possible to add to 2.5?

Giving the amount of hair-tearing over uuid and __index__, this seems like 
an especially bad day to ask for a new-feature variance in a time of 
feature freeze ;-).

Some quick questions:
* I presume you gain the new functionality by directly calling the 
factored-out render_doc and printing thru your own pager.  Does everyone?
* Would making pager() a parameter of doc() make sense?
* Is pager() the only part of the original doc() that can generate 
ImportError, ErrorDuringImport?  If not, the try/except should be in 
render_doc also or instead.
* Why generalize the doc() signature?  Bad calls will be traced as caught 
in render_doc instead of doc.  Couldn't that potentially break a bad_call 
test?

Terry Jan Reedy

> this is the code of pydoc, starting at line 1457
>
> <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
> def doc(thing, title='Python Library Documentation: %s', forceload=0):
>    """Display text documentation, given an object or a path to an 
> object."""
>    try:
>        object, name = resolve(thing, forceload)
>        desc = describe(object)
>        module = inspect.getmodule(object)
>        if name and '.' in name:
>            desc += ' in ' + name[:name.rfind('.')]
>        elif module and module is not object:
>            desc += ' in module ' + module.__name__
>        if not (inspect.ismodule(object) or
>                inspect.isclass(object) or
>                inspect.isroutine(object) or
>                isinstance(object, property)):
>            # If the passed object is a piece of data or an instance,
>            # document its available methods instead of its value.
>            object = type(object)
>            desc += ' object'
>        pager(title % desc + '\n\n' + text.document(object, name))
>    except (ImportError, ErrorDuringImport), value:
>        print value
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>
> this is the suggested code
>
> <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
> def render_doc(thing, title='Python Library Documentation: %s', 
> forceload=0):
>    """generate the text"""
>    object, name = resolve(thing, forceload)
>    desc = describe(object)
>    module = inspect.getmodule(object)
>    if name and '.' in name:
>        desc += ' in ' + name[:name.rfind('.')]
>    elif module and module is not object:
>        desc += ' in module ' + module.__name__
>    if not (inspect.ismodule(object) or
>            inspect.isclass(object) or
>            inspect.isroutine(object) or
>            isinstance(object, property)):
>        # If the passed object is a piece of data or an instance,
>        # document its available methods instead of its value.
>        object = type(object)
>        desc += ' object'
>    return title % desc + '\n\n' + text.document(object, name)
>
> def doc(*args, **kwargs):
>    """Display text documentation, given an object or a path to an 
> object."""
>    try:
>        text = render_doc(*args, **kwargs)
>        pager(text)
>    except (ImportError, ErrorDuringImport), value:
>        print value
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> http://mail.python.org/mailman/options/python-dev/python-python-dev%40m.gmane.org
> 




From guido at python.org  Fri Jul 28 20:31:09 2006
From: guido at python.org (Guido van Rossum)
Date: Fri, 28 Jul 2006 11:31:09 -0700
Subject: [Python-Dev] Bad interaction of __index__ and sequence repeat
In-Reply-To: <44CA552C.9020805@gmail.com>
References: <20060728101133.GA339@code0.codespeak.net>
	<1f7befae0607280855l3f71febbo95ff5751fe31d99c@mail.gmail.com>
	<44CA3C3D.90805@gmail.com> <44CA552C.9020805@gmail.com>
Message-ID: <ca471dc20607281131m34f752b4pb1ff933f2d2262d9@mail.gmail.com>

On 7/28/06, Nick Coghlan <ncoghlan at gmail.com> wrote:
> Nick Coghlan wrote:
> > Tim Peters wrote:
> >> OTOH, in the long discussion about PEP 357, I'm not sure anyone except
> >> Travis was clear on whether nb_index was meant to apply only to
> >> sequence /slicing/ or was meant to apply "everywhere an object gets
> >> used in an index-like context".  Clipping makes sense only for the
> >> former, but it looks like the implementation treats it more like the
> >> latter.  This was probably exacerbated by:
> >>
> >>     http://mail.python.org/pipermail/python-dev/2006-February/060663.html
> >>
> >>     [Travis]
> >>     There are other places in Python that check specifically for int objects
> >>     and long integer objects and fail with anything else.  Perhaps all of
> >>     these should aslo call the __index__ slot.
> >>
> >>     [Guido]
> >>     Right, absolutely.
> >>
> >> This is a mess :-)
> >
> > I've been trawling through the code a bit, and I don't think it's as bad as
> > all that.
>
> Damn, it really is a mess. . . nb_index returns the Pyssize_t directly, and a
> whole heap of the code expects errors to be signalled via returning -1 before
> checking PyErr_Occurred().
>
> To get it to work without clipping everywhere, wrap_lenfunc (typeobject.c),
> _PyEval_SliceIndex (ceval.c), PyNumber_Index (abstract.c) and sequence_repeat
> (abstract.c) all had to be modified to recognize PY_SSIZE_T_MIN and
> PY_SSIZE_T_MAX as potential error flags (in order to clear the overflow error
> for _PyEval_SliceIndex, and in order to propagate the exception for the other
> three).
>
> And using this approach still means that (2**100).__index__() raises an
> OverflowError.
>
> It would probably be cleaner to change the signature of nb_index to return a
> PyObject *, and let the code that uses it worry about how (or even whether!)
> to convert PyLong results to a Py_ssize_t.

No time to look through the code here, but IMO it's acceptable (at
least for 2.5) if (2**100).__index__() raises OverflowError, as long
as x[:2**100] silently clips. __index__() is primarily meant to return
a value useful for indexing concrete sequences, and 2**100 isn't.

Certainly the exception is preferrable to the silently truncated
result currently returned.

Fortunately there's some extra time since we're now going to do a third beta.

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From tim.peters at gmail.com  Fri Jul 28 20:39:28 2006
From: tim.peters at gmail.com (Tim Peters)
Date: Fri, 28 Jul 2006 14:39:28 -0400
Subject: [Python-Dev] Bad interaction of __index__ and sequence repeat
In-Reply-To: <44CA552C.9020805@gmail.com>
References: <20060728101133.GA339@code0.codespeak.net>
	<1f7befae0607280855l3f71febbo95ff5751fe31d99c@mail.gmail.com>
	<44CA3C3D.90805@gmail.com> <44CA552C.9020805@gmail.com>
Message-ID: <1f7befae0607281139ycb89b13n3ac3d829499eeab1@mail.gmail.com>

[Tim]
>>> ...
>>> This is a mess :-)

[Nick Coghlan]
>> I've been trawling through the code a bit, and I don't think it's as bad as
>> all that.

[also Nick, but older & wiser ;-)]
> Damn, it really is a mess. . . nb_index returns the Pyssize_t directly,

Bingo.  It's a /conceptual/ mess.  Best I can make out, Travis only
cared about sequence slicing (not indexing), and then the machinery
got hijacked to become a more general "can you faithfully act like an
integer?" thing -- but kept a signature that made sense only for the
original slicing use (where clipping is fine).

> and a whole heap of the code expects errors to be signalled via returning -1 before
> checking PyErr_Occurred().
>
> To get it to work without clipping everywhere, wrap_lenfunc (typeobject.c),
> _PyEval_SliceIndex (ceval.c), PyNumber_Index (abstract.c) and sequence_repeat
> (abstract.c) all had to be modified to recognize PY_SSIZE_T_MIN and
> PY_SSIZE_T_MAX as potential error flags (in order to clear the overflow error
> for _PyEval_SliceIndex, and in order to propagate the exception for the other
> three).
>
> And using this approach still means that (2**100).__index__() raises an
> OverflowError.
>
> It would probably be cleaner to change the signature of nb_index to return a
> PyObject *,

Given that the more-general use is what everyone else either wanted,
or simply /assumed/, in the original discussions, I expect it would
be, although with the understanding that the PyObject * returned must
be NULL (in case of error), or a Python int or long.

> and let the code that uses it worry about how (or even whether!)
> to convert PyLong results to a Py_ssize_t.

A utility function or two could help, like one that converted to
Py_ssize_t with clipping, and another that did the same but raised
OverflowError if Py_ssize_t isn't big enough (and in the latter case a
caller would do the usual business of checking for a -1 return and
PyErr_Occurred())..

From syfou at users.sourceforge.net  Fri Jul 28 21:24:19 2006
From: syfou at users.sourceforge.net (Sylvain Fourmanoit)
Date: Fri, 28 Jul 2006 15:24:19 -0400 (EDT)
Subject: [Python-Dev] New miniconf module
In-Reply-To: <20060727103331.GA31912@code0.codespeak.net>
References: <Pine.LNX.4.64.0607261613130.6037@sylvain>
	<Pine.LNX.4.64.0607261402410.5956@sylvain>
	<44C7B8C2.1060904@blueyonder.co.uk>
	<Pine.LNX.4.64.0607261613130.6037@sylvain>
	<5.1.1.6.0.20060726180017.0262c990@sparrow.telecommunity.com>
	<Pine.LNX.4.64.0607270200300.6000@sylvain>
	<20060727103331.GA31912@code0.codespeak.net>
Message-ID: <Pine.LNX.4.64.0607281438120.10450@sylvain>

Armin Rigo wrote:
> In the same spirit, maybe it could be slightly re-oriented towards a 
> dumper/loader for more than config files; for example, it could provide 
> a safe inverse of repr() for common built-in types

New version of miniconf (version 1.2.0) is out [1][2], including a 
unrepr() function; that's the only change this time.

Michael Foord wrote:
> ConfigObj [3] gained an 'unrepr' mode a while back. The code is simple, 
> and originally came from CherryPy.

Thanks for the link! I completely missed ConfigObj. It indeed shares a 
lot with my code. At the core, it use many of the same ideas and 
implementation techniques... In many ways, it is also a lot more 
advanced, specialized than miniconf: for instance, it introduce a new, 
specialized Config File format, while my effort aimed at keeping things 
minimal.

Armin Rigo wrote:
> If it goes in that direction, I'd suggest to rename the module to give
> it a name closer to existing persistence-related modules already in the
> stdlib.

I am not especially fond of the current miniconf name either; I didn't 
find something more suitable, yet evocative of what it does; I would be 
glad to hear any suggestion you or the rest of the developers would have.

Yours,

-- 
Sylvain <syfou at users.sourceforge.net>

The only difference between a car salesman and a computer salesman is
that the car salesman knows he's lying.

[1]http://cheeseshop.python.org/pypi?:action=display&name=miniconf&version=1.2.0
[2]http://sourceforge.net/tracker/index.php?func=detail&aid=1527597&group_id=5470&atid=355470
[3]http://www.voidspace.org.uk/python/configobj.html

P.-S. I am leaving the civilization (where I have some sort of network 
access) from July the 29th to August the 13th: I will be glad to address 
any comment, bug report or suggestion Python developers might want to 
discuss in the meantime as soon as I will be back.

From fuzzyman at voidspace.org.uk  Fri Jul 28 21:38:34 2006
From: fuzzyman at voidspace.org.uk (Michael Foord)
Date: Fri, 28 Jul 2006 20:38:34 +0100
Subject: [Python-Dev] New miniconf module
In-Reply-To: <Pine.LNX.4.64.0607281438120.10450@sylvain>
References: <Pine.LNX.4.64.0607261613130.6037@sylvain>	<Pine.LNX.4.64.0607261402410.5956@sylvain>	<44C7B8C2.1060904@blueyonder.co.uk>	<Pine.LNX.4.64.0607261613130.6037@sylvain>	<5.1.1.6.0.20060726180017.0262c990@sparrow.telecommunity.com>	<Pine.LNX.4.64.0607270200300.6000@sylvain>	<20060727103331.GA31912@code0.codespeak.net>
	<Pine.LNX.4.64.0607281438120.10450@sylvain>
Message-ID: <44CA67BA.9060803@voidspace.org.uk>

Sylvain Fourmanoit wrote:
> Armin Rigo wrote:
>   
>> In the same spirit, maybe it could be slightly re-oriented towards a 
>> dumper/loader for more than config files; for example, it could provide 
>> a safe inverse of repr() for common built-in types
>>     
>
> New version of miniconf (version 1.2.0) is out [1][2], including a 
> unrepr() function; that's the only change this time.
>
> Michael Foord wrote:
>   
>> ConfigObj [3] gained an 'unrepr' mode a while back. The code is simple, 
>> and originally came from CherryPy.
>>     
>
> Thanks for the link! I completely missed ConfigObj. It indeed shares a 
> lot with my code. At the core, it use many of the same ideas and 
> implementation techniques... In many ways, it is also a lot more 
> advanced, specialized than miniconf: for instance, it introduce a new, 
> specialized Config File format, while my effort aimed at keeping things 
> minimal.
>
>   
Cool - I made the post so you could reuse the unrepr code, but I'm glad 
you like ConfigObj. :-)

If miniconf can store and restore instances of classes (which I guess it 
does if you see it as an improvement of pickle ?) then it does more than 
ConfigObj unrepr mode - which simply allows you to store basic 
data-types in config files (amongst all the other things it does...).

Michael
http://www.voidspace.org.uk/python/index.shtml


From rhettinger at ewtllc.com  Fri Jul 28 21:36:39 2006
From: rhettinger at ewtllc.com (Raymond Hettinger)
Date: Fri, 28 Jul 2006 12:36:39 -0700
Subject: [Python-Dev] Py2.5 release schedule
In-Reply-To: <ee2a432c0607280859r6c293fa5l78c66011829ff466@mail.gmail.com>
References: <44C9A184.7030701@ewtllc.com>	<200607281539.50794.anthony@interlink.com.au>	<2mirlixdg4.fsf@starship.python.net>
	<ee2a432c0607280859r6c293fa5l78c66011829ff466@mail.gmail.com>
Message-ID: <44CA6747.2090603@ewtllc.com>

Neal Norwitz wrote:

>Anthony and I talked about still having b3 on Aug 1.  rc1 around Aug
>17-18 (just before the Google sprint which Martin, Jeremy and I will
>be attending).  Final around 24-29.  We didn't discuss with Martin
>yet, so these dates are quite tentative.
>  
>

If it doesn't muck-up your workflow, I would like to see all of these 
dates bumped back by about a week.

In particular, it may be too aggressive to expect the __index__ issue be 
fully resolved by 8/1.  Also, I have a few error-traps to add to 
setobject.c and need to review one of the implementation decisions for 
str.rpartition().  This weekend, I'll run the head on a few third-party 
packages to see if their test-suites still pass -- if not, it would be 
nice to have a bit of time to fix whatever arises.

Raymond




From fdrake at acm.org  Fri Jul 28 22:07:46 2006
From: fdrake at acm.org (Fred L. Drake, Jr.)
Date: Fri, 28 Jul 2006 16:07:46 -0400
Subject: [Python-Dev] Release manager pronouncement needed: PEP 302 Fix
In-Reply-To: <ee2a432c0607272149xc22baf4u2945da575237a83@mail.gmail.com>
References: <5.1.1.6.0.20060726142456.036d9008@sparrow.telecommunity.com>
	<5.1.1.6.0.20060727171324.0262c318@sparrow.telecommunity.com>
	<ee2a432c0607272149xc22baf4u2945da575237a83@mail.gmail.com>
Message-ID: <200607281607.46709.fdrake@acm.org>

On Friday 28 July 2006 00:49, Neal Norwitz wrote:
 > Based on this comment, is it really acceptable to just document a
 > behaviour change?  ISTM there should really only be 2 choices:  fix
 > 2.5 properly or revert the change.  This seemed to be Armin's
 > position.

I agree those are the only reasonable solutions.  I'd rather see things fixed, 
but I don't know how much time Phillip has to work on it.  I'll be working on 
the straigtening out the xmlcore issue tonight/tomorrow.


  -Fred

-- 
Fred L. Drake, Jr.   <fdrake at acm.org>

From bob at redivi.com  Fri Jul 28 22:35:40 2006
From: bob at redivi.com (Bob Ippolito)
Date: Fri, 28 Jul 2006 13:35:40 -0700
Subject: [Python-Dev] struct module and coercing floats to integers
Message-ID: <9E51A030-B610-414B-98E0-1923F1C3E862@redivi.com>

It seems that the pre-2.5 struct module has some additional  
undocumented behavior[1] that didn't percolate into the new version:  
http://python.org/sf/1530559

Python 2.4 and previous will coerce floats to integers when necessary  
as such without any kind of complaint:

$ python2.4 -c "import struct; print repr(struct.pack('>H',  
0.9999999999999999))"
'\x00\x00'

Python 2.5 refuses to coerce float to int:

$ python2.5 -c "import struct; print repr(struct.pack('>H',  
0.9999999999999999))"
Traceback (most recent call last):
   File "<string>", line 1, in <module>
   File "/Users/bob/src/python/Lib/struct.py", line 63, in pack
     return o.pack(*args)
TypeError: unsupported operand type(s) for &: 'float' and 'long'

The available options are to:

1. Reinstate the pre-2.5 weirdness
2. Reinstate the pre-2.5 weirdness with a DeprecationWarning
3. Break existing code that relies on undocumented behavior (seems  
more like a bug than lack of specification)

Either 2 or 3 seems reasonable to me, with a preference for 3 because  
none of my code depends on old bugs in the struct module :)

As far as precedent goes, the array module *used* to coerce floats  
silently, but it's had a DeprecationWarning since at least Python 2.3  
(but perhaps even earlier). Maybe it's time to promote that warning  
to an exception for Python 2.5?

[1] The pre-2.5 behavior should really be considered a bug, the  
documentation says "Return a string containing the values v1, v2, ...  
packed according to the given format. The arguments must match the  
values required by the format exactly." I wouldn't consider arbitrary  
floating point numbers to match the value required by an integer  
format exactly. Floats are not in general interchangeable with  
integers in Python anyway (e.g. list indexes, etc.).

-bob


From pje at telecommunity.com  Fri Jul 28 22:42:00 2006
From: pje at telecommunity.com (Phillip J. Eby)
Date: Fri, 28 Jul 2006 16:42:00 -0400
Subject: [Python-Dev] Release manager pronouncement needed: PEP 302 Fix
In-Reply-To: <200607281607.46709.fdrake@acm.org>
References: <ee2a432c0607272149xc22baf4u2945da575237a83@mail.gmail.com>
	<5.1.1.6.0.20060726142456.036d9008@sparrow.telecommunity.com>
	<5.1.1.6.0.20060727171324.0262c318@sparrow.telecommunity.com>
	<ee2a432c0607272149xc22baf4u2945da575237a83@mail.gmail.com>
Message-ID: <5.1.1.6.0.20060728163317.0377cbf8@sparrow.telecommunity.com>

At 04:07 PM 7/28/2006 -0400, Fred L. Drake, Jr. wrote:
>On Friday 28 July 2006 00:49, Neal Norwitz wrote:
>  > Based on this comment, is it really acceptable to just document a
>  > behaviour change?  ISTM there should really only be 2 choices:  fix
>  > 2.5 properly or revert the change.  This seemed to be Armin's
>  > position.
>
>I agree those are the only reasonable solutions.  I'd rather see things 
>fixed,
>but I don't know how much time Phillip has to work on it.  I'll be working on
>the straigtening out the xmlcore issue tonight/tomorrow.

I'm testing a semi-final version of the fix now.  It's minimally intrusive 
in that it only adds an "imp.NullImporter" type to replace the "False" 
value, and it keeps "None" to mean that the built-in import machinery 
should be used.


From martin at v.loewis.de  Fri Jul 28 22:47:34 2006
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Fri, 28 Jul 2006 22:47:34 +0200
Subject: [Python-Dev] Py2.5 release schedule
In-Reply-To: <ee2a432c0607280859r6c293fa5l78c66011829ff466@mail.gmail.com>
References: <44C9A184.7030701@ewtllc.com>	<200607281539.50794.anthony@interlink.com.au>	<2mirlixdg4.fsf@starship.python.net>
	<ee2a432c0607280859r6c293fa5l78c66011829ff466@mail.gmail.com>
Message-ID: <44CA77E6.7060200@v.loewis.de>

Neal Norwitz wrote:
> Anthony and I talked about still having b3 on Aug 1.  rc1 around Aug
> 17-18 (just before the Google sprint which Martin, Jeremy and I will
> be attending).  Final around 24-29.  We didn't discuss with Martin
> yet, so these dates are quite tentative.

That doesn't work for me. The final release must either happen before
Aug 19, or after Sep 9, or somebody else must roll the Windows binaries.

Regards,
Martin

From martin at v.loewis.de  Fri Jul 28 22:55:46 2006
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Fri, 28 Jul 2006 22:55:46 +0200
Subject: [Python-Dev] Release manager pronouncement needed: PEP 302 Fix
In-Reply-To: <5.1.1.6.0.20060728112800.03a7fb48@sparrow.telecommunity.com>
References: <5.1.1.6.0.20060727114924.0262c980@sparrow.telecommunity.com>	<20060727103920.GB31912@code0.codespeak.net>	<5.1.1.6.0.20060726142456.036d9008@sparrow.telecommunity.com>	<20060727103920.GB31912@code0.codespeak.net>	<5.1.1.6.0.20060727114924.0262c980@sparrow.telecommunity.com>
	<5.1.1.6.0.20060728112800.03a7fb48@sparrow.telecommunity.com>
Message-ID: <44CA79D2.7000209@v.loewis.de>

Phillip J. Eby wrote:
> The issue is that a proper fix that caches existence requires adding new 
> types to import.c and thus might appear to be more of a feature.  I was 
> therefore reluctant to embark upon the work without some assurance that it 
> wouldn't be rejected as adding a last-minute feature.

So do you have a patch, or are going to write one?

Regards,
Martin

From fuzzyman at voidspace.org.uk  Fri Jul 28 23:20:31 2006
From: fuzzyman at voidspace.org.uk (Michael Foord)
Date: Fri, 28 Jul 2006 22:20:31 +0100
Subject: [Python-Dev] Patch Against shutil.copytree Bug
Message-ID: <44CA7F9F.3080402@voidspace.org.uk>

Hello all,

Sourceforge is being anal and won't let me log in.

Attached is a patch for the shutil test, which would pickup bug 
"[1525866] Bug in shutil.copytree on Windows".

It passes against a bug-fixed shutil [1] on my XP box, and fails on an 
un-fixed one.

It only tests basic functionality of copytree. I did add tests that 
copystat had worked, but the mtime results were consistently off on 
Windows [2] so I removed that test.

I would appreciate it if someone could post this patch to sourceforge. 
IMO this bug needs fixing before 2.5 final, the fix is trivial and 
shutil.copytree is broken on windows without it.

All the best,


Michael Foord

[1] Bugfix supplied by Thomas Heller and is attached to the sourceforge 
bug report.
[2] And consistently off in a weird way FWIW...
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: test_shutil.diff
Url: http://mail.python.org/pipermail/python-dev/attachments/20060728/b83b9152/attachment.diff 

From gward-1337f07a94b43060ff5c1ea922ed93d6 at python.net  Wed Jul 26 04:32:13 2006
From: gward-1337f07a94b43060ff5c1ea922ed93d6 at python.net (Greg Ward)
Date: Tue, 25 Jul 2006 22:32:13 -0400
Subject: [Python-Dev] httplib and bad response chunking
Message-ID: <20060726023213.GA3106@cthulhu.gerg.ca>

So I accidentally discovered the other day that httplib does not handle
a particular type of mangled HTTP response very well.  In particular, it
tends to blow up with an undocumented ValueError when the server screws
up "chunked" encoding.  I'm not the first to discover this, either: see
http://www.python.org/sf/1486335 .

<digression>
HTTP 1.1 response chunking allows clients to know how many bytes of
response to expect for dynamic content, i.e. when it's not possible to
include a "Content-length" header.  A chunked response might look like
this:

  0005\r\nabcd\n\r\n0004\r\nabc\n\r\n0\r\n\r\n

which means:
  0x0005 bytes in first chunk, which is "abcd\n"
  0x0004 bytes in second chunk, which is "abc\n"

Each chunk size is terminated with "\r\n"; each chunk is terminated with
"\r\n"; end of response is indicated by a chunk of 0 bytes, hence the
"\r\n\r\n" at the end.

Details in RFC 2616:
  http://www.w3.org/Protocols/rfc2616/rfc2616-sec3.html#sec3.6.1
</digression>

Anyways, what I discovered in the wild the other day was a response like
this:

  0005\r\nabcd\n\r\n0004\r\nabc\n\r\n\r\n

i.e. the chunk-size for the terminating empty chunk was missing.
This cause httplib.py to blow up with ValueError because it tried to
call

  int(line, 16)

assuming that 'line' contained a hex number, when in fact it was the
empty string.  Oops.

IMHO the minimal fix is to turn ValueError into HTTPException (or a
subclass thereof); httplib should not raise ValueError just because some
server sends a bad response.  (The server in question was Apache 2.0.52
running PHP 4.3.9 sending a big hairy error page because the database
was down.)

Where I'm getting hung up is how far to test this stuff.  I have
discovered other hypothetical cases of bad chunking that cause httplib
to go into an infinite loop or block forever on socket.readline().
Should we worry about those cases as well, despite not having seen them
happen in the wild?  More annoying, I can reproduce the "block forever"
case using a real socket, but not using the StringIO-based FakeSocket
class in test_httplib.

Anyways, I've cobbled together a crude hack to test_httplib.py that
exposes the problem:

  http://sourceforge.net/tracker/download.php?group_id=5470&atid=105470&file_id=186245&aid=1486335

Feedback welcome.  (Fixing the inadvertent ValueError is trivial, so I'm
concentrating on getting the tests right first.)

Oh yeah, my patch is relative to the 2.4 branch.

        Greg
-- 
Greg Ward <gward at python.net>                         http://www.gerg.ca/
I don't believe there really IS a GAS SHORTAGE.. I think it's all just
a BIG HOAX on the part of the plastic sign salesmen -- to sell more numbers!!

From richardjones at optushome.com.au  Wed Jul 26 06:26:29 2006
From: richardjones at optushome.com.au (Richard Jones)
Date: Wed, 26 Jul 2006 14:26:29 +1000
Subject: [Python-Dev] More tracker demos online
In-Reply-To: <ea6msv$hgf$1@sea.gmane.org>
References: <44C6748C.4060701@v.loewis.de> <ea6msv$hgf$1@sea.gmane.org>
Message-ID: <200607261426.29143.richardjones@optushome.com.au>

On Wednesday 26 July 2006 13:17, Terry Reedy wrote:
> ""Martin v. L?wis"" <martin at v.loewis.de> wrote in message
> news:44C6748C.4060701 at v.loewis.de...
>
> > Currently, we have two running tracker demos online:
> >
> > Roundup:
> > http://efod.se/python-tracker/
> >
> > Jira:
> > http://jira.python.atlassian.com/secure/Dashboard.jspa
>
> What user name and passwords will they accept, if any?

For the Roundup tracker at least you should be able to recover your password 
using your sourceforge email address.


    Richard

From oliphant.travis at ieee.org  Fri Jul 28 18:15:51 2006
From: oliphant.travis at ieee.org (Travis Oliphant)
Date: Fri, 28 Jul 2006 10:15:51 -0600
Subject: [Python-Dev] Bad interaction of __index__ and sequence repeat
In-Reply-To: <44CA2D4F.1020601@gmail.com>
References: <20060728101133.GA339@code0.codespeak.net>
	<44CA082B.6030908@blueyonder.co.uk> <44CA2D4F.1020601@gmail.com>
Message-ID: <44CA3837.3040506@ieee.org>

Nick Coghlan wrote:
> David Hopwood wrote:
>> Armin Rigo wrote:
>>> Hi,
>>>
>>> There is an oversight in the design of __index__() that only just
>>> surfaced :-(  It is responsible for the following behavior, on a 32-bit
>>> machine with >= 2GB of RAM:
>>>
>>>     >>> s = 'x' * (2**100)       # works!
>>>     >>> len(s)
>>>     2147483647
>>>
>>> This is because PySequence_Repeat(v, w) works by applying 
>>> w.__index__ in
>>> order to call v->sq_repeat.  However, __index__ is defined to clip the
>>> result to fit in a Py_ssize_t.
>>
>> Clipping the result sounds like it would *never* be a good idea. What 
>> was
>> the rationale for that? It should throw an exception.
>
> A simple demonstration of the clipping behaviour that works on 
> machines with limited memory:
>
> >>> (2**100).__index__()
> 2147483647
> >>> (-2**100).__index__()
> -2147483648
>
> PEP 357 doesn't even mention the issue, and the comment on long_index 
> in the code doesn't give a rationale - it just notes that the function 
> clips the result.
I can't think of the rationale so it was probably an unclear one and 
should be thought of as a bug.  The fact that it isn't discussed in the 
PEP means it wasn't thought about clearly.  I think I had the vague idea 
that .__index_() should always succeed.  But, this shows a problem with 
that notion.
>
> I'm inclined to call it a bug, too, but I've cc'ed Travis to see if he 
> can shed some light on the question - the implementation of long_index 
> explicitly suppresses the overflow error generated by 
> _long_as_ssize_t, so the current behaviour appears to be deliberate.

If it was deliberate, it was a hurried decision and one that should be 
re-thought and probably changed.  I think the idea came from the fact 
that out-of-bounds slicing returns empty lists and since __index__ was 
primarily developed to allow integer-like objects to be used in slicing 
it adopted that behavior.  In fact it looks like the comment above 
_long_index contains words from the comment above _PyEval_SliceIndex 
showing the direct borrowing of the idea.   But, _long_index is clearly 
the wrong place to handle the situation since it is used by more than 
just the slicing code.  An error return is already handled by the 
_Eval_SliceIndex code anyway.

I say it's a bug that should be fixed.  Don't clear the error, raise it.


-Travis


From pje at telecommunity.com  Sat Jul 29 00:00:36 2006
From: pje at telecommunity.com (Phillip J. Eby)
Date: Fri, 28 Jul 2006 18:00:36 -0400
Subject: [Python-Dev] Release manager pronouncement needed: PEP 302 Fix
In-Reply-To: <44CA79D2.7000209@v.loewis.de>
References: <5.1.1.6.0.20060728112800.03a7fb48@sparrow.telecommunity.com>
	<5.1.1.6.0.20060727114924.0262c980@sparrow.telecommunity.com>
	<20060727103920.GB31912@code0.codespeak.net>
	<5.1.1.6.0.20060726142456.036d9008@sparrow.telecommunity.com>
	<20060727103920.GB31912@code0.codespeak.net>
	<5.1.1.6.0.20060727114924.0262c980@sparrow.telecommunity.com>
	<5.1.1.6.0.20060728112800.03a7fb48@sparrow.telecommunity.com>
Message-ID: <5.1.1.6.0.20060728173543.01b7c120@sparrow.telecommunity.com>

At 10:55 PM 7/28/2006 +0200, Martin v. L?wis wrote:
>Phillip J. Eby wrote:
> > The issue is that a proper fix that caches existence requires adding new
> > types to import.c and thus might appear to be more of a feature.  I was
> > therefore reluctant to embark upon the work without some assurance that it
> > wouldn't be rejected as adding a last-minute feature.
>
>So do you have a patch, or are going to write one?

Yes, it's checked in as r50916.

It ultimately turned out to be simpler than I thought; only one new type 
(imp.NullImporter) was required.


From martin at v.loewis.de  Sat Jul 29 00:44:46 2006
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sat, 29 Jul 2006 00:44:46 +0200
Subject: [Python-Dev] Bad interaction of __index__ and sequence repeat
In-Reply-To: <44CA3837.3040506@ieee.org>
References: <20060728101133.GA339@code0.codespeak.net>	<44CA082B.6030908@blueyonder.co.uk>
	<44CA2D4F.1020601@gmail.com> <44CA3837.3040506@ieee.org>
Message-ID: <44CA935E.4010901@v.loewis.de>

Travis Oliphant wrote:
> I say it's a bug that should be fixed.  Don't clear the error, raise it.

Several people have said this, but I don't think it can work.

If you raise an OverflowError in __index__, the slicing code cannot know
whether this meant as overflow or underflow (in a signed sense).

Regards,
Martin

From jjl at pobox.com  Sat Jul 29 00:59:37 2006
From: jjl at pobox.com (John J Lee)
Date: Fri, 28 Jul 2006 22:59:37 +0000 (UTC)
Subject: [Python-Dev] httplib and bad response chunking
In-Reply-To: <20060726023213.GA3106@cthulhu.gerg.ca>
References: <20060726023213.GA3106@cthulhu.gerg.ca>
Message-ID: <Pine.LNX.4.64.0607282251320.8644@localhost>

On Tue, 25 Jul 2006, Greg Ward wrote:
[...]
> Where I'm getting hung up is how far to test this stuff.

Stop when you run out of time ;-)

> I have
> discovered other hypothetical cases of bad chunking that cause httplib
> to go into an infinite loop or block forever on socket.readline().
> Should we worry about those cases as well, despite not having seen them
> happen in the wild?  More annoying, I can reproduce the "block forever"
> case using a real socket, but not using the StringIO-based FakeSocket
> class in test_httplib.

They have been seen in the wild :-)

http://python.org/sf/1411097


The IP address referenced isn't under my control, I don't know if it still 
provokes the error, but the problem is clear.


John


From rrr at ronadam.com  Sat Jul 29 01:06:02 2006
From: rrr at ronadam.com (Ron Adam)
Date: Fri, 28 Jul 2006 18:06:02 -0500
Subject: [Python-Dev] patching pydoc?
In-Reply-To: <1d85506f0607280635q3a693682l230c7821dc6f408f@mail.gmail.com>
References: <1d85506f0607280635q3a693682l230c7821dc6f408f@mail.gmail.com>
Message-ID: <44CA985A.3030408@ronadam.com>

tomer filiba wrote:
> i have a problem with pydoc in rpyc. i wanted help(obj), where obj
> is a NetProxy object, to work as if it were local.
> 
> i followed the code starting from site.help to pydoc.doc, which is the
> ultimate function that generates and prints the text. i expected there
> would be some function in the middle that prepares the text, and
> another that writes it to the pager, but to my disappointment pydoc.doc
> does both.
> 
> this means i can't transfer the document to my local machine (it's
> printed directly to the remote console).
> 
> therefore, i would like to split this behavior into two parts:
> * render_doc - a function that returns the document text
> * doc - a function that calls render_doc and sends it to the pager
> 
> this way no existing code breaks (no existing function signatures
> are changed) and i gain help on remote objects.
> i hope people would be in favor, as it's not such a big change anyway.
> 
> 
> is it possible to add to 2.5?
> 
> -tomer


Hi Tomer,

This doesn't address your small patch since the following described 
project is, not complete, will be either be a separate module or 
package, (if/when it's completed), and because it's more of a complete 
rewrite than not, would be ready and tested no sooner than 2.6. But you 
might be interested in taking a look at what I've attempted to do so far.

(This is too premature to consider adding to python distribution but 
maybe after completing, and after it is used in the community it could 
be a candidate.)


I am/was working on cleaning up pydoc by trying to separate the various 
formating, server, and output routines, because they are currently very 
much intertwined making enhancing and/or maintaining pydoc more 
difficult than it could be.  This has resulted rewriting most of the top 
level functions, which after having some success, has been put on hold 
because I have other things to do that are more immediate at the moment.

What I have done (so far) is separate the file pydoc.py into multiple 
files in order to begin to better organize the code, And rewriting each 
of those files so they are less interdependent on each other and more 
complete in them selves with the possibility of putting it back together 
in a single file or a package at some point.  (I'm leaning on it being a 
package myself.)

My attempt has focused on having pydoc create an intermediate xml-tree 
data structure that is then be used as the source to generate text, 
html, and other outputs.  This separates the introspective and 
formatting functions of pydoc.  The xml to html function mostly work. 
(some dependency link tags still need to be reimplemented.)

Another improvement I've made is having the html server prepend an html 
search bar to each page which eliminates the need to have a separate TK 
search window.  This makes interactively browsing with pydoc very easy 
and nice. :-)  It also uses a style sheet to format the html pages, 
making it easier to change colors, indents, fonts, and borders, etc...


If you are interested in working on this or would just like to take a 
look, let me know, I can put it in a zip file and send it to you.  It's 
not as well organized as I would like at the moment, but the xml and 
html generation functions work.  No attempt to create plain text output 
from the xml has been made yet.  I intended to write the xml to text 
(and paging help) functions from scratch once the html output is 
complete and fully working. (mostly done) I think it will not be 
difficult to get the output very similar to the existing pydoc help text 
form.


I'm thinking it may be better to share it at this time rather than let 
it sit on my hard drive and possibly get waylayed, or worse, forgotten 
indefinitely.

If this is the "right approach", and is "desirable", maybe it could make 
a good community project.  Possibly someone could post it and/or manage 
it someplace?  It may need a bit more cleaning up before it's ready for 
that though.


Some additional thoughts of why to do this are:

I'm hoping there may be a way to run the html server on a web site to 
generate documentation directly (pydoc-live?) instead of pre-generating 
html files and placing them on the web site.  (Possible?,  Has that 
already been done?)  I know pydoc html files are available on line, but 
I think they are all pregenerated by pydoc first then uploaded.  That 
may be preferred in most cases because it would require less server 
overhead and be more secure, but live pydocs would give more immediate 
and dynamic feedback when a project is being updated and changed often.

Another far off (and more ambitious 3.0 ?) goal may be to use the pydoc 
generated xml as a source to insert doc strings and other information 
directly into the python user manuals.  That may improve both the python 
library manuals as well as the pydoc help information by having them 
more closely tied together.  Inconsistencies would be found sooner. The 
doc strings could be the brief quick ref documents, which it currently 
isn't always brief and/or present, and the python manual could include 
the pydoc output (or selected parts of it) along with the longer 
explanation and discussion.  Having pydoc produce xml as an intermediate 
format makes these types of things easier to do.


Cheers,
    Ron Adam



From tomerfiliba at gmail.com  Sat Jul 29 02:11:19 2006
From: tomerfiliba at gmail.com (tomer filiba)
Date: Sat, 29 Jul 2006 02:11:19 +0200
Subject: [Python-Dev] patching pydoc?
Message-ID: <1d85506f0607281711q1152506bn4c3bc8a2bcadb0ef@mail.gmail.com>

> Giving the amount of hair-tearing over uuid and __index__, this seems like
> an especially bad day to ask for a new-feature variance in a time of
> feature freeze ;-).

yeah, i guess so.

> * Would making pager() a parameter of doc() make sense?

not at all.

> * I presume you gain the new functionality by directly calling the
> factored-out render_doc and printing thru your own pager.  Does everyone?

i don't quite get the question, but yes, i plan to call render_doc
instead of doc, getting the doc text, and printing it on my own console
using a pager/whatever.

about the signature, okay, it can easily be fixed. i didn't think about that
too much. about the other point -- ImportError etc -- i tried to go "dumb"
as much as possible and not make any change that would break
something. pydoc is poorly written anyway, and as ron adam noted,
there is a need for at least a cleaned-up version of pydoc, or even a
redesigned package. and it ought to be a package. it's a module of
~2000 lines. that's way too much for a module.

i made the minimal changes needed to separate the "generate text" part
from the "generate and display" part. and i kept all the semantics in tact
(including who handles the errors)

and therefore, i don't expect it to break any tests (after restoring the
original doc signature, that is). i'd hate to wait for 2.6 for such a small
change.

today my solution is to replace pydoc.pager by a fake pager that hands
over the result. this is of course not thread safe, as multiple threads
calling help() would mix their outputs. not very likely, but i'd love to
see this code go away asap.


thanks for the comments,
-tomer

> From: Terry Reedy <tjreedy <at> udel.edu>
> Subject: Re: patching pydoc?
> Newsgroups: gmane.comp.python.devel
> Date: 2006-07-28 18:29:50 GMT (5 hours and 27 minutes ago)
>
> "tomer filiba" <tomerfiliba <at> gmail.com> wrote in message
> news:1d85506f0607280635q3a693682l230c7821dc6f408f <at> mail.gmail.com...
> ...
> > therefore, i would like to split this behavior into two parts:
> > * render_doc - a function that returns the document text
> > * doc - a function that calls render_doc and sends it to the pager
> >
> > this way no existing code breaks (no existing function signatures
> > are changed) and i gain help on remote objects.
> > i hope people would be in favor, as it's not such a big change anyway.
> > is it possible to add to 2.5?
>
> Giving the amount of hair-tearing over uuid and __index__, this seems like
> an especially bad day to ask for a new-feature variance in a time of
> feature freeze ;-).
>
> Some quick questions:
> * I presume you gain the new functionality by directly calling the
> factored-out render_doc and printing thru your own pager.  Does everyone?
> * Would making pager() a parameter of doc() make sense?
> * Is pager() the only part of the original doc() that can generate
> ImportError, ErrorDuringImport?  If not, the try/except should be in
> render_doc also or instead.
> * Why generalize the doc() signature?  Bad calls will be traced as caught
> in render_doc instead of doc.  Couldn't that potentially break a bad_call
> test?
>
> Terry Jan Reedy

From chrism at plope.com  Sat Jul 29 02:02:10 2006
From: chrism at plope.com (Chris McDonough)
Date: Fri, 28 Jul 2006 20:02:10 -0400
Subject: [Python-Dev] cgi.FieldStorage DOS (sf bug #1112549)
Message-ID: <D38E3187-946F-4649-9E6D-F366E1F7E52C@plope.com>

 From the initial bugreport
(http://sourceforge.net/tracker/index.php? 
func=detail&aid=1112549&group_id=5470&atid=105470)

"""
Various parts of cgi.FieldStorage call its
"read_lines_to_outerboundary", "read_lines" and
"skip_lines" methods. These methods use the
"readline" method of the file object that represents an
input stream. The input stream is typically data
supplied by an untrusted source (such as a user
uploading a file from a web browser). The input data
is not required by the RFC 822/1521/1522/1867
specifications to contain any newline characters. For
example, it is within the bounds of the specification
to supply a a multipart/form-data input stream with a
"file-data" part that consists of a 2GB string composed
entirely of "x" characters (which happens to be
something I did that led me to noticing this bug).
"""

This bug has been around for about a year but I just worked up a  
patch yesterday that applies OK against current SVN.  It's attached  
to the issue.  Would someone be so kind as to check it in?  Guido has  
already reviewed it, I believe.

- C


From david.nospam.hopwood at blueyonder.co.uk  Sat Jul 29 02:14:21 2006
From: david.nospam.hopwood at blueyonder.co.uk (David Hopwood)
Date: Sat, 29 Jul 2006 01:14:21 +0100
Subject: [Python-Dev] Bad interaction of __index__ and sequence repeat
In-Reply-To: <44CA935E.4010901@v.loewis.de>
References: <20060728101133.GA339@code0.codespeak.net>	<44CA082B.6030908@blueyonder.co.uk>	<44CA2D4F.1020601@gmail.com>
	<44CA3837.3040506@ieee.org> <44CA935E.4010901@v.loewis.de>
Message-ID: <44CAA85D.5010907@blueyonder.co.uk>

Martin v. L?wis wrote:
> Travis Oliphant wrote:
> 
>>I say it's a bug that should be fixed.  Don't clear the error, raise it.
> 
> Several people have said this, but I don't think it can work.
> 
> If you raise an OverflowError in __index__, the slicing code cannot know
> whether this meant as overflow or underflow (in a signed sense).

Why not use IndexError for an underflow, and OverflowError for an overflow?

-- 
David Hopwood <david.nospam.hopwood at blueyonder.co.uk>




From cvaughn at gmail.com  Sat Jul 29 03:34:36 2006
From: cvaughn at gmail.com (Charles Vaughn)
Date: Fri, 28 Jul 2006 20:34:36 -0500
Subject: [Python-Dev] Eliminating loops
Message-ID: <3bcbd0d70607281834g3b21146bi5dfd700cea47d4f2@mail.gmail.com>

I'm looking for a way of modifying the compiler to eliminate any loops and
recursion from code.  It's for a high speed data processing application.
The alternative is a custom language that is little more than gloryfied
assembly.  I'd like to be able to use everything else around Python, but we
can't allow the users to create more than O(1) complexity.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20060728/0e0b1009/attachment.html 

From brett at python.org  Sat Jul 29 03:40:43 2006
From: brett at python.org (Brett Cannon)
Date: Fri, 28 Jul 2006 18:40:43 -0700
Subject: [Python-Dev] Eliminating loops
In-Reply-To: <3bcbd0d70607281834g3b21146bi5dfd700cea47d4f2@mail.gmail.com>
References: <3bcbd0d70607281834g3b21146bi5dfd700cea47d4f2@mail.gmail.com>
Message-ID: <bbaeab100607281840g6ea3db63y5eb89b11bc1a424@mail.gmail.com>

On 7/28/06, Charles Vaughn <cvaughn at gmail.com> wrote:
>
> I'm looking for a way of modifying the compiler to eliminate any loops and
> recursion from code.  It's for a high speed data processing application.
> The alternative is a custom language that is little more than gloryfied
> assembly.  I'd like to be able to use everything else around Python, but we
> can't allow the users to create more than O(1) complexity.
>

Well, the problem is how are you going to infer how many times to unroll the
loop?  range() and xrange() can change since they are only built-ins.  You
would need type inference on some locally defined variable that was set to a
syntactically defined atomic type in order to know the proper number of
times to unroll the loop and that might not work for your needs (on top of
being a pain to write out a list with however many values you want to have
the loop iterate over).

-Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20060728/ed111499/attachment-0001.htm 

From greg.ewing at canterbury.ac.nz  Sat Jul 29 04:50:21 2006
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sat, 29 Jul 2006 14:50:21 +1200
Subject: [Python-Dev] Bad interaction of __index__ and sequence repeat
In-Reply-To: <20060728101133.GA339@code0.codespeak.net>
References: <20060728101133.GA339@code0.codespeak.net>
Message-ID: <44CACCED.4090307@canterbury.ac.nz>

Armin Rigo wrote:

> This is because PySequence_Repeat(v, w) works by applying w.__index__ in
> order to call v->sq_repeat.

Why does it do that? Shouldn't __index__ only be used for
numbers which are going to be used as an index?

> However, __index__ is defined to clip the
> result to fit in a Py_ssize_t.

Why is it defined to do this instead of raising OverflowError?

--
Greg

From greg.ewing at canterbury.ac.nz  Sat Jul 29 05:09:46 2006
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sat, 29 Jul 2006 15:09:46 +1200
Subject: [Python-Dev] Bad interaction of __index__ and sequence repeat
In-Reply-To: <1f7befae0607280855l3f71febbo95ff5751fe31d99c@mail.gmail.com>
References: <20060728101133.GA339@code0.codespeak.net>
	<1f7befae0607280855l3f71febbo95ff5751fe31d99c@mail.gmail.com>
Message-ID: <44CAD17A.9090902@canterbury.ac.nz>

Tim Peters wrote:

> So, e.g., plain a[i] shouldn't use __index__ either if i is already
> int or long.  I don't see any justification for invoking nb_index in
> sequence_repeat(), although if someone thinks it should, then as for
> plain indexing it certainly shouldn't invoke nb_index if the incoming
> count is an int or long to begin with.

Hmmm. So that means anything accepting an integer index
needs to do its own range checking anyway. So having
__index__ do clipping is at best unnecessary and at worst
counterproductive, since it could suppress an overflow
or range exception that *should* be produced by the code
using the index, and would be if it got the equivalent
index value as an int or long directly.

--
Greg

From greg.ewing at canterbury.ac.nz  Sat Jul 29 05:11:57 2006
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sat, 29 Jul 2006 15:11:57 +1200
Subject: [Python-Dev] Bad interaction of __index__ and sequence repeat
In-Reply-To: <ca471dc20607280905y7447d005r2ab0ec664a69948@mail.gmail.com>
References: <20060728101133.GA339@code0.codespeak.net>
	<44CA082B.6030908@blueyonder.co.uk> <44CA2D4F.1020601@gmail.com>
	<ca471dc20607280905y7447d005r2ab0ec664a69948@mail.gmail.com>
Message-ID: <44CAD1FD.5090602@canterbury.ac.nz>

Guido van Rossum wrote:

> In my recollection I tried to avoid this exact behavior. I wanted
> __index__() to just return the unclipped int or long value, but have a
> C API that clipped it for use in slice operations.

So is there still a chance to fix it?

--
Greg

From murman at gmail.com  Sat Jul 29 05:14:58 2006
From: murman at gmail.com (Michael Urman)
Date: Fri, 28 Jul 2006 22:14:58 -0500
Subject: [Python-Dev] struct module and coercing floats to integers
In-Reply-To: <9E51A030-B610-414B-98E0-1923F1C3E862@redivi.com>
References: <9E51A030-B610-414B-98E0-1923F1C3E862@redivi.com>
Message-ID: <dcbbbb410607282014w30459351ifb09c56075997be1@mail.gmail.com>

On 7/28/06, Bob Ippolito <bob at redivi.com> wrote:
> http://python.org/sf/1530559
>
> [1] The pre-2.5 behavior should really be considered a bug, the
> documentation says "Return a string containing the values v1, v2, ...
> packed according to the given format. The arguments must match the
> values required by the format exactly." I wouldn't consider arbitrary
> floating point numbers to match the value required by an integer
> format exactly. Floats are not in general interchangeable with
> integers in Python anyway (e.g. list indexes, etc.).

While it may be a bug, it's not as hard to run into, nor as illogical
as the presentation here makes it sound. The original code[1] took a
float value between 0 and 2, and wanted to use
    pack('>H', round(value * 32768))
The workaround is a trivial change
    pack('>H', int(round(value * 32768)))
but the timeframe is less than ideal, as working code will suddenly
stop and recieve only mildly helpful error message. The fact that
round returns a float rather than an int, while intentional, does not
feature prominently in one's mine when the first version yielded the
expected results.

I would appreciate option 2 which retains compatibility but warns that
the construct is bad. I will accept any of the options, as it's clear
that floats don't make sense. It's just unfortunate that the previous
implementation let them through in a way the new implementation does
not.

[1] http://www.sacredchao.net/quodlibet/changeset/3706

Michael
-- 
Michael Urman  http://www.tortall.net/mu/blog

From greg.ewing at canterbury.ac.nz  Sat Jul 29 05:44:33 2006
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sat, 29 Jul 2006 15:44:33 +1200
Subject: [Python-Dev] Rounding float to int directly (Re: struct module and
 coercing floats to integers)
In-Reply-To: <dcbbbb410607282014w30459351ifb09c56075997be1@mail.gmail.com>
References: <9E51A030-B610-414B-98E0-1923F1C3E862@redivi.com>
	<dcbbbb410607282014w30459351ifb09c56075997be1@mail.gmail.com>
Message-ID: <44CAD9A1.8080000@canterbury.ac.nz>

Michael Urman wrote:
> The fact that
> round returns a float rather than an int, while intentional, does not
> feature prominently in one's mine when the first version yielded the
> expected results.

As an aside, does anyone else think that it would be
useful to have a builtin which rounds and converts to
an int in one go? Whenever I use round(), I almost
always want the result as an int, and making me do
it in two steps seems unnecessarily bothersome.

Since automatic float->int coercion is being increasingly
disallowed, use cases for this are becoming more and more
common.

--
Greg

From ncoghlan at gmail.com  Sat Jul 29 05:49:51 2006
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 29 Jul 2006 13:49:51 +1000
Subject: [Python-Dev] Bad interaction of __index__ and sequence repeat
In-Reply-To: <44CA935E.4010901@v.loewis.de>
References: <20060728101133.GA339@code0.codespeak.net>	<44CA082B.6030908@blueyonder.co.uk>
	<44CA2D4F.1020601@gmail.com> <44CA3837.3040506@ieee.org>
	<44CA935E.4010901@v.loewis.de>
Message-ID: <44CADADF.1040801@gmail.com>

Martin v. L?wis wrote:
> Travis Oliphant wrote:
>> I say it's a bug that should be fixed.  Don't clear the error, raise it.
> 
> Several people have said this, but I don't think it can work.
> 
> If you raise an OverflowError in __index__, the slicing code cannot know
> whether this meant as overflow or underflow (in a signed sense).

It can actually, but you have to allow 3 possible error return values from 
nb_index (-1, PY_SSIZE_T_MIN and PY_SSIZE_T_MAX).

This is ugly as hell [1], so I'm going to try to work out a patch that changes 
the signature of nb_index to return PyObject * (leaving the client code in the 
core to decide whether to clip, overflow, or leave the result alone).

Cheers,
Nick.

[1] http://www.python.org/sf/1530738

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
             http://www.boredomandlaziness.org

From ncoghlan at gmail.com  Sat Jul 29 05:53:21 2006
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 29 Jul 2006 13:53:21 +1000
Subject: [Python-Dev] New miniconf module
In-Reply-To: <Pine.LNX.4.64.0607281438120.10450@sylvain>
References: <Pine.LNX.4.64.0607261613130.6037@sylvain>	<Pine.LNX.4.64.0607261402410.5956@sylvain>	<44C7B8C2.1060904@blueyonder.co.uk>	<Pine.LNX.4.64.0607261613130.6037@sylvain>	<5.1.1.6.0.20060726180017.0262c990@sparrow.telecommunity.com>	<Pine.LNX.4.64.0607270200300.6000@sylvain>	<20060727103331.GA31912@code0.codespeak.net>
	<Pine.LNX.4.64.0607281438120.10450@sylvain>
Message-ID: <44CADBB1.6090401@gmail.com>

Sylvain Fourmanoit wrote:
> Armin Rigo wrote:
>> If it goes in that direction, I'd suggest to rename the module to give
>> it a name closer to existing persistence-related modules already in the
>> stdlib.
> 
> I am not especially fond of the current miniconf name either; I didn't 
> find something more suitable, yet evocative of what it does; I would be 
> glad to hear any suggestion you or the rest of the developers would have.

pyson :)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
             http://www.boredomandlaziness.org

From jcarlson at uci.edu  Sat Jul 29 10:05:53 2006
From: jcarlson at uci.edu (Josiah Carlson)
Date: Sat, 29 Jul 2006 01:05:53 -0700
Subject: [Python-Dev] Eliminating loops
In-Reply-To: <3bcbd0d70607281834g3b21146bi5dfd700cea47d4f2@mail.gmail.com>
References: <3bcbd0d70607281834g3b21146bi5dfd700cea47d4f2@mail.gmail.com>
Message-ID: <20060729004957.E623.JCARLSON@uci.edu>


"Charles Vaughn" <cvaughn at gmail.com> wrote:
> I'm looking for a way of modifying the compiler to eliminate any loops and
> recursion from code.  It's for a high speed data processing application.
> The alternative is a custom language that is little more than gloryfied
> assembly.  I'd like to be able to use everything else around Python, but we
> can't allow the users to create more than O(1) complexity.

One of the larger, if not largest advances in computer science in the
last 50 years was the design and implementation of looping as a
construct, not just an artifact of structured gotos, but as a method of
implementing algorithms.

With your proposed removal of loops and recursion, you are effectively
saying that users should be given a turing machine because you are
afraid of them doing foolish things with the language.  Well, since the
user is going to do foolish things with the language anyways, about all
you can really do is to test, analyze, and verify.  Oh, and educate.

What do I mean?  If your users have access to sequences of any type, and
they can't perform 'x in y', then they are going to write their own
contains/index function...

def index(x,y):
    if x == y[0]:
        return 0
    elif x == y[1]:
        return 1
    elif x == y[2]:
        return 2
    ...

If they aren't given acess to sequences, then they will do what used to
be done in QuakeC back in the day...

def index(x,y):
    if x == y0:
        return 0
    elif x == y1:
        return 1
    elif x == y2:
        return 2
    ...


While algorithm design and analysis isn't something that everyone can do
(some just can't handle the math), the users really should understand at
least a bit about algorithms before they work on the "high speed data
processing" application.

 - Josiah


From g.brandl at gmx.net  Sat Jul 29 10:07:42 2006
From: g.brandl at gmx.net (Georg Brandl)
Date: Sat, 29 Jul 2006 10:07:42 +0200
Subject: [Python-Dev] Using Python docs
Message-ID: <eaf4u4$gfc$1@sea.gmane.org>


Regarding bug 469773, I think it would be great to have such a
document "Using Python", containing the manual page and platform-
specific hints on how to invoke the interpreter and scripts
(e.g. explaining the shebang for Unices).

I'd be willing to help write up such a document.

Another thing that could be helpful is a list of "frequently needed
documentation sections", that is, a list of keywords and respective
links to topics that are hard to find for newbies, such as the section
"String formatting" or some locations in the reference manual.

Georg


From arigo at tunes.org  Sat Jul 29 10:56:12 2006
From: arigo at tunes.org (Armin Rigo)
Date: Sat, 29 Jul 2006 10:56:12 +0200
Subject: [Python-Dev] Bad interaction of __index__ and sequence repeat
In-Reply-To: <ca471dc20607281131m34f752b4pb1ff933f2d2262d9@mail.gmail.com>
References: <20060728101133.GA339@code0.codespeak.net>
	<1f7befae0607280855l3f71febbo95ff5751fe31d99c@mail.gmail.com>
	<44CA3C3D.90805@gmail.com> <44CA552C.9020805@gmail.com>
	<ca471dc20607281131m34f752b4pb1ff933f2d2262d9@mail.gmail.com>
Message-ID: <20060729085612.GA30795@code0.codespeak.net>

Hi Guido,

On Fri, Jul 28, 2006 at 11:31:09AM -0700, Guido van Rossum wrote:
> No time to look through the code here, but IMO it's acceptable (at
> least for 2.5) if (2**100).__index__() raises OverflowError, as long
> as x[:2**100] silently clips. __index__() is primarily meant to return
> a value useful for indexing concrete sequences, and 2**100 isn't.

If nb_index keeps returning a Py_ssize_t with clipping, it means that
there is no way to write in pure Python an object that emulates a long
-- only an int.  Sounds inconsistent with the int/long unification trend
for pure Python code.  It would make it awkward to write, say, pure
Python classes that pretend to be very large sequences, because using
__index__ in such code wouldn't work.

Another example of this is that if places like sequence_repeat are made
to use the following pseudo-logic:

    if isinstance(w, long) and w > sys.maxint:
        raise OverflowError
    else:
        i = w.__index__()

then if an object 'l' is an emulated pseudo-long, then  "x"*l  will
still silently clip the pseudo-long to sys.maxint.

I'm more in favor of changing nb_index to return a PyObject *, since now
is our last chance to do so.  A pair of API functions can be added to
return a Py_ssize_t with either the proper clipping, or the proper
OverflowError'ing.


A bientot,

Armin.

From ncoghlan at gmail.com  Sat Jul 29 16:06:53 2006
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 30 Jul 2006 00:06:53 +1000
Subject: [Python-Dev] Bad interaction of __index__ and sequence repeat
In-Reply-To: <20060728101133.GA339@code0.codespeak.net>
References: <20060728101133.GA339@code0.codespeak.net>
Message-ID: <44CB6B7D.1040001@gmail.com>

Armin Rigo wrote:
> Hi,
> 
> There is an oversight in the design of __index__() that only just
> surfaced :-(  It is responsible for the following behavior, on a 32-bit
> machine with >= 2GB of RAM:
> 
>     >>> s = 'x' * (2**100)       # works!
>     >>> len(s)
>     2147483647
> 
> This is because PySequence_Repeat(v, w) works by applying w.__index__ in
> order to call v->sq_repeat.  However, __index__ is defined to clip the
> result to fit in a Py_ssize_t.  This means that the above problem exists
> with all sequences, not just strings, given enough RAM to create such
> sequences with 2147483647 items.
> 
> For reference, in 2.4 we correctly get an OverflowError.
> 
> Argh!  What should be done about it?

I've now got a patch on SF that aims to fix this properly [1].

The gist of the patch:

1. Redesign the PyNumber_Index C API to serve the actual use cases in the 
interpreter core and the standard library.

   The PEP 357 abstract C API as written was bypassed by nearly all of the 
uses in the core and the standard library. The patch redesigns that API to 
reduce code duplication between the various parts of the code base that were 
previously calling nb_index directly.

   The principal change is to provide an "is_index" output variable that the 
various mp_subscript implementations can use to determine whether or not the 
passed in object was an index or not, rather than having to repeat the type 
check everywhere. The rationale for doing it this way:
   a. you may want to try something else (e.g. the mp_subscript 
implementations in the standard library try indexing before checking to see if 
the object is a slice object)
   b. a different error message may be wanted (e.g. the normal indexing 
related Type Error doesn't make sense for sequence repetition)
   c. a separate checking function would lead to repeating the check on common 
code paths (e.g. if an mp_subscript implementation did the type check first, 
and then PyNumber_Check did it again to see whether or not to raise an error)

   The output variable solves the problem nicely: either pass in NULL to get 
the default behaviour of raising a sequence indexing TypeError, or pass in a 
pointer to a C int in order to be told whether or not the typecheck succeeded 
without an exception actually being set if it fails (if the typecheck passes, 
but the actual call fails, the exception state is set as normal).

   Additionally, PyNumber_Index is redefined to raise an IndexError for values 
that cannot be represented as a Py_ssize_t. The choice of IndexError was made 
based on the dominant usage in the standard library (IndexError is the correct 
error to raise so that an mp_subscript implementation does the right thing). 
There are only a few places that need to override the IndexError to replace it 
with OverflowError (the length argument to slice.indices, sequence repetition, 
the mmap constructor), whereas all of the sequence objects (list, tuple, 
string, unicode, array), as well as PyObject_Get/Set/DelItem, need it to raise 
IndexError.

   Raising IndexError also benefits sequences implemented in Python, which can 
simply do:

   def __getitem__(self, idx):
      if isinstance(idx, slice):
          return self._get_slice(idx)
      idx = operator.index(idx) # Will raise IndexError on overflow

   A second API function PyNumber_SliceIndex is added so that the clipping 
semantics are still available where needed and _PyEval_SliceIndex is modified 
to use the new public API. This is exposed to Python code as 
operator.sliceindex().

   With the redesigned C API, the *only* code that calls the nb_index slot 
directly is the two functions in abstract.c. Everything else uses one or the 
other of those interfaces. Code duplication was significantly reduced as a 
result, and it should be much easier for 3rd party C libraries to do what they 
need to do (i.e. implementing mp_subscript slots).

2. Redefine nb_index to return a PyObject *

   Returning the PyInt/PyLong objects directly from nb_index greatly 
simplified the implementation of the nb_index methods for the affected 
classes. For classic classes, instance_index could be modified to simply 
return the result of calling __index__, as could slot_nb_index for new-style 
classes. For the standard library classes, the existing int_int method, and 
the long_long method could be used instead of needing new functions.

   This convenience should hold true for extension classes - instead of 
needing to implement __index__ separately, they should be able to reuse their 
existing __int__ or __long__ implementations.

   The other benefit is that the logic to downconvert to Py_ssize_t that was 
formerly invoked by long's __index__ method is now instead invoked by 
PyNumber_Index and PyNumber_SliceIndex. This means that directly calling an 
__index__() method allows large long results to be passed through unaffected, 
but calling the indexing operator will raise IndexError if the long is outside 
the memory address space:

   (2 ** 100).__index__() == (2**100)  # This works
   operator.index(2**100)              # This raises IndexError

The patch includes additions to test_index.py to cover these limit cases, as 
well as the necessary updates to the C API and operator module documentation.

Cheers,
Nick.

[1] http://www.python.org/sf/1530738

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
             http://www.boredomandlaziness.org

From david.nospam.hopwood at blueyonder.co.uk  Sat Jul 29 16:09:03 2006
From: david.nospam.hopwood at blueyonder.co.uk (David Hopwood)
Date: Sat, 29 Jul 2006 15:09:03 +0100
Subject: [Python-Dev] New miniconf module
In-Reply-To: <44CADBB1.6090401@gmail.com>
References: <Pine.LNX.4.64.0607261613130.6037@sylvain>	<Pine.LNX.4.64.0607261402410.5956@sylvain>	<44C7B8C2.1060904@blueyonder.co.uk>	<Pine.LNX.4.64.0607261613130.6037@sylvain>	<5.1.1.6.0.20060726180017.0262c990@sparrow.telecommunity.com>	<Pine.LNX.4.64.0607270200300.6000@sylvain>	<20060727103331.GA31912@code0.codespeak.net>	<Pine.LNX.4.64.0607281438120.10450@sylvain>
	<44CADBB1.6090401@gmail.com>
Message-ID: <44CB6BFF.4010700@blueyonder.co.uk>

Nick Coghlan wrote:
> Sylvain Fourmanoit wrote:
>>Armin Rigo wrote:
>>
>>>If it goes in that direction, I'd suggest to rename the module to give
>>>it a name closer to existing persistence-related modules already in the
>>>stdlib.
>>
>>I am not especially fond of the current miniconf name either; I didn't 
>>find something more suitable, yet evocative of what it does; I would be 
>>glad to hear any suggestion you or the rest of the developers would have.
> 
> pyson :)

Following the pattern of JSON, it would be "PYON" (PYthon Object Notation).

-- 
David Hopwood <david.nospam.hopwood at blueyonder.co.uk>



From ncoghlan at iinet.net.au  Sat Jul 29 16:19:27 2006
From: ncoghlan at iinet.net.au (Nick Coghlan)
Date: Sun, 30 Jul 2006 00:19:27 +1000
Subject: [Python-Dev] Bad interaction of __index__ and sequence repeat
In-Reply-To: <44CB6B7D.1040001@gmail.com>
References: <20060728101133.GA339@code0.codespeak.net>
	<44CB6B7D.1040001@gmail.com>
Message-ID: <44CB6E6F.5010507@iinet.net.au>

Nick Coghlan wrote:
>   The other benefit is that the logic to downconvert to Py_ssize_t that 
> was formerly invoked by long's __index__ method is now instead invoked 
> by PyNumber_Index and PyNumber_SliceIndex. This means that directly 
> calling an __index__() method allows large long results to be passed 
> through unaffected, but calling the indexing operator will raise 
> IndexError if the long is outside the memory address space:
> 
>   (2 ** 100).__index__() == (2**100)  # This works
>   operator.index(2**100)              # This raises IndexError
> 
> The patch includes additions to test_index.py to cover these limit 
> cases, as well as the necessary updates to the C API and operator module 
> documentation.

I forgot to mention the main benefit of this: when working with a 
pseudo-sequence rather than a concrete one, __index__() can be used directly 
to ensure you are working with integral data types while still allowing access 
to the full range of representable integer values.

operator.index is available for when what you have really is a concrete data 
set that is limited to the memory capacity of a single machine, and 
operator.sliceindex for when you want to clamp at the memory address space 
limits rather than raising an exception.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
             http://www.boredomandlaziness.org

From tonynelson at georgeanelson.com  Sat Jul 29 17:29:03 2006
From: tonynelson at georgeanelson.com (Tony Nelson)
Date: Sat, 29 Jul 2006 11:29:03 -0400
Subject: [Python-Dev] Testing Socket Timeouts patch 1519025
Message-ID: <v04020a01c0f129ae3d64@[192.168.123.162]>

I'm trying to write a test for my Socket Timeouts patch [1], which fixes
signal handling (notably Ctl-C == SIGINT == KeyboarInterrupt) on socket
operations using a timeout.  I don't see a portable way to send a signal,
and asking the test runner to press Ctl-C is a non-starter.  A "real"
signal is needed to interrupt the select() (or equivalent) call, because
that's what wasn't being handled correctly.  The bug should happen on the
other platforms I don't know how to test on.

Is there a portable way to send a signal?  SIGINT would be best, but
another signal (such as SIGALRM) would do, I think.

If not, should I write the test to only work on systems implementing
SIGALRM, the signal I'm using now, or implementing kill(), or what?

[1] <http://www.python.org/sf/1519025>
____________________________________________________________________
TonyN.:'                       <mailto:tonynelson at georgeanelson.com>
      '                              <http://www.georgeanelson.com/>

From ncoghlan at iinet.net.au  Sat Jul 29 18:55:13 2006
From: ncoghlan at iinet.net.au (Nick Coghlan)
Date: Sun, 30 Jul 2006 02:55:13 +1000
Subject: [Python-Dev] Bad interaction of __index__ and sequence repeat
In-Reply-To: <44CB6B7D.1040001@gmail.com>
References: <20060728101133.GA339@code0.codespeak.net>
	<44CB6B7D.1040001@gmail.com>
Message-ID: <44CB92F1.9040304@iinet.net.au>

Nick Coghlan wrote:
> Armin Rigo wrote:
>> Hi,
>>
>> There is an oversight in the design of __index__() that only just
>> surfaced :-(  It is responsible for the following behavior, on a 32-bit
>> machine with >= 2GB of RAM:
>>
>>     >>> s = 'x' * (2**100)       # works!
>>     >>> len(s)
>>     2147483647
>>
>> This is because PySequence_Repeat(v, w) works by applying w.__index__ in
>> order to call v->sq_repeat.  However, __index__ is defined to clip the
>> result to fit in a Py_ssize_t.  This means that the above problem exists
>> with all sequences, not just strings, given enough RAM to create such
>> sequences with 2147483647 items.
>>
>> For reference, in 2.4 we correctly get an OverflowError.
>>
>> Argh!  What should be done about it?
> 
> I've now got a patch on SF that aims to fix this properly [1].

I revised this patch to further reduce the code duplication associated with 
the indexing code in the standard library.

The patch now has three new functions in the abstract C API:

   PyNumber_Index (used in a dozen or so places)
     - raises IndexError on overflow
   PyNumber_AsSsize_t (used in 3 places)
     - raises OverflowError on overflow
   PyNumber_AsClippedSsize_t() (used once, by _PyEval_SliceIndex)
     - clips to PY_SSIZE_T_MIN/MAX on overflow

All 3 have an int * output argument allowing type errors to be flagged 
directly to the caller rather than through PyErr_Occurred().

Of the 3, only PyNumber_Index is exposed through the operator module.

Probably the most interesting thing now would be for Travis to review it, and 
see whether it makes things easier to handle for the Numeric scalar types 
(given the amount of code the patch deleted from the builtin and standard 
library data types, hopefully the benefits to Numeric will be comparable).

Cheers,
Nick.

[1] http://www.python.org/sf/1530738


-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
             http://www.boredomandlaziness.org

From nnorwitz at gmail.com  Sat Jul 29 21:31:29 2006
From: nnorwitz at gmail.com (Neal Norwitz)
Date: Sat, 29 Jul 2006 12:31:29 -0700
Subject: [Python-Dev] test_uuid
Message-ID: <ee2a432c0607291231s6ad5a9bdt8a3cfa599a52acb3@mail.gmail.com>

Ping,

I just checked in a change to disable testing 2 uuid functions
(_ifconfig_get_node and unixdll_getnode) that fail on many platforms.
Here's the message:

"""
Disable these tests until they are reliable across platforms. These
problems may mask more important, real problems.

One or both methods are known to fail on: Solaris, OpenBSD, Debian, Ubuntu.
They pass on Windows and some Linux boxes.
"""

Can you fix these issues or at least provide guidance how they should be fixed?

n

From python-dev at zesty.ca  Sat Jul 29 22:00:50 2006
From: python-dev at zesty.ca (Ka-Ping Yee)
Date: Sat, 29 Jul 2006 15:00:50 -0500 (CDT)
Subject: [Python-Dev] uuid test suite failing
In-Reply-To: <ee2a432c0607280024j56bf7facmd9ebdd26aafc013f@mail.gmail.com>
References: <eaamnu$mrq$1@sea.gmane.org> <20060727163647.GA4175@rogue.amk.ca>
	<ee2a432c0607280024j56bf7facmd9ebdd26aafc013f@mail.gmail.com>
Message-ID: <Pine.LNX.4.58.0607291458310.23895@server1.LFW.org>

On Fri, 28 Jul 2006, Neal Norwitz wrote:
> This only fixes 1 of the 2 failures in test_uuid.  The other one is
> due to _unixdll_getnode() failing.  This is because
> _uuid_generate_time is None because we couldn't find it in the uuid
> library.  This is just broken, not sure if it's the code or the test
> though.  We should handle the case if _uuid_generate_time and the
> others are None better.  I don't know what to do in this case.

The design intention is this: each of the various *_getnode() functions
is supposed to work on the platform for which it was written.  For
example, _windll_getnode() is supposed to work on Windows, and will
raise an exception on other platforms; if it raises an exception on
Windows, something is wrong (the code's expectations of the OS are
not met).  When uuid_generate_time is unavailable, _unixdll_getnode()
is supposed to fail.

The getnode() function is just supposed to get an available MAC
address; that's why it catches any exceptions raised by the
*_getnode() functions.


-- ?!ng

From python-dev at zesty.ca  Sat Jul 29 22:05:16 2006
From: python-dev at zesty.ca (Ka-Ping Yee)
Date: Sat, 29 Jul 2006 15:05:16 -0500 (CDT)
Subject: [Python-Dev] test_uuid
In-Reply-To: <ee2a432c0607291231s6ad5a9bdt8a3cfa599a52acb3@mail.gmail.com>
References: <ee2a432c0607291231s6ad5a9bdt8a3cfa599a52acb3@mail.gmail.com>
Message-ID: <Pine.LNX.4.58.0607291501220.23895@server1.LFW.org>

On Sat, 29 Jul 2006, Neal Norwitz wrote:
> I just checked in a change to disable testing 2 uuid functions
> (_ifconfig_get_node and unixdll_getnode) that fail on many platforms.
> Here's the message:
>
> """
> Disable these tests until they are reliable across platforms. These
> problems may mask more important, real problems.
>
> One or both methods are known to fail on: Solaris, OpenBSD, Debian, Ubuntu.
> They pass on Windows and some Linux boxes.
> """

_ifconfig_get_node() should work on all Linuxes.  (Thanks for fixing
it to work on more types of Unix.)

It's okay for unixdll_getnode to fail when the necessary shared library
is not available.

Ideally, test_uuid should serve as a record of which platforms we expect
these routines to work on.  The uuid module as a whole isn't broken if
one of these routines fails; it just means that we don't have complete
platform coverage and/or test_uuid has inaccurate expectations of which
routines work on which platforms.


-- ?!ng

From jcarlson at uci.edu  Sat Jul 29 23:38:38 2006
From: jcarlson at uci.edu (Josiah Carlson)
Date: Sat, 29 Jul 2006 14:38:38 -0700
Subject: [Python-Dev] Testing Socket Timeouts patch 1519025
In-Reply-To: <v04020a01c0f129ae3d64@[192.168.123.162]>
References: <v04020a01c0f129ae3d64@[192.168.123.162]>
Message-ID: <20060729143048.E630.JCARLSON@uci.edu>


Tony Nelson <tonynelson at georgeanelson.com> wrote:
> 
> I'm trying to write a test for my Socket Timeouts patch [1], which fixes
> signal handling (notably Ctl-C == SIGINT == KeyboarInterrupt) on socket
> operations using a timeout.  I don't see a portable way to send a signal,
> and asking the test runner to press Ctl-C is a non-starter.  A "real"
> signal is needed to interrupt the select() (or equivalent) call, because
> that's what wasn't being handled correctly.  The bug should happen on the
> other platforms I don't know how to test on.
> 
> Is there a portable way to send a signal?  SIGINT would be best, but
> another signal (such as SIGALRM) would do, I think.

According to my (limited) research on signals, Windows signal support is
horrible.  I have not been able to have Python send signals of any kind
other than SIGABRT, and then only to the currently running process,
which kills it (regardless of whether you have a signal handler or not).

> If not, should I write the test to only work on systems implementing
> SIGALRM, the signal I'm using now, or implementing kill(), or what?

I think that most non-Windows platforms should have non-braindead signal
support, though the signal module seems to be severely lacking in
sending any signal except for SIGALRM, and the os module has its fingers
on SIGABRT.


If someone is looking for a project for 2.6 that digs into all sorts of
platform-specific nastiness, they could add actual signal sending to the
signal module (at least for unix systems).

 - Josiah


From exarkun at divmod.com  Sat Jul 29 23:44:28 2006
From: exarkun at divmod.com (Jean-Paul Calderone)
Date: Sat, 29 Jul 2006 17:44:28 -0400
Subject: [Python-Dev] Testing Socket Timeouts patch 1519025
In-Reply-To: <20060729143048.E630.JCARLSON@uci.edu>
Message-ID: <20060729214428.1717.1630815704.divmod.quotient.6244@ohm>

On Sat, 29 Jul 2006 14:38:38 -0700, Josiah Carlson <jcarlson at uci.edu> wrote:
>
>If someone is looking for a project for 2.6 that digs into all sorts of
>platform-specific nastiness, they could add actual signal sending to the
>signal module (at least for unix systems).
>

Maybe I am missing something obvious, but what is necessary beyond
os.kill()?

What /would/ be useful is a complete sigaction wrapper, but that's a
completely separate topic.

Jean-Paul

From tonynelson at georgeanelson.com  Sun Jul 30 00:18:41 2006
From: tonynelson at georgeanelson.com (Tony Nelson)
Date: Sat, 29 Jul 2006 18:18:41 -0400
Subject: [Python-Dev] Testing Socket Timeouts patch 1519025
In-Reply-To: <20060729143048.E630.JCARLSON@uci.edu>
References: <v04020a01c0f129ae3d64@[192.168.123.162]>
	<v04020a01c0f129ae3d64@[192.168.123.162]>
Message-ID: <v04020a01c0f18c283570@[192.168.123.162]>

At 2:38 PM -0700 7/29/06, Josiah Carlson wrote:
>Tony Nelson <tonynelson at georgeanelson.com> wrote:
>>
>> I'm trying to write a test for my Socket Timeouts patch [1], which fixes
>> signal handling (notably Ctl-C == SIGINT == KeyboarInterrupt) on socket
>> operations using a timeout.  I don't see a portable way to send a signal,
>> and asking the test runner to press Ctl-C is a non-starter.  A "real"
>> signal is needed to interrupt the select() (or equivalent) call, because
>> that's what wasn't being handled correctly.  The bug should happen on the
>> other platforms I don't know how to test on.
>>
>> Is there a portable way to send a signal?  SIGINT would be best, but
>> another signal (such as SIGALRM) would do, I think.
>
>According to my (limited) research on signals, Windows signal support is
>horrible.  I have not been able to have Python send signals of any kind
>other than SIGABRT, and then only to the currently running process,
>which kills it (regardless of whether you have a signal handler or not).

Hmm, OK, darn, thanks.  MSWindows does allow users to press Ctl-C to send a
KeyboardInterrupt, so it's just too bad if I can't find a way to test it
from a script.


>> If not, should I write the test to only work on systems implementing
>> SIGALRM, the signal I'm using now, or implementing kill(), or what?
>
>I think that most non-Windows platforms should have non-braindead signal
>support, though the signal module seems to be severely lacking in
>sending any signal except for SIGALRM, and the os module has its fingers
>on SIGABRT.

The test now checks "hasattr(signal, 'alarm')" before proceeding, so at
least it won't die horribly.


>If someone is looking for a project for 2.6 that digs into all sorts of
>platform-specific nastiness, they could add actual signal sending to the
>signal module (at least for unix systems).

Isn't signal sending the province of kill (2) (or os.kill()) in python)?
Not that I know much about it.

BTW, I picked SIGALRM because I could do it all with one thread.  Reading
POSIX, ISTM that if I sent the signal from another thread, it would bounce
off that thread to the main thread during the call to kill(), at which
point I got the willies.  OTOH, if kill() is more widely available than
alarm(), I'll give it a try, but going by the docs, I'd say it isn't.
____________________________________________________________________
TonyN.:'                       <mailto:tonynelson at georgeanelson.com>
      '                              <http://www.georgeanelson.com/>

From martin at v.loewis.de  Sun Jul 30 09:42:14 2006
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sun, 30 Jul 2006 09:42:14 +0200
Subject: [Python-Dev] Testing Socket Timeouts patch 1519025
In-Reply-To: <v04020a01c0f18c283570@[192.168.123.162]>
References: <v04020a01c0f129ae3d64@[192.168.123.162]>	<v04020a01c0f129ae3d64@[192.168.123.162]>
	<v04020a01c0f18c283570@[192.168.123.162]>
Message-ID: <44CC62D6.9070703@v.loewis.de>

Tony Nelson schrieb:
> Hmm, OK, darn, thanks.  MSWindows does allow users to press Ctl-C to send a
> KeyboardInterrupt, so it's just too bad if I can't find a way to test it
> from a script.

You can use GenerateConsoleCtrlEvent to send Ctrl-C to all processes
that share the console of the calling process.

> BTW, I picked SIGALRM because I could do it all with one thread.  Reading
> POSIX, ISTM that if I sent the signal from another thread, it would bounce
> off that thread to the main thread during the call to kill(), at which
> point I got the willies.  OTOH, if kill() is more widely available than
> alarm(), I'll give it a try, but going by the docs, I'd say it isn't.

Indeed, alarm should be available on any POSIX system.

Regards,
Martin

From phd at mail2.phd.pp.ru  Sun Jul 30 11:29:57 2006
From: phd at mail2.phd.pp.ru (Oleg Broytmann)
Date: Sun, 30 Jul 2006 13:29:57 +0400
Subject: [Python-Dev] first draft of bug guidelines for
	www.python.org/dev/
In-Reply-To: <bbaeab100607202052t68ac220cs70b32a7c6a3190a2@mail.gmail.com>
References: <bbaeab100607202052t68ac220cs70b32a7c6a3190a2@mail.gmail.com>
Message-ID: <20060730092957.GB29899@phd.pp.ru>

On Thu, Jul 20, 2006 at 08:52:34PM -0700, Brett Cannon wrote:
> * Summary
>     A one-line describing the problem so as to make it easy for
> developers to spot whether they have the expertise needed to work on
> the bug.

   Summary is also displayed as a title on index and search pages, so it is
important to make the summary right - short and descriptive.

Oleg.
-- 
     Oleg Broytmann            http://phd.pp.ru/            phd at phd.pp.ru
           Programmers don't die, they just GOSUB without RETURN.

From martin at v.loewis.de  Sun Jul 30 15:14:13 2006
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sun, 30 Jul 2006 15:14:13 +0200
Subject: [Python-Dev] Which version of distutils to ship with Python 2.5?
In-Reply-To: <200607271701.58223.anthony@interlink.com.au>
References: <43aa6ff70607261924w65e01325o854f87d4c109026a@mail.gmail.com>
	<44C85FF2.3030701@v.loewis.de>
	<200607271701.58223.anthony@interlink.com.au>
Message-ID: <44CCB0A5.30706@v.loewis.de>

Anthony Baxter schrieb:
>> In any case, I bumped the version number to 2.5, according to the
>> policy discussed in
>>
> 
> Could this not simply use the Python version number directly, instead?

See the prior discussion at

http://mail.python.org/pipermail/distutils-sig/2005-January/004366.html

Some people still believe (at least, believed in January 2005), that
distutils is developed independently of Python, and thus deserves
its own version number.

Of course, Andrew Kuchling officially declared in r1982 of pep 291
that there won't be any further stand-alone distutils releases,
and therefore, backwards compatibility with 2.1 is not necessary
anymore.

So I changed distutils.__version__ again, to be derived from
sys.version_info.

I left the numerous comments still in distutils that compatibility
with 2.1 is desired. We should remove these after 2.5 is released
(or perhaps even before that).

Regards,
Martin

From jcarlson at uci.edu  Sun Jul 30 18:50:36 2006
From: jcarlson at uci.edu (Josiah Carlson)
Date: Sun, 30 Jul 2006 09:50:36 -0700
Subject: [Python-Dev] Testing Socket Timeouts patch 1519025
In-Reply-To: <20060729214428.1717.1630815704.divmod.quotient.6244@ohm>
References: <20060729143048.E630.JCARLSON@uci.edu>
	<20060729214428.1717.1630815704.divmod.quotient.6244@ohm>
Message-ID: <20060730093610.E634.JCARLSON@uci.edu>


Jean-Paul Calderone <exarkun at divmod.com> wrote:
> 
> On Sat, 29 Jul 2006 14:38:38 -0700, Josiah Carlson <jcarlson at uci.edu> wrote:
> >
> >If someone is looking for a project for 2.6 that digs into all sorts of
> >platform-specific nastiness, they could add actual signal sending to the
> >signal module (at least for unix systems).
> 
> Maybe I am missing something obvious, but what is necessary beyond
> os.kill()?

I note that os.kill() does everything necessary for posix systems. 
I didn't notice that it took an argument for the kind of signal.

A new project for someone: combine all of the methods available to
Windows into a single function.


> What /would/ be useful is a complete sigaction wrapper, but that's a
> completely separate topic.

Like atexit, only a stack per signal?

 - Josiah


From tonynelson at georgeanelson.com  Sun Jul 30 19:10:32 2006
From: tonynelson at georgeanelson.com (Tony Nelson)
Date: Sun, 30 Jul 2006 13:10:32 -0400
Subject: [Python-Dev] Testing Socket Timeouts patch 1519025
In-Reply-To: <44CC62D6.9070703@v.loewis.de>
References: <v04020a01c0f18c283570@[192.168.123.162]>
	<v04020a01c0f129ae3d64@[192.168.123.162]>
	<v04020a01c0f129ae3d64@[192.168.123.162]>
	<v04020a01c0f18c283570@[192.168.123.162]>
Message-ID: <v04020a01c0f286dddce0@[192.168.123.162]>

At 9:42 AM +0200 7/30/06, Martin v. L?wis wrote:
>Tony Nelson schrieb:
>> Hmm, OK, darn, thanks.  MSWindows does allow users to press Ctl-C to send a
>> KeyboardInterrupt, so it's just too bad if I can't find a way to test it
>> from a script.
>
>You can use GenerateConsoleCtrlEvent to send Ctrl-C to all processes
>that share the console of the calling process.

That looks like it would work, but it seems prone to overkill.  To avoid
killing all the processes running from a console, the test would need to be
run in a subprocess in a new process group.  If the test simply sends the
event to its own process, all the other processes in its process group
would receive the event as well, and probably die.  I would expect that all
the processes sharing the console would die, but even if they didn't when I
tried it, I couldn't be sure that it wouldn't happen elsewhere, say when
run from a .bat file.

Martin, your advice is usually spot-on, but I don't always understand it.
Maybe using it here is just complicated.  I expect that
GenerateConsoleCtrlEvent() can be called through the ctypes module, though
that would make backporting the test to 2.4 a bit more difficult.  It looks
like the subprocess module can be passed the needed creation flag to make a
new process group.  The subprocess can send the event to itself, and could
return the test result in its result code, so that part isn't so bad.  To
avoid adding a new file to the distribution, test_socket.test_main() could
be modified to look for a command line argument requesting the particular
test action.


>> BTW, I picked SIGALRM because I could do it all with one thread.  Reading
>> POSIX, ISTM that if I sent the signal from another thread, it would bounce
>> off that thread to the main thread during the call to kill(), at which
>> point I got the willies.  OTOH, if kill() is more widely available than
>> alarm(), I'll give it a try, but going by the docs, I'd say it isn't.
>
>Indeed, alarm should be available on any POSIX system.

Well, if alarm() is available, then the test will work.  If not, it will be
silently skipped, as are some other tests already in test_socket.py.  I
can't offhand tell if MSWindows supports alarm(), but RiscOS and OS2 do not.
____________________________________________________________________
TonyN.:'                       <mailto:tonynelson at georgeanelson.com>
      '                              <http://www.georgeanelson.com/>

From exarkun at divmod.com  Sun Jul 30 19:20:54 2006
From: exarkun at divmod.com (Jean-Paul Calderone)
Date: Sun, 30 Jul 2006 13:20:54 -0400
Subject: [Python-Dev] Testing Socket Timeouts patch 1519025
In-Reply-To: <20060730093610.E634.JCARLSON@uci.edu>
Message-ID: <20060730172054.1717.102253317.divmod.quotient.7639@ohm>

On Sun, 30 Jul 2006 09:50:36 -0700, Josiah Carlson <jcarlson at uci.edu> wrote:
>
>Jean-Paul Calderone <exarkun at divmod.com> wrote:
>>
>> On Sat, 29 Jul 2006 14:38:38 -0700, Josiah Carlson <jcarlson at uci.edu> wrote:
>> >
>> >If someone is looking for a project for 2.6 that digs into all sorts of
>> >platform-specific nastiness, they could add actual signal sending to the
>> >signal module (at least for unix systems).
>>
>> Maybe I am missing something obvious, but what is necessary beyond
>> os.kill()?
>
>I note that os.kill() does everything necessary for posix systems.
>I didn't notice that it took an argument for the kind of signal.
>
>A new project for someone: combine all of the methods available to
>Windows into a single function.
>
>> What /would/ be useful is a complete sigaction wrapper, but that's a
>> completely separate topic.
>
>Like atexit, only a stack per signal?

I just mean a complete wrapping of sigaction(2).  In particular, I
need this functionality to properly install a SIGCHLD handler which
does not interfer with various I/O functions by installing the handler
with the SA_RESTART flag, which is not currently possible using the
signal.signal function.

Jean-Paul

From barry at python.org  Sun Jul 30 21:44:08 2006
From: barry at python.org (Barry Warsaw)
Date: Sun, 30 Jul 2006 15:44:08 -0400
Subject: [Python-Dev] Py2.5 release schedule
In-Reply-To: <BBBDED8E-ACDD-4176-985A-144188B52F1E@python.org>
References: <44C9A184.7030701@ewtllc.com>
	<200607281539.50794.anthony@interlink.com.au>
	<BBBDED8E-ACDD-4176-985A-144188B52F1E@python.org>
Message-ID: <3E70BA16-8DAA-487D-9901-B1597D912222@python.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Re: Python 2.5 compatibility

On Jul 28, 2006, at 8:57 AM, Barry Warsaw wrote:

> +1.  It would give me more type to port and test a few of my  
> applications to the new version.
>
> I'm still working on Mailman but the most painful thing so far has  
> been the conversion of exceptions to new-style classes, and even  
> that wasn't /too/ painful.

I believe I've finished porting Mailman to Python 2.5.  None of the  
issues were insurmountable, but here they are FTR:

1) Exceptions are new-style classes but Mailman was doing one  
specific test against the type of an object to see if it needed to be  
instantiated.  This test was written as:

if isinstance(obj, ClassType)

which fails in Python 2.5.  I actually rewrote it like so:

if isinstance(obj, ClassType) or isinstance(obj, type(type))

in MM2.1 because it has to be backward compatible to Python 2.1.

2) There was one place where I was commonly raising a string and that  
generates deprecation warnings, so I changed that to a class.  There  
are a few other legacy string exceptions defined in the code, but  
they are rarely used and should all get rewritten as well.

3) Cookie.py's repr changed so that trailing semicolons are no longer  
output on the end of text lines.  I understand this change was made  
so that Python cookies were more RFC compliant, but it broke some  
homegrown cookie text splitting we were doing.  I changed this code  
to split on lines first.

All in all, not too bad although the Cookie.py change took a while to  
track down!

- -Barry

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.3 (Darwin)

iQCVAwUBRM0MDnEjvBPtnXfVAQJ9ZAP/VdtM79SXgx7s/X0aEIu4HDZva7TkYyi6
dRzlgAtEV5BN1yYn+vzw8PBCtdy+9N3yYtv/zqdQP54mZDjsaGaNw6MiS0jsETRy
248hj3otL/00WTrKWh8/OvDlLW8KUNQI4MWBOMKJ/TqYW5Es4fJGEMtbO/xqGXXD
/wgWmmLOOAE=
=Mu8m
-----END PGP SIGNATURE-----

From fdrake at acm.org  Sun Jul 30 21:55:40 2006
From: fdrake at acm.org (Fred L. Drake, Jr.)
Date: Sun, 30 Jul 2006 15:55:40 -0400
Subject: [Python-Dev] Py2.5 release schedule
In-Reply-To: <3E70BA16-8DAA-487D-9901-B1597D912222@python.org>
References: <44C9A184.7030701@ewtllc.com>
	<BBBDED8E-ACDD-4176-985A-144188B52F1E@python.org>
	<3E70BA16-8DAA-487D-9901-B1597D912222@python.org>
Message-ID: <200607301555.40541.fdrake@acm.org>

On Sunday 30 July 2006 15:44, Barry Warsaw wrote:
 > if isinstance(obj, ClassType) or isinstance(obj, type(type))

Looks like you've got a possible name clash in the second isinstance.  ;-)


  -Fred

-- 
Fred L. Drake, Jr.   <fdrake at acm.org>

From g.brandl at gmx.net  Sun Jul 30 22:17:41 2006
From: g.brandl at gmx.net (Georg Brandl)
Date: Sun, 30 Jul 2006 22:17:41 +0200
Subject: [Python-Dev] Py2.5 release schedule
In-Reply-To: <3E70BA16-8DAA-487D-9901-B1597D912222@python.org>
References: <44C9A184.7030701@ewtllc.com>	<200607281539.50794.anthony@interlink.com.au>	<BBBDED8E-ACDD-4176-985A-144188B52F1E@python.org>
	<3E70BA16-8DAA-487D-9901-B1597D912222@python.org>
Message-ID: <eaj42s$6n0$1@sea.gmane.org>

Barry Warsaw wrote:

> if isinstance(obj, ClassType)
> 
> which fails in Python 2.5.  I actually rewrote it like so:
> 
> if isinstance(obj, ClassType) or isinstance(obj, type(type))

The second "type" seems to be superfluous. ;)

Georg


From fdrake at acm.org  Sun Jul 30 22:27:16 2006
From: fdrake at acm.org (Fred L. Drake, Jr.)
Date: Sun, 30 Jul 2006 16:27:16 -0400
Subject: [Python-Dev] Py2.5 release schedule
In-Reply-To: <eaj42s$6n0$1@sea.gmane.org>
References: <44C9A184.7030701@ewtllc.com>
	<3E70BA16-8DAA-487D-9901-B1597D912222@python.org>
	<eaj42s$6n0$1@sea.gmane.org>
Message-ID: <200607301627.17309.fdrake@acm.org>

On Sunday 30 July 2006 16:17, Georg Brandl wrote:
 > The second "type" seems to be superfluous. ;)

I was thinking it suggested there was a local named "type".  But if not, yeah.

I get the impression Barry's pretty new to this "Python thing."  Wonder what 
he's been up to.  ;-)


  -Fred

-- 
Fred L. Drake, Jr.   <fdrake at acm.org>

From barry at python.org  Sun Jul 30 23:07:22 2006
From: barry at python.org (Barry Warsaw)
Date: Sun, 30 Jul 2006 17:07:22 -0400
Subject: [Python-Dev] Py2.5 release schedule
In-Reply-To: <200607301627.17309.fdrake@acm.org>
References: <44C9A184.7030701@ewtllc.com>
	<3E70BA16-8DAA-487D-9901-B1597D912222@python.org>
	<eaj42s$6n0$1@sea.gmane.org> <200607301627.17309.fdrake@acm.org>
Message-ID: <ADB1B68C-7C90-4466-BE43-921FDFA7ECEE@python.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Jul 30, 2006, at 4:27 PM, Fred L. Drake, Jr. wrote:

> On Sunday 30 July 2006 16:17, Georg Brandl wrote:
>> The second "type" seems to be superfluous. ;)
>
> I was thinking it suggested there was a local named "type".  But if  
> not, yeah.
>
> I get the impression Barry's pretty new to this "Python thing."   
> Wonder what
> he's been up to.  ;-)

As I mentioned, this has to be compatible with Python 2.1:

Python 2.1.3+ (#1, Apr 25 2005, 22:52:02)
[GCC 3.3.5-20050130 (Gentoo Linux 3.3.5.20050130-r1,  
ssp-3.3.5.20050130-1, pie- on linux2
Type "copyright", "credits" or "license" for more information.
 >>> isinstance(Exception, type)
Traceback (most recent call last):
   File "<stdin>", line 1, in ?
TypeError: isinstance() arg 2 must be a class or type
 >>> isinstance(Exception, type(type))
0

Python 2.5b2 (trunk:50835, Jul 25 2006, 23:27:51)
[GCC 3.4.6 (Gentoo 3.4.6-r1, ssp-3.4.5-1.0, pie-8.7.9)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
 >>> isinstance(Exception, type)
True
 >>> isinstance(Exception, type(type))
True

I thought that was rather clever actually. :)

- -Barry

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.3 (Darwin)

iQCVAwUBRM0fj3EjvBPtnXfVAQI1AQP+ID6BSbJ/4TL7cizvMjrxHD6JVjFKBzD6
7FCwXpvELQt7vlDWGrXWi+Lai/93nGqD42VYRSgtHqFP2gMYKEkM+TaLl91YFuSh
B6jO7l5wW7SGlyLQQiibmPBp8uGDG30F1ylM9e9y9c69Hy1LJEa1sG8/AS/FiA+n
fpQ/WlTNRvA=
=uGDZ
-----END PGP SIGNATURE-----

From martin at v.loewis.de  Sun Jul 30 23:34:30 2006
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sun, 30 Jul 2006 23:34:30 +0200
Subject: [Python-Dev] Py2.5 release schedule
In-Reply-To: <200607301555.40541.fdrake@acm.org>
References: <44C9A184.7030701@ewtllc.com>	<BBBDED8E-ACDD-4176-985A-144188B52F1E@python.org>	<3E70BA16-8DAA-487D-9901-B1597D912222@python.org>
	<200607301555.40541.fdrake@acm.org>
Message-ID: <44CD25E6.5040608@v.loewis.de>

Fred L. Drake, Jr. schrieb:
> On Sunday 30 July 2006 15:44, Barry Warsaw wrote:
>  > if isinstance(obj, ClassType) or isinstance(obj, type(type))
> 
> Looks like you've got a possible name clash in the second isinstance.  ;-)

Nah, that's rather an entry to the obfuscated Python contest.
The two occurrences of type really mean to refer to the same
thing; this is the test whether obj _is a_ new-style class.

Normally, you would write isinstance(obj, type), but that gives
a TypeError in 2.1 (isinstance() arg 2 must be a class or type).
In 2.1, type(type) is FunctionType, so the test should
fail (in the context, as obj ought to be a string, an exception
object, or an exception type). In 2.2 and later, we have

>>> type(type) is type
1 # sometimes True instead

I think I would have rewritten as

try:
  # Instantiate it if possible and necessary
  exc = exc()
except AttributeError: # no __call__; it's already an object
  pass

(assuming that the mailman exceptions don't have __call__) or as

if not isinstance(exc, Exception):
   exc = exc()

(assuming that string exceptions are gone, but the code below
 already assumes that exc should be an object that supports
 exc.reason_notice())

Regards,
Martin

From martin at v.loewis.de  Sun Jul 30 23:42:54 2006
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sun, 30 Jul 2006 23:42:54 +0200
Subject: [Python-Dev] Testing Socket Timeouts patch 1519025
In-Reply-To: <v04020a01c0f286dddce0@[192.168.123.162]>
References: <v04020a01c0f18c283570@[192.168.123.162]>	<v04020a01c0f129ae3d64@[192.168.123.162]>	<v04020a01c0f129ae3d64@[192.168.123.162]>	<v04020a01c0f18c283570@[192.168.123.162]>
	<v04020a01c0f286dddce0@[192.168.123.162]>
Message-ID: <44CD27DE.5060804@v.loewis.de>

Tony Nelson schrieb:
>> You can use GenerateConsoleCtrlEvent to send Ctrl-C to all processes
>> that share the console of the calling process.
[...]
> Martin, your advice is usually spot-on, but I don't always understand it.
> Maybe using it here is just complicated.  

This was really just in response to your remark that you couldn't
find a way to send Ctrl-C programmatically. I researched (in
the C library sources) how SIGINT was *generated* (through
SetConsoleCtrlHandler), and that let me to a way to generate

I didn't mean to suggest that you *should* use GenerateConsoleCtrlEvent,
only that you could if you wanted to.

> I expect that
> GenerateConsoleCtrlEvent() can be called through the ctypes module, though
> that would make backporting the test to 2.4 a bit more difficult.

Well, if there was general utility to that API, I would prefer exposing
it in the nt module. It doesn't quite fit into kill(2), as it doesn't
allow to specify a pid of the target process, so perhaps it doesn't
have general utility. In any case, that would have to wait for 2.6.

Regards,
Martin


From tonynelson at georgeanelson.com  Mon Jul 31 01:23:23 2006
From: tonynelson at georgeanelson.com (Tony Nelson)
Date: Sun, 30 Jul 2006 19:23:23 -0400
Subject: [Python-Dev] Testing Socket Timeouts patch 1519025
In-Reply-To: <44CD27DE.5060804@v.loewis.de>
References: <v04020a01c0f286dddce0@[192.168.123.162]>
	<v04020a01c0f18c283570@[192.168.123.162]>
	<v04020a01c0f129ae3d64@[192.168.123.162]>
	<v04020a01c0f129ae3d64@[192.168.123.162]>
	<v04020a01c0f18c283570@[192.168.123.162]>
	<v04020a01c0f286dddce0@[192.168.123.162]>
Message-ID: <v04020a04c0f2da888957@[192.168.123.162]>

At 11:42 PM +0200 7/30/06, Martin v. L?wis wrote:
>Tony Nelson schrieb:
>>> You can use GenerateConsoleCtrlEvent to send Ctrl-C to all processes
>>> that share the console of the calling process.
>[...]
>> Martin, your advice is usually spot-on, but I don't always understand it.
>> Maybe using it here is just complicated.
>
>This was really just in response to your remark that you couldn't
>find a way to send Ctrl-C programmatically. I researched (in
>the C library sources) how SIGINT was *generated* (through
>SetConsoleCtrlHandler), and that let me to a way to generate [one.]

Well, fine work there!

>I didn't mean to suggest that you *should* use GenerateConsoleCtrlEvent,
>only that you could if you wanted to.

Hmm.  Well, it would make the test possible on MSWindows as well as on OS's
implementing alarm(2).  If I figure out how to build Python on MSWindows, I
might give it a try.  I tried to get MSVC 7.1 via the .Net SDK, but it
installed VS 8 instead, so I'm not quite sure how to proceed.


>> I expect that
>> GenerateConsoleCtrlEvent() can be called through the ctypes module, though
>> that would make backporting the test to 2.4 a bit more difficult.
>
>Well, if there was general utility to that API, I would prefer exposing
>it in the nt module. It doesn't quite fit into kill(2), as it doesn't
>allow to specify a pid of the target process, so perhaps it doesn't
>have general utility. In any case, that would have to wait for 2.6.

A Process Group ID is the PID of the first process put in it, so it's sort
of a PID.  It just means a collection of processes, probably more than one.
It seems to be mostly applicable to MSWindows, and isn't a suitable way to
implement a form of kill(2).

I hope that the Socket Timeouts patch 1519025 can make it into 2.5, or
2.5.1, as it is a bug fix.  As such, it would probably be better to punt
the test on MSWindows than to do a tricky fancy test that might have its
own issues.
____________________________________________________________________
TonyN.:'                       <mailto:tonynelson at georgeanelson.com>
      '                              <http://www.georgeanelson.com/>

From tonynelson at georgeanelson.com  Mon Jul 31 04:04:59 2006
From: tonynelson at georgeanelson.com (Tony Nelson)
Date: Sun, 30 Jul 2006 22:04:59 -0400
Subject: [Python-Dev] Testing Socket Timeouts patch 1519025
In-Reply-To: <v04020a04c0f2da888957@[192.168.123.162]>
References: <44CD27DE.5060804@v.loewis.de>
	<v04020a01c0f286dddce0@[192.168.123.162]>
	<v04020a01c0f18c283570@[192.168.123.162]>
	<v04020a01c0f129ae3d64@[192.168.123.162]>
	<v04020a01c0f129ae3d64@[192.168.123.162]>
	<v04020a01c0f18c283570@[192.168.123.162]>
	<v04020a01c0f286dddce0@[192.168.123.162]>
Message-ID: <v04020a06c0f315064900@[192.168.123.162]>

At 7:23 PM -0400 7/30/06, Tony Nelson wrote:
 ...
>...I tried to get MSVC 7.1 via the .Net SDK, but it
>installed VS 8 instead, so I'm not quite sure how to proceed.
 ...

David Murmann suggested off-list that I'd probably installed the 2.0 .Net
SDK, and that I should install the 1.1 .Net SDK, which is the correct one.
Now I can try to build Python on MSWindows.
____________________________________________________________________
TonyN.:'                       <mailto:tonynelson at georgeanelson.com>
      '                              <http://www.georgeanelson.com/>

From martin at v.loewis.de  Mon Jul 31 04:34:03 2006
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Mon, 31 Jul 2006 04:34:03 +0200
Subject: [Python-Dev] Testing Socket Timeouts patch 1519025
In-Reply-To: <v04020a04c0f2da888957@[192.168.123.162]>
References: <v04020a01c0f286dddce0@[192.168.123.162]>	<v04020a01c0f18c283570@[192.168.123.162]>	<v04020a01c0f129ae3d64@[192.168.123.162]>	<v04020a01c0f129ae3d64@[192.168.123.162]>	<v04020a01c0f18c283570@[192.168.123.162]>	<v04020a01c0f286dddce0@[192.168.123.162]>
	<v04020a04c0f2da888957@[192.168.123.162]>
Message-ID: <44CD6C1B.6040401@v.loewis.de>

Tony Nelson schrieb:
> Hmm.  Well, it would make the test possible on MSWindows as well as on OS's
> implementing alarm(2).  If I figure out how to build Python on MSWindows, I
> might give it a try.  I tried to get MSVC 7.1 via the .Net SDK, but it
> installed VS 8 instead, so I'm not quite sure how to proceed.

The .NET SDK (any version) is not suitable to build Python. You really
need VS 2003; if you don't have it anymore, you might be able to find
a copy of the free version of the VC Toolkit 2003 (VCToolkitSetup.exe)
somewhere.

Of course, just for testing, you can also install VS Express 2005, and
use the PCbuild8 projects directory; these changes should work the
same under both versions.

Regards,
Martin

From tonynelson at georgeanelson.com  Mon Jul 31 06:39:07 2006
From: tonynelson at georgeanelson.com (Tony Nelson)
Date: Mon, 31 Jul 2006 00:39:07 -0400
Subject: [Python-Dev] Testing Socket Timeouts patch 1519025
In-Reply-To: <44CD6C1B.6040401@v.loewis.de>
References: <v04020a04c0f2da888957@[192.168.123.162]>
	<v04020a01c0f286dddce0@[192.168.123.162]>
	<v04020a01c0f18c283570@[192.168.123.162]>
	<v04020a01c0f129ae3d64@[192.168.123.162]>
	<v04020a01c0f129ae3d64@[192.168.123.162]>
	<v04020a01c0f18c283570@[192.168.123.162]>
	<v04020a01c0f286dddce0@[192.168.123.162]>
	<v04020a04c0f2da888957@[192.168.123.162]>
Message-ID: <v04020a07c0f320d50fb8@[192.168.123.162]>

At 4:34 AM +0200 7/31/06, Martin v. L?wis wrote:
>Tony Nelson schrieb:
>>Hmm. Well, it would make the test possible on MSWindows as well as on
>>OS's implementing alarm(2).  If I figure out how to build Python on
>>MSWindows, I might give it a try.  I tried to get MSVC 7.1 via the .Net
>>SDK, but it installed VS 8 instead, so I'm not quite sure how to proceed.
>
>The .NET SDK (any version) is not suitable to build Python.

I do see the warning in the instructions about it not be an optimizing
compiler.  I've managed to build python.exe and the rt.bat tests mostly
work -- 2 tests fail, test_popen, and test_cmd_line because of popen()
failing.

Hmm, actually, this might be a real problem with the MSWindows version of
posix_popen() in Modules/posixmodule.c.  The path to my built python.exe is:

    "E:\Documents and Settings\Tony Nelson\My Documents\Python\pydev\trunk\PCBuild\python.exe"

(lots of spaces in it).  It seems properly quoted in the test and when I do
it by hand, but in a call to popen() it doesn't work:

    popen('"E:\Documents and Settings\Tony Nelson\My Documents\Python\pydev\trunk\PCBuild\python.exe" -c "import sys;sys.version_info"')

The returned file object repr resembles one that does work.  If I just use
"python.exe" from within the PCBuild directory:

    popen('python.exe -c "import sys;sys.version_info"')

I get the right version, and that's the only 2.5b2 python I've got, so the
built python must be working, but the path, even quoted, isn't accepted by
MSWindows XP SP2.  Should I report a bug?  It may well just be MSWindows
weirdness, and not something that posixmodule.c can do anything about. 
OTOH, it does work from the command line.  I'll bet I wouldn't have seen a
thing if I'd checked out to "E:\pydev" instead.

>You really need VS 2003; if you don't have it anymore, you might be able
>to find a copy of the free version of the VC Toolkit 2003
>(VCToolkitSetup.exe) somewhere.

I really never had VS 2003.  It doesn't appear to be on microsoft.com
anymore.  I'm reluctant to try to steal a copy.


>Of course, just for testing, you can also install VS Express 2005, and
>use the PCbuild8 projects directory; these changes should work the
>same under both versions.

I'll try that if I have any real trouble with the non-optimized python or
if you insist that it's necessary.
____________________________________________________________________
TonyN.:'                       <mailto:tonynelson at georgeanelson.com>
      '                              <http://www.georgeanelson.com/>

From greg at electricrain.com  Mon Jul 31 06:48:55 2006
From: greg at electricrain.com (Gregory P. Smith)
Date: Sun, 30 Jul 2006 21:48:55 -0700
Subject: [Python-Dev] httplib and bad response chunking
In-Reply-To: <20060726023213.GA3106@cthulhu.gerg.ca>
References: <20060726023213.GA3106@cthulhu.gerg.ca>
Message-ID: <20060731044855.GK22921@zot.electricrain.com>

On Tue, Jul 25, 2006 at 10:32:13PM -0400, Greg Ward wrote:
>
> what I discovered in the wild the other day was a response like this:
> 
>   0005\r\nabcd\n\r\n0004\r\nabc\n\r\n\r\n
> 
> i.e. the chunk-size for the terminating empty chunk was missing.
> This cause httplib.py to blow up with ValueError because it tried to
> call
> 
>   int(line, 16)
> 
> assuming that 'line' contained a hex number, when in fact it was the
> empty string.  Oops.
> 
> IMHO the minimal fix is to turn ValueError into HTTPException (or a
> subclass thereof); httplib should not raise ValueError just because some
> server sends a bad response.  (The server in question was Apache 2.0.52
> running PHP 4.3.9 sending a big hairy error page because the database
> was down.)

IMNSHO httplib should be fixed and this shouldn't be an error at all
as its in the wild and will only show up more and more in the future.
Plus file a bug with the apache or php project as appropriate for
having a non-RFC compliant response.  This is part of the good old
network programming addage of being lenient in what you accept.

-g


From tonynelson at georgeanelson.com  Mon Jul 31 06:58:23 2006
From: tonynelson at georgeanelson.com (Tony Nelson)
Date: Mon, 31 Jul 2006 00:58:23 -0400
Subject: [Python-Dev] Testing Socket Timeouts patch 1519025
In-Reply-To: <v04020a07c0f320d50fb8@[192.168.123.162]>
References: <44CD6C1B.6040401@v.loewis.de>
	<v04020a04c0f2da888957@[192.168.123.162]>
	<v04020a01c0f286dddce0@[192.168.123.162]>
	<v04020a01c0f18c283570@[192.168.123.162]>
	<v04020a01c0f129ae3d64@[192.168.123.162]>
	<v04020a01c0f129ae3d64@[192.168.123.162]>
	<v04020a01c0f18c283570@[192.168.123.162]>
	<v04020a01c0f286dddce0@[192.168.123.162]>
	<v04020a04c0f2da888957@[192.168.123.162]>
Message-ID: <v04020a08c0f33d81cce8@[192.168.123.162]>

At 12:39 AM -0400 7/31/06, Tony Nelson wrote:

>    popen('"E:\Documents and Settings\Tony Nelson\My
>Documents\Python\pydev\trunk\PCBuild\python.exe" -c "import
>sys;sys.version_info"')

Ehh, I must admit that I retyped that.  Obviously what I typed would not
work, but what I used was:

    python = '"' + sys.executable + '"'
    popen(python + ' -c "import sys;sys.version_info"'

So there wasn't a problem with backslashes.  I've also been using raw
strings.  And, as I said, the file objects looked OK, with backslashes
where they should be.  Sorry for the mistyping.
____________________________________________________________________
TonyN.:'                       <mailto:tonynelson at georgeanelson.com>
      '                              <http://www.georgeanelson.com/>

From tonynelson at georgeanelson.com  Mon Jul 31 07:14:36 2006
From: tonynelson at georgeanelson.com (Tony Nelson)
Date: Mon, 31 Jul 2006 01:14:36 -0400
Subject: [Python-Dev] Testing Socket Timeouts patch 1519025
In-Reply-To: <v04020a08c0f33d81cce8@[192.168.123.162]>
References: <v04020a07c0f320d50fb8@[192.168.123.162]>
	<44CD6C1B.6040401@v.loewis.de>	<v04020a04c0f2da888957@[192.168.123.162]>
	<v04020a01c0f286dddce0@[192.168.123.162]>
	<v04020a01c0f18c283570@[192.168.123.162]>
	<v04020a01c0f129ae3d64@[192.168.123.162]>
	<v04020a01c0f129ae3d64@[192.168.123.162]>
	<v04020a01c0f18c283570@[192.168.123.162]>
	<v04020a01c0f286dddce0@[192.168.123.162]>
	<v04020a04c0f2da888957@[192.168.123.162]>
Message-ID: <v04020a09c0f34051763e@[192.168.123.162]>

At 12:58 AM -0400 7/31/06, Tony Nelson wrote:
>At 12:39 AM -0400 7/31/06, Tony Nelson wrote:
>
>>    popen('"E:\Documents and Settings\Tony Nelson\My
>>Documents\Python\pydev\trunk\PCBuild\python.exe" -c "import
>>sys;sys.version_info"')
>
>Ehh, I must admit that I retyped that.  Obviously what I typed would not
>work, but what I used was:
>
>    python = '"' + sys.executable + '"'
>    popen(python + ' -c "import sys;sys.version_info"'
>
>So there wasn't a problem with backslashes.  I've also been using raw
>strings.  And, as I said, the file objects looked OK, with backslashes
>where they should be.  Sorry for the mistyping.

OK, I recognize the bug now.  It's that quote parsing bug in MSWindows
(which I can find again if you want) which can be worked around by using an
extra quote at the front (and maybe also the back):

    popen('""E:\Documents ...

Not really a bug in Python at all.
____________________________________________________________________
TonyN.:'                       <mailto:tonynelson at georgeanelson.com>
      '                              <http://www.georgeanelson.com/>

From gward-1337f07a94b43060ff5c1ea922ed93d6 at python.net  Mon Jul 31 04:09:02 2006
From: gward-1337f07a94b43060ff5c1ea922ed93d6 at python.net (Greg Ward)
Date: Sun, 30 Jul 2006 22:09:02 -0400
Subject: [Python-Dev] httplib and bad response chunking
In-Reply-To: <Pine.LNX.4.64.0607282251320.8644@localhost>
References: <20060726023213.GA3106@cthulhu.gerg.ca>
	<Pine.LNX.4.64.0607282251320.8644@localhost>
Message-ID: <20060731020902.GA2950@cthulhu.gerg.ca>

[me, on 25 July]
> I have
> discovered other hypothetical cases of bad chunking that cause httplib
> to go into an infinite loop or block forever on socket.readline().
> Should we worry about those cases as well, despite not having seen them
> happen in the wild?  More annoying, I can reproduce the "block forever"
> case using a real socket, but not using the StringIO-based FakeSocket
> class in test_httplib.

[John J Lee]
> They have been seen in the wild :-)
> 
> http://python.org/sf/1411097

Thanks -- that was really all the encouragement I needed to keep banging
away at this bug.

Did you look at the crude attempt at testing for this bug that I hacked
into test_httplib.py?  I posted it to bug #1486335 here:

  http://sourceforge.net/tracker/download.php?group_id=5470&atid=105470&file_id=186245&aid=1486335

The idea is simple: put various chunked responses into strings and then
feed those strings to HTTPConnection.  The trouble is that StringIO does
not behave the same as a real socket: where HTTPResponse fails one way
reading from a real socket (eg. infinite loop), it fails differently (or
not at all) reading from a StringIO.  Makes testing with the FakeSocket
class in test_httplib.py problematic.

Maybe the right way to test httplib is to spawn a server process
(thread?) to listen on some random port, feed various HTTP responses at
HTTPConnection/HTTPResponse, and see what happens.  I'm not sure how to
do that portably, though.  Well, I'll see if I can whip up a Unix-y
solution and see if anyone knows how to make it portable.

        Greg
-- 
Greg Ward <gward at python.net>                         http://www.gerg.ca/
Be careful: sometimes, you're only standing on the shoulders of idiots.

From rhettinger at ewtllc.com  Mon Jul 31 20:34:26 2006
From: rhettinger at ewtllc.com (Raymond Hettinger)
Date: Mon, 31 Jul 2006 11:34:26 -0700
Subject: [Python-Dev] Rounding float to int directly (Re: struct module
 and coercing floats to integers)
In-Reply-To: <44CAD9A1.8080000@canterbury.ac.nz>
References: <9E51A030-B610-414B-98E0-1923F1C3E862@redivi.com>	<dcbbbb410607282014w30459351ifb09c56075997be1@mail.gmail.com>
	<44CAD9A1.8080000@canterbury.ac.nz>
Message-ID: <44CE4D32.6040909@ewtllc.com>

Greg Ewing wrote:

>As an aside, does anyone else think that it would be
>useful to have a builtin which rounds and converts to
>an int in one go? Whenever I use round(), I almost
>always want the result as an int, and making me do
>it in two steps seems unnecessarily bothersome.
>  
>
I think this would harm more than it would help.  It more confusing to 
have several rounding-thingies to choose from than it is have an 
explicit two-step.


BTW,  I thought the traditional idiom (for positive numbers) was:  int(x+.5)


>Since automatic float->int coercion is being increasingly
>disallowed, use cases for this are becoming more and more
>common.
>
>--
>Greg
>_______________________________________________
>Python-Dev mailing list
>Python-Dev at python.org
>http://mail.python.org/mailman/listinfo/python-dev
>Unsubscribe: http://mail.python.org/mailman/options/python-dev/rhettinger%40ewtllc.com
>  
>


From chad at zetaweb.com  Mon Jul 31 20:38:49 2006
From: chad at zetaweb.com (Chad Whitacre)
Date: Mon, 31 Jul 2006 14:38:49 -0400
Subject: [Python-Dev] Patch submitted, now what?
Message-ID: <44CE4E39.5010006@zetaweb.com>

Dear All,

Last week I submitted a patch (my first), and now I'm wondering what my 
expectations should be. Do I sit around and wait? How long? Do I notify 
this list? Do I notify a specific person, say, an author or reviewer of 
the original code I modified? Do I use SF's assignment mechanism? Who do 
I assign it to?

These are the questions I have, unanswered (afaict) by the patch 
documentation I've found:

   http://www.python.org/dev/patches/
   http://www.python.org/dev/patches/style/
   http://www.python.org/patches/
   http://www.python.org/patches/style.html
   http://www.python.org/dev/tools/#patch-tracking


Any insight?




chad


P.S. For the interested, here is my patch:

   "Expose case-insensitivity of string.Template"
   http://www.python.org/sf/1528167

From brett at python.org  Mon Jul 31 20:53:14 2006
From: brett at python.org (Brett Cannon)
Date: Mon, 31 Jul 2006 11:53:14 -0700
Subject: [Python-Dev] Patch submitted, now what?
In-Reply-To: <44CE4E39.5010006@zetaweb.com>
References: <44CE4E39.5010006@zetaweb.com>
Message-ID: <bbaeab100607311153q52753e27yf371e2ad19d61405@mail.gmail.com>

On 7/31/06, Chad Whitacre <chad at zetaweb.com> wrote:
>
> Dear All,
>
> Last week I submitted a patch (my first),


Thanks!

and now I'm wondering what my
> expectations should be. Do I sit around and wait? How long? Do I notify
> this list? Do I notify a specific person, say, an author or reviewer of
> the original code I modified? Do I use SF's assignment mechanism? Who do
> I assign it to?


Let it sit for now.  We get email notifications when new patches come in.
Since we are all volunteers it can take a little while before we get to it.

And don't assign it to anyone.  Let us handle that.

These are the questions I have, unanswered (afaict) by the patch
> documentation I've found:
>
>    http://www.python.org/dev/patches/
>    http://www.python.org/dev/patches/style/
>    http://www.python.org/patches/
>    http://www.python.org/patches/style.html
>    http://www.python.org/dev/tools/#patch-tracking


Those docs needs to be overhauled.  I am planning to consolidate into a
single patch guidelines doc in a month or so.

Any insight?
>
>
>
>
> chad
>
>
> P.S. For the interested, here is my patch:
>
>    "Expose case-insensitivity of string.Template"
>    http://www.python.org/sf/1528167
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/brett%40python.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20060731/161ca6a3/attachment.htm 

From chad at zetaweb.com  Mon Jul 31 21:43:20 2006
From: chad at zetaweb.com (Chad Whitacre)
Date: Mon, 31 Jul 2006 15:43:20 -0400
Subject: [Python-Dev] Patch submitted, now what?
In-Reply-To: <bbaeab100607311153q52753e27yf371e2ad19d61405@mail.gmail.com>
References: <44CE4E39.5010006@zetaweb.com>
	<bbaeab100607311153q52753e27yf371e2ad19d61405@mail.gmail.com>
Message-ID: <44CE5D58.6060407@zetaweb.com>

Brett,

Thanks for the helpful reply.


> Let it sit for now.  We get email notifications when new patches come in.

Can I ask who "we" are? Is that the seven SF "Project Admins?" Is that 
the 68 SF "Developers?"

And is every patch eventually responded to? Or do some simply fall by 
the wayside?


> Those docs needs to be overhauled.  I am planning to consolidate into a
> single patch guidelines doc in a month or so.

Great! I'd be happy to review it when the time comes, if that'd be helpful.

Thanks again.




chad

From brett at python.org  Mon Jul 31 21:56:58 2006
From: brett at python.org (Brett Cannon)
Date: Mon, 31 Jul 2006 12:56:58 -0700
Subject: [Python-Dev] Patch submitted, now what?
In-Reply-To: <44CE5D58.6060407@zetaweb.com>
References: <44CE4E39.5010006@zetaweb.com>
	<bbaeab100607311153q52753e27yf371e2ad19d61405@mail.gmail.com>
	<44CE5D58.6060407@zetaweb.com>
Message-ID: <bbaeab100607311256u281af267wd5b4d79b7db7b203@mail.gmail.com>

On 7/31/06, Chad Whitacre <chad at zetaweb.com> wrote:
>
> Brett,
>
> Thanks for the helpful reply.
>
>
> > Let it sit for now.  We get email notifications when new patches come
> in.
>
> Can I ask who "we" are? Is that the seven SF "Project Admins?" Is that
> the 68 SF "Developers?"


"We" is most of the developers on python-dev.  There is a Python patches
mailing list that most developers subscribe to.

And is every patch eventually responded to? Or do some simply fall by
> the wayside?


The latter unfortunatley.  Since this all relies on people's volunteer
efforts the patch usually has to matter to someone to lead to them putting
the time and effort into getting it checked in.

> Those docs needs to be overhauled.  I am planning to consolidate into a
> > single patch guidelines doc in a month or so.
>
> Great! I'd be happy to review it when the time comes, if that'd be
> helpful.


Yep, it would be.

-Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20060731/67f04a64/attachment.html 

From skip at pobox.com  Mon Jul 31 22:06:47 2006
From: skip at pobox.com (skip at pobox.com)
Date: Mon, 31 Jul 2006 15:06:47 -0500
Subject: [Python-Dev] Patch submitted, now what?
In-Reply-To: <bbaeab100607311256u281af267wd5b4d79b7db7b203@mail.gmail.com>
References: <44CE4E39.5010006@zetaweb.com>
	<bbaeab100607311153q52753e27yf371e2ad19d61405@mail.gmail.com>
	<44CE5D58.6060407@zetaweb.com>
	<bbaeab100607311256u281af267wd5b4d79b7db7b203@mail.gmail.com>
Message-ID: <17614.25303.255526.960212@montanaro.dyndns.org>


    Chad> And is every patch eventually responded to? Or do some simply fall
    Chad> by the wayside?

    Brett> The latter unfortunatley.  Since this all relies on people's
    Brett> volunteer efforts the patch usually has to matter to someone to
    Brett> lead to them putting the time and effort into getting it checked
    Brett> in.

It's worth noting that a number of people will look at a patch after the
submitter has reviewed five other patches or bug reports (does it look okay
to you?  does Python build with it applied?  do all unit tests pass? if it
needs documentation does it have it? etc).  Do that, attach comments to each
of the five, then send a note here listing the five you reviewed and the
patch id of your patch and one of those patch angels will take a look at
your patch (if they haven't already).

Skip

From chad at zetaweb.com  Mon Jul 31 22:07:10 2006
From: chad at zetaweb.com (Chad Whitacre)
Date: Mon, 31 Jul 2006 16:07:10 -0400
Subject: [Python-Dev] Patch submitted, now what?
In-Reply-To: <bbaeab100607311256u281af267wd5b4d79b7db7b203@mail.gmail.com>
References: <44CE4E39.5010006@zetaweb.com>	
	<bbaeab100607311153q52753e27yf371e2ad19d61405@mail.gmail.com>	
	<44CE5D58.6060407@zetaweb.com>
	<bbaeab100607311256u281af267wd5b4d79b7db7b203@mail.gmail.com>
Message-ID: <44CE62EE.2010801@zetaweb.com>

Brett,

> "We" is most of the developers on python-dev.  There is a Python patches
> mailing list that most developers subscribe to.

Helpful, thanks.


> The latter unfortunatley.  Since this all relies on people's volunteer
> efforts the patch usually has to matter to someone to lead to them putting
> the time and effort into getting it checked in.

Hrm, ok, also good to know. So it sounds like I wait another week or so 
and then maybe approach some folks individually.



chad

From chad at zetaweb.com  Mon Jul 31 22:08:32 2006
From: chad at zetaweb.com (Chad Whitacre)
Date: Mon, 31 Jul 2006 16:08:32 -0400
Subject: [Python-Dev] Patch submitted, now what?
In-Reply-To: <17614.25303.255526.960212@montanaro.dyndns.org>
References: <44CE4E39.5010006@zetaweb.com>
	<bbaeab100607311153q52753e27yf371e2ad19d61405@mail.gmail.com>
	<44CE5D58.6060407@zetaweb.com>
	<bbaeab100607311256u281af267wd5b4d79b7db7b203@mail.gmail.com>
	<17614.25303.255526.960212@montanaro.dyndns.org>
Message-ID: <44CE6340.30505@zetaweb.com>

Skip,

> It's worth noting that a number of people will look at a patch after the
> submitter has reviewed five other patches or bug reports

Also helpful, thanks!



chad

From martin at v.loewis.de  Mon Jul 31 22:48:36 2006
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Mon, 31 Jul 2006 22:48:36 +0200
Subject: [Python-Dev] Patch submitted, now what?
In-Reply-To: <44CE4E39.5010006@zetaweb.com>
References: <44CE4E39.5010006@zetaweb.com>
Message-ID: <44CE6CA4.1080908@v.loewis.de>

Chad Whitacre schrieb:

[I notice that my message comes across pretty negative. In a single
 sentence: We are all volunteers with limited time, and we contribute
 to Python because its fun and because it helps us solve our own
 problems.]
> Last week I submitted a patch (my first), and now I'm wondering what my 
> expectations should be.

As a starting point, it would be better if you had provided the patch
number (1528167) in this message. That makes it easier to answer the
questions. [I didn't read until the end of the message yet]

> Do I sit around and wait? How long? 

It's a new feature, so it can't possibly get into Python 2.5. That means
that no action is likely taken before October. It might take years until
the patch gets considered. As others have mentioned, there are ways to
expedite processing.

> Do I notify this list?

If you want discussion of the patch, you can do that. Depending on
how many people care about that aspect of Python, you may or may
not get a response.

I personally don't see myself using $-interpolation in the next
few years, so I doubt I will react in any way (the Python library
would just be the same for me if that feature didn't exist, and your
patch isn't a bug fix).

Others are more interested in the feature (or else it wouldn't
have gotten into the standard library), so they might respond.

> Do I notify a specific person, say, an author or reviewer of 
> the original code I modified? 

You might do that, although some may consider this at the
edge of spamming.

> Do I use SF's assignment mechanism? 

Please don't. If your assignment is "incorrect", this is a guarantee
that the patch will sit there forever: the person assigned won't react
because he doesn't feel responsible if he isn't, and nobody else
will feel responsible because the patch is assigned.

As Skip explains, I have a 5-for-1-rule for people who really want
to push their patches: If you review 5 patches, I will review yours
(despite me normally ignoring patches to this part of the library).

Regards,
Martin

From chad at zetaweb.com  Mon Jul 31 23:07:23 2006
From: chad at zetaweb.com (Chad Whitacre)
Date: Mon, 31 Jul 2006 17:07:23 -0400
Subject: [Python-Dev] Patch submitted, now what?
In-Reply-To: <44CE6CA4.1080908@v.loewis.de>
References: <44CE4E39.5010006@zetaweb.com> <44CE6CA4.1080908@v.loewis.de>
Message-ID: <44CE710B.4010905@zetaweb.com>

Martin,

Thanks for the reply.

> I notice that my message comes across pretty negative. 

No worries. I'm a volunteer too. :)


> It's a new feature, so it can't possibly get into Python 2.5. That means
> that no action is likely taken before October. It might take years until
> the patch gets considered.

Helpful, thanks.


> As Skip explains, I have a 5-for-1-rule for people who really want
> to push their patches: If you review 5 patches, I will review yours

Yeah, this hit the bulls-eye for me. The 5-for-1 rule is a great way to 
specify how newcomers can influence patch workflow w/o becoming 
spammers. [Brett: +1 on mentioning this in the updated Patch Guidelines]

Watch for another post once I've found the time to volunteer to review 5 
other patches. :)


Thanks again, all! Consider my original questions answered.




chad

From oliphant.travis at ieee.org  Mon Jul 31 20:28:09 2006
From: oliphant.travis at ieee.org (Travis Oliphant)
Date: Mon, 31 Jul 2006 12:28:09 -0600
Subject: [Python-Dev] Bad interaction of __index__ and sequence repeat
In-Reply-To: <44CB92F1.9040304@iinet.net.au>
References: <20060728101133.GA339@code0.codespeak.net>
	<44CB6B7D.1040001@gmail.com> <44CB92F1.9040304@iinet.net.au>
Message-ID: <44CE4BB9.60903@ieee.org>

Nick Coghlan wrote:
> Nick Coghlan wrote:
>> Armin Rigo wrote:
>>> Hi,
>>>
>>> There is an oversight in the design of __index__() that only just
>>> surfaced :-(  It is responsible for the following behavior, on a 32-bit
>>> machine with >= 2GB of RAM:
>>>
>>>     >>> s = 'x' * (2**100)       # works!
>>>     >>> len(s)
>>>     2147483647
>>>
>>> This is because PySequence_Repeat(v, w) works by applying 
>>> w.__index__ in
>>> order to call v->sq_repeat.  However, __index__ is defined to clip the
>>> result to fit in a Py_ssize_t.  This means that the above problem 
>>> exists
>>> with all sequences, not just strings, given enough RAM to create such
>>> sequences with 2147483647 items.
>>>
>>> For reference, in 2.4 we correctly get an OverflowError.
>>>
>>> Argh!  What should be done about it?
>>
>> I've now got a patch on SF that aims to fix this properly [1].
>
> I revised this patch to further reduce the code duplication associated 
> with the indexing code in the standard library.
>
> The patch now has three new functions in the abstract C API:
>
>   PyNumber_Index (used in a dozen or so places)
>     - raises IndexError on overflow
>   PyNumber_AsSsize_t (used in 3 places)
>     - raises OverflowError on overflow
>   PyNumber_AsClippedSsize_t() (used once, by _PyEval_SliceIndex)
>     - clips to PY_SSIZE_T_MIN/MAX on overflow
>
> All 3 have an int * output argument allowing type errors to be flagged 
> directly to the caller rather than through PyErr_Occurred().
>
> Of the 3, only PyNumber_Index is exposed through the operator module.
>
> Probably the most interesting thing now would be for Travis to review 
> it, and see whether it makes things easier to handle for the Numeric 
> scalar types (given the amount of code the patch deleted from the 
> builtin and standard library data types, hopefully the benefits to 
> Numeric will be comparable).


I noticed most of the checks for PyInt where removed in the patch.  If I 
remember correctly, I left these in for "optimization."   Other than 
that, I think the patch is great.

As far as helping with NumPy,  I think it will help to be able to remove 
special-checks for all the different integer-types.  But, this has not 
yet been done in the NumPy code.

-Travis