From ziade.tarek at gmail.com  Sat Jan 10 11:48:37 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Sat, 10 Jan 2009 11:48:37 +0100
Subject: [Python-ideas] Make getpass.getpass use system password keyrings ?
Message-ID: <94bdd2610901100248i4cda2ce4gd894869d8463d3fb@mail.gmail.com>

Hello,

What about having an option in getpass to store and reuse passwords in
system keyrings ?

    getpass(prompt[, stream])

would become:

    getpass(prompt[, stream, keyring])

where keyring would be a callable that can be use to retrieve the
password from a keyring system
and store it the first time.

The getpass module could provide some keyring support for:

- ssh-agent under Linux
- keychain under Mac OS X
- ...

And let the developers use their own keyring system by providing a callable.

Regards
Tarek

-- 
Tarek Ziad? | Association AfPy | www.afpy.org
Blog FR | http://programmation-python.org
Blog EN | http://tarekziade.wordpress.com/


From gerald.britton at gmail.com  Sat Jan 10 16:35:49 2009
From: gerald.britton at gmail.com (Gerald Britton)
Date: Sat, 10 Jan 2009 10:35:49 -0500
Subject: [Python-ideas] Add "while" clauses to generator expressions
Message-ID: <5d1a32000901100735t2433528br85b4f502a51e3664@mail.gmail.com>

I've been using Python generators for a while now.  e.g.

a=(i for i in range(10))

a.next()
a.next()
...etc.

I also find the "if" clause handy:

a = (i for i in range(10) if i%2==0)

(I know that range(0,12,2) will do the same thing, but it's the idea I
like, especially for more complex predicates.)

I would like to know if anyone has thought of adding a "while" clause
as well, like this:

a = (i for i in range(100) while i <=50)

(Again, this could be done with range(0,51) but then the predicate can
be more complicated.)

Why would this be helpful?  Consider the "in ..." part of the
generator.  You could be referring to something that is ordered --
sorted names for example.  Then you might want to stop your iterator
when you reach "Morgan" so you would like to write:

name = (n for n in names while n <= "Morgan")
name.next()
...etc...

Of course,

name = (n for n in names if n <= "Morgan")

will work, but it will look at every item in "names."  Since "names"
is sorted, this is a waste of time.  Imagine you want to stop at
"Baker".  Your "if" clause will look at and discard most names in the
list, assuming a normal distribution of English names.

Now, you could do the same thing with a generator function:

def leMorgan(names):
  i = 0
  while names[i] <= "Morgan":
    yield names[i]
    i+=1

and use it like this:

name=leMorgan(names)
name.next()
...etc...

but I think that adding a while clause to the generator expression is
simpler and clearer (and it keeps it all in one place!)

I know that this is functionally equivalent to the takewhile function
in itertools and my motivation is the same.   I just think that this
could be done nicely within the context of the existing syntax. This
is also convenient when the "in ..." clause refers to another
(possibly infinite) generator.  For a simple example, suppose I want
to run through some natural numbers.  I can write an infinte generator
function like this:

def genN(n=0):
  while 1:
    yield n
    n+=1

Then, I might use this in another generator:

p = (n for n in genN() if prime(n) while n <= 100)

to get the prime numbers under 100 (assuming I have a predicate
"prime" that works as one would hope).  Of course you could do this
with range(101) instead of genN, but this is just an example to
demonstrate the idea.

Without the "while" clause, this will not work:

p = (n for n in genN() if prime(n) if n <= 100)

This will actually NEVER terminate, since EVERY item in genN() (which
is an infinite generator) will be tested for <= 100.

So...What do others think?  Is this a loony idea?  Is there a better
way?  Also, can anyone think of a similarly syntax to replicate the
dropwhile function?


From grosser.meister.morti at gmx.net  Sat Jan 10 18:06:08 2009
From: grosser.meister.morti at gmx.net (=?ISO-8859-1?Q?Mathias_Panzenb=F6ck?=)
Date: Sat, 10 Jan 2009 18:06:08 +0100
Subject: [Python-ideas] Add "while" clauses to generator expressions
In-Reply-To: <5d1a32000901100735t2433528br85b4f502a51e3664@mail.gmail.com>
References: <5d1a32000901100735t2433528br85b4f502a51e3664@mail.gmail.com>
Message-ID: <4968D580.4030904@gmx.net>

Gerald Britton schrieb:
> I've been using Python generators for a while now.  e.g.
> 
> a=(i for i in range(10))
> 
> a.next()
> a.next()
> ...etc.
> 
> I also find the "if" clause handy:
> 
> a = (i for i in range(10) if i%2==0)
> 
> (I know that range(0,12,2) will do the same thing, but it's the idea I
> like, especially for more complex predicates.)
> 
> I would like to know if anyone has thought of adding a "while" clause
> as well, like this:
> 
> a = (i for i in range(100) while i <=50)
> 

from itertools takewhile
a = takewhile(lambda i: i <= 50, xrange(100))


From grosser.meister.morti at gmx.net  Sat Jan 10 18:07:43 2009
From: grosser.meister.morti at gmx.net (=?ISO-8859-1?Q?Mathias_Panzenb=F6ck?=)
Date: Sat, 10 Jan 2009 18:07:43 +0100
Subject: [Python-ideas] Add "while" clauses to generator expressions
In-Reply-To: <4968D580.4030904@gmx.net>
References: <5d1a32000901100735t2433528br85b4f502a51e3664@mail.gmail.com>
	<4968D580.4030904@gmx.net>
Message-ID: <4968D5DF.7070202@gmx.net>

Mathias Panzenb?ck schrieb:
> Gerald Britton schrieb:
>> I've been using Python generators for a while now.  e.g.
>>
>> a=(i for i in range(10))
>>
>> a.next()
>> a.next()
>> ...etc.
>>
>> I also find the "if" clause handy:
>>
>> a = (i for i in range(10) if i%2==0)
>>
>> (I know that range(0,12,2) will do the same thing, but it's the idea I
>> like, especially for more complex predicates.)
>>
>> I would like to know if anyone has thought of adding a "while" clause
>> as well, like this:
>>
>> a = (i for i in range(100) while i <=50)
>>
> 
> from itertools takewhile
> a = takewhile(lambda i: i <= 50, xrange(100))

typo:
from itertools import takewhile
a = takewhile(lambda i: i <= 50, xrange(100))


From gerald.britton at gmail.com  Sat Jan 10 20:06:02 2009
From: gerald.britton at gmail.com (Gerald Britton)
Date: Sat, 10 Jan 2009 14:06:02 -0500
Subject: [Python-ideas] Add "while" clauses to generator expressions
In-Reply-To: <4968D5DF.7070202@gmx.net>
References: <5d1a32000901100735t2433528br85b4f502a51e3664@mail.gmail.com>
	<4968D580.4030904@gmx.net> <4968D5DF.7070202@gmx.net>
Message-ID: <5d1a32000901101106x31cd15e7v8d1a0a352baec52e@mail.gmail.com>

Yes of course. However I am advocating adding this to the syntax
rather than using the takewhile function, thus making it part of the
generator expression proper and avoiding the function call and module
include. From the little I know about the implementation of the
corresponding "if" clause, this should be relatively easy.



On 1/10/09, Mathias Panzenb?ck <grosser.meister.morti at gmx.net> wrote:
> Mathias Panzenb?ck schrieb:
>> Gerald Britton schrieb:
>>> I've been using Python generators for a while now.  e.g.
>>>
>>> a=(i for i in range(10))
>>>
>>> a.next()
>>> a.next()
>>> ...etc.
>>>
>>> I also find the "if" clause handy:
>>>
>>> a = (i for i in range(10) if i%2==0)
>>>
>>> (I know that range(0,12,2) will do the same thing, but it's the idea I
>>> like, especially for more complex predicates.)
>>>
>>> I would like to know if anyone has thought of adding a "while" clause
>>> as well, like this:
>>>
>>> a = (i for i in range(100) while i <=50)
>>>
>>
>> from itertools takewhile
>> a = takewhile(lambda i: i <= 50, xrange(100))
>
> typo:
> from itertools import takewhile
> a = takewhile(lambda i: i <= 50, xrange(100))
>

-- 
Sent from my mobile device


From steven.bethard at gmail.com  Sat Jan 10 20:33:25 2009
From: steven.bethard at gmail.com (Steven Bethard)
Date: Sat, 10 Jan 2009 11:33:25 -0800
Subject: [Python-ideas] Add "while" clauses to generator expressions
In-Reply-To: <5d1a32000901100735t2433528br85b4f502a51e3664@mail.gmail.com>
References: <5d1a32000901100735t2433528br85b4f502a51e3664@mail.gmail.com>
Message-ID: <d11dcfba0901101133q58f5807ao373db4f072dea00f@mail.gmail.com>

On Sat, Jan 10, 2009 at 7:35 AM, Gerald Britton
<gerald.britton at gmail.com> wrote:
> I would like to know if anyone has thought of adding a "while" clause
> as well, like this:
>
> a = (i for i in range(100) while i <=50)

I think this could end up being confusing. Current generator
expressions turn into an equivalent generator function by simply
indenting the clauses and adding a yield, for example:

    (i for i in range(100) if i % 2 == 0)

is equivalent to:

    def gen():
        for i in range(100):
            if i % 2 == 0:
                yield i

Now you're proposing syntax that would no longer work like this.
Taking your example:

    (i for i in range(100) while i <= 50)

I would expect this to mean:

    def gen():
        for i in range(100):
            while i <= 50:
                yield i

In short, -1. You're proposing to use an existing keyword in a new way
that doesn't match how generator expressions are evaluated.

Steve
-- 
I'm not *in*-sane. Indeed, I am so far *out* of sane that you appear a
tiny blip on the distant coast of sanity.
        --- Bucky Katt, Get Fuzzy


From gerald.britton at gmail.com  Sat Jan 10 23:34:24 2009
From: gerald.britton at gmail.com (Gerald Britton)
Date: Sat, 10 Jan 2009 17:34:24 -0500
Subject: [Python-ideas] Add "while" clauses to generator expressions
In-Reply-To: <d11dcfba0901101133q58f5807ao373db4f072dea00f@mail.gmail.com>
References: <5d1a32000901100735t2433528br85b4f502a51e3664@mail.gmail.com>
	<d11dcfba0901101133q58f5807ao373db4f072dea00f@mail.gmail.com>
Message-ID: <5d1a32000901101434k7e7bb6cdv1f625e628e2e5b26@mail.gmail.com>

I guess I don't think it would be confusing. On the contrary, I
believe that the expressions would read naturally and be a nice
simplification.  Of course it won't work just like "if" but that is
just the point!  I can (and do) accomplish the same thing with
"takewhile", but if the same thing can be sone with a little addition
to the generator expression, why not do it?

On 1/10/09, Steven Bethard <steven.bethard at gmail.com> wrote:
> On Sat, Jan 10, 2009 at 7:35 AM, Gerald Britton
> <gerald.britton at gmail.com> wrote:
>> I would like to know if anyone has thought of adding a "while" clause
>> as well, like this:
>>
>> a = (i for i in range(100) while i <=50)
>
> I think this could end up being confusing. Current generator
> expressions turn into an equivalent generator function by simply
> indenting the clauses and adding a yield, for example:
>
>     (i for i in range(100) if i % 2 == 0)
>
> is equivalent to:
>
>     def gen():
>         for i in range(100):
>             if i % 2 == 0:
>                 yield i
>
> Now you're proposing syntax that would no longer work like this.
> Taking your example:
>
>     (i for i in range(100) while i <= 50)
>
> I would expect this to mean:
>
>     def gen():
>         for i in range(100):
>             while i <= 50:
>                 yield i
>
> In short, -1. You're proposing to use an existing keyword in a new way
> that doesn't match how generator expressions are evaluated.
>
> Steve
> --
> I'm not *in*-sane. Indeed, I am so far *out* of sane that you appear a
> tiny blip on the distant coast of sanity.
>         --- Bucky Katt, Get Fuzzy
>

-- 
Sent from my mobile device


From steven.bethard at gmail.com  Sat Jan 10 23:45:59 2009
From: steven.bethard at gmail.com (Steven Bethard)
Date: Sat, 10 Jan 2009 14:45:59 -0800
Subject: [Python-ideas] Add "while" clauses to generator expressions
In-Reply-To: <5d1a32000901101434k7e7bb6cdv1f625e628e2e5b26@mail.gmail.com>
References: <5d1a32000901100735t2433528br85b4f502a51e3664@mail.gmail.com>
	<d11dcfba0901101133q58f5807ao373db4f072dea00f@mail.gmail.com>
	<5d1a32000901101434k7e7bb6cdv1f625e628e2e5b26@mail.gmail.com>
Message-ID: <d11dcfba0901101445y12378241w6c1e6b844d5aa669@mail.gmail.com>

[Fixing the top-posting. Note that only Guido is allowed to top-post
around here. ;-)]

On 1/10/09, Steven Bethard <steven.bethard at gmail.com> wrote:
> I think this could end up being confusing. Current generator
> expressions turn into an equivalent generator function by simply
> indenting the clauses and adding a yield, for example:
>
>     (i for i in range(100) if i % 2 == 0)
>
> is equivalent to:
>
>     def gen():
>         for i in range(100):
>             if i % 2 == 0:
>                 yield i
>
> Now you're proposing syntax that would no longer work like this.
> Taking your example:
>
>     (i for i in range(100) while i <= 50)
>
> I would expect this to mean:
>
>     def gen():
>         for i in range(100):
>             while i <= 50:
>                 yield i

On Sat, Jan 10, 2009 at 2:34 PM, Gerald Britton
<gerald.britton at gmail.com> wrote:
> I guess I don't think it would be confusing. On the contrary, I
> believe that the expressions would read naturally and be a nice
> simplification.  Of course it won't work just like "if" but that is
> just the point!  I can (and do) accomplish the same thing with
> "takewhile", but if the same thing can be sone with a little addition
> to the generator expression, why not do it?

I'm probably just repeating myself here, but the reason not to do it
is that the current generator expressions translate almost directly
into the corresponding generator statements. Using "while" in the way
you've suggested breaks this symmetry, and would make Python harder to
learn.

Steve
-- 
I'm not *in*-sane. Indeed, I am so far *out* of sane that you appear a
tiny blip on the distant coast of sanity.
        --- Bucky Katt, Get Fuzzy


From gerald.britton at gmail.com  Sun Jan 11 00:20:40 2009
From: gerald.britton at gmail.com (Gerald Britton)
Date: Sat, 10 Jan 2009 18:20:40 -0500
Subject: [Python-ideas] Add "while" clauses to generator expressions
In-Reply-To: <d11dcfba0901101445y12378241w6c1e6b844d5aa669@mail.gmail.com>
References: <5d1a32000901100735t2433528br85b4f502a51e3664@mail.gmail.com>
	<d11dcfba0901101133q58f5807ao373db4f072dea00f@mail.gmail.com>
	<5d1a32000901101434k7e7bb6cdv1f625e628e2e5b26@mail.gmail.com>
	<d11dcfba0901101445y12378241w6c1e6b844d5aa669@mail.gmail.com>
Message-ID: <5d1a32000901101520j264c3979v62543ed2707cda2d@mail.gmail.com>

Hmmm.  I can't really see what you are saying.  Your example doesn't
quite get the intention.  The "while" clause in the example would
translate like this:

     def gen():
         for i in range(100):
             if i <= 50:
                 yield i
             else:
                break

Also, I think that for a new python user,

(n for n in range(100) while n*n < 50)

is easier to understand and use than:

(n for n in takewhile(lambda n:n*n < 50, range(100)))

My proposed version is shorter (less chance for typos), has one less
set of parentheses (always a good thing) and reads naturally.  Also,
it is directly analogous to:

(n for n in range(100) if n*n < 50)

except that the "while" version stops when n reaches 8.  The "if"
version doesn't stop until n reaches 99.


On Sat, Jan 10, 2009 at 5:45 PM, Steven Bethard
<steven.bethard at gmail.com> wrote:
> [Fixing the top-posting. Note that only Guido is allowed to top-post
> around here. ;-)]
>
> On 1/10/09, Steven Bethard <steven.bethard at gmail.com> wrote:
>> I think this could end up being confusing. Current generator
>> expressions turn into an equivalent generator function by simply
>> indenting the clauses and adding a yield, for example:
>>
>>     (i for i in range(100) if i % 2 == 0)
>>
>> is equivalent to:
>>
>>     def gen():
>>         for i in range(100):
>>             if i % 2 == 0:
>>                 yield i
>>
>> Now you're proposing syntax that would no longer work like this.
>> Taking your example:
>>
>>     (i for i in range(100) while i <= 50)
>>
>> I would expect this to mean:
>>
>>     def gen():
>>         for i in range(100):
>>             while i <= 50:
>>                 yield i
>
> On Sat, Jan 10, 2009 at 2:34 PM, Gerald Britton
> <gerald.britton at gmail.com> wrote:
>> I guess I don't think it would be confusing. On the contrary, I
>> believe that the expressions would read naturally and be a nice
>> simplification.  Of course it won't work just like "if" but that is
>> just the point!  I can (and do) accomplish the same thing with
>> "takewhile", but if the same thing can be sone with a little addition
>> to the generator expression, why not do it?
>
> I'm probably just repeating myself here, but the reason not to do it
> is that the current generator expressions translate almost directly
> into the corresponding generator statements. Using "while" in the way
> you've suggested breaks this symmetry, and would make Python harder to
> learn.
>
> Steve
> --
> I'm not *in*-sane. Indeed, I am so far *out* of sane that you appear a
> tiny blip on the distant coast of sanity.
>        --- Bucky Katt, Get Fuzzy
>


From grosser.meister.morti at gmx.net  Sun Jan 11 00:44:02 2009
From: grosser.meister.morti at gmx.net (=?ISO-8859-1?Q?Mathias_Panzenb=F6ck?=)
Date: Sun, 11 Jan 2009 00:44:02 +0100
Subject: [Python-ideas] Add "while" clauses to generator expressions
In-Reply-To: <5d1a32000901101520j264c3979v62543ed2707cda2d@mail.gmail.com>
References: <5d1a32000901100735t2433528br85b4f502a51e3664@mail.gmail.com>	<d11dcfba0901101133q58f5807ao373db4f072dea00f@mail.gmail.com>	<5d1a32000901101434k7e7bb6cdv1f625e628e2e5b26@mail.gmail.com>	<d11dcfba0901101445y12378241w6c1e6b844d5aa669@mail.gmail.com>
	<5d1a32000901101520j264c3979v62543ed2707cda2d@mail.gmail.com>
Message-ID: <496932C2.2070507@gmx.net>

I've got an idea:
x = (n for n in xrange(100) if n < 22 else break)

but I don't mean that seriously (giving it -1). :P

	-panzi


From g.brandl at gmx.net  Sun Jan 11 10:14:41 2009
From: g.brandl at gmx.net (Georg Brandl)
Date: Sun, 11 Jan 2009 10:14:41 +0100
Subject: [Python-ideas] Add "while" clauses to generator expressions
In-Reply-To: <5d1a32000901101520j264c3979v62543ed2707cda2d@mail.gmail.com>
References: <5d1a32000901100735t2433528br85b4f502a51e3664@mail.gmail.com>	<d11dcfba0901101133q58f5807ao373db4f072dea00f@mail.gmail.com>	<5d1a32000901101434k7e7bb6cdv1f625e628e2e5b26@mail.gmail.com>	<d11dcfba0901101445y12378241w6c1e6b844d5aa669@mail.gmail.com>
	<5d1a32000901101520j264c3979v62543ed2707cda2d@mail.gmail.com>
Message-ID: <gkcdb5$fl5$1@ger.gmane.org>

Gerald Britton schrieb:
> Hmmm.  I can't really see what you are saying.  Your example doesn't
> quite get the intention.  The "while" clause in the example would
> translate like this:
> 
>      def gen():
>          for i in range(100):
>              if i <= 50:
>                  yield i
>              else:
>                 break
> 
> Also, I think that for a new python user,
> 
> (n for n in range(100) while n*n < 50)
> 
> is easier to understand and use than:
> 
> (n for n in takewhile(lambda n:n*n < 50, range(100)))
>
> My proposed version is shorter (less chance for typos), has one less
> set of parentheses (always a good thing) and reads naturally.

The other is needlessly complicated, though.

takewhile(lambda n: n*n < 50, range(100))

is just as fine, and only 3 characters longer than the proposed while-exp.

Georg

-- 
Thus spake the Lord: Thou shalt indent with four spaces. No more, no less.
Four shall be the number of spaces thou shalt indent, and the number of thy
indenting shall be four. Eight shalt thou not indent, nor either indent thou
two, excepting that thou then proceed to four. Tabs are right out.



From bborcic at gmail.com  Sun Jan 11 13:05:18 2009
From: bborcic at gmail.com (Boris Borcic)
Date: Sun, 11 Jan 2009 13:05:18 +0100
Subject: [Python-ideas] Add "while" clauses to generator expressions
In-Reply-To: <5d1a32000901100735t2433528br85b4f502a51e3664@mail.gmail.com>
References: <5d1a32000901100735t2433528br85b4f502a51e3664@mail.gmail.com>
Message-ID: <gkcnbr$8j4$1@ger.gmane.org>

Gerald Britton wrote:
> as well, like this:
> 
> a = (i for i in range(100) while i <=50)

there was a longish thread on the subject back in august '08, where I pointed out

> you can do something like
> 
>>>> def yet(b) :
>     if b : raise StopIteration
> 
>     
>>>> list(x for x in range(0,10) if not yet(x>4))
> [0, 1, 2, 3, 4]

but I was then answered that this "worked by accident" which apparently meant 
that 3.0 pedagogical doctrine wants [x for x...] to be strictly equivalent to 
list(x for x...), while this equivalence in fact doesn't extend to similar uses 
of StopIteration.

Cheers, BB



From greg at krypto.org  Mon Jan 12 08:59:08 2009
From: greg at krypto.org (Gregory P. Smith)
Date: Sun, 11 Jan 2009 23:59:08 -0800
Subject: [Python-ideas] Make getpass.getpass use system password
	keyrings ?
In-Reply-To: <94bdd2610901100248i4cda2ce4gd894869d8463d3fb@mail.gmail.com>
References: <94bdd2610901100248i4cda2ce4gd894869d8463d3fb@mail.gmail.com>
Message-ID: <52dc1c820901112359w29e688bdlb72616af2d83563b@mail.gmail.com>

On Sat, Jan 10, 2009 at 2:48 AM, Tarek Ziad? <ziade.tarek at gmail.com> wrote:

> Hello,
>
> What about having an option in getpass to store and reuse passwords in
> system keyrings ?
>
>    getpass(prompt[, stream])
>
> would become:
>
>    getpass(prompt[, stream, keyring])
>
> where keyring would be a callable that can be use to retrieve the
> password from a keyring system
> and store it the first time.
>
> The getpass module could provide some keyring support for:
>
> - ssh-agent under Linux
> - keychain under Mac OS X
> - ...
>
> And let the developers use their own keyring system by providing a
> callable.
>
> Regards
> Tarek


That sounds like a nice idea.  I suggest starting by writing a library
keyring get/put functions for the various keyrings out there and releasing
it on its own.

(ssh-agent, gnome and kde keychains, mac os x keychain, does windows have an
api for this?, others?, pwsafe?)

-gps
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20090111/e0b1b0e9/attachment.html>

From ziade.tarek at gmail.com  Mon Jan 12 15:53:27 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Mon, 12 Jan 2009 15:53:27 +0100
Subject: [Python-ideas] Make getpass.getpass use system password
	keyrings ?
In-Reply-To: <52dc1c820901112359w29e688bdlb72616af2d83563b@mail.gmail.com>
References: <94bdd2610901100248i4cda2ce4gd894869d8463d3fb@mail.gmail.com>
	<52dc1c820901112359w29e688bdlb72616af2d83563b@mail.gmail.com>
Message-ID: <94bdd2610901120653g27013a52sfbb7daa7dba04ee5@mail.gmail.com>

On Mon, Jan 12, 2009 at 8:59 AM, Gregory P. Smith <greg at krypto.org> wrote:
>
> On Sat, Jan 10, 2009 at 2:48 AM, Tarek Ziad? <ziade.tarek at gmail.com> wrote:
>>
>> Hello,
>>
>> What about having an option in getpass to store and reuse passwords in
>> system keyrings ?
>>
>>    getpass(prompt[, stream])
>>
>> would become:
>>
>>    getpass(prompt[, stream, keyring])
>>
>> where keyring would be a callable that can be use to retrieve the
>> password from a keyring system
>> and store it the first time.
>>
>> The getpass module could provide some keyring support for:
>>
>> - ssh-agent under Linux
>> - keychain under Mac OS X
>> - ...
>>
>> And let the developers use their own keyring system by providing a
>> callable.
>>
>> Regards
>> Tarek
>
> That sounds like a nice idea.  I suggest starting by writing a library
> keyring get/put functions for the various keyrings out there and releasing
> it on its own.

Right, sounds like the best approach

(My use case is for Distutils 's register/upload commands)

> (ssh-agent, gnome and kde keychains, mac os x keychain, does windows have an
> api for this?, others?, pwsafe?)

I don't know about Windows, I need to digg,

I'll set a list of existing keyrings.

Cheers
Tarek


> -gps



-- 
Tarek Ziad? | Association AfPy | www.afpy.org
Blog FR | http://programmation-python.org
Blog EN | http://tarekziade.wordpress.com/


From mail at timgolden.me.uk  Mon Jan 12 16:06:31 2009
From: mail at timgolden.me.uk (Tim Golden)
Date: Mon, 12 Jan 2009 15:06:31 +0000
Subject: [Python-ideas] Make getpass.getpass use system
 password	keyrings ?
In-Reply-To: <94bdd2610901120653g27013a52sfbb7daa7dba04ee5@mail.gmail.com>
References: <94bdd2610901100248i4cda2ce4gd894869d8463d3fb@mail.gmail.com>	<52dc1c820901112359w29e688bdlb72616af2d83563b@mail.gmail.com>
	<94bdd2610901120653g27013a52sfbb7daa7dba04ee5@mail.gmail.com>
Message-ID: <496B5C77.1010402@timgolden.me.uk>

Tarek Ziad? wrote:
> I don't know about Windows, I need to digg,


Assuming I understand what the requirement is, you need
the authentication functions. Specifically, I think things
like CredRead, CredWrite and CredUICmdLinePromptForCredentials
would be useful. (getpass may be using these already; I haven't 
checked).

http://msdn.microsoft.com/en-us/library/aa374804(VS.85).aspx

http://msdn.microsoft.com/en-us/library/aa375187(VS.85).aspx

http://msdn.microsoft.com/en-us/library/aa375171(VS.85).aspx


TJG


From sturla at molden.no  Tue Jan 13 21:07:05 2009
From: sturla at molden.no (Sturla Molden)
Date: Tue, 13 Jan 2009 21:07:05 +0100
Subject: [Python-ideas] allow tuples in a with statement?
Message-ID: <496CF469.5010605@molden.no>


Sometimes multiple arguments in a with statement would be convinient.

with open('in.txt','rt'), \
      open('out.txt','wt'),\
      open('err.txt','wt') \
      as fstdin, fstdout, fstderr:

But today this has to be written as:

with open('in.txt','rt') as fstdin:
    with open('out.txt','wt') as fstdout:
       with open('err.txt','wt') as fstderr:

with the effect of causing multiple redundant levels of intendation.


Regards,
Sturla Molden





From mail at timgolden.me.uk  Tue Jan 13 21:43:32 2009
From: mail at timgolden.me.uk (Tim Golden)
Date: Tue, 13 Jan 2009 20:43:32 +0000
Subject: [Python-ideas] allow tuples in a with statement?
In-Reply-To: <496CF469.5010605@molden.no>
References: <496CF469.5010605@molden.no>
Message-ID: <496CFCF4.5040009@timgolden.me.uk>

Sturla Molden wrote:
> Sometimes multiple arguments in a with statement would be convinient.
> 
> with open('in.txt','rt'), \
>      open('out.txt','wt'),\
>      open('err.txt','wt') \
>      as fstdin, fstdout, fstderr:
> 
> But today this has to be written as:
> 
> with open('in.txt','rt') as fstdin:
>    with open('out.txt','wt') as fstdout:
>       with open('err.txt','wt') as fstderr:
> 
> with the effect of causing multiple redundant levels of intendation.

contextlib.nested ?

TJG


From fredrik.johansson at gmail.com  Tue Jan 13 21:57:12 2009
From: fredrik.johansson at gmail.com (Fredrik Johansson)
Date: Tue, 13 Jan 2009 21:57:12 +0100
Subject: [Python-ideas] allow tuples in a with statement?
In-Reply-To: <496CF469.5010605@molden.no>
References: <496CF469.5010605@molden.no>
Message-ID: <3d0cebfb0901131257g6a441cb5u9752c8acf3231817@mail.gmail.com>

On Tue, Jan 13, 2009 at 9:07 PM, Sturla Molden <sturla at molden.no> wrote:
>
> Sometimes multiple arguments in a with statement would be convinient.

This has occurred to me as well.

In sympy, we have considered using the with statement for assumptions
on variables. Tuples would provide a very natural syntax for multiple
assumptions:

with x>0, y>0:
    simplify(sqrt((x+y)**2))

Fredrik


From carl at carlsensei.com  Wed Jan 14 08:12:34 2009
From: carl at carlsensei.com (Carl Johnson)
Date: Tue, 13 Jan 2009 21:12:34 -1000
Subject: [Python-ideas] With-expression
Message-ID: <82936ECA-C3F2-40D2-90CE-3EBA009AF314@carlsensei.com>

 From the department of doomed ideas:

      data = f.read() with open("file.txt") as f

Has the idea of a with-expression already been preemptively shot down?

I guess the objection is that the above could easily be rewritten as

     data = read_and_close("file.txt")

if one defines the appropriate function. The counter-objection is that  
why do I need to write a micro-function when the with-statement  
already does what I want, except it takes up two lines when it would  
still be clear as one line. But then the counter-counter-objection is  
that you can write

     with open("file.txt") as f: data = f.read()

today. To which the counter-counter-counter-objection is, yeah, but  
that looks cluttered and ugly in a way that the first example  
doesn't... Except the first example is sort of cluttered.

I dunno, what do other people think about this? Doomed or super-doomed?

-- Carl


From pyideas at rebertia.com  Wed Jan 14 09:24:33 2009
From: pyideas at rebertia.com (Chris Rebert)
Date: Wed, 14 Jan 2009 00:24:33 -0800
Subject: [Python-ideas] With-expression
In-Reply-To: <82936ECA-C3F2-40D2-90CE-3EBA009AF314@carlsensei.com>
References: <82936ECA-C3F2-40D2-90CE-3EBA009AF314@carlsensei.com>
Message-ID: <50697b2c0901140024o21639c93k8d592f4dcbe2f35d@mail.gmail.com>

On Tue, Jan 13, 2009 at 11:12 PM, Carl Johnson <carl at carlsensei.com> wrote:
> From the department of doomed ideas:
>
>     data = f.read() with open("file.txt") as f
>
> Has the idea of a with-expression already been preemptively shot down?
>
> I guess the objection is that the above could easily be rewritten as
>
>    data = read_and_close("file.txt")
>
> if one defines the appropriate function. The counter-objection is that why
> do I need to write a micro-function when the with-statement already does
> what I want, except it takes up two lines when it would still be clear as
> one line. But then the counter-counter-objection is that you can write
>
>    with open("file.txt") as f: data = f.read()
>
> today. To which the counter-counter-counter-objection is, yeah, but that
> looks cluttered and ugly in a way that the first example doesn't... Except
> the first example is sort of cluttered.
>
> I dunno, what do other people think about this? Doomed or super-doomed?

Fairly doomed, IMHO. To be consistent at all, Python would also have
to allow the related Perlisms of:

x **= 2 while x < y
x += y for y in z
x.sort() if should_sort

which seem pretty gross to me as the control structure gets somewhat
hidden. And as you pointed out, it's just plain unnecessary since such
code can already be written as a one-liner, with the advantage of less
reformatting if it later becomes a multiliner. Also, having a
consistent order of block-statement before block-body enhances the
uniformity of the language. About the only good examples of block
bodies coming before block statements which I can think of offhand are
do-while and Haskell's `where` clause, which both seem to me to be
sufficiently special cases as to not have bearing here.

That's my 2? anyway.

Cheers,
Chris

--
Follow the path of the Iguana...
http://rebertia.com


From tjreedy at udel.edu  Wed Jan 14 12:20:08 2009
From: tjreedy at udel.edu (Terry Reedy)
Date: Wed, 14 Jan 2009 06:20:08 -0500
Subject: [Python-ideas] With-expression
In-Reply-To: <82936ECA-C3F2-40D2-90CE-3EBA009AF314@carlsensei.com>
References: <82936ECA-C3F2-40D2-90CE-3EBA009AF314@carlsensei.com>
Message-ID: <gkkhp8$kb3$1@ger.gmane.org>

Carl Johnson wrote:
>  From the department of doomed ideas:
> 
>      data = f.read() with open("file.txt") as f
> 
> Has the idea of a with-expression already been preemptively shot down?
> 
> I guess the objection is that the above could easily be rewritten as
> 
>     data = read_and_close("file.txt")
> 
> if one defines the appropriate function. The counter-objection is that 
> why do I need to write a micro-function when the with-statement already 
> does what I want, except it takes up two lines when it would still be 
> clear as one line. But then the counter-counter-objection is that you 
> can write
> 
>     with open("file.txt") as f: data = f.read()
> 
> today. To which the counter-counter-counter-objection is, yeah, but that 
> looks cluttered and ugly in a way that the first example doesn't... 
> Except the first example is sort of cluttered.
> 
> I dunno, what do other people think about this? Doomed or super-doomed?

Twisted and super-doomed.



From grosser.meister.morti at gmx.net  Wed Jan 14 12:33:02 2009
From: grosser.meister.morti at gmx.net (=?ISO-8859-1?Q?Mathias_Panzenb=F6ck?=)
Date: Wed, 14 Jan 2009 12:33:02 +0100
Subject: [Python-ideas] With-expression
In-Reply-To: <82936ECA-C3F2-40D2-90CE-3EBA009AF314@carlsensei.com>
References: <82936ECA-C3F2-40D2-90CE-3EBA009AF314@carlsensei.com>
Message-ID: <496DCD6E.30009@gmx.net>

strippedLines = [sline for line in lines with line.strip() as sline if sline]

or something like that. :P
(-1)

	-panzi

Carl Johnson schrieb:
> From the department of doomed ideas:
>
>      data = f.read() with open("file.txt") as f
>
> Has the idea of a with-expression already been preemptively shot down?
>
> I guess the objection is that the above could easily be rewritten as
>
>     data = read_and_close("file.txt")
>
> if one defines the appropriate function. The counter-objection is that
> why do I need to write a micro-function when the with-statement already
> does what I want, except it takes up two lines when it would still be
> clear as one line. But then the counter-counter-objection is that you
> can write
>
>     with open("file.txt") as f: data = f.read()
>
> today. To which the counter-counter-counter-objection is, yeah, but that
> looks cluttered and ugly in a way that the first example doesn't...
> Except the first example is sort of cluttered.
>
> I dunno, what do other people think about this? Doomed or super-doomed?
>
> -- Carl


From zac256 at gmail.com  Sun Jan 18 02:35:48 2009
From: zac256 at gmail.com (Zac Burns)
Date: Sat, 17 Jan 2009 17:35:48 -0800
Subject: [Python-ideas] defaultdict factory argument
Message-ID: <333edbe80901171735w7951a2eep6d236a419e8bf0f9@mail.gmail.com>

I think defaultdict would be more useful if one could initialize it
with the option to have the key that is being looked up passed to
default_factory.

--
Zachary Burns
(407)590-4814
Aim - Zac256FL
Production Engineer (Digital Overlord)
Zindagi Games


From pyideas at rebertia.com  Sun Jan 18 02:44:44 2009
From: pyideas at rebertia.com (Chris Rebert)
Date: Sat, 17 Jan 2009 17:44:44 -0800
Subject: [Python-ideas] defaultdict factory argument
In-Reply-To: <333edbe80901171735w7951a2eep6d236a419e8bf0f9@mail.gmail.com>
References: <333edbe80901171735w7951a2eep6d236a419e8bf0f9@mail.gmail.com>
Message-ID: <50697b2c0901171744p71fc8ddfw1778048ff7ee78eb@mail.gmail.com>

On Sat, Jan 17, 2009 at 5:35 PM, Zac Burns <zac256 at gmail.com> wrote:
> I think defaultdict would be more useful if one could initialize it
> with the option to have the key that is being looked up passed to
> default_factory.

While I don't disagree in principle, giving an example use-case would
probably help.

Cheers,
Chris

--
Follow the path of the Iguana...
http://rebertia.com


From zac256 at gmail.com  Sun Jan 18 03:01:44 2009
From: zac256 at gmail.com (Zac Burns)
Date: Sat, 17 Jan 2009 18:01:44 -0800
Subject: [Python-ideas] defaultdict factory argument
In-Reply-To: <50697b2c0901171744p71fc8ddfw1778048ff7ee78eb@mail.gmail.com>
References: <333edbe80901171735w7951a2eep6d236a419e8bf0f9@mail.gmail.com>
	<50697b2c0901171744p71fc8ddfw1778048ff7ee78eb@mail.gmail.com>
Message-ID: <333edbe80901171801n7de3f2c1s79e5cf378a24bb6b@mail.gmail.com>

For one, it could be used to cache function calls.

I'm not sure how well defaultdict and WeakValueDictionary work
together (when combined/subclassed), but I can also imagine using a
class as the default_factory and using a defaultdict to return a new
instance of a class if none yet existed instantiated with the same
input parameters or returning the already existing one if one did
exist with those parameters.

--
Zachary Burns
(407)590-4814
Aim - Zac256FL
Production Engineer (Digital Overlord)
Zindagi Games



On Sat, Jan 17, 2009 at 5:44 PM, Chris Rebert <pyideas at rebertia.com> wrote:
> On Sat, Jan 17, 2009 at 5:35 PM, Zac Burns <zac256 at gmail.com> wrote:
>> I think defaultdict would be more useful if one could initialize it
>> with the option to have the key that is being looked up passed to
>> default_factory.
>
> While I don't disagree in principle, giving an example use-case would
> probably help.
>
> Cheers,
> Chris
>
> --
> Follow the path of the Iguana...
> http://rebertia.com
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>


From steven.bethard at gmail.com  Sun Jan 18 03:32:28 2009
From: steven.bethard at gmail.com (Steven Bethard)
Date: Sat, 17 Jan 2009 18:32:28 -0800
Subject: [Python-ideas] defaultdict factory argument
In-Reply-To: <333edbe80901171735w7951a2eep6d236a419e8bf0f9@mail.gmail.com>
References: <333edbe80901171735w7951a2eep6d236a419e8bf0f9@mail.gmail.com>
Message-ID: <d11dcfba0901171832u127d0becnfd0f9230c67f9248@mail.gmail.com>

On Sat, Jan 17, 2009 at 5:35 PM, Zac Burns <zac256 at gmail.com> wrote:
> I think defaultdict would be more useful if one could initialize it
> with the option to have the key that is being looked up passed to
> default_factory.

Are you sure you can't get what you want by using __missing__?

http://www.python.org/doc/2.5.2/lib/defaultdict-objects.html

Steve
-- 
I'm not *in*-sane. Indeed, I am so far *out* of sane that you appear a
tiny blip on the distant coast of sanity.
        --- Bucky Katt, Get Fuzzy


From zac256 at gmail.com  Sun Jan 18 06:22:05 2009
From: zac256 at gmail.com (Zac Burns)
Date: Sat, 17 Jan 2009 21:22:05 -0800
Subject: [Python-ideas] defaultdict factory argument
In-Reply-To: <d11dcfba0901171832u127d0becnfd0f9230c67f9248@mail.gmail.com>
References: <333edbe80901171735w7951a2eep6d236a419e8bf0f9@mail.gmail.com>
	<d11dcfba0901171832u127d0becnfd0f9230c67f9248@mail.gmail.com>
Message-ID: <333edbe80901172122v5675a70cs7c1984331c9fd40f@mail.gmail.com>

Ah, excellent.
--
Zachary Burns
(407)590-4814
Aim - Zac256FL
Production Engineer (Digital Overlord)
Zindagi Games



On Sat, Jan 17, 2009 at 6:32 PM, Steven Bethard
<steven.bethard at gmail.com> wrote:
> On Sat, Jan 17, 2009 at 5:35 PM, Zac Burns <zac256 at gmail.com> wrote:
>> I think defaultdict would be more useful if one could initialize it
>> with the option to have the key that is being looked up passed to
>> default_factory.
>
> Are you sure you can't get what you want by using __missing__?
>
> http://www.python.org/doc/2.5.2/lib/defaultdict-objects.html
>
> Steve
> --
> I'm not *in*-sane. Indeed, I am so far *out* of sane that you appear a
> tiny blip on the distant coast of sanity.
>        --- Bucky Katt, Get Fuzzy
>


From tjreedy at udel.edu  Sun Jan 18 06:46:06 2009
From: tjreedy at udel.edu (Terry Reedy)
Date: Sun, 18 Jan 2009 00:46:06 -0500
Subject: [Python-ideas] defaultdict factory argument
In-Reply-To: <333edbe80901171801n7de3f2c1s79e5cf378a24bb6b@mail.gmail.com>
References: <333edbe80901171735w7951a2eep6d236a419e8bf0f9@mail.gmail.com>	<50697b2c0901171744p71fc8ddfw1778048ff7ee78eb@mail.gmail.com>
	<333edbe80901171801n7de3f2c1s79e5cf378a24bb6b@mail.gmail.com>
Message-ID: <gkufmr$5j9$1@ger.gmane.org>

Continuing the OP's top posting style..

if not hasattr(d,key):
   d[key] = <long computation>
return d[key]

or variations, including try:...except:.. work for that

Zac Burns wrote:
> For one, it could be used to cache function calls.

>> While I don't disagree in principle, giving an example use-case would
>> probably help.



From ziade.tarek at gmail.com  Sun Jan 18 13:40:40 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Sun, 18 Jan 2009 13:40:40 +0100
Subject: [Python-ideas] More things in the standard library equipped with
	__enter__ and __exit__
Message-ID: <94bdd2610901180440u54b452fnd7f4ba55eb9768ac@mail.gmail.com>

Hello,

I have entered this yesterday : http://bugs.python.org/issue4972.
about the fact that ftplib.FTP could be
used in a "with" loop.

But I spotted that imaplib.IMAP4/IMAP4_SSL and smtplib.SMTP could be
equipped as well,
so an __exit__ call takes care of closing or quiting.

This change could be done together with some examples in the documentation.

If this sounds like a good idea, I can start a patch in #4972,

Regards
Tarek

-- 
Tarek Ziad? | Association AfPy | www.afpy.org
Blog FR | http://programmation-python.org
Blog EN | http://tarekziade.wordpress.com/


From ironfroggy at gmail.com  Sun Jan 18 14:31:07 2009
From: ironfroggy at gmail.com (Calvin Spealman)
Date: Sun, 18 Jan 2009 08:31:07 -0500
Subject: [Python-ideas] More things in the standard library equipped
	with __enter__ and __exit__
In-Reply-To: <94bdd2610901180440u54b452fnd7f4ba55eb9768ac@mail.gmail.com>
References: <94bdd2610901180440u54b452fnd7f4ba55eb9768ac@mail.gmail.com>
Message-ID: <76fd5acf0901180531r7cee8373n4fe4bf2be2a2acba@mail.gmail.com>

+1 on both

On Sun, Jan 18, 2009 at 7:40 AM, Tarek Ziad? <ziade.tarek at gmail.com> wrote:
> Hello,
>
> I have entered this yesterday : http://bugs.python.org/issue4972.
> about the fact that ftplib.FTP could be
> used in a "with" loop.
>
> But I spotted that imaplib.IMAP4/IMAP4_SSL and smtplib.SMTP could be
> equipped as well,
> so an __exit__ call takes care of closing or quiting.
>
> This change could be done together with some examples in the documentation.
>
> If this sounds like a good idea, I can start a patch in #4972,
>
> Regards
> Tarek
>
> --
> Tarek Ziad? | Association AfPy | www.afpy.org
> Blog FR | http://programmation-python.org
> Blog EN | http://tarekziade.wordpress.com/
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>



-- 
Read my blog! I depend on your acceptance of my opinion! I am interesting!
http://techblog.ironfroggy.com/
Follow me if you're into that sort of thing: http://www.twitter.com/ironfroggy


From brett at python.org  Sun Jan 18 23:32:06 2009
From: brett at python.org (Brett Cannon)
Date: Sun, 18 Jan 2009 14:32:06 -0800
Subject: [Python-ideas] More things in the standard library equipped
	with __enter__ and __exit__
In-Reply-To: <94bdd2610901180440u54b452fnd7f4ba55eb9768ac@mail.gmail.com>
References: <94bdd2610901180440u54b452fnd7f4ba55eb9768ac@mail.gmail.com>
Message-ID: <bbaeab100901181432r2dc7494dy898fa1bebdab095f@mail.gmail.com>

On Sun, Jan 18, 2009 at 04:40, Tarek Ziad? <ziade.tarek at gmail.com> wrote:
> Hello,
>
> I have entered this yesterday : http://bugs.python.org/issue4972.
> about the fact that ftplib.FTP could be
> used in a "with" loop.
>
> But I spotted that imaplib.IMAP4/IMAP4_SSL and smtplib.SMTP could be
> equipped as well,
> so an __exit__ call takes care of closing or quiting.
>
> This change could be done together with some examples in the documentation.
>
> If this sounds like a good idea, I can start a patch in #4972,

They should be separate patches, but yes, more context managers are a
good thing (we are not calling them "'with' loops" since they don't
loop =).

-Brett


From leif.walsh at gmail.com  Sun Jan 18 23:41:15 2009
From: leif.walsh at gmail.com (Leif Walsh)
Date: Sun, 18 Jan 2009 17:41:15 -0500
Subject: [Python-ideas] With-expression
In-Reply-To: <82936ECA-C3F2-40D2-90CE-3EBA009AF314@carlsensei.com>
References: <82936ECA-C3F2-40D2-90CE-3EBA009AF314@carlsensei.com>
Message-ID: <cc7430500901181441xcd47df6v3844b42ac0b6d567@mail.gmail.com>

On Wed, Jan 14, 2009 at 2:12 AM, Carl Johnson <carl at carlsensei.com> wrote:
> From the department of doomed ideas:
>
>     data = f.read() with open("file.txt") as f

"Who do you think we are, Ruby?"

Just kidding, but -1.

-- 
Cheers,
Leif


From josiah.carlson at gmail.com  Tue Jan 20 02:46:27 2009
From: josiah.carlson at gmail.com (Josiah Carlson)
Date: Mon, 19 Jan 2009 17:46:27 -0800
Subject: [Python-ideas] With-expression
In-Reply-To: <cc7430500901181441xcd47df6v3844b42ac0b6d567@mail.gmail.com>
References: <82936ECA-C3F2-40D2-90CE-3EBA009AF314@carlsensei.com>
	<cc7430500901181441xcd47df6v3844b42ac0b6d567@mail.gmail.com>
Message-ID: <e6511dbf0901191746p5e4cd6c1gb8e1f52398a4b55c@mail.gmail.com>

On Sun, Jan 18, 2009 at 2:41 PM, Leif Walsh <leif.walsh at gmail.com> wrote:
> On Wed, Jan 14, 2009 at 2:12 AM, Carl Johnson <carl at carlsensei.com> wrote:
>> From the department of doomed ideas:
>>
>>     data = f.read() with open("file.txt") as f
>
> "Who do you think we are, Ruby?"
>
> Just kidding, but -1.

I always preferred:

data = open("file.txt").read()

Which works all the way back as far as I can remember.

And as for:
strippedLines = [sline for line in lines with line.strip() as sline if sline]

Try:
strippedLines = [sline for sline in (line.strip() for line in lines) if sline]

The request gets a big fat -1 from me.

 - Josiah


From greg at krypto.org  Tue Jan 20 02:47:56 2009
From: greg at krypto.org (Gregory P. Smith)
Date: Mon, 19 Jan 2009 17:47:56 -0800
Subject: [Python-ideas] With-expression
In-Reply-To: <e6511dbf0901191746p5e4cd6c1gb8e1f52398a4b55c@mail.gmail.com>
References: <82936ECA-C3F2-40D2-90CE-3EBA009AF314@carlsensei.com>
	<cc7430500901181441xcd47df6v3844b42ac0b6d567@mail.gmail.com>
	<e6511dbf0901191746p5e4cd6c1gb8e1f52398a4b55c@mail.gmail.com>
Message-ID: <52dc1c820901191747s3fee79a4u119d5623a3796205@mail.gmail.com>

On Mon, Jan 19, 2009 at 5:46 PM, Josiah Carlson <josiah.carlson at gmail.com>wrote:

> On Sun, Jan 18, 2009 at 2:41 PM, Leif Walsh <leif.walsh at gmail.com> wrote:
> > On Wed, Jan 14, 2009 at 2:12 AM, Carl Johnson <carl at carlsensei.com>
> wrote:
> >> From the department of doomed ideas:
> >>
> >>     data = f.read() with open("file.txt") as f
> >
> > "Who do you think we are, Ruby?"
> >
> > Just kidding, but -1.
>
> I always preferred:
>
> data = open("file.txt").read()
>
> Which works all the way back as far as I can remember.
>

In C Python yes that "works" but it makes absolutely no guarantees about
when that file is actually closed.  Don't write code that depends on a
particular GC behavior.  Use with. :)


>
> And as for:
> strippedLines = [sline for line in lines with line.strip() as sline if
> sline]
>
> Try:
> strippedLines = [sline for sline in (line.strip() for line in lines) if
> sline]
>
> The request gets a big fat -1 from me.
>
>  - Josiah
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20090119/40022dc5/attachment.html>

From sturla at molden.no  Tue Jan 20 14:00:31 2009
From: sturla at molden.no (Sturla Molden)
Date: Tue, 20 Jan 2009 14:00:31 +0100
Subject: [Python-ideas] With-expression
In-Reply-To: <52dc1c820901191747s3fee79a4u119d5623a3796205@mail.gmail.com>
References: <82936ECA-C3F2-40D2-90CE-3EBA009AF314@carlsensei.com>	<cc7430500901181441xcd47df6v3844b42ac0b6d567@mail.gmail.com>	<e6511dbf0901191746p5e4cd6c1gb8e1f52398a4b55c@mail.gmail.com>
	<52dc1c820901191747s3fee79a4u119d5623a3796205@mail.gmail.com>
Message-ID: <4975CAEF.4090304@molden.no>

On 1/20/2009 2:47 AM, Gregory P. Smith wrote:

>     I always preferred:
> 
>     data = open("file.txt").read()
> 
>     Which works all the way back as far as I can remember.
> 
> 
> In C Python yes that "works" but it makes absolutely no guarantees about 
> when that file is actually closed.  Don't write code that depends on a 
> particular GC behavior. 

Then why not introduce that guarantee? Indeterministic GC behaviour is a 
Jython and IronPython problem.

And if you are worried about not closing the file, why not close it 
manually?

S.M.




From ironfroggy at gmail.com  Tue Jan 20 15:38:09 2009
From: ironfroggy at gmail.com (Calvin Spealman)
Date: Tue, 20 Jan 2009 09:38:09 -0500
Subject: [Python-ideas] With-expression
In-Reply-To: <4975CAEF.4090304@molden.no>
References: <82936ECA-C3F2-40D2-90CE-3EBA009AF314@carlsensei.com>
	<cc7430500901181441xcd47df6v3844b42ac0b6d567@mail.gmail.com>
	<e6511dbf0901191746p5e4cd6c1gb8e1f52398a4b55c@mail.gmail.com>
	<52dc1c820901191747s3fee79a4u119d5623a3796205@mail.gmail.com>
	<4975CAEF.4090304@molden.no>
Message-ID: <76fd5acf0901200638h20135aearaeeee4fe785e40e2@mail.gmail.com>

On Tue, Jan 20, 2009 at 8:00 AM, Sturla Molden <sturla at molden.no> wrote:
> On 1/20/2009 2:47 AM, Gregory P. Smith wrote:
>
>>    I always preferred:
>>
>>    data = open("file.txt").read()
>>
>>    Which works all the way back as far as I can remember.
>>
>>
>> In C Python yes that "works" but it makes absolutely no guarantees about
>> when that file is actually closed.  Don't write code that depends on a
>> particular GC behavior.
>
> Then why not introduce that guarantee? Indeterministic GC behaviour is a
> Jython and IronPython problem.

We cannot and should not define aspects of the language which do
(especially on purpose) specifically deny other implementations from
respecting that change.

> And if you are worried about not closing the file, why not close it
> manually?
>
> S.M.
>
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>



-- 
Read my blog! I depend on your acceptance of my opinion! I am interesting!
http://techblog.ironfroggy.com/
Follow me if you're into that sort of thing: http://www.twitter.com/ironfroggy


From grosser.meister.morti at gmx.net  Tue Jan 20 21:25:53 2009
From: grosser.meister.morti at gmx.net (=?ISO-8859-1?Q?Mathias_Panzenb=F6ck?=)
Date: Tue, 20 Jan 2009 21:25:53 +0100
Subject: [Python-ideas] With-expression
In-Reply-To: <76fd5acf0901200638h20135aearaeeee4fe785e40e2@mail.gmail.com>
References: <82936ECA-C3F2-40D2-90CE-3EBA009AF314@carlsensei.com>	<cc7430500901181441xcd47df6v3844b42ac0b6d567@mail.gmail.com>	<e6511dbf0901191746p5e4cd6c1gb8e1f52398a4b55c@mail.gmail.com>	<52dc1c820901191747s3fee79a4u119d5623a3796205@mail.gmail.com>	<4975CAEF.4090304@molden.no>
	<76fd5acf0901200638h20135aearaeeee4fe785e40e2@mail.gmail.com>
Message-ID: <49763351.5040202@gmx.net>

Calvin Spealman schrieb:
> On Tue, Jan 20, 2009 at 8:00 AM, Sturla Molden <sturla at molden.no> wrote:
>> On 1/20/2009 2:47 AM, Gregory P. Smith wrote:
>>
>>>    I always preferred:
>>>
>>>    data = open("file.txt").read()
>>>
>>>    Which works all the way back as far as I can remember.
>>>
>>>
>>> In C Python yes that "works" but it makes absolutely no guarantees about
>>> when that file is actually closed.  Don't write code that depends on a
>>> particular GC behavior.
>> Then why not introduce that guarantee? Indeterministic GC behaviour is a
>> Jython and IronPython problem.
>
> We cannot and should not define aspects of the language which do
> (especially on purpose) specifically deny other implementations from
> respecting that change.
>

And who knows, maybe someday Pythons GC will change its strategy to something
more like Java? Then this would break a lot of code. (Well, most likely it would
anyway.)

	-panzi


From ironfroggy at gmail.com  Wed Jan 21 02:32:21 2009
From: ironfroggy at gmail.com (Calvin Spealman)
Date: Tue, 20 Jan 2009 20:32:21 -0500
Subject: [Python-ideas] min_fields argument to str.split()
Message-ID: <76fd5acf0901201732r158d3b9bhf865ee9dc7a4c6e0@mail.gmail.com>

This might be a silly idea, but I was wondering about forcing split()
to return at least X number of items. For example, I might be getting
a comma separated string and i want to split it up into names, but I
might have less than all of them. If it is just 0, 1, or 2, I can use
partition(), but any more and that doesn't work. Besides, I don't care
if the separator is there, just to get the values. Might also make
sense to give the values to give by default.

Example of implementing this:

def split(self, sep=None, max_splits=None, min_items=None):
    parts = self.split(sep, max_splits)
    if len(parts) < min_items:
        parts.extend([None] * (min_items - len(parts)))
    return parts

Use would be like this:

a, b, c, d = "1,2,3".split(',', None, 4)

Probably not a great idea, but I'm tossing it out there, anyway.

-- 
Read my blog! I depend on your acceptance of my opinion! I am interesting!
http://techblog.ironfroggy.com/
Follow me if you're into that sort of thing: http://www.twitter.com/ironfroggy


From algorias at yahoo.com  Wed Jan 21 02:50:51 2009
From: algorias at yahoo.com (Vitor Bosshard)
Date: Tue, 20 Jan 2009 17:50:51 -0800 (PST)
Subject: [Python-ideas] min_fields argument to str.split()
References: <76fd5acf0901201732r158d3b9bhf865ee9dc7a4c6e0@mail.gmail.com>
Message-ID: <162626.90154.qm@web54406.mail.yahoo.com>



----- Mensaje original ----
> De: Calvin Spealman <ironfroggy at gmail.com>
> Para: python-ideas at python.org
> Enviado: martes, 20 de enero, 2009 22:32:21
> Asunto: [Python-ideas] min_fields argument to str.split()
> 
> This might be a silly idea, but I was wondering about forcing split()
> to return at least X number of items. For example, I might be getting
> a comma separated string and i want to split it up into names, but I
> might have less than all of them. If it is just 0, 1, or 2, I can use
> partition(), but any more and that doesn't work. Besides, I don't care
> if the separator is there, just to get the values. Might also make
> sense to give the values to give by default.
> 
> Example of implementing this:
> 
> def split(self, sep=None, max_splits=None, min_items=None):
>     parts = self.split(sep, max_splits)
>     if len(parts) < min_items:
>         parts.extend([None] * (min_items - len(parts)))
>     return parts
> 
> Use would be like this:
> 
> a, b, c, d = "1,2,3".split(',', None, 4)


How about this?

def splitter(string,sep=None,max_splits=-1, min_items=None):
    parts = string.split(sep,max_splits)
    for part in parts:
        yield part
    for i in range(min_items-len(parts)):
        yield


a, b, c, d = splitter("1,2,3", ",", -1, 4)


Vitor


      ?Todo sobre la Liga Mexicana de f?tbol! Estadisticas, resultados, calendario, fotos y m?s:&lt;
http://espanol.sports.yahoo.com/


From charlie137 at gmail.com  Wed Jan 21 03:36:05 2009
From: charlie137 at gmail.com (Guillaume Chereau)
Date: Wed, 21 Jan 2009 10:36:05 +0800
Subject: [Python-ideas] standard library proposal for "tasklets" using
	generators
Message-ID: <8e9327d40901201836w40866b67w476f8cd20c579558@mail.gmail.com>

Hello,
Thanks to the introduction of new generator features, as described in
PEP 0342, it is possible to write what I would call "tasklet" (I am no
sure what should be the term exactly).
tasklet are a way to write callback based code that looks like thread. Example :

(assuming we have a function sleep(sec, callback) that will call the
callback after `sec` seconds)

@tasklet
task1():
    print "sleep"
    yield Wait(sleep, 5)
    print "called after sleep"
    yield 10 # returned value

@tasklet
task2():
    for i in range(10):
        value = yield task1()

(the same thing using callback only would be quite difficult to read)

The library should provide -the names are not very important- a
Tasklet class, plus a tasklet decorator to turn any generator into a
tasklet, plus a Wait function to turn a callback function into a
tasklet, plus maybe a few other functions.

In many cases this kind of library would be very useful. I am
currently using something very similar for a project I am working on
[0]. the kiwi project also have a similar library [1] (but the syntax
is not really nice because they don't take advantage of the PEP 0342
features.)

I guess more and more people may try to use similar library in the
future, but the problem is that the implementation of it is really not
trivial, so I would like to suggest having such a library in the
standard set of python libraries.

Do you think this is a good idea ?

cheers,

Guillaume

[0] http://git.openmoko.org/?p=tichy.git;a=blob;f=tichy/tasklet.py
[1] http://www.async.com.br/projects/kiwi/api/kiwi.tasklet.html

-- 
http://charlie137.blogspot.com/


From guido at python.org  Wed Jan 21 03:51:37 2009
From: guido at python.org (Guido van Rossum)
Date: Tue, 20 Jan 2009 18:51:37 -0800
Subject: [Python-ideas] min_fields argument to str.split()
In-Reply-To: <76fd5acf0901201732r158d3b9bhf865ee9dc7a4c6e0@mail.gmail.com>
References: <76fd5acf0901201732r158d3b9bhf865ee9dc7a4c6e0@mail.gmail.com>
Message-ID: <ca471dc20901201851o5a024d28rc14c9922f810ef9@mail.gmail.com>

On Tue, Jan 20, 2009 at 5:32 PM, Calvin Spealman <ironfroggy at gmail.com> wrote:
> This might be a silly idea, but I was wondering about forcing split()
> to return at least X number of items. For example, I might be getting
> a comma separated string and i want to split it up into names, but I
> might have less than all of them. If it is just 0, 1, or 2, I can use
> partition(), but any more and that doesn't work. Besides, I don't care
> if the separator is there, just to get the values. Might also make
> sense to give the values to give by default.
>
> Example of implementing this:
>
> def split(self, sep=None, max_splits=None, min_items=None):
>    parts = self.split(sep, max_splits)
>    if len(parts) < min_items:
>        parts.extend([None] * (min_items - len(parts)))
>    return parts
>
> Use would be like this:
>
> a, b, c, d = "1,2,3".split(',', None, 4)
>
> Probably not a great idea, but I'm tossing it out there, anyway.

Doesn't strike me as easy to use; the None values returned would most
likely cause a traceback a few lines down. Returning '' might be
better, but I still think it's very marginal functionality and makes
the API ugly due to the large number of optional arguments.

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)


From guido at python.org  Wed Jan 21 03:56:48 2009
From: guido at python.org (Guido van Rossum)
Date: Tue, 20 Jan 2009 18:56:48 -0800
Subject: [Python-ideas] standard library proposal for "tasklets" using
	generators
In-Reply-To: <8e9327d40901201836w40866b67w476f8cd20c579558@mail.gmail.com>
References: <8e9327d40901201836w40866b67w476f8cd20c579558@mail.gmail.com>
Message-ID: <ca471dc20901201856v77bc8dc2h6c22cd02b741c413@mail.gmail.com>

The pattern of invoking all sorts of stuff that may wait indefinitely
using yield has been proposed before, but I find it creates rather
awkward code compared to using plain threads. Not something I'd like
to encourage by having standard decorators and other APIs around this
idea. I believe Twisted has something like this, and that's probably a
good place to isolate such patterns; in their case they really
want/need this, so they put up with the awkwardness. Putting the
necessary decorators/APIs in the language doesn't do much about the
awkwardness, so I don't think we should do this.

On Tue, Jan 20, 2009 at 6:36 PM, Guillaume Chereau <charlie137 at gmail.com> wrote:
> Hello,
> Thanks to the introduction of new generator features, as described in
> PEP 0342, it is possible to write what I would call "tasklet" (I am no
> sure what should be the term exactly).
> tasklet are a way to write callback based code that looks like thread. Example :
>
> (assuming we have a function sleep(sec, callback) that will call the
> callback after `sec` seconds)
>
> @tasklet
> task1():
>    print "sleep"
>    yield Wait(sleep, 5)
>    print "called after sleep"
>    yield 10 # returned value
>
> @tasklet
> task2():
>    for i in range(10):
>        value = yield task1()
>
> (the same thing using callback only would be quite difficult to read)
>
> The library should provide -the names are not very important- a
> Tasklet class, plus a tasklet decorator to turn any generator into a
> tasklet, plus a Wait function to turn a callback function into a
> tasklet, plus maybe a few other functions.
>
> In many cases this kind of library would be very useful. I am
> currently using something very similar for a project I am working on
> [0]. the kiwi project also have a similar library [1] (but the syntax
> is not really nice because they don't take advantage of the PEP 0342
> features.)
>
> I guess more and more people may try to use similar library in the
> future, but the problem is that the implementation of it is really not
> trivial, so I would like to suggest having such a library in the
> standard set of python libraries.
>
> Do you think this is a good idea ?
>
> cheers,
>
> Guillaume
>
> [0] http://git.openmoko.org/?p=tichy.git;a=blob;f=tichy/tasklet.py
> [1] http://www.async.com.br/projects/kiwi/api/kiwi.tasklet.html
>
> --
> http://charlie137.blogspot.com/
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>



-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)


From taleinat at gmail.com  Wed Jan 21 11:29:17 2009
From: taleinat at gmail.com (Tal Einat)
Date: Wed, 21 Jan 2009 12:29:17 +0200
Subject: [Python-ideas] min_fields argument to str.split()
In-Reply-To: <76fd5acf0901201732r158d3b9bhf865ee9dc7a4c6e0@mail.gmail.com>
References: <76fd5acf0901201732r158d3b9bhf865ee9dc7a4c6e0@mail.gmail.com>
Message-ID: <7afdee2f0901210229q4b591777t9d18fa9d7b56e332@mail.gmail.com>

Calvin Spealman wrote:
> This might be a silly idea, but I was wondering about forcing split()
> to return at least X number of items. For example, I might be getting
> a comma separated string and i want to split it up into names, but I
> might have less than all of them. If it is just 0, 1, or 2, I can use
> partition(), but any more and that doesn't work. Besides, I don't care
> if the separator is there, just to get the values. Might also make
> sense to give the values to give by default.
>
> Example of implementing this:
>
> def split(self, sep=None, max_splits=None, min_items=None):
>    parts = self.split(sep, max_splits)
>    if len(parts) < min_items:
>        parts.extend([None] * (min_items - len(parts)))
>    return parts
>
> Use would be like this:
>
> a, b, c, d = "1,2,3".split(',', None, 4)
>
> Probably not a great idea, but I'm tossing it out there, anyway.

This is not a common thing to do, and it is easy enough to implement
in a single assignment even without a helper function:

a, b, c, d = ("1,2,3".split(',', 4) + [None] * 4)[:4]

This also has the advantage of giving you explicit control of the
filler value for missing pieces.

(Note: be careful not to use a _mutable_ filler value in this manner!)

- Tal


From charlie137 at gmail.com  Wed Jan 21 11:45:55 2009
From: charlie137 at gmail.com (Guillaume Chereau)
Date: Wed, 21 Jan 2009 18:45:55 +0800
Subject: [Python-ideas] Fwd: standard library proposal for "tasklets" using
	generators
In-Reply-To: <8e9327d40901202000o34908c1dv27e7ef8ac969a14c@mail.gmail.com>
References: <8e9327d40901201836w40866b67w476f8cd20c579558@mail.gmail.com>
	<ca471dc20901201856v77bc8dc2h6c22cd02b741c413@mail.gmail.com>
	<8e9327d40901202000o34908c1dv27e7ef8ac969a14c@mail.gmail.com>
Message-ID: <8e9327d40901210245v63dbbc9bn2132626858e6811a@mail.gmail.com>

Just realized i clicked "reply" instead of 'reply to all"


---------- Forwarded message ----------
From: Guillaume Chereau <charlie137 at gmail.com>
Date: Wed, Jan 21, 2009 at 12:00 PM
Subject: Re: [Python-ideas] standard library proposal for "tasklets"
using  generators
To: Guido van Rossum <guido at python.org>


Thanks for the answer,

I understand the concern here.

I still think having such a library, if not as a standard python lib,
but at least as an independent one, that could then be used by all the
project using such pattern (twisted, kiwi, gobject, etc.) would be a
good idea.
I will try to contact those different projects for suggestions.

I also have to say one of my secret hope was that the pypy interpreter
could eventually understand this kind of 'tasklet' generators and
generate C or other language code out of it, since pypy already has
support for tasklets, but using a syntax that can't be interpreted by
cpython.

- Guillaume

On Wed, Jan 21, 2009 at 10:56 AM, Guido van Rossum <guido at python.org> wrote:
> The pattern of invoking all sorts of stuff that may wait indefinitely
> using yield has been proposed before, but I find it creates rather
> awkward code compared to using plain threads. Not something I'd like
> to encourage by having standard decorators and other APIs around this
> idea. I believe Twisted has something like this, and that's probably a
> good place to isolate such patterns; in their case they really
> want/need this, so they put up with the awkwardness. Putting the
> necessary decorators/APIs in the language doesn't do much about the
> awkwardness, so I don't think we should do this.
>
> On Tue, Jan 20, 2009 at 6:36 PM, Guillaume Chereau <charlie137 at gmail.com> wrote:
>> Hello,
>> Thanks to the introduction of new generator features, as described in
>> PEP 0342, it is possible to write what I would call "tasklet" (I am no
>> sure what should be the term exactly).
>> tasklet are a way to write callback based code that looks like thread. Example :
>>
>> (assuming we have a function sleep(sec, callback) that will call the
>> callback after `sec` seconds)
>>
>> @tasklet
>> task1():
>>    print "sleep"
>>    yield Wait(sleep, 5)
>>    print "called after sleep"
>>    yield 10 # returned value
>>
>> @tasklet
>> task2():
>>    for i in range(10):
>>        value = yield task1()
>>
>> (the same thing using callback only would be quite difficult to read)
>>
>> The library should provide -the names are not very important- a
>> Tasklet class, plus a tasklet decorator to turn any generator into a
>> tasklet, plus a Wait function to turn a callback function into a
>> tasklet, plus maybe a few other functions.
>>
>> In many cases this kind of library would be very useful. I am
>> currently using something very similar for a project I am working on
>> [0]. the kiwi project also have a similar library [1] (but the syntax
>> is not really nice because they don't take advantage of the PEP 0342
>> features.)
>>
>> I guess more and more people may try to use similar library in the
>> future, but the problem is that the implementation of it is really not
>> trivial, so I would like to suggest having such a library in the
>> standard set of python libraries.
>>
>> Do you think this is a good idea ?
>>
>> cheers,
>>
>> Guillaume
>>
>> [0] http://git.openmoko.org/?p=tichy.git;a=blob;f=tichy/tasklet.py
>> [1] http://www.async.com.br/projects/kiwi/api/kiwi.tasklet.html
>>
>> --
>> http://charlie137.blogspot.com/
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> http://mail.python.org/mailman/listinfo/python-ideas
>>
>
>
>
> --
> --Guido van Rossum (home page: http://www.python.org/~guido/)
>



--
http://charlie137.blogspot.com/



-- 
http://charlie137.blogspot.com/


From arnodel at googlemail.com  Wed Jan 21 14:32:56 2009
From: arnodel at googlemail.com (Arnaud Delobelle)
Date: Wed, 21 Jan 2009 13:32:56 +0000
Subject: [Python-ideas] min_fields argument to str.split()
In-Reply-To: <7afdee2f0901210229q4b591777t9d18fa9d7b56e332@mail.gmail.com>
References: <76fd5acf0901201732r158d3b9bhf865ee9dc7a4c6e0@mail.gmail.com>
	<7afdee2f0901210229q4b591777t9d18fa9d7b56e332@mail.gmail.com>
Message-ID: <9bfc700a0901210532k57119aa8h6fb8746c5cbd02ea@mail.gmail.com>

2009/1/21 Tal Einat <taleinat at gmail.com>:
> Calvin Spealman wrote:
>> This might be a silly idea, but I was wondering about forcing split()
>> to return at least X number of items. For example, I might be getting
>> a comma separated string and i want to split it up into names, but I
>> might have less than all of them. If it is just 0, 1, or 2, I can use
>> partition(), but any more and that doesn't work. Besides, I don't care
>> if the separator is there, just to get the values. Might also make
>> sense to give the values to give by default.
>>
>> Example of implementing this:
>>
>> def split(self, sep=None, max_splits=None, min_items=None):
>>    parts = self.split(sep, max_splits)
>>    if len(parts) < min_items:
>>        parts.extend([None] * (min_items - len(parts)))
>>    return parts
>>
>> Use would be like this:
>>
>> a, b, c, d = "1,2,3".split(',', None, 4)
>>
>> Probably not a great idea, but I'm tossing it out there, anyway.
>
> This is not a common thing to do, and it is easy enough to implement
> in a single assignment even without a helper function:
>
> a, b, c, d = ("1,2,3".split(',', 4) + [None] * 4)[:4]
>
> This also has the advantage of giving you explicit control of the
> filler value for missing pieces.

How about this?

>>> from itertools import *
>>> def fixlen(iterable, len, item=None):
...     return islice(chain(iterable, repeat(item)), len)
...
>>> list(fixlen([1, 2, 3], 5))
[1, 2, 3, None, None]

-- 
Arnaud


From grosser.meister.morti at gmx.net  Wed Jan 21 14:55:08 2009
From: grosser.meister.morti at gmx.net (=?ISO-8859-1?Q?Mathias_Panzenb=F6ck?=)
Date: Wed, 21 Jan 2009 14:55:08 +0100
Subject: [Python-ideas] min_fields argument to str.split()
In-Reply-To: <9bfc700a0901210532k57119aa8h6fb8746c5cbd02ea@mail.gmail.com>
References: <76fd5acf0901201732r158d3b9bhf865ee9dc7a4c6e0@mail.gmail.com>	<7afdee2f0901210229q4b591777t9d18fa9d7b56e332@mail.gmail.com>
	<9bfc700a0901210532k57119aa8h6fb8746c5cbd02ea@mail.gmail.com>
Message-ID: <4977293C.5080103@gmx.net>

Arnaud Delobelle schrieb:
>
> How about this?
>
>>>> from itertools import *
>>>> def fixlen(iterable, len, item=None):
> ...     return islice(chain(iterable, repeat(item)), len)
> ...
>>>> list(fixlen([1, 2, 3], 5))
> [1, 2, 3, None, None]
>

I like this. Maybe an addition to itertools?

+1


From algorias at yahoo.com  Wed Jan 21 15:14:57 2009
From: algorias at yahoo.com (Vitor Bosshard)
Date: Wed, 21 Jan 2009 06:14:57 -0800 (PST)
Subject: [Python-ideas] min_fields argument to str.split()
References: <76fd5acf0901201732r158d3b9bhf865ee9dc7a4c6e0@mail.gmail.com>
	<7afdee2f0901210229q4b591777t9d18fa9d7b56e332@mail.gmail.com>
	<9bfc700a0901210532k57119aa8h6fb8746c5cbd02ea@mail.gmail.com>
	<4977293C.5080103@gmx.net>
Message-ID: <439790.10501.qm@web54406.mail.yahoo.com>

MIME-Version: 1.0
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: quoted-printable

----- Mensaje original ----=0A> De: Mathias Panzenb=F6ck <grosser.meister.m=
orti at gmx.net>=0A> Para: Arnaud Delobelle <arnodel at googlemail.com>=0A> CC: p=
ython-ideas at python.org=0A> Enviado: mi=E9rcoles, 21 de enero, 2009 10:55:08=
=0A> Asunto: Re: [Python-ideas] min_fields argument to str.split()=0A> =0A>=
 Arnaud Delobelle schrieb:=0A> >=0A> > How about this?=0A> >=0A> >>>> from =
itertools import *=0A> >>>> def fixlen(iterable, len, item=3DNone):=0A> > .=
..=A0 =A0 return islice(chain(iterable, repeat(item)), len)=0A> > ...=0A> >=
>>> list(fixlen([1, 2, 3], 5))=0A> > [1, 2, 3, None, None]=0A> >=0A> =0A> I=
 like this. Maybe an addition to itertools?=0A> =0A> +1=0A=0A+1 from me too=
.=A0Way better than the generator I proposed. Having this as a function als=
o makes it=A0applicable to much more than just one use case.=0A=0AIf you se=
e the following code:=0A=0Aa,b,c,d =3D fixlen(str.split(),4)=0A=0Ayou immed=
iately know it's not a regular unpacking operation; it will be the first pl=
ace you look if something goes wrong.=0A=0A=0AVitor=0A=0A=0A      =A1Todo s=
obre la Liga Mexicana de f=FAtbol! Estadisticas, resultados, calendario, fo=
tos y m=E1s:&lt;=0Ahttp://espanol.sports.yahoo.com/


From brett at python.org  Wed Jan 21 21:22:58 2009
From: brett at python.org (Brett Cannon)
Date: Wed, 21 Jan 2009 12:22:58 -0800
Subject: [Python-ideas] Fwd: standard library proposal for "tasklets"
	using generators
In-Reply-To: <8e9327d40901210245v63dbbc9bn2132626858e6811a@mail.gmail.com>
References: <8e9327d40901201836w40866b67w476f8cd20c579558@mail.gmail.com>
	<ca471dc20901201856v77bc8dc2h6c22cd02b741c413@mail.gmail.com>
	<8e9327d40901202000o34908c1dv27e7ef8ac969a14c@mail.gmail.com>
	<8e9327d40901210245v63dbbc9bn2132626858e6811a@mail.gmail.com>
Message-ID: <bbaeab100901211222q174213c5u36167ade880fb303@mail.gmail.com>

On Wed, Jan 21, 2009 at 02:45, Guillaume Chereau <charlie137 at gmail.com> wrote:
> Just realized i clicked "reply" instead of 'reply to all"
>
>
> ---------- Forwarded message ----------
> From: Guillaume Chereau <charlie137 at gmail.com>
> Date: Wed, Jan 21, 2009 at 12:00 PM
> Subject: Re: [Python-ideas] standard library proposal for "tasklets"
> using  generators
> To: Guido van Rossum <guido at python.org>
>
>
> Thanks for the answer,
>
> I understand the concern here.
>
> I still think having such a library, if not as a standard python lib,
> but at least as an independent one, that could then be used by all the
> project using such pattern (twisted, kiwi, gobject, etc.) would be a
> good idea.
> I will try to contact those different projects for suggestions.
>

If the package really takes off and it turns out there is a good
design that leads to clean code this can be re-evaluated. But getting
a third-party library out there that has wide usage is a good first
step.

-Brett



> I also have to say one of my secret hope was that the pypy interpreter
> could eventually understand this kind of 'tasklet' generators and
> generate C or other language code out of it, since pypy already has
> support for tasklets, but using a syntax that can't be interpreted by
> cpython.
>
> - Guillaume
>
> On Wed, Jan 21, 2009 at 10:56 AM, Guido van Rossum <guido at python.org> wrote:
>> The pattern of invoking all sorts of stuff that may wait indefinitely
>> using yield has been proposed before, but I find it creates rather
>> awkward code compared to using plain threads. Not something I'd like
>> to encourage by having standard decorators and other APIs around this
>> idea. I believe Twisted has something like this, and that's probably a
>> good place to isolate such patterns; in their case they really
>> want/need this, so they put up with the awkwardness. Putting the
>> necessary decorators/APIs in the language doesn't do much about the
>> awkwardness, so I don't think we should do this.
>>
>> On Tue, Jan 20, 2009 at 6:36 PM, Guillaume Chereau <charlie137 at gmail.com> wrote:
>>> Hello,
>>> Thanks to the introduction of new generator features, as described in
>>> PEP 0342, it is possible to write what I would call "tasklet" (I am no
>>> sure what should be the term exactly).
>>> tasklet are a way to write callback based code that looks like thread. Example :
>>>
>>> (assuming we have a function sleep(sec, callback) that will call the
>>> callback after `sec` seconds)
>>>
>>> @tasklet
>>> task1():
>>>    print "sleep"
>>>    yield Wait(sleep, 5)
>>>    print "called after sleep"
>>>    yield 10 # returned value
>>>
>>> @tasklet
>>> task2():
>>>    for i in range(10):
>>>        value = yield task1()
>>>
>>> (the same thing using callback only would be quite difficult to read)
>>>
>>> The library should provide -the names are not very important- a
>>> Tasklet class, plus a tasklet decorator to turn any generator into a
>>> tasklet, plus a Wait function to turn a callback function into a
>>> tasklet, plus maybe a few other functions.
>>>
>>> In many cases this kind of library would be very useful. I am
>>> currently using something very similar for a project I am working on
>>> [0]. the kiwi project also have a similar library [1] (but the syntax
>>> is not really nice because they don't take advantage of the PEP 0342
>>> features.)
>>>
>>> I guess more and more people may try to use similar library in the
>>> future, but the problem is that the implementation of it is really not
>>> trivial, so I would like to suggest having such a library in the
>>> standard set of python libraries.
>>>
>>> Do you think this is a good idea ?
>>>
>>> cheers,
>>>
>>> Guillaume
>>>
>>> [0] http://git.openmoko.org/?p=tichy.git;a=blob;f=tichy/tasklet.py
>>> [1] http://www.async.com.br/projects/kiwi/api/kiwi.tasklet.html
>>>
>>> --
>>> http://charlie137.blogspot.com/
>>> _______________________________________________
>>> Python-ideas mailing list
>>> Python-ideas at python.org
>>> http://mail.python.org/mailman/listinfo/python-ideas
>>>
>>
>>
>>
>> --
>> --Guido van Rossum (home page: http://www.python.org/~guido/)
>>
>
>
>
> --
> http://charlie137.blogspot.com/
>
>
>
> --
> http://charlie137.blogspot.com/
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>


From algorias at gmail.com  Thu Jan 22 02:58:06 2009
From: algorias at gmail.com (Vitor Bosshard)
Date: Wed, 21 Jan 2009 22:58:06 -0300
Subject: [Python-ideas] min_fields argument to str.split()
In-Reply-To: <2987c46d0901211748n6b2559fcoe0d67c45f38f6a6f@mail.gmail.com>
References: <2987c46d0901211748n6b2559fcoe0d67c45f38f6a6f@mail.gmail.com>
Message-ID: <2987c46d0901211758m1e99d76bo879960e8eee6c079@mail.gmail.com>

NOTE: It was brought to my attention that my last e-mail message arrived
completely garbled. Here it is again, as I meant to send it. My Apologies to
everyone.


> >
> > How about this?
> >
> >>>> from itertools import *
> >>>> def fixlen(iterable, len, item=None):
> > ...    return islice(chain(iterable, repeat(item)), len)
> > ...
> >>>> list(fixlen([1, 2, 3], 5))
> > [1, 2, 3, None, None]
> >
>
> I like this. Maybe an addition to itertools?
>
> +1


+1 from me too.

Way better than the generator I proposed. Having this as a function also
makes it applicable to much more than just one use case.

If you see the following code:

a,b,c,d = fixlen(str.split(),4)

You immediately know it's not a regular unpacking operation; it will be the
first place you look if something goes wrong.


Vitor
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20090121/e1fb3ef0/attachment.html>

From sturla at molden.no  Thu Jan 22 11:53:13 2009
From: sturla at molden.no (Sturla Molden)
Date: Thu, 22 Jan 2009 11:53:13 +0100
Subject: [Python-ideas] Suggestion for a new thread API for Python (OpenMP
	inspired)
Message-ID: <49785019.5070606@molden.no>


I have lately been using OpenMP to write parallel C and Fortran code. I 
must admit I am impressed. OpenMP is a much better abstraction for 
writing concurrent code than using Win32/posix threads directly (or 
similarly threading.Thread i Java or Python). What matters most is that 
code can be written as sequential, tested, and then parallelised using 
compiler pragmas. This is much easier than writing code intended to be 
parallel from the start.

Not only is the abstraction more easy to apply, it also leads to fewer 
problems with deadlocks, race conditions, livelocks, etc.

I was thinking something similar could be created for Python, e.g. on 
top of the existing thread or threading modules, and possibly 
multiprocessing. I believe a context manager could be used for this 
purpose. What I have in mind is an API that would look approximately 
like this (OpenMP pragmas for C on top, proposed Python equivalent below):


#pragma omp parallel
with pymp.Pool() as pool:

#pragma omp for
for item in pool.parallel(<iterable>):

#pragma omp for shedule(guided)
for item in pool.parallel(<iterable>, shed='guided'):

#pragma omp parallel for
with pymp.Pool() as pool:
    for item in pool.parallel(<iterable>):

#pragma omp barrier
pool.barrier()

#pragma omp section
pool.section(fun, *args, **kwargs)

#pragma omp parallel sections
with pymp.Pool() as pool:
    pool.section(fun1, *args, **kwargs)
    pool.section(fun2, *args, **kwargs)

#pragma omp master
if pool.master:

#pragma omp critical
#pragma omp atomic
with pool.lock:

#pragma omp single
with pool.single():

#pragma omp ordered
with pool.ordered():


This is all trivial to program, except for the context manager on top. 
It has somehow to get access to the code block below, spawn multiple 
threads, and execute that block in each of the threads. I am not sure 
how to grab the next executable block as a Python object (so I could 
pass it to eval or exec), so a little help would be appreciated :)


Regards,
Sturla Molden



From jnoller at gmail.com  Thu Jan 22 15:17:28 2009
From: jnoller at gmail.com (Jesse Noller)
Date: Thu, 22 Jan 2009 09:17:28 -0500
Subject: [Python-ideas] Suggestion for a new thread API for Python
	(OpenMP inspired)
In-Reply-To: <49785019.5070606@molden.no>
References: <49785019.5070606@molden.no>
Message-ID: <4222a8490901220617h4d1888yc7850481ca63acbf@mail.gmail.com>

On Thu, Jan 22, 2009 at 5:53 AM, Sturla Molden <sturla at molden.no> wrote:
>
> I have lately been using OpenMP to write parallel C and Fortran code. I must
> admit I am impressed. OpenMP is a much better abstraction for writing
> concurrent code than using Win32/posix threads directly (or similarly
> threading.Thread i Java or Python). What matters most is that code can be
> written as sequential, tested, and then parallelised using compiler pragmas.
> This is much easier than writing code intended to be parallel from the
> start.
>
> Not only is the abstraction more easy to apply, it also leads to fewer
> problems with deadlocks, race conditions, livelocks, etc.
>
> I was thinking something similar could be created for Python, e.g. on top of
> the existing thread or threading modules, and possibly multiprocessing. I
> believe a context manager could be used for this purpose. What I have in
> mind is an API that would look approximately like this (OpenMP pragmas for C
> on top, proposed Python equivalent below):
>
>
> #pragma omp parallel
> with pymp.Pool() as pool:
>
> #pragma omp for
> for item in pool.parallel(<iterable>):
>
> #pragma omp for shedule(guided)
> for item in pool.parallel(<iterable>, shed='guided'):
>
> #pragma omp parallel for
> with pymp.Pool() as pool:
>   for item in pool.parallel(<iterable>):
>
> #pragma omp barrier
> pool.barrier()
>
> #pragma omp section
> pool.section(fun, *args, **kwargs)
>
> #pragma omp parallel sections
> with pymp.Pool() as pool:
>   pool.section(fun1, *args, **kwargs)
>   pool.section(fun2, *args, **kwargs)
>
> #pragma omp master
> if pool.master:
>
> #pragma omp critical
> #pragma omp atomic
> with pool.lock:
>
> #pragma omp single
> with pool.single():
>
> #pragma omp ordered
> with pool.ordered():
>
>
> This is all trivial to program, except for the context manager on top. It
> has somehow to get access to the code block below, spawn multiple threads,
> and execute that block in each of the threads. I am not sure how to grab the
> next executable block as a Python object (so I could pass it to eval or
> exec), so a little help would be appreciated :)
>

Hi Sturla,

 Interesting that you bring this up - while I'm not in the know about
openMP - I have been sketching out some improvements to threading and
multiprocessing that follow some of this thinking.

The first batch would be adding context managers where appropriate to
the multiprocessing module (e.g. pool and managers/etc). The second is
adding a library of decorators where you could say:

@multiprocessing.Pool(5, 'apply')
def myfunc()
  ....

Or something along those lines. In the case of getting the function
object back, decorators make more sense for some of the use cases.

Now, for threading - I personally feel that there's some improvements
to be made by adding a series of abstractions and utilities on top of
threading ala multiprocessing.pool and the like. See
java.util.concurrent for some good examples of threading abstractions.

The same decorators and contextmanagers could be used for any
threading abstractions added.

Just some thoughts - right now I've sworn off writing new features
until I zero out the multiprocessing bug queue, so these might be done
in 2025.

-jesse


From sturla at molden.no  Thu Jan 22 16:03:06 2009
From: sturla at molden.no (Sturla Molden)
Date: Thu, 22 Jan 2009 16:03:06 +0100
Subject: [Python-ideas] Suggestion for a new thread API for Python
 (OpenMP inspired)
In-Reply-To: <4222a8490901220617h4d1888yc7850481ca63acbf@mail.gmail.com>
References: <49785019.5070606@molden.no>
	<4222a8490901220617h4d1888yc7850481ca63acbf@mail.gmail.com>
Message-ID: <49788AAA.2040406@molden.no>

On 1/22/2009 3:17 PM, Jesse Noller wrote:

> Just some thoughts - right now I've sworn off writing new features
> until I zero out the multiprocessing bug queue, so these might be done
> in 2025.

I was actually thinking about writing it myself. :)

I think the OpenMP style of abstracting concurrency fits better with the 
mind, and therefore is more 'pythonic' than the Java inspired threading 
module. Not just for the sake of speed on SMPs, but also for other 
concurrent tasks for which threads (or processes) can be used.

A skeleton of such a context manager would look roughly like this.

Anyway I am stuck with what to exec in the static method 
Pool._threadproc, in order to execute the correct code object. I somehow 
have to exec (or eval) the code object in the frame from f_lineno or 
f_lasti.

I have noticed the Queue bug. Very annoying indeed.


import thread
import sys
import os
from math import ceil, log


class Pool(object):

     def __init__(self, nthreads=None, masterpool=None, tid=0):
         self.nthreads = nthreads
         self.masterpool = masterpool
         self.tid = tid
         if nthreads is None:
             self.nthreads = self._num_processors()


     def __enter__(self):
         if self.masterpool is None:
             frame = sys._getframe().f_back
             code = frame.f_code
             lasti = frame.f_lasti
             localdict = frame.f_locals
             globaldict = frame.f_globals

             # find name of self in frame
             for name in localdict:
                 if localdict[name] is self:
                     selfname = name
                     break

             # spawn threads
             for tid in range(1,self.nthreads):
                 pool = Pool(nthreads=self.nthreads,
			masterpool=self, tid=tid)
                 _localdict = localdict.copy()
                 _globaldict = globaldict.copy()
                 _localdict[selfname] = pool
                 thread.start_new_thread(self._threadproc,
                         (code,lasti,_globaldict,_localdict))


     def __exit__(self, exc_type, exc_val, exc_tb):
         self.barrier()
         if self.masterpool is not None:
             thread.exit()
         # ++ more cleanup


     @staticmethod
     def _threadproc(code, lasti, globals, locals):
         # we must somehow execute the code object from
         # the last instruction (lasti)
         # Something like 'exec code in globals, locals'


     @staticmethod
     def _num_processors():
         if os.name == 'nt':
             return int(os.getenv('NUMBER_OF_PROCESSORS'))
         elif sys.platform == 'linux2':
             # Linux ... I think this works
             # each cpu is separated by a blank line, and there
             # is a blank line at the end
             retv = 0
             with open('/proc/cpuinfo','rt') as cpuinfo:
                 for line in cpuinfo:
                     if not len(line): retv += 1
             return retv
         else:
             raise RuntimeError, 'unknown platform'


     def barrier(self):
         ''' dissemination barrier '''
         nt = self.nthreads
         if nt == 1: return
         tid = self.tid
         log2 = lambda x :
         for k in range(int(ceil(log(nt)/log(2)))):
             # send event to thread (tid + 2**k) % nt
             # wait for event from thread (tid - 2**k) % nt


Regards,
S.M.



From sturla at molden.no  Thu Jan 22 16:26:25 2009
From: sturla at molden.no (Sturla Molden)
Date: Thu, 22 Jan 2009 16:26:25 +0100
Subject: [Python-ideas] Suggestion for a new thread API for Python
 (OpenMP inspired)
In-Reply-To: <4222a8490901220617h4d1888yc7850481ca63acbf@mail.gmail.com>
References: <49785019.5070606@molden.no>
	<4222a8490901220617h4d1888yc7850481ca63acbf@mail.gmail.com>
Message-ID: <49789021.2080101@molden.no>

On 1/22/2009 3:17 PM, Jesse Noller wrote:

>  Interesting that you bring this up - while I'm not in the know about
> openMP - I have been sketching out some improvements to threading and
> multiprocessing that follow some of this thinking.

Here is a toy example of what I have in mind. Say you would want to 
compute the DFT of some signal (real apps would use an FFT in C for 
this, but never mind). In Python using an O(n**2) algorithm, this would 
look like somthing like this:

def real_dft(x):
    ''' DFT for a real valued sequence x '''
    r = []
    N = len(x)
    M = N//2 + 1 if N%2 else N//2
    for n in range(M):
       s = 0j
       for k in range(N):
          tmp = 2*pi*k*n/N
          s += x[k] * (cos(tmp) - 1j*sin(tmp))
       r.append(s)
    return r

Then, one could 'magically' transform this algorithm into to a parallel 
one simply by inserting directives from the 'pymp' module:

def real_dft(x):
    ''' DFT for a real valued sequence x '''
    ''' parallelized '''
    r = []
    N = len(x)
    M = N//2 + 1 if N%2 else N//2
    with Pool() as pool:
       for n in pool.parallel(range(M)):
          s = 0j
          for k in range(N):
             tmp = 2*pi*k*n/N
             s += x[k] * (cos(tmp) - 1j*sin(tmp))
          with pool.ordered():
             r.append(s)
    return r

The idea is that 'parallelizing' a sequential algorithm like this is 
much easier than writing a parallel one from scratch using the 
abstractions in threading or multiprocessing.


Sturla Molden




From stargaming at gmail.com  Thu Jan 22 19:25:49 2009
From: stargaming at gmail.com (Robert Lehmann)
Date: Thu, 22 Jan 2009 18:25:49 +0000 (UTC)
Subject: [Python-ideas] min_fields argument to str.split()
References: <76fd5acf0901201732r158d3b9bhf865ee9dc7a4c6e0@mail.gmail.com>
	<7afdee2f0901210229q4b591777t9d18fa9d7b56e332@mail.gmail.com>
	<9bfc700a0901210532k57119aa8h6fb8746c5cbd02ea@mail.gmail.com>
	<4977293C.5080103@gmx.net>
Message-ID: <gladnd$88u$1@ger.gmane.org>

On Wed, 21 Jan 2009 14:55:08 +0100, Mathias Panzenb?ck wrote:

> Arnaud Delobelle schrieb:
>>
>> How about this?
>>
>>>>> from itertools import *
>>>>> def fixlen(iterable, len, item=None):
>> ...     return islice(chain(iterable, repeat(item)), len) ...
>>>>> list(fixlen([1, 2, 3], 5))
>> [1, 2, 3, None, None]
>>
>>
> I like this. Maybe an addition to itertools?
> 
> +1

I opened an issue_ at the Python bug tracker.

.. _issue: http://bugs.python.org/issue5034

-- 
Robert "Stargaming" Lehmann



From joeedh at gmail.com  Fri Jan 23 07:59:01 2009
From: joeedh at gmail.com (joe)
Date: Thu, 22 Jan 2009 23:59:01 -0700
Subject: [Python-ideas] optimized VM ideas
Message-ID: <ca3ef29a0901222259w7ff82eb8x3117e68aa913f218@mail.gmail.com>

So, I've been kicking around some ideas for an optimized python VM.  I freely
admit I'm an amateur at this, but I find the problem of making python code
run faster fascinating.  My ideas arose from observing that google V8's JIT
compiler and type system are much simpler compared to TraceMonkey, but is
also faster, and also learning that SquirrelFish is allegedy faster than V8,
even though it doesn't compile to native code at all (which V8 and I
believe TraceMonkey both do).

This leads me to believe that relatively simple, more general concepts in
VM design can have a bigger impact then specific, highly complicated JIT
solutions, in the context of dynamic languages that can't be easily typed
at compile time.

So I've thought of a few ideas for a more (new) streamlined python VM:

* Simplify the cpython object model as much as possible, while still allowing
  most of the power of the current model.

* Either keep referencing counting, or experiment with some of the newer
  techniques such as pointer escaping. Object models that exclusively rely
  on cyclic GC's have many issues and are hard to get right.

* Possibly modify the bytecode to be register-based, as in SquirrelFish.
  Not sure if this is worth it with python code.

* Use direct threading (which is basically optimizing switch statements to
  be only one or two instructions) for the bytecode loop.

* Remove string lookups for member access entirely, and replaced with a
  system of unique identifyers.  The idea is you would use a hash in the
  types to map a member id to an index.  Hashing ints is faster then strings,
  and I've even thought about experimenting with using collapsed arrays instead
  of hashes.  Of course, the design would still need to support string lookups
  when necessary.  I've thought about this a lot, and I think you'd need the
  same general idea as V8's hidden classes for this to work right (though
  instead of classes, it'd just be member/unique id lookup maps).

I'm not sure I'll have the time to anytime soon to prototype these ideas, but I
thought I'd kick them out there and see what people say.  Note, I'm in no way
suggesting any sort of change to the existing cpython VM (it's way, way too
early for that kind of talk).

references:

v8's design: http://code.google.com/apis/v8/design.html
squirrelfish's design:
http://blog.mozilla.com/dmandelin/2008/06/03/squirrelfish/

Joe


From joeedh at gmail.com  Fri Jan 23 08:20:22 2009
From: joeedh at gmail.com (joe)
Date: Fri, 23 Jan 2009 00:20:22 -0700
Subject: [Python-ideas] optimized VM ideas
In-Reply-To: <ca3ef29a0901222259w7ff82eb8x3117e68aa913f218@mail.gmail.com>
References: <ca3ef29a0901222259w7ff82eb8x3117e68aa913f218@mail.gmail.com>
Message-ID: <ca3ef29a0901222320m42794deflf18f0fd1010032be@mail.gmail.com>

Ah, should've done more fact checking *before* posting the first mail.
 Actually SquirrelFish does compile to native code nowadays, in a
nicely simple way.

On Thu, Jan 22, 2009 at 11:59 PM, joe <joeedh at gmail.com> wrote:
> So, I've been kicking around some ideas for an optimized python VM.  I freely
> admit I'm an amateur at this, but I find the problem of making python code
> run faster fascinating.  My ideas arose from observing that google V8's JIT
> compiler and type system are much simpler compared to TraceMonkey, but is
> also faster, and also learning that SquirrelFish is allegedy faster than V8,
> even though it doesn't compile to native code at all (which V8 and I
> believe TraceMonkey both do).
>
> This leads me to believe that relatively simple, more general concepts in
> VM design can have a bigger impact then specific, highly complicated JIT
> solutions, in the context of dynamic languages that can't be easily typed
> at compile time.
>
> So I've thought of a few ideas for a more (new) streamlined python VM:
>
> * Simplify the cpython object model as much as possible, while still allowing
>  most of the power of the current model.
>
> * Either keep referencing counting, or experiment with some of the newer
>  techniques such as pointer escaping. Object models that exclusively rely
>  on cyclic GC's have many issues and are hard to get right.
>
> * Possibly modify the bytecode to be register-based, as in SquirrelFish.
>  Not sure if this is worth it with python code.
>
> * Use direct threading (which is basically optimizing switch statements to
>  be only one or two instructions) for the bytecode loop.
>
> * Remove string lookups for member access entirely, and replaced with a
>  system of unique identifyers.  The idea is you would use a hash in the
>  types to map a member id to an index.  Hashing ints is faster then strings,
>  and I've even thought about experimenting with using collapsed arrays instead
>  of hashes.  Of course, the design would still need to support string lookups
>  when necessary.  I've thought about this a lot, and I think you'd need the
>  same general idea as V8's hidden classes for this to work right (though
>  instead of classes, it'd just be member/unique id lookup maps).
>
> I'm not sure I'll have the time to anytime soon to prototype these ideas, but I
> thought I'd kick them out there and see what people say.  Note, I'm in no way
> suggesting any sort of change to the existing cpython VM (it's way, way too
> early for that kind of talk).
>
> references:
>
> v8's design: http://code.google.com/apis/v8/design.html
> squirrelfish's design:
> http://blog.mozilla.com/dmandelin/2008/06/03/squirrelfish/
>
> Joe
>


From greg at krypto.org  Fri Jan 23 09:17:15 2009
From: greg at krypto.org (Gregory P. Smith)
Date: Fri, 23 Jan 2009 00:17:15 -0800
Subject: [Python-ideas] optimized VM ideas
In-Reply-To: <ca3ef29a0901222259w7ff82eb8x3117e68aa913f218@mail.gmail.com>
References: <ca3ef29a0901222259w7ff82eb8x3117e68aa913f218@mail.gmail.com>
Message-ID: <52dc1c820901230017r5021294el34e1726079f551c5@mail.gmail.com>

On Thu, Jan 22, 2009 at 10:59 PM, joe <joeedh at gmail.com> wrote:
...

>
> * Use direct threading (which is basically optimizing switch statements to
>  be only one or two instructions) for the bytecode loop.


fyi - http://bugs.python.org/issue4753 does this (at least when using gcc).

The optimization is not about removing instructions per se. but in getting
rid of the single switch statement's unpredictable branch that causes modern
cpus to stall while determining the correct place to jump rather than
speculatively guessing correctly a significant portion of the time.

...

>
> * Remove string lookups for member access entirely, and replaced with a
>  system of unique identifyers.  The idea is you would use a hash in the
>  types to map a member id to an index.  Hashing ints is faster then
> strings,
>  and I've even thought about experimenting with using collapsed arrays
> instead
>  of hashes.  Of course, the design would still need to support string
> lookups
>  when necessary.  I've thought about this a lot, and I think you'd need the
>  same general idea as V8's hidden classes for this to work right (though
>  instead of classes, it'd just be member/unique id lookup maps).


Python strings are already immutable, their hash is computed only once.
strings used in code as attributes are interned so that all occurances of
that string are the same object making the most common lookups for attribute
accesses a table lookup with a pointer equality check.


> I'm not sure I'll have the time to anytime soon to prototype these ideas,
> but I
> thought I'd kick them out there and see what people say.  Note, I'm in no
> way
> suggesting any sort of change to the existing cpython VM (it's way, way too
> early for that kind of talk).


Anyways, the ideas are great and are definately all things people are
considering.  This should be an great year for python performance and all
language VM performance in general.

happy hacking,
Greg
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20090123/8f7baac0/attachment.html>

From santagada at gmail.com  Fri Jan 23 10:36:52 2009
From: santagada at gmail.com (Leonardo Santagada)
Date: Fri, 23 Jan 2009 07:36:52 -0200
Subject: [Python-ideas] optimized VM ideas
In-Reply-To: <ca3ef29a0901222259w7ff82eb8x3117e68aa913f218@mail.gmail.com>
References: <ca3ef29a0901222259w7ff82eb8x3117e68aa913f218@mail.gmail.com>
Message-ID: <1CB5B7C3-53DD-432B-ABE9-A8C35FB370CF@gmail.com>


On Jan 23, 2009, at 4:59 AM, joe wrote:

> So, I've been kicking around some ideas for an optimized python VM.   
> I freely
> admit I'm an amateur at this, but I find the problem of making  
> python code
> run faster fascinating.  My ideas arose from observing that google  
> V8's JIT
> compiler and type system are much simpler compared to TraceMonkey,  
> but is
> also faster, and also learning that SquirrelFish is allegedy faster  
> than V8,
> even though it doesn't compile to native code at all (which V8 and I
> believe TraceMonkey both do).

I don't think TraceMonkey is slower than V8 I believe the last time I  
looked TraceMonkey was faster than V8 and it is becoming even faster  
at each interaction.

The way TraceMonkey works reminds me a bit of Psyco, although I might  
be mixing it with the PyPy JIT. But talking about Psyco, why people  
don't go help Psyco if all they want is a JIT? It is not like the idea  
of having a JIT on Python is even new... Psyco was optimizing code  
even before Webkit/V8 existed.


> This leads me to believe that relatively simple, more general  
> concepts in
> VM design can have a bigger impact then specific, highly complicated  
> JIT
> solutions, in the context of dynamic languages that can't be easily  
> typed
> at compile time.

I believe in the exact oposite, and TraceMonkey is probably one of the  
proofs of that...


> So I've thought of a few ideas for a more (new) streamlined python VM:
>
> * Simplify the cpython object model as much as possible, while still  
> allowing
>  most of the power of the current model.

This would modify the language, so it might be interesting, but would  
generate something which is not Python.

> * Either keep referencing counting, or experiment with some of the  
> newer
>  techniques such as pointer escaping. Object models that exclusively  
> rely
>  on cyclic GC's have many issues and are hard to get right.

Don't know, but a good GC is way faster than what CPython is doing  
already, but maybe it is a good idea to explore some others  
perspectives on this.

> * Possibly modify the bytecode to be register-based, as in  
> SquirrelFish.
>  Not sure if this is worth it with python code.

Maybe it would help a bit. I don't think it would help more than 10%  
tops (but I am completely guessing here)

> * Use direct threading (which is basically optimizing switch  
> statements to
>  be only one or two instructions) for the bytecode loop.

The problem with this is (besides the error someone has already stated  
about your phrasing) that python has really complex bytecodes, so this  
would also only gain around 10% and it only works with compilers that  
accept goto labels which the MSVC for example does not (maybe there  
are more compilers that also doesn't).

> * Remove string lookups for member access entirely, and replaced  
> with a
>  system of unique identifyers.  The idea is you would use a hash in  
> the
>  types to map a member id to an index.  Hashing ints is faster then  
> strings,
>  and I've even thought about experimenting with using collapsed  
> arrays instead
>  of hashes.  Of course, the design would still need to support  
> string lookups
>  when necessary.  I've thought about this a lot, and I think you'd  
> need the
>  same general idea as V8's hidden classes for this to work right  
> (though
>  instead of classes, it'd just be member/unique id lookup maps).

A form of hidden classes is already part of PyPy (but I think that  
only the jit does this). But you can simply remove string lookups as  
people can implement special methods to track this on the current  
Python. As I said before I don't believe changing the semantics of  
python for the sake of performance is even possible.

> I'm not sure I'll have the time to anytime soon to prototype these  
> ideas, but I
> thought I'd kick them out there and see what people say.  Note, I'm  
> in no way
> suggesting any sort of change to the existing cpython VM (it's way,  
> way too
> early for that kind of talk).

If you are not talking about changing CPython VM why not look at Psyco  
and PyPy? :)

> references:
>
> v8's design: http://code.google.com/apis/v8/design.html
> squirrelfish's design:
> http://blog.mozilla.com/dmandelin/2008/06/03/squirrelfish/
>
> Joe

--
Leonardo Santagada
santagada at gmail.com





From solipsis at pitrou.net  Fri Jan 23 13:19:39 2009
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Fri, 23 Jan 2009 12:19:39 +0000 (UTC)
Subject: [Python-ideas] optimized VM ideas
References: <ca3ef29a0901222259w7ff82eb8x3117e68aa913f218@mail.gmail.com>
	<1CB5B7C3-53DD-432B-ABE9-A8C35FB370CF@gmail.com>
Message-ID: <loom.20090123T115823-938@post.gmane.org>

Leonardo Santagada <santagada at ...> writes:
> 
> The way TraceMonkey works reminds me a bit of Psyco, although I might  
> be mixing it with the PyPy JIT.

I'm not sure it works like Psyco. Read the paper on trace trees, it is quite
interesting, and it may be doable to retrofit at least the trace construction
part in the current bytecode execution loop.
http://www.ics.uci.edu/~franz/Site/pubs-pdf/ICS-TR-06-16.pdf

> But talking about Psyco, why people  
> don't go help Psyco if all they want is a JIT? It is not like the idea  
> of having a JIT on Python is even new... Psyco was optimizing code  
> even before Webkit/V8 existed.

Three problems with Psyco:
1) it is not 100% compatible with Python semantics
2) it only works on i386 (not even x86-64)
3) it's (roughly) unmaintained

> Don't know, but a good GC is way faster than what CPython is doing  
> already, but maybe it is a good idea to explore some others  
> perspectives on this.

Without changing how the GC works, it's probably possible to improve things a
bit.
See e.g. http://bugs.python.org/issue4688

> > * Possibly modify the bytecode to be register-based, as in  
> > SquirrelFish.
> >  Not sure if this is worth it with python code.
> 
> Maybe it would help a bit. I don't think it would help more than 10%  
> tops (but I am completely guessing here)

I had thought about this and it seems a problem would be reference counting.
You have to DECREF a register as soon as it isn't used anymore, which adds
some bookkeeping.

> The problem with this is (besides the error someone has already stated  
> about your phrasing) that python has really complex bytecodes, so this  
> would also only gain around 10% and it only works with compilers that  
> accept goto labels which the MSVC for example does not (maybe there  
> are more compilers that also doesn't).

Well the bytecodes are not that complex. A bunch of them are completely
implemented inline the evaluation loop. Also, since it is a stack machine,
there are many trivial bytecodes such as LOAD_FAST, POP_TOP etc.

The threaded code patch in http://bugs.python.org/issue4753 has given between
0% and 20% speedups on pybench totals amongst the various posters (and a 5%
slowdown in one case).

cheers

Antoine.




From jnoller at gmail.com  Fri Jan 23 15:14:28 2009
From: jnoller at gmail.com (Jesse Noller)
Date: Fri, 23 Jan 2009 09:14:28 -0500
Subject: [Python-ideas] Suggestion for a new thread API for Python
	(OpenMP inspired)
In-Reply-To: <49789021.2080101@molden.no>
References: <49785019.5070606@molden.no>
	<4222a8490901220617h4d1888yc7850481ca63acbf@mail.gmail.com>
	<49789021.2080101@molden.no>
Message-ID: <4222a8490901230614y5485e371p8f6e546563740794@mail.gmail.com>

On Thu, Jan 22, 2009 at 10:26 AM, Sturla Molden <sturla at molden.no> wrote:
> On 1/22/2009 3:17 PM, Jesse Noller wrote:
>
>>  Interesting that you bring this up - while I'm not in the know about
>> openMP - I have been sketching out some improvements to threading and
>> multiprocessing that follow some of this thinking.
>
> Here is a toy example of what I have in mind. Say you would want to compute
> the DFT of some signal (real apps would use an FFT in C for this, but never
> mind). In Python using an O(n**2) algorithm, this would look like somthing
> like this:
>
> def real_dft(x):
>   ''' DFT for a real valued sequence x '''
>   r = []
>   N = len(x)
>   M = N//2 + 1 if N%2 else N//2
>   for n in range(M):
>      s = 0j
>      for k in range(N):
>         tmp = 2*pi*k*n/N
>         s += x[k] * (cos(tmp) - 1j*sin(tmp))
>      r.append(s)
>   return r
>
> Then, one could 'magically' transform this algorithm into to a parallel one
> simply by inserting directives from the 'pymp' module:
>
> def real_dft(x):
>   ''' DFT for a real valued sequence x '''
>   ''' parallelized '''
>   r = []
>   N = len(x)
>   M = N//2 + 1 if N%2 else N//2
>   with Pool() as pool:
>      for n in pool.parallel(range(M)):
>         s = 0j
>         for k in range(N):
>            tmp = 2*pi*k*n/N
>            s += x[k] * (cos(tmp) - 1j*sin(tmp))
>         with pool.ordered():
>            r.append(s)
>   return r
>
> The idea is that 'parallelizing' a sequential algorithm like this is much
> easier than writing a parallel one from scratch using the abstractions in
> threading or multiprocessing.
>
>
> Sturla Molden

Interesting - this is a slightly more extreme series of changes then I
was thinking, a good way to approach this would not to be a patch
against python-core (unless you find bugs) but rather as a separate
package hosted outside of core and posted to pypi. I think it has
merit - but would require more use/eyeballs on it than just a few of
us.

-jesse


From santagada at gmail.com  Sat Jan 24 22:08:27 2009
From: santagada at gmail.com (Leonardo Santagada)
Date: Sat, 24 Jan 2009 19:08:27 -0200
Subject: [Python-ideas] optimized VM ideas
In-Reply-To: <ca3ef29a0901232045g187fa9ceufa297643c501c42a@mail.gmail.com>
References: <ca3ef29a0901222259w7ff82eb8x3117e68aa913f218@mail.gmail.com>
	<1CB5B7C3-53DD-432B-ABE9-A8C35FB370CF@gmail.com>
	<ca3ef29a0901231953hb84b7aeied434e4ab9be4b77@mail.gmail.com>
	<ca3ef29a0901232045g187fa9ceufa297643c501c42a@mail.gmail.com>
Message-ID: <715FC3DA-61A5-45C0-A679-C63BFB9073E6@gmail.com>

You should probably at least use a recognized python test suite like  
pystones or richards, or use some real world applications for this.

look at the benchmarks for the pypy project for example:
http://tuatara.cs.uni-duesseldorf.de/benchmark.html

Good luck on you endeavor

On Jan 24, 2009, at 2:45 AM, joe wrote:

> Ok, I ran a little test script through CodeAnalyst.  basically it is
> just a loop that does member lookups and function calls (this is in
> py2.5 btw, will have to install 3.x and test that too).
>
> The results were kindof interesting.  CodeAnalyst is sample-based, so
> I don't know if it's completely reliable.  It is usually reasonably
> correct though.
>
> Much of the time was spent in PyEval_EvalFrameEx (which makes sense,
> it being the code that interprets and executes the bytecode).  14633
> samples were spent in it, of which around 2000 were spent in what
> appears to be the switch jump lookup (I'm not sure, I'm not the best
> at reading assembly code).  After PyEval_EvalFrameEx, the four next
> most expensive calls were PyWrapper_New (at 2879 samples)
> PyInstance_New (at 1418) PyDict_GetItem (at 1298 samples) and
> PyObject_SetAttr (at 1155).
>
> Here's the (rather hastily thrown together) script:
>
> class A:
> 	t = 0
> 	
> class Bleh:
> 	a = 0
> 	b = 0
> 	c = 0
> 	
> 	def __init__(self):
> 		self.a = A()
> 		self.b = []
> 		self.c = ""
> 		
> 	def doit(self, no=0):
> 		if no: return
> 		
> 		while 1:
> 			a = self.a
> 			b = self.b
> 			c = self.c
> 			self.a = a
> 			self.b = b
> 			self.c = c
> 			self.doit(1)
> b = Bleh()
> b.doit()
>
> So, it looks like indirect or direct threading could help (if I'm
> reading the results and asm code right).  Also the attribute lookups
> did appear to take a significant portion of the time, as did
> PyInstance_New and PyWrapper_New (need to look up what those last two
> do).  This is just preliminary though, that isn't a particularly good
> script for profiling, I suspect.
>
> Joe

--
Leonardo Santagada
santagada at gmail.com





From grosser.meister.morti at gmx.net  Sun Jan 25 00:48:20 2009
From: grosser.meister.morti at gmx.net (=?ISO-8859-15?Q?Mathias_Panzenb=F6ck?=)
Date: Sun, 25 Jan 2009 00:48:20 +0100
Subject: [Python-ideas] contextmanager for locale
Message-ID: <497BA8C4.7030702@gmx.net>

When I generate certain files I have to write a date formated as english text,
but the error messages (if there are any) I'd like to keep in the system
language. I propose to add this (or a similar) contextmanager:

@contextmanager
def localectx(category, newlocale=None):
	oldlocale = locale.getlocale(category)
	locale.setlocale(category, newlocale)
	try:
		yield
	finally:
		locale.setlocale(category, oldlocale)

Example usage:

with localectx(locale.LC_TIME,('en','UTF8')):
	sdate = time.strftime('%a %b %d %Y')

spec_file.write('''
%%changelog

* %s Maintainer Name <maintainer at example.com>
''' % sdate)


	-panzi


From grosser.meister.morti at gmx.net  Sun Jan 25 00:53:59 2009
From: grosser.meister.morti at gmx.net (=?ISO-8859-1?Q?Mathias_Panzenb=F6ck?=)
Date: Sun, 25 Jan 2009 00:53:59 +0100
Subject: [Python-ideas] contextmanager for locale
In-Reply-To: <497BA8C4.7030702@gmx.net>
References: <497BA8C4.7030702@gmx.net>
Message-ID: <497BAA17.20108@gmx.net>

Off topic:

Here would be a with-expression or inline with (like the inline if) handy:

spec_file.write('''
%%changelog

* %s Maintainer Name <maintainer at example.com>
''' % (time.strftime('%a %b %d %Y') with localectx(locale.LC_TIME,('en','UTF8'))))

However, this would be a very long line. Not very clean code.

	-panzi


From santagada at gmail.com  Sat Jan 24 22:04:35 2009
From: santagada at gmail.com (Leonardo Santagada)
Date: Sat, 24 Jan 2009 19:04:35 -0200
Subject: [Python-ideas] optimized VM ideas
In-Reply-To: <ca3ef29a0901231953hb84b7aeied434e4ab9be4b77@mail.gmail.com>
References: <ca3ef29a0901222259w7ff82eb8x3117e68aa913f218@mail.gmail.com>
	<1CB5B7C3-53DD-432B-ABE9-A8C35FB370CF@gmail.com>
	<ca3ef29a0901231953hb84b7aeied434e4ab9be4b77@mail.gmail.com>
Message-ID: <40C4AE64-4D37-49FC-93FC-87915B135E57@gmail.com>


On Jan 24, 2009, at 1:53 AM, joe wrote:

> On Fri, Jan 23, 2009 at 2:36 AM, Leonardo Santagada <santagada at gmail.com 
> > wrote:
>>
>> I don't think TraceMonkey is slower than V8 I believe the last time  
>> I looked
>> TraceMonkey was faster than V8 and it is becoming even faster at each
>> interaction.
>>
>> The way TraceMonkey works reminds me a bit of Psyco, although I  
>> might be
>> mixing it with the PyPy JIT. But talking about Psyco, why people  
>> don't go
>> help Psyco if all they want is a JIT? It is not like the idea of  
>> having a
>> JIT on Python is even new... Psyco was optimizing code even before  
>> Webkit/V8
>> existed.
>>
>>
>>> This leads me to believe that relatively simple, more general  
>>> concepts in
>>> VM design can have a bigger impact then specific, highly  
>>> complicated JIT
>>> solutions, in the context of dynamic languages that can't be  
>>> easily typed
>>> at compile time.
>>
>> I believe in the exact oposite, and TraceMonkey is probably one of  
>> the
>> proofs of that...
>>
> Well, perhaps there's validity to both sides; V8 is still pretty fast.
> I didn't know they'd improved tracemonkey that much, interesting.
> Trace trees struck me as fairly different from psyco's design (it's
> been a while since I looked at the papers for both though); Psyco
> essentially was designed to optimized specific forms of code, while
> trace trees is more generalized.

Yes I confused the design of pypy jit with the psyco one, sorry.

>>> So I've thought of a few ideas for a more (new) streamlined python  
>>> VM:
>>>
>>> * Simplify the cpython object model as much as possible, while still
>>> allowing
>>> most of the power of the current model.
>>
>> This would modify the language, so it might be interesting, but would
>> generate something which is not Python.
>
> I don't think so.  I didn't mean modify the python object model, I
> meant use a modified version of cpython's implementation of it.  The
> cpython C API isn't part of the language standard, after all, and it's
> kindof inefficient and complex, imho.

I never messed much with the cpython implementation of it, but maybe.  
If I remember correctly the idea for it is to be reasonably simple so  
that it is easy to write extension modules and incorporate it with c  
programs.


>>
>>> * Either keep referencing counting, or experiment with some of the  
>>> newer
>>> techniques such as pointer escaping. Object models that  
>>> exclusively rely
>>> on cyclic GC's have many issues and are hard to get right.
>>
>> Don't know, but a good GC is way faster than what CPython is doing  
>> already,
>> but maybe it is a good idea to explore some others perspectives on  
>> this.
>>
>
> I disagree.  I dislike GCs as a generalized solution, you don't always
> need the overhead of a full GC.  Reference counting + a GC seems like
> a better compromise, since there's less work for it to do (or rather,
> the work is spread over time in smaller amounts).

A generational GC does exactly that, it spread the work of a  
collection around, not as fine grained as refcounting but maybe  
simpler to implement if in the end you also want to get rid of the  
GIL... but yes this is still a point that is not a decided matter...

>>> * Possibly modify the bytecode to be register-based, as in  
>>> SquirrelFish.
>>> Not sure if this is worth it with python code.
>>
>> Maybe it would help a bit. I don't think it would help more than  
>> 10% tops
>> (but I am completely guessing here)
>
> Ah, 10% sounds like it would be worth it, actually.  The simple code
> generation SquirrelFish does is interesting too, it essentially
> compiles the opcodes to native code.  The code for simpler opcodes
> (and simple execution paths in the more complex ones) are inlined in
> the native code stream, while more complex opcodes are called as
> functions.

There is a tool on pypy to do this also, you should look at it (I  
always forget the name, but it would be easy to find on pypy site).


>>> * Use direct threading (which is basically optimizing switch  
>>> statements to
>>> be only one or two instructions) for the bytecode loop.
>>
>> The problem with this is (besides the error someone has already  
>> stated about
>> your phrasing) that python has really complex bytecodes, so this  
>> would also
>> only gain around 10% and it only works with compilers that accept  
>> goto
>> labels which the MSVC for example does not (maybe there are more  
>> compilers
>> that also doesn't).
>
> Python's bytecode isn't all that complex, when I looked at it.  It's
> not that much worse then Squirrelfish's original bytecode
> specification (which I need to look at again, btw, not sure what
> they're doing now).  I was kindof surprised, thought it'd be much
> worse.

There was a discussion about this on pypy-dev only a week ago, you  
might have some fun looking at the archives.

>>> * Remove string lookups for member access entirely, and replaced  
>>> with a
>>> system of unique identifyers.  The idea is you would use a hash in  
>>> the
>>> types to map a member id to an index.  Hashing ints is faster then
>>> strings,
>>> and I've even thought about experimenting with using collapsed  
>>> arrays
>>> instead
>>> of hashes.  Of course, the design would still need to support string
>>> lookups
>>> when necessary.  I've thought about this a lot, and I think you'd  
>>> need
>>> the
>>> same general idea as V8's hidden classes for this to work right  
>>> (though
>>> instead of classes, it'd just be member/unique id lookup maps).
>>
>> A form of hidden classes is already part of PyPy (but I think that  
>> only the
>> jit does this). But you can simply remove string lookups as people  
>> can
>> implement special methods to track this on the current Python. As I  
>> said
>> before I don't believe changing the semantics of python for the  
>> sake of
>> performance is even possible.
>
> Obviously you'd have to go through the effort to *not* change the
> language semantics, which would mean still allowing things like
> __get/setattr__ and __get/setattribute__, a working __dict__, etc.

Thats one of the reason I think that a JIT is probably the only answer  
for performance in python. At any time you could appen any of the  
special methods to a class...

>>> I'm not sure I'll have the time to anytime soon to prototype these  
>>> ideas,
>>> but I
>>> thought I'd kick them out there and see what people say.  Note,  
>>> I'm in no
>>> way
>>> suggesting any sort of change to the existing cpython VM (it's  
>>> way, way
>>> too
>>> early for that kind of talk).
>>
>> If you are not talking about changing CPython VM why not look at  
>> Psyco and
>> PyPy? :)
>
> I looked at Psyco.  It didn't look like it had much further potential
> to me.  It only optimizes certain situations; it's not very
> generalized.  PyPy looks interesting though.


Psyco optimizes a lot of situations, the next version I gathered is  
going to optimize generator expressions, but yes, pypy I guess is the  
long term answer to performance and python.

ps1: I thought you just forgot to send this to the list so I am  
replying to it, hope you don't mind.

ps2: I'm not part of pypy core team or anything, so my view is just,  
well, my view.
--
Leonardo Santagada
santagada at gmail.com





From ben+python at benfinney.id.au  Sun Jan 25 23:11:10 2009
From: ben+python at benfinney.id.au (Ben Finney)
Date: Mon, 26 Jan 2009 09:11:10 +1100
Subject: [Python-ideas] Draft PEP: Standard daemon process library
Message-ID: <87wscj11fl.fsf@benfinney.id.au>

Howdy all,

I am preparing a PEP, and corresponding reference implementation, for
a standard implementation of the steps needed to turn a program into a
well-behaved Unix daemon.

This message is a call for comments, support, and suggestions, prior
to submitting this PEP to the editor.


:PEP:               XXX
:Title:             Standard daemon process library
:Version:           0.1
:Last-Modified:     2009-01-26 08:44
:Author:            Ben Finney <ben+python at benfinney.id.au>
:Status:            Draft
:Type:              Standards Track
:Content-Type:      text/x-rst
:Created:           2009-01-26
:Python-Version:    3.1
:Post-History:


========
Abstract
========

Writing a program to become a well-behaved Unix daemon is somewhat
complex and tricky to get right, yet the steps are largely similar for
any daemon regardless of what else the program may need to do.

This PEP introduces a module to the Python standard library that
provides a simple interface to the task of becoming a daemon process.


..  contents::
..
    Table of Contents: 
    Abstract
    Specification
      Example usage
      Interface
      ``Daemon`` objects
      ``DaemonError`` objects
    Motivation
    Rationale
      Correct daemon behaviour
    Reference Implementation
    References
    Copyright


=============
Specification
=============

Example usage
=============

Simple example of usage::

    import daemon

    from spam import do_main_program

    this_daemon = daemon.Daemon()
    this_daemon.start()

    do_main_program()

More complex example usage::

    import os
    import grp
    import signal
    import daemon

    from spam import (
        initial_program_setup,
        do_main_program,
        program_cleanup,
        reload_program_config,
        )

    initial_program_setup()
    important_file = open('spam.data', 'w')
    interesting_file = open('eggs.data', 'w')

    this_daemon = daemon.Daemon()

    this_daemon.files_preserve = [important_file, interesting_file]

    this_daemon.working_directory = '/var/lib/foo'
    this_daemon.umask = 0o002

    mail_gid = grp.getgrnam('mail').gr_gid
    this_daemon.gid = mail_gid

    this_daemon.terminate_callback = program_cleanup
    this_daemon.reload_callback = reload_program_config
    this_daemon.reload_signals = [signal.SIGHUP, signal.SIGUSR1]

    this_daemon.start()

    do_main_program()

Interface
=========

A new module, `daemon`, is added to the standard library.

The module defines a class, `Daemon`, used to represent the settings
for a daemon process.

An exception class, `DaemonError`, is defined for exceptions raised
from the module.

``Daemon`` objects
==================

A `Daemon` instance represents the behaviour settings for the process
when it becomes a daemon. The behaviour is customised by setting
attributes on the instance, before calling the `start` method.

The following attributes are defined.

`files_preserve`
    :Default: ``None``

    List of files that should *not* be closed when starting the
    daemon. If ``None``, all open file descriptors will be closed.

    Elements of the list are file descriptors (as returned by a file
    object's `fileno()` method) or Python `file` objects. Each
    specifies a file that is not to be closed during daemon start.

`chroot_directory`
    :Default: ``None``

    Full path to a directory to set as the effective root directory of
    the process. If ``None``, specifies that the root directory is not
    to be changed.

`working_directory`
    :Default: ``'/'``

    Full path of the working directory to which the process should
    change on daemon start.

    Since a filesystem cannot be unmounted if a process has its
    current working directory on that filesystem, this should either
    be left at default or set to a directory that is a sensible ?home
    directory? for the daemon while it is running.

`lockfile_directory`
    :Default: ``'/var/run'``

    Absolute directory path to contain the daemon's lockfile. If
    ``None``, the lockfile behaviour for this daemon is skipped.

`lockfile_name`
    :Default: ``None``

    Base name of the lockfile for this daemon, without directory or
    extension. If ``None``, the name is derived from the process
    command line.

`umask`
    :Default: ``0``

    File access creation mask (?umask?) to set for the process on
    daemon start.

    Since a process inherits its umask from its parent process,
    starting the daemon will reset the umask to this value so that
    files are created by the daemon with access modes as it expects.

`ignore_signals`
    :Default: ``[signal.SIGTTOU, signal.SIGTTIN, signal.SIGTSTP]``

    List of signals that the process should ignore (by setting the
    signal action to ``signal.SIG_IGN``) on daemon start.

`terminate_signals`
    :Default: ``[signal.SIGTERM]``

    List of signals that the process should interpret as a request to
    terminate cleanly.

`terminate_callback`
    :Default: ``None``

    Callable to invoke when the process receives any of the
    `terminate_signals` signals, before then terminating the process.

`reload_signals`
    :Default: ``[signal.SIGHUP]``

    List of signals that the process should interpret as a request to
    reload runtime configuration.

`reload_callback`
    :Default: ``None``

    Callable to invoke when the process receives any of the
    `reload_signals` signals.

`uid`
    :Default: ``None``

    The user ID (?uid?) value to switch the process to on daemon start.

`gid`
    :Default: ``None``

    The group ID (?gid?) value to switch the process to on daemon start.

`prevent_core`
    :Default: ``True``

    If true, prevents the generation of core files, in order to avoid
    leaking sensitive information from daemons run as `root`.

`stdout`
    :Default: ``None``

    File-like object, open for writing (in append mode, 'w+'), that
    will be used as the new value of `sys.stdout`. If it represents an
    actual file, it should be listed in `files_preserve` to prevent it
    being closed during daemon start. If ``None``, then `sys.stdout`
    is not re-bound.

`stderr`
    :Default: ``None``

    File-like object, open for writing (in append mode, 'w+'), that
    will be used as the new value of `sys.stderr`. If it represents an
    actual file, it should be listed in `files_preserve` to prevent it
    being closed during daemon start. If ``None``, then `sys.stderr`
    is not re-bound.


The following methods are defined.

`start()`
    :Return: ``None``

    Start the daemon. This performs the following steps:

    * If the `chroot_directory` attribute is not ``None``:

      * Set the effective root directory of the process to that
        directory (via `os.chroot`). This allows running the daemon
        process inside a ?chroot gaol? as a means of limiting the
        system's exposure to rogue behaviour by the process.

    * If the `lockfile_directory` attribute is not ``None``:

      * Look in that directory for a file named '`lockfile_name`.pid';
        if it exists, raise a `DaemonError` to prevent multiple
        instances of the daemon process.

    * Close all open file descriptors, excluding those listed in the
      `files_preserve` attribute.

    * Change current working directory to the path specified by the
      `working_directory` attribute.

    * Reset the file access creation mask to the value specified by
      the `umask` attribute.

    * Detach the current process into its own process group, and
      disassociate from any controlling terminal.

      This step is skipped if it is determined to be redundant: if the
      process was started by `init`, by `initd`, or by `inetd`.

    * Set signal handlers as specified by the `ignore_signals`,
      `terminate_signals`, `terminate_callback`, `reload_signals`,
      `reload_callback` attributes.

    * If the `prevent_core` attribute is true:

      * Set the resource limits for the process to prevent any core
        dump from the process.

    * Set the process uid and gid to the true uid and gid of the
      process, to relinquish any elevated privilege.

    * If the `lockfile_directory` attribute is not ``None``:

      * Create the lockfile for this daemon in that directory, by
        writing a text line containing the current process ID (?pid?)
        to a file named '`lockfile_name`.pid'.

    * If either of the attributes `uid` or `gid` are not ``None``:

      * Set the process uid and/or gid to the specified values.

    * If either of the attributes `stdout` or `stderr` are not
      ``None``:

      * Bind the names `sys.stdout` and/or `sys.stderr` to the
        corresponding objects.

`reload()`
    :Return: ``None``

    Reload the daemon configuration. The meaning of this is entirely
    defined by the customisation of this daemon: if the
    `reload_callback` attribute is not ``None``, call that object. The
    return value is discarded.

`stop()`
    :Return: ``None``

    Stop the daemon. This performs the following steps:

    * If the `terminate_callback` attribute is not ``None``:

      * Call that object. The return value is discarded.

    * If the `lockfile_directory` attribute is not ``None``:

      * Delete the lockfile for this daemon.

    * Raise a `SystemExit` exception.


``DaemonError`` objects
=======================

The `DaemonError` class inherits from `Exception`. The module
implementation will raise an instance of `DaemonError` when an error
occurs in processing daemon behaviour.


==========
Motivation
==========

The majority of programs written to be Unix daemons either implement
behaviour very similar to that in the `Specification`_, or are
poorly-behaved daemons by the `Rationale`_.

Since these steps should be much the same in most implementations but
are very particular and easy to omit or implement incorrectly, they
are a prime target for a standard well-tested implementation in the
standard library.


=========
Rationale
=========

Correct daemon behaviour
========================

According to Stevens in [stevens]_ ?2.6, a program should perform the
following steps to become a Unix daemon process.

* Close all open file descriptors.

* Change current working directory.

* Reset the file access creation mask.

* Run in the background.

* Disassociate from process group.

* Ignore terminal I/O signals.

* Disassociate from control terminal.

* Don't reacquire a control terminal.

* Correctly handle the following circumstances:

  * Started by System V `init` process.

  * Daemon termination by ``SIGTERM`` signal.

  * Children generate ``SIGCLD`` signal.

The `daemon` package [daemon]_ lists (in its summary of features)
behaviour that should be performed when turning a program into a
well-behaved Unix daemon process. The following are appropriate for a
daemon started once the program is already running:

* Sets up the correct process context for a daemon.

* Behaves sensibly when started by `initd(8)` or `inetd(8)`.

* Revokes any suid or sgid privileges to reduce security risks in case
  daemon is incorrectly installed with special privileges.

* Prevents the generation of core files to prevent leaking sensitive
  information from daemons run as root (optional).

* Names the daemon by creating and locking a PID file to guarantee
  that only one daemon with the given name can execute at any given
  time (optional).

* Sets the user and group under which to run the daemon (optional,
  root only).

* Creates a chroot gaol (optional, root only).

* Captures the daemon's stdout and stderr and directs them to syslog
  (optional).


========================
Reference Implementation
========================

The `python-daemon` package [python-daemon]_.

As of 2009-01-26, the package is under active development and is not
yet a full implementation of this PEP.


==========
References
==========

..  [stevens]

    `Unix Network Programming`, W. Richard Stevens, 1994 Prentice
    Hall.

..  [daemon]

    The (non-Python) `daemon` package,
    `<http://www.libslack.org/daemon/>`_ by ?raf? <raf at raf.org>.

..  [python-daemon]

    The `python-daemon` package at the Python Package Index,
    `<http://pypi.python.org/pypi/python-daemon>`_. This is the
    successor to [bda.daemon]_.

..  [bda.daemon]

    The `bda.daemon` package at the Python Package Index,
    `<http://pypi.python.org/pypi/bda.daemon>`_. This is an
    implementation of [cookbook-66012]_.

..  [cookbook-66012]

    Many good ideas were contributed by the community to Python
    cookbook recipe 66012, ?Fork a daemon process on Unix?
    `<http://code.activestate.com/recipes/66012/>`_.


=========
Copyright
=========

This work is hereby placed in the public domain. To the extent that
placing a work in the public domain is not legally possible, the
copyright holder hereby grants to all recipients of this work all
rights and freedoms that would otherwise be restricted by copyright.



-- 
 \       ?Pinky, are you pondering what I'm pondering?? ?Well, I think |
  `\   so (hiccup), but Kevin Costner with an English accent?? ?_Pinky |
_o__)                                                   and The Brain_ |
Ben Finney



From tjreedy at udel.edu  Mon Jan 26 01:05:32 2009
From: tjreedy at udel.edu (Terry Reedy)
Date: Sun, 25 Jan 2009 19:05:32 -0500
Subject: [Python-ideas] Draft PEP: Standard daemon process library
In-Reply-To: <87wscj11fl.fsf@benfinney.id.au>
References: <87wscj11fl.fsf@benfinney.id.au>
Message-ID: <gliuo9$shj$1@ger.gmane.org>

Ben Finney wrote:
> Howdy all,
> 
> I am preparing a PEP, and corresponding reference implementation, for
> a standard implementation of the steps needed to turn a program into a
> well-behaved Unix daemon.

[complete spec omitted]

> ========================
> Reference Implementation
> ========================
> 
> The `python-daemon` package [python-daemon]_.
> 
> As of 2009-01-26, the package is under active development and is not
> yet a full implementation of this PEP.

If I were wanting to make a *nix daemon, I would love to have this.

As I understand things, you should finish the implementation, post it to 
PyPI, and get people to use and test it, (all of which require no 
approvals).  When stable, propose it for the stdlib along with a 
credible promise to maintain it for at least 3 years.

The question will then be whether this unix-only module is generally 
useful enough to be in the stdlib.  Many good modules are not.

tjr



From janssen at parc.com  Mon Jan 26 04:38:35 2009
From: janssen at parc.com (Bill Janssen)
Date: Sun, 25 Jan 2009 19:38:35 PST
Subject: [Python-ideas] Draft PEP: Standard daemon process library
In-Reply-To: <gliuo9$shj$1@ger.gmane.org>
References: <87wscj11fl.fsf@benfinney.id.au> <gliuo9$shj$1@ger.gmane.org>
Message-ID: <8637.1232941115@pippin.parc.xerox.com>

Terry Reedy <tjreedy at udel.edu> wrote:

> The question will then be whether this unix-only module is generally
> useful enough to be in the stdlib.  Many good modules are not.

Right.  Personally, I think it's pretty easy to make a daemon on Unix.
If this module supported both Unix and Windows, though, that would be
interesting.

Bill


From andrew-pythonideas at puzzling.org  Mon Jan 26 06:01:29 2009
From: andrew-pythonideas at puzzling.org (Andrew Bennetts)
Date: Mon, 26 Jan 2009 16:01:29 +1100
Subject: [Python-ideas] Draft PEP: Standard daemon process library
In-Reply-To: <8637.1232941115@pippin.parc.xerox.com>
References: <87wscj11fl.fsf@benfinney.id.au> <gliuo9$shj$1@ger.gmane.org>
	<8637.1232941115@pippin.parc.xerox.com>
Message-ID: <20090126050129.GB11101@steerpike.home.puzzling.org>

Bill Janssen wrote:
> Terry Reedy <tjreedy at udel.edu> wrote:
> 
> > The question will then be whether this unix-only module is generally
> > useful enough to be in the stdlib.  Many good modules are not.
> 
> Right.  Personally, I think it's pretty easy to make a daemon on Unix.

It's pretty easy, but still fiddly enough that I'd rather use a library
than worry about the details myself.

> If this module supported both Unix and Windows, though, that would be
> interesting.

That would be lovely, but even a Unix-only module would be worthwhile,
IMO.

-Andrew.



From pyideas at rebertia.com  Mon Jan 26 06:23:14 2009
From: pyideas at rebertia.com (Chris Rebert)
Date: Sun, 25 Jan 2009 21:23:14 -0800
Subject: [Python-ideas] Draft PEP: Standard daemon process library
In-Reply-To: <20090126050129.GB11101@steerpike.home.puzzling.org>
References: <87wscj11fl.fsf@benfinney.id.au> <gliuo9$shj$1@ger.gmane.org>
	<8637.1232941115@pippin.parc.xerox.com>
	<20090126050129.GB11101@steerpike.home.puzzling.org>
Message-ID: <50697b2c0901252123x7e7f2e83r72e24125bb09ec59@mail.gmail.com>

On Sun, Jan 25, 2009 at 9:01 PM, Andrew Bennetts
<andrew-pythonideas at puzzling.org> wrote:
> Bill Janssen wrote:
>> Terry Reedy <tjreedy at udel.edu> wrote:
>>
>> > The question will then be whether this unix-only module is generally
>> > useful enough to be in the stdlib.  Many good modules are not.
>>
>> Right.  Personally, I think it's pretty easy to make a daemon on Unix.
>
> It's pretty easy, but still fiddly enough that I'd rather use a library
> than worry about the details myself.

I strongly second this sentiment. It's such an oft-used recipe that it
deserves a canonical, fully correct library implementation.

>> If this module supported both Unix and Windows, though, that would be
>> interesting.
>
> That would be lovely, but even a Unix-only module would be worthwhile,
> IMO.

Though it would probably improve its chances of getting added to the std lib.

Cheers,
Chris

--
Follow the path of the Iguana...
http://rebertia.com


From phd at phd.pp.ru  Mon Jan 26 09:30:53 2009
From: phd at phd.pp.ru (Oleg Broytmann)
Date: Mon, 26 Jan 2009 11:30:53 +0300
Subject: [Python-ideas] Draft PEP: Standard daemon process library
In-Reply-To: <87wscj11fl.fsf@benfinney.id.au>
References: <87wscj11fl.fsf@benfinney.id.au>
Message-ID: <20090126083052.GB12811@phd.pp.ru>

On Mon, Jan 26, 2009 at 09:11:10AM +1100, Ben Finney wrote:
> ..  [cookbook-66012]
> 
>     Many good ideas were contributed by the community to Python
>     cookbook recipe 66012, ???Fork a daemon process on Unix???
>     `<http://code.activestate.com/recipes/66012/>`_.

   I use this one (with some local modifications) and I'm pretty happy
about it. Having such a module in the std lib would be a big plus.

Oleg.
-- 
     Oleg Broytmann            http://phd.pp.ru/            phd at phd.pp.ru
           Programmers don't die, they just GOSUB without RETURN.


From skip at pobox.com  Mon Jan 26 14:04:13 2009
From: skip at pobox.com (skip at pobox.com)
Date: Mon, 26 Jan 2009 07:04:13 -0600
Subject: [Python-ideas] Draft PEP: Standard daemon process library
In-Reply-To: <gliuo9$shj$1@ger.gmane.org>
References: <gliuo9$shj$1@ger.gmane.org>
Message-ID: <18813.46285.532475.389582@montanaro.dyndns.org>


    > I am preparing a PEP, and corresponding reference implementation, for
    > a standard implementation of the steps needed to turn a program into a
    > well-behaved Unix daemon.

I missed Ben's original post and am now reading mail offline so I can't
dredge it up, but I've used a module called daemon from PyPI.  Might well be
worth a look as an existing proof of concept if nothing else.

-- 
Skip Montanaro - skip at pobox.com - http://smontanaro.dyndns.org/


From denis.spir at free.fr  Mon Jan 26 19:42:08 2009
From: denis.spir at free.fr (spir)
Date: Mon, 26 Jan 2009 19:42:08 +0100
Subject: [Python-ideas] import imports
Message-ID: <20090126194208.47f33aad@o>

Hello,

Here are some ideas around import. I wish to receive comments and to learn whether they fit python. As I am not a highly experimented programmer, there may be some features or practices I do know know or do not know well. The following ideas are not dependant of each other, but as they all concern the import topic, I thought they could be grouped in a single message anyway.



=== import transmission ===

The module attribute __all__ allows defining names to be exported. I always use it for my own modules, and also sometimes define it in external modules, when absent, or adapt it to my needs. Then I'm happy with a secure "import *".
I find nevertheless an issue in the common case of import "transmission" from module to module, where all imported names usually have to repeated to export. This is not only ugly, imo, but also unpracticle when a single change in an name export list has to be corrected anywhere else; there can easily be a whole chain or even a 'tree' of imports.
I have found a way to deal with that, illustrated in below in a case where module M0 imports M1 and M2, which itself imports M3:

*** M3 ***
M3_names = [actual list of relevant names]
__all__ = ["M3_names"] + M3_names

*** M2 ***
from M3 import *
M2_names = [actual list of relevant /local/ names] + M3_names
__all__ = ["M2_names"] + M2_names

*** M1 ***
M1_names = [actual list of relevant names]
__all__ = ["M1_names"] + M1_names

*** M0 ***
from M1 import *
from M2 import *
M0_names = [actual list of relevant /local/ names] + M1_names + M2_names
__all__ = ["M0_names"] + M0_names

This has the advantage to avoid repetition, and allow a change in a name export list with no need of "chain correction". Still, it is only a personal trick. I wonder whether such a practice could be officialised with a kind of __imported_names__ module attribute, then understood and usable by all:

from Mx import *
__all__ = __imported_names__ + [actual list of relevant /local/ names]



=== semantics of "import *" ===

"import *" is not recommanded for good reasons, as a blind and risky practice. Now, when this is done consciously, kwowing that the imported module defines __all__, then the meaning is totally different, if not opposite. As a consequence, I am not happy at all with the present ambiguity. My proposal would be:

* Either a slightly different syntactic form, meaning "import all names defined for export". [I would like "import **", for '*' and '**' are associated in other cases and this is all about named things. But this point is no issue.] As an additional advantage, imo important, trying this kind of import from a module that does not define __all__ would raise an exception; for instance: ImportError("Module <module name> does not define any name list for export.")

* Or change the semantics of "import *" to require __all__ to be defined in the pointed module. Obviously, this raises compatibility issues that may or may not be adressed using warnings (and/or "from __future__ import import" ;-}) in a change path.



=== package module lookup ===

There is a similar issue at the package/application level: I do not know of any practice or feature that simply allows changing a filename, a directory name, or the organisation of the package's directory tree, without having then to correct all dependance information inside the concerned modules.
I would love a kind of import lookup to be based on the following assumption:
"The main/startup module resides in the package's root dir."
Then any import lookup would automatically explore local, relative, sub directories. E basta!
This would also encourage the good practice of never writing absolute imports -- which is a plague.

There are probably many good ways to implement that feature clearly and efficiently. Here is proposed an example that only aims at clarifying the feature's meaning & usefulness; but does not pretend to be good. The purpose is python to know not only which modules exist, or which modules are intended to be imported, but also where actually they are -- or should be. This does not only address import from client code but also intra-module dependances.

A file called __init__.py is used to identify a package as package and provide information about it. I propose to add a __dir__ attribute to a package, defined inside its __init__.py, that would hold a module directory (in the sense of 'index'). This could be implemented as a python dict holding a set of name:path bindings -- while actually it is pure information. It would be either a complement, or an alternative, to a package's __all__ attribute that lists the module names:

__all__ = ["main", "compute", "userinput"]
__dir__ = ["main":"/", "compute":"/tools", "userinput":"/interface/dialogs"]

Note that the pathes are *relative*. The main point is that __dir__'s value would be set by python itself when it is absent from __init__.py or when it needs beeing updated after any change in the package's tree. So that the process of module lookup would be as follows (provided I properly understand it):

* If current module is not part of a package: usual module lookup is performed.
* If it is part of a package which __dir__ is not yet defined, perform a local tree traversal starting from the module's root dir to reference the modules, then write the result into __dir__. Use it to import.
* If __dir__ is defined, try and import using it. If import fails (maybe the package's organisation has changed), while the module is referenced by __all__, update __dir__ as necessary. Then, try import again.
* When intra-package import finally fails, use PYTHONPATH to try external import, as usual.

Obviously, as long as the package's organisation does not change, a single traversal is necessary for all further imports during all further program executions. Intra-package imports may be faster, even if it is not the primary purpose of the proposal.
Actually, this seems so simple that python aware editors may even provide this automatically: meaning maintain __dir__ inside an __init__.py file. (This is close to the notion of "project" that many editors provide.)


Denis


------
la vida e estranya


From guido at python.org  Mon Jan 26 19:45:43 2009
From: guido at python.org (Guido van Rossum)
Date: Mon, 26 Jan 2009 10:45:43 -0800
Subject: [Python-ideas] Draft PEP: Standard daemon process library
In-Reply-To: <87wscj11fl.fsf@benfinney.id.au>
References: <87wscj11fl.fsf@benfinney.id.au>
Message-ID: <ca471dc20901261045i33cb689ag8b78f8c30c240ba9@mail.gmail.com>

On Sun, Jan 25, 2009 at 2:11 PM, Ben Finney <ben+python at benfinney.id.au> wrote:
> I am preparing a PEP, and corresponding reference implementation, for
> a standard implementation of the steps needed to turn a program into a
> well-behaved Unix daemon.
>
> This message is a call for comments, support, and suggestions, prior
> to submitting this PEP to the editor.

I would like to note that there are many existing packages that
provide support for daemonization (including one I wrote for Zope long
ago, which still appears to be around under the name zdaemon). Twisted
also seems to have one. I also recall hearing a talk at PyCon a few
years ago about a descendant of my zdaemon on steroids.

I have an inkling that this is one of those problems where the many
solutions that are hard to compare, because they have different APIs
or make different assumptions about the ultimate goals. The basics are
very simple (just follow the recipe from the Stevens book), but that's
a bit rough, and beyond that everyone seems to have a different idea
on what use cases need to be support and what the most flexible API is
to support all desirable use cases.

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)


From ben+python at benfinney.id.au  Mon Jan 26 23:40:42 2009
From: ben+python at benfinney.id.au (Ben Finney)
Date: Tue, 27 Jan 2009 09:40:42 +1100
Subject: [Python-ideas] Draft PEP: Standard daemon process library
References: <87wscj11fl.fsf@benfinney.id.au>
	<ca471dc20901261045i33cb689ag8b78f8c30c240ba9@mail.gmail.com>
Message-ID: <87ocxt1yj9.fsf@benfinney.id.au>

Guido van Rossum writes:

> On Sun, Jan 25, 2009 at 2:11 PM, Ben Finney wrote:
> > I am preparing a PEP, and corresponding reference implementation,
> > for a standard implementation of the steps needed to turn a
> > program into a well-behaved Unix daemon.
> 
> I would like to note that there are many existing packages that
> provide support for daemonization [?]

Good point, thanks (and thanks to others in this thread for drawing my
attention to some more that I hadn't yet found).

I will make note, in the next revision, of the existing equivalent
Python works of which I'm aware, and how their implementation differs
from the intent of this PEP. If anyone feels I'm missing any, I would
be happy to be informed.

> I have an inkling that this is one of those problems where the many
> solutions that are hard to compare, because they have different APIs
> or make different assumptions about the ultimate goals.

This suspicion could well be true.

On the other hand, there already seems to be a good amount of positive
support for the PEP, so I'd like to attempt to come to a sensible API
that covers most use cases.

Is ?python-ideas? the right forum to hash this out? Or is now the
right time to subject it to the inevitable by posting to
?comp.lang.python??

-- 
 \       ?Pinky, are you pondering what I'm pondering?? ?Well, I think |
  `\   so, Brain, but it's a miracle that this one grew back.? ?_Pinky |
_o__)                                                   and The Brain_ |
Ben Finney



From guido at python.org  Mon Jan 26 23:50:14 2009
From: guido at python.org (Guido van Rossum)
Date: Mon, 26 Jan 2009 14:50:14 -0800
Subject: [Python-ideas] Draft PEP: Standard daemon process library
In-Reply-To: <87ocxt1yj9.fsf@benfinney.id.au>
References: <87wscj11fl.fsf@benfinney.id.au>
	<ca471dc20901261045i33cb689ag8b78f8c30c240ba9@mail.gmail.com>
	<87ocxt1yj9.fsf@benfinney.id.au>
Message-ID: <ca471dc20901261450s62d9c1f5vb79c5ea106a6872a@mail.gmail.com>

On Mon, Jan 26, 2009 at 2:40 PM, Ben Finney <ben+python at benfinney.id.au> wrote:
> Guido van Rossum writes:
>
>> On Sun, Jan 25, 2009 at 2:11 PM, Ben Finney wrote:
>> > I am preparing a PEP, and corresponding reference implementation,
>> > for a standard implementation of the steps needed to turn a
>> > program into a well-behaved Unix daemon.
>>
>> I would like to note that there are many existing packages that
>> provide support for daemonization [?]
>
> Good point, thanks (and thanks to others in this thread for drawing my
> attention to some more that I hadn't yet found).
>
> I will make note, in the next revision, of the existing equivalent
> Python works of which I'm aware, and how their implementation differs
> from the intent of this PEP. If anyone feels I'm missing any, I would
> be happy to be informed.
>
>> I have an inkling that this is one of those problems where the many
>> solutions that are hard to compare, because they have different APIs
>> or make different assumptions about the ultimate goals.
>
> This suspicion could well be true.
>
> On the other hand, there already seems to be a good amount of positive
> support for the PEP, so I'd like to attempt to come to a sensible API
> that covers most use cases.
>
> Is 'python-ideas' the right forum to hash this out? Or is now the
> right time to subject it to the inevitable by posting to
> 'comp.lang.python'?

Depends on how much bikeshedding you want. I expect that you'll get
enough of that without involving c.k.py...

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)


From jseutter at gmail.com  Mon Jan 26 23:55:31 2009
From: jseutter at gmail.com (Jerry Seutter)
Date: Mon, 26 Jan 2009 15:55:31 -0700
Subject: [Python-ideas] Draft PEP: Standard daemon process library
In-Reply-To: <87ocxt1yj9.fsf@benfinney.id.au>
References: <87wscj11fl.fsf@benfinney.id.au>
	<ca471dc20901261045i33cb689ag8b78f8c30c240ba9@mail.gmail.com>
	<87ocxt1yj9.fsf@benfinney.id.au>
Message-ID: <2c8d48d70901261455l3be44eax5a997df5e7ea083f@mail.gmail.com>

It would be nice if this functionality could work on windows machine as well
as unix flavors.

I wrote the package at http://daemonize.sourceforge.net in frustration after
seeing daemons built poorly several times.  Some of the instances had been
done incorrectly even after the developer had been pointed to the
ActiveState page.  Understanding why a daemon should be written the way it
is requires a fair bit of historical knowledge that younger people
occasionally do not have.

Some of the recent grads hired at my workplace did not know what a daemon
was.  "Oh, like a web container?"

Jerry Seutter

On Mon, Jan 26, 2009 at 3:40 PM, Ben Finney
<ben+python at benfinney.id.au<ben%2Bpython at benfinney.id.au>
> wrote:

> Guido van Rossum writes:
>
> > On Sun, Jan 25, 2009 at 2:11 PM, Ben Finney wrote:
> > > I am preparing a PEP, and corresponding reference implementation,
> > > for a standard implementation of the steps needed to turn a
> > > program into a well-behaved Unix daemon.
> >
> > I would like to note that there are many existing packages that
> > provide support for daemonization [?]
>
> Good point, thanks (and thanks to others in this thread for drawing my
> attention to some more that I hadn't yet found).
>
> I will make note, in the next revision, of the existing equivalent
> Python works of which I'm aware, and how their implementation differs
> from the intent of this PEP. If anyone feels I'm missing any, I would
> be happy to be informed.
>
> > I have an inkling that this is one of those problems where the many
> > solutions that are hard to compare, because they have different APIs
> > or make different assumptions about the ultimate goals.
>
> This suspicion could well be true.
>
> On the other hand, there already seems to be a good amount of positive
> support for the PEP, so I'd like to attempt to come to a sensible API
> that covers most use cases.
>
> Is 'python-ideas' the right forum to hash this out? Or is now the
> right time to subject it to the inevitable by posting to
> 'comp.lang.python'?
>
> --
>  \       "Pinky, are you pondering what I'm pondering?" "Well, I think |
>  `\   so, Brain, but it's a miracle that this one grew back." ?_Pinky |
> _o__)                                                   and The Brain_ |
> Ben Finney
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20090126/a7fbf326/attachment.html>

From guido at python.org  Tue Jan 27 00:02:52 2009
From: guido at python.org (Guido van Rossum)
Date: Mon, 26 Jan 2009 15:02:52 -0800
Subject: [Python-ideas] Draft PEP: Standard daemon process library
In-Reply-To: <2c8d48d70901261455l3be44eax5a997df5e7ea083f@mail.gmail.com>
References: <87wscj11fl.fsf@benfinney.id.au>
	<ca471dc20901261045i33cb689ag8b78f8c30c240ba9@mail.gmail.com>
	<87ocxt1yj9.fsf@benfinney.id.au>
	<2c8d48d70901261455l3be44eax5a997df5e7ea083f@mail.gmail.com>
Message-ID: <ca471dc20901261502s31026dacpeb5f2120c2adb8de@mail.gmail.com>

Was that you in the YouTube video that was one of the first hits for
"PyCon daemon" ? That was hilarious. :-)

On Mon, Jan 26, 2009 at 2:55 PM, Jerry Seutter <jseutter at gmail.com> wrote:
> It would be nice if this functionality could work on windows machine as well
> as unix flavors.
>
> I wrote the package at http://daemonize.sourceforge.net in frustration after
> seeing daemons built poorly several times.  Some of the instances had been
> done incorrectly even after the developer had been pointed to the
> ActiveState page.  Understanding why a daemon should be written the way it
> is requires a fair bit of historical knowledge that younger people
> occasionally do not have.
> Some of the recent grads hired at my workplace did not know what a daemon
> was.  "Oh, like a web container?"
> Jerry Seutter
> On Mon, Jan 26, 2009 at 3:40 PM, Ben Finney <ben+python at benfinney.id.au>
> wrote:
>>
>> Guido van Rossum writes:
>>
>> > On Sun, Jan 25, 2009 at 2:11 PM, Ben Finney wrote:
>> > > I am preparing a PEP, and corresponding reference implementation,
>> > > for a standard implementation of the steps needed to turn a
>> > > program into a well-behaved Unix daemon.
>> >
>> > I would like to note that there are many existing packages that
>> > provide support for daemonization [?]
>>
>> Good point, thanks (and thanks to others in this thread for drawing my
>> attention to some more that I hadn't yet found).
>>
>> I will make note, in the next revision, of the existing equivalent
>> Python works of which I'm aware, and how their implementation differs
>> from the intent of this PEP. If anyone feels I'm missing any, I would
>> be happy to be informed.
>>
>> > I have an inkling that this is one of those problems where the many
>> > solutions that are hard to compare, because they have different APIs
>> > or make different assumptions about the ultimate goals.
>>
>> This suspicion could well be true.
>>
>> On the other hand, there already seems to be a good amount of positive
>> support for the PEP, so I'd like to attempt to come to a sensible API
>> that covers most use cases.
>>
>> Is 'python-ideas' the right forum to hash this out? Or is now the
>> right time to subject it to the inevitable by posting to
>> 'comp.lang.python'?
>>
>> --
>>  \       "Pinky, are you pondering what I'm pondering?" "Well, I think |
>>  `\   so, Brain, but it's a miracle that this one grew back." ?_Pinky |
>> _o__)                                                   and The Brain_ |
>> Ben Finney
>>
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> http://mail.python.org/mailman/listinfo/python-ideas
>
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>
>



-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)


From jseutter at gmail.com  Tue Jan 27 00:08:18 2009
From: jseutter at gmail.com (Jerry Seutter)
Date: Mon, 26 Jan 2009 16:08:18 -0700
Subject: [Python-ideas] Draft PEP: Standard daemon process library
In-Reply-To: <ca471dc20901261502s31026dacpeb5f2120c2adb8de@mail.gmail.com>
References: <87wscj11fl.fsf@benfinney.id.au>
	<ca471dc20901261045i33cb689ag8b78f8c30c240ba9@mail.gmail.com>
	<87ocxt1yj9.fsf@benfinney.id.au>
	<2c8d48d70901261455l3be44eax5a997df5e7ea083f@mail.gmail.com>
	<ca471dc20901261502s31026dacpeb5f2120c2adb8de@mail.gmail.com>
Message-ID: <2c8d48d70901261508w3183a3a9r8ff979f173cb2157@mail.gmail.com>

Yeah, that was me. :)

On Mon, Jan 26, 2009 at 4:02 PM, Guido van Rossum <guido at python.org> wrote:

> Was that you in the YouTube video that was one of the first hits for
> "PyCon daemon" ? That was hilarious. :-)
>
> On Mon, Jan 26, 2009 at 2:55 PM, Jerry Seutter <jseutter at gmail.com> wrote:
> > It would be nice if this functionality could work on windows machine as
> well
> > as unix flavors.
> >
> > I wrote the package at http://daemonize.sourceforge.net in frustration
> after
> > seeing daemons built poorly several times.  Some of the instances had
> been
> > done incorrectly even after the developer had been pointed to the
> > ActiveState page.  Understanding why a daemon should be written the way
> it
> > is requires a fair bit of historical knowledge that younger people
> > occasionally do not have.
> > Some of the recent grads hired at my workplace did not know what a daemon
> > was.  "Oh, like a web container?"
> > Jerry Seutter
> > On Mon, Jan 26, 2009 at 3:40 PM, Ben Finney <ben+python at benfinney.id.au<ben%2Bpython at benfinney.id.au>
> >
> > wrote:
> >>
> >> Guido van Rossum writes:
> >>
> >> > On Sun, Jan 25, 2009 at 2:11 PM, Ben Finney wrote:
> >> > > I am preparing a PEP, and corresponding reference implementation,
> >> > > for a standard implementation of the steps needed to turn a
> >> > > program into a well-behaved Unix daemon.
> >> >
> >> > I would like to note that there are many existing packages that
> >> > provide support for daemonization [?]
> >>
> >> Good point, thanks (and thanks to others in this thread for drawing my
> >> attention to some more that I hadn't yet found).
> >>
> >> I will make note, in the next revision, of the existing equivalent
> >> Python works of which I'm aware, and how their implementation differs
> >> from the intent of this PEP. If anyone feels I'm missing any, I would
> >> be happy to be informed.
> >>
> >> > I have an inkling that this is one of those problems where the many
> >> > solutions that are hard to compare, because they have different APIs
> >> > or make different assumptions about the ultimate goals.
> >>
> >> This suspicion could well be true.
> >>
> >> On the other hand, there already seems to be a good amount of positive
> >> support for the PEP, so I'd like to attempt to come to a sensible API
> >> that covers most use cases.
> >>
> >> Is 'python-ideas' the right forum to hash this out? Or is now the
> >> right time to subject it to the inevitable by posting to
> >> 'comp.lang.python'?
> >>
> >> --
> >>  \       "Pinky, are you pondering what I'm pondering?" "Well, I think |
> >>  `\   so, Brain, but it's a miracle that this one grew back." ?_Pinky |
> >> _o__)                                                   and The Brain_ |
> >> Ben Finney
> >>
> >> _______________________________________________
> >> Python-ideas mailing list
> >> Python-ideas at python.org
> >> http://mail.python.org/mailman/listinfo/python-ideas
> >
> >
> > _______________________________________________
> > Python-ideas mailing list
> > Python-ideas at python.org
> > http://mail.python.org/mailman/listinfo/python-ideas
> >
> >
>
>
>
> --
> --Guido van Rossum (home page: http://www.python.org/~guido/)
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20090126/ae91bf5a/attachment.html>

From ben+python at benfinney.id.au  Tue Jan 27 01:45:53 2009
From: ben+python at benfinney.id.au (Ben Finney)
Date: Tue, 27 Jan 2009 11:45:53 +1100
Subject: [Python-ideas] Draft PEP (version 0.2): Standard daemon process
	library
References: <87wscj11fl.fsf@benfinney.id.au>
Message-ID: <87ab9d1sqm.fsf@benfinney.id.au>

Significant changes: new section ?Other daemon implementations? with
brief descriptions of their relationship to this PEP.


:PEP:               XXX
:Title:             Standard daemon process library
:Version:           0.2
:Last-Modified:     2009-01-27 11:28
:Author:            Ben Finney <ben+python at benfinney.id.au>
:Status:            Draft
:Type:              Standards Track
:Content-Type:      text/x-rst
:Created:           2009-01-26
:Python-Version:    3.1
:Post-History:


========
Abstract
========

Writing a program to become a well-behaved Unix daemon is somewhat
complex and tricky to get right, yet the steps are largely similar for
any daemon regardless of what else the program may need to do.

This PEP introduces a module to the Python standard library that
provides a simple interface to the task of becoming a daemon process.


..  contents::
..
    Table of Contents: 
    Abstract
    Specification
      Example usage
      Interface
      ``Daemon`` objects
      ``DaemonError`` objects
    Motivation
    Rationale
      Correct daemon behaviour
    Reference Implementation
      Other daemon implementations
    References
    Copyright


=============
Specification
=============

Example usage
=============

Simple example of usage::

    import daemon

    from spam import do_main_program

    this_daemon = daemon.Daemon()
    this_daemon.start()

    do_main_program()

More complex example usage::

    import os
    import grp
    import signal
    import daemon

    from spam import (
        initial_program_setup,
        do_main_program,
        program_cleanup,
        reload_program_config,
        )

    initial_program_setup()
    important_file = open('spam.data', 'w')
    interesting_file = open('eggs.data', 'w')

    this_daemon = daemon.Daemon()

    this_daemon.files_preserve = [important_file, interesting_file]

    this_daemon.working_directory = '/var/lib/foo'
    this_daemon.umask = 0o002

    mail_gid = grp.getgrnam('mail').gr_gid
    this_daemon.gid = mail_gid

    this_daemon.terminate_callback = program_cleanup
    this_daemon.reload_callback = reload_program_config
    this_daemon.reload_signals = [signal.SIGHUP, signal.SIGUSR1]

    this_daemon.start()

    do_main_program()

Interface
=========

A new module, `daemon`, is added to the standard library.

The module defines a class, `Daemon`, used to represent the settings
for a daemon process.

An exception class, `DaemonError`, is defined for exceptions raised
from the module.

``Daemon`` objects
==================

A `Daemon` instance represents the behaviour settings for the process
when it becomes a daemon. The behaviour is customised by setting
attributes on the instance, before calling the `start` method.

The following attributes are defined.

`files_preserve`
    :Default: ``None``

    List of files that should *not* be closed when starting the
    daemon. If ``None``, all open file descriptors will be closed.

    Elements of the list are file descriptors (as returned by a file
    object's `fileno()` method) or Python `file` objects. Each
    specifies a file that is not to be closed during daemon start.

`chroot_directory`
    :Default: ``None``

    Full path to a directory to set as the effective root directory of
    the process. If ``None``, specifies that the root directory is not
    to be changed.

`working_directory`
    :Default: ``'/'``

    Full path of the working directory to which the process should
    change on daemon start.

    Since a filesystem cannot be unmounted if a process has its
    current working directory on that filesystem, this should either
    be left at default or set to a directory that is a sensible ?home
    directory? for the daemon while it is running.

`lockfile_directory`
    :Default: ``'/var/run'``

    Absolute directory path to contain the daemon's lockfile. If
    ``None``, the lockfile behaviour for this daemon is skipped.

`lockfile_name`
    :Default: ``None``

    Base name of the lockfile for this daemon, without directory or
    suffix. If ``None``, the name is derived from the process command
    line.

`umask`
    :Default: ``0``

    File access creation mask (?umask?) to set for the process on
    daemon start.

    Since a process inherits its umask from its parent process,
    starting the daemon will reset the umask to this value so that
    files are created by the daemon with access modes as it expects.

`ignore_signals`
    :Default: ``[signal.SIGTTOU, signal.SIGTTIN, signal.SIGTSTP]``

    List of signals that the process should ignore (by setting the
    signal action to ``signal.SIG_IGN``) on daemon start.

`terminate_signals`
    :Default: ``[signal.SIGTERM]``

    List of signals that the process should interpret as a request to
    terminate cleanly.

`terminate_callback`
    :Default: ``None``

    Callable to invoke when the process receives any of the
    `terminate_signals` signals, before then terminating the process.

`reload_signals`
    :Default: ``[signal.SIGHUP]``

    List of signals that the process should interpret as a request to
    reload runtime configuration.

`reload_callback`
    :Default: ``None``

    Callable to invoke when the process receives any of the
    `reload_signals` signals.

`uid`
    :Default: ``None``

    The user ID (?uid?) value to switch the process to on daemon start.

`gid`
    :Default: ``None``

    The group ID (?gid?) value to switch the process to on daemon start.

`prevent_core`
    :Default: ``True``

    If true, prevents the generation of core files, in order to avoid
    leaking sensitive information from daemons run as `root`.

`stdout`
    :Default: ``None``

    File-like object, open for writing (in append mode, 'w+'), that
    will be used as the new value of `sys.stdout`. If it represents an
    actual file, it should be listed in `files_preserve` to prevent it
    being closed during daemon start. If ``None``, then `sys.stdout`
    is not re-bound.

`stderr`
    :Default: ``None``

    File-like object, open for writing (in append mode, 'w+'), that
    will be used as the new value of `sys.stderr`. If it represents an
    actual file, it should be listed in `files_preserve` to prevent it
    being closed during daemon start. If ``None``, then `sys.stderr`
    is not re-bound.


The following methods are defined.

`start()`
    :Return: ``None``

    Start the daemon. This performs the following steps:

    * If the `chroot_directory` attribute is not ``None``:

      * Set the effective root directory of the process to that
        directory (via `os.chroot`). This allows running the daemon
        process inside a ?chroot gaol? as a means of limiting the
        system's exposure to rogue behaviour by the process.

    * If the `lockfile_directory` attribute is not ``None``:

      * Look in that directory for a file named '`lockfile_name`.pid';
        if it exists, raise a `DaemonError` to prevent multiple
        instances of the daemon process.

    * Close all open file descriptors, excluding those listed in the
      `files_preserve` attribute.

    * Change current working directory to the path specified by the
      `working_directory` attribute.

    * Reset the file access creation mask to the value specified by
      the `umask` attribute.

    * Detach the current process into its own process group, and
      disassociate from any controlling terminal.

      This step is skipped if it is determined to be redundant: if the
      process was started by `init`, by `initd`, or by `inetd`.

    * Set signal handlers as specified by the `ignore_signals`,
      `terminate_signals`, `reload_signals` attributes.

    * If the `prevent_core` attribute is true:

      * Set the resource limits for the process to prevent any core
        dump from the process.

    * Set the process uid and gid to the true uid and gid of the
      process, to relinquish any elevated privilege.

    * If the `lockfile_directory` attribute is not ``None``:

      * Create the lockfile for this daemon in that directory, by
        writing a text line containing the current process ID (?pid?)
        to a file named '`lockfile_name`.pid'.

    * If either of the attributes `uid` or `gid` are not ``None``:

      * Set the process uid and/or gid to the specified values.

    * If either of the attributes `stdout` or `stderr` are not
      ``None``:

      * Bind the names `sys.stdout` and/or `sys.stderr` to the
        corresponding objects.

`reload()`
    :Return: ``None``

    Reload the daemon configuration. The meaning of this is entirely
    defined by the customisation of this daemon: if the
    `reload_callback` attribute is not ``None``, call that object. The
    return value is discarded.

`stop()`
    :Return: ``None``

    Stop the daemon. This performs the following steps:

    * If the `terminate_callback` attribute is not ``None``:

      * Call that object. The return value is discarded.

    * If the `lockfile_directory` attribute is not ``None``:

      * Delete the lockfile for this daemon.

    * Raise a `SystemExit` exception.


``DaemonError`` objects
=======================

The `DaemonError` class inherits from `Exception`. The module
implementation will raise an instance of `DaemonError` when an error
occurs in processing daemon behaviour.


==========
Motivation
==========

The majority of programs written to be Unix daemons either implement
behaviour very similar to that in the `Specification`_, or are
poorly-behaved daemons by the `Correct daemon behaviour`_.

Since these steps should be much the same in most implementations but
are very particular and easy to omit or implement incorrectly, they
are a prime target for a standard well-tested implementation in the
standard library.


=========
Rationale
=========

Correct daemon behaviour
========================

According to Stevens in [stevens]_ ?2.6, a program should perform the
following steps to become a Unix daemon process.

* Close all open file descriptors.

* Change current working directory.

* Reset the file access creation mask.

* Run in the background.

* Disassociate from process group.

* Ignore terminal I/O signals.

* Disassociate from control terminal.

* Don't reacquire a control terminal.

* Correctly handle the following circumstances:

  * Started by System V `init` process.

  * Daemon termination by ``SIGTERM`` signal.

  * Children generate ``SIGCLD`` signal.

The `daemon` tool [slack-daemon]_ lists (in its summary of features)
behaviour that should be performed when turning a program into a
well-behaved Unix daemon process. It differs from this PEP's intent in
that it invokes a *separate* program as a daemon process. The
following features are appropriate for a daemon that starts itself
once the program is already running:

* Sets up the correct process context for a daemon.

* Behaves sensibly when started by `initd(8)` or `inetd(8)`.

* Revokes any suid or sgid privileges to reduce security risks in case
  daemon is incorrectly installed with special privileges.

* Prevents the generation of core files to prevent leaking sensitive
  information from daemons run as root (optional).

* Names the daemon by creating and locking a PID file to guarantee
  that only one daemon with the given name can execute at any given
  time (optional).

* Sets the user and group under which to run the daemon (optional,
  root only).

* Creates a chroot gaol (optional, root only).

* Captures the daemon's stdout and stderr and directs them to syslog
  (optional).


========================
Reference Implementation
========================

The `python-daemon` package [python-daemon]_.

As of 2009-01-26, the package is under active development and is not
yet a full implementation of this PEP.

Other daemon implementations
============================

Prior to this PEP, several existing third-party Python libraries or
tools implemented some of this PEP's `correct daemon behaviour`_.

The `reference implementation`_ is a fairly direct successor from the
following implementations:

* Many good ideas were contributed by the community to Python cookbook
  recipe 66012 [cookbook-66012]_.

* The `bda.daemon` library [bda.daemon]_ is an implementation (by
  Robert Niederreiter et al) of [cookbook-66012]_. It is the
  predecessor of [python-daemon]_.

Other Python daemon implementations that differ from this PEP:

* The `zdaemon` tool [zdaemon]_ was written for
  the Zope project. Like [slack-daemon]_, it differs from this
  specification because it is used to run another program as a daemon
  process.

* The Python library `daemon` [clapper-daemon]_ is (according to its
  homepage) no longer maintained. As of version 1.0.1, it implements
  the basic steps from [stevens]_.

* The `daemonize` library [seutter-daemonize]_ also implements the
  basic steps from [stevens]_.

* Twisted [twisted]_ includes, perhaps unsurprisingly, an
  implementation of a process daemonisation API that is integrated
  with the rest of the Twisted framework; it differs significantly
  from the API in this PEP.

* The Python `initd` library [dagitses-initd]_, which uses
  [clapper-daemon]_, implements an equivalent of Unix `initd(8)` for
  controlling a daemon process.


==========
References
==========

..  [stevens]

    `Unix Network Programming`, W. Richard Stevens, 1994 Prentice
    Hall.

..  [slack-daemon]

    The (non-Python) ?libslack? implementation of a `daemon` tool
    `<http://www.libslack.org/daemon/>`_ by ?raf? <raf at raf.org>.

..  [python-daemon]

    The `python-daemon` library
    `<http://pypi.python.org/pypi/python-daemon/>`_ by Ben Finney et
    al.

..  [cookbook-66012]

    Python Cookbook recipe 66012, ?Fork a daemon process on Unix?
    `<http://code.activestate.com/recipes/66012/>`_.

..  [bda.daemon]

    The `bda.daemon` library
    `<http://pypi.python.org/pypi/bda.daemon/>`_ by Robert
    Niederreiter et al.

..  [zdaemon]

    The `zdaemon` tool `<http://pypi.python.org/pypi/zdaemon/>`_ by
    Guido van Rossum et al.

..  [clapper-daemon]

    The `daemon` library `<http://pypi.python.org/pypi/daemon/>`_ by
    Brian Clapper.

..  [seutter-daemonize]

    The `daemonize` library `<http://daemonize.sourceforge.net/>`_ by
    Jerry Seutter.

..  [twisted]

    The `Twisted` application framework
    `<http://pypi.python.org/pypi/Twisted/>`_ by Glyph Lefkowitz et
    al.

..  [dagitses-initd]

    The Python `initd` library `<http://pypi.python.org/pypi/initd/>`_
    by Michael Andreas Dagitses.


=========
Copyright
=========

This work is hereby placed in the public domain. To the extent that
placing a work in the public domain is not legally possible, the
copyright holder hereby grants to all recipients of this work all
rights and freedoms that would otherwise be restricted by copyright.


-- 
 \      ?I went to a garage sale. ?How much for the garage?? ?It's not |
  `\                                        for sale.?? ?Steven Wright |
_o__)                                                                  |
Ben Finney



From ben+python at benfinney.id.au  Tue Jan 27 01:51:16 2009
From: ben+python at benfinney.id.au (Ben Finney)
Date: Tue, 27 Jan 2009 11:51:16 +1100
Subject: [Python-ideas] Draft PEP: Standard daemon process library
References: <87wscj11fl.fsf@benfinney.id.au> <gliuo9$shj$1@ger.gmane.org>
	<8637.1232941115@pippin.parc.xerox.com>
	<20090126050129.GB11101@steerpike.home.puzzling.org>
Message-ID: <8763k11shn.fsf@benfinney.id.au>

Andrew Bennetts writes:

> Bill Janssen wrote:
> > If this module supported both Unix and Windows, though, that would
> > be interesting.
> 
> That would be lovely, but even a Unix-only module would be
> worthwhile, IMO.

Jerry Seutter writes:

> It would be nice if this functionality could work on windows machine
> as well as unix flavors.


My knowledge of programming on Windows is quite limited. AIUI, the
steps required to make a Windows program into a daemon process are
markedly different from the canonical Stevens procedure, correct?

In that case, how would an implementation on Windows need to change
the API given in this PEP?

-- 
 \     ?I wish there was a knob on the TV to turn up the intelligence. |
  `\          There's a knob called ?brightness? but it doesn't work.? |
_o__)                                             ?Eugene P. Gallagher |
Ben Finney



From lie.1296 at gmail.com  Mon Jan 26 09:41:33 2009
From: lie.1296 at gmail.com (Lie Ryan)
Date: Mon, 26 Jan 2009 08:41:33 +0000 (UTC)
Subject: [Python-ideas] Draft PEP: Standard daemon process library
References: <87wscj11fl.fsf@benfinney.id.au> <gliuo9$shj$1@ger.gmane.org>
	<8637.1232941115@pippin.parc.xerox.com>
	<20090126050129.GB11101@steerpike.home.puzzling.org>
	<50697b2c0901252123x7e7f2e83r72e24125bb09ec59@mail.gmail.com>
Message-ID: <gljsvs$hgh$2@ger.gmane.org>

On Sun, 25 Jan 2009 21:23:14 -0800, Chris Rebert wrote:
>>> If this module supported both Unix and Windows, though, that would be
>>> interesting.
>>
>> That would be lovely, but even a Unix-only module would be worthwhile,
>> IMO.
> 
> Though it would probably improve its chances of getting added to the std
> lib.
> 
> Cheers,
> Chris

I think you could have a much better chance if you revise the PEP 
proposal by also considering Windows' daemon (i.e. Windows Service). Make 
it so that Unix's and Unix-like's daemon and Windows Service have as 
similar interface as possible (then create the implementation, caveat: 
"it's not going to be me").



From ben+python at benfinney.id.au  Tue Jan 27 01:58:13 2009
From: ben+python at benfinney.id.au (Ben Finney)
Date: Tue, 27 Jan 2009 11:58:13 +1100
Subject: [Python-ideas] Draft PEP: Standard daemon process library
References: <87wscj11fl.fsf@benfinney.id.au> <gliuo9$shj$1@ger.gmane.org>
Message-ID: <871vup1s62.fsf@benfinney.id.au>

Terry Reedy <tjreedy at udel.edu> writes:

> If I were wanting to make a *nix daemon, I would love to have this.

I'm glad to have many comments like this in support of this PEP.
Thanks, everyone!

> As I understand things, you should finish the implementation, post
> it to PyPI, and get people to use and test it, (all of which require
> no approvals).

Sure, but before I ?finish? the implementation, I want to reach
consensus on the suitability of the proposed API. To quote PEP 1:

    It is better to finish the specification and rationale first and
    reach consensus on it before writing code.

> When stable, propose it for the stdlib along with a credible promise
> to maintain it for at least 3 years.

My intention is to continue supporting this code after inclusion in
the standard library, yes.

-- 
 \       ?Liberty, n. One of imagination's most precious possessions.? |
  `\                   ?Ambrose Bierce, _The Devil's Dictionary_, 1906 |
_o__)                                                                  |
Ben Finney



From alan.gauld at btinternet.com  Mon Jan 26 23:38:24 2009
From: alan.gauld at btinternet.com (Alan Gauld)
Date: Mon, 26 Jan 2009 22:38:24 -0000
Subject: [Python-ideas] import imports
References: <20090126194208.47f33aad@o>
Message-ID: <glle15$7rb$1@ger.gmane.org>

"spir" <denis.spir at free.fr> wrote

> Here are some ideas around import. I wish to receive comments

> The module attribute __all__ allows defining names to be exported.
> I always use it for my own modules, and also sometimes define it in
> external modules, when absent, or adapt it to my needs.

I've never defined it in any of my modules. It encourages people
to use from X import * which is nearly always a bad idea.

> Then I'm happy with a secure "import *".

I'm not sure what you mean by a secure import *.
The only danger is that of name collisions and limiting the
exported names only helps by a tiny amount. There is still
a good chance that another module will have the same
name exported.

> ...unpracticle when a single change in an name export list has
> to be corrected anywhere else; there can easily be a whole
> chain or even a 'tree' of imports.

Exactly, thats why its a bad idea IMHO.
You can never predict what other modules your module might
be used with and to start customising other peoples modules
to your own envirohnment leads to madness in my view!

> "import *" is not recommanded for good reasons, as a
> blind and risky practice. Now, when this is done consciously,
> knowing that the imported module defines __all__, then
> the meaning is totally different, if not opposite.

Not at all. Two modules could still have conflicting names.
I don;t see how you  can avoid it. Thats why using import foo
is better.

import foo also has the advantage of forcing code to explicitly
flag where imported functions and classes are found. This
makes debugging and maintence far easier.
import * is a "bad idea" for all sorts of reasons.
Namespaces are powerful tools - lets use them.

> === package module lookup ===
>
> There is a similar issue at the package/application level:
> I do not know of any practice or feature that simply allows
> changing a filename, a directory name, or the organisation
> of the package's directory tree, without having then to
> correct all dependance information inside the concerned modules.

This is a more valid issue although if you always import both
package and subpackages directly and especially if you use
aliases to protect your code references from change

import foo as f

means even if foo changes name you only need to change
the import statement not all the references to f.x


> I would love a kind of import lookup to be based on the following 
> assumption:
> "The main/startup module resides in the package's root dir."

But most packages don;t have a startup or main module. They
are libraries that the application writer uses. Thus the app
writer provides the main or startup module not the package writer.

> This would also encourage the good practice of never
> writing absolute imports -- which is a plague.

I'm not sure what you mean by absolute imports. If you mean
giving the physical path to the file then You cannot really do
that in Python. I don't understand the comment?

> The purpose is python to know not only which modules exist,
> or which modules are intended to be imported, but also
> where actually they are -- or should be.

I thought it did that already? It looks in sys.path for modules
and when it finds them it sets the __file__ attribute if appropriate..

I got lost around here. I assume you are having problems
with modules that I've never encountered?

-- 
Alan Gauld
Author of the Learn to Program web site
http://www.freenetpages.co.uk/hp/alan.gauld 




From stephen at xemacs.org  Tue Jan 27 03:13:18 2009
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Tue, 27 Jan 2009 11:13:18 +0900
Subject: [Python-ideas] Draft PEP: Standard daemon process library
In-Reply-To: <871vup1s62.fsf@benfinney.id.au>
References: <87wscj11fl.fsf@benfinney.id.au> <gliuo9$shj$1@ger.gmane.org>
	<871vup1s62.fsf@benfinney.id.au>
Message-ID: <871vupfqdd.fsf@xemacs.org>

Ben Finney writes:

 > Sure, but before I "finish" the implementation, I want to reach
 > consensus on the suitability of the proposed API. To quote PEP 1:
 > 
 >     It is better to finish the specification and rationale first and
 >     reach consensus on it before writing code.

The people who are posting "show us the code" are mostly folks with
long experience in the PEP process.  I think you should take it as an
indication that in this case the definition of "suitable API" is an
"I'll know it when I've tried it" kind of thing.

What you might find useful is to avoid writing any code yourself, but
see how well your proposed spec fits the existing code, of which
several (potential) examples have been posted.


From janssen at parc.com  Tue Jan 27 07:46:23 2009
From: janssen at parc.com (Bill Janssen)
Date: Mon, 26 Jan 2009 22:46:23 PST
Subject: [Python-ideas] Draft PEP: Standard daemon process library
In-Reply-To: <8763k11shn.fsf@benfinney.id.au>
References: <87wscj11fl.fsf@benfinney.id.au> <gliuo9$shj$1@ger.gmane.org>
	<8637.1232941115@pippin.parc.xerox.com>
	<20090126050129.GB11101@steerpike.home.puzzling.org>
	<8763k11shn.fsf@benfinney.id.au>
Message-ID: <21108.1233038783@pippin.parc.xerox.com>

Ben Finney <ben+python at benfinney.id.au> wrote:

> My knowledge of programming on Windows is quite limited. AIUI, the
> steps required to make a Windows program into a daemon process are
> markedly different from the canonical Stevens procedure, correct?
> 
> In that case, how would an implementation on Windows need to change
> the API given in this PEP?

Yes, that's the question.  You should figure out the answer, or find some
more folks who can help you figure out the answer, and update the PEP to
reflect it.

Bill


From ben+python at benfinney.id.au  Tue Jan 27 08:53:36 2009
From: ben+python at benfinney.id.au (Ben Finney)
Date: Tue, 27 Jan 2009 18:53:36 +1100
Subject: [Python-ideas] Draft PEP: Standard daemon process library
References: <87wscj11fl.fsf@benfinney.id.au> <gliuo9$shj$1@ger.gmane.org>
	<8637.1232941115@pippin.parc.xerox.com>
	<20090126050129.GB11101@steerpike.home.puzzling.org>
	<8763k11shn.fsf@benfinney.id.au>
	<21108.1233038783@pippin.parc.xerox.com>
Message-ID: <87wschyykf.fsf@benfinney.id.au>

Bill Janssen <janssen at parc.com> writes:

> Ben Finney <ben+python at benfinney.id.au> wrote:
> 
> > [?] how would an implementation on Windows need to change the API
> > given in this PEP?
> 
> Yes, that's the question. You should figure out the answer, or find
> some more folks who can help you figure out the answer, and update
> the PEP to reflect it.

Based on what I can see in the Python Cookbook (e.g.
<URL:http://code.activestate.com/recipes/551780/>), Windows doesn't
even call it a ?daemon?, instead calling it a ?service?. The
customisation options and expected behaviour are wildly different from
what a Unix programmer would expect.

So I think that a module for creating Unix-style daemons, with API and
options geared toward making that easy, has no business also getting
involved with creating Windows services. It would result in an unduly
complicated API for no benefit to either Unix programmers or Windows
programmers.

If someone else wants to offer a separate PEP to include a
?winservice? module to the standard library, that seems a better way
forward for that use case.

-- 
 \         ?I'm beginning to think that life is just one long Yoko Ono |
  `\   album; no rhyme or reason, just a lot of incoherent shrieks and |
_o__)                                      then it's over.? ?Ian Wolff |
Ben Finney



From ben+python at benfinney.id.au  Tue Jan 27 09:19:01 2009
From: ben+python at benfinney.id.au (Ben Finney)
Date: Tue, 27 Jan 2009 19:19:01 +1100
Subject: [Python-ideas] Draft PEP (version 0.3): Standard daemon process
	library
References: <87wscj11fl.fsf@benfinney.id.au>
Message-ID: <87r62pyxe2.fsf@benfinney.id.au>

Significant changes in this version: allow all configuration options
to be set either via Daemon constructor arguments or attribute
assignment. Thanks to Mike Meyer.


:PEP:               XXX
:Title:             Standard daemon process library
:Version:           0.3
:Last-Modified:     2009-01-27 19:13
:Author:            Ben Finney <ben+python at benfinney.id.au>
:Status:            Draft
:Type:              Standards Track
:Content-Type:      text/x-rst
:Created:           2009-01-26
:Python-Version:    3.1
:Post-History:


========
Abstract
========

Writing a program to become a well-behaved Unix daemon is somewhat
complex and tricky to get right, yet the steps are largely similar for
any daemon regardless of what else the program may need to do.

This PEP introduces a module to the Python standard library that
provides a simple interface to the task of becoming a daemon process.


..  contents::
..
    Table of Contents: 
    Abstract
    Specification
      Example usage
      Interface
      ``Daemon`` objects
      ``DaemonError`` objects
    Motivation
    Rationale
      Correct daemon behaviour
    Reference Implementation
      Other daemon implementations
    References
    Copyright


=============
Specification
=============

Example usage
=============

Simple example of usage::

    import daemon

    from spam import do_main_program

    this_daemon = daemon.Daemon()
    this_daemon.start()

    do_main_program()

More complex example usage::

    import os
    import grp
    import signal
    import daemon

    from spam import (
        initial_program_setup,
        do_main_program,
        program_cleanup,
        reload_program_config,
        )

    initial_program_setup()
    important_file = open('spam.data', 'w')
    interesting_file = open('eggs.data', 'w')

    this_daemon = daemon.Daemon(
        working_directory='/var/lib/foo',
        umask=0o002,
        terminate_callback=program_cleanup,
        reload_callback=reload_program_config,
        reload_signals=[signal.SIGHUP, signal.SIGUSR1],
        )

    this_daemon.files_preserve = [important_file, interesting_file]

    mail_gid = grp.getgrnam('mail').gr_gid
    this_daemon.gid = mail_gid

    this_daemon.start()

    do_main_program()

Interface
=========

A new module, `daemon`, is added to the standard library.

The module defines a class, `Daemon`, used to represent the settings
and controls for a daemon process.

An exception class, `DaemonError`, is defined for exceptions raised
from the module.

``Daemon`` objects
==================

A `Daemon` instance represents the behaviour settings for the process
when it becomes a daemon. The behaviour is customised by setting
options on the instance, before calling the `start` method.

Each option can be passed as a keyword argument to the `Daemon`
constructor, or subsequently altered by assigning to an attribute on
the instance at any time prior to calling `start`. That is, for an
option named `wibble`, the following invocation::

    foo = daemon.Daemon(wibble=bar)
    foo.start()

is equivalent to::

    foo = daemon.Daemon()
    foo.wibble = bar
    foo.start()

The following options are defined.

`files_preserve`
    :Default: ``None``

    List of files that should *not* be closed when starting the
    daemon. If ``None``, all open file descriptors will be closed.

    Elements of the list are file descriptors (as returned by a file
    object's `fileno()` method) or Python `file` objects. Each
    specifies a file that is not to be closed during daemon start.

`chroot_directory`
    :Default: ``None``

    Full path to a directory to set as the effective root directory of
    the process. If ``None``, specifies that the root directory is not
    to be changed.

`working_directory`
    :Default: ``'/'``

    Full path of the working directory to which the process should
    change on daemon start.

    Since a filesystem cannot be unmounted if a process has its
    current working directory on that filesystem, this should either
    be left at default or set to a directory that is a sensible ?home
    directory? for the daemon while it is running.

`lockfile_directory`
    :Default: ``'/var/run'``

    Absolute directory path to contain the daemon's lockfile. If
    ``None``, the lockfile behaviour for this daemon is skipped.

`lockfile_name`
    :Default: ``None``

    Base name of the lockfile for this daemon, without directory or
    suffix. If ``None``, the name is derived from the process command
    line.

`umask`
    :Default: ``0``

    File access creation mask (?umask?) to set for the process on
    daemon start.

    Since a process inherits its umask from its parent process,
    starting the daemon will reset the umask to this value so that
    files are created by the daemon with access modes as it expects.

`ignore_signals`
    :Default: ``[signal.SIGTTOU, signal.SIGTTIN, signal.SIGTSTP]``

    List of signals that the process should ignore (by setting the
    signal action to ``signal.SIG_IGN``) on daemon start.

`terminate_signals`
    :Default: ``[signal.SIGTERM]``

    List of signals that the process should interpret as a request to
    terminate cleanly.

`terminate_callback`
    :Default: ``None``

    Callable to invoke when the process receives any of the
    `terminate_signals` signals, before then terminating the process.

`reload_signals`
    :Default: ``[signal.SIGHUP]``

    List of signals that the process should interpret as a request to
    reload runtime configuration.

`reload_callback`
    :Default: ``None``

    Callable to invoke when the process receives any of the
    `reload_signals` signals.

`uid`
    :Default: ``None``

    The user ID (?uid?) value to switch the process to on daemon start.

`gid`
    :Default: ``None``

    The group ID (?gid?) value to switch the process to on daemon start.

`prevent_core`
    :Default: ``True``

    If true, prevents the generation of core files, in order to avoid
    leaking sensitive information from daemons run as `root`.

`stdout`
    :Default: ``None``

    File-like object, open for writing (in append mode, 'w+'), that
    will be used as the new value of `sys.stdout`. If it represents an
    actual file, it should be listed in `files_preserve` to prevent it
    being closed during daemon start. If ``None``, then `sys.stdout`
    is not re-bound.

`stderr`
    :Default: ``None``

    File-like object, open for writing (in append mode, 'w+'), that
    will be used as the new value of `sys.stderr`. If it represents an
    actual file, it should be listed in `files_preserve` to prevent it
    being closed during daemon start. If ``None``, then `sys.stderr`
    is not re-bound.


The following methods are defined.

`start()`
    :Return: ``None``

    Start the daemon. This performs the following steps:

    * If the `chroot_directory` attribute is not ``None``:

      * Set the effective root directory of the process to that
        directory (via `os.chroot`). This allows running the daemon
        process inside a ?chroot gaol? as a means of limiting the
        system's exposure to rogue behaviour by the process.

    * If the `lockfile_directory` attribute is not ``None``:

      * Look in that directory for a file named '`lockfile_name`.pid';
        if it exists, raise a `DaemonError` to prevent multiple
        instances of the daemon process.

    * Close all open file descriptors, excluding those listed in the
      `files_preserve` attribute.

    * Change current working directory to the path specified by the
      `working_directory` attribute.

    * Reset the file access creation mask to the value specified by
      the `umask` attribute.

    * Detach the current process into its own process group, and
      disassociate from any controlling terminal.

      This step is skipped if it is determined to be redundant: if the
      process was started by `init`, by `initd`, or by `inetd`.

    * Set signal handlers as specified by the `ignore_signals`,
      `terminate_signals`, `reload_signals` attributes.

    * If the `prevent_core` attribute is true:

      * Set the resource limits for the process to prevent any core
        dump from the process.

    * Set the process uid and gid to the true uid and gid of the
      process, to relinquish any elevated privilege.

    * If the `lockfile_directory` attribute is not ``None``:

      * Create the lockfile for this daemon in that directory, by
        writing a text line containing the current process ID (?pid?)
        to a file named '`lockfile_name`.pid'.

    * If either of the attributes `uid` or `gid` are not ``None``:

      * Set the process uid and/or gid to the specified values.

    * If either of the attributes `stdout` or `stderr` are not
      ``None``:

      * Bind the names `sys.stdout` and/or `sys.stderr` to the
        corresponding objects.

`reload()`
    :Return: ``None``

    Reload the daemon configuration. The meaning of this is entirely
    defined by the customisation of this daemon: if the
    `reload_callback` attribute is not ``None``, call that object. The
    return value is discarded.

`stop()`
    :Return: ``None``

    Stop the daemon. This performs the following steps:

    * If the `terminate_callback` attribute is not ``None``:

      * Call that object. The return value is discarded.

    * If the `lockfile_directory` attribute is not ``None``:

      * Delete the lockfile for this daemon.

    * Raise a `SystemExit` exception.


``DaemonError`` objects
=======================

The `DaemonError` class inherits from `Exception`. The module
implementation will raise an instance of `DaemonError` when an error
occurs in processing daemon behaviour.


==========
Motivation
==========

The majority of programs written to be Unix daemons either implement
behaviour very similar to that in the `specification`_, or are
poorly-behaved daemons by the `correct daemon behaviour`_.

Since these steps should be much the same in most implementations but
are very particular and easy to omit or implement incorrectly, they
are a prime target for a standard well-tested implementation in the
standard library.


=========
Rationale
=========

Correct daemon behaviour
========================

According to Stevens in [stevens]_ ?2.6, a program should perform the
following steps to become a Unix daemon process.

* Close all open file descriptors.

* Change current working directory.

* Reset the file access creation mask.

* Run in the background.

* Disassociate from process group.

* Ignore terminal I/O signals.

* Disassociate from control terminal.

* Don't reacquire a control terminal.

* Correctly handle the following circumstances:

  * Started by System V `init` process.

  * Daemon termination by ``SIGTERM`` signal.

  * Children generate ``SIGCLD`` signal.

The `daemon` tool [slack-daemon]_ lists (in its summary of features)
behaviour that should be performed when turning a program into a
well-behaved Unix daemon process. It differs from this PEP's intent in
that it invokes a *separate* program as a daemon process. The
following features are appropriate for a daemon that starts itself
once the program is already running:

* Sets up the correct process context for a daemon.

* Behaves sensibly when started by `initd(8)` or `inetd(8)`.

* Revokes any suid or sgid privileges to reduce security risks in case
  daemon is incorrectly installed with special privileges.

* Prevents the generation of core files to prevent leaking sensitive
  information from daemons run as root (optional).

* Names the daemon by creating and locking a PID file to guarantee
  that only one daemon with the given name can execute at any given
  time (optional).

* Sets the user and group under which to run the daemon (optional,
  root only).

* Creates a chroot gaol (optional, root only).

* Captures the daemon's stdout and stderr and directs them to syslog
  (optional).


========================
Reference Implementation
========================

The `python-daemon` package [python-daemon]_.

As of 2009-01-26, the package is under active development and is not
yet a full implementation of this PEP.

Other daemon implementations
============================

Prior to this PEP, several existing third-party Python libraries or
tools implemented some of this PEP's `correct daemon behaviour`_.

The `reference implementation`_ is a fairly direct successor from the
following implementations:

* Many good ideas were contributed by the community to Python cookbook
  recipe 66012 [cookbook-66012]_.

* The `bda.daemon` library [bda.daemon]_ is an implementation of
  [cookbook-66012]_. It is the predecessor of [python-daemon]_.

Other Python daemon implementations that differ from this PEP:

* The `zdaemon` tool [zdaemon]_ was written for the Zope project. Like
  [slack-daemon]_, it differs from this specification because it is
  used to run another program as a daemon process.

* The Python library `daemon` [clapper-daemon]_ is (according to its
  homepage) no longer maintained. As of version 1.0.1, it implements
  the basic steps from [stevens]_.

* The `daemonize` library [seutter-daemonize]_ also implements the
  basic steps from [stevens]_.

* Twisted [twisted]_ includes, perhaps unsurprisingly, an
  implementation of a process daemonisation API that is integrated
  with the rest of the Twisted framework; it differs significantly
  from the API in this PEP.

* The Python `initd` library [dagitses-initd]_, which uses
  [clapper-daemon]_, implements an equivalent of Unix `initd(8)` for
  controlling a daemon process.


==========
References
==========

..  [stevens]

    `Unix Network Programming`, W. Richard Stevens, 1994 Prentice
    Hall.

..  [slack-daemon]

    The (non-Python) ?libslack? implementation of a `daemon` tool
    `<http://www.libslack.org/daemon/>`_ by ?raf? <raf at raf.org>.

..  [python-daemon]

    The `python-daemon` library
    `<http://pypi.python.org/pypi/python-daemon/>`_ by Ben Finney et
    al.

..  [cookbook-66012]

    Python Cookbook recipe 66012, ?Fork a daemon process on Unix?
    `<http://code.activestate.com/recipes/66012/>`_.

..  [bda.daemon]

    The `bda.daemon` library
    `<http://pypi.python.org/pypi/bda.daemon/>`_ by Robert
    Niederreiter et al.

..  [zdaemon]

    The `zdaemon` tool `<http://pypi.python.org/pypi/zdaemon/>`_ by
    Guido van Rossum et al.

..  [clapper-daemon]

    The `daemon` library `<http://pypi.python.org/pypi/daemon/>`_ by
    Brian Clapper.

..  [seutter-daemonize]

    The `daemonize` library `<http://daemonize.sourceforge.net/>`_ by
    Jerry Seutter.

..  [twisted]

    The `Twisted` application framework
    `<http://pypi.python.org/pypi/Twisted/>`_ by Glyph Lefkowitz et
    al.

..  [dagitses-initd]

    The Python `initd` library `<http://pypi.python.org/pypi/initd/>`_
    by Michael Andreas Dagitses.


=========
Copyright
=========

This work is hereby placed in the public domain. To the extent that
placing a work in the public domain is not legally possible, the
copyright holder hereby grants to all recipients of this work all
rights and freedoms that would otherwise be restricted by copyright.


-- 
 \            ?Program testing can be a very effective way to show the |
  `\        presence of bugs, but is hopelessly inadequate for showing |
_o__)                              their absence.? ?Edsger W. Dijkstra |
Ben Finney



From phd at phd.pp.ru  Tue Jan 27 10:00:59 2009
From: phd at phd.pp.ru (Oleg Broytmann)
Date: Tue, 27 Jan 2009 12:00:59 +0300
Subject: [Python-ideas] Draft PEP: Standard daemon process library
In-Reply-To: <87wschyykf.fsf@benfinney.id.au>
References: <87wscj11fl.fsf@benfinney.id.au> <gliuo9$shj$1@ger.gmane.org>
	<8637.1232941115@pippin.parc.xerox.com>
	<20090126050129.GB11101@steerpike.home.puzzling.org>
	<8763k11shn.fsf@benfinney.id.au>
	<21108.1233038783@pippin.parc.xerox.com>
	<87wschyykf.fsf@benfinney.id.au>
Message-ID: <20090127090058.GA302@phd.pp.ru>

On Tue, Jan 27, 2009 at 06:53:36PM +1100, Ben Finney wrote:
> Based on what I can see in the Python Cookbook (e.g.
> <URL:http://code.activestate.com/recipes/551780/>), Windows doesn't
> even call it a ???daemon???, instead calling it a ???service???. The
> customisation options and expected behaviour are wildly different from
> what a Unix programmer would expect.

   Not only it's a completely different API - it requires pywin32 which is
not in the stdlib.

> So I think that a module for creating Unix-style daemons, with API and
> options geared toward making that easy, has no business also getting
> involved with creating Windows services. It would result in an unduly
> complicated API for no benefit to either Unix programmers or Windows
> programmers.

   +1 for unix-only daemonization, -1 for making a universal unix/w32 API.

Oleg.
-- 
     Oleg Broytmann            http://phd.pp.ru/            phd at phd.pp.ru
           Programmers don't die, they just GOSUB without RETURN.


From denis.spir at free.fr  Tue Jan 27 10:39:05 2009
From: denis.spir at free.fr (spir)
Date: Tue, 27 Jan 2009 10:39:05 +0100
Subject: [Python-ideas] import imports
In-Reply-To: <glle15$7rb$1@ger.gmane.org>
References: <20090126194208.47f33aad@o>
	<glle15$7rb$1@ger.gmane.org>
Message-ID: <20090127103905.63234099@o>

[Sorry, forgot to answer one of the questions in my previous reply.]

Le Mon, 26 Jan 2009 22:38:24 -0000,
"Alan Gauld" <alan.gauld at btinternet.com> a ?crit :

> "spir" <denis.spir at free.fr> wrote
 
> > === package module lookup ===

[...]

> > This would also encourage the good practice of never
> > writing absolute imports -- which is a plague.
> 
> I'm not sure what you mean by absolute imports. If you mean
> giving the physical path to the file then You cannot really do
> that in Python. I don't understand the comment?

What I call "absolute import" happens when you give python an absolute path to look for a module (even if its your own). For sure python does not let you write an absolute path in the import instruction itself, but people do that by appending to PYTHONPATH, using 'sys.path'. Some installation instructions even explicitely tell the users to do that.
More commonly, application/package authors often rely on absolute pathes for internal imports (meaning that you cannot even chose the dir's name! not to talk about the actual location...) or external ones.

sys.path.append("path/to/dir")
import foo

Actually, for external imports, the only solutions I know are either to use environment variables, or to rely on python standard locations for packages (which are relative to python's root path, itself defined as an environment variable).
For intra-package imports, there are now facilities to write relative imports directly:
http://www.python.org/dev/peps/pep-0328/#rationale-for-relative-imports

Denis
------
la vida e estranya


From python-ideas-list at trentnelson.com  Tue Jan 27 17:06:44 2009
From: python-ideas-list at trentnelson.com (Trent Nelson)
Date: Tue, 27 Jan 2009 16:06:44 +0000
Subject: [Python-ideas] Draft PEP: Standard daemon process library
In-Reply-To: <20090127090058.GA302@phd.pp.ru>
References: <87wscj11fl.fsf@benfinney.id.au> <gliuo9$shj$1@ger.gmane.org>
	<8637.1232941115@pippin.parc.xerox.com>
	<20090126050129.GB11101@steerpike.home.puzzling.org>
	<8763k11shn.fsf@benfinney.id.au>
	<21108.1233038783@pippin.parc.xerox.com>
	<87wschyykf.fsf@benfinney.id.au> <20090127090058.GA302@phd.pp.ru>
Message-ID: <20090127160643.GC37589@wind.teleri.net>

On Tue, Jan 27, 2009 at 12:00:59PM +0300, Oleg Broytmann wrote:
> On Tue, Jan 27, 2009 at 06:53:36PM +1100, Ben Finney wrote:
> > Based on what I can see in the Python Cookbook (e.g.
> > <URL:http://code.activestate.com/recipes/551780/>), Windows doesn't
> > even call it a ???daemon???, instead calling it a ???service???. The
> > customisation options and expected behaviour are wildly different from
> > what a Unix programmer would expect.
> 
>    Not only it's a completely different API - it requires pywin32 which is
> not in the stdlib.
> 
> > So I think that a module for creating Unix-style daemons, with API and
> > options geared toward making that easy, has no business also getting
> > involved with creating Windows services. It would result in an unduly
> > complicated API for no benefit to either Unix programmers or Windows
> > programmers.
> 
>    +1 for unix-only daemonization, -1 for making a universal unix/w32 API.

    Meh, Unix-only stuff is such a pain because most developers or
    projects could cobble together Unix-only stuff themselves; the
    same can rarely be said for Windows-equivalent stuff.

    It's not hard to abstract a general service interface; Qt has
    been doing it for a long time:

    http://doc.trolltech.com/solutions/qtservice/qtservice.html

        Trent.


From phd at phd.pp.ru  Tue Jan 27 17:19:33 2009
From: phd at phd.pp.ru (Oleg Broytmann)
Date: Tue, 27 Jan 2009 19:19:33 +0300
Subject: [Python-ideas] Draft PEP: Standard daemon process library
In-Reply-To: <20090127160643.GC37589@wind.teleri.net>
References: <87wscj11fl.fsf@benfinney.id.au> <gliuo9$shj$1@ger.gmane.org>
	<8637.1232941115@pippin.parc.xerox.com>
	<20090126050129.GB11101@steerpike.home.puzzling.org>
	<8763k11shn.fsf@benfinney.id.au>
	<21108.1233038783@pippin.parc.xerox.com>
	<87wschyykf.fsf@benfinney.id.au> <20090127090058.GA302@phd.pp.ru>
	<20090127160643.GC37589@wind.teleri.net>
Message-ID: <20090127161933.GB28125@phd.pp.ru>

On Tue, Jan 27, 2009 at 04:06:44PM +0000, Trent Nelson wrote:
> On Tue, Jan 27, 2009 at 12:00:59PM +0300, Oleg Broytmann wrote:
> >    Not only it's a completely different API - it requires pywin32 which is
> > not in the stdlib.
[skip]
> >    +1 for unix-only daemonization, -1 for making a universal unix/w32 API.
> 
>     It's not hard to abstract a general service interface

   Do you propose to include pywin32 into python?

Oleg.
-- 
     Oleg Broytmann            http://phd.pp.ru/            phd at phd.pp.ru
           Programmers don't die, they just GOSUB without RETURN.


From mail at timgolden.me.uk  Tue Jan 27 17:23:57 2009
From: mail at timgolden.me.uk (Tim Golden)
Date: Tue, 27 Jan 2009 16:23:57 +0000
Subject: [Python-ideas] Draft PEP: Standard daemon process library
In-Reply-To: <20090127161933.GB28125@phd.pp.ru>
References: <87wscj11fl.fsf@benfinney.id.au>
	<gliuo9$shj$1@ger.gmane.org>	<8637.1232941115@pippin.parc.xerox.com>	<20090126050129.GB11101@steerpike.home.puzzling.org>	<8763k11shn.fsf@benfinney.id.au>	<21108.1233038783@pippin.parc.xerox.com>	<87wschyykf.fsf@benfinney.id.au>
	<20090127090058.GA302@phd.pp.ru>	<20090127160643.GC37589@wind.teleri.net>
	<20090127161933.GB28125@phd.pp.ru>
Message-ID: <497F351D.30700@timgolden.me.uk>

Oleg Broytmann wrote:
> On Tue, Jan 27, 2009 at 04:06:44PM +0000, Trent Nelson wrote:
>> On Tue, Jan 27, 2009 at 12:00:59PM +0300, Oleg Broytmann wrote:
>>>    Not only it's a completely different API - it requires pywin32 which is
>>> not in the stdlib.
> [skip]
>>>    +1 for unix-only daemonization, -1 for making a universal unix/w32 API.
>>     It's not hard to abstract a general service interface
> 
>    Do you propose to include pywin32 into python?

Ummm.. Python already has a slew of windows-specific code in
it. You could -- if it seemed suitable -- copy the pywin32
code with appropriate attribution, but you don't need the
whole thing. And you don't necessarily need it at all.

TJG


From jnoller at gmail.com  Tue Jan 27 17:30:00 2009
From: jnoller at gmail.com (Jesse Noller)
Date: Tue, 27 Jan 2009 11:30:00 -0500
Subject: [Python-ideas] Draft PEP: Standard daemon process library
In-Reply-To: <497F351D.30700@timgolden.me.uk>
References: <87wscj11fl.fsf@benfinney.id.au>
	<8637.1232941115@pippin.parc.xerox.com>
	<20090126050129.GB11101@steerpike.home.puzzling.org>
	<8763k11shn.fsf@benfinney.id.au>
	<21108.1233038783@pippin.parc.xerox.com>
	<87wschyykf.fsf@benfinney.id.au> <20090127090058.GA302@phd.pp.ru>
	<20090127160643.GC37589@wind.teleri.net>
	<20090127161933.GB28125@phd.pp.ru> <497F351D.30700@timgolden.me.uk>
Message-ID: <4222a8490901270830l275a7e55t1514e51944d86089@mail.gmail.com>

On Tue, Jan 27, 2009 at 11:23 AM, Tim Golden <mail at timgolden.me.uk> wrote:
> Oleg Broytmann wrote:
>>
>> On Tue, Jan 27, 2009 at 04:06:44PM +0000, Trent Nelson wrote:
>>>
>>> On Tue, Jan 27, 2009 at 12:00:59PM +0300, Oleg Broytmann wrote:
>>>>
>>>>   Not only it's a completely different API - it requires pywin32 which
>>>> is
>>>> not in the stdlib.
>>
>> [skip]
>>>>
>>>>   +1 for unix-only daemonization, -1 for making a universal unix/w32
>>>> API.
>>>
>>>    It's not hard to abstract a general service interface
>>
>>   Do you propose to include pywin32 into python?
>
> Ummm.. Python already has a slew of windows-specific code in
> it. You could -- if it seemed suitable -- copy the pywin32
> code with appropriate attribution, but you don't need the
> whole thing. And you don't necessarily need it at all.
>
> TJG

I think adding the windows services functionality distracts from the
reasonable goal of the PEP Ben is proposing.

-jesse


From mail at timgolden.me.uk  Tue Jan 27 18:10:49 2009
From: mail at timgolden.me.uk (Tim Golden)
Date: Tue, 27 Jan 2009 17:10:49 +0000
Subject: [Python-ideas] Draft PEP: Standard daemon process library
In-Reply-To: <4222a8490901270830l275a7e55t1514e51944d86089@mail.gmail.com>
References: <87wscj11fl.fsf@benfinney.id.au>	<8637.1232941115@pippin.parc.xerox.com>	<20090126050129.GB11101@steerpike.home.puzzling.org>	<8763k11shn.fsf@benfinney.id.au>	<21108.1233038783@pippin.parc.xerox.com>	<87wschyykf.fsf@benfinney.id.au>
	<20090127090058.GA302@phd.pp.ru>	<20090127160643.GC37589@wind.teleri.net>	<20090127161933.GB28125@phd.pp.ru>
	<497F351D.30700@timgolden.me.uk>
	<4222a8490901270830l275a7e55t1514e51944d86089@mail.gmail.com>
Message-ID: <497F4019.6080601@timgolden.me.uk>

Jesse Noller wrote:
> On Tue, Jan 27, 2009 at 11:23 AM, Tim Golden <mail at timgolden.me.uk> wrote:
>> Oleg Broytmann wrote:
>>> On Tue, Jan 27, 2009 at 04:06:44PM +0000, Trent Nelson wrote:
>>>> On Tue, Jan 27, 2009 at 12:00:59PM +0300, Oleg Broytmann wrote:
>>>>>   Not only it's a completely different API - it requires pywin32 which
>>>>> is
>>>>> not in the stdlib.
>>> [skip]
>>>>>   +1 for unix-only daemonization, -1 for making a universal unix/w32
>>>>> API.
>>>>    It's not hard to abstract a general service interface
>>>   Do you propose to include pywin32 into python?
>> Ummm.. Python already has a slew of windows-specific code in
>> it. You could -- if it seemed suitable -- copy the pywin32
>> code with appropriate attribution, but you don't need the
>> whole thing. And you don't necessarily need it at all.
>>
>> TJG
> 
> I think adding the windows services functionality distracts from the
> reasonable goal of the PEP Ben is proposing.

I'm inclined to agree with you. Or at least I think a PEP proposing
a cross-platform daemon/service framework would be quite different
from the one Ben is authoring. 

I was really just commenting on Oleg's suggestion that anything 
making use of win32 code needed to use pywin32.


TJG


From phd at phd.pp.ru  Tue Jan 27 18:27:17 2009
From: phd at phd.pp.ru (Oleg Broytmann)
Date: Tue, 27 Jan 2009 20:27:17 +0300
Subject: [Python-ideas] Draft PEP: Standard daemon process library
In-Reply-To: <497F4019.6080601@timgolden.me.uk>
References: <20090126050129.GB11101@steerpike.home.puzzling.org>
	<8763k11shn.fsf@benfinney.id.au>
	<21108.1233038783@pippin.parc.xerox.com>
	<87wschyykf.fsf@benfinney.id.au> <20090127090058.GA302@phd.pp.ru>
	<20090127160643.GC37589@wind.teleri.net>
	<20090127161933.GB28125@phd.pp.ru> <497F351D.30700@timgolden.me.uk>
	<4222a8490901270830l275a7e55t1514e51944d86089@mail.gmail.com>
	<497F4019.6080601@timgolden.me.uk>
Message-ID: <20090127172717.GA9501@phd.pp.ru>

On Tue, Jan 27, 2009 at 05:10:49PM +0000, Tim Golden wrote:
> I was really just commenting on Oleg's suggestion that anything making 
> use of win32 code needed to use pywin32.

   I haven't said that. I have said (not exactly in these words) "If you
want to add an implementation of w32 services to the stdlib you must (or at
least should) to add pywin32 to the stdlib."

Oleg.
-- 
     Oleg Broytmann            http://phd.pp.ru/            phd at phd.pp.ru
           Programmers don't die, they just GOSUB without RETURN.


From ben+python at benfinney.id.au  Tue Jan 27 22:18:51 2009
From: ben+python at benfinney.id.au (Ben Finney)
Date: Wed, 28 Jan 2009 08:18:51 +1100
Subject: [Python-ideas] Draft PEP: Standard daemon process library
References: <87wscj11fl.fsf@benfinney.id.au>
	<8637.1232941115@pippin.parc.xerox.com>
	<20090126050129.GB11101@steerpike.home.puzzling.org>
	<8763k11shn.fsf@benfinney.id.au>
	<21108.1233038783@pippin.parc.xerox.com>
	<87wschyykf.fsf@benfinney.id.au> <20090127090058.GA302@phd.pp.ru>
	<20090127160643.GC37589@wind.teleri.net>
	<20090127161933.GB28125@phd.pp.ru> <497F351D.30700@timgolden.me.uk>
	<4222a8490901270830l275a7e55t1514e51944d86089@mail.gmail.com>
Message-ID: <87eiyozbus.fsf@benfinney.id.au>

Jesse Noller <jnoller at gmail.com> writes:

> I think adding the windows services functionality distracts from the
> reasonable goal of the PEP Ben is proposing.

Thanks, this is my position also. I have documented this now, and it
will appear in the next version of the PEP.

-- 
 \        ?The problem with television is that the people must sit and |
  `\    keep their eyes glued on a screen: the average American family |
_o__)                 hasn't time for it.? ?_The New York Times_, 1939 |
Ben Finney



From python-ideas-list at trentnelson.com  Wed Jan 28 02:29:07 2009
From: python-ideas-list at trentnelson.com (Trent Nelson)
Date: Wed, 28 Jan 2009 01:29:07 +0000
Subject: [Python-ideas] Draft PEP: Standard daemon process library
In-Reply-To: <87eiyozbus.fsf@benfinney.id.au>
References: <20090126050129.GB11101@steerpike.home.puzzling.org>
	<8763k11shn.fsf@benfinney.id.au>
	<21108.1233038783@pippin.parc.xerox.com>
	<87wschyykf.fsf@benfinney.id.au> <20090127090058.GA302@phd.pp.ru>
	<20090127160643.GC37589@wind.teleri.net>
	<20090127161933.GB28125@phd.pp.ru> <497F351D.30700@timgolden.me.uk>
	<4222a8490901270830l275a7e55t1514e51944d86089@mail.gmail.com>
	<87eiyozbus.fsf@benfinney.id.au>
Message-ID: <20090128012906.GH57568@wind.teleri.net>

On Wed, Jan 28, 2009 at 08:18:51AM +1100, Ben Finney wrote:
> Jesse Noller <jnoller at gmail.com> writes:
> 
> > I think adding the windows services functionality distracts from the
> > reasonable goal of the PEP Ben is proposing.
> 
> Thanks, this is my position also. I have documented this now, and it
> will appear in the next version of the PEP.

    I disagree.  Partly because I'm in a bit of a devil's advocate mood
    at the moment.  But mostly because I know what will happen: the PEP
    gets approved, the daemon module gets written, and, because it's so
    useful, uptake is immediate, people start writing code that uses it.
    
    Sweet.

    Or is it?  Don't get me wrong, I *heart* Unix.  But I get paid to
    develop in Windows.  Because frankly, that's where the big bucks are
    and it's an extremely satisfying transition to jettison all emotional
    reasoning and just work for whoever is willing to pay you the most.
    I highly recommend it.  But for now, I digress.

    So, back to your Unix-only daemon module.  It's good.  Great, in
    fact.  All sorts of projects start using it.  Subversion uses it
    to write some funky persistent daemon that speeds up hooks by
    8000% percent.  Well that's just super!  I'd like to use that for
    one of my clients, but, argh, no, wait, I can't, it's trying to
    use some daemon module that Google tells me is Unix-only.

    After poking around the source, I'm perplexed.  It's not doing
    anything uniquely Unix-xy, just your normal, run-of-the-mill 
    start/stop type stuff.  Why on earth, I think to myself, is this
    great 3rd party Python utility using a Unix-only daemon module,
    thus requiring me to hack it to bits in order for it to run on 
    Windows, wasting my time and my client's time.

    Because it was there.


        Trent.


From jnoller at gmail.com  Wed Jan 28 02:49:52 2009
From: jnoller at gmail.com (Jesse Noller)
Date: Tue, 27 Jan 2009 20:49:52 -0500
Subject: [Python-ideas] Draft PEP: Standard daemon process library
In-Reply-To: <20090128012906.GH57568@wind.teleri.net>
References: <20090126050129.GB11101@steerpike.home.puzzling.org>
	<21108.1233038783@pippin.parc.xerox.com>
	<87wschyykf.fsf@benfinney.id.au> <20090127090058.GA302@phd.pp.ru>
	<20090127160643.GC37589@wind.teleri.net>
	<20090127161933.GB28125@phd.pp.ru> <497F351D.30700@timgolden.me.uk>
	<4222a8490901270830l275a7e55t1514e51944d86089@mail.gmail.com>
	<87eiyozbus.fsf@benfinney.id.au>
	<20090128012906.GH57568@wind.teleri.net>
Message-ID: <4222a8490901271749g1a9fc616lc6e6da2987be6040@mail.gmail.com>

On Tue, Jan 27, 2009 at 8:29 PM, Trent Nelson
<python-ideas-list at trentnelson.com> wrote:
> On Wed, Jan 28, 2009 at 08:18:51AM +1100, Ben Finney wrote:
>> Jesse Noller <jnoller at gmail.com> writes:
>>
>> > I think adding the windows services functionality distracts from the
>> > reasonable goal of the PEP Ben is proposing.
>>
>> Thanks, this is my position also. I have documented this now, and it
>> will appear in the next version of the PEP.
>
>    I disagree.  Partly because I'm in a bit of a devil's advocate mood
>    at the moment.  But mostly because I know what will happen: the PEP
>    gets approved, the daemon module gets written, and, because it's so
>    useful, uptake is immediate, people start writing code that uses it.

>    Or is it?  Don't get me wrong, I *heart* Unix.  But I get paid to
>    develop in Windows.  Because frankly, that's where the big bucks are
>    and it's an extremely satisfying transition to jettison all emotional
>    reasoning and just work for whoever is willing to pay you the most.
>    I highly recommend it.  But for now, I digress.
>

Don't worry; I think most of us would gladly work for money. In fact,
I would hazard most of us like money a lot.

>    So, back to your Unix-only daemon module.  It's good.  Great, in
>    fact.  All sorts of projects start using it.  Subversion uses it
>    to write some funky persistent daemon that speeds up hooks by
>    8000% percent.  Well that's just super!  I'd like to use that for
>    one of my clients, but, argh, no, wait, I can't, it's trying to
>    use some daemon module that Google tells me is Unix-only.
>
>    After poking around the source, I'm perplexed.  It's not doing
>    anything uniquely Unix-xy, just your normal, run-of-the-mill
>    start/stop type stuff.  Why on earth, I think to myself, is this
>    great 3rd party Python utility using a Unix-only daemon module,
>    thus requiring me to hack it to bits in order for it to run on
>    Windows, wasting my time and my client's time.

Windows Services != Unix daemons. Admittedly, I haven't had to do it
in awhile, but from what I remember, windows services are nowhere near
the same type of beast as a unix daemon, and trying to make a unified
daemon library that glosses over the differences might cause seizures
and habitual smoking.

A unix-only daemon - and then a complimenting windows-only service
library - makes sense. Ideally, they would have close to the same API,
but I don't think log jamming all of the vagaries of both into one
beast makes sense. Then again, my windows-ability is out of date and
rusty.

-jesse


From ben+python at benfinney.id.au  Wed Jan 28 03:15:41 2009
From: ben+python at benfinney.id.au (Ben Finney)
Date: Wed, 28 Jan 2009 13:15:41 +1100
Subject: [Python-ideas] Draft PEP: Standard daemon process library
References: <20090126050129.GB11101@steerpike.home.puzzling.org>
	<8763k11shn.fsf@benfinney.id.au>
	<21108.1233038783@pippin.parc.xerox.com>
	<87wschyykf.fsf@benfinney.id.au> <20090127090058.GA302@phd.pp.ru>
	<20090127160643.GC37589@wind.teleri.net>
	<20090127161933.GB28125@phd.pp.ru> <497F351D.30700@timgolden.me.uk>
	<4222a8490901270830l275a7e55t1514e51944d86089@mail.gmail.com>
	<87eiyozbus.fsf@benfinney.id.au>
	<20090128012906.GH57568@wind.teleri.net>
Message-ID: <87ab9cyy42.fsf@benfinney.id.au>

Trent Nelson writes:

> On Wed, Jan 28, 2009 at 08:18:51AM +1100, Ben Finney wrote:
> > Jesse Noller writes:
> > 
> > > I think adding the windows services functionality distracts from
> > > the reasonable goal of the PEP Ben is proposing.
> > 
> > Thanks, this is my position also.
> 
> I disagree. Partly because I'm in a bit of a devil's advocate mood
> at the moment. But mostly because I know what will happen: [?]

I respectfully suggest that this isn't true, because your hypothetical
situation continues:

> After poking around the source, I'm perplexed. It's not doing
> anything uniquely Unix-xy,

Your disagreement seems to rest on this assertion, and I don't see how
your hypothetical described can be true. Do you really see the
described behaviour in the current draft PEP as ?not doing anything
uniquely Unix-y??

> just your normal, run-of-the-mill start/stop type stuff.

As far as I can tell, beyond trivial use cases, there is both wide
discrepancy and very little overlap in ?start/stop type stuff?
between a Unix daemon and an MS Windows service. At least, none of the
(admittedly few) instructions on MS Windows services I've seen
demonstrate otherwise.

Someone who thinks otherwise is welcome to attempt a PEP that brings
all that behaviour under a single banner *and* design the API to be
both flexible enough to be useful in many use cases and simple enough
to be worth learning instead of doing it manually. That person will
not be me.

-- 
 \      ?Contentment is a pearl of great price, and whosoever procures |
  `\        it at the expense of ten thousand desires makes a wise and |
_o__)                                      happy purchase.? ?J. Balguy |
Ben Finney



From python-ideas-list at trentnelson.com  Wed Jan 28 03:21:45 2009
From: python-ideas-list at trentnelson.com (Trent Nelson)
Date: Wed, 28 Jan 2009 02:21:45 +0000
Subject: [Python-ideas] Draft PEP: Standard daemon process library
In-Reply-To: <4222a8490901271749g1a9fc616lc6e6da2987be6040@mail.gmail.com>
References: <21108.1233038783@pippin.parc.xerox.com>
	<87wschyykf.fsf@benfinney.id.au> <20090127090058.GA302@phd.pp.ru>
	<20090127160643.GC37589@wind.teleri.net>
	<20090127161933.GB28125@phd.pp.ru> <497F351D.30700@timgolden.me.uk>
	<4222a8490901270830l275a7e55t1514e51944d86089@mail.gmail.com>
	<87eiyozbus.fsf@benfinney.id.au>
	<20090128012906.GH57568@wind.teleri.net>
	<4222a8490901271749g1a9fc616lc6e6da2987be6040@mail.gmail.com>
Message-ID: <20090128022145.GC64950@wind.teleri.net>

On Tue, Jan 27, 2009 at 08:49:52PM -0500, Jesse Noller wrote:
> On Tue, Jan 27, 2009 at 8:29 PM, Trent Nelson
> <python-ideas-list at trentnelson.com> wrote:
> > On Wed, Jan 28, 2009 at 08:18:51AM +1100, Ben Finney wrote:
> >> Jesse Noller <jnoller at gmail.com> writes:

> Windows Services != Unix daemons. Admittedly, I haven't had to do it
> in awhile, but from what I remember, windows services are nowhere near
> the same type of beast as a unix daemon, and trying to make a unified
> daemon library that glosses over the differences might cause seizures
> and habitual smoking.
> 
> A unix-only daemon - and then a complimenting windows-only service
> library - makes sense. Ideally, they would have close to the same API,
> but I don't think log jamming all of the vagaries of both into one
> beast makes sense. Then again, my windows-ability is out of date and
> rusty.

    See, I disagree with that as well.  Take a look at what Qt did with
    their QtService class:

        http://doc.trolltech.com/solutions/qtservice/qtservice.html

    The Trolls are particularly good at abstracting wildly different
    Unix/Windows-isms into a single, usable, unified interface.  The
    key is not thinking in terms of writing a Windows service or a
    Unix daemon, but in terms of providing an abstraction that makes
    sense to implement as a daemon on Unix and a service on Windows.

    I'd like to hear arguments for why that QtService interface is
    perceived as being deficient.  If the argument is simply that the
    PEP author is not familiar with Windows, then I'm sure those of
    us that are would much rather help out with a cross-platform
    solution than have yet another avenue for people to write Python
    code that doesn't work on the most widely deployed platform in 
    the Universe.*

        Trent.


[*] Citation needed.


From ben+python at benfinney.id.au  Wed Jan 28 03:48:46 2009
From: ben+python at benfinney.id.au (Ben Finney)
Date: Wed, 28 Jan 2009 13:48:46 +1100
Subject: [Python-ideas] Draft PEP: Standard daemon process library
References: <21108.1233038783@pippin.parc.xerox.com>
	<87wschyykf.fsf@benfinney.id.au> <20090127090058.GA302@phd.pp.ru>
	<20090127160643.GC37589@wind.teleri.net>
	<20090127161933.GB28125@phd.pp.ru> <497F351D.30700@timgolden.me.uk>
	<4222a8490901270830l275a7e55t1514e51944d86089@mail.gmail.com>
	<87eiyozbus.fsf@benfinney.id.au>
	<20090128012906.GH57568@wind.teleri.net>
	<4222a8490901271749g1a9fc616lc6e6da2987be6040@mail.gmail.com>
	<20090128022145.GC64950@wind.teleri.net>
Message-ID: <8763k0ywkx.fsf@benfinney.id.au>

Trent Nelson writes:

> On Tue, Jan 27, 2009 at 08:49:52PM -0500, Jesse Noller wrote:
> > Windows Services != Unix daemons. Admittedly, I haven't had to do
> > it in awhile, but from what I remember, windows services are
> > nowhere near the same type of beast as a unix daemon, and trying
> > to make a unified daemon library that glosses over the differences
> > might cause seizures and habitual smoking.
> > 
> > A unix-only daemon - and then a complimenting windows-only service
> > library - makes sense. Ideally, they would have close to the same
> > API, but I don't think log jamming all of the vagaries of both
> > into one beast makes sense. Then again, my windows-ability is out
> > of date and rusty.
> 
> See, I disagree with that as well.  Take a look at what Qt did with
> their QtService class:
> 
>     http://doc.trolltech.com/solutions/qtservice/qtservice.html

Thanks for mentioning that again; I saw another reference to it
earlier, but didn't respond.

> I'd like to hear arguments for why that QtService interface is
> perceived as being deficient.

Off the top of my head:

The QtService model seems to be ?start an additional process and
maintain it at arms length, with a bunch of communication channels
into and out of it?. Contrariwise, this PEP addresses the use case of
?turn the *current* program into a daemon process, so that everything
further this program does is running inside that process?.

It's too big, as a result of the above difference in focus. When
writing a Unix daemon program, I don't want the great majority of what
that interface offers.

It misses a lot, again as a result of the mismatch in purpose. After
setting up a QtService, I need to continue chattering with it to find
out what's going on inside it, and to do things like change its state
or log a message. The current PEP, on the other hand, doesn't have the
model of managing a separate process; the daemon is the *current*
program, so one can just do things like output or state changes as one
normally would.

In short: It addresses a different problem to what I'm addressing with
this PEP.

-- 
 \         ?If nature has made any one thing less susceptible than all |
  `\    others of exclusive property, it is the action of the thinking |
_o__)                          power called an idea? ?Thomas Jefferson |
Ben Finney



From python-ideas-list at trentnelson.com  Wed Jan 28 04:03:53 2009
From: python-ideas-list at trentnelson.com (Trent Nelson)
Date: Wed, 28 Jan 2009 03:03:53 +0000
Subject: [Python-ideas] Draft PEP: Standard daemon process library
In-Reply-To: <87ab9cyy42.fsf@benfinney.id.au>
References: <21108.1233038783@pippin.parc.xerox.com>
	<87wschyykf.fsf@benfinney.id.au> <20090127090058.GA302@phd.pp.ru>
	<20090127160643.GC37589@wind.teleri.net>
	<20090127161933.GB28125@phd.pp.ru> <497F351D.30700@timgolden.me.uk>
	<4222a8490901270830l275a7e55t1514e51944d86089@mail.gmail.com>
	<87eiyozbus.fsf@benfinney.id.au>
	<20090128012906.GH57568@wind.teleri.net>
	<87ab9cyy42.fsf@benfinney.id.au>
Message-ID: <20090128030351.GD64950@wind.teleri.net>

On Wed, Jan 28, 2009 at 01:15:41PM +1100, Ben Finney wrote:
> Trent Nelson writes:
> 
> > On Wed, Jan 28, 2009 at 08:18:51AM +1100, Ben Finney wrote:
> > > Jesse Noller writes:
> > > 
> > > > I think adding the windows services functionality distracts from
> > > > the reasonable goal of the PEP Ben is proposing.
> > > 
> > > Thanks, this is my position also.
> > 
> > I disagree. Partly because I'm in a bit of a devil's advocate mood
> > at the moment. But mostly because I know what will happen: [???]
> 
> I respectfully suggest that this isn't true, because your hypothetical
> situation continues:
> 
> > After poking around the source, I'm perplexed. It's not doing
> > anything uniquely Unix-xy,
> 
> Your disagreement seems to rest on this assertion, and I don't see how
> your hypothetical described can be true. Do you really see the
> described behaviour in the current draft PEP as ???not doing anything
> uniquely Unix-y????

    No, the PEP is nothing but Unix-isms, that's my point.  The 3rd
    party code I'm hypothetically wading through has been coded to
    that interface, but semantically, it's not doing anything unique
    to Unix.

> `prevent_core`
>    :Default: ``True``
>
>        If true, prevents the generation of core files, in order to
>        avoid leaking sensitive information from daemons run as `root`.

    I was going to suggest service.set_secure(True); but then I wondered
    what is this actually achieving?  Won't the resulting .core will be
    owned by root?  

>`lockfile_directory`
>    :Default: ``'/var/run'``
>
>    Absolute directory path to contain the daemon's lockfile. If
>    ``None``, the lockfile behaviour for this daemon is skipped.
>
>`lockfile_name`
>    :Default: ``None``
>
>    Base name of the lockfile for this daemon, without directory or
>    suffix. If ``None``, the name is derived from the process command
>    line.

    All you're doing is ensuring multiple instances of the daemon don't
    run.  On Windows, you'd use a mutex, on Unix, sure, lockfile makes
    sense.  A cross-platform interface wouldn't bother exposing this
    implementation detail.

> > just your normal, run-of-the-mill start/stop type stuff.
> 
> As far as I can tell, beyond trivial use cases, there is both wide
> discrepancy and very little overlap in ???start/stop type stuff???
> between a Unix daemon and an MS Windows service. At least, none of the
> (admittedly few) instructions on MS Windows services I've seen
> demonstrate otherwise.

    Semantically the overlap is almost identical, which is why I'm
    interested in hearing why the QtService-type approach couldn't
    be leveraged here.  (The implementation details differ wildly,
    which is what you're referring to.)
 
> Someone who thinks otherwise is welcome to attempt a PEP that brings
> all that behaviour under a single banner *and* design the API to be
> both flexible enough to be useful in many use cases and simple enough
> to be worth learning instead of doing it manually. That person will
> not be me.

    class Service(object):
        EventType = namedtuple(
            Success = 1,
            Error = 2,
            Warning = 3,
            Information = 4,
        )
        StartupType = namedtuple(
            Auto = 0,
            Manual = 1
        )
        def __init__(self, name, desc=None, startup_type=StartupType.Auto)
            self.name = name
            self.description = desc

        def parse_arguments(args): pass
        def is_running(self): pass
        def is_installed(self): pass
        def install(self): pass
        def uninstall(self): pass
        def report_event(self, msg, event_type, id, category, data): pass

        def exec(args): pass
        def terminate(): pass
        def request_pause(): pass
        def request_resume(): pass
        def send_command(code): pass

        def run(args): raise NotImplementedError()

        def start(): pass
        def stop(): pass
        def pause(): pass
        def resume(): pass
        
    }:>


        Trent.


From ben+python at benfinney.id.au  Wed Jan 28 04:24:27 2009
From: ben+python at benfinney.id.au (Ben Finney)
Date: Wed, 28 Jan 2009 14:24:27 +1100
Subject: [Python-ideas] Draft PEP: Standard daemon process library
References: <21108.1233038783@pippin.parc.xerox.com>
	<87wschyykf.fsf@benfinney.id.au> <20090127090058.GA302@phd.pp.ru>
	<20090127160643.GC37589@wind.teleri.net>
	<20090127161933.GB28125@phd.pp.ru> <497F351D.30700@timgolden.me.uk>
	<4222a8490901270830l275a7e55t1514e51944d86089@mail.gmail.com>
	<87eiyozbus.fsf@benfinney.id.au>
	<20090128012906.GH57568@wind.teleri.net>
	<87ab9cyy42.fsf@benfinney.id.au>
	<20090128030351.GD64950@wind.teleri.net>
Message-ID: <871vuoyuxg.fsf@benfinney.id.au>

(I'm only addressing commentary on the current PEP here. Comparisons
to MS Windows services have been addressed in other messages.)

Trent Nelson writes:

> On Wed, Jan 28, 2009 at 01:15:41PM +1100, Ben Finney wrote:
> > `prevent_core`
> >    :Default: ``True``
> >
> >        If true, prevents the generation of core files, in order to
> >        avoid leaking sensitive information from daemons run as
> >        `root`.
> 
>     I was going to suggest service.set_secure(True)

That name implies something emphatically not true: that a program's
security is a binary toggle, and it can be turned on or off.

> but then I wondered what is this actually achieving? Won't the
> resulting .core will be owned by root?

Yes. Which is less secure than never creating the core file at all.

> >`lockfile_directory`
> >    :Default: ``'/var/run'``
> >
> >    Absolute directory path to contain the daemon's lockfile. If
> >    ``None``, the lockfile behaviour for this daemon is skipped.
> >
> >`lockfile_name`
> >    :Default: ``None``
> >
> >    Base name of the lockfile for this daemon, without directory or
> >    suffix. If ``None``, the name is derived from the process command
> >    line.
> 
>     All you're doing is ensuring multiple instances of the daemon
>     don't run.

Also allowing the location and name of the lock file to be customised.

>  On Windows, you'd use a mutex, on Unix, sure, lockfile makes sense.
>  A cross-platform interface wouldn't bother exposing this
>  implementation detail.

The details are exposed precisely so that they can be customised by
programs that need it. This is common, for example, to allow for
differing conventions or standards of filesystem layout.

-- 
 \         ?Pinky, are you pondering what I'm pondering?? ?I think so, |
  `\    Brain, but I find scratching just makes it worse.? ?_Pinky and |
_o__)                                                       The Brain_ |
Ben Finney



From python-ideas-list at trentnelson.com  Wed Jan 28 04:38:42 2009
From: python-ideas-list at trentnelson.com (Trent Nelson)
Date: Wed, 28 Jan 2009 03:38:42 +0000
Subject: [Python-ideas] Draft PEP: Standard daemon process library
In-Reply-To: <871vuoyuxg.fsf@benfinney.id.au>
References: <20090127090058.GA302@phd.pp.ru>
	<20090127160643.GC37589@wind.teleri.net>
	<20090127161933.GB28125@phd.pp.ru> <497F351D.30700@timgolden.me.uk>
	<4222a8490901270830l275a7e55t1514e51944d86089@mail.gmail.com>
	<87eiyozbus.fsf@benfinney.id.au>
	<20090128012906.GH57568@wind.teleri.net>
	<87ab9cyy42.fsf@benfinney.id.au>
	<20090128030351.GD64950@wind.teleri.net>
	<871vuoyuxg.fsf@benfinney.id.au>
Message-ID: <20090128033842.GF64950@wind.teleri.net>

On Wed, Jan 28, 2009 at 02:24:27PM +1100, Ben Finney wrote:
> (I'm only addressing commentary on the current PEP here. Comparisons
> to MS Windows services have been addressed in other messages.)

    (I've run out of steam trying to argue the virtues of cross-
     platform modules, so I'm only commenting on Ben's commentary
     on my commentary.)

> Trent Nelson writes:
> > On Wed, Jan 28, 2009 at 01:15:41PM +1100, Ben Finney wrote:
> > > `prevent_core`
> > >    :Default: ``True``
> > >
> > >        If true, prevents the generation of core files, in order to
> > >        avoid leaking sensitive information from daemons run as
> > >        `root`.
> > 
> >     I was going to suggest service.set_secure(True)
> 
> That name implies something emphatically not true: that a program's
> security is a binary toggle, and it can be turned on or off.
> 
> > but then I wondered what is this actually achieving? Won't the
> > resulting .core will be owned by root?
> 
> Yes. Which is less secure than never creating the core file at all.

    I must be missing something.  This seems like a false sense of
    security.  The core file is owned by root.  Someone will need
    root access in order to read it.  If someone has root access, 
    and has malevolent intentions, it's game over, .core file or
    not.


        Trent.


From python-ideas-list at trentnelson.com  Wed Jan 28 04:41:05 2009
From: python-ideas-list at trentnelson.com (Trent Nelson)
Date: Wed, 28 Jan 2009 03:41:05 +0000
Subject: [Python-ideas]  Draft PEP: Standard daemon process library]
Message-ID: <20090128034105.GG64950@wind.teleri.net>

On Wed, Jan 28, 2009 at 01:48:46PM +1100, Ben Finney wrote:
> Trent Nelson writes:
> 
> > On Tue, Jan 27, 2009 at 08:49:52PM -0500, Jesse Noller wrote:
> > > Windows Services != Unix daemons. Admittedly, I haven't had to do
> > > it in awhile, but from what I remember, windows services are
> > > nowhere near the same type of beast as a unix daemon, and trying
> > > to make a unified daemon library that glosses over the differences
> > > might cause seizures and habitual smoking.
> > > 
> > > A unix-only daemon - and then a complimenting windows-only service
> > > library - makes sense. Ideally, they would have close to the same
> > > API, but I don't think log jamming all of the vagaries of both
> > > into one beast makes sense. Then again, my windows-ability is out
> > > of date and rusty.
> > 
> > See, I disagree with that as well.  Take a look at what Qt did with
> > their QtService class:
> > 
> >     http://doc.trolltech.com/solutions/qtservice/qtservice.html
> 
> Thanks for mentioning that again; I saw another reference to it
> earlier, but didn't respond.
> 
> > I'd like to hear arguments for why that QtService interface is
> > perceived as being deficient.
> 
> Off the top of my head:
> 
> The QtService model seems to be ???start an additional process and
> maintain it at arms length, with a bunch of communication channels
> into and out of it???. Contrariwise, this PEP addresses the use case of
> ???turn the *current* program into a daemon process, so that everything
> further this program does is running inside that process???.

    What are you left with at the end of the day?  Something you can 
    start, stop, query status, and log out and have it still run.  How
    you got there (daemonising the current process) surely isn't as
    important as where you ended, especially if it nukes any chance
    of cross-platformness?

> It's too big, as a result of the above difference in focus. When
> writing a Unix daemon program, I don't want the great majority of what
> that interface offers.

    So I can't persuade you from shifting focus from "When writing a
    Unix daemon program, I ...", to "When writing a cross-platform
    daemon/service, I ..."?

    Sidebar: this definition on the Qt page caught my attention:

        "A Windows service or Unix daemon (a "service"), 
         is a program that runs regardless of whether a 
         user is logged in or not."

    If you're vehemently against the notion of a cross-platform
    implementation that utilised Windows services; what about a
    Windows implementation that mimicked the Unix behaviour as 
    best it could?  That is, upon daemonisation, the Windows 
    process would change process group or whatever such that
    it would keep running after you log out, until killed.

    Not pretty, but it avoids all the problems of Unix-only.

> It misses a lot, again as a result of the mismatch in purpose. After
> setting up a QtService, I need to continue chattering with it to find
> out what's going on inside it, and to do things like change its state
> or log a message. The current PEP, on the other hand, doesn't have the
> model of managing a separate process; the daemon is the *current*
> program, so one can just do things like output or state changes as one
> normally would.
> 
> In short: It addresses a different problem to what I'm addressing with
> this PEP.

    Indeed it does, but that's 'cause I'm trying to change your focus
    from writing a Unix-only daemon to writing a cross-platform service
    that just happens to be implemented as a daemon on Unix and as a
    service on Windows ;-)

    Sure, the interface wouldn't be the same as if it were Unix-only,
    but that's what cross-platformness is all about, isn't it?


        Trent.



From ben+python at benfinney.id.au  Wed Jan 28 05:21:42 2009
From: ben+python at benfinney.id.au (Ben Finney)
Date: Wed, 28 Jan 2009 15:21:42 +1100
Subject: [Python-ideas] Draft PEP: Standard daemon process library]
References: <20090128034105.GG64950@wind.teleri.net>
Message-ID: <87iqo0xdpl.fsf@benfinney.id.au>

Trent Nelson writes:

>     What are you left with at the end of the day? Something you can
>     start, stop, query status, and log out and have it still run.

This presupposes a point of view (?you?) that is someone *running*
the program. For that point of view, this PEP is irrelevant.

The audience for the current PEP, on the other hand, is a programmer
*writing* such a program, who sees the API as addressing just one of
the many and varied things their program needs to do.

>     How you got there (daemonising the current process) surely isn't
>     as important as where you ended

I'm specifying an API, and trying to make it so people will want to
use it for the purpose it's designed for. So yes, to the extent that
?how you got there? includes programming to an API, I think it is
very important in this context.

>     So I can't persuade you from shifting focus from "When writing a
>     Unix daemon program, I ...", to "When writing a cross-platform
>     daemon/service, I ..."?

I could potentially be persuaded, but the ?run an additional process
as a service? model is definitely not a problem I need solved. Given
that, I'm not likely to be a good champion for it.

Meanwhile, the focus of this PEP *does* meet a need shared by many
(?daemonise the current running program?), and I see value in
keeping it focussed on that.

>     Sidebar: this definition on the Qt page caught my attention:
> 
>         "A Windows service or Unix daemon (a "service"), 
>          is a program that runs regardless of whether a 
>          user is logged in or not."

I noticed that too, and it didn't strike any chord with me. It seems
targeted to people who want to write an MS Windows service that also
happens to run on Unix. Those people doubtless exist, and deserve to
find solutions to their challenges; but I'm not one of them.

> If you're vehemently against the notion of a cross-platform
> implementation that utilised Windows services

You are arguing against a position I've never taken. I repeat: if
someone else wants what you're describing, more power to them, and I
don't intend to get in their way. But it's not going to be me that
writes that PEP nor the implementation for it.

-- 
 \      ?If you fell down yesterday, stand up today.? ?_The Anatomy of |
  `\                                   Frustration_, H. G. Wells, 1936 |
_o__)                                                                  |
Ben Finney



From stephen at xemacs.org  Wed Jan 28 05:28:10 2009
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Wed, 28 Jan 2009 13:28:10 +0900
Subject: [Python-ideas] Draft PEP: Standard daemon process library
In-Reply-To: <871vuoyuxg.fsf@benfinney.id.au>
References: <21108.1233038783@pippin.parc.xerox.com>
	<87wschyykf.fsf@benfinney.id.au> <20090127090058.GA302@phd.pp.ru>
	<20090127160643.GC37589@wind.teleri.net>
	<20090127161933.GB28125@phd.pp.ru> <497F351D.30700@timgolden.me.uk>
	<4222a8490901270830l275a7e55t1514e51944d86089@mail.gmail.com>
	<87eiyozbus.fsf@benfinney.id.au>
	<20090128012906.GH57568@wind.teleri.net>
	<87ab9cyy42.fsf@benfinney.id.au>
	<20090128030351.GD64950@wind.teleri.net>
	<871vuoyuxg.fsf@benfinney.id.au>
Message-ID: <87k58gdpgl.fsf@xemacs.org>

Ben Finney writes:

 > The details [of the lockfiles implementing a mutex] are exposed
 > precisely so that they can be customised by programs that need
 > it. This is common, for example, to allow for differing conventions
 > or standards of filesystem layout.

Sure, but the question being asked is why should users of the daemon
process library need to worry about that, separately from other users
of lockfiles or mutexes?  If they need to worry about it only because
there's no satisfactory module implementing it widely available, then
that could be done in a subsidiary module that handles OS-specific
details, and optionally exposes OS-specific special features (a la os
and os.posix).  That would also make refactoring (close to) trivial
when such a module did become widely available.

It's obvious you don't want to go in that direction, which is IMHO
good and sufficient reason for you not to do so.  But as it stands IMO
it's a weakness in the PEP, even if Unix-only is otherwise acceptable
for the stdlib.


From solipsis at pitrou.net  Thu Jan 29 01:57:42 2009
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 29 Jan 2009 00:57:42 +0000 (UTC)
Subject: [Python-ideas] Draft PEP: Standard daemon process library
References: <20090126050129.GB11101@steerpike.home.puzzling.org>
	<21108.1233038783@pippin.parc.xerox.com>
	<87wschyykf.fsf@benfinney.id.au> <20090127090058.GA302@phd.pp.ru>
	<20090127160643.GC37589@wind.teleri.net>
	<20090127161933.GB28125@phd.pp.ru> <497F351D.30700@timgolden.me.uk>
	<4222a8490901270830l275a7e55t1514e51944d86089@mail.gmail.com>
	<87eiyozbus.fsf@benfinney.id.au>
	<20090128012906.GH57568@wind.teleri.net>
	<4222a8490901271749g1a9fc616lc6e6da2987be6040@mail.gmail.com>
Message-ID: <loom.20090129T005452-128@post.gmane.org>

Jesse Noller <jnoller at ...> writes:
> Windows Services != Unix daemons. Admittedly, I haven't had to do it
> in awhile, but from what I remember, windows services are nowhere near
> the same type of beast as a unix daemon, and trying to make a unified
> daemon library that glosses over the differences might cause seizures
> and habitual smoking.

That would be like a portable library which would allow to write multiprocess
programs by glossing over the differences between Unix and Windows processes...
Wouldn't it?





From jnoller at gmail.com  Thu Jan 29 02:12:58 2009
From: jnoller at gmail.com (Jesse Noller)
Date: Wed, 28 Jan 2009 20:12:58 -0500
Subject: [Python-ideas] Draft PEP: Standard daemon process library
In-Reply-To: <loom.20090129T005452-128@post.gmane.org>
References: <20090126050129.GB11101@steerpike.home.puzzling.org>
	<21108.1233038783@pippin.parc.xerox.com>
	<87wschyykf.fsf@benfinney.id.au> <20090127090058.GA302@phd.pp.ru>
	<20090127160643.GC37589@wind.teleri.net>
	<20090127161933.GB28125@phd.pp.ru> <497F351D.30700@timgolden.me.uk>
	<4222a8490901270830l275a7e55t1514e51944d86089@mail.gmail.com>
	<87eiyozbus.fsf@benfinney.id.au>
	<20090128012906.GH57568@wind.teleri.net>
	<4222a8490901271749g1a9fc616lc6e6da2987be6040@mail.gmail.com>
	<loom.20090129T005452-128@post.gmane.org>
Message-ID: <5B769383-4274-4EE4-A252-F101FBCB1794@gmail.com>



On Jan 28, 2009, at 7:57 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:

> Jesse Noller <jnoller at ...> writes:
>> Windows Services != Unix daemons. Admittedly, I haven't had to do it
>> in awhile, but from what I remember, windows services are nowhere  
>> near
>> the same type of beast as a unix daemon, and trying to make a unified
>> daemon library that glosses over the differences might cause seizures
>> and habitual smoking.
>
> That would be like a portable library which would allow to write  
> multiprocess
> programs by glossing over the differences between Unix and Windows  
> processes...
> Wouldn't it?
>

Touch? good sir, touch? 

From jnoller at gmail.com  Thu Jan 29 02:48:50 2009
From: jnoller at gmail.com (Jesse Noller)
Date: Wed, 28 Jan 2009 20:48:50 -0500
Subject: [Python-ideas] Draft PEP: Standard daemon process library
In-Reply-To: <loom.20090129T005452-128@post.gmane.org>
References: <20090126050129.GB11101@steerpike.home.puzzling.org>
	<20090127090058.GA302@phd.pp.ru>
	<20090127160643.GC37589@wind.teleri.net>
	<20090127161933.GB28125@phd.pp.ru> <497F351D.30700@timgolden.me.uk>
	<4222a8490901270830l275a7e55t1514e51944d86089@mail.gmail.com>
	<87eiyozbus.fsf@benfinney.id.au>
	<20090128012906.GH57568@wind.teleri.net>
	<4222a8490901271749g1a9fc616lc6e6da2987be6040@mail.gmail.com>
	<loom.20090129T005452-128@post.gmane.org>
Message-ID: <4222a8490901281748w7f4b0b9cw662982ebdac01774@mail.gmail.com>

On Wed, Jan 28, 2009 at 7:57 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> Jesse Noller <jnoller at ...> writes:
>> Windows Services != Unix daemons. Admittedly, I haven't had to do it
>> in awhile, but from what I remember, windows services are nowhere near
>> the same type of beast as a unix daemon, and trying to make a unified
>> daemon library that glosses over the differences might cause seizures
>> and habitual smoking.
>
> That would be like a portable library which would allow to write multiprocess
> programs by glossing over the differences between Unix and Windows processes...
> Wouldn't it?
>

Since Antoine called me to the mat on this one, and I've been chewing
on this more today - I think the cross-platformedness (is that a
word?) idea is a good one, even if the Qt design doesn't necessarily
apply here completely, having a single abstraction to both (but
different modules under that abstraction based on os) is attractive
(as Stephen pointed out).

Heck, a daemon for OS/X is going to need it's own underlying
implementation as well: the proper way to install a daemon/service is
via launchd [1] which I had to monkey with both for the OS/X buildbot
I have, and for an application today.

jesse

[1] http://developer.apple.com/documentation/Darwin/Reference/ManPages/man8/launchd.8.html#//apple_ref/doc/man/8/launchd


From andrew-pythonideas at puzzling.org  Thu Jan 29 02:57:03 2009
From: andrew-pythonideas at puzzling.org (Andrew Bennetts)
Date: Thu, 29 Jan 2009 12:57:03 +1100
Subject: [Python-ideas] Draft PEP: Standard daemon process library
In-Reply-To: <4222a8490901281748w7f4b0b9cw662982ebdac01774@mail.gmail.com>
References: <20090127090058.GA302@phd.pp.ru>
	<20090127160643.GC37589@wind.teleri.net>
	<20090127161933.GB28125@phd.pp.ru> <497F351D.30700@timgolden.me.uk>
	<4222a8490901270830l275a7e55t1514e51944d86089@mail.gmail.com>
	<87eiyozbus.fsf@benfinney.id.au>
	<20090128012906.GH57568@wind.teleri.net>
	<4222a8490901271749g1a9fc616lc6e6da2987be6040@mail.gmail.com>
	<loom.20090129T005452-128@post.gmane.org>
	<4222a8490901281748w7f4b0b9cw662982ebdac01774@mail.gmail.com>
Message-ID: <20090129015703.GG9167@steerpike.home.puzzling.org>

Jesse Noller wrote:
> 
> Since Antoine called me to the mat on this one, and I've been chewing
> on this more today - I think the cross-platformedness (is that a
> word?) idea is a good one, even if the Qt design doesn't necessarily
> apply here completely, having a single abstraction to both (but
> different modules under that abstraction based on os) is attractive
> (as Stephen pointed out).

Perhaps such an abstraction should be built on top of Ben Finney's
proposed unix daemon module, rather than built in?

-Andrew.



From jnoller at gmail.com  Thu Jan 29 03:06:38 2009
From: jnoller at gmail.com (Jesse Noller)
Date: Wed, 28 Jan 2009 21:06:38 -0500
Subject: [Python-ideas] Draft PEP: Standard daemon process library
In-Reply-To: <20090129015703.GG9167@steerpike.home.puzzling.org>
References: <20090127090058.GA302@phd.pp.ru> <20090127161933.GB28125@phd.pp.ru>
	<497F351D.30700@timgolden.me.uk>
	<4222a8490901270830l275a7e55t1514e51944d86089@mail.gmail.com>
	<87eiyozbus.fsf@benfinney.id.au>
	<20090128012906.GH57568@wind.teleri.net>
	<4222a8490901271749g1a9fc616lc6e6da2987be6040@mail.gmail.com>
	<loom.20090129T005452-128@post.gmane.org>
	<4222a8490901281748w7f4b0b9cw662982ebdac01774@mail.gmail.com>
	<20090129015703.GG9167@steerpike.home.puzzling.org>
Message-ID: <4222a8490901281806s34c259f7xbbbfaf62b67ad5da@mail.gmail.com>

On Wed, Jan 28, 2009 at 8:57 PM, Andrew Bennetts
<andrew-pythonideas at puzzling.org> wrote:
> Jesse Noller wrote:
>>
>> Since Antoine called me to the mat on this one, and I've been chewing
>> on this more today - I think the cross-platformedness (is that a
>> word?) idea is a good one, even if the Qt design doesn't necessarily
>> apply here completely, having a single abstraction to both (but
>> different modules under that abstraction based on os) is attractive
>> (as Stephen pointed out).
>
> Perhaps such an abstraction should be built on top of Ben Finney's
> proposed unix daemon module, rather than built in?
>
> -Andrew.
>
>

Perhaps; but it makes sense to nest it under the abstraction -
daemons.unixdaemon/winservice/osxdaemon


From ben+python at benfinney.id.au  Thu Jan 29 03:21:00 2009
From: ben+python at benfinney.id.au (Ben Finney)
Date: Thu, 29 Jan 2009 13:21:00 +1100
Subject: [Python-ideas] Draft PEP: Standard daemon process library
References: <20090127090058.GA302@phd.pp.ru>
	<20090127160643.GC37589@wind.teleri.net>
	<20090127161933.GB28125@phd.pp.ru> <497F351D.30700@timgolden.me.uk>
	<4222a8490901270830l275a7e55t1514e51944d86089@mail.gmail.com>
	<87eiyozbus.fsf@benfinney.id.au>
	<20090128012906.GH57568@wind.teleri.net>
	<4222a8490901271749g1a9fc616lc6e6da2987be6040@mail.gmail.com>
	<loom.20090129T005452-128@post.gmane.org>
	<4222a8490901281748w7f4b0b9cw662982ebdac01774@mail.gmail.com>
	<20090129015703.GG9167@steerpike.home.puzzling.org>
Message-ID: <87zlhax377.fsf@benfinney.id.au>

Andrew Bennetts writes:

> Jesse Noller wrote:
> > 
> > having a single abstraction to both

Both? What is the referent here? (I couldn't find any in the context
provided.)

> > (but different modules under that abstraction based on os) is
> > attractive (as Stephen pointed out).
> 
> Perhaps such an abstraction should be built on top of Ben Finney's
> proposed unix daemon module, rather than built in?

+1.

It seems Jesse and I have been simultaneously masticating. Having now
chewed on this issue overnight, I can't see any reason why, given the
existence of an implementation of the current PEP, that it couldn't be
used to implement some *other* PEP dealing with a separate-process,
service-based model.

That I can't think of any reason why not is likely to mean that there
are half a dozen reasons I'm unaware of :-)

-- 
 \         ?I went to the museum where they had all the heads and arms |
  `\      from the statues that are in all the other museums.? ?Steven |
_o__)                                                           Wright |
Ben Finney



From ben+python at benfinney.id.au  Thu Jan 29 03:35:21 2009
From: ben+python at benfinney.id.au (Ben Finney)
Date: Thu, 29 Jan 2009 13:35:21 +1100
Subject: [Python-ideas] Draft PEP: Standard daemon process library
References: <20090127090058.GA302@phd.pp.ru> <20090127161933.GB28125@phd.pp.ru>
	<497F351D.30700@timgolden.me.uk>
	<4222a8490901270830l275a7e55t1514e51944d86089@mail.gmail.com>
	<87eiyozbus.fsf@benfinney.id.au>
	<20090128012906.GH57568@wind.teleri.net>
	<4222a8490901271749g1a9fc616lc6e6da2987be6040@mail.gmail.com>
	<loom.20090129T005452-128@post.gmane.org>
	<4222a8490901281748w7f4b0b9cw662982ebdac01774@mail.gmail.com>
	<20090129015703.GG9167@steerpike.home.puzzling.org>
	<4222a8490901281806s34c259f7xbbbfaf62b67ad5da@mail.gmail.com>
Message-ID: <87vdryx2ja.fsf@benfinney.id.au>

Jesse Noller <jnoller at gmail.com> writes:

> Perhaps; but it makes sense to nest it under the abstraction -
> daemons.unixdaemon/winservice/osxdaemon

I really don't think it's helpful to conflate this ?service? model
with the much simpler concept of a Unix daemon.

To implement a service on Unix, it's most logical to use a daemon; but
the reverse is definitely not true (which is a large part of my
resistance to expanding the current PEP to conflate the two concepts).

Instead, I think that people who want a ?service? would be better
served by a top-level module named ?service?, to preserve that concept
as separate from the concept of a daemon.

In other words, my understanding is: ?import service? should get an
API (so far in no PEP that I know of) for a distinct purpose, that of
implementing an MS Windows-style service, which uses whatever
OS-specific implementation makes sense.

?import daemon? gets a totally different API for a different purpose,
that of turning the current running program into a daemon, which
AFAICT only makes a whole lot of sense on Unix and is what I'm
proposing in this PEP.

It makes much more sense for ?daemon? to stay simple and continue to
mean the Unix-specific concept of ?daemon?, without necessarily
obstructing whoever wants to implement ?service?, than for ?daemon? in
Python to mean something other than the Unix meaning.

-- 
 \     ?No matter how far down the wrong road you've gone, turn back.? |
  `\                                                  ?Turkish proverb |
_o__)                                                                  |
Ben Finney



From ben+python at benfinney.id.au  Thu Jan 29 03:42:44 2009
From: ben+python at benfinney.id.au (Ben Finney)
Date: Thu, 29 Jan 2009 13:42:44 +1100
Subject: [Python-ideas] Draft PEP: Standard daemon process library
References: <21108.1233038783@pippin.parc.xerox.com>
	<87wschyykf.fsf@benfinney.id.au> <20090127090058.GA302@phd.pp.ru>
	<20090127160643.GC37589@wind.teleri.net>
	<20090127161933.GB28125@phd.pp.ru> <497F351D.30700@timgolden.me.uk>
	<4222a8490901270830l275a7e55t1514e51944d86089@mail.gmail.com>
	<87eiyozbus.fsf@benfinney.id.au>
	<20090128012906.GH57568@wind.teleri.net>
	<87ab9cyy42.fsf@benfinney.id.au>
	<20090128030351.GD64950@wind.teleri.net>
	<871vuoyuxg.fsf@benfinney.id.au> <87k58gdpgl.fsf@xemacs.org>
Message-ID: <87r62mx26z.fsf@benfinney.id.au>

"Stephen J. Turnbull" writes:

> the question being asked is why should users of the daemon process
> library need to worry about that, separately from other users of
> lockfiles or mutexes?

Perhaps my mistake is in calling it a ?lockfile?. It's actually a
rather specific Unix-native concept: the ?pidfile?, a combination
sentinel, lockfile, and primitive text interface. It isn't even used
much like other lockfiles (in that it wouldn't make sense, for example
to wrap a context manager around it).

I might return the terminology in the PEP back to ?pidfile? to be
clear.

To answer the question, the location of a pidfile warrants specific
mention (separate from discussion of lockfiles) in at least two OS
standards that I'm aware of, as well as howtos and other discussions.
For that reason, it's better to have a simple way to tell a program
the desired location and name of its pidfile in particular.

-- 
 \      ?[I]t is impossible for anyone to begin to learn that which he |
  `\                thinks he already knows.? ?Epictetus, _Discourses_ |
_o__)                                                                  |
Ben Finney



From janssen at parc.com  Thu Jan 29 06:14:09 2009
From: janssen at parc.com (Bill Janssen)
Date: Wed, 28 Jan 2009 21:14:09 PST
Subject: [Python-ideas] Draft PEP: Standard daemon process library
In-Reply-To: <87vdryx2ja.fsf@benfinney.id.au>
References: <20090127090058.GA302@phd.pp.ru> <20090127161933.GB28125@phd.pp.ru>
	<497F351D.30700@timgolden.me.uk>
	<4222a8490901270830l275a7e55t1514e51944d86089@mail.gmail.com>
	<87eiyozbus.fsf@benfinney.id.au>
	<20090128012906.GH57568@wind.teleri.net>
	<4222a8490901271749g1a9fc616lc6e6da2987be6040@mail.gmail.com>
	<loom.20090129T005452-128@post.gmane.org>
	<4222a8490901281748w7f4b0b9cw662982ebdac01774@mail.gmail.com>
	<20090129015703.GG9167@steerpike.home.puzzling.org>
	<4222a8490901281806s34c259f7xbbbfaf62b67ad5da@mail.gmail.com>
	<87vdryx2ja.fsf@benfinney.id.au>
Message-ID: <42979.1233206049@pippin.parc.xerox.com>

Ben Finney <ben+python at benfinney.id.au> wrote:

> ?import daemon? gets a totally different API for a different purpose,
> that of turning the current running program into a daemon, which
> AFAICT only makes a whole lot of sense on Unix and is what I'm
> proposing in this PEP.

I think that's an implementation detail.  Unix daemons and Windows
services seem more similar than different.  The details about how they
are established/started are different, but that's what a good API covers
over.  That would be worth designing.

There are a number of Unix-only modules for this purpose already, aren't
there?  I see at least three of them on PyPI, including yours, Ben.
What's the point of a new one?  And what's the point of adding a new
Unix-only module to the stdlib?  Aren't we trying to remove
platform-only modules?  Didn't we remove the Mac modules in 3.x?  And
isn't win32all a separate distribution?

Bill


From ben+python at benfinney.id.au  Thu Jan 29 07:00:19 2009
From: ben+python at benfinney.id.au (Ben Finney)
Date: Thu, 29 Jan 2009 17:00:19 +1100
Subject: [Python-ideas] Draft PEP: Standard daemon process library
References: <20090127090058.GA302@phd.pp.ru> <20090127161933.GB28125@phd.pp.ru>
	<497F351D.30700@timgolden.me.uk>
	<4222a8490901270830l275a7e55t1514e51944d86089@mail.gmail.com>
	<87eiyozbus.fsf@benfinney.id.au>
	<20090128012906.GH57568@wind.teleri.net>
	<4222a8490901271749g1a9fc616lc6e6da2987be6040@mail.gmail.com>
	<loom.20090129T005452-128@post.gmane.org>
	<4222a8490901281748w7f4b0b9cw662982ebdac01774@mail.gmail.com>
	<20090129015703.GG9167@steerpike.home.puzzling.org>
	<4222a8490901281806s34c259f7xbbbfaf62b67ad5da@mail.gmail.com>
	<87vdryx2ja.fsf@benfinney.id.au>
	<42979.1233206049@pippin.parc.xerox.com>
Message-ID: <87mydawt1o.fsf@benfinney.id.au>

Bill Janssen <janssen at parc.com> writes:

> There are a number of Unix-only modules for this purpose already,
> aren't there? I see at least three of them on PyPI, including yours,
> Ben.

Yes, none of which meet the criteria listed in the PEP for
?well-behaved daemon?. They're useful, to be sure; but they're lacking
when judged against what a well-behaved Unix daemon should be doing.

> What's the point of a new one?

Perhaps I've mistakenly given the impression that I'm re-inventing the
wheel; sorry for that false impression if so.

To implement this PEP fully, I've taken the existing implementation
that was closest to the requirements (?bda.daemon?), and based this
work on altering it to conform with this PEP. I'm in active
correspondence with the author of ?bda.daemon?.

> And what's the point of adding a new Unix-only module to the stdlib?

To have this common, fairly standardised but tricky, use case
implemented once and easily available to any Python programmer,
without them needing to know the fine details of what's required to
make it work internally.

Having multiple partial implementations in PyPI and the Cookbook leads
only to the situation where a prospective user has to either know
intimate details of this rather fiddly task to judge each of them, or
make a choice arbitrarily.

-- 
 \      ?I don't know half of you half as well as I should like, and I |
  `\   like less than half of you half as well as you deserve.? ?Bilbo |
_o__)                                                          Baggins |
Ben Finney



From ben+python at benfinney.id.au  Thu Jan 29 07:03:05 2009
From: ben+python at benfinney.id.au (Ben Finney)
Date: Thu, 29 Jan 2009 17:03:05 +1100
Subject: [Python-ideas] Draft PEP (version 0.4): Standard daemon process
	library
References: <87wscj11fl.fsf@benfinney.id.au>
Message-ID: <87iqnywsx2.fsf@benfinney.id.au>

Significant changes in this version: New section distinguishing a Unix
daemon from a ?service?; clarify that the API makes the *current*
program into a daemon process; consistently discuss PID files (instead
of lock files).


:PEP:               XXX
:Title:             Standard daemon process library
:Version:           0.4
:Last-Modified:     2009-01-29 16:32
:Author:            Ben Finney <ben+python at benfinney.id.au>
:Status:            Draft
:Type:              Standards Track
:Content-Type:      text/x-rst
:Created:           2009-01-26
:Python-Version:    3.1
:Post-History:


========
Abstract
========

Writing a program to become a well-behaved Unix daemon is somewhat
complex and tricky to get right, yet the steps are largely similar for
any daemon regardless of what else the program may need to do.

This PEP introduces a module to the Python standard library that
provides a simple interface to the task of becoming a daemon process.


..  contents::
..
    Table of Contents: 
    Abstract
    Specification
      Example usage
      Interface
      ``Daemon`` objects
      ``DaemonError`` objects
    Motivation
    Rationale
      Correct daemon behaviour
      A daemon is not a service
    Reference Implementation
      Other daemon implementations
    References
    Copyright


=============
Specification
=============

Example usage
=============

Simple example of usage::

    import daemon

    from spam import do_main_program

    this_daemon = daemon.Daemon()
    this_daemon.start()

    do_main_program()

More complex example usage::

    import os
    import grp
    import signal
    import daemon

    from spam import (
        initial_program_setup,
        do_main_program,
        program_cleanup,
        reload_program_config,
        )

    initial_program_setup()
    important_file = open('spam.data', 'w')
    interesting_file = open('eggs.data', 'w')

    this_daemon = daemon.Daemon(
        working_directory='/var/lib/foo',
        umask=0o002,
        terminate_callback=program_cleanup,
        reload_callback=reload_program_config,
        reload_signals=[signal.SIGHUP, signal.SIGUSR1],
        )

    this_daemon.files_preserve = [important_file, interesting_file]

    mail_gid = grp.getgrnam('mail').gr_gid
    this_daemon.gid = mail_gid

    this_daemon.start()

    do_main_program()

Interface
=========

A new module, `daemon`, is added to the standard library.

The module defines a class, `Daemon`, used to represent the settings
and controls for a daemon process.

An exception class, `DaemonError`, is defined for exceptions raised
from the module.

``Daemon`` objects
==================

A `Daemon` instance represents the behaviour settings for the process
when it becomes a daemon. The behaviour is customised by setting
options on the instance, before calling the `start` method.

Each option can be passed as a keyword argument to the `Daemon`
constructor, or subsequently altered by assigning to an attribute on
the instance at any time prior to calling `start`. That is, for an
option named `wibble`, the following invocation::

    foo = daemon.Daemon(wibble=bar)
    foo.start()

is equivalent to::

    foo = daemon.Daemon()
    foo.wibble = bar
    foo.start()

The following options are defined.

`files_preserve`
    :Default: ``None``

    List of files that should *not* be closed when starting the
    daemon. If ``None``, all open file descriptors will be closed.

    Elements of the list are file descriptors (as returned by a file
    object's `fileno()` method) or Python `file` objects. Each
    specifies a file that is not to be closed during daemon start.

`chroot_directory`
    :Default: ``None``

    Full path to a directory to set as the effective root directory of
    the process. If ``None``, specifies that the root directory is not
    to be changed.

`working_directory`
    :Default: ``'/'``

    Full path of the working directory to which the process should
    change on daemon start.

    Since a filesystem cannot be unmounted if a process has its
    current working directory on that filesystem, this should either
    be left at default or set to a directory that is a sensible ?home
    directory? for the daemon while it is running.

`pidfile_directory`
    :Default: ``'/var/run'``

    Absolute directory path to contain the daemon's PID file. If
    ``None``, the PID file behaviour for this daemon is skipped.

`pidfile_name`
    :Default: ``None``

    Base name of the PID file for this daemon, without directory or
    suffix. If ``None``, the name is derived from the process command
    line.

`umask`
    :Default: ``0``

    File access creation mask (?umask?) to set for the process on
    daemon start.

    Since a process inherits its umask from its parent process,
    starting the daemon will reset the umask to this value so that
    files are created by the daemon with access modes as it expects.

`ignore_signals`
    :Default: ``[signal.SIGTTOU, signal.SIGTTIN, signal.SIGTSTP]``

    List of signals that the process should ignore (by setting the
    signal action to ``signal.SIG_IGN``) on daemon start.

`terminate_signals`
    :Default: ``[signal.SIGTERM]``

    List of signals that the process should interpret as a request to
    terminate cleanly.

`terminate_callback`
    :Default: ``None``

    Callable to invoke when the process receives any of the
    `terminate_signals` signals, before then terminating the process.

`reload_signals`
    :Default: ``[signal.SIGHUP]``

    List of signals that the process should interpret as a request to
    reload runtime configuration.

`reload_callback`
    :Default: ``None``

    Callable to invoke when the process receives any of the
    `reload_signals` signals.

`uid`
    :Default: ``None``

    The user ID (?uid?) value to switch the process to on daemon start.

`gid`
    :Default: ``None``

    The group ID (?gid?) value to switch the process to on daemon start.

`prevent_core`
    :Default: ``True``

    If true, prevents the generation of core files, in order to avoid
    leaking sensitive information from daemons run as `root`.

`stdout`
    :Default: ``None``

    File-like object, open for writing (in append mode, 'w+'), that
    will be used as the new value of `sys.stdout`. If it represents an
    actual file, it should be listed in `files_preserve` to prevent it
    being closed during daemon start. If ``None``, then `sys.stdout`
    is not re-bound.

`stderr`
    :Default: ``None``

    File-like object, open for writing (in append mode, 'w+'), that
    will be used as the new value of `sys.stderr`. If it represents an
    actual file, it should be listed in `files_preserve` to prevent it
    being closed during daemon start. If ``None``, then `sys.stderr`
    is not re-bound.


The following methods are defined.

`start()`
    :Return: ``None``

    Start the daemon, turning the current program into a daemon
    process. This performs the following steps:

    * If the `chroot_directory` attribute is not ``None``:

      * Set the effective root directory of the process to that
        directory (via `os.chroot`). This allows running the daemon
        process inside a ?chroot gaol? as a means of limiting the
        system's exposure to rogue behaviour by the process.

    * If the `pidfile_directory` attribute is not ``None``:

      * Look in that directory for a file named '`pidfile_name`.pid';
        if it exists, raise a `DaemonError` to prevent multiple
        instances of the daemon process.

    * Close all open file descriptors, excluding those listed in the
      `files_preserve` attribute.

    * Change current working directory to the path specified by the
      `working_directory` attribute.

    * Reset the file access creation mask to the value specified by
      the `umask` attribute.

    * Detach the current process into its own process group, and
      disassociate from any controlling terminal.

      This step is skipped if it is determined to be redundant: if the
      process was started by `init`, by `initd`, or by `inetd`.

    * Set signal handlers as specified by the `ignore_signals`,
      `terminate_signals`, `reload_signals` attributes.

    * If the `prevent_core` attribute is true:

      * Set the resource limits for the process to prevent any core
        dump from the process.

    * Set the process uid and gid to the true uid and gid of the
      process, to relinquish any elevated privilege.

    * If the `pidfile_directory` attribute is not ``None``:

      * Create the PID file for this daemon in that directory, by
        writing a text line containing the current process ID (?PID?)
        to a file named '`pidfile_name`.pid'.

    * If either of the attributes `uid` or `gid` are not ``None``:

      * Set the process uid and/or gid to the specified values.

    * If either of the attributes `stdout` or `stderr` are not
      ``None``:

      * Bind the names `sys.stdout` and/or `sys.stderr` to the
        corresponding objects.

    When the function returns, the running program is a daemon
    process.

`reload()`
    :Return: ``None``

    Reload the daemon configuration. The meaning of this is entirely
    defined by the customisation of this daemon: if the
    `reload_callback` attribute is not ``None``, call that object. The
    return value is discarded.

`stop()`
    :Return: ``None``

    Stop the daemon, ending the program. This performs the following
    steps:

    * If the `terminate_callback` attribute is not ``None``:

      * Call that object. The return value is discarded.

    * If the `pidfile_directory` attribute is not ``None``:

      * Delete the PID file for this daemon.

    * Raise a `SystemExit` exception.


``DaemonError`` objects
=======================

The `DaemonError` class inherits from `Exception`. The module
implementation will raise an instance of `DaemonError` when an error
occurs in processing daemon behaviour.


==========
Motivation
==========

The majority of programs written to be Unix daemons either implement
behaviour very similar to that in the `specification`_, or are
poorly-behaved daemons by the `correct daemon behaviour`_.

Since these steps should be much the same in most implementations but
are very particular and easy to omit or implement incorrectly, they
are a prime target for a standard well-tested implementation in the
standard library.


=========
Rationale
=========

Correct daemon behaviour
========================

According to Stevens in [stevens]_ ?2.6, a program should perform the
following steps to become a Unix daemon process.

* Close all open file descriptors.

* Change current working directory.

* Reset the file access creation mask.

* Run in the background.

* Disassociate from process group.

* Ignore terminal I/O signals.

* Disassociate from control terminal.

* Don't reacquire a control terminal.

* Correctly handle the following circumstances:

  * Started by System V `init` process.

  * Daemon termination by ``SIGTERM`` signal.

  * Children generate ``SIGCLD`` signal.

The `daemon` tool [slack-daemon]_ lists (in its summary of features)
behaviour that should be performed when turning a program into a
well-behaved Unix daemon process. It differs from this PEP's intent in
that it invokes a *separate* program as a daemon process. The
following features are appropriate for a daemon that starts itself
once the program is already running:

* Sets up the correct process context for a daemon.

* Behaves sensibly when started by `initd(8)` or `inetd(8)`.

* Revokes any suid or sgid privileges to reduce security risks in case
  daemon is incorrectly installed with special privileges.

* Prevents the generation of core files to prevent leaking sensitive
  information from daemons run as root (optional).

* Names the daemon by creating and locking a PID file to guarantee
  that only one daemon with the given name can execute at any given
  time (optional).

* Sets the user and group under which to run the daemon (optional,
  root only).

* Creates a chroot gaol (optional, root only).

* Captures the daemon's stdout and stderr and directs them to syslog
  (optional).

A daemon is not a service
=========================

This PEP addresses only Unix-style daemons, for which the above
correct behaviour is relevant, as opposed to comparable behaviours on
other operating systems.

There is a related concept in many systems, called a ?service?. A
service differs from the model in this PEP, in that rather than having
the *current* program continue to run as a daemon process, a service
starts an *additional* process to run in the background, and the
current process communicates with that additional process via some
defined channels.

The Unix-style daemon model in this PEP can be used, among other
things, to implement the background-process part of a service; but
this PEP does not address the other aspects of setting up and managing
a service. It is the opinion of this PEP's author that combining the
two disparate systems of behaviour under a single API would be
contrary to the focus of this PEP.


========================
Reference Implementation
========================

The `python-daemon` package [python-daemon]_.

As of 2009-01-26, the package is under active development and is not
yet a full implementation of this PEP.

Other daemon implementations
============================

Prior to this PEP, several existing third-party Python libraries or
tools implemented some of this PEP's `correct daemon behaviour`_.

The `reference implementation`_ is a fairly direct successor from the
following implementations:

* Many good ideas were contributed by the community to Python cookbook
  recipe 66012 [cookbook-66012]_.

* The `bda.daemon` library [bda.daemon]_ is an implementation of
  [cookbook-66012]_. It is the predecessor of [python-daemon]_.

Other Python daemon implementations that differ from this PEP:

* The `zdaemon` tool [zdaemon]_ was written for the Zope project. Like
  [slack-daemon]_, it differs from this specification because it is
  used to run another program as a daemon process.

* The Python library `daemon` [clapper-daemon]_ is (according to its
  homepage) no longer maintained. As of version 1.0.1, it implements
  the basic steps from [stevens]_.

* The `daemonize` library [seutter-daemonize]_ also implements the
  basic steps from [stevens]_.

* Twisted [twisted]_ includes, perhaps unsurprisingly, an
  implementation of a process daemonisation API that is integrated
  with the rest of the Twisted framework; it differs significantly
  from the API in this PEP.

* The Python `initd` library [dagitses-initd]_, which uses
  [clapper-daemon]_, implements an equivalent of Unix `initd(8)` for
  controlling a daemon process.


==========
References
==========

..  [stevens]

    `Unix Network Programming`, W. Richard Stevens, 1994 Prentice
    Hall.

..  [slack-daemon]

    The (non-Python) ?libslack? implementation of a `daemon` tool
    `<http://www.libslack.org/daemon/>`_ by ?raf? <raf at raf.org>.

..  [python-daemon]

    The `python-daemon` library
    `<http://pypi.python.org/pypi/python-daemon/>`_ by Ben Finney et
    al.

..  [cookbook-66012]

    Python Cookbook recipe 66012, ?Fork a daemon process on Unix?
    `<http://code.activestate.com/recipes/66012/>`_.

..  [bda.daemon]

    The `bda.daemon` library
    `<http://pypi.python.org/pypi/bda.daemon/>`_ by Robert
    Niederreiter et al.

..  [zdaemon]

    The `zdaemon` tool `<http://pypi.python.org/pypi/zdaemon/>`_ by
    Guido van Rossum et al.

..  [clapper-daemon]

    The `daemon` library `<http://pypi.python.org/pypi/daemon/>`_ by
    Brian Clapper.

..  [seutter-daemonize]

    The `daemonize` library `<http://daemonize.sourceforge.net/>`_ by
    Jerry Seutter.

..  [twisted]

    The `Twisted` application framework
    `<http://pypi.python.org/pypi/Twisted/>`_ by Glyph Lefkowitz et
    al.

..  [dagitses-initd]

    The Python `initd` library `<http://pypi.python.org/pypi/initd/>`_
    by Michael Andreas Dagitses.


=========
Copyright
=========

This work is hereby placed in the public domain. To the extent that
placing a work in the public domain is not legally possible, the
copyright holder hereby grants to all recipients of this work all
rights and freedoms that would otherwise be restricted by copyright.


..
    Local variables:
    mode: rst
    coding: utf-8
    time-stamp-start: "^:Last-Modified:[         ]+"
    time-stamp-end: "$"
    time-stamp-line-limit: 20
    time-stamp-format: "%:y-%02m-%02d %02H:%02M"
    End:
    vim: filetype=rst fileencoding=utf-8 :



From phd at phd.pp.ru  Thu Jan 29 13:01:49 2009
From: phd at phd.pp.ru (Oleg Broytmann)
Date: Thu, 29 Jan 2009 15:01:49 +0300
Subject: [Python-ideas] Draft PEP: Standard daemon process library
In-Reply-To: <87r62mx26z.fsf@benfinney.id.au>
References: <20090127161933.GB28125@phd.pp.ru> <497F351D.30700@timgolden.me.uk>
	<4222a8490901270830l275a7e55t1514e51944d86089@mail.gmail.com>
	<87eiyozbus.fsf@benfinney.id.au>
	<20090128012906.GH57568@wind.teleri.net>
	<87ab9cyy42.fsf@benfinney.id.au>
	<20090128030351.GD64950@wind.teleri.net>
	<871vuoyuxg.fsf@benfinney.id.au> <87k58gdpgl.fsf@xemacs.org>
	<87r62mx26z.fsf@benfinney.id.au>
Message-ID: <20090129120149.GA29303@phd.pp.ru>

On Thu, Jan 29, 2009 at 01:42:44PM +1100, Ben Finney wrote:
> rather specific Unix-native concept: the "pidfile", a combination
> sentinel, lockfile, and primitive text interface.

   IWBN to have a main() function in the module to be used from the command
line:

$ python -m daemon --pidfile=/var/run/mydaemon.pid status
Daemon pid 1221 is running

$ python -m daemon --pidfile=/var/run/mydaemon.pid stop

$ python -m daemon --pidfile=/var/run/mydaemon.pid status
Cannot find /var/run/mydaemon.pid

   A note on implementation detail: IWBN if the pidfile is removed on
stop(). Please do not put the burden unto the user.

> I might return the terminology in the PEP back to "pidfile" to be
> clear.

   +1

Oleg.
-- 
     Oleg Broytmann            http://phd.pp.ru/            phd at phd.pp.ru
           Programmers don't die, they just GOSUB without RETURN.


From solipsis at pitrou.net  Thu Jan 29 13:53:51 2009
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 29 Jan 2009 12:53:51 +0000 (UTC)
Subject: [Python-ideas] Draft PEP: Standard daemon process library
References: <20090127090058.GA302@phd.pp.ru> <20090127161933.GB28125@phd.pp.ru>
	<497F351D.30700@timgolden.me.uk>
	<4222a8490901270830l275a7e55t1514e51944d86089@mail.gmail.com>
	<87eiyozbus.fsf@benfinney.id.au>
	<20090128012906.GH57568@wind.teleri.net>
	<4222a8490901271749g1a9fc616lc6e6da2987be6040@mail.gmail.com>
	<loom.20090129T005452-128@post.gmane.org>
	<4222a8490901281748w7f4b0b9cw662982ebdac01774@mail.gmail.com>
	<20090129015703.GG9167@steerpike.home.puzzling.org>
	<4222a8490901281806s34c259f7xbbbfaf62b67ad5da@mail.gmail.com>
	<87vdryx2ja.fsf@benfinney.id.au>
Message-ID: <loom.20090129T124638-317@post.gmane.org>

Ben Finney <ben+python at ...> writes:
> 
> Instead, I think that people who want a ?service? would be better
> served by a top-level module named ?service?, to preserve that concept
> as separate from the concept of a daemon.

I may be misstaken, but I think the usual desire is to have *something* which
runs in the background and doesn't get killed when you kill the parent
application or terminal which launched it.

Whatever it is implemented in terms of "daemon", "service" or anything else
doesn't really matter in those cases.




From amcnabb at mcnabbs.org  Thu Jan 29 15:56:05 2009
From: amcnabb at mcnabbs.org (Andrew McNabb)
Date: Thu, 29 Jan 2009 07:56:05 -0700
Subject: [Python-ideas] Draft PEP (version 0.4): Standard daemon	process
	library
In-Reply-To: <87iqnywsx2.fsf@benfinney.id.au>
References: <87wscj11fl.fsf@benfinney.id.au> <87iqnywsx2.fsf@benfinney.id.au>
Message-ID: <20090129145605.GB3133@mcnabbs.org>

On Thu, Jan 29, 2009 at 05:03:05PM +1100, Ben Finney wrote:
> 
> There is a related concept in many systems, called a ?service?. A
> service differs from the model in this PEP, in that rather than having
> the *current* program continue to run as a daemon process, a service
> starts an *additional* process to run in the background, and the
> current process communicates with that additional process via some
> defined channels.

In my opinion, the difference between a daemon and a service is just a
by-product of the difference between fork and CreateProcess.  If the
multiprocessing package can solve this problem, I don't see why a
daemon/service package wouldn't be able to, too.


> As of 2009-01-26, the package is under active development and is not
> yet a full implementation of this PEP.

I'm still confused why there's a PEP before there's a stable
implementation and a base of users.


-- 
Andrew McNabb
http://www.mcnabbs.org/andrew/
PGP Fingerprint: 8A17 B57C 6879 1863 DE55  8012 AB4D 6098 8826 6868


From solipsis at pitrou.net  Thu Jan 29 17:54:53 2009
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 29 Jan 2009 16:54:53 +0000 (UTC)
Subject: [Python-ideas]
	=?utf-8?q?Draft_PEP_=28version_0=2E4=29=3A_Standar?=
	=?utf-8?q?d_daemon_process=09library?=
References: <87wscj11fl.fsf@benfinney.id.au> <87iqnywsx2.fsf@benfinney.id.au>
Message-ID: <loom.20090129T161440-214@post.gmane.org>


Hello,

> `files_preserve`
>     :Default: ``None``
> 
>     List of files that should *not* be closed when starting the
>     daemon.

Why does it have to be explicit? If I have an open file lying around, it
should be obvious that I don't want it closed. Otherwise I'd have closed it
myself...
The exception is of course the three stdio streams, which should be treated
separately.

> `umask`
>     :Default: ``0``
> 
>     File access creation mask (?umask?) to set for the process on
>     daemon start.
> 
>     Since a process inherits its umask from its parent process,
>     starting the daemon will reset the umask to this value so that
>     files are created by the daemon with access modes as it expects.

Why this behaviour? It's easy enough to call os.umask() manually if you want it,
but there are certainly situations where you don't want to change the
umask (situations where the daemon is meant to act on behalf of the user who
started it).
Or, perhaps, umask could be made mand

(regardless of which, 0 isn't a sensible umask default!)

>     * If the `pidfile_directory` attribute is not ``None``:
> 
>       * Look in that directory for a file named '`pidfile_name`.pid';
>         if it exists, raise a `DaemonError` to prevent multiple
>         instances of the daemon process.

It should first check that there does exist a processus with that number, so
that stale pid files don't cause too many problems.

Regards

Antoine.




From solipsis at pitrou.net  Thu Jan 29 18:02:16 2009
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 29 Jan 2009 17:02:16 +0000 (UTC)
Subject: [Python-ideas]
	=?utf-8?q?Draft_PEP_=28version_0=2E4=29=3A_Standar?=
	=?utf-8?q?d_daemon_process=09library?=
References: <87wscj11fl.fsf@benfinney.id.au> <87iqnywsx2.fsf@benfinney.id.au>
	<loom.20090129T161440-214@post.gmane.org>
Message-ID: <loom.20090129T170141-101@post.gmane.org>

Antoine Pitrou <solipsis at ...> writes:
> Or, perhaps, umask could be made mand

Finishing my sentence:
Or, perhaps, umask could be made mandatory only when the uid is changed.





From george.sakkis at gmail.com  Thu Jan 29 19:33:39 2009
From: george.sakkis at gmail.com (George Sakkis)
Date: Thu, 29 Jan 2009 13:33:39 -0500
Subject: [Python-ideas] Draft PEP (version 0.4): Standard daemon process
	library
In-Reply-To: <20090129145605.GB3133@mcnabbs.org>
References: <87wscj11fl.fsf@benfinney.id.au> <87iqnywsx2.fsf@benfinney.id.au>
	<20090129145605.GB3133@mcnabbs.org>
Message-ID: <91ad5bf80901291033g10f2585bt39283e2c8389c118@mail.gmail.com>

On Thu, Jan 29, 2009 at 9:56 AM, Andrew McNabb <amcnabb at mcnabbs.org> wrote:

>> As of 2009-01-26, the package is under active development and is not
>> yet a full implementation of this PEP.
>
> I'm still confused why there's a PEP before there's a stable
> implementation and a base of users.

Let alone the *need* to be included in the stdlib, when Pypi seems
perfectly appropriate for such package, especially if it's unix-only.

George


From python-ideas-list at trentnelson.com  Thu Jan 29 20:54:01 2009
From: python-ideas-list at trentnelson.com (Trent Nelson)
Date: Thu, 29 Jan 2009 19:54:01 +0000
Subject: [Python-ideas] Draft PEP: Standard daemon process library
In-Reply-To: <87vdryx2ja.fsf@benfinney.id.au>
References: <497F351D.30700@timgolden.me.uk>
	<4222a8490901270830l275a7e55t1514e51944d86089@mail.gmail.com>
	<87eiyozbus.fsf@benfinney.id.au>
	<20090128012906.GH57568@wind.teleri.net>
	<4222a8490901271749g1a9fc616lc6e6da2987be6040@mail.gmail.com>
	<loom.20090129T005452-128@post.gmane.org>
	<4222a8490901281748w7f4b0b9cw662982ebdac01774@mail.gmail.com>
	<20090129015703.GG9167@steerpike.home.puzzling.org>
	<4222a8490901281806s34c259f7xbbbfaf62b67ad5da@mail.gmail.com>
	<87vdryx2ja.fsf@benfinney.id.au>
Message-ID: <20090129195401.GB60708@wind.teleri.net>

On Thu, Jan 29, 2009 at 01:35:21PM +1100, Ben Finney wrote:

> It makes much more sense for ???daemon??? to stay simple and continue to
> mean the Unix-specific concept of ???daemon???, without necessarily
> obstructing whoever wants to implement ???service???, than for ???daemon???
> in Python to mean something other than the Unix meaning.

    There are two types of developers that will utilise this module.
    Those that are writing something for themselves, that never has to
    run anywhere else other than Unix.  Or those that work primarily or
    exclusively in Unix, but understand that others might want to run
    their code elsewhere.

    If you build it, they will come.  By building the latter solution,
    sure, you've inconvenienced the guy that wants a Unix-only daemon,
    but you've also enabled everyone else who wants to write stuff that
    has a chance of running unmodified on other platforms.

    And jumping on the bandwagon of a few others; I like the argument
    that something Unix-only like this could just as easily belong in
    PyPI, whereas a cross-platform solution is more viable for stdlib
    inclusion.

    We should make it harder to write non-portable code, not easier.

        Trent.


From ben+python at benfinney.id.au  Thu Jan 29 21:30:54 2009
From: ben+python at benfinney.id.au (Ben Finney)
Date: Fri, 30 Jan 2009 07:30:54 +1100
Subject: [Python-ideas] Draft PEP (version 0.4): Standard daemon	process
	library
References: <87wscj11fl.fsf@benfinney.id.au> <87iqnywsx2.fsf@benfinney.id.au>
	<20090129145605.GB3133@mcnabbs.org>
Message-ID: <8763jxx3b5.fsf@benfinney.id.au>

Andrew McNabb <amcnabb at mcnabbs.org> writes:

> I'm still confused why there's a PEP before there's a stable
> implementation and a base of users.

Because that's exactly what PEP 1 explicitly advises. Am I wrong to be
following PEP 1?

-- 
 \     ?We should strive to do things in [Gandhi's] spirit? not to use |
  `\   violence in fighting for our cause, but by non-participation in |
_o__)                       what we believe is evil.? ?Albert Einstein |
Ben Finney



From ben+python at benfinney.id.au  Thu Jan 29 22:06:38 2009
From: ben+python at benfinney.id.au (Ben Finney)
Date: Fri, 30 Jan 2009 08:06:38 +1100
Subject: [Python-ideas] Draft PEP (version 0.4): Standard daemon
	process	library
References: <87wscj11fl.fsf@benfinney.id.au> <87iqnywsx2.fsf@benfinney.id.au>
	<loom.20090129T161440-214@post.gmane.org>
Message-ID: <871vulx1nl.fsf@benfinney.id.au>

Antoine Pitrou <solipsis at pitrou.net> writes:

> Hello,
> 
> > `files_preserve`
> >     :Default: ``None``
> > 
> >     List of files that should *not* be closed when starting the
> >     daemon.
> 
> Why does it have to be explicit? If I have an open file lying
> around, it should be obvious that I don't want it closed. Otherwise
> I'd have closed it myself...

Is it so obvious? In Python programs, I far more often see files left
open simply because the programmer expects them to be cleaned up at
the appropriate time. By the references cited in this PEP, the time of
starting the daemon is such an appropriate time to close all open file
descriptors.

> > `umask`
> >     :Default: ``0``
> > 
> >     File access creation mask (?umask?) to set for the process on
> >     daemon start.
> > 
> >     Since a process inherits its umask from its parent process,
> >     starting the daemon will reset the umask to this value so that
> >     files are created by the daemon with access modes as it
> >     expects.
> 
> Why this behaviour?

Because the umask is inherited from the environment of the parent
process of the current program, which is good when the current program
is conceptually an extension of the parent program's environment, but
not in the case where one is intentionally disassociating from that
environment.

> It's easy enough to call os.umask() manually if you want it, but
> there are certainly situations where you don't want to change the
> umask (situations where the daemon is meant to act on behalf of the
> user who started it).

Those situations exist, but I would argue they are not the common
case.

> Or, perhaps, umask could be made [mandatory when the options ?uid?
> or ?gid? are set.]

That diverges from the referenced ?correct behaviour?, in particular
the Stevens reference. I think if this PEP is worth implementing, it's
for exactly the purpose of providing a Python implementation of
defined correct daemon behaviour; that definition includes re-setting
the umask.

> (regardless of which, 0 isn't a sensible umask default!)

The default umask ?0? is chosen in the references as the umask that
allows file access modes for files created by the daemon to work as
expected. Quoting Stevens:

    This prevents any files created by the daemon from having their
    access bits modified. For example, if a daemon specifically
    created a file with a mode of 0660, so that only the user and
    group could read and write the file, but the ``umask`` value was
    060, the group read and write permissions would be turned off. If
    the daemon required the group permissions to be on, so that some
    other process in that group could access the file, this ``umask``
    value prevents it.

-- 
 \        ?I don't accept the currently fashionable assertion that any |
  `\       view is automatically as worthy of respect as any equal and |
_o__)                                   opposite view.? ?Douglas Adams |
Ben Finney



From ben+python at benfinney.id.au  Thu Jan 29 22:09:16 2009
From: ben+python at benfinney.id.au (Ben Finney)
Date: Fri, 30 Jan 2009 08:09:16 +1100
Subject: [Python-ideas] Draft PEP (version 0.4): Standard daemon
	process	library
References: <87wscj11fl.fsf@benfinney.id.au> <87iqnywsx2.fsf@benfinney.id.au>
	<loom.20090129T161440-214@post.gmane.org>
Message-ID: <87wscdvmyr.fsf@benfinney.id.au>

Antoine Pitrou <solipsis at pitrou.net> writes:

> Hello,
> [?]
> >     * If the `pidfile_directory` attribute is not ``None``:
> > 
> >       * Look in that directory for a file named '`pidfile_name`.pid';
> >         if it exists, raise a `DaemonError` to prevent multiple
> >         instances of the daemon process.
> 
> It should first check that there does exist a processus with that
> number, so that stale pid files don't cause too many problems.

Good point, thanks.

What would be appropriate behaviour in the case of a stale PID file?
Abort the daemonisation attempt? Delete the stale lock file silently
and continue as though it didn't exist?

-- 
 \       ?I bought some powdered water, but I don't know what to add.? |
  `\                                                    ?Steven Wright |
_o__)                                                                  |
Ben Finney



From ben+python at benfinney.id.au  Thu Jan 29 22:14:07 2009
From: ben+python at benfinney.id.au (Ben Finney)
Date: Fri, 30 Jan 2009 08:14:07 +1100
Subject: [Python-ideas] Draft PEP: Standard daemon process library
References: <20090127161933.GB28125@phd.pp.ru> <497F351D.30700@timgolden.me.uk>
	<4222a8490901270830l275a7e55t1514e51944d86089@mail.gmail.com>
	<87eiyozbus.fsf@benfinney.id.au>
	<20090128012906.GH57568@wind.teleri.net>
	<87ab9cyy42.fsf@benfinney.id.au>
	<20090128030351.GD64950@wind.teleri.net>
	<871vuoyuxg.fsf@benfinney.id.au> <87k58gdpgl.fsf@xemacs.org>
	<87r62mx26z.fsf@benfinney.id.au> <20090129120149.GA29303@phd.pp.ru>
Message-ID: <87pri5vmqo.fsf@benfinney.id.au>

Oleg Broytmann <phd at phd.pp.ru> writes:

>    IWBN to have a main() function in the module to be used from the
> command line:
> 
> $ python -m daemon --pidfile=/var/run/mydaemon.pid status
> Daemon pid 1221 is running
> 
> $ python -m daemon --pidfile=/var/run/mydaemon.pid stop
> 
> $ python -m daemon --pidfile=/var/run/mydaemon.pid status
> Cannot find /var/run/mydaemon.pid

Interesting. Should that be defined in this PEP, or merely added to
the reference implementation as a bonus?

>    A note on implementation detail: IWBN if the pidfile is removed on
> stop(). Please do not put the burden unto the user.

Yes, that's already addressed in the defined behaviour for `stop()`.

-- 
 \       ?Know what I hate most? Rhetorical questions.? ?Henry N. Camp |
  `\                                                                   |
_o__)                                                                  |
Ben Finney



From ben+python at benfinney.id.au  Thu Jan 29 22:18:05 2009
From: ben+python at benfinney.id.au (Ben Finney)
Date: Fri, 30 Jan 2009 08:18:05 +1100
Subject: [Python-ideas] Draft PEP (version 0.4): Standard daemon process
	library
References: <87wscj11fl.fsf@benfinney.id.au> <87iqnywsx2.fsf@benfinney.id.au>
	<20090129145605.GB3133@mcnabbs.org>
	<91ad5bf80901291033g10f2585bt39283e2c8389c118@mail.gmail.com>
Message-ID: <87ocxpvmk2.fsf@benfinney.id.au>

George Sakkis writes:

> Let alone the *need* to be included in the stdlib, when Pypi seems
> perfectly appropriate for such package, especially if it's
> unix-only.

I addressed this in Message-ID: <87mydawt1o.fsf at benfinney.id.au>
yesterday (a reply to Bill Janssen).

-- 
 \        ?My doctor told me to stop having intimate dinners for four. |
  `\               Unless there are three other people.? ?Orson Welles |
_o__)                                                                  |
Ben Finney



From phd at phd.pp.ru  Thu Jan 29 22:22:15 2009
From: phd at phd.pp.ru (Oleg Broytmann)
Date: Fri, 30 Jan 2009 00:22:15 +0300
Subject: [Python-ideas] Draft PEP: Standard daemon process library
In-Reply-To: <87pri5vmqo.fsf@benfinney.id.au>
References: <4222a8490901270830l275a7e55t1514e51944d86089@mail.gmail.com>
	<87eiyozbus.fsf@benfinney.id.au>
	<20090128012906.GH57568@wind.teleri.net>
	<87ab9cyy42.fsf@benfinney.id.au>
	<20090128030351.GD64950@wind.teleri.net>
	<871vuoyuxg.fsf@benfinney.id.au> <87k58gdpgl.fsf@xemacs.org>
	<87r62mx26z.fsf@benfinney.id.au> <20090129120149.GA29303@phd.pp.ru>
	<87pri5vmqo.fsf@benfinney.id.au>
Message-ID: <20090129212215.GA17415@phd.pp.ru>

On Fri, Jan 30, 2009 at 08:14:07AM +1100, Ben Finney wrote:
> Oleg Broytmann <phd at phd.pp.ru> writes:
> >    IWBN to have a main() function in the module to be used from the
> > command line:
> > 
> > $ python -m daemon --pidfile=/var/run/mydaemon.pid status
> > Daemon pid 1221 is running
> > 
> > $ python -m daemon --pidfile=/var/run/mydaemon.pid stop
> > 
> > $ python -m daemon --pidfile=/var/run/mydaemon.pid status
> > Cannot find /var/run/mydaemon.pid
> 
> Interesting. Should that be defined in this PEP, or merely added to
> the reference implementation as a bonus?

   I think an implementation would be enough.

> >    A note on implementation detail: IWBN if the pidfile is removed on
> > stop(). Please do not put the burden unto the user.
> 
> Yes, that's already addressed in the defined behaviour for `stop()`.

   Thank you!

Oleg.
-- 
     Oleg Broytmann            http://phd.pp.ru/            phd at phd.pp.ru
           Programmers don't die, they just GOSUB without RETURN.


From ben+python at benfinney.id.au  Thu Jan 29 22:22:45 2009
From: ben+python at benfinney.id.au (Ben Finney)
Date: Fri, 30 Jan 2009 08:22:45 +1100
Subject: [Python-ideas] Draft PEP (version 0.4): Standard daemon	process
	library
References: <87wscj11fl.fsf@benfinney.id.au> <87iqnywsx2.fsf@benfinney.id.au>
	<20090129145605.GB3133@mcnabbs.org>
Message-ID: <87k58dvmca.fsf@benfinney.id.au>

Andrew McNabb <amcnabb at mcnabbs.org> writes:

> In my opinion, the difference between a daemon and a service is just
> a by-product of the difference between fork and CreateProcess.

This PEP addresses the case where a daemon is specifically what the
programmer wants.

I would like it to *also* be useful as the obvious thing to use for
implementing a service on Unix, but it is designed to also allow for
the case when a service is *not* what the programmer wants.

> If the multiprocessing package can solve this problem, I don't see
> why a daemon/service package wouldn't be able to, too.

A service package would be ideal for the use case you suggest. That's
not what this PEP proposes, though.

-- 
 \           ?Why, I'd horse-whip you if I had a horse.? ?Groucho Marx |
  `\                                                                   |
_o__)                                                                  |
Ben Finney



From ben+python at benfinney.id.au  Thu Jan 29 22:28:33 2009
From: ben+python at benfinney.id.au (Ben Finney)
Date: Fri, 30 Jan 2009 08:28:33 +1100
Subject: [Python-ideas] Draft PEP: Standard daemon process library
References: <497F351D.30700@timgolden.me.uk>
	<4222a8490901270830l275a7e55t1514e51944d86089@mail.gmail.com>
	<87eiyozbus.fsf@benfinney.id.au>
	<20090128012906.GH57568@wind.teleri.net>
	<4222a8490901271749g1a9fc616lc6e6da2987be6040@mail.gmail.com>
	<loom.20090129T005452-128@post.gmane.org>
	<4222a8490901281748w7f4b0b9cw662982ebdac01774@mail.gmail.com>
	<20090129015703.GG9167@steerpike.home.puzzling.org>
	<4222a8490901281806s34c259f7xbbbfaf62b67ad5da@mail.gmail.com>
	<87vdryx2ja.fsf@benfinney.id.au>
	<20090129195401.GB60708@wind.teleri.net>
Message-ID: <87fxj1vm2m.fsf@benfinney.id.au>

Trent Nelson writes:

>     There are two types of developers that will utilise this module.
>     Those that are writing something for themselves, that never has to
>     run anywhere else other than Unix.

Surely there is also the type that are writing *for others*, and want
the program to be both well-behaved and work on any Unix-like OS?

>  Or those that work primarily or exclusively in Unix, but understand
>  that others might want to run their code elsewhere.

Those people would be well served by a putative ?service? interface.
I would welcome such a PEP from you, since you seem to have fairly
solid ideas of what it should look like.

>     We should make it harder to write non-portable code, not easier.

It's a bit of a stretch to go from ?designed to work on any Unix-like
OS? to ?non-portable?.

-- 
 \     ?Under democracy one party always devotes its chief energies to |
  `\       trying to prove that the other party is unfit to rule ? and |
_o__)         both commonly succeed, and are right.? ?Henry L. Mencken |
Ben Finney



From solipsis at pitrou.net  Thu Jan 29 22:53:18 2009
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 29 Jan 2009 21:53:18 +0000 (UTC)
Subject: [Python-ideas]
	=?utf-8?q?Draft_PEP_=28version_0=2E4=29=3A_Standar?=
	=?utf-8?q?d_daemon=09process=09library?=
References: <87wscj11fl.fsf@benfinney.id.au> <87iqnywsx2.fsf@benfinney.id.au>
	<loom.20090129T161440-214@post.gmane.org>
	<871vulx1nl.fsf@benfinney.id.au>
Message-ID: <loom.20090129T214059-805@post.gmane.org>

Ben Finney <ben+python at ...> writes:
> 
> Is it so obvious? In Python programs, I far more often see files left
> open simply because the programmer expects them to be cleaned up at
> the appropriate time.

What if I use a library which keeps its own files open in the background? Do I
have to monkeypatch the library so as to get at the file descriptors and pass
them by hand?

> > It's easy enough to call os.umask() manually if you want it, but
> > there are certainly situations where you don't want to change the
> > umask (situations where the daemon is meant to act on behalf of the
> > user who started it).
> 
> Those situations exist, but I would argue they are not the common
> case.

Well, perhaps, but it's easier to leave the umask as is rather than ask the
programmer to query the current umask and then pass it as an argument (not to
mention that the current os.umask() function does not support querying without
modifying...).

> That diverges from the referenced ?correct behaviour?, in particular
> the Stevens reference. I think if this PEP is worth implementing, it's
> for exactly the purpose of providing a Python implementation of
> defined correct daemon behaviour; that definition includes re-setting
> the umask.

Well, I think the PEP should implement a behaviour which is at the same time
useful and sane. Whether it scrupulously conforms to something which isn't an
official standard shouldn't be considered important.

> The default umask ?0? is chosen in the references as the umask that
> allows file access modes for files created by the daemon to work as
> expected.

And then, if I create a file using the standard Python idiom (the open()
function), it will be readable and writable by anyone. It isn't sane at all.

Regards,

Antoine.




From solipsis at pitrou.net  Thu Jan 29 23:07:32 2009
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 29 Jan 2009 22:07:32 +0000 (UTC)
Subject: [Python-ideas]
	=?utf-8?q?Draft_PEP_=28version_0=2E4=29=3A_Standar?=
	=?utf-8?q?d_daemon=09process=09library?=
References: <87wscj11fl.fsf@benfinney.id.au> <87iqnywsx2.fsf@benfinney.id.au>
	<loom.20090129T161440-214@post.gmane.org>
	<87wscdvmyr.fsf@benfinney.id.au>
Message-ID: <loom.20090129T220650-495@post.gmane.org>

Ben Finney <ben+python at ...> writes:
> 
> What would be appropriate behaviour in the case of a stale PID file?
> Abort the daemonisation attempt? Delete the stale lock file silently
> and continue as though it didn't exist?

Delete the stale lock file silently and continue as though it didn't exist.
(and, of course, create another one for the current process)




From ben+python at benfinney.id.au  Fri Jan 30 01:09:20 2009
From: ben+python at benfinney.id.au (Ben Finney)
Date: Fri, 30 Jan 2009 11:09:20 +1100
Subject: [Python-ideas] Draft PEP: Standard daemon process library
References: <87wscj11fl.fsf@benfinney.id.au> <gliuo9$shj$1@ger.gmane.org>
	<871vup1s62.fsf@benfinney.id.au> <871vupfqdd.fsf@xemacs.org>
Message-ID: <87pri5u027.fsf@benfinney.id.au>

"Stephen J. Turnbull" <stephen at xemacs.org> writes:

> The people who are posting "show us the code" are mostly folks with
> long experience in the PEP process. I think you should take it as an
> indication that in this case the definition of "suitable API" is an
> "I'll know it when I've tried it" kind of thing.

I am working on ?python-daemon? concurrently with this PEP now, and
intend to keep publishing updates to that implementation as it
approaches conformance with the PEP.

> What you might find useful is to avoid writing any code yourself, but
> see how well your proposed spec fits the existing code, of which
> several (potential) examples have been posted.

Fortunately I've based ?python-daemon? on what I determined to be the
closest starting point for the well-behaved daemon behaviour.

-- 
 \      ?Any intelligent fool can make things bigger and more complex? |
  `\    It takes a touch of genius ? and a lot of courage ? to move in |
_o__)                        the opposite direction.? ?Albert Einstein |
Ben Finney



From stephen at xemacs.org  Fri Jan 30 03:12:05 2009
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Fri, 30 Jan 2009 11:12:05 +0900
Subject: [Python-ideas] Draft PEP: Standard daemon process library
In-Reply-To: <87fxj1vm2m.fsf@benfinney.id.au>
References: <497F351D.30700@timgolden.me.uk>
	<4222a8490901270830l275a7e55t1514e51944d86089@mail.gmail.com>
	<87eiyozbus.fsf@benfinney.id.au>
	<20090128012906.GH57568@wind.teleri.net>
	<4222a8490901271749g1a9fc616lc6e6da2987be6040@mail.gmail.com>
	<loom.20090129T005452-128@post.gmane.org>
	<4222a8490901281748w7f4b0b9cw662982ebdac01774@mail.gmail.com>
	<20090129015703.GG9167@steerpike.home.puzzling.org>
	<4222a8490901281806s34c259f7xbbbfaf62b67ad5da@mail.gmail.com>
	<87vdryx2ja.fsf@benfinney.id.au>
	<20090129195401.GB60708@wind.teleri.net>
	<87fxj1vm2m.fsf@benfinney.id.au>
Message-ID: <87y6wtczka.fsf@xemacs.org>

Ben Finney writes:

 > Those people would be well served by a putative "service" interface.
 > I would welcome such a PEP from you, since you seem to have fairly
 > solid ideas of what it should look like.

AFAICS it looks like yours, except with the requirement of "works on
Windows" and a strong preference for "no other-OS-specific clutter
with a simple 'import daemonize'".  Bill added "plays nicely with Mac
OS X".  I don't see on what grounds you object to exploring the
possibility, since you say you don't know much about Windows services.

Note that nobody is suggesting that you write the code that implements
daemons via Windows services (of course they'd love it if you did!),
only that you open up the API design to changes that would make some
programs work on all platforms without conditioning on platform, and
make all programs work with minimal platform-specific code.

So I suggest that you bat the ball back into their court, and announce
on Python-Dev that you're willing but not able to generalize your PEP
to Windows, and ask for coauthors to help.  If there are no takers,
then you're in a strong position to argue that Unix-like OSes are the
important case anyway, nobody cares about Windows services in Python.
If there are, then somebody else does the work and everybody is happy.

No?

 > >     We should make it harder to write non-portable code, not easier.
 > 
 > It's a bit of a stretch to go from "designed to work on any Unix-like
 > OS" to "non-portable".

That is not a Pythonic attitude.  Here, "portable code" means
"designed to work in any instance of Python conditioning only on 'this
instance supports the feature'."



From ben+python at benfinney.id.au  Fri Jan 30 04:30:17 2009
From: ben+python at benfinney.id.au (Ben Finney)
Date: Fri, 30 Jan 2009 14:30:17 +1100
Subject: [Python-ideas] Draft PEP: Standard daemon process library
References: <497F351D.30700@timgolden.me.uk>
	<4222a8490901270830l275a7e55t1514e51944d86089@mail.gmail.com>
	<87eiyozbus.fsf@benfinney.id.au>
	<20090128012906.GH57568@wind.teleri.net>
	<4222a8490901271749g1a9fc616lc6e6da2987be6040@mail.gmail.com>
	<loom.20090129T005452-128@post.gmane.org>
	<4222a8490901281748w7f4b0b9cw662982ebdac01774@mail.gmail.com>
	<20090129015703.GG9167@steerpike.home.puzzling.org>
	<4222a8490901281806s34c259f7xbbbfaf62b67ad5da@mail.gmail.com>
	<87vdryx2ja.fsf@benfinney.id.au>
	<20090129195401.GB60708@wind.teleri.net>
	<87fxj1vm2m.fsf@benfinney.id.au> <87y6wtczka.fsf@xemacs.org>
Message-ID: <878wottqra.fsf@benfinney.id.au>

"Stephen J. Turnbull" <stephen at xemacs.org> writes:

> Ben Finney writes:
> 
>  > Those people would be well served by a putative "service" interface.
>  > I would welcome such a PEP from you, since you seem to have fairly
>  > solid ideas of what it should look like.
> 
> AFAICS it looks like yours

I can't see how anyone could compare the latest version of this PEP
with the suggestions being made by Trent for an API, and conclude that
they are similar.

This might because the PEP isn't communicating very well. Does the
section ?A daemon is not a service? help to explain the fundamental
difference between the model suggested by Trent, versus the model in
the PEP?

If not, is it made clearer by the addition of nearly a dozen extra
methods in Trent's suggestion, and several extra concepts, that are
extraneous to the task described in the PEP?

All of which seem ideally suited to a putative ?service? API.

> I don't see on what grounds you object to exploring the possibility,
> since you say you don't know much about Windows services.

I have no objection to someone exploring a ?service? interface.
Indeed, I applaud such efforts. But that's not what I'm addressing
with this PEP.

As for the possibility of using the daemon interface as the
Unix-specific implementation of the background-process *component* of
a service interface, I believe it would be very well suited for that.

But I am implementing the daemon API *now* because it's a distinct
task, that I've seen needed many times and implemented poorly in many
places. I would be delighted if someone who knows what they want from
a ?service? interface could build that, entirely different, API
using a daemon as part of the implementation.

> Note that nobody is suggesting that you write the code that
> implements daemons via Windows services (of course they'd love it if
> you did!), only that you open up the API design to changes that
> would make some programs work on all platforms without conditioning
> on platform, and make all programs work with minimal
> platform-specific code.

I maintain that it would be better to specify a platform-agnostic
?service? API, that uses the ?daemon? API as the Unix-specific
implementation of the background-process component of the service.

Meanwhile, I and others have a need for a simple, standardised,
one-obvious-way ?daemonise the current program? library, *entirely
distinct from* any of ther extra baggage that comes with a ?spawn a
separate process that I can then communicate with through channels?
API. I intend to satisfy that with this PEP.

-- 
 \      ?People demand freedom of speech to make up for the freedom of |
  `\   thought which they avoid.? ?Soren Aabye Kierkegaard (1813-1855) |
_o__)                                                                  |
Ben Finney



From ben+python at benfinney.id.au  Fri Jan 30 04:44:39 2009
From: ben+python at benfinney.id.au (Ben Finney)
Date: Fri, 30 Jan 2009 14:44:39 +1100
Subject: [Python-ideas] =?utf-8?q?Cross-platform_file_locking=2C_PID_files?=
 =?utf-8?b?LCBhbmQgdGhlIOKAnGRhZW1vbuKAnSBQRVAgKHdhczogRHJhZnQgUEVQOiBT?=
 =?utf-8?q?tandard_daemon_process_library=29?=
References: <21108.1233038783@pippin.parc.xerox.com>
	<87wschyykf.fsf@benfinney.id.au> <20090127090058.GA302@phd.pp.ru>
	<20090127160643.GC37589@wind.teleri.net>
	<20090127161933.GB28125@phd.pp.ru> <497F351D.30700@timgolden.me.uk>
	<4222a8490901270830l275a7e55t1514e51944d86089@mail.gmail.com>
	<87eiyozbus.fsf@benfinney.id.au>
	<20090128012906.GH57568@wind.teleri.net>
	<87ab9cyy42.fsf@benfinney.id.au>
	<20090128030351.GD64950@wind.teleri.net>
	<871vuoyuxg.fsf@benfinney.id.au> <87k58gdpgl.fsf@xemacs.org>
Message-ID: <874ozhtq3c.fsf_-_@benfinney.id.au>

"Stephen J. Turnbull" writes:

> If [the daemon interface deals with file locking] only because
> there's no satisfactory module implementing it widely available,
> then that could be done in a subsidiary module that handles
> OS-specific details, and optionally exposes OS-specific special
> features (a la os and os.posix). That would also make refactoring
> (close to) trivial when such a module did become widely available.

Skip Montanaro has a PyPI package, ?lockfile? (currently marked
?Beta?) <URL:http://pypi.python.org/pypi/lockfile> that has such
ambitions.

I think the interface he proposes is elegant and I intend to submit an
additional ?PID File lock? capability, to cover the behaviour
expected of PID files in particular.

How should I alter this PEP if I'm proposing to use functionality
currently present in another package that exists only in PyPI?

Should I make an even lower-level PEP to suggest inclusion of Skip's
?lockfile? package? Is that wise? Is it polite?

-- 
 \          ?Are you pondering what I'm pondering?? ?Well, I think so, |
  `\    Brain, but if Jimmy cracks corn, and no one cares, why does he |
_o__)                           keep doing it?? ?_Pinky and The Brain_ |
Ben Finney



From stephen at xemacs.org  Fri Jan 30 06:20:30 2009
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Fri, 30 Jan 2009 14:20:30 +0900
Subject: [Python-ideas] Draft PEP: Standard daemon process library
In-Reply-To: <878wottqra.fsf@benfinney.id.au>
References: <497F351D.30700@timgolden.me.uk>
	<4222a8490901270830l275a7e55t1514e51944d86089@mail.gmail.com>
	<87eiyozbus.fsf@benfinney.id.au>
	<20090128012906.GH57568@wind.teleri.net>
	<4222a8490901271749g1a9fc616lc6e6da2987be6040@mail.gmail.com>
	<loom.20090129T005452-128@post.gmane.org>
	<4222a8490901281748w7f4b0b9cw662982ebdac01774@mail.gmail.com>
	<20090129015703.GG9167@steerpike.home.puzzling.org>
	<4222a8490901281806s34c259f7xbbbfaf62b67ad5da@mail.gmail.com>
	<87vdryx2ja.fsf@benfinney.id.au>
	<20090129195401.GB60708@wind.teleri.net>
	<87fxj1vm2m.fsf@benfinney.id.au> <87y6wtczka.fsf@xemacs.org>
	<878wottqra.fsf@benfinney.id.au>
Message-ID: <87r62lcqu9.fsf@xemacs.org>

Ben Finney writes:

 > [C]ompare the latest version of this PEP [...].

I'm assuming the one posted on 1/29 in
<87iqnywsx2.fsf at benfinney.id.au>, or fairly close to that.

 > This might because the PEP isn't communicating very well. Does the
 > section "A daemon is not a service" help to explain the fundamental
 > difference between the model suggested by Trent, versus the model in
 > the PEP?

That is quite clear, and you've said "daemonize-me != start-a-
service" in discussion here.  However, you've also said that "daemon
!= service" (including in the title of that section!), which clearly
is mostly false when those terms are referring to the processes that
are running in the background.

I think that focusing on the distinction between "*becoming* a daemon"
and "*starting* a service" is distracting.  The point being that
*becoming* a well-behaved service surely requires becoming a well-
behaved background process.  The real issue is not starting a separate
process (just don't provide a way to do it), but rather it is that it
also requires all the communication setup as well, which is out-of-
scope for a daemonization PEP on any platform (apart from closing file
descriptors, which is specific to the fork(2) implementation of
process creation).  To prevent the misinterpretation I made, I would
phrase that title "A Service is Not Just a Daemon", implying "this PEP
is not a full-service PEP".<wink>

BTW, IMO you can delete the sentence "It is the opinion of this PEP's
author that combining the two disparate systems of behaviour under a
single API would be contrary to the focus of this PEP." from the PEP.
It's redundant.

 > Meanwhile, I and others have a need for a simple, standardised,
 > one-obvious-way "daemonise the current program" library, *entirely
 > distinct from* any of ther extra baggage that comes with a "spawn a
 > separate process that I can then communicate with through channels"
 > API. I intend to satisfy that with this PEP.

Well, that's what I thought was in question here.  It seems to me that
there are likely to be additional aspects to "daemonize me" for Mac OS
X and Windows, but that the basic start-reload-stop methods should
probably apply.  I'm not familiar enough with the process of
daemonization on Mac OS X (Bill mentioned interaction with launchd) or
Windows to help at reasonable expense, but I certainly hope that those
who are will review to see if a similar API might be useful to them.
Unfortunately there doesn't seem to be that much common ground among
the advocates, though.

Steve


From stephen at xemacs.org  Fri Jan 30 06:39:52 2009
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Fri, 30 Jan 2009 14:39:52 +0900
Subject: [Python-ideas]  Cross-platform file locking, PID files ,
 and the "daemon" PEP (was: Draft PEP: S tandard daemon process
 library)
In-Reply-To: <874ozhtq3c.fsf_-_@benfinney.id.au>
References: <21108.1233038783@pippin.parc.xerox.com>
	<87wschyykf.fsf@benfinney.id.au> <20090127090058.GA302@phd.pp.ru>
	<20090127160643.GC37589@wind.teleri.net>
	<20090127161933.GB28125@phd.pp.ru> <497F351D.30700@timgolden.me.uk>
	<4222a8490901270830l275a7e55t1514e51944d86089@mail.gmail.com>
	<87eiyozbus.fsf@benfinney.id.au>
	<20090128012906.GH57568@wind.teleri.net>
	<87ab9cyy42.fsf@benfinney.id.au>
	<20090128030351.GD64950@wind.teleri.net>
	<871vuoyuxg.fsf@benfinney.id.au> <87k58gdpgl.fsf@xemacs.org>
	<874ozhtq3c.fsf_-_@benfinney.id.au>
Message-ID: <87ocxpcpxz.fsf@xemacs.org>

Ben Finney writes:

 > Skip Montanaro has a PyPI package, ?lockfile? (currently marked
 > "Beta") <URL:http://pypi.python.org/pypi/lockfile> that has such
 > ambitions.

 > Should I make an even lower-level PEP to suggest inclusion of Skip's
 > ?lockfile? package? Is that wise? Is it polite?

I would send your proposal for PIDlockfile to Skip.  In your PEP,
describe the relevant parts of the API explicitly in the PEP, with a
footnote that you are deliberately extending the lockfile interface
and currently intend to remain compatible with that interface.

Then talk to Skip about a PEP for including his package, or perhaps
extending your PEP to include it.  Which is better depends on details
and how aggressively Skip would want to pursue inclusion of lockfile
if your daemon package doesn't make it or takes a long time.

I don't know about "polite", but I think it would be unwise to try to
PEP lockfile without Skip's explicit support.



From ben+python at benfinney.id.au  Fri Jan 30 06:46:13 2009
From: ben+python at benfinney.id.au (Ben Finney)
Date: Fri, 30 Jan 2009 16:46:13 +1100
Subject: [Python-ideas] Skeletal PEP (version 0.1): Background process
	services
Message-ID: <87pri5s5wa.fsf@benfinney.id.au>

Howdy all,

Many people evidently want a proposal for a ?service? interface, and
saw part of what they wanted in the separate discussion of the
?Standard daemon process library?. As discussed there, a Unix daemon
is a quite distinct, yet complementary, concept to a service.

This is a skeleton of a PEP, presented in order that discussion of
what people want from a ?service? has a concrete focus. The
specification is based on work from Trent Nelson.

Note that I'm *not* intending to propose, implement, or champion this
PEP. I present it as is because there is a clear desire to see
something like it, and in the hope that those with more enthusiasm for
it can take it and improve on it to the point where it gets a
champion.


:PEP:               XXX
:Title:             Background process services
:Version:           0.1
:Last-Modified:     2009-01-30 16:40
:Author:            Ben Finney <ben+python at benfinney.id.au>
:Status:            Draft
:Type:              Standards Track
:Content-Type:      text/x-rst
:Created:           2009-01-30
:Python-Version:    3.1
:Post-History:


========
Abstract
========

This PEP introduces a package to the Python standard library that
provides an interface to the task of creating a separate background
process, and maintaining an ongoing service channel with that process.


..  contents::
..
    Table of Contents:
    Abstract
    Specification
      Interface
      ``Service`` objects
    Motivation
    Rationale
    Reference Implementation
    References
    Copyright


=============
Specification
=============


Interface
=========

A new package, `service`, is added to the standard library.

The package defines a class, `Service`, used to set up and maintain
the service.


``Service`` objects
===================

A `Service` instance represents a control channel to a separate,
background process that runs concurrently with the existing program.
The existing program communicates with that background process via the
corresponding `Service` instance.

`__init__(self, func, name, desc=None, startup_type=StartupType.Auto)`
    Initialises the parameters of a new service [?]

    The `func` argument is a callable, which will be called to run in
    the background process when the service is started.

    [?]

`parse_arguments(args)`
    [?]

`is_running()`
    Returns ``True`` if the background process is currently running [?]

`is_installed()`
    Returns ``True`` if [?]

`install()`
    [?]

`uninstall()`
    [?]

`report_event(self, msg, event_type, id, category, data)`
    [?]

`exec()`
    [?]

`terminate()`
    [?]

`request_pause()`
    [?]

`request_resume()`
    [?]

`send_command(code)`
    [?]

`run(args)`
    [?]

`start()`
    Start a new background process running the `func` callable [?]

    * On Unix, this creates a new daemon process.

    * On MacOS, [?].

    * On MS Windows, [?].

`stop()`
    Stop the background process [?]

`pause()`
    [?]

`resume()`
    [?]


==========
Motivation
==========

[?]


=========
Rationale
=========

[?]


========================
Reference Implementation
========================

[?]


==========
References
==========

[?]


=========
Copyright
=========

This work is hereby placed in the public domain. To the extent that
placing a work in the public domain is not legally possible, the
copyright holder hereby grants to all recipients of this work all
rights and freedoms that would otherwise be restricted by copyright.


..
    Local variables:
    mode: rst
    coding: utf-8
    time-stamp-start: "^:Last-Modified:[         ]+"
    time-stamp-end: "$"
    time-stamp-line-limit: 20
    time-stamp-format: "%:y-%02m-%02d %02H:%02M"
    End:
    vim: filetype=rst fileencoding=utf-8 :



From nate at binkert.org  Fri Jan 30 22:38:42 2009
From: nate at binkert.org (nathan binkert)
Date: Fri, 30 Jan 2009 13:38:42 -0800
Subject: [Python-ideas] Draft PEP: Standard daemon process library
In-Reply-To: <87r62lcqu9.fsf@xemacs.org>
References: <497F351D.30700@timgolden.me.uk>
	<4222a8490901281748w7f4b0b9cw662982ebdac01774@mail.gmail.com>
	<20090129015703.GG9167@steerpike.home.puzzling.org>
	<4222a8490901281806s34c259f7xbbbfaf62b67ad5da@mail.gmail.com>
	<87vdryx2ja.fsf@benfinney.id.au>
	<20090129195401.GB60708@wind.teleri.net>
	<87fxj1vm2m.fsf@benfinney.id.au> <87y6wtczka.fsf@xemacs.org>
	<878wottqra.fsf@benfinney.id.au> <87r62lcqu9.fsf@xemacs.org>
Message-ID: <217accd40901301338t65c1729cp8db16623c423346c@mail.gmail.com>

It's tough to follow this whole thread as it has spread all over the
place, so my apologies if this has been mentioned.

Why not try to make both the unix only daemon and the generic service
available?  The latter could use the former for its implementation?

  Nate


From ben+python at benfinney.id.au  Fri Jan 30 23:40:20 2009
From: ben+python at benfinney.id.au (Ben Finney)
Date: Sat, 31 Jan 2009 09:40:20 +1100
Subject: [Python-ideas] Draft PEP: Standard daemon process library
References: <497F351D.30700@timgolden.me.uk>
	<4222a8490901281748w7f4b0b9cw662982ebdac01774@mail.gmail.com>
	<20090129015703.GG9167@steerpike.home.puzzling.org>
	<4222a8490901281806s34c259f7xbbbfaf62b67ad5da@mail.gmail.com>
	<87vdryx2ja.fsf@benfinney.id.au>
	<20090129195401.GB60708@wind.teleri.net>
	<87fxj1vm2m.fsf@benfinney.id.au> <87y6wtczka.fsf@xemacs.org>
	<878wottqra.fsf@benfinney.id.au> <87r62lcqu9.fsf@xemacs.org>
	<217accd40901301338t65c1729cp8db16623c423346c@mail.gmail.com>
Message-ID: <87ab981kq3.fsf@benfinney.id.au>

nathan binkert <nate at binkert.org> writes:

> It's tough to follow this whole thread as it has spread all over the
> place, so my apologies if this has been mentioned.
> 
> Why not try to make both the unix only daemon and the generic service
> available?  The latter could use the former for its implementation?

Yes, this is my position. I'm glad to see that I'm not alone, thanks :-)

-- 
 \      ?Puritanism: The haunting fear that someone, somewhere, may be |
  `\                                         happy.? ?Henry L. Mencken |
_o__)                                                                  |
Ben Finney



From gagsl-py2 at yahoo.com.ar  Sat Jan 31 11:36:16 2009
From: gagsl-py2 at yahoo.com.ar (Gabriel Genellina)
Date: Sat, 31 Jan 2009 08:36:16 -0200
Subject: [Python-ideas] Draft PEP (version 0.4): Standard daemon process
	library
References: <87wscj11fl.fsf@benfinney.id.au> <87iqnywsx2.fsf@benfinney.id.au>
Message-ID: <op.uomaeqtzx6zn5v@gabriel2.softlabbsas.com.ar>

En Thu, 29 Jan 2009 04:03:05 -0200, Ben Finney  
<ben+python-TqlCGjI+HWGnbCmf7pGUHw at public.gmane.org> escribi?:

> :PEP:               XXX
> :Title:             Standard daemon process library
> :Version:           0.4

After reading this PEP, I see it as a big list of implementation details.  
A more high level view would be more suitable for a multiplatform  
implementation.
Like when using most library modules, I trust the authors, I hope they've  
done their job well, that they've read their Stevens and Pietrek, and the  
code follows the best practices. For the most part, I don't care about the  
implementation - just that it don't gets in my way.

I'd say that a daemon is just a background process, detached from any  
terminal and without any user interactivity, that keeps running  
independently of the logged user. *How* to get there, it's the library job.

So I need a method to "become a daemon". Stopping a daemon process must be  
done orderly so a "stop" method is required too. It needs some way to  
react to external requests (signals, or control requests on Windows):  
they're a short list, we can use methods following a naming convention  
(e.g. "on_reload"). And that's all, at least conceptually; probably we  
need some attributes to customize the behaviour.

I think it's a lot easier to implement such abstract view on different  
systems. Excluding Windows right from the start --because of so many  
details that should be emulated, and are mostly irrelevant-- would be a  
bad idea.

-- 
Gabriel Genellina