From Nikolaus at rath.org  Mon Oct 18 04:02:45 2010
From: Nikolaus at rath.org (Nikolaus Rath)
Date: Sun, 17 Oct 2010 22:02:45 -0400
Subject: [capi-sig] What can I do without GIL?
Message-ID: <4CBBAAC5.6010806@rath.org>

Hello,

I a thread does not hold the GIL, is it totally forbidden to call any
Py* API functions at all, or are there some exceptions?

More concretely, am I allowed to create fresh Python objects with
PyBytes_FromString et. al. without holding the GIL?


Thanks,

   -Nikolaus

-- 
 ?Time flies like an arrow, fruit flies like a Banana.?

  PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6  02CF A9AD B7F8 AE4E 425C

From Nikolaus at rath.org  Mon Oct 18 04:00:36 2010
From: Nikolaus at rath.org (Nikolaus Rath)
Date: Sun, 17 Oct 2010 22:00:36 -0400
Subject: [capi-sig] How to initialize state for C created threads
Message-ID: <4CBBAA44.4020709@rath.org>

Hello,

I am starting threads in a C extension. If I understand correctly,
successive sequences of PyGILState_Ensure() and PyGILState_Release will
create and destroy the Python thread state over and over again, so e.g.
objects stored with threading.Local() will not persist.

Can someone explain to me how I can initialize and destroy the thread
state manually (i.e., right after creating and destroying the thread
with pthreads), so that PyGILState_* will just acquire and release the GIL?

I guess I have to call PyThreadState_New(), but I am not sure what to do
with the returned pointer and where to get the PyInterpreter argument
from...


Thanks,

   -Nikolaus

-- 
 ?Time flies like an arrow, fruit flies like a Banana.?

  PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6  02CF A9AD B7F8 AE4E 425C

From skippy.hammond at gmail.com  Mon Oct 18 05:55:07 2010
From: skippy.hammond at gmail.com (Mark Hammond)
Date: Mon, 18 Oct 2010 14:55:07 +1100
Subject: [capi-sig] How to initialize state for C created threads
In-Reply-To: <4CBBAA44.4020709@rath.org>
References: <4CBBAA44.4020709@rath.org>
Message-ID: <4CBBC51B.1070106@gmail.com>

On 18/10/2010 1:00 PM, Nikolaus Rath wrote:
> Hello,
>
> I am starting threads in a C extension. If I understand correctly,
> successive sequences of PyGILState_Ensure() and PyGILState_Release will
> create and destroy the Python thread state over and over again, so e.g.
> objects stored with threading.Local() will not persist.
>
> Can someone explain to me how I can initialize and destroy the thread
> state manually (i.e., right after creating and destroying the thread
> with pthreads), so that PyGILState_* will just acquire and release the GIL?

You can just Ensure as your thread starts and the Release as it 
terminates.  You can then use the normal 
Py_BEGIN_ALLOW_THREADS/Py_END_ALLOW_THREADS macros to release the GIL as 
appropriate.

HTH,

Mark

From hniksic at xemacs.org  Mon Oct 18 09:51:24 2010
From: hniksic at xemacs.org (Hrvoje Niksic)
Date: Mon, 18 Oct 2010 09:51:24 +0200
Subject: [capi-sig] What can I do without GIL?
In-Reply-To: <4CBBAAC5.6010806@rath.org> (Nikolaus Rath's message of "Sun, 17
	Oct 2010 22:02:45 -0400")
References: <4CBBAAC5.6010806@rath.org>
Message-ID: <8762x07xc3.fsf@xemacs.org>

Nikolaus Rath <Nikolaus at rath.org> writes:

> I a thread does not hold the GIL, is it totally forbidden to call any
> Py* API functions at all, or are there some exceptions?
>
> More concretely, am I allowed to create fresh Python objects with
> PyBytes_FromString et. al. without holding the GIL?

You are not allowed to create fresh Python objects without holding the
GIL.  The objects might implement a caching strategy which would surely
rely on the GIL for locking.

The only Python API calls that could conceivably be safe without holding
the GIL are accessors to C members of immutable objects whose reference
you're owning - say, PyString_AS_STRING, or PyInt_AS_LONG.

From python_capi at behnel.de  Mon Oct 18 12:35:44 2010
From: python_capi at behnel.de (Stefan Behnel)
Date: Mon, 18 Oct 2010 12:35:44 +0200
Subject: [capi-sig] What can I do without GIL?
In-Reply-To: <4CBBAAC5.6010806@rath.org>
References: <4CBBAAC5.6010806@rath.org>
Message-ID: <4CBC2300.7080907@behnel.de>

Nikolaus Rath, 18.10.2010 04:02:
> I a thread does not hold the GIL, is it totally forbidden to call any
> Py* API functions at all, or are there some exceptions?
>
> More concretely, am I allowed to create fresh Python objects with
> PyBytes_FromString et. al. without holding the GIL?

No. Anything that requires Python memory management and especially 
reference counting must be protected by the GIL. You can read the C pointer 
and length of a Python string without the GIL (using the C-API macros) and 
you can acquire and release thread locks, but you can't create new objects.

Even simple looking functions may raise exceptions in some corner cases 
(such as memory errors), which also requires the GIL to work. So don't 
expect there to be much that you can do without holding it. The best way to 
tell is to look through the CPython implementation of the function you want 
to call, including all other functions that it calls, and to check for 
anything that looks like reference counting or exception handling.

Stefan

From stefan_ml at behnel.de  Mon Oct 18 08:53:26 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 18 Oct 2010 08:53:26 +0200
Subject: [capi-sig] What can I do without GIL?
In-Reply-To: <4CBBAAC5.6010806@rath.org>
References: <4CBBAAC5.6010806@rath.org>
Message-ID: <4CBBEEE6.9030508@behnel.de>

Nikolaus Rath, 18.10.2010 04:02:
> I a thread does not hold the GIL, is it totally forbidden to call any
> Py* API functions at all, or are there some exceptions?
>
> More concretely, am I allowed to create fresh Python objects with
> PyBytes_FromString et. al. without holding the GIL?

No. Anything that requires Python memory management and especially 
reference counting must be protected by the GIL. You can read the C pointer 
and length of a Python string without the GIL (using the C-API macros) and 
you can acquire and release thread locks, but you can't create new objects.

Even simple looking functions may raise exceptions in some corner cases 
(such as memory errors), which also requires the GIL to work. So don't 
expect there to be much that you can do without holding it. The best way to 
tell is to look through the CPython implementation of the function you want 
to call, including all other functions that it calls, and to check for 
anything that looks like reference counting or exception handling.

Stefan

From Nikolaus at rath.org  Sun Oct 24 18:09:49 2010
From: Nikolaus at rath.org (Nikolaus Rath)
Date: Sun, 24 Oct 2010 12:09:49 -0400
Subject: [capi-sig] What can I do without GIL?
In-Reply-To: <4CBBEEE6.9030508@behnel.de>
References: <4CBBAAC5.6010806@rath.org> <4CBBEEE6.9030508@behnel.de>
Message-ID: <4CC45A4D.4080400@rath.org>

On 10/18/2010 02:53 AM, Stefan Behnel wrote:
> Nikolaus Rath, 18.10.2010 04:02:
>> I a thread does not hold the GIL, is it totally forbidden to call any
>> Py* API functions at all, or are there some exceptions?
>>
>> More concretely, am I allowed to create fresh Python objects with
>> PyBytes_FromString et. al. without holding the GIL?
> 
> No. Anything that requires Python memory management and especially
> reference counting must be protected by the GIL. You can read the C
> pointer and length of a Python string without the GIL (using the C-API
> macros) and you can acquire and release thread locks, but you can't
> create new objects.

I see, thanks.

I'd like to follow up with another question directly: if I call
PyBytes_AsString() on a Python object (and keep a reference to that
object), can I rely on the char* to stay valid when I am releasing the GIL?

I just learned that in OCAML, the interpreter may happily move data
around so that pointers are not guaranteed to stay valid if the GIL is
released.

Is this a problem with Python as well, or can I rely on the string
staying at the same memory location all the time?


Thanks,

   -Nikolaus

-- 
 ?Time flies like an arrow, fruit flies like a Banana.?

  PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6  02CF A9AD B7F8 AE4E 425C

From python_capi at behnel.de  Sun Oct 24 18:57:11 2010
From: python_capi at behnel.de (Stefan Behnel)
Date: Sun, 24 Oct 2010 18:57:11 +0200
Subject: [capi-sig] What can I do without GIL?
In-Reply-To: <4CC45A4D.4080400@rath.org>
References: <4CBBAAC5.6010806@rath.org> <4CBBEEE6.9030508@behnel.de>
	<4CC45A4D.4080400@rath.org>
Message-ID: <4CC46567.8060202@behnel.de>

Nikolaus Rath, 24.10.2010 18:09:
> On 10/18/2010 02:53 AM, Stefan Behnel wrote:
>> Nikolaus Rath, 18.10.2010 04:02:
>>> I a thread does not hold the GIL, is it totally forbidden to call any
>>> Py* API functions at all, or are there some exceptions?
>>>
>>> More concretely, am I allowed to create fresh Python objects with
>>> PyBytes_FromString et. al. without holding the GIL?
>>
>> No. Anything that requires Python memory management and especially
>> reference counting must be protected by the GIL. You can read the C
>> pointer and length of a Python string without the GIL (using the C-API
>> macros) and you can acquire and release thread locks, but you can't
>> create new objects.
>
> I'd like to follow up with another question directly: if I call
> PyBytes_AsString() on a Python object (and keep a reference to that
> object), can I rely on the char* to stay valid when I am releasing the GIL?

First of all, you shouldn't call PyBytes_AsString() without holding the GIL 
because it may raise an exception. You can call PyBytes_AS_STRING(), 
though. As I said, check the implementation.


> I just learned that in OCAML, the interpreter may happily move data
> around so that pointers are not guaranteed to stay valid if the GIL is
> released.
>
> Is this a problem with Python as well, or can I rely on the string
> staying at the same memory location all the time?

CPython doesn't do that. Strings are immutable, so they never need to be 
reallocated by Python runtime operations. And CPython is tuned for making 
it easy to interact with C code through its C-API. One feature is that the 
memory management doesn't move stuff around that may be referenced by some 
C code somewhere.

Stefan

From amnorvend at gmail.com  Sun Oct 24 19:22:08 2010
From: amnorvend at gmail.com (Jason Baker)
Date: Sun, 24 Oct 2010 10:22:08 -0700
Subject: [capi-sig] What can I do without GIL?
In-Reply-To: <4CC45A4D.4080400@rath.org>
References: <4CBBAAC5.6010806@rath.org> <4CBBEEE6.9030508@behnel.de>
	<4CC45A4D.4080400@rath.org>
Message-ID: <F0C9F783-1B73-4012-89AD-11FEEDCE4FA7@gmail.com>

On Oct 24, 2010, at 9:09 AM, Nikolaus Rath <Nikolaus at rath.org> wrote:
> I just learned that in OCAML, the interpreter may happily move data
> around so that pointers are not guaranteed to stay valid if the GIL is
> released.
> 
> Is this a problem with Python as well, or can I rely on the string
> staying at the same memory location all the time?

OCaml has a compacting garbage collector.  Every once in a while, it will move everything around to reduce memory fragmentation.  As far as I know, Python doesn't do this.

From Nikolaus at rath.org  Sun Oct 24 19:18:21 2010
From: Nikolaus at rath.org (Nikolaus Rath)
Date: Sun, 24 Oct 2010 13:18:21 -0400
Subject: [capi-sig] What can I do without GIL?
In-Reply-To: <4CC46567.8060202@behnel.de>
References: <4CBBAAC5.6010806@rath.org> <4CBBEEE6.9030508@behnel.de>
	<4CC45A4D.4080400@rath.org> <4CC46567.8060202@behnel.de>
Message-ID: <4CC46A5D.5040001@rath.org>

On 10/24/2010 12:57 PM, Stefan Behnel wrote:
> Nikolaus Rath, 24.10.2010 18:09:
>> On 10/18/2010 02:53 AM, Stefan Behnel wrote:
>>> Nikolaus Rath, 18.10.2010 04:02:
>>>> I a thread does not hold the GIL, is it totally forbidden to call any
>>>> Py* API functions at all, or are there some exceptions?
>>>>
>>>> More concretely, am I allowed to create fresh Python objects with
>>>> PyBytes_FromString et. al. without holding the GIL?
>>>
>>> No. Anything that requires Python memory management and especially
>>> reference counting must be protected by the GIL. You can read the C
>>> pointer and length of a Python string without the GIL (using the C-API
>>> macros) and you can acquire and release thread locks, but you can't
>>> create new objects.
>>
>> I'd like to follow up with another question directly: if I call
>> PyBytes_AsString() on a Python object (and keep a reference to that
>> object), can I rely on the char* to stay valid when I am releasing the
>> GIL?
> 
> First of all, you shouldn't call PyBytes_AsString() without holding the
> GIL because it may raise an exception. You can call PyBytes_AS_STRING(),
> though. As I said, check the implementation.

That's what I meant. I want to call PyBytes_* and *afterwards* release
the GIL.

>> I just learned that in OCAML, the interpreter may happily move data
>> around so that pointers are not guaranteed to stay valid if the GIL is
>> released.
>>
>> Is this a problem with Python as well, or can I rely on the string
>> staying at the same memory location all the time?
> 
> CPython doesn't do that. Strings are immutable, so they never need to be
> reallocated by Python runtime operations. And CPython is tuned for
> making it easy to interact with C code through its C-API. One feature is
> that the memory management doesn't move stuff around that may be
> referenced by some C code somewhere.

Great, thanks again!



   -Nikolaus

-- 
 ?Time flies like an arrow, fruit flies like a Banana.?

  PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6  02CF A9AD B7F8 AE4E 425C

From hniksic at xemacs.org  Mon Oct 25 12:36:16 2010
From: hniksic at xemacs.org (Hrvoje Niksic)
Date: Mon, 25 Oct 2010 12:36:16 +0200
Subject: [capi-sig] What can I do without GIL?
In-Reply-To: <4CC45A4D.4080400@rath.org> (Nikolaus Rath's message of "Sun, 24
	Oct 2010 12:09:49 -0400")
References: <4CBBAAC5.6010806@rath.org> <4CBBEEE6.9030508@behnel.de>
	<4CC45A4D.4080400@rath.org>
Message-ID: <87pquy36fz.fsf@xemacs.org>

Nikolaus Rath <Nikolaus at rath.org> writes:

> I'd like to follow up with another question directly: if I call
> PyBytes_AsString() on a Python object (and keep a reference to that
> object), can I rely on the char* to stay valid when I am releasing the
> GIL?
[...]
> Is this a problem with Python as well, or can I rely on the string
> staying at the same memory location all the time?

CPython string data is stored inside the PyStringObject structure, so it
is pretty much guaranteed to be immovable in the current implementation.

typedef struct {
    PyObject_VAR_HEAD
    long ob_shash;
    int ob_sstate;
    char ob_sval[1];    // PyString_AS_STRING returns pointer to this member
} PyStringObject;

From amnorvend at gmail.com  Sat Oct 30 20:14:45 2010
From: amnorvend at gmail.com (Jason Baker)
Date: Sat, 30 Oct 2010 11:14:45 -0700
Subject: [capi-sig] How do I make a subclass of a C class iterable?
Message-ID: <AANLkTimo8AQAyL3B+N-b9nxQAQiLOZb0tGSJSc+Apmj8@mail.gmail.com>

My package has an optional C extension.  Here's an abbreviated version of
what I have:

try:
    from pysistence._persistent_list import PList as BasePList
except ImportError:
    class BasePList(object):
        __slots__ = ['_first', '_rest']
        def __init__(self, first, rest=None):
            self._first = first
            self._rest = rest
        def cons(self, item):
            return self.__class__(item, self)

class PList(BasePList):
    ...
    def __iter__(self):
        ....
    ...

So what I'm basically doing is subclassing a base class (which is defined in
Python or C) and adding an __iter__ method.  This works fine if I don't use
the C extension, but if I do use it I get the following error:

Traceback (most recent call last):
  File
"/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/timeit.py",
line 297, in main
    x = t.timeit(number)
  File
"/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/timeit.py",
line 193, in timeit
    timing = self.inner(it, self.timer)
  File "<timeit-src>", line 11, in inner
    for i in iter(datalist):
TypeError: 'pysistence.persistent_list.PList' object is not iterable

Is there a way to define the __iter__ method in Python, or am I going to
have to do that in C?