From sturla at molden.no  Mon Jul  2 14:49:40 2012
From: sturla at molden.no (Sturla Molden)
Date: Mon, 02 Jul 2012 14:49:40 +0200
Subject: [Cython] Automatic C++ conversions
In-Reply-To: <4FEE34E8.9050807@behnel.de>
References: <CADiQ+QAR7y+pxbD2_uPoKfA+r5MmHT07=DLQoTEStYLfE1x_+g@mail.gmail.com>
	<4FEC29E9.9030609@behnel.de>
	<CADiQ+QCdq52RHSmoK-mOkEF96P7yGj25zTB+wfd+LmgzCNcNtA@mail.gmail.com>
	<4FEC499F.5050505@behnel.de>
	<CADiQ+QBTj1M3K9NjqXVBiHeL0kR_MHmhqnAhaN2mkKbb-Ke0wA@mail.gmail.com>
	<4FEE34E8.9050807@behnel.de>
Message-ID: <4FF198E4.9020304@molden.no>

On 30.06.2012 01:06, Stefan Behnel wrote:

>> std::string<=>  bytes
>> std::map<=>  dict
>> iterable =>  std::vector =>  list
>> iterable =>  std::list =>  list
>> iterable =>  std::set =>  set
>> 2-iterable =>  std::pair =>  2-tuple
>
> Very cool.

I think (in C++11) std::unordered_set and std::unordered_map should be 
used instead. They are hash-based with O(1) lookup.

std::set and std::map are binary search threes with average O(log n) 
lookup and worst-case O(n**2).

Also beware that C++11 has a std:tuple type.


Sturla Molden

From sturla at molden.no  Mon Jul  2 14:52:09 2012
From: sturla at molden.no (Sturla Molden)
Date: Mon, 02 Jul 2012 14:52:09 +0200
Subject: [Cython] Automatic C++ conversions
In-Reply-To: <4FF198E4.9020304@molden.no>
References: <CADiQ+QAR7y+pxbD2_uPoKfA+r5MmHT07=DLQoTEStYLfE1x_+g@mail.gmail.com>
	<4FEC29E9.9030609@behnel.de>
	<CADiQ+QCdq52RHSmoK-mOkEF96P7yGj25zTB+wfd+LmgzCNcNtA@mail.gmail.com>
	<4FEC499F.5050505@behnel.de>
	<CADiQ+QBTj1M3K9NjqXVBiHeL0kR_MHmhqnAhaN2mkKbb-Ke0wA@mail.gmail.com>
	<4FEE34E8.9050807@behnel.de> <4FF198E4.9020304@molden.no>
Message-ID: <4FF19979.3020501@molden.no>

On 02.07.2012 14:49, Sturla Molden wrote:

> I think (in C++11) std::unordered_set and std::unordered_map should be
> used instead. They are hash-based with O(1) lookup.
>
> std::set and std::map are binary search threes with average O(log n)
> lookup and worst-case O(n**2).

Sorry typo, that should be worst-case O(n).


Sturla

From robertwb at gmail.com  Mon Jul  2 18:09:00 2012
From: robertwb at gmail.com (Robert Bradshaw)
Date: Mon, 2 Jul 2012 09:09:00 -0700
Subject: [Cython] Automatic C++ conversions
In-Reply-To: <4FF198E4.9020304@molden.no>
References: <CADiQ+QAR7y+pxbD2_uPoKfA+r5MmHT07=DLQoTEStYLfE1x_+g@mail.gmail.com>
	<4FEC29E9.9030609@behnel.de>
	<CADiQ+QCdq52RHSmoK-mOkEF96P7yGj25zTB+wfd+LmgzCNcNtA@mail.gmail.com>
	<4FEC499F.5050505@behnel.de>
	<CADiQ+QBTj1M3K9NjqXVBiHeL0kR_MHmhqnAhaN2mkKbb-Ke0wA@mail.gmail.com>
	<4FEE34E8.9050807@behnel.de> <4FF198E4.9020304@molden.no>
Message-ID: <CADiQ+QBe1rRy+JJ2gaKnPJ-P8uWeGwmKpTGmEhZDGSf4Xh_fNQ@mail.gmail.com>

On Mon, Jul 2, 2012 at 5:49 AM, Sturla Molden <sturla at molden.no> wrote:
> On 30.06.2012 01:06, Stefan Behnel wrote:
>
>>> std::string<=> ?bytes
>>> std::map<=> ?dict
>>> iterable => ?std::vector => ?list
>>> iterable => ?std::list => ?list
>>> iterable => ?std::set => ?set
>>> 2-iterable => ?std::pair => ?2-tuple
>>
>>
>> Very cool.
>
>
> I think (in C++11) std::unordered_set and std::unordered_map should be used
> instead. They are hash-based with O(1) lookup.
>
> std::set and std::map are binary search threes with average O(log n) lookup
> and worst-case O(n**2).
>
> Also beware that C++11 has a std:tuple type.

The object => C++ coercion is always explicit, there's no choice made
here. This is used to do, e.g.

    cdef map<int, vector<double>> my_cpp_map = o

or

    cdef extern from "mylibrary.h":
        cdef map<long, double> my_func(vector<long>)

    def f(o):
        return my_func(o)

It would make sense to support these unordered versions as well, and
fairly straightforward.

- Robert

From lists at onerussian.com  Tue Jul  3 00:56:44 2012
From: lists at onerussian.com (Yaroslav Halchenko)
Date: Mon, 2 Jul 2012 18:56:44 -0400
Subject: [Cython] e:  Automatic C++ conversions
Message-ID: <20120702225644.GR5788@onerussian.com>

I was thinking about updating Debian package for cython 0.16 but ran into
the failing unittests so decided first to check with experts: anything obvious
which comes to mind from seeing those?

...
compiling (c) and running sequential_parallel ... sequential_parallel.c: In function '__pyx_pf_19sequential_parallel_56test_chunksize':
sequential_parallel.c:12405:7: warning: variable '__pyx_t_6' set but not used [-Wunused-but-set-variable]
sequential_parallel.c:12404:7: warning: variable '__pyx_t_5' set but not used [-Wunused-but-set-variable]
sequential_parallel.c:12402:7: warning: variable '__pyx_t_3' set but not used [-Wunused-but-set-variable]
sequential_parallel.c: In function '__pyx_pw_19sequential_parallel_25test_nested_break_continue':
sequential_parallel.c:5887:29: warning: '__pyx_v_j' may be used uninitialized in this function [-Wmaybe-uninitialized]
sequential_parallel.c:5590:7: note: '__pyx_v_j' was declared here
sequential_parallel.c: In function '__pyx_f_19sequential_parallel_parallel_exc_cpdef_unnested.isra.29':
sequential_parallel.c:8915:3: warning: '__pyx_r' may be used uninitialized in this function [-Wmaybe-uninitialized]
sequential_parallel.c:8681:7: note: '__pyx_r' was declared here
sequential_parallel.c: In function '__pyx_f_19sequential_parallel_parallel_exc_cpdef.isra.30':
sequential_parallel.c:8614:3: warning: '__pyx_r' may be used uninitialized in this function [-Wmaybe-uninitialized]
sequential_parallel.c:8259:7: note: '__pyx_r' was declared here
sequential_parallel.c: In function '__pyx_pw_19sequential_parallel_33test_parallel_exc_cdef':
sequential_parallel.c:8222:78: warning: '__pyx_r' may be used uninitialized in this function [-Wuninitialized]
sequential_parallel.c:7946:7: note: '__pyx_r' was declared here
sequential_parallel.c:8233:69: warning: '__pyx_r' may be used uninitialized in this function [-Wuninitialized]
sequential_parallel.c:7578:7: note: '__pyx_r' was declared here
Fatal Python error: deletion of interned string failed
Compiling /tmp/buildd/.cython/inline/_cython_inline_a998a12d3d1450b2368ca75e07a27942.pyx because it changed.
Cythonizing /tmp/buildd/.cython/inline/_cython_inline_a998a12d3d1450b2368ca75e07a27942.pyx
ERROR
compiling (cpp) and running sequential_parallel ... sequential_parallel.cpp: In function 'PyObject* __pyx_pf_19sequential_parallel_56test_chunksize(PyObject*)':
sequential_parallel.cpp:12402:7: warning: variable '__pyx_t_3' set but not used [-Wunused-but-set-variable]
sequential_parallel.cpp:12404:7: warning: variable '__pyx_t_5' set but not used [-Wunused-but-set-variable]
sequential_parallel.cpp:12405:7: warning: variable '__pyx_t_6' set but not used [-Wunused-but-set-variable]
sequential_parallel.cpp: In function 'PyObject* __pyx_pw_19sequential_parallel_25test_nested_break_continue(PyObject*, PyObject*)':
sequential_parallel.cpp:5887:39: warning: '__pyx_v_j' may be used uninitialized in this function [-Wmaybe-uninitialized]
sequential_parallel.cpp:5590:7: note: '__pyx_v_j' was declared here
sequential_parallel.cpp: In function 'int _ZL48__pyx_f_19sequential_parallel_parallel_exc_cpdefi.isra.30()':
sequential_parallel.cpp:8614:10: warning: '__pyx_r' may be used uninitialized in this function [-Wmaybe-uninitialized]
sequential_parallel.cpp:8259:7: note: '__pyx_r' was declared here
sequential_parallel.cpp: In function 'int _ZL57__pyx_f_19sequential_parallel_parallel_exc_cpdef_unnestedi.isra.29()':
sequential_parallel.cpp:8915:10: warning: '__pyx_r' may be used uninitialized in this function [-Wmaybe-uninitialized]
sequential_parallel.cpp:8681:7: note: '__pyx_r' was declared here
sequential_parallel.cpp: In function 'PyObject* __pyx_pw_19sequential_parallel_33test_parallel_exc_cdef(PyObject*, PyObject*)':
sequential_parallel.cpp:8222:75: warning: '__pyx_r' may be used uninitialized in this function [-Wuninitialized]
sequential_parallel.cpp:7946:7: note: '__pyx_r' was declared here
sequential_parallel.cpp:8233:66: warning: '__pyx_r' may be used uninitialized in this function [-Wuninitialized]
sequential_parallel.cpp:7578:7: note: '__pyx_r' was declared here
ERROR
...
======================================================================
ERROR: compiling (c) and running parallel
----------------------------------------------------------------------
Traceback (most recent call last):
  File "runtests.py", line 707, in run
    self.run_tests(result)
  File "runtests.py", line 723, in run_tests
    self.run_doctests(self.module, result)
  File "runtests.py", line 729, in run_doctests
    run_forked_test(result, run_test, self.shortDescription(), self.fork)
  File "runtests.py", line 778, in run_forked_test
    (module_name, result_code & 255))
Exception: Tests in module 'parallel' were unexpectedly killed by signal 11

======================================================================
ERROR: compiling (cpp) and running parallel
----------------------------------------------------------------------
Traceback (most recent call last):
  File "runtests.py", line 707, in run
    self.run_tests(result)
  File "runtests.py", line 723, in run_tests
    self.run_doctests(self.module, result)
  File "runtests.py", line 729, in run_doctests
    run_forked_test(result, run_test, self.shortDescription(), self.fork)
  File "runtests.py", line 778, in run_forked_test
    (module_name, result_code & 255))
Exception: Tests in module 'parallel' were unexpectedly killed by signal 11

======================================================================
ERROR: compiling (c) and running sequential_parallel
----------------------------------------------------------------------
Traceback (most recent call last):
  File "runtests.py", line 707, in run
    self.run_tests(result)
  File "runtests.py", line 723, in run_tests
    self.run_doctests(self.module, result)
  File "runtests.py", line 729, in run_doctests
    run_forked_test(result, run_test, self.shortDescription(), self.fork)
  File "runtests.py", line 778, in run_forked_test
    (module_name, result_code & 255))
Exception: Tests in module 'sequential_parallel' were unexpectedly killed by signal 6

======================================================================
ERROR: compiling (cpp) and running sequential_parallel
----------------------------------------------------------------------
Traceback (most recent call last):
  File "runtests.py", line 707, in run
    self.run_tests(result)
  File "runtests.py", line 723, in run_tests
    self.run_doctests(self.module, result)
  File "runtests.py", line 729, in run_doctests
    run_forked_test(result, run_test, self.shortDescription(), self.fork)
  File "runtests.py", line 778, in run_forked_test
    (module_name, result_code & 255))
Exception: Tests in module 'sequential_parallel' were unexpectedly killed by signal 11

----------------------------------------------------------------------
Ran 6358 tests in 954.316s

FAILED (errors=4)


-- 
Yaroslav O. Halchenko
Postdoctoral Fellow,   Department of Psychological and Brain Sciences
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834                       Fax: +1 (603) 646-1419
WWW:   http://www.linkedin.com/in/yarik        

From lists at onerussian.com  Tue Jul  3 01:04:37 2012
From: lists at onerussian.com (Yaroslav Halchenko)
Date: Mon, 2 Jul 2012 19:04:37 -0400
Subject: [Cython] Exception: Tests in module 'parallel' were unexpectedly
 killed by signal 11
In-Reply-To: <20120702225644.GR5788@onerussian.com>
References: <20120702225644.GR5788@onerussian.com>
Message-ID: <20120702230437.GS5788@onerussian.com>

please pardon my sloppy cut/paste which ruined the subject line in the
original post ;)  So the topic is:

On Mon, 02 Jul 2012, Yaroslav Halchenko wrote:
> I was thinking about updating Debian package for cython 0.16 but ran into
> the failing unittests so decided first to check with experts: anything obvious
> which comes to mind from seeing those?

> ...
> compiling (c) and running sequential_parallel ... sequential_parallel.c: In function '__pyx_pf_19sequential_parallel_56test_chunksize':
> sequential_parallel.c:12405:7: warning: variable '__pyx_t_6' set but not used [-Wunused-but-set-variable]
> sequential_parallel.c:12404:7: warning: variable '__pyx_t_5' set but not used [-Wunused-but-set-variable]
> sequential_parallel.c:12402:7: warning: variable '__pyx_t_3' set but not used [-Wunused-but-set-variable]
> sequential_parallel.c: In function '__pyx_pw_19sequential_parallel_25test_nested_break_continue':
> sequential_parallel.c:5887:29: warning: '__pyx_v_j' may be used uninitialized in this function [-Wmaybe-uninitialized]
> sequential_parallel.c:5590:7: note: '__pyx_v_j' was declared here
> sequential_parallel.c: In function '__pyx_f_19sequential_parallel_parallel_exc_cpdef_unnested.isra.29':
> sequential_parallel.c:8915:3: warning: '__pyx_r' may be used uninitialized in this function [-Wmaybe-uninitialized]
> sequential_parallel.c:8681:7: note: '__pyx_r' was declared here
> sequential_parallel.c: In function '__pyx_f_19sequential_parallel_parallel_exc_cpdef.isra.30':
> sequential_parallel.c:8614:3: warning: '__pyx_r' may be used uninitialized in this function [-Wmaybe-uninitialized]
> sequential_parallel.c:8259:7: note: '__pyx_r' was declared here
> sequential_parallel.c: In function '__pyx_pw_19sequential_parallel_33test_parallel_exc_cdef':
> sequential_parallel.c:8222:78: warning: '__pyx_r' may be used uninitialized in this function [-Wuninitialized]
> sequential_parallel.c:7946:7: note: '__pyx_r' was declared here
> sequential_parallel.c:8233:69: warning: '__pyx_r' may be used uninitialized in this function [-Wuninitialized]
> sequential_parallel.c:7578:7: note: '__pyx_r' was declared here
> Fatal Python error: deletion of interned string failed
> Compiling /tmp/buildd/.cython/inline/_cython_inline_a998a12d3d1450b2368ca75e07a27942.pyx because it changed.
> Cythonizing /tmp/buildd/.cython/inline/_cython_inline_a998a12d3d1450b2368ca75e07a27942.pyx
> ERROR
> ...
> ======================================================================
> ERROR: compiling (c) and running parallel
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File "runtests.py", line 707, in run
>     self.run_tests(result)
>   File "runtests.py", line 723, in run_tests
>     self.run_doctests(self.module, result)
>   File "runtests.py", line 729, in run_doctests
>     run_forked_test(result, run_test, self.shortDescription(), self.fork)
>   File "runtests.py", line 778, in run_forked_test
>     (module_name, result_code & 255))
> Exception: Tests in module 'parallel' were unexpectedly killed by signal 11

> ======================================================================
-- 
Yaroslav O. Halchenko
Postdoctoral Fellow,   Department of Psychological and Brain Sciences
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834                       Fax: +1 (603) 646-1419
WWW:   http://www.linkedin.com/in/yarik        

From stefan_ml at behnel.de  Tue Jul  3 06:13:09 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 03 Jul 2012 06:13:09 +0200
Subject: [Cython] Exception: Tests in module 'parallel' were
 unexpectedly killed by signal 11
In-Reply-To: <20120702230437.GS5788@onerussian.com>
References: <20120702225644.GR5788@onerussian.com>
	<20120702230437.GS5788@onerussian.com>
Message-ID: <4FF27155.5030900@behnel.de>

Yaroslav Halchenko, 03.07.2012 01:04:
> please pardon my sloppy cut/paste which ruined the subject line in the
> original post ;)  So the topic is:
> 
> On Mon, 02 Jul 2012, Yaroslav Halchenko wrote:
>> I was thinking about updating Debian package for cython 0.16 but ran into
>> the failing unittests so decided first to check with experts: anything obvious
>> which comes to mind from seeing those?

Yes, they are known to be broken currently.

https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests/

AFAIK, Mark is working on fixing them.

Stefan

From stefan_ml at behnel.de  Tue Jul  3 06:58:35 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 03 Jul 2012 06:58:35 +0200
Subject: [Cython] Cython methods for C/C++ types
Message-ID: <4FF27BFB.80508@behnel.de>

Hi,

the discussion on allowing for automatic user provided ways to coerce
between Python types and C++ types got me thinking that this might hide a
rather interesting general feature: methods for low-level types. I faintly
remember that this idea has come up in our discussions before, but here's a
draft CEP for it:

http://wiki.cython.org/enhancements/ctypemethods

Basically, it would allow writing this in .pxd files:

"""
cdef extern from "...":
    cdef cppclass MyClass:
        cdef int cython_method_here(self):
            return 1

ctypedef double mydouble:
    cdef double cython_method_here(self):
        return self ** 2

cdef struct mystruct:
    int x, y

    cdef int cython_method_here(self):
        return self.x + self.y

cdef union myunion:
    int a
    double b

    cdef int cython_method_here(self):
        return self.a if ImSureImAnInt else <int>self.b
"""

The C code for these methods would then be generated into the modules that
use these types, similar to what we allow with "__getbuffer__()". Calls
would be direct function calls as with "final" methods.

I think it fits into what's there in a very natural way.

Stefan

From stefan_ml at behnel.de  Tue Jul  3 09:14:43 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 03 Jul 2012 09:14:43 +0200
Subject: [Cython] [cython-users] C++: how to handle failures of 'new'?
In-Reply-To: <CADiQ+QAFYMgycY5-RkTf7SOwDb0-qM93Y8+sEzA=+1P9DSyOyA@mail.gmail.com>
References: <4FECA49B.8090404@behnel.de>
	<CADiQ+QBeBw=FYsNPpSU_OJOU48m7E1=csxMxfenMQLs2jhJgPw@mail.gmail.com>
	<4FED40E1.1040301@behnel.de>
	<CADiQ+QAFYMgycY5-RkTf7SOwDb0-qM93Y8+sEzA=+1P9DSyOyA@mail.gmail.com>
Message-ID: <4FF29BE3.6030201@behnel.de>

Robert Bradshaw, 29.06.2012 11:08:
> On Thu, Jun 28, 2012 at 10:45 PM, Stefan Behnel wrote:
>> Robert Bradshaw, 28.06.2012 21:46:
>>> On Thu, Jun 28, 2012 at 11:38 AM, Stefan Behnel wrote:
>>>> currently, when I write "new CppClass()" in Cython, it generates a straight
>>>> call to the "new" operator. It doesn't do any error handling. And the
>>>> current documentation doesn't even mention this case.
>>>>
>>>> Is there a "standard" way to handle this? It seems that C++ has different
>>>> ways to deal with failures here but raises an exception by default. Would
>>>> you declare the constructor(s) with an "except +MemoryError"? Is there a
>>>> reason Cython shouldn't be doing this automatically (if nothing else was
>>>> declared) ?
>>>
>>> I think it certainly makes sense to declare the default constructor as
>>> "except +" (and std::bad_alloc should become MemoryError),
>>
>> Right. The code in the constructor can raise other exceptions that must
>> also be handled properly. An explicit "except +" will handle that.
>>
>>
>>> but whether
>>> to implicitly annotate declared constructors is less clear, especially
>>> as there's no way to un-annotate them.
>>
>> I agree, but sadly, it's the default behaviour that is wrong. I'm sure we
>> made lots of users run into this trap already. I fixed the documentation
>> for now, but the bottom line is that we require users to take care of
>> proper declarations themselves. Otherwise, the code that we generate is
>> incorrect, although it's 100% certain that an allocation error can occur,
>> even if the constructor code doesn't raise any exceptions itself.
> 
> This is always the case.

Sure, we always rely on correct declarations. However, this is a case where
we *know* that bad things can happen, even if there is no declaration for
them. It would be nice to play safe by default.

IMHO, only the second best solution would be to raise a warning when we
encounter a "new" without an exception declared, so that users are urged
into writing safe code.


>> Apparently, changing the behaviour of the "new" operator requires a special
>> annotation "std::nothrow", which then returns NULL on allocation failures.
>> You can pass that from Cython by hacking up a cname, e.g.
>>
>>    Rectangle "(std::nothrow) Rectangle" (int w, int h)
>>
>> I'm sure there are users out there who figured this out (I mean, I did...)
>> and use it in their code, so I agree that this isn't easy to handle because
>> Cython simply wouldn't know what the actual error behaviour is for a given
>> constructor and how to correctly detect an error.
>>
>> This problem applies only to heap allocation in that form. However, stack
>> allocation and the new exttype field allocation suffer from similar
>> problems when the default constructor raises an exception. Exttype fields
>> are a particularly nasty case because the user has no control over the
>> allocation. A C++ exception in the C++ class constructor would terminate
>> the exttype constructor unexpectedly and thus leak resources (in the best
>> case - no idea how CPython reacts if you throw a C++ exception through its
>> type instantiation code).
> 
> If the default constructor raises an exception then it should be
> declared (to not do so is an error on the users part). New raising
> bad_alloc is a bit of a special case, but doesn't appl to the stack or
> exttype allocations.

Right. In those two cases, it would definitely be the fault of the user.


>> Similarly, a C++ exception in the constructor of a stack allocated object
>> would then originate from the function entry code and potentially hit the
>> Python function wrapper etc. Again, potentially leaking resources or worse.
>>
>> To me, this sounds like we should do something about it. At least for the
>> implicit calls to the default constructor, we should generate "except +"
>> code automatically because there is no other way to handle them safely.
> 
> If no constructor is declared, it should be "except +" just to be
> safe

Ok, I'll fix it.


> but otherwise I don't see how this is any different than
> forgetting to declare exceptions on any other function. Unfortunately
> catching exceptions  (with custom per-object handling) on a set of
> stack allocated objects seems difficult if not impossible (without
> resorting to ugly hacks like using placement new everywhere).

I don't know what happens if a C++ exception is not being caught, but I
guess it would simply crash the application. That's a bit more visible than
just printing a warning when a Python exception is being ignored due to a
missing declaration. It's really unfortunate that our documentation didn't
even mention the need for this, because it's not immediately obvious that
Cython won't handle errors in "new", and testing for memory errors isn't
quite what people commonly do in their test suites.

Apart from that, I agree, users have to take care to properly declare the
API they are using.

Stefan

From markflorisson88 at gmail.com  Tue Jul  3 12:12:21 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Tue, 3 Jul 2012 11:12:21 +0100
Subject: [Cython] Cython methods for C/C++ types
In-Reply-To: <4FF27BFB.80508@behnel.de>
References: <4FF27BFB.80508@behnel.de>
Message-ID: <CANg26EV5f29XkKygVAdh=XYs43vhWzozpeywfFXxoaY_Mi4bCw@mail.gmail.com>

On 3 July 2012 05:58, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Hi,
>
> the discussion on allowing for automatic user provided ways to coerce
> between Python types and C++ types got me thinking that this might hide a
> rather interesting general feature: methods for low-level types. I faintly
> remember that this idea has come up in our discussions before, but here's a
> draft CEP for it:
>
> http://wiki.cython.org/enhancements/ctypemethods
>
> Basically, it would allow writing this in .pxd files:
>
> """
> cdef extern from "...":
>     cdef cppclass MyClass:
>         cdef int cython_method_here(self):
>             return 1
>
> ctypedef double mydouble:
>     cdef double cython_method_here(self):
>         return self ** 2
>
> cdef struct mystruct:
>     int x, y
>
>     cdef int cython_method_here(self):
>         return self.x + self.y
>
> cdef union myunion:
>     int a
>     double b
>
>     cdef int cython_method_here(self):
>         return self.a if ImSureImAnInt else <int>self.b
> """
>
> The C code for these methods would then be generated into the modules that
> use these types, similar to what we allow with "__getbuffer__()". Calls
> would be direct function calls as with "final" methods.
>
> I think it fits into what's there in a very natural way.
>
> Stefan
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

Looks a bit like C++. I have no strong feelings either way, it would
simplify the memoryview implementation slightly, which special-cases
the memoryview struct with method attributes. I would prefer to keep
the language simple however, and have people write functions operating
on structs. It's really the same thing written differently, and
methods don't really bring an advantage since the data in the struct
cannot be hidden or readonly.

From markflorisson88 at gmail.com  Tue Jul  3 12:15:51 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Tue, 3 Jul 2012 11:15:51 +0100
Subject: [Cython] Cython methods for C/C++ types
In-Reply-To: <4FF27BFB.80508@behnel.de>
References: <4FF27BFB.80508@behnel.de>
Message-ID: <CANg26EV8TDzwEFt5ZWWL6wNsAptdPqMMatHHQV3ae-0gkA8XcQ@mail.gmail.com>

On 3 July 2012 05:58, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Hi,
>
> the discussion on allowing for automatic user provided ways to coerce
> between Python types and C++ types got me thinking that this might hide a
> rather interesting general feature: methods for low-level types. I faintly
> remember that this idea has come up in our discussions before, but here's a
> draft CEP for it:
>
> http://wiki.cython.org/enhancements/ctypemethods

I see this CEP also mentions overriding C++ methods. But I think if it
doesn't support virtual methods, then overriding base class methods by
calling another function is pretty horrible.

> Basically, it would allow writing this in .pxd files:
>
> """
> cdef extern from "...":
>     cdef cppclass MyClass:
>         cdef int cython_method_here(self):
>             return 1
>
> ctypedef double mydouble:
>     cdef double cython_method_here(self):
>         return self ** 2
>
> cdef struct mystruct:
>     int x, y
>
>     cdef int cython_method_here(self):
>         return self.x + self.y
>
> cdef union myunion:
>     int a
>     double b
>
>     cdef int cython_method_here(self):
>         return self.a if ImSureImAnInt else <int>self.b
> """
>
> The C code for these methods would then be generated into the modules that
> use these types, similar to what we allow with "__getbuffer__()". Calls
> would be direct function calls as with "final" methods.
>
> I think it fits into what's there in a very natural way.
>
> Stefan
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

From markflorisson88 at gmail.com  Tue Jul  3 12:16:50 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Tue, 3 Jul 2012 11:16:50 +0100
Subject: [Cython] Exception: Tests in module 'parallel' were
 unexpectedly killed by signal 11
In-Reply-To: <20120702230437.GS5788@onerussian.com>
References: <20120702225644.GR5788@onerussian.com>
	<20120702230437.GS5788@onerussian.com>
Message-ID: <CANg26EU=WYBCXdReQKieoTdC1zRZ7rzRxOFiED1E8zYZf0ZpvA@mail.gmail.com>

On 3 July 2012 00:04, Yaroslav Halchenko <lists at onerussian.com> wrote:
> please pardon my sloppy cut/paste which ruined the subject line in the
> original post ;)  So the topic is:
>
> On Mon, 02 Jul 2012, Yaroslav Halchenko wrote:
>> I was thinking about updating Debian package for cython 0.16 but ran into
>> the failing unittests so decided first to check with experts: anything obvious
>> which comes to mind from seeing those?
>
>> ...
>> compiling (c) and running sequential_parallel ... sequential_parallel.c: In function '__pyx_pf_19sequential_parallel_56test_chunksize':
>> sequential_parallel.c:12405:7: warning: variable '__pyx_t_6' set but not used [-Wunused-but-set-variable]
>> sequential_parallel.c:12404:7: warning: variable '__pyx_t_5' set but not used [-Wunused-but-set-variable]
>> sequential_parallel.c:12402:7: warning: variable '__pyx_t_3' set but not used [-Wunused-but-set-variable]
>> sequential_parallel.c: In function '__pyx_pw_19sequential_parallel_25test_nested_break_continue':
>> sequential_parallel.c:5887:29: warning: '__pyx_v_j' may be used uninitialized in this function [-Wmaybe-uninitialized]
>> sequential_parallel.c:5590:7: note: '__pyx_v_j' was declared here
>> sequential_parallel.c: In function '__pyx_f_19sequential_parallel_parallel_exc_cpdef_unnested.isra.29':
>> sequential_parallel.c:8915:3: warning: '__pyx_r' may be used uninitialized in this function [-Wmaybe-uninitialized]
>> sequential_parallel.c:8681:7: note: '__pyx_r' was declared here
>> sequential_parallel.c: In function '__pyx_f_19sequential_parallel_parallel_exc_cpdef.isra.30':
>> sequential_parallel.c:8614:3: warning: '__pyx_r' may be used uninitialized in this function [-Wmaybe-uninitialized]
>> sequential_parallel.c:8259:7: note: '__pyx_r' was declared here
>> sequential_parallel.c: In function '__pyx_pw_19sequential_parallel_33test_parallel_exc_cdef':
>> sequential_parallel.c:8222:78: warning: '__pyx_r' may be used uninitialized in this function [-Wuninitialized]
>> sequential_parallel.c:7946:7: note: '__pyx_r' was declared here
>> sequential_parallel.c:8233:69: warning: '__pyx_r' may be used uninitialized in this function [-Wuninitialized]
>> sequential_parallel.c:7578:7: note: '__pyx_r' was declared here
>> Fatal Python error: deletion of interned string failed
>> Compiling /tmp/buildd/.cython/inline/_cython_inline_a998a12d3d1450b2368ca75e07a27942.pyx because it changed.
>> Cythonizing /tmp/buildd/.cython/inline/_cython_inline_a998a12d3d1450b2368ca75e07a27942.pyx
>> ERROR
>> ...
>> ======================================================================
>> ERROR: compiling (c) and running parallel
>> ----------------------------------------------------------------------
>> Traceback (most recent call last):
>>   File "runtests.py", line 707, in run
>>     self.run_tests(result)
>>   File "runtests.py", line 723, in run_tests
>>     self.run_doctests(self.module, result)
>>   File "runtests.py", line 729, in run_doctests
>>     run_forked_test(result, run_test, self.shortDescription(), self.fork)
>>   File "runtests.py", line 778, in run_forked_test
>>     (module_name, result_code & 255))
>> Exception: Tests in module 'parallel' were unexpectedly killed by signal 11
>
>> ======================================================================
> --
> Yaroslav O. Halchenko
> Postdoctoral Fellow,   Department of Psychological and Brain Sciences
> Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
> Phone: +1 (603) 646-9834                       Fax: +1 (603) 646-1419
> WWW:   http://www.linkedin.com/in/yarik
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

Thanks for the report, I'll have a look.

From stefan_ml at behnel.de  Tue Jul  3 12:49:08 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 03 Jul 2012 12:49:08 +0200
Subject: [Cython] Cython methods for C/C++ types
In-Reply-To: <CANg26EV8TDzwEFt5ZWWL6wNsAptdPqMMatHHQV3ae-0gkA8XcQ@mail.gmail.com>
References: <4FF27BFB.80508@behnel.de>
	<CANg26EV8TDzwEFt5ZWWL6wNsAptdPqMMatHHQV3ae-0gkA8XcQ@mail.gmail.com>
Message-ID: <4FF2CE24.1090906@behnel.de>

mark florisson, 03.07.2012 12:15:
> On 3 July 2012 05:58, Stefan Behnel wrote:
>> the discussion on allowing for automatic user provided ways to coerce
>> between Python types and C++ types got me thinking that this might hide a
>> rather interesting general feature: methods for low-level types. I faintly
>> remember that this idea has come up in our discussions before, but here's a
>> draft CEP for it:
>>
>> http://wiki.cython.org/enhancements/ctypemethods
> 
> I see this CEP also mentions overriding C++ methods. But I think if it
> doesn't support virtual methods, then overriding base class methods by
> calling another function is pretty horrible.

Right, I didn't think that part through. If we don't know the real type at
compile time and get a subtype at runtime then we'll call the wrong code.

I agree that that's not a desirable feature.

Stefan

From d.s.seljebotn at astro.uio.no  Tue Jul  3 18:11:05 2012
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Tue, 03 Jul 2012 18:11:05 +0200
Subject: [Cython] [cython-users] C++: how to handle failures of 'new'?
In-Reply-To: <4FF29BE3.6030201@behnel.de>
References: <4FECA49B.8090404@behnel.de>
	<CADiQ+QBeBw=FYsNPpSU_OJOU48m7E1=csxMxfenMQLs2jhJgPw@mail.gmail.com>
	<4FED40E1.1040301@behnel.de>
	<CADiQ+QAFYMgycY5-RkTf7SOwDb0-qM93Y8+sEzA=+1P9DSyOyA@mail.gmail.com>
	<4FF29BE3.6030201@behnel.de>
Message-ID: <4FF31999.6090501@astro.uio.no>

On 07/03/2012 09:14 AM, Stefan Behnel wrote:
> Robert Bradshaw, 29.06.2012 11:08:
>> On Thu, Jun 28, 2012 at 10:45 PM, Stefan Behnel wrote:
>>> Robert Bradshaw, 28.06.2012 21:46:
>>>> On Thu, Jun 28, 2012 at 11:38 AM, Stefan Behnel wrote:
>>>>> currently, when I write "new CppClass()" in Cython, it generates a straight
>>>>> call to the "new" operator. It doesn't do any error handling. And the
>>>>> current documentation doesn't even mention this case.
>>>>>
>>>>> Is there a "standard" way to handle this? It seems that C++ has different
>>>>> ways to deal with failures here but raises an exception by default. Would
>>>>> you declare the constructor(s) with an "except +MemoryError"? Is there a
>>>>> reason Cython shouldn't be doing this automatically (if nothing else was
>>>>> declared) ?
>>>>
>>>> I think it certainly makes sense to declare the default constructor as
>>>> "except +" (and std::bad_alloc should become MemoryError),
>>>
>>> Right. The code in the constructor can raise other exceptions that must
>>> also be handled properly. An explicit "except +" will handle that.
>>>
>>>
>>>> but whether
>>>> to implicitly annotate declared constructors is less clear, especially
>>>> as there's no way to un-annotate them.
>>>
>>> I agree, but sadly, it's the default behaviour that is wrong. I'm sure we
>>> made lots of users run into this trap already. I fixed the documentation
>>> for now, but the bottom line is that we require users to take care of
>>> proper declarations themselves. Otherwise, the code that we generate is
>>> incorrect, although it's 100% certain that an allocation error can occur,
>>> even if the constructor code doesn't raise any exceptions itself.
>>
>> This is always the case.
>
> Sure, we always rely on correct declarations. However, this is a case where
> we *know* that bad things can happen, even if there is no declaration for
> them. It would be nice to play safe by default.
>
> IMHO, only the second best solution would be to raise a warning when we
> encounter a "new" without an exception declared, so that users are urged
> into writing safe code.
>
>
>>> Apparently, changing the behaviour of the "new" operator requires a special
>>> annotation "std::nothrow", which then returns NULL on allocation failures.
>>> You can pass that from Cython by hacking up a cname, e.g.
>>>
>>>     Rectangle "(std::nothrow) Rectangle" (int w, int h)
>>>
>>> I'm sure there are users out there who figured this out (I mean, I did...)
>>> and use it in their code, so I agree that this isn't easy to handle because
>>> Cython simply wouldn't know what the actual error behaviour is for a given
>>> constructor and how to correctly detect an error.
>>>
>>> This problem applies only to heap allocation in that form. However, stack
>>> allocation and the new exttype field allocation suffer from similar
>>> problems when the default constructor raises an exception. Exttype fields
>>> are a particularly nasty case because the user has no control over the
>>> allocation. A C++ exception in the C++ class constructor would terminate
>>> the exttype constructor unexpectedly and thus leak resources (in the best
>>> case - no idea how CPython reacts if you throw a C++ exception through its
>>> type instantiation code).
>>
>> If the default constructor raises an exception then it should be
>> declared (to not do so is an error on the users part). New raising
>> bad_alloc is a bit of a special case, but doesn't appl to the stack or
>> exttype allocations.
>
> Right. In those two cases, it would definitely be the fault of the user.
>
>
>>> Similarly, a C++ exception in the constructor of a stack allocated object
>>> would then originate from the function entry code and potentially hit the
>>> Python function wrapper etc. Again, potentially leaking resources or worse.
>>>
>>> To me, this sounds like we should do something about it. At least for the
>>> implicit calls to the default constructor, we should generate "except +"
>>> code automatically because there is no other way to handle them safely.
>>
>> If no constructor is declared, it should be "except +" just to be
>> safe
>
> Ok, I'll fix it.
>
>
>> but otherwise I don't see how this is any different than
>> forgetting to declare exceptions on any other function. Unfortunately
>> catching exceptions  (with custom per-object handling) on a set of
>> stack allocated objects seems difficult if not impossible (without
>> resorting to ugly hacks like using placement new everywhere).
>
> I don't know what happens if a C++ exception is not being caught, but I
> guess it would simply crash the application. That's a bit more visible than

Yep.

> just printing a warning when a Python exception is being ignored due to a
> missing declaration. It's really unfortunate that our documentation didn't
> even mention the need for this, because it's not immediately obvious that
> Cython won't handle errors in "new", and testing for memory errors isn't
> quite what people commonly do in their test suites.
>
> Apart from that, I agree, users have to take care to properly declare the
> API they are using.

Is there any time you do NOT want a "catch (...) {}" block? I can't see 
a C++ exception propagating to Python-land doing anything useful ever.

So shouldn't we just make --cplus turn *all* external functions and 
methods (whether C-like or C++-like) into "except +"? (Or keep except+ 
for manual translation, but always have a catch(...)".

Performance overhead is the only reason I can think of to not do this, 
although IIRC C++ catch blocks are only dealt with during stack unwinds 
and doesn't cost anything/much (?) when they're not triggered.

"except -1" should then actually mean both; "except + except -1". So 
it's more a question of just adding catch(...) *everywhere*, than making 
"except +" the default.

Dag

From stefan_ml at behnel.de  Tue Jul  3 18:38:16 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 03 Jul 2012 18:38:16 +0200
Subject: [Cython] [cython-users] C++: how to handle failures of 'new'?
In-Reply-To: <4FF31999.6090501@astro.uio.no>
References: <4FECA49B.8090404@behnel.de>
	<CADiQ+QBeBw=FYsNPpSU_OJOU48m7E1=csxMxfenMQLs2jhJgPw@mail.gmail.com>
	<4FED40E1.1040301@behnel.de>
	<CADiQ+QAFYMgycY5-RkTf7SOwDb0-qM93Y8+sEzA=+1P9DSyOyA@mail.gmail.com>
	<4FF29BE3.6030201@behnel.de> <4FF31999.6090501@astro.uio.no>
Message-ID: <4FF31FF8.10507@behnel.de>

Dag Sverre Seljebotn, 03.07.2012 18:11:
> On 07/03/2012 09:14 AM, Stefan Behnel wrote:
>> I don't know what happens if a C++ exception is not being caught, but I
>> guess it would simply crash the application. That's a bit more visible than
> 
> Yep.
> 
>> just printing a warning when a Python exception is being ignored due to a
>> missing declaration. It's really unfortunate that our documentation didn't
>> even mention the need for this, because it's not immediately obvious that
>> Cython won't handle errors in "new", and testing for memory errors isn't
>> quite what people commonly do in their test suites.
>>
>> Apart from that, I agree, users have to take care to properly declare the
>> API they are using.
> 
> Is there any time you do NOT want a "catch (...) {}" block? I can't see a
> C++ exception propagating to Python-land doing anything useful ever.

That would have been my intuition, too.


> So shouldn't we just make --cplus turn *all* external functions and methods
> (whether C-like or C++-like) into "except +"? (Or keep except+ for manual
> translation, but always have a catch(...)".
>
> Performance overhead is the only reason I can think of to not do this,
> although IIRC C++ catch blocks are only dealt with during stack unwinds and
> doesn't cost anything/much (?) when they're not triggered.
> 
> "except -1" should then actually mean both; "except + except -1". So it's
> more a question of just adding catch(...) *everywhere*, than making "except
> +" the default.

I have no idea if there is a performance impact, but if there isn't, always
catching all exceptions sounds like a reasonable thing to do. After all, we
have no support for catching C++ exceptions on user side.

Stefan

From robertwb at gmail.com  Tue Jul  3 19:58:53 2012
From: robertwb at gmail.com (Robert Bradshaw)
Date: Tue, 3 Jul 2012 10:58:53 -0700
Subject: [Cython] [cython-users] C++: how to handle failures of 'new'?
In-Reply-To: <4FF31FF8.10507@behnel.de>
References: <4FECA49B.8090404@behnel.de>
	<CADiQ+QBeBw=FYsNPpSU_OJOU48m7E1=csxMxfenMQLs2jhJgPw@mail.gmail.com>
	<4FED40E1.1040301@behnel.de>
	<CADiQ+QAFYMgycY5-RkTf7SOwDb0-qM93Y8+sEzA=+1P9DSyOyA@mail.gmail.com>
	<4FF29BE3.6030201@behnel.de> <4FF31999.6090501@astro.uio.no>
	<4FF31FF8.10507@behnel.de>
Message-ID: <CADiQ+QA8nRcpCUsmMx=zb50XQAnOXLVky5pxOxQu=DQxF+BMfw@mail.gmail.com>

On Tue, Jul 3, 2012 at 9:38 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Dag Sverre Seljebotn, 03.07.2012 18:11:
>> On 07/03/2012 09:14 AM, Stefan Behnel wrote:
>>> I don't know what happens if a C++ exception is not being caught, but I
>>> guess it would simply crash the application. That's a bit more visible than
>>
>> Yep.
>>
>>> just printing a warning when a Python exception is being ignored due to a
>>> missing declaration. It's really unfortunate that our documentation didn't
>>> even mention the need for this, because it's not immediately obvious that
>>> Cython won't handle errors in "new", and testing for memory errors isn't
>>> quite what people commonly do in their test suites.
>>>
>>> Apart from that, I agree, users have to take care to properly declare the
>>> API they are using.
>>
>> Is there any time you do NOT want a "catch (...) {}" block? I can't see a
>> C++ exception propagating to Python-land doing anything useful ever.
>
> That would have been my intuition, too.

If it's actually embedded, with the main driver in C++, one might want
it to propagate up.

>> So shouldn't we just make --cplus turn *all* external functions and methods
>> (whether C-like or C++-like) into "except +"? (Or keep except+ for manual
>> translation, but always have a catch(...)".
>>
>> Performance overhead is the only reason I can think of to not do this,
>> although IIRC C++ catch blocks are only dealt with during stack unwinds and
>> doesn't cost anything/much (?) when they're not triggered.
>>
>> "except -1" should then actually mean both; "except + except -1". So it's
>> more a question of just adding catch(...) *everywhere*, than making "except
>> +" the default.
>
> I have no idea if there is a performance impact, but if there isn't, always
> catching all exceptions sounds like a reasonable thing to do. After all, we
> have no support for catching C++ exceptions on user side.

This is a bit like following every C call with "except *" (though the
performance ratios are unclear). It just seems a lot to wrap every
single line of a non-trivial C++ using function with try..catch
blocks. I also don't think this would play well with Pynac (from Sage)
which is a C++ library with Cython callbacks that may call back into
the library and raise C++ exceptions (but that does feel a bit risky
anyways).

- Robert

From stefan_ml at behnel.de  Tue Jul  3 20:11:30 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 03 Jul 2012 20:11:30 +0200
Subject: [Cython] [cython-users] C++: how to handle failures of 'new'?
In-Reply-To: <CADiQ+QA8nRcpCUsmMx=zb50XQAnOXLVky5pxOxQu=DQxF+BMfw@mail.gmail.com>
References: <4FECA49B.8090404@behnel.de>
	<CADiQ+QBeBw=FYsNPpSU_OJOU48m7E1=csxMxfenMQLs2jhJgPw@mail.gmail.com>
	<4FED40E1.1040301@behnel.de>
	<CADiQ+QAFYMgycY5-RkTf7SOwDb0-qM93Y8+sEzA=+1P9DSyOyA@mail.gmail.com>
	<4FF29BE3.6030201@behnel.de> <4FF31999.6090501@astro.uio.no>
	<4FF31FF8.10507@behnel.de>
	<CADiQ+QA8nRcpCUsmMx=zb50XQAnOXLVky5pxOxQu=DQxF+BMfw@mail.gmail.com>
Message-ID: <4FF335D2.3070706@behnel.de>

Robert Bradshaw, 03.07.2012 19:58:
> On Tue, Jul 3, 2012 at 9:38 AM, Stefan Behnel wrote:
>> Dag Sverre Seljebotn, 03.07.2012 18:11:
>>> On 07/03/2012 09:14 AM, Stefan Behnel wrote:
>>>> I don't know what happens if a C++ exception is not being caught, but I
>>>> guess it would simply crash the application. That's a bit more visible than
>>>
>>> Yep.
>>>
>>>> just printing a warning when a Python exception is being ignored due to a
>>>> missing declaration. It's really unfortunate that our documentation didn't
>>>> even mention the need for this, because it's not immediately obvious that
>>>> Cython won't handle errors in "new", and testing for memory errors isn't
>>>> quite what people commonly do in their test suites.
>>>>
>>>> Apart from that, I agree, users have to take care to properly declare the
>>>> API they are using.
>>>
>>> Is there any time you do NOT want a "catch (...) {}" block? I can't see a
>>> C++ exception propagating to Python-land doing anything useful ever.
>>
>> That would have been my intuition, too.
> 
> If it's actually embedded, with the main driver in C++, one might want
> it to propagate up.

But what kind of a propagation would that be? On the way out, it could
induce anything, from side effects to resource leaks to crashes, depending
on what the state of the surrounding code is. It would leave the whole
system in an unpredictable state. I cannot imagine anyone really wanting this.


>>> So shouldn't we just make --cplus turn *all* external functions and methods
>>> (whether C-like or C++-like) into "except +"? (Or keep except+ for manual
>>> translation, but always have a catch(...)".
>>>
>>> Performance overhead is the only reason I can think of to not do this,
>>> although IIRC C++ catch blocks are only dealt with during stack unwinds and
>>> doesn't cost anything/much (?) when they're not triggered.
>>>
>>> "except -1" should then actually mean both; "except + except -1". So it's
>>> more a question of just adding catch(...) *everywhere*, than making "except
>>> +" the default.
>>
>> I have no idea if there is a performance impact, but if there isn't, always
>> catching all exceptions sounds like a reasonable thing to do. After all, we
>> have no support for catching C++ exceptions on user side.
> 
> This is a bit like following every C call with "except *" (though the
> performance ratios are unclear). It just seems a lot to wrap every
> single line of a non-trivial C++ using function with try..catch
> blocks.

But if users are correct about their declarations, we'd end up with the
same thing. I think it's worth a try.

Stefan

From robertwb at gmail.com  Tue Jul  3 20:23:07 2012
From: robertwb at gmail.com (Robert Bradshaw)
Date: Tue, 3 Jul 2012 11:23:07 -0700
Subject: [Cython] [cython-users] C++: how to handle failures of 'new'?
In-Reply-To: <4FF335D2.3070706@behnel.de>
References: <4FECA49B.8090404@behnel.de>
	<CADiQ+QBeBw=FYsNPpSU_OJOU48m7E1=csxMxfenMQLs2jhJgPw@mail.gmail.com>
	<4FED40E1.1040301@behnel.de>
	<CADiQ+QAFYMgycY5-RkTf7SOwDb0-qM93Y8+sEzA=+1P9DSyOyA@mail.gmail.com>
	<4FF29BE3.6030201@behnel.de> <4FF31999.6090501@astro.uio.no>
	<4FF31FF8.10507@behnel.de>
	<CADiQ+QA8nRcpCUsmMx=zb50XQAnOXLVky5pxOxQu=DQxF+BMfw@mail.gmail.com>
	<4FF335D2.3070706@behnel.de>
Message-ID: <CADiQ+QATCzuJ2vNXJm-QxueG9s_DBPyCBRGhZ1cnMpEMQvMtzQ@mail.gmail.com>

On Tue, Jul 3, 2012 at 11:11 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Robert Bradshaw, 03.07.2012 19:58:
>> On Tue, Jul 3, 2012 at 9:38 AM, Stefan Behnel wrote:
>>> Dag Sverre Seljebotn, 03.07.2012 18:11:
>>>> On 07/03/2012 09:14 AM, Stefan Behnel wrote:
>>>>> I don't know what happens if a C++ exception is not being caught, but I
>>>>> guess it would simply crash the application. That's a bit more visible than
>>>>
>>>> Yep.
>>>>
>>>>> just printing a warning when a Python exception is being ignored due to a
>>>>> missing declaration. It's really unfortunate that our documentation didn't
>>>>> even mention the need for this, because it's not immediately obvious that
>>>>> Cython won't handle errors in "new", and testing for memory errors isn't
>>>>> quite what people commonly do in their test suites.
>>>>>
>>>>> Apart from that, I agree, users have to take care to properly declare the
>>>>> API they are using.
>>>>
>>>> Is there any time you do NOT want a "catch (...) {}" block? I can't see a
>>>> C++ exception propagating to Python-land doing anything useful ever.
>>>
>>> That would have been my intuition, too.
>>
>> If it's actually embedded, with the main driver in C++, one might want
>> it to propagate up.
>
> But what kind of a propagation would that be? On the way out, it could
> induce anything, from side effects to resource leaks to crashes, depending
> on what the state of the surrounding code is. It would leave the whole
> system in an unpredictable state. I cannot imagine anyone really wanting this.
>
>
>>>> So shouldn't we just make --cplus turn *all* external functions and methods
>>>> (whether C-like or C++-like) into "except +"? (Or keep except+ for manual
>>>> translation, but always have a catch(...)".
>>>>
>>>> Performance overhead is the only reason I can think of to not do this,
>>>> although IIRC C++ catch blocks are only dealt with during stack unwinds and
>>>> doesn't cost anything/much (?) when they're not triggered.
>>>>
>>>> "except -1" should then actually mean both; "except + except -1". So it's
>>>> more a question of just adding catch(...) *everywhere*, than making "except
>>>> +" the default.
>>>
>>> I have no idea if there is a performance impact, but if there isn't, always
>>> catching all exceptions sounds like a reasonable thing to do. After all, we
>>> have no support for catching C++ exceptions on user side.
>>
>> This is a bit like following every C call with "except *" (though the
>> performance ratios are unclear). It just seems a lot to wrap every
>> single line of a non-trivial C++ using function with try..catch
>> blocks.
>
> But if users are correct about their declarations, we'd end up with the
> same thing. I think it's worth a try.

Most C++ code (that I've ever run into) doesn't use exceptions,
because exception handling is so broken in C++ anyways.

- Robert

From d.s.seljebotn at astro.uio.no  Tue Jul  3 20:43:44 2012
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Tue, 03 Jul 2012 20:43:44 +0200
Subject: [Cython] [cython-users] C++: how to handle failures of 'new'?
In-Reply-To: <CADiQ+QATCzuJ2vNXJm-QxueG9s_DBPyCBRGhZ1cnMpEMQvMtzQ@mail.gmail.com>
References: <4FECA49B.8090404@behnel.de>
	<CADiQ+QBeBw=FYsNPpSU_OJOU48m7E1=csxMxfenMQLs2jhJgPw@mail.gmail.com>
	<4FED40E1.1040301@behnel.de>
	<CADiQ+QAFYMgycY5-RkTf7SOwDb0-qM93Y8+sEzA=+1P9DSyOyA@mail.gmail.com>
	<4FF29BE3.6030201@behnel.de> <4FF31999.6090501@astro.uio.no>
	<4FF31FF8.10507@behnel.de>
	<CADiQ+QA8nRcpCUsmMx=zb50XQAnOXLVky5pxOxQu=DQxF+BMfw@mail.gmail.com>
	<4FF335D2.3070706@behnel.de>
	<CADiQ+QATCzuJ2vNXJm-QxueG9s_DBPyCBRGhZ1cnMpEMQvMtzQ@mail.gmail.com>
Message-ID: <4FF33D60.5080804@astro.uio.no>

On 07/03/2012 08:23 PM, Robert Bradshaw wrote:
> On Tue, Jul 3, 2012 at 11:11 AM, Stefan Behnel<stefan_ml at behnel.de>  wrote:
>> Robert Bradshaw, 03.07.2012 19:58:
>>> On Tue, Jul 3, 2012 at 9:38 AM, Stefan Behnel wrote:
>>>> Dag Sverre Seljebotn, 03.07.2012 18:11:
>>>>> On 07/03/2012 09:14 AM, Stefan Behnel wrote:
>>>>>> I don't know what happens if a C++ exception is not being caught, but I
>>>>>> guess it would simply crash the application. That's a bit more visible than
>>>>>
>>>>> Yep.
>>>>>
>>>>>> just printing a warning when a Python exception is being ignored due to a
>>>>>> missing declaration. It's really unfortunate that our documentation didn't
>>>>>> even mention the need for this, because it's not immediately obvious that
>>>>>> Cython won't handle errors in "new", and testing for memory errors isn't
>>>>>> quite what people commonly do in their test suites.
>>>>>>
>>>>>> Apart from that, I agree, users have to take care to properly declare the
>>>>>> API they are using.
>>>>>
>>>>> Is there any time you do NOT want a "catch (...) {}" block? I can't see a
>>>>> C++ exception propagating to Python-land doing anything useful ever.
>>>>
>>>> That would have been my intuition, too.
>>>
>>> If it's actually embedded, with the main driver in C++, one might want
>>> it to propagate up.
>>
>> But what kind of a propagation would that be? On the way out, it could
>> induce anything, from side effects to resource leaks to crashes, depending
>> on what the state of the surrounding code is. It would leave the whole
>> system in an unpredictable state. I cannot imagine anyone really wanting this.
>>
>>
>>>>> So shouldn't we just make --cplus turn *all* external functions and methods
>>>>> (whether C-like or C++-like) into "except +"? (Or keep except+ for manual
>>>>> translation, but always have a catch(...)".
>>>>>
>>>>> Performance overhead is the only reason I can think of to not do this,
>>>>> although IIRC C++ catch blocks are only dealt with during stack unwinds and
>>>>> doesn't cost anything/much (?) when they're not triggered.
>>>>>
>>>>> "except -1" should then actually mean both; "except + except -1". So it's
>>>>> more a question of just adding catch(...) *everywhere*, than making "except
>>>>> +" the default.
>>>>
>>>> I have no idea if there is a performance impact, but if there isn't, always
>>>> catching all exceptions sounds like a reasonable thing to do. After all, we
>>>> have no support for catching C++ exceptions on user side.
>>>
>>> This is a bit like following every C call with "except *" (though the
>>> performance ratios are unclear). It just seems a lot to wrap every
>>> single line of a non-trivial C++ using function with try..catch
>>> blocks.

It seems "a lot" of just what exactly? Generated code? Binary size? Time 
spent in GCC parser?

Though I guess one might want to try to pull out the try-catch to at 
least only one per code line rather than one per SimpleCallNode.

"except *" only has a point when calling functions using the CPython 
API, but most external C functions are pure C, not 
CPython-API-using-functions. OTOH, all external C++ functions are C++ :-)

(Also, if we wrote Cython from scratch now I'm pretty sure the "except 
*" defaults would be a tad different.)

>>
>> But if users are correct about their declarations, we'd end up with the
>> same thing. I think it's worth a try.
>
> Most C++ code (that I've ever run into) doesn't use exceptions,
> because exception handling is so broken in C++ anyways.

Except for the fact that any code touching "new" could be raising 
exceptions? That propagates.

There is a lot of C++ code out there using exceptions. I'd guess that 
both mathematical code and Google-written code is unlike most C++ code 
out there :-) Many C++ programmers go on and on about RAII and auto_ptrs 
and so on, and that doesn't have much point unless you throw an 
exception now and then (OK, there's the occasional return statement 
where it matters well).

Dag

From d.s.seljebotn at astro.uio.no  Tue Jul  3 20:47:48 2012
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Tue, 03 Jul 2012 20:47:48 +0200
Subject: [Cython] [cython-users] C++: how to handle failures of 'new'?
In-Reply-To: <CADiQ+QATCzuJ2vNXJm-QxueG9s_DBPyCBRGhZ1cnMpEMQvMtzQ@mail.gmail.com>
References: <4FECA49B.8090404@behnel.de>
	<CADiQ+QBeBw=FYsNPpSU_OJOU48m7E1=csxMxfenMQLs2jhJgPw@mail.gmail.com>
	<4FED40E1.1040301@behnel.de>
	<CADiQ+QAFYMgycY5-RkTf7SOwDb0-qM93Y8+sEzA=+1P9DSyOyA@mail.gmail.com>
	<4FF29BE3.6030201@behnel.de> <4FF31999.6090501@astro.uio.no>
	<4FF31FF8.10507@behnel.de>
	<CADiQ+QA8nRcpCUsmMx=zb50XQAnOXLVky5pxOxQu=DQxF+BMfw@mail.gmail.com>
	<4FF335D2.3070706@behnel.de>
	<CADiQ+QATCzuJ2vNXJm-QxueG9s_DBPyCBRGhZ1cnMpEMQvMtzQ@mail.gmail.com>
Message-ID: <4FF33E54.1080108@astro.uio.no>

On 07/03/2012 08:23 PM, Robert Bradshaw wrote:
> On Tue, Jul 3, 2012 at 11:11 AM, Stefan Behnel<stefan_ml at behnel.de>  wrote:
>> Robert Bradshaw, 03.07.2012 19:58:
>>> On Tue, Jul 3, 2012 at 9:38 AM, Stefan Behnel wrote:
>>>> Dag Sverre Seljebotn, 03.07.2012 18:11:
>>>>> On 07/03/2012 09:14 AM, Stefan Behnel wrote:
>>>>>> I don't know what happens if a C++ exception is not being caught, but I
>>>>>> guess it would simply crash the application. That's a bit more visible than
>>>>>
>>>>> Yep.
>>>>>
>>>>>> just printing a warning when a Python exception is being ignored due to a
>>>>>> missing declaration. It's really unfortunate that our documentation didn't
>>>>>> even mention the need for this, because it's not immediately obvious that
>>>>>> Cython won't handle errors in "new", and testing for memory errors isn't
>>>>>> quite what people commonly do in their test suites.
>>>>>>
>>>>>> Apart from that, I agree, users have to take care to properly declare the
>>>>>> API they are using.
>>>>>
>>>>> Is there any time you do NOT want a "catch (...) {}" block? I can't see a
>>>>> C++ exception propagating to Python-land doing anything useful ever.
>>>>
>>>> That would have been my intuition, too.
>>>
>>> If it's actually embedded, with the main driver in C++, one might want
>>> it to propagate up.
>>
>> But what kind of a propagation would that be? On the way out, it could
>> induce anything, from side effects to resource leaks to crashes, depending
>> on what the state of the surrounding code is. It would leave the whole
>> system in an unpredictable state. I cannot imagine anyone really wanting this.
>>
>>
>>>>> So shouldn't we just make --cplus turn *all* external functions and methods
>>>>> (whether C-like or C++-like) into "except +"? (Or keep except+ for manual
>>>>> translation, but always have a catch(...)".
>>>>>
>>>>> Performance overhead is the only reason I can think of to not do this,
>>>>> although IIRC C++ catch blocks are only dealt with during stack unwinds and
>>>>> doesn't cost anything/much (?) when they're not triggered.
>>>>>
>>>>> "except -1" should then actually mean both; "except + except -1". So it's
>>>>> more a question of just adding catch(...) *everywhere*, than making "except
>>>>> +" the default.
>>>>
>>>> I have no idea if there is a performance impact, but if there isn't, always
>>>> catching all exceptions sounds like a reasonable thing to do. After all, we
>>>> have no support for catching C++ exceptions on user side.
>>>
>>> This is a bit like following every C call with "except *" (though the
>>> performance ratios are unclear). It just seems a lot to wrap every
>>> single line of a non-trivial C++ using function with try..catch
>>> blocks.
>>
>> But if users are correct about their declarations, we'd end up with the
>> same thing. I think it's worth a try.
>
> Most C++ code (that I've ever run into) doesn't use exceptions,
> because exception handling is so broken in C++ anyways.

Did you mean that most C++ code doesn't use try/except? That's true, but 
my experience is that throw is used.

Dag

From robertwb at gmail.com  Tue Jul  3 21:15:10 2012
From: robertwb at gmail.com (Robert Bradshaw)
Date: Tue, 3 Jul 2012 12:15:10 -0700
Subject: [Cython] [cython-users] C++: how to handle failures of 'new'?
In-Reply-To: <4FF33D60.5080804@astro.uio.no>
References: <4FECA49B.8090404@behnel.de>
	<CADiQ+QBeBw=FYsNPpSU_OJOU48m7E1=csxMxfenMQLs2jhJgPw@mail.gmail.com>
	<4FED40E1.1040301@behnel.de>
	<CADiQ+QAFYMgycY5-RkTf7SOwDb0-qM93Y8+sEzA=+1P9DSyOyA@mail.gmail.com>
	<4FF29BE3.6030201@behnel.de> <4FF31999.6090501@astro.uio.no>
	<4FF31FF8.10507@behnel.de>
	<CADiQ+QA8nRcpCUsmMx=zb50XQAnOXLVky5pxOxQu=DQxF+BMfw@mail.gmail.com>
	<4FF335D2.3070706@behnel.de>
	<CADiQ+QATCzuJ2vNXJm-QxueG9s_DBPyCBRGhZ1cnMpEMQvMtzQ@mail.gmail.com>
	<4FF33D60.5080804@astro.uio.no>
Message-ID: <CADiQ+QD0etaANiRy=03=yTjCi60xCpN+GVV5=mueWuQnBotrDQ@mail.gmail.com>

On Tue, Jul 3, 2012 at 11:43 AM, Dag Sverre Seljebotn
<d.s.seljebotn at astro.uio.no> wrote:
> On 07/03/2012 08:23 PM, Robert Bradshaw wrote:
>>
>> On Tue, Jul 3, 2012 at 11:11 AM, Stefan Behnel<stefan_ml at behnel.de>
>> wrote:
>>>
>>> Robert Bradshaw, 03.07.2012 19:58:
>>>>
>>>> On Tue, Jul 3, 2012 at 9:38 AM, Stefan Behnel wrote:
>>>>>
>>>>> Dag Sverre Seljebotn, 03.07.2012 18:11:
>>>>>>
>>>>>> On 07/03/2012 09:14 AM, Stefan Behnel wrote:
>>>>>>>
>>>>>>> I don't know what happens if a C++ exception is not being caught, but
>>>>>>> I
>>>>>>> guess it would simply crash the application. That's a bit more
>>>>>>> visible than
>>>>>>
>>>>>>
>>>>>> Yep.
>>>>>>
>>>>>>> just printing a warning when a Python exception is being ignored due
>>>>>>> to a
>>>>>>> missing declaration. It's really unfortunate that our documentation
>>>>>>> didn't
>>>>>>> even mention the need for this, because it's not immediately obvious
>>>>>>> that
>>>>>>> Cython won't handle errors in "new", and testing for memory errors
>>>>>>> isn't
>>>>>>> quite what people commonly do in their test suites.
>>>>>>>
>>>>>>> Apart from that, I agree, users have to take care to properly declare
>>>>>>> the
>>>>>>> API they are using.
>>>>>>
>>>>>>
>>>>>> Is there any time you do NOT want a "catch (...) {}" block? I can't
>>>>>> see a
>>>>>> C++ exception propagating to Python-land doing anything useful ever.
>>>>>
>>>>>
>>>>> That would have been my intuition, too.
>>>>
>>>>
>>>> If it's actually embedded, with the main driver in C++, one might want
>>>> it to propagate up.
>>>
>>>
>>> But what kind of a propagation would that be? On the way out, it could
>>> induce anything, from side effects to resource leaks to crashes,
>>> depending
>>> on what the state of the surrounding code is. It would leave the whole
>>> system in an unpredictable state. I cannot imagine anyone really wanting
>>> this.
>>>
>>>
>>>>>> So shouldn't we just make --cplus turn *all* external functions and
>>>>>> methods
>>>>>> (whether C-like or C++-like) into "except +"? (Or keep except+ for
>>>>>> manual
>>>>>> translation, but always have a catch(...)".
>>>>>>
>>>>>> Performance overhead is the only reason I can think of to not do this,
>>>>>> although IIRC C++ catch blocks are only dealt with during stack
>>>>>> unwinds and
>>>>>> doesn't cost anything/much (?) when they're not triggered.
>>>>>>
>>>>>> "except -1" should then actually mean both; "except + except -1". So
>>>>>> it's
>>>>>> more a question of just adding catch(...) *everywhere*, than making
>>>>>> "except
>>>>>> +" the default.
>>>>>
>>>>>
>>>>> I have no idea if there is a performance impact, but if there isn't,
>>>>> always
>>>>> catching all exceptions sounds like a reasonable thing to do. After
>>>>> all, we
>>>>> have no support for catching C++ exceptions on user side.
>>>>
>>>>
>>>> This is a bit like following every C call with "except *" (though the
>>>> performance ratios are unclear). It just seems a lot to wrap every
>>>> single line of a non-trivial C++ using function with try..catch
>>>> blocks.
>
>
> It seems "a lot" of just what exactly? Generated code? Binary size? Time
> spent in GCC parser?

All of the above. And we should take a look at the runtime overhead
(which is hopefully nil, but who knows.)

> Though I guess one might want to try to pull out the try-catch to at least
> only one per code line rather than one per SimpleCallNode.

Or even higher, if possible. It's still a lot.

> "except *" only has a point when calling functions using the CPython API,
> but most external C functions are pure C, not CPython-API-using-functions.
> OTOH, all external C++ functions are C++ :-)

Fair point.

> (Also, if we wrote Cython from scratch now I'm pretty sure the "except *"
> defaults would be a tad different.)

For sure.

>>> But if users are correct about their declarations, we'd end up with the
>>> same thing. I think it's worth a try.
>>
>>
>> Most C++ code (that I've ever run into) doesn't use exceptions,
>> because exception handling is so broken in C++ anyways.
>
>
> Except for the fact that any code touching "new" could be raising
> exceptions? That propagates.

I would guess most of the time people don't bother catching these and
let the program die, as there's often no sane recovery (the same as
MemoryErrors in Python, though I guess C++ is less often used from an
event loop).

> There is a lot of C++ code out there using exceptions. I'd guess that both
> mathematical code and Google-written code is unlike most C++ code out there
> :-) Many C++ programmers go on and on about RAII and auto_ptrs and so on,
> and that doesn't have much point unless you throw an exception now and then
> (OK, there's the occasional return statement where it matters well).

True, I've seen a small subset of the C++ code that's out there. Maybe
numerical computations use it a lot?

+1 to making catch-everywhere a directive at least. Lets see what the
impact is before we decide to make it the default.

- Robert

From d.s.seljebotn at astro.uio.no  Tue Jul  3 22:39:24 2012
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Tue, 03 Jul 2012 22:39:24 +0200
Subject: [Cython] [cython-users] C++: how to handle failures of 'new'?
In-Reply-To: <CADiQ+QD0etaANiRy=03=yTjCi60xCpN+GVV5=mueWuQnBotrDQ@mail.gmail.com>
References: <4FECA49B.8090404@behnel.de>
	<CADiQ+QBeBw=FYsNPpSU_OJOU48m7E1=csxMxfenMQLs2jhJgPw@mail.gmail.com>
	<4FED40E1.1040301@behnel.de>
	<CADiQ+QAFYMgycY5-RkTf7SOwDb0-qM93Y8+sEzA=+1P9DSyOyA@mail.gmail.com>
	<4FF29BE3.6030201@behnel.de> <4FF31999.6090501@astro.uio.no>
	<4FF31FF8.10507@behnel.de>
	<CADiQ+QA8nRcpCUsmMx=zb50XQAnOXLVky5pxOxQu=DQxF+BMfw@mail.gmail.com>
	<4FF335D2.3070706@behnel.de>
	<CADiQ+QATCzuJ2vNXJm-QxueG9s_DBPyCBRGhZ1cnMpEMQvMtzQ@mail.gmail.com>
	<4FF33D60.5080804@astro.uio.no>
	<CADiQ+QD0etaANiRy=03=yTjCi60xCpN+GVV5=mueWuQnBotrDQ@mail.gmail.com>
Message-ID: <4FF3587C.5060402@astro.uio.no>

On 07/03/2012 09:15 PM, Robert Bradshaw wrote:
> On Tue, Jul 3, 2012 at 11:43 AM, Dag Sverre Seljebotn
> <d.s.seljebotn at astro.uio.no>  wrote:
>> On 07/03/2012 08:23 PM, Robert Bradshaw wrote:
>>>
>>> On Tue, Jul 3, 2012 at 11:11 AM, Stefan Behnel<stefan_ml at behnel.de>
>>> wrote:
>>>>
>>>> Robert Bradshaw, 03.07.2012 19:58:
>>>>>
>>>>> On Tue, Jul 3, 2012 at 9:38 AM, Stefan Behnel wrote:
>>>>>>
>>>>>> Dag Sverre Seljebotn, 03.07.2012 18:11:
>>>>>>>
>>>>>>> On 07/03/2012 09:14 AM, Stefan Behnel wrote:
>>>>>>>>
>>>>>>>> I don't know what happens if a C++ exception is not being caught, but
>>>>>>>> I
>>>>>>>> guess it would simply crash the application. That's a bit more
>>>>>>>> visible than
>>>>>>>
>>>>>>>
>>>>>>> Yep.
>>>>>>>
>>>>>>>> just printing a warning when a Python exception is being ignored due
>>>>>>>> to a
>>>>>>>> missing declaration. It's really unfortunate that our documentation
>>>>>>>> didn't
>>>>>>>> even mention the need for this, because it's not immediately obvious
>>>>>>>> that
>>>>>>>> Cython won't handle errors in "new", and testing for memory errors
>>>>>>>> isn't
>>>>>>>> quite what people commonly do in their test suites.
>>>>>>>>
>>>>>>>> Apart from that, I agree, users have to take care to properly declare
>>>>>>>> the
>>>>>>>> API they are using.
>>>>>>>
>>>>>>>
>>>>>>> Is there any time you do NOT want a "catch (...) {}" block? I can't
>>>>>>> see a
>>>>>>> C++ exception propagating to Python-land doing anything useful ever.
>>>>>>
>>>>>>
>>>>>> That would have been my intuition, too.
>>>>>
>>>>>
>>>>> If it's actually embedded, with the main driver in C++, one might want
>>>>> it to propagate up.
>>>>
>>>>
>>>> But what kind of a propagation would that be? On the way out, it could
>>>> induce anything, from side effects to resource leaks to crashes,
>>>> depending
>>>> on what the state of the surrounding code is. It would leave the whole
>>>> system in an unpredictable state. I cannot imagine anyone really wanting
>>>> this.
>>>>
>>>>
>>>>>>> So shouldn't we just make --cplus turn *all* external functions and
>>>>>>> methods
>>>>>>> (whether C-like or C++-like) into "except +"? (Or keep except+ for
>>>>>>> manual
>>>>>>> translation, but always have a catch(...)".
>>>>>>>
>>>>>>> Performance overhead is the only reason I can think of to not do this,
>>>>>>> although IIRC C++ catch blocks are only dealt with during stack
>>>>>>> unwinds and
>>>>>>> doesn't cost anything/much (?) when they're not triggered.
>>>>>>>
>>>>>>> "except -1" should then actually mean both; "except + except -1". So
>>>>>>> it's
>>>>>>> more a question of just adding catch(...) *everywhere*, than making
>>>>>>> "except
>>>>>>> +" the default.
>>>>>>
>>>>>>
>>>>>> I have no idea if there is a performance impact, but if there isn't,
>>>>>> always
>>>>>> catching all exceptions sounds like a reasonable thing to do. After
>>>>>> all, we
>>>>>> have no support for catching C++ exceptions on user side.
>>>>>
>>>>>
>>>>> This is a bit like following every C call with "except *" (though the
>>>>> performance ratios are unclear). It just seems a lot to wrap every
>>>>> single line of a non-trivial C++ using function with try..catch
>>>>> blocks.
>>
>>
>> It seems "a lot" of just what exactly? Generated code? Binary size? Time
>> spent in GCC parser?
>
> All of the above. And we should take a look at the runtime overhead
> (which is hopefully nil, but who knows.)
>
>> Though I guess one might want to try to pull out the try-catch to at least
>> only one per code line rather than one per SimpleCallNode.
>
> Or even higher, if possible. It's still a lot.
>
>> "except *" only has a point when calling functions using the CPython API,
>> but most external C functions are pure C, not CPython-API-using-functions.
>> OTOH, all external C++ functions are C++ :-)
>
> Fair point.
>
>> (Also, if we wrote Cython from scratch now I'm pretty sure the "except *"
>> defaults would be a tad different.)
>
> For sure.
>
>>>> But if users are correct about their declarations, we'd end up with the
>>>> same thing. I think it's worth a try.
>>>
>>>
>>> Most C++ code (that I've ever run into) doesn't use exceptions,
>>> because exception handling is so broken in C++ anyways.
>>
>>
>> Except for the fact that any code touching "new" could be raising
>> exceptions? That propagates.
>
> I would guess most of the time people don't bother catching these and
> let the program die, as there's often no sane recovery (the same as
> MemoryErrors in Python, though I guess C++ is less often used from an
> event loop).

The point is to provide the user with the stack trace of exactly where 
you ran out of memory. Even if you can't follow it deep into C++-land, 
the Python-side stack trace can be a worth a lot.

>
>> There is a lot of C++ code out there using exceptions. I'd guess that both
>> mathematical code and Google-written code is unlike most C++ code out there
>> :-) Many C++ programmers go on and on about RAII and auto_ptrs and so on,
>> and that doesn't have much point unless you throw an exception now and then
>> (OK, there's the occasional return statement where it matters well).
>
> True, I've seen a small subset of the C++ code that's out there. Maybe
> numerical computations use it a lot?

One library I'm using for distributed dense linear algebra (Elemental) 
does. Numerical computations libraries are very different from one another.

Dag

From d.s.seljebotn at astro.uio.no  Tue Jul  3 22:42:55 2012
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Tue, 03 Jul 2012 22:42:55 +0200
Subject: [Cython] [cython-users] C++: how to handle failures of 'new'?
In-Reply-To: <4FF3587C.5060402@astro.uio.no>
References: <4FECA49B.8090404@behnel.de>
	<CADiQ+QBeBw=FYsNPpSU_OJOU48m7E1=csxMxfenMQLs2jhJgPw@mail.gmail.com>
	<4FED40E1.1040301@behnel.de>
	<CADiQ+QAFYMgycY5-RkTf7SOwDb0-qM93Y8+sEzA=+1P9DSyOyA@mail.gmail.com>
	<4FF29BE3.6030201@behnel.de> <4FF31999.6090501@astro.uio.no>
	<4FF31FF8.10507@behnel.de>
	<CADiQ+QA8nRcpCUsmMx=zb50XQAnOXLVky5pxOxQu=DQxF+BMfw@mail.gmail.com>
	<4FF335D2.3070706@behnel.de>
	<CADiQ+QATCzuJ2vNXJm-QxueG9s_DBPyCBRGhZ1cnMpEMQvMtzQ@mail.gmail.com>
	<4FF33D60.5080804@astro.uio.no>
	<CADiQ+QD0etaANiRy=03=yTjCi60xCpN+GVV5=mueWuQnBotrDQ@mail.gmail.com>
	<4FF3587C.5060402@astro.uio.no>
Message-ID: <4FF3594F.3080507@astro.uio.no>

On 07/03/2012 10:39 PM, Dag Sverre Seljebotn wrote:
> On 07/03/2012 09:15 PM, Robert Bradshaw wrote:
>> On Tue, Jul 3, 2012 at 11:43 AM, Dag Sverre Seljebotn
>> <d.s.seljebotn at astro.uio.no> wrote:
>>> On 07/03/2012 08:23 PM, Robert Bradshaw wrote:
>>>>
>>>> On Tue, Jul 3, 2012 at 11:11 AM, Stefan Behnel<stefan_ml at behnel.de>
>>>> wrote:
>>>>>
>>>>> Robert Bradshaw, 03.07.2012 19:58:
>>>>>>
>>>>>> On Tue, Jul 3, 2012 at 9:38 AM, Stefan Behnel wrote:
>>>>>>>
>>>>>>> Dag Sverre Seljebotn, 03.07.2012 18:11:
>>>>>>>>
>>>>>>>> On 07/03/2012 09:14 AM, Stefan Behnel wrote:
>>>>>>>>>
>>>>>>>>> I don't know what happens if a C++ exception is not being
>>>>>>>>> caught, but
>>>>>>>>> I
>>>>>>>>> guess it would simply crash the application. That's a bit more
>>>>>>>>> visible than
>>>>>>>>
>>>>>>>>
>>>>>>>> Yep.
>>>>>>>>
>>>>>>>>> just printing a warning when a Python exception is being
>>>>>>>>> ignored due
>>>>>>>>> to a
>>>>>>>>> missing declaration. It's really unfortunate that our
>>>>>>>>> documentation
>>>>>>>>> didn't
>>>>>>>>> even mention the need for this, because it's not immediately
>>>>>>>>> obvious
>>>>>>>>> that
>>>>>>>>> Cython won't handle errors in "new", and testing for memory errors
>>>>>>>>> isn't
>>>>>>>>> quite what people commonly do in their test suites.
>>>>>>>>>
>>>>>>>>> Apart from that, I agree, users have to take care to properly
>>>>>>>>> declare
>>>>>>>>> the
>>>>>>>>> API they are using.
>>>>>>>>
>>>>>>>>
>>>>>>>> Is there any time you do NOT want a "catch (...) {}" block? I can't
>>>>>>>> see a
>>>>>>>> C++ exception propagating to Python-land doing anything useful
>>>>>>>> ever.
>>>>>>>
>>>>>>>
>>>>>>> That would have been my intuition, too.
>>>>>>
>>>>>>
>>>>>> If it's actually embedded, with the main driver in C++, one might
>>>>>> want
>>>>>> it to propagate up.
>>>>>
>>>>>
>>>>> But what kind of a propagation would that be? On the way out, it could
>>>>> induce anything, from side effects to resource leaks to crashes,
>>>>> depending
>>>>> on what the state of the surrounding code is. It would leave the whole
>>>>> system in an unpredictable state. I cannot imagine anyone really
>>>>> wanting
>>>>> this.
>>>>>
>>>>>
>>>>>>>> So shouldn't we just make --cplus turn *all* external functions and
>>>>>>>> methods
>>>>>>>> (whether C-like or C++-like) into "except +"? (Or keep except+ for
>>>>>>>> manual
>>>>>>>> translation, but always have a catch(...)".
>>>>>>>>
>>>>>>>> Performance overhead is the only reason I can think of to not do
>>>>>>>> this,
>>>>>>>> although IIRC C++ catch blocks are only dealt with during stack
>>>>>>>> unwinds and
>>>>>>>> doesn't cost anything/much (?) when they're not triggered.
>>>>>>>>
>>>>>>>> "except -1" should then actually mean both; "except + except
>>>>>>>> -1". So
>>>>>>>> it's
>>>>>>>> more a question of just adding catch(...) *everywhere*, than making
>>>>>>>> "except
>>>>>>>> +" the default.
>>>>>>>
>>>>>>>
>>>>>>> I have no idea if there is a performance impact, but if there isn't,
>>>>>>> always
>>>>>>> catching all exceptions sounds like a reasonable thing to do. After
>>>>>>> all, we
>>>>>>> have no support for catching C++ exceptions on user side.
>>>>>>
>>>>>>
>>>>>> This is a bit like following every C call with "except *" (though the
>>>>>> performance ratios are unclear). It just seems a lot to wrap every
>>>>>> single line of a non-trivial C++ using function with try..catch
>>>>>> blocks.
>>>
>>>
>>> It seems "a lot" of just what exactly? Generated code? Binary size? Time
>>> spent in GCC parser?
>>
>> All of the above. And we should take a look at the runtime overhead
>> (which is hopefully nil, but who knows.)
>>
>>> Though I guess one might want to try to pull out the try-catch to at
>>> least
>>> only one per code line rather than one per SimpleCallNode.
>>
>> Or even higher, if possible. It's still a lot.
>>
>>> "except *" only has a point when calling functions using the CPython
>>> API,
>>> but most external C functions are pure C, not
>>> CPython-API-using-functions.
>>> OTOH, all external C++ functions are C++ :-)
>>
>> Fair point.
>>
>>> (Also, if we wrote Cython from scratch now I'm pretty sure the
>>> "except *"
>>> defaults would be a tad different.)
>>
>> For sure.
>>
>>>>> But if users are correct about their declarations, we'd end up with
>>>>> the
>>>>> same thing. I think it's worth a try.
>>>>
>>>>
>>>> Most C++ code (that I've ever run into) doesn't use exceptions,
>>>> because exception handling is so broken in C++ anyways.
>>>
>>>
>>> Except for the fact that any code touching "new" could be raising
>>> exceptions? That propagates.
>>
>> I would guess most of the time people don't bother catching these and
>> let the program die, as there's often no sane recovery (the same as
>> MemoryErrors in Python, though I guess C++ is less often used from an
>> event loop).
>
> The point is to provide the user with the stack trace of exactly where
> you ran out of memory. Even if you can't follow it deep into C++-land,
> the Python-side stack trace can be a worth a lot.

Note that I think (I just skimmed gcc documentation) the abort will 
happen the moment you pop back to a C function (unless you compile the C 
code with -fexceptions). I.e. once you're back in the CPython eval loop 
you die, it doesn't propagate back to main(). That means you could crash 
in a lot of callback cases too; your expectation of some kind of 
propagation for callbacks is only true if there's C++ all the way 
(according to my skimming, anyway).

Dag

>
>>
>>> There is a lot of C++ code out there using exceptions. I'd guess that
>>> both
>>> mathematical code and Google-written code is unlike most C++ code out
>>> there
>>> :-) Many C++ programmers go on and on about RAII and auto_ptrs and so
>>> on,
>>> and that doesn't have much point unless you throw an exception now
>>> and then
>>> (OK, there's the occasional return statement where it matters well).
>>
>> True, I've seen a small subset of the C++ code that's out there. Maybe
>> numerical computations use it a lot?
>
> One library I'm using for distributed dense linear algebra (Elemental)
> does. Numerical computations libraries are very different from one another.
>
> Dag
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel


From sturla at molden.no  Wed Jul  4 02:58:14 2012
From: sturla at molden.no (Sturla Molden)
Date: Wed, 04 Jul 2012 02:58:14 +0200
Subject: [Cython] [cython-users] C++: how to handle failures of 'new'?
In-Reply-To: <4FF33D60.5080804@astro.uio.no>
References: <4FECA49B.8090404@behnel.de>
	<CADiQ+QBeBw=FYsNPpSU_OJOU48m7E1=csxMxfenMQLs2jhJgPw@mail.gmail.com>
	<4FED40E1.1040301@behnel.de>
	<CADiQ+QAFYMgycY5-RkTf7SOwDb0-qM93Y8+sEzA=+1P9DSyOyA@mail.gmail.com>
	<4FF29BE3.6030201@behnel.de> <4FF31999.6090501@astro.uio.no>
	<4FF31FF8.10507@behnel.de>
	<CADiQ+QA8nRcpCUsmMx=zb50XQAnOXLVky5pxOxQu=DQxF+BMfw@mail.gmail.com>
	<4FF335D2.3070706@behnel.de>
	<CADiQ+QATCzuJ2vNXJm-QxueG9s_DBPyCBRGhZ1cnMpEMQvMtzQ@mail.gmail.com>
	<4FF33D60.5080804@astro.uio.no>
Message-ID: <4FF39526.9070704@molden.no>

Den 03.07.2012 20:43, skrev Dag Sverre Seljebotn:
>
> Except for the fact that any code touching "new" could be raising 
> exceptions? That propagates.
>
> There is a lot of C++ code out there using exceptions. I'd guess that 
> both mathematical code and Google-written code is unlike most C++ code 
> out there :-) Many C++ programmers go on and on about RAII and 
> auto_ptrs and so on, and that doesn't have much point unless you throw 
> an exception now and then (OK, there's the occasional return statement 
> where it matters well).
>

Usually there is just one C++ exception to care about: std::bad_alloc. 
It is important to know that it can be raised almost anywhere in C++ 
code that are using the STL.

The majority of C++ programs never attempt to catch std::bad_alloc. 
Instead a program will set a global handler routine using 
std::set_new_handler. Usually a callback for memory failure will just 
display an error message and call std::exit with status 
std::EXIT_FAILURE. ||

When interfacing from Python I think a std::bad_alloc should be 
translated to a MemoryError exception if possible.  Though others might 
argue that one should never try to recover from a memory failure. 
Arguably, recovering std::bad_alloc might not be possible if the heap is 
exhausted.

Sturla





-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20120704/e840679b/attachment.html>

From stefan_ml at behnel.de  Wed Jul  4 08:06:07 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 04 Jul 2012 08:06:07 +0200
Subject: [Cython] [cython-users] C++: how to handle failures of 'new'?
In-Reply-To: <4FF39526.9070704@molden.no>
References: <4FECA49B.8090404@behnel.de>
	<CADiQ+QBeBw=FYsNPpSU_OJOU48m7E1=csxMxfenMQLs2jhJgPw@mail.gmail.com>
	<4FED40E1.1040301@behnel.de>
	<CADiQ+QAFYMgycY5-RkTf7SOwDb0-qM93Y8+sEzA=+1P9DSyOyA@mail.gmail.com>
	<4FF29BE3.6030201@behnel.de> <4FF31999.6090501@astro.uio.no>
	<4FF31FF8.10507@behnel.de>
	<CADiQ+QA8nRcpCUsmMx=zb50XQAnOXLVky5pxOxQu=DQxF+BMfw@mail.gmail.com>
	<4FF335D2.3070706@behnel.de>
	<CADiQ+QATCzuJ2vNXJm-QxueG9s_DBPyCBRGhZ1cnMpEMQvMtzQ@mail.gmail.com>
	<4FF33D60.5080804@astro.uio.no> <4FF39526.9070704@molden.no>
Message-ID: <4FF3DD4F.4040600@behnel.de>

Sturla Molden, 04.07.2012 02:58:
> Den 03.07.2012 20:43, skrev Dag Sverre Seljebotn:
>> Except for the fact that any code touching "new" could be raising
>> exceptions? That propagates.
>>
>> There is a lot of C++ code out there using exceptions. I'd guess that
>> both mathematical code and Google-written code is unlike most C++ code
>> out there :-) Many C++ programmers go on and on about RAII and auto_ptrs
>> and so on, and that doesn't have much point unless you throw an exception
>> now and then (OK, there's the occasional return statement where it
>> matters well).
> 
> Usually there is just one C++ exception to care about: std::bad_alloc. It
> is important to know that it can be raised almost anywhere in C++ code that
> are using the STL.
> 
> The majority of C++ programs never attempt to catch std::bad_alloc. Instead
> a program will set a global handler routine using std::set_new_handler.

Interesting. Doesn't make much sense to do that in Cython, though (and
users wouldn't like it if they intend to use it themselves).


> Usually a callback for memory failure will just display an error message
> and call std::exit with status std::EXIT_FAILURE. ||
> 
> When interfacing from Python I think a std::bad_alloc should be translated
> to a MemoryError exception if possible.  Though others might argue that one
> should never try to recover from a memory failure. Arguably, recovering
> std::bad_alloc might not be possible if the heap is exhausted.

Usually, the intention is more to exit gracefully, i.e. to pass through all
finally clauses, than to actually handle the error and continue. And I
would expect that going back along that path would eventually free up some
memory, so that this should actually work in most cases.

Also, the allocation may have failed on a larger block of memory, which is
then unused when the exception gets raised and can be used by cleanup code.
I really don't think the world is all that dark here.

Stefan

From sturla at molden.no  Wed Jul  4 13:03:24 2012
From: sturla at molden.no (Sturla Molden)
Date: Wed, 4 Jul 2012 13:03:24 +0200
Subject: [Cython] [cython-users] C++: how to handle failures of 'new'?
In-Reply-To: <4FF3DD4F.4040600@behnel.de>
References: <4FECA49B.8090404@behnel.de>
	<CADiQ+QBeBw=FYsNPpSU_OJOU48m7E1=csxMxfenMQLs2jhJgPw@mail.gmail.com>
	<4FED40E1.1040301@behnel.de>
	<CADiQ+QAFYMgycY5-RkTf7SOwDb0-qM93Y8+sEzA=+1P9DSyOyA@mail.gmail.com>
	<4FF29BE3.6030201@behnel.de> <4FF31999.6090501@astro.uio.no>
	<4FF31FF8.10507@behnel.de>
	<CADiQ+QA8nRcpCUsmMx=zb50XQAnOXLVky5pxOxQu=DQxF+BMfw@mail.gmail.com>
	<4FF335D2.3070706@behnel.de>
	<CADiQ+QATCzuJ2vNXJm-QxueG9s_DBPyCBRGhZ1cnMpEMQvMtzQ@mail.gmail.com>
	<4FF33D60.5080804@astro.uio.no> <4FF39526.9070704@molden.no>
	<4FF3DD4F.4040600@behnel.de>
Message-ID: <13DAD479-7405-4C53-9FCC-4C5FABA3EE9E@molden.no>



Den 4. juli 2012 kl. 08:06 skrev Stefan Behnel <stefan_ml at behnel.de>:

> 
> 
> Also, the allocation may have failed on a larger block of memory, which is
> then unused when the exception gets raised and can be used by cleanup code.
> I really don't think the world is all that dark here.
> 
> 


Indeed. But how to tell? malloc a small buffer and see if it works?

Sturla




From stefan_ml at behnel.de  Wed Jul  4 14:33:17 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 04 Jul 2012 14:33:17 +0200
Subject: [Cython] [cython-users] C++: how to handle failures of 'new'?
In-Reply-To: <13DAD479-7405-4C53-9FCC-4C5FABA3EE9E@molden.no>
References: <4FECA49B.8090404@behnel.de>
	<CADiQ+QBeBw=FYsNPpSU_OJOU48m7E1=csxMxfenMQLs2jhJgPw@mail.gmail.com>
	<4FED40E1.1040301@behnel.de>
	<CADiQ+QAFYMgycY5-RkTf7SOwDb0-qM93Y8+sEzA=+1P9DSyOyA@mail.gmail.com>
	<4FF29BE3.6030201@behnel.de> <4FF31999.6090501@astro.uio.no>
	<4FF31FF8.10507@behnel.de>
	<CADiQ+QA8nRcpCUsmMx=zb50XQAnOXLVky5pxOxQu=DQxF+BMfw@mail.gmail.com>
	<4FF335D2.3070706@behnel.de>
	<CADiQ+QATCzuJ2vNXJm-QxueG9s_DBPyCBRGhZ1cnMpEMQvMtzQ@mail.gmail.com>
	<4FF33D60.5080804@astro.uio.no> <4FF39526.9070704@molden.no>
	<4FF3DD4F.4040600@behnel.de>
	<13DAD479-7405-4C53-9FCC-4C5FABA3EE9E@molden.no>
Message-ID: <4FF4380D.1070503@behnel.de>

Sturla Molden, 04.07.2012 13:03:
> Den 4. juli 2012 kl. 08:06 skrev Stefan Behnel:
>> Also, the allocation may have failed on a larger block of memory, which is
>> then unused when the exception gets raised and can be used by cleanup code.
>> I really don't think the world is all that dark here.
> 
> Indeed. But how to tell? malloc a small buffer and see if it works?

In the worst case, you'd get another memory error during cleanup and it
would keep rippling up the stack.

Stefan

From sturla at molden.no  Wed Jul  4 15:44:01 2012
From: sturla at molden.no (Sturla Molden)
Date: Wed, 4 Jul 2012 15:44:01 +0200
Subject: [Cython] [cython-users] C++: how to handle failures of 'new'?
In-Reply-To: <4FF4380D.1070503@behnel.de>
References: <4FECA49B.8090404@behnel.de>
	<CADiQ+QBeBw=FYsNPpSU_OJOU48m7E1=csxMxfenMQLs2jhJgPw@mail.gmail.com>
	<4FED40E1.1040301@behnel.de>
	<CADiQ+QAFYMgycY5-RkTf7SOwDb0-qM93Y8+sEzA=+1P9DSyOyA@mail.gmail.com>
	<4FF29BE3.6030201@behnel.de> <4FF31999.6090501@astro.uio.no>
	<4FF31FF8.10507@behnel.de>
	<CADiQ+QA8nRcpCUsmMx=zb50XQAnOXLVky5pxOxQu=DQxF+BMfw@mail.gmail.com>
	<4FF335D2.3070706@behnel.de>
	<CADiQ+QATCzuJ2vNXJm-QxueG9s_DBPyCBRGhZ1cnMpEMQvMtzQ@mail.gmail.com>
	<4FF33D60.5080804@astro.uio.no> <4FF39526.9070704@molden.no>
	<4FF3DD4F.4040600@behnel.de>
	<13DAD479-7405-4C53-9FCC-4C5FABA3EE9E@molden.no>
	<4FF4380D.1070503@behnel.de>
Message-ID: <CBB44D5B-644E-46AA-9621-5A15DB4E3172@molden.no>



Sendt fra min iPad

Den 4. juli 2012 kl. 14:33 skrev Stefan Behnel <stefan_ml at behnel.de>:

>> 
>> Indeed. But how to tell? malloc a small buffer and see if it works?
> 
> In the worst case, you'd get another memory error during cleanup and it
> would keep rippling up the stack.
> 
> 

Which is why I wrote 'malloc' instead of 'new'. It doesn't throw a new exception. :-)

Sturla







From d.s.seljebotn at astro.uio.no  Wed Jul  4 21:50:19 2012
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Wed, 04 Jul 2012 21:50:19 +0200
Subject: [Cython] [cython-users] C++: how to handle failures of 'new'?
In-Reply-To: <CBB44D5B-644E-46AA-9621-5A15DB4E3172@molden.no>
References: <4FECA49B.8090404@behnel.de>
	<CADiQ+QBeBw=FYsNPpSU_OJOU48m7E1=csxMxfenMQLs2jhJgPw@mail.gmail.com>
	<4FED40E1.1040301@behnel.de>
	<CADiQ+QAFYMgycY5-RkTf7SOwDb0-qM93Y8+sEzA=+1P9DSyOyA@mail.gmail.com>
	<4FF29BE3.6030201@behnel.de> <4FF31999.6090501@astro.uio.no>
	<4FF31FF8.10507@behnel.de>
	<CADiQ+QA8nRcpCUsmMx=zb50XQAnOXLVky5pxOxQu=DQxF+BMfw@mail.gmail.com>
	<4FF335D2.3070706@behnel.de>
	<CADiQ+QATCzuJ2vNXJm-QxueG9s_DBPyCBRGhZ1cnMpEMQvMtzQ@mail.gmail.com>
	<4FF33D60.5080804@astro.uio.no> <4FF39526.9070704@molden.no>
	<4FF3DD4F.4040600@behnel.de>
	<13DAD479-7405-4C53-9FCC-4C5FABA3EE9E@molden.no>
	<4FF4380D.1070503@behnel.de>
	<CBB44D5B-644E-46AA-9621-5A15DB4E3172@molden.no>
Message-ID: <0ae59ec2-a344-488a-976f-44cca8ddb460@email.android.com>



Sturla Molden <sturla at molden.no> wrote:

>
>
>Sendt fra min iPad
>
>Den 4. juli 2012 kl. 14:33 skrev Stefan Behnel <stefan_ml at behnel.de>:
>
>>> 
>>> Indeed. But how to tell? malloc a small buffer and see if it works?
>> 
>> In the worst case, you'd get another memory error during cleanup and
>it
>> would keep rippling up the stack.
>> 
>> 
>
>Which is why I wrote 'malloc' instead of 'new'. It doesn't throw a new
>exception. :-)

I think Stefan's point was you don't need to tell.

I don't understand what Sturla is getting at either...are you saying that you can't rely on the C++ exception type (std::bad_alloc or whatever it is?) and translate that to MemoryError? I think one would translate a catch(...) to a RuntimeError (or a subclass)

At any rate probing will never work, the failing allocation *could* be that a 50 GB buffer was requested (in fact that's when it is extremely convenient to have a stack trace).

Dag


>
>Sturla
>
>
>
>
>
>
>_______________________________________________
>cython-devel mailing list
>cython-devel at python.org
>http://mail.python.org/mailman/listinfo/cython-devel

-- 
Sent from my Android phone with K-9 Mail. Please excuse my brevity.

From stefan_ml at behnel.de  Thu Jul  5 19:36:28 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 05 Jul 2012 19:36:28 +0200
Subject: [Cython] Cython+PyPy benchmarks
Message-ID: <4FF5D09C.2030404@behnel.de>

Hi,

I set up a Jenkins job to run a couple of (simple) benchmarks comparing
Cython's current performance under CPython and PyPy. Note that these are
C-API intensive benchmarks by design.

https://sage.math.washington.edu:8091/hudson/job/cython-devel-cybenchmarks-pypy/lastSuccessfulBuild/artifact/bench_chart.html

Basically, PyPy's cpyext is currently about 100-200x slower than CPython's
native C-API for these kinds of benchmarks. That's because it hasn't been
optimised in any way, correctness and completeness are still the main goals
in its development (and they're not there yet).

The one major performance issue in cpyext is currently the creation and
deallocation of the PyObject representation for each object, which
obviously has a huge impact on everything. I profiled the nbody benchmark
and it showed that almost 80% of the runtime is currently spent in creating
and discarding PyObject instances. Here's the call graph:

http://cython.org/callgrind-pypy-nbody.png

The up side of this is that there is likely a lot of low hanging fruit in
cpyext (plus some more tweaks in Cython), given that no optimisation at all
has been done so far. It shouldn't be too hard to drop the factor
substantially.

I also think we should add a couple of more C-ish benchmarks to see how
much overhead there really is for less C-API intensive code.

Stefan

From markflorisson88 at gmail.com  Thu Jul  5 20:47:43 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Thu, 5 Jul 2012 19:47:43 +0100
Subject: [Cython] [cython-users] C++: how to handle failures of 'new'?
In-Reply-To: <CADiQ+QD0etaANiRy=03=yTjCi60xCpN+GVV5=mueWuQnBotrDQ@mail.gmail.com>
References: <4FECA49B.8090404@behnel.de>
	<CADiQ+QBeBw=FYsNPpSU_OJOU48m7E1=csxMxfenMQLs2jhJgPw@mail.gmail.com>
	<4FED40E1.1040301@behnel.de>
	<CADiQ+QAFYMgycY5-RkTf7SOwDb0-qM93Y8+sEzA=+1P9DSyOyA@mail.gmail.com>
	<4FF29BE3.6030201@behnel.de> <4FF31999.6090501@astro.uio.no>
	<4FF31FF8.10507@behnel.de>
	<CADiQ+QA8nRcpCUsmMx=zb50XQAnOXLVky5pxOxQu=DQxF+BMfw@mail.gmail.com>
	<4FF335D2.3070706@behnel.de>
	<CADiQ+QATCzuJ2vNXJm-QxueG9s_DBPyCBRGhZ1cnMpEMQvMtzQ@mail.gmail.com>
	<4FF33D60.5080804@astro.uio.no>
	<CADiQ+QD0etaANiRy=03=yTjCi60xCpN+GVV5=mueWuQnBotrDQ@mail.gmail.com>
Message-ID: <CANg26EVTZC0mfxbcka_v8fE3try5YkLgc8x-VyvBh0kooT97iA@mail.gmail.com>

On 3 July 2012 20:15, Robert Bradshaw <robertwb at gmail.com> wrote:
> On Tue, Jul 3, 2012 at 11:43 AM, Dag Sverre Seljebotn
> <d.s.seljebotn at astro.uio.no> wrote:
>> On 07/03/2012 08:23 PM, Robert Bradshaw wrote:
>>>
>>> On Tue, Jul 3, 2012 at 11:11 AM, Stefan Behnel<stefan_ml at behnel.de>
>>> wrote:
>>>>
>>>> Robert Bradshaw, 03.07.2012 19:58:
>>>>>
>>>>> On Tue, Jul 3, 2012 at 9:38 AM, Stefan Behnel wrote:
>>>>>>
>>>>>> Dag Sverre Seljebotn, 03.07.2012 18:11:
>>>>>>>
>>>>>>> On 07/03/2012 09:14 AM, Stefan Behnel wrote:
>>>>>>>>
>>>>>>>> I don't know what happens if a C++ exception is not being caught, but
>>>>>>>> I
>>>>>>>> guess it would simply crash the application. That's a bit more
>>>>>>>> visible than
>>>>>>>
>>>>>>>
>>>>>>> Yep.
>>>>>>>
>>>>>>>> just printing a warning when a Python exception is being ignored due
>>>>>>>> to a
>>>>>>>> missing declaration. It's really unfortunate that our documentation
>>>>>>>> didn't
>>>>>>>> even mention the need for this, because it's not immediately obvious
>>>>>>>> that
>>>>>>>> Cython won't handle errors in "new", and testing for memory errors
>>>>>>>> isn't
>>>>>>>> quite what people commonly do in their test suites.
>>>>>>>>
>>>>>>>> Apart from that, I agree, users have to take care to properly declare
>>>>>>>> the
>>>>>>>> API they are using.
>>>>>>>
>>>>>>>
>>>>>>> Is there any time you do NOT want a "catch (...) {}" block? I can't
>>>>>>> see a
>>>>>>> C++ exception propagating to Python-land doing anything useful ever.
>>>>>>
>>>>>>
>>>>>> That would have been my intuition, too.
>>>>>
>>>>>
>>>>> If it's actually embedded, with the main driver in C++, one might want
>>>>> it to propagate up.
>>>>
>>>>
>>>> But what kind of a propagation would that be? On the way out, it could
>>>> induce anything, from side effects to resource leaks to crashes,
>>>> depending
>>>> on what the state of the surrounding code is. It would leave the whole
>>>> system in an unpredictable state. I cannot imagine anyone really wanting
>>>> this.
>>>>
>>>>
>>>>>>> So shouldn't we just make --cplus turn *all* external functions and
>>>>>>> methods
>>>>>>> (whether C-like or C++-like) into "except +"? (Or keep except+ for
>>>>>>> manual
>>>>>>> translation, but always have a catch(...)".
>>>>>>>
>>>>>>> Performance overhead is the only reason I can think of to not do this,
>>>>>>> although IIRC C++ catch blocks are only dealt with during stack
>>>>>>> unwinds and
>>>>>>> doesn't cost anything/much (?) when they're not triggered.
>>>>>>>
>>>>>>> "except -1" should then actually mean both; "except + except -1". So
>>>>>>> it's
>>>>>>> more a question of just adding catch(...) *everywhere*, than making
>>>>>>> "except
>>>>>>> +" the default.
>>>>>>
>>>>>>
>>>>>> I have no idea if there is a performance impact, but if there isn't,
>>>>>> always
>>>>>> catching all exceptions sounds like a reasonable thing to do. After
>>>>>> all, we
>>>>>> have no support for catching C++ exceptions on user side.
>>>>>
>>>>>
>>>>> This is a bit like following every C call with "except *" (though the
>>>>> performance ratios are unclear). It just seems a lot to wrap every
>>>>> single line of a non-trivial C++ using function with try..catch
>>>>> blocks.
>>
>>
>> It seems "a lot" of just what exactly? Generated code? Binary size? Time
>> spent in GCC parser?
>
> All of the above. And we should take a look at the runtime overhead
> (which is hopefully nil, but who knows.)
>
>> Though I guess one might want to try to pull out the try-catch to at least
>> only one per code line rather than one per SimpleCallNode.
>
> Or even higher, if possible. It's still a lot.

Why would you have to do that? Can't you just insert a try/catch per
try/except or try/finally block, or if absent, the function body. That
will still work with the way temporaries are cleaned up. (It should
also be implemented for parallel/prange sections).

>> "except *" only has a point when calling functions using the CPython API,
>> but most external C functions are pure C, not CPython-API-using-functions.
>> OTOH, all external C++ functions are C++ :-)
>
> Fair point.
>
>> (Also, if we wrote Cython from scratch now I'm pretty sure the "except *"
>> defaults would be a tad different.)
>
> For sure.
>
>>>> But if users are correct about their declarations, we'd end up with the
>>>> same thing. I think it's worth a try.
>>>
>>>
>>> Most C++ code (that I've ever run into) doesn't use exceptions,
>>> because exception handling is so broken in C++ anyways.
>>
>>
>> Except for the fact that any code touching "new" could be raising
>> exceptions? That propagates.
>
> I would guess most of the time people don't bother catching these and
> let the program die, as there's often no sane recovery (the same as
> MemoryErrors in Python, though I guess C++ is less often used from an
> event loop).
>
>> There is a lot of C++ code out there using exceptions. I'd guess that both
>> mathematical code and Google-written code is unlike most C++ code out there
>> :-) Many C++ programmers go on and on about RAII and auto_ptrs and so on,
>> and that doesn't have much point unless you throw an exception now and then
>> (OK, there's the occasional return statement where it matters well).
>
> True, I've seen a small subset of the C++ code that's out there. Maybe
> numerical computations use it a lot?
>
> +1 to making catch-everywhere a directive at least. Lets see what the
> impact is before we decide to make it the default.
>
> - Robert
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

From stefan_ml at behnel.de  Thu Jul  5 22:11:34 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 05 Jul 2012 22:11:34 +0200
Subject: [Cython] [cython-users] C++: how to handle failures of 'new'?
In-Reply-To: <CANg26EVTZC0mfxbcka_v8fE3try5YkLgc8x-VyvBh0kooT97iA@mail.gmail.com>
References: <4FECA49B.8090404@behnel.de>
	<CADiQ+QBeBw=FYsNPpSU_OJOU48m7E1=csxMxfenMQLs2jhJgPw@mail.gmail.com>
	<4FED40E1.1040301@behnel.de>
	<CADiQ+QAFYMgycY5-RkTf7SOwDb0-qM93Y8+sEzA=+1P9DSyOyA@mail.gmail.com>
	<4FF29BE3.6030201@behnel.de> <4FF31999.6090501@astro.uio.no>
	<4FF31FF8.10507@behnel.de>
	<CADiQ+QA8nRcpCUsmMx=zb50XQAnOXLVky5pxOxQu=DQxF+BMfw@mail.gmail.com>
	<4FF335D2.3070706@behnel.de>
	<CADiQ+QATCzuJ2vNXJm-QxueG9s_DBPyCBRGhZ1cnMpEMQvMtzQ@mail.gmail.com>
	<4FF33D60.5080804@astro.uio.no>
	<CADiQ+QD0etaANiRy=03=yTjCi60xCpN+GVV5=mueWuQnBotrDQ@mail.gmail.com>
	<CANg26EVTZC0mfxbcka_v8fE3try5YkLgc8x-VyvBh0kooT97iA@mail.gmail.com>
Message-ID: <4FF5F4F6.60104@behnel.de>

mark florisson, 05.07.2012 20:47:
> On 3 July 2012 20:15, Robert Bradshaw wrote:
>> On Tue, Jul 3, 2012 at 11:43 AM, Dag Sverre Seljebotn
>>> Though I guess one might want to try to pull out the try-catch to at least
>>> only one per code line rather than one per SimpleCallNode.
>>
>> Or even higher, if possible. It's still a lot.
> 
> Why would you have to do that? Can't you just insert a try/catch per
> try/except or try/finally block, or if absent, the function body. That
> will still work with the way temporaries are cleaned up. (It should
> also be implemented for parallel/prange sections).

My first reaction was, "sure, smart idea". It certainly sounds like a good
idea to unify the exception handling between C++ and Python into the same
syntactic structures.

But does it allow to handle different declarations for multiple C++
functions that get called? E.g. "except +" for one and "except
+MemoryError" for another, but both called in the same try-whatever block?
That would just lead to nested try-except blocks, I guess, thus making the
outer exception clauses mostly a fallback for exceptions that users forgot
to declare or couldn't properly handle for some reason.

I think it's worth a try to see if it works.

BTW, is there a reason we can't allow users to declare C++ exceptions in
their .pxd files, and then support catching them in Python try-except
syntax? Just verbatimly translating them to the C++ structures, based on
the type of the exception that gets caught?

(Although, given the discussion so far, maybe try-finally is more important
than try-catch, and the former can't know when it needs to be mapped into
C++ code and when not ...)

Stefan

From d.s.seljebotn at astro.uio.no  Thu Jul  5 22:46:16 2012
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Thu, 05 Jul 2012 22:46:16 +0200
Subject: [Cython] [cython-users] C++: how to handle failures of 'new'?
In-Reply-To: <CANg26EVTZC0mfxbcka_v8fE3try5YkLgc8x-VyvBh0kooT97iA@mail.gmail.com>
References: <4FECA49B.8090404@behnel.de>
	<CADiQ+QBeBw=FYsNPpSU_OJOU48m7E1=csxMxfenMQLs2jhJgPw@mail.gmail.com>
	<4FED40E1.1040301@behnel.de>
	<CADiQ+QAFYMgycY5-RkTf7SOwDb0-qM93Y8+sEzA=+1P9DSyOyA@mail.gmail.com>
	<4FF29BE3.6030201@behnel.de> <4FF31999.6090501@astro.uio.no>
	<4FF31FF8.10507@behnel.de>
	<CADiQ+QA8nRcpCUsmMx=zb50XQAnOXLVky5pxOxQu=DQxF+BMfw@mail.gmail.com>
	<4FF335D2.3070706@behnel.de>
	<CADiQ+QATCzuJ2vNXJm-QxueG9s_DBPyCBRGhZ1cnMpEMQvMtzQ@mail.gmail.com>
	<4FF33D60.5080804@astro.uio.no>
	<CADiQ+QD0etaANiRy=03=yTjCi60xCpN+GVV5=mueWuQnBotrDQ@mail.gmail.com>
	<CANg26EVTZC0mfxbcka_v8fE3try5YkLgc8x-VyvBh0kooT97iA@mail.gmail.com>
Message-ID: <23685ab8-5945-418f-b3f6-0c7a92a03bea@email.android.com>



mark florisson <markflorisson88 at gmail.com> wrote:

>On 3 July 2012 20:15, Robert Bradshaw <robertwb at gmail.com> wrote:
>> On Tue, Jul 3, 2012 at 11:43 AM, Dag Sverre Seljebotn
>> <d.s.seljebotn at astro.uio.no> wrote:
>>> On 07/03/2012 08:23 PM, Robert Bradshaw wrote:
>>>>
>>>> On Tue, Jul 3, 2012 at 11:11 AM, Stefan Behnel<stefan_ml at behnel.de>
>>>> wrote:
>>>>>
>>>>> Robert Bradshaw, 03.07.2012 19:58:
>>>>>>
>>>>>> On Tue, Jul 3, 2012 at 9:38 AM, Stefan Behnel wrote:
>>>>>>>
>>>>>>> Dag Sverre Seljebotn, 03.07.2012 18:11:
>>>>>>>>
>>>>>>>> On 07/03/2012 09:14 AM, Stefan Behnel wrote:
>>>>>>>>>
>>>>>>>>> I don't know what happens if a C++ exception is not being
>caught, but
>>>>>>>>> I
>>>>>>>>> guess it would simply crash the application. That's a bit more
>>>>>>>>> visible than
>>>>>>>>
>>>>>>>>
>>>>>>>> Yep.
>>>>>>>>
>>>>>>>>> just printing a warning when a Python exception is being
>ignored due
>>>>>>>>> to a
>>>>>>>>> missing declaration. It's really unfortunate that our
>documentation
>>>>>>>>> didn't
>>>>>>>>> even mention the need for this, because it's not immediately
>obvious
>>>>>>>>> that
>>>>>>>>> Cython won't handle errors in "new", and testing for memory
>errors
>>>>>>>>> isn't
>>>>>>>>> quite what people commonly do in their test suites.
>>>>>>>>>
>>>>>>>>> Apart from that, I agree, users have to take care to properly
>declare
>>>>>>>>> the
>>>>>>>>> API they are using.
>>>>>>>>
>>>>>>>>
>>>>>>>> Is there any time you do NOT want a "catch (...) {}" block? I
>can't
>>>>>>>> see a
>>>>>>>> C++ exception propagating to Python-land doing anything useful
>ever.
>>>>>>>
>>>>>>>
>>>>>>> That would have been my intuition, too.
>>>>>>
>>>>>>
>>>>>> If it's actually embedded, with the main driver in C++, one might
>want
>>>>>> it to propagate up.
>>>>>
>>>>>
>>>>> But what kind of a propagation would that be? On the way out, it
>could
>>>>> induce anything, from side effects to resource leaks to crashes,
>>>>> depending
>>>>> on what the state of the surrounding code is. It would leave the
>whole
>>>>> system in an unpredictable state. I cannot imagine anyone really
>wanting
>>>>> this.
>>>>>
>>>>>
>>>>>>>> So shouldn't we just make --cplus turn *all* external functions
>and
>>>>>>>> methods
>>>>>>>> (whether C-like or C++-like) into "except +"? (Or keep except+
>for
>>>>>>>> manual
>>>>>>>> translation, but always have a catch(...)".
>>>>>>>>
>>>>>>>> Performance overhead is the only reason I can think of to not
>do this,
>>>>>>>> although IIRC C++ catch blocks are only dealt with during stack
>>>>>>>> unwinds and
>>>>>>>> doesn't cost anything/much (?) when they're not triggered.
>>>>>>>>
>>>>>>>> "except -1" should then actually mean both; "except + except
>-1". So
>>>>>>>> it's
>>>>>>>> more a question of just adding catch(...) *everywhere*, than
>making
>>>>>>>> "except
>>>>>>>> +" the default.
>>>>>>>
>>>>>>>
>>>>>>> I have no idea if there is a performance impact, but if there
>isn't,
>>>>>>> always
>>>>>>> catching all exceptions sounds like a reasonable thing to do.
>After
>>>>>>> all, we
>>>>>>> have no support for catching C++ exceptions on user side.
>>>>>>
>>>>>>
>>>>>> This is a bit like following every C call with "except *" (though
>the
>>>>>> performance ratios are unclear). It just seems a lot to wrap
>every
>>>>>> single line of a non-trivial C++ using function with try..catch
>>>>>> blocks.
>>>
>>>
>>> It seems "a lot" of just what exactly? Generated code? Binary size?
>Time
>>> spent in GCC parser?
>>
>> All of the above. And we should take a look at the runtime overhead
>> (which is hopefully nil, but who knows.)
>>
>>> Though I guess one might want to try to pull out the try-catch to at
>least
>>> only one per code line rather than one per SimpleCallNode.
>>
>> Or even higher, if possible. It's still a lot.
>
>Why would you have to do that? Can't you just insert a try/catch per
>try/except or try/finally block, or if absent, the function body. That
>will still work with the way temporaries are cleaned up. (It should
>also be implemented for parallel/prange sections).

One disadvantage is that you don't get source code line for the .pyx file in the stack trace. Which is often exactly the information you are looking for (even worse, since C++ stack isn't in the stack trace, the lineno for what seems like the ' ultimate cause' is not there). Having to surround statements with try/except just to pinpoint which one is raising the exception would be incredibly irritating.

Dag

>
>>> "except *" only has a point when calling functions using the CPython
>API,
>>> but most external C functions are pure C, not
>CPython-API-using-functions.
>>> OTOH, all external C++ functions are C++ :-)
>>
>> Fair point.
>>
>>> (Also, if we wrote Cython from scratch now I'm pretty sure the
>"except *"
>>> defaults would be a tad different.)
>>
>> For sure.
>>
>>>>> But if users are correct about their declarations, we'd end up
>with the
>>>>> same thing. I think it's worth a try.
>>>>
>>>>
>>>> Most C++ code (that I've ever run into) doesn't use exceptions,
>>>> because exception handling is so broken in C++ anyways.
>>>
>>>
>>> Except for the fact that any code touching "new" could be raising
>>> exceptions? That propagates.
>>
>> I would guess most of the time people don't bother catching these and
>> let the program die, as there's often no sane recovery (the same as
>> MemoryErrors in Python, though I guess C++ is less often used from an
>> event loop).
>>
>>> There is a lot of C++ code out there using exceptions. I'd guess
>that both
>>> mathematical code and Google-written code is unlike most C++ code
>out there
>>> :-) Many C++ programmers go on and on about RAII and auto_ptrs and
>so on,
>>> and that doesn't have much point unless you throw an exception now
>and then
>>> (OK, there's the occasional return statement where it matters well).
>>
>> True, I've seen a small subset of the C++ code that's out there.
>Maybe
>> numerical computations use it a lot?
>>
>> +1 to making catch-everywhere a directive at least. Lets see what the
>> impact is before we decide to make it the default.
>>
>> - Robert
>> _______________________________________________
>> cython-devel mailing list
>> cython-devel at python.org
>> http://mail.python.org/mailman/listinfo/cython-devel
>_______________________________________________
>cython-devel mailing list
>cython-devel at python.org
>http://mail.python.org/mailman/listinfo/cython-devel

-- 
Sent from my Android phone with K-9 Mail. Please excuse my brevity.

From markflorisson88 at gmail.com  Thu Jul  5 23:02:02 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Thu, 5 Jul 2012 22:02:02 +0100
Subject: [Cython] [cython-users] C++: how to handle failures of 'new'?
In-Reply-To: <23685ab8-5945-418f-b3f6-0c7a92a03bea@email.android.com>
References: <4FECA49B.8090404@behnel.de>
	<CADiQ+QBeBw=FYsNPpSU_OJOU48m7E1=csxMxfenMQLs2jhJgPw@mail.gmail.com>
	<4FED40E1.1040301@behnel.de>
	<CADiQ+QAFYMgycY5-RkTf7SOwDb0-qM93Y8+sEzA=+1P9DSyOyA@mail.gmail.com>
	<4FF29BE3.6030201@behnel.de> <4FF31999.6090501@astro.uio.no>
	<4FF31FF8.10507@behnel.de>
	<CADiQ+QA8nRcpCUsmMx=zb50XQAnOXLVky5pxOxQu=DQxF+BMfw@mail.gmail.com>
	<4FF335D2.3070706@behnel.de>
	<CADiQ+QATCzuJ2vNXJm-QxueG9s_DBPyCBRGhZ1cnMpEMQvMtzQ@mail.gmail.com>
	<4FF33D60.5080804@astro.uio.no>
	<CADiQ+QD0etaANiRy=03=yTjCi60xCpN+GVV5=mueWuQnBotrDQ@mail.gmail.com>
	<CANg26EVTZC0mfxbcka_v8fE3try5YkLgc8x-VyvBh0kooT97iA@mail.gmail.com>
	<23685ab8-5945-418f-b3f6-0c7a92a03bea@email.android.com>
Message-ID: <CANg26EWp-fqysQeE58hVDeWtWZJ9p+sPem8MOwaEs1tUdqcKZg@mail.gmail.com>

On 5 July 2012 21:46, Dag Sverre Seljebotn <d.s.seljebotn at astro.uio.no> wrote:
>
>
> mark florisson <markflorisson88 at gmail.com> wrote:
>
>>On 3 July 2012 20:15, Robert Bradshaw <robertwb at gmail.com> wrote:
>>> On Tue, Jul 3, 2012 at 11:43 AM, Dag Sverre Seljebotn
>>> <d.s.seljebotn at astro.uio.no> wrote:
>>>> On 07/03/2012 08:23 PM, Robert Bradshaw wrote:
>>>>>
>>>>> On Tue, Jul 3, 2012 at 11:11 AM, Stefan Behnel<stefan_ml at behnel.de>
>>>>> wrote:
>>>>>>
>>>>>> Robert Bradshaw, 03.07.2012 19:58:
>>>>>>>
>>>>>>> On Tue, Jul 3, 2012 at 9:38 AM, Stefan Behnel wrote:
>>>>>>>>
>>>>>>>> Dag Sverre Seljebotn, 03.07.2012 18:11:
>>>>>>>>>
>>>>>>>>> On 07/03/2012 09:14 AM, Stefan Behnel wrote:
>>>>>>>>>>
>>>>>>>>>> I don't know what happens if a C++ exception is not being
>>caught, but
>>>>>>>>>> I
>>>>>>>>>> guess it would simply crash the application. That's a bit more
>>>>>>>>>> visible than
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Yep.
>>>>>>>>>
>>>>>>>>>> just printing a warning when a Python exception is being
>>ignored due
>>>>>>>>>> to a
>>>>>>>>>> missing declaration. It's really unfortunate that our
>>documentation
>>>>>>>>>> didn't
>>>>>>>>>> even mention the need for this, because it's not immediately
>>obvious
>>>>>>>>>> that
>>>>>>>>>> Cython won't handle errors in "new", and testing for memory
>>errors
>>>>>>>>>> isn't
>>>>>>>>>> quite what people commonly do in their test suites.
>>>>>>>>>>
>>>>>>>>>> Apart from that, I agree, users have to take care to properly
>>declare
>>>>>>>>>> the
>>>>>>>>>> API they are using.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Is there any time you do NOT want a "catch (...) {}" block? I
>>can't
>>>>>>>>> see a
>>>>>>>>> C++ exception propagating to Python-land doing anything useful
>>ever.
>>>>>>>>
>>>>>>>>
>>>>>>>> That would have been my intuition, too.
>>>>>>>
>>>>>>>
>>>>>>> If it's actually embedded, with the main driver in C++, one might
>>want
>>>>>>> it to propagate up.
>>>>>>
>>>>>>
>>>>>> But what kind of a propagation would that be? On the way out, it
>>could
>>>>>> induce anything, from side effects to resource leaks to crashes,
>>>>>> depending
>>>>>> on what the state of the surrounding code is. It would leave the
>>whole
>>>>>> system in an unpredictable state. I cannot imagine anyone really
>>wanting
>>>>>> this.
>>>>>>
>>>>>>
>>>>>>>>> So shouldn't we just make --cplus turn *all* external functions
>>and
>>>>>>>>> methods
>>>>>>>>> (whether C-like or C++-like) into "except +"? (Or keep except+
>>for
>>>>>>>>> manual
>>>>>>>>> translation, but always have a catch(...)".
>>>>>>>>>
>>>>>>>>> Performance overhead is the only reason I can think of to not
>>do this,
>>>>>>>>> although IIRC C++ catch blocks are only dealt with during stack
>>>>>>>>> unwinds and
>>>>>>>>> doesn't cost anything/much (?) when they're not triggered.
>>>>>>>>>
>>>>>>>>> "except -1" should then actually mean both; "except + except
>>-1". So
>>>>>>>>> it's
>>>>>>>>> more a question of just adding catch(...) *everywhere*, than
>>making
>>>>>>>>> "except
>>>>>>>>> +" the default.
>>>>>>>>
>>>>>>>>
>>>>>>>> I have no idea if there is a performance impact, but if there
>>isn't,
>>>>>>>> always
>>>>>>>> catching all exceptions sounds like a reasonable thing to do.
>>After
>>>>>>>> all, we
>>>>>>>> have no support for catching C++ exceptions on user side.
>>>>>>>
>>>>>>>
>>>>>>> This is a bit like following every C call with "except *" (though
>>the
>>>>>>> performance ratios are unclear). It just seems a lot to wrap
>>every
>>>>>>> single line of a non-trivial C++ using function with try..catch
>>>>>>> blocks.
>>>>
>>>>
>>>> It seems "a lot" of just what exactly? Generated code? Binary size?
>>Time
>>>> spent in GCC parser?
>>>
>>> All of the above. And we should take a look at the runtime overhead
>>> (which is hopefully nil, but who knows.)
>>>
>>>> Though I guess one might want to try to pull out the try-catch to at
>>least
>>>> only one per code line rather than one per SimpleCallNode.
>>>
>>> Or even higher, if possible. It's still a lot.
>>
>>Why would you have to do that? Can't you just insert a try/catch per
>>try/except or try/finally block, or if absent, the function body. That
>>will still work with the way temporaries are cleaned up. (It should
>>also be implemented for parallel/prange sections).
>
> One disadvantage is that you don't get source code line for the .pyx file in the stack trace. Which is often exactly the information you are looking for (even worse, since C++ stack isn't in the stack trace, the lineno for what seems like the ' ultimate cause' is not there). Having to surround statements with try/except just to pinpoint which one is raising the exception would be incredibly irritating.
>
> Dag

Oh yeah, good point. Maybe we could use these zero-cost exceptions for
cdef functions in Cython though, instead of error checks (if it
appears to make any significant difference). Basically instead of the
'error' argument in CEP 526. It'd need version that ABI as well...

>>
>>>> "except *" only has a point when calling functions using the CPython
>>API,
>>>> but most external C functions are pure C, not
>>CPython-API-using-functions.
>>>> OTOH, all external C++ functions are C++ :-)
>>>
>>> Fair point.
>>>
>>>> (Also, if we wrote Cython from scratch now I'm pretty sure the
>>"except *"
>>>> defaults would be a tad different.)
>>>
>>> For sure.
>>>
>>>>>> But if users are correct about their declarations, we'd end up
>>with the
>>>>>> same thing. I think it's worth a try.
>>>>>
>>>>>
>>>>> Most C++ code (that I've ever run into) doesn't use exceptions,
>>>>> because exception handling is so broken in C++ anyways.
>>>>
>>>>
>>>> Except for the fact that any code touching "new" could be raising
>>>> exceptions? That propagates.
>>>
>>> I would guess most of the time people don't bother catching these and
>>> let the program die, as there's often no sane recovery (the same as
>>> MemoryErrors in Python, though I guess C++ is less often used from an
>>> event loop).
>>>
>>>> There is a lot of C++ code out there using exceptions. I'd guess
>>that both
>>>> mathematical code and Google-written code is unlike most C++ code
>>out there
>>>> :-) Many C++ programmers go on and on about RAII and auto_ptrs and
>>so on,
>>>> and that doesn't have much point unless you throw an exception now
>>and then
>>>> (OK, there's the occasional return statement where it matters well).
>>>
>>> True, I've seen a small subset of the C++ code that's out there.
>>Maybe
>>> numerical computations use it a lot?
>>>
>>> +1 to making catch-everywhere a directive at least. Lets see what the
>>> impact is before we decide to make it the default.
>>>
>>> - Robert
>>> _______________________________________________
>>> cython-devel mailing list
>>> cython-devel at python.org
>>> http://mail.python.org/mailman/listinfo/cython-devel
>>_______________________________________________
>>cython-devel mailing list
>>cython-devel at python.org
>>http://mail.python.org/mailman/listinfo/cython-devel
>
> --
> Sent from my Android phone with K-9 Mail. Please excuse my brevity.
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

From barry at python.org  Fri Jul  6 00:29:19 2012
From: barry at python.org (Barry Warsaw)
Date: Thu, 5 Jul 2012 18:29:19 -0400
Subject: [Cython] Odd behavior with std::string and .decode()
Message-ID: <20120705182919.5d526c85@resist.wooz.org>

I'm currently exploring using Cython to provide new Python 3 bindings for
Xapian.  I'm pretty much a Cython n00b but the documentation is great, and I
was able to pretty quickly get something really simple working.  I'm using
Cython 0.15 on Ubuntu 12.04 with Python 3.2 and Xapian 1.2.12.  I've pushed my
current branch to github:

https://github.com/warsaw/xapian/tree/py3/xapian-bindings/python3

There you'll see my xapianlib.pxd and xapian.pyx files.

Where I'm seeing some odd behavior is in trying to expose the
Xapian::TermGenerator.get_description() method.  This returns a std::string
and I'm trying to create a `description` property that coerces this to unicode
before returning it to Python.  Here's the relevant code:

-----snip snip-----
cdef class TermGenerator:
    cdef xapianlib.TermGenerator * _this

    def __cinit__(self):
        self._this = new xapianlib.TermGenerator()

    def __dealloc__(self):
        del self._this

    property description:
        def __get__(self):
            as_bytes = <char *>self._this.get_description().c_str()
            #return as_bytes
            return as_bytes.decode('utf-8')
-----snip snip-----

I'm sure I'm doing something naive or stupid, but the problem is that
as written above, .description is returning nonsense.

% python
Python 3.2.3 (default, May  3 2012, 15:51:42) 
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import xapian
>>> tg = xapian.TermGenerator()
>>> tg.description
'\x00\x00\x00\x00_\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'

If instead, I return just the bytes object (i.e. what
.get_description().c_str() returns), then I get more like what I expect.

% python
Python 3.2.3 (default, May  3 2012, 15:51:42) 
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import xapian
>>> tg = xapian.TermGenerator()
>>> tg.description
b'Xapian::TermGenerator(stem=Xapian::Stem(none), doc=Document(Xapian::Document::Internal()), termpos=0)'
>>> tg.description.decode('utf-8')
'Xapian::TermGenerator(stem=Xapian::Stem(none), doc=Document(Xapian::Document::Internal()), termpos=0)'

I looked at the generated code in the first example, but didn't really see
anything obvious.  There are no NULs in the char* description afaict.  I
haven't yet tested Cython 0.16 or 0.17 to see if this behaves differently.

Is this a bug or am I doing something stupid?

Cheers,
-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20120705/5e089012/attachment.pgp>

From d.s.seljebotn at astro.uio.no  Fri Jul  6 04:34:09 2012
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Fri, 06 Jul 2012 04:34:09 +0200
Subject: [Cython] [cython-users] C++: how to handle failures of 'new'?
In-Reply-To: <CANg26EWp-fqysQeE58hVDeWtWZJ9p+sPem8MOwaEs1tUdqcKZg@mail.gmail.com>
References: <4FECA49B.8090404@behnel.de>
	<CADiQ+QBeBw=FYsNPpSU_OJOU48m7E1=csxMxfenMQLs2jhJgPw@mail.gmail.com>
	<4FED40E1.1040301@behnel.de>
	<CADiQ+QAFYMgycY5-RkTf7SOwDb0-qM93Y8+sEzA=+1P9DSyOyA@mail.gmail.com>
	<4FF29BE3.6030201@behnel.de> <4FF31999.6090501@astro.uio.no>
	<4FF31FF8.10507@behnel.de>
	<CADiQ+QA8nRcpCUsmMx=zb50XQAnOXLVky5pxOxQu=DQxF+BMfw@mail.gmail.com>
	<4FF335D2.3070706@behnel.de>
	<CADiQ+QATCzuJ2vNXJm-QxueG9s_DBPyCBRGhZ1cnMpEMQvMtzQ@mail.gmail.com>
	<4FF33D60.5080804@astro.uio.no>
	<CADiQ+QD0etaANiRy=03=yTjCi60xCpN+GVV5=mueWuQnBotrDQ@mail.gmail.com>
	<CANg26EVTZC0mfxbcka_v8fE3try5YkLgc8x-VyvBh0kooT97iA@mail.gmail.com>
	<23685ab8-5945-418f-b3f6-0c7a92a03bea@email.android.com>
	<CANg26EWp-fqysQeE58hVDeWtWZJ9p+sPem8MOwaEs1tUdqcKZg@mail.gmail.com>
Message-ID: <84789ed7-3426-4d18-818e-e680bea9487b@email.android.com>



mark florisson <markflorisson88 at gmail.com> wrote:

>On 5 July 2012 21:46, Dag Sverre Seljebotn <d.s.seljebotn at astro.uio.no>
>wrote:
>>
>>
>> mark florisson <markflorisson88 at gmail.com> wrote:
>>
>>>On 3 July 2012 20:15, Robert Bradshaw <robertwb at gmail.com> wrote:
>>>> On Tue, Jul 3, 2012 at 11:43 AM, Dag Sverre Seljebotn
>>>> <d.s.seljebotn at astro.uio.no> wrote:
>>>>> On 07/03/2012 08:23 PM, Robert Bradshaw wrote:
>>>>>>
>>>>>> On Tue, Jul 3, 2012 at 11:11 AM, Stefan
>Behnel<stefan_ml at behnel.de>
>>>>>> wrote:
>>>>>>>
>>>>>>> Robert Bradshaw, 03.07.2012 19:58:
>>>>>>>>
>>>>>>>> On Tue, Jul 3, 2012 at 9:38 AM, Stefan Behnel wrote:
>>>>>>>>>
>>>>>>>>> Dag Sverre Seljebotn, 03.07.2012 18:11:
>>>>>>>>>>
>>>>>>>>>> On 07/03/2012 09:14 AM, Stefan Behnel wrote:
>>>>>>>>>>>
>>>>>>>>>>> I don't know what happens if a C++ exception is not being
>>>caught, but
>>>>>>>>>>> I
>>>>>>>>>>> guess it would simply crash the application. That's a bit
>more
>>>>>>>>>>> visible than
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Yep.
>>>>>>>>>>
>>>>>>>>>>> just printing a warning when a Python exception is being
>>>ignored due
>>>>>>>>>>> to a
>>>>>>>>>>> missing declaration. It's really unfortunate that our
>>>documentation
>>>>>>>>>>> didn't
>>>>>>>>>>> even mention the need for this, because it's not immediately
>>>obvious
>>>>>>>>>>> that
>>>>>>>>>>> Cython won't handle errors in "new", and testing for memory
>>>errors
>>>>>>>>>>> isn't
>>>>>>>>>>> quite what people commonly do in their test suites.
>>>>>>>>>>>
>>>>>>>>>>> Apart from that, I agree, users have to take care to
>properly
>>>declare
>>>>>>>>>>> the
>>>>>>>>>>> API they are using.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Is there any time you do NOT want a "catch (...) {}" block? I
>>>can't
>>>>>>>>>> see a
>>>>>>>>>> C++ exception propagating to Python-land doing anything
>useful
>>>ever.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> That would have been my intuition, too.
>>>>>>>>
>>>>>>>>
>>>>>>>> If it's actually embedded, with the main driver in C++, one
>might
>>>want
>>>>>>>> it to propagate up.
>>>>>>>
>>>>>>>
>>>>>>> But what kind of a propagation would that be? On the way out, it
>>>could
>>>>>>> induce anything, from side effects to resource leaks to crashes,
>>>>>>> depending
>>>>>>> on what the state of the surrounding code is. It would leave the
>>>whole
>>>>>>> system in an unpredictable state. I cannot imagine anyone really
>>>wanting
>>>>>>> this.
>>>>>>>
>>>>>>>
>>>>>>>>>> So shouldn't we just make --cplus turn *all* external
>functions
>>>and
>>>>>>>>>> methods
>>>>>>>>>> (whether C-like or C++-like) into "except +"? (Or keep
>except+
>>>for
>>>>>>>>>> manual
>>>>>>>>>> translation, but always have a catch(...)".
>>>>>>>>>>
>>>>>>>>>> Performance overhead is the only reason I can think of to not
>>>do this,
>>>>>>>>>> although IIRC C++ catch blocks are only dealt with during
>stack
>>>>>>>>>> unwinds and
>>>>>>>>>> doesn't cost anything/much (?) when they're not triggered.
>>>>>>>>>>
>>>>>>>>>> "except -1" should then actually mean both; "except + except
>>>-1". So
>>>>>>>>>> it's
>>>>>>>>>> more a question of just adding catch(...) *everywhere*, than
>>>making
>>>>>>>>>> "except
>>>>>>>>>> +" the default.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I have no idea if there is a performance impact, but if there
>>>isn't,
>>>>>>>>> always
>>>>>>>>> catching all exceptions sounds like a reasonable thing to do.
>>>After
>>>>>>>>> all, we
>>>>>>>>> have no support for catching C++ exceptions on user side.
>>>>>>>>
>>>>>>>>
>>>>>>>> This is a bit like following every C call with "except *"
>(though
>>>the
>>>>>>>> performance ratios are unclear). It just seems a lot to wrap
>>>every
>>>>>>>> single line of a non-trivial C++ using function with try..catch
>>>>>>>> blocks.
>>>>>
>>>>>
>>>>> It seems "a lot" of just what exactly? Generated code? Binary
>size?
>>>Time
>>>>> spent in GCC parser?
>>>>
>>>> All of the above. And we should take a look at the runtime overhead
>>>> (which is hopefully nil, but who knows.)
>>>>
>>>>> Though I guess one might want to try to pull out the try-catch to
>at
>>>least
>>>>> only one per code line rather than one per SimpleCallNode.
>>>>
>>>> Or even higher, if possible. It's still a lot.
>>>
>>>Why would you have to do that? Can't you just insert a try/catch per
>>>try/except or try/finally block, or if absent, the function body.
>That
>>>will still work with the way temporaries are cleaned up. (It should
>>>also be implemented for parallel/prange sections).
>>
>> One disadvantage is that you don't get source code line for the .pyx
>file in the stack trace. Which is often exactly the information you are
>looking for (even worse, since C++ stack isn't in the stack trace, the
>lineno for what seems like the ' ultimate cause' is not there). Having
>to surround statements with try/except just to pinpoint which one is
>raising the exception would be incredibly irritating.
>>
>> Dag
>
>Oh yeah, good point. Maybe we could use these zero-cost exceptions for
>cdef functions in Cython though, instead of error checks (if it
>appears to make any significant difference). Basically instead of the
>'error' argument in CEP 526. It'd need version that ABI as well...
>

Wait, one can just do

lineno = 25;
a()
linono = 27;
b()

And wrap the block with a try catch.

I'm not sure if I like C++ exceptions internally in Cython myself. It would mean that C-compiled Cython modules would not be able to call C++-compiled Cython modules, which I think would create a lot of headache. And I think we want to keep the ability to compile with a C compiler if you don't call C++ code...

Dag

>>>
>>>>> "except *" only has a point when calling functions using the
>CPython
>>>API,
>>>>> but most external C functions are pure C, not
>>>CPython-API-using-functions.
>>>>> OTOH, all external C++ functions are C++ :-)
>>>>
>>>> Fair point.
>>>>
>>>>> (Also, if we wrote Cython from scratch now I'm pretty sure the
>>>"except *"
>>>>> defaults would be a tad different.)
>>>>
>>>> For sure.
>>>>
>>>>>>> But if users are correct about their declarations, we'd end up
>>>with the
>>>>>>> same thing. I think it's worth a try.
>>>>>>
>>>>>>
>>>>>> Most C++ code (that I've ever run into) doesn't use exceptions,
>>>>>> because exception handling is so broken in C++ anyways.
>>>>>
>>>>>
>>>>> Except for the fact that any code touching "new" could be raising
>>>>> exceptions? That propagates.
>>>>
>>>> I would guess most of the time people don't bother catching these
>and
>>>> let the program die, as there's often no sane recovery (the same as
>>>> MemoryErrors in Python, though I guess C++ is less often used from
>an
>>>> event loop).
>>>>
>>>>> There is a lot of C++ code out there using exceptions. I'd guess
>>>that both
>>>>> mathematical code and Google-written code is unlike most C++ code
>>>out there
>>>>> :-) Many C++ programmers go on and on about RAII and auto_ptrs and
>>>so on,
>>>>> and that doesn't have much point unless you throw an exception now
>>>and then
>>>>> (OK, there's the occasional return statement where it matters
>well).
>>>>
>>>> True, I've seen a small subset of the C++ code that's out there.
>>>Maybe
>>>> numerical computations use it a lot?
>>>>
>>>> +1 to making catch-everywhere a directive at least. Lets see what
>the
>>>> impact is before we decide to make it the default.
>>>>
>>>> - Robert
>>>> _______________________________________________
>>>> cython-devel mailing list
>>>> cython-devel at python.org
>>>> http://mail.python.org/mailman/listinfo/cython-devel
>>>_______________________________________________
>>>cython-devel mailing list
>>>cython-devel at python.org
>>>http://mail.python.org/mailman/listinfo/cython-devel
>>
>> --
>> Sent from my Android phone with K-9 Mail. Please excuse my brevity.
>> _______________________________________________
>> cython-devel mailing list
>> cython-devel at python.org
>> http://mail.python.org/mailman/listinfo/cython-devel
>_______________________________________________
>cython-devel mailing list
>cython-devel at python.org
>http://mail.python.org/mailman/listinfo/cython-devel

-- 
Sent from my Android phone with K-9 Mail. Please excuse my brevity.

From stefan_ml at behnel.de  Fri Jul  6 06:48:37 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 06 Jul 2012 06:48:37 +0200
Subject: [Cython] Odd behavior with std::string and .decode()
In-Reply-To: <20120705182919.5d526c85@resist.wooz.org>
References: <20120705182919.5d526c85@resist.wooz.org>
Message-ID: <4FF66E25.3070808@behnel.de>

Hi Barry,

Barry Warsaw, 06.07.2012 00:29:
> I'm currently exploring using Cython to provide new Python 3 bindings for
> Xapian.  I'm pretty much a Cython n00b but the documentation is great, and I
> was able to pretty quickly get something really simple working.  I'm using
> Cython 0.15 on Ubuntu 12.04 with Python 3.2 and Xapian 1.2.12.  I've pushed my
> current branch to github:
> 
> https://github.com/warsaw/xapian/tree/py3/xapian-bindings/python3
> 
> There you'll see my xapianlib.pxd and xapian.pyx files.
> 
> Where I'm seeing some odd behavior is in trying to expose the
> Xapian::TermGenerator.get_description() method.  This returns a std::string
> and I'm trying to create a `description` property that coerces this to unicode
> before returning it to Python.  Here's the relevant code:
> 
> -----snip snip-----
> cdef class TermGenerator:
>     cdef xapianlib.TermGenerator * _this
> 
>     def __cinit__(self):
>         self._this = new xapianlib.TermGenerator()
> 
>     def __dealloc__(self):
>         del self._this
> 
>     property description:
>         def __get__(self):
>             as_bytes = <char *>self._this.get_description().c_str()
>             #return as_bytes
>             return as_bytes.decode('utf-8')
> -----snip snip-----
> 
> I'm sure I'm doing something naive or stupid, but the problem is that
> as written above, .description is returning nonsense.
> 
> % python
> Python 3.2.3 (default, May  3 2012, 15:51:42) 
> [GCC 4.6.3] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
> >>> import xapian
> >>> tg = xapian.TermGenerator()
> >>> tg.description
> '\x00\x00\x00\x00_\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'
> 
> If instead, I return just the bytes object (i.e. what
> .get_description().c_str() returns), then I get more like what I expect.
> 
> % python
> Python 3.2.3 (default, May  3 2012, 15:51:42) 
> [GCC 4.6.3] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
> >>> import xapian
> >>> tg = xapian.TermGenerator()
> >>> tg.description
> b'Xapian::TermGenerator(stem=Xapian::Stem(none), doc=Document(Xapian::Document::Internal()), termpos=0)'
> >>> tg.description.decode('utf-8')
> 'Xapian::TermGenerator(stem=Xapian::Stem(none), doc=Document(Xapian::Document::Internal()), termpos=0)'

This is very weird behaviour indeed. I wouldn't know why that should
happen. What "return as_bytes.decode('utf-8')" does is that is calls
strlen() to see how long the string is, then it calls the UTF-8 decode
C-API function with that.

The string that get_description() returns is allocated internally in the
C++ object, right? So it can't suddenly die or something?

One thing I would generally suggest is to do this:

    descr = self._this.get_description()
    return descr.data()[:descr.size()].decode('utf-8')

Avoids the call to strlen() by explicitly slicing the pointer. Also avoids
needing to make sure the C string is 0-terminated.


> I looked at the generated code in the first example, but didn't really see
> anything obvious.  There are no NULs in the char* description afaict.  I
> haven't yet tested Cython 0.16 or 0.17 to see if this behaves differently.

I wouldn't know any differences out of the top of my head, except that 0.17
has generally better support for STL containers and std:string (but that's
unrelated to this failure). I'm planning to enable direct support for
cpp_string.decode(...) as well, but that's not implemented yet. It would
basically generate the verbose code above automatically.


> Is this a bug or am I doing something stupid?

Definitely not doing something stupid, but I have no idea why this should
go wrong.

Stefan

From markflorisson88 at gmail.com  Fri Jul  6 11:33:00 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Fri, 6 Jul 2012 10:33:00 +0100
Subject: [Cython] [cython-users] C++: how to handle failures of 'new'?
In-Reply-To: <84789ed7-3426-4d18-818e-e680bea9487b@email.android.com>
References: <4FECA49B.8090404@behnel.de>
	<CADiQ+QBeBw=FYsNPpSU_OJOU48m7E1=csxMxfenMQLs2jhJgPw@mail.gmail.com>
	<4FED40E1.1040301@behnel.de>
	<CADiQ+QAFYMgycY5-RkTf7SOwDb0-qM93Y8+sEzA=+1P9DSyOyA@mail.gmail.com>
	<4FF29BE3.6030201@behnel.de> <4FF31999.6090501@astro.uio.no>
	<4FF31FF8.10507@behnel.de>
	<CADiQ+QA8nRcpCUsmMx=zb50XQAnOXLVky5pxOxQu=DQxF+BMfw@mail.gmail.com>
	<4FF335D2.3070706@behnel.de>
	<CADiQ+QATCzuJ2vNXJm-QxueG9s_DBPyCBRGhZ1cnMpEMQvMtzQ@mail.gmail.com>
	<4FF33D60.5080804@astro.uio.no>
	<CADiQ+QD0etaANiRy=03=yTjCi60xCpN+GVV5=mueWuQnBotrDQ@mail.gmail.com>
	<CANg26EVTZC0mfxbcka_v8fE3try5YkLgc8x-VyvBh0kooT97iA@mail.gmail.com>
	<23685ab8-5945-418f-b3f6-0c7a92a03bea@email.android.com>
	<CANg26EWp-fqysQeE58hVDeWtWZJ9p+sPem8MOwaEs1tUdqcKZg@mail.gmail.com>
	<84789ed7-3426-4d18-818e-e680bea9487b@email.android.com>
Message-ID: <CANg26EUTjNtQz6ZX=Ve_cq9r3A3AEUJfKBPv4Yc2T-M5yVGoqA@mail.gmail.com>

On 6 July 2012 03:34, Dag Sverre Seljebotn <d.s.seljebotn at astro.uio.no> wrote:
>
>
> mark florisson <markflorisson88 at gmail.com> wrote:
>
>>On 5 July 2012 21:46, Dag Sverre Seljebotn <d.s.seljebotn at astro.uio.no>
>>wrote:
>>>
>>>
>>> mark florisson <markflorisson88 at gmail.com> wrote:
>>>
>>>>On 3 July 2012 20:15, Robert Bradshaw <robertwb at gmail.com> wrote:
>>>>> On Tue, Jul 3, 2012 at 11:43 AM, Dag Sverre Seljebotn
>>>>> <d.s.seljebotn at astro.uio.no> wrote:
>>>>>> On 07/03/2012 08:23 PM, Robert Bradshaw wrote:
>>>>>>>
>>>>>>> On Tue, Jul 3, 2012 at 11:11 AM, Stefan
>>Behnel<stefan_ml at behnel.de>
>>>>>>> wrote:
>>>>>>>>
>>>>>>>> Robert Bradshaw, 03.07.2012 19:58:
>>>>>>>>>
>>>>>>>>> On Tue, Jul 3, 2012 at 9:38 AM, Stefan Behnel wrote:
>>>>>>>>>>
>>>>>>>>>> Dag Sverre Seljebotn, 03.07.2012 18:11:
>>>>>>>>>>>
>>>>>>>>>>> On 07/03/2012 09:14 AM, Stefan Behnel wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> I don't know what happens if a C++ exception is not being
>>>>caught, but
>>>>>>>>>>>> I
>>>>>>>>>>>> guess it would simply crash the application. That's a bit
>>more
>>>>>>>>>>>> visible than
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Yep.
>>>>>>>>>>>
>>>>>>>>>>>> just printing a warning when a Python exception is being
>>>>ignored due
>>>>>>>>>>>> to a
>>>>>>>>>>>> missing declaration. It's really unfortunate that our
>>>>documentation
>>>>>>>>>>>> didn't
>>>>>>>>>>>> even mention the need for this, because it's not immediately
>>>>obvious
>>>>>>>>>>>> that
>>>>>>>>>>>> Cython won't handle errors in "new", and testing for memory
>>>>errors
>>>>>>>>>>>> isn't
>>>>>>>>>>>> quite what people commonly do in their test suites.
>>>>>>>>>>>>
>>>>>>>>>>>> Apart from that, I agree, users have to take care to
>>properly
>>>>declare
>>>>>>>>>>>> the
>>>>>>>>>>>> API they are using.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Is there any time you do NOT want a "catch (...) {}" block? I
>>>>can't
>>>>>>>>>>> see a
>>>>>>>>>>> C++ exception propagating to Python-land doing anything
>>useful
>>>>ever.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> That would have been my intuition, too.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> If it's actually embedded, with the main driver in C++, one
>>might
>>>>want
>>>>>>>>> it to propagate up.
>>>>>>>>
>>>>>>>>
>>>>>>>> But what kind of a propagation would that be? On the way out, it
>>>>could
>>>>>>>> induce anything, from side effects to resource leaks to crashes,
>>>>>>>> depending
>>>>>>>> on what the state of the surrounding code is. It would leave the
>>>>whole
>>>>>>>> system in an unpredictable state. I cannot imagine anyone really
>>>>wanting
>>>>>>>> this.
>>>>>>>>
>>>>>>>>
>>>>>>>>>>> So shouldn't we just make --cplus turn *all* external
>>functions
>>>>and
>>>>>>>>>>> methods
>>>>>>>>>>> (whether C-like or C++-like) into "except +"? (Or keep
>>except+
>>>>for
>>>>>>>>>>> manual
>>>>>>>>>>> translation, but always have a catch(...)".
>>>>>>>>>>>
>>>>>>>>>>> Performance overhead is the only reason I can think of to not
>>>>do this,
>>>>>>>>>>> although IIRC C++ catch blocks are only dealt with during
>>stack
>>>>>>>>>>> unwinds and
>>>>>>>>>>> doesn't cost anything/much (?) when they're not triggered.
>>>>>>>>>>>
>>>>>>>>>>> "except -1" should then actually mean both; "except + except
>>>>-1". So
>>>>>>>>>>> it's
>>>>>>>>>>> more a question of just adding catch(...) *everywhere*, than
>>>>making
>>>>>>>>>>> "except
>>>>>>>>>>> +" the default.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I have no idea if there is a performance impact, but if there
>>>>isn't,
>>>>>>>>>> always
>>>>>>>>>> catching all exceptions sounds like a reasonable thing to do.
>>>>After
>>>>>>>>>> all, we
>>>>>>>>>> have no support for catching C++ exceptions on user side.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> This is a bit like following every C call with "except *"
>>(though
>>>>the
>>>>>>>>> performance ratios are unclear). It just seems a lot to wrap
>>>>every
>>>>>>>>> single line of a non-trivial C++ using function with try..catch
>>>>>>>>> blocks.
>>>>>>
>>>>>>
>>>>>> It seems "a lot" of just what exactly? Generated code? Binary
>>size?
>>>>Time
>>>>>> spent in GCC parser?
>>>>>
>>>>> All of the above. And we should take a look at the runtime overhead
>>>>> (which is hopefully nil, but who knows.)
>>>>>
>>>>>> Though I guess one might want to try to pull out the try-catch to
>>at
>>>>least
>>>>>> only one per code line rather than one per SimpleCallNode.
>>>>>
>>>>> Or even higher, if possible. It's still a lot.
>>>>
>>>>Why would you have to do that? Can't you just insert a try/catch per
>>>>try/except or try/finally block, or if absent, the function body.
>>That
>>>>will still work with the way temporaries are cleaned up. (It should
>>>>also be implemented for parallel/prange sections).
>>>
>>> One disadvantage is that you don't get source code line for the .pyx
>>file in the stack trace. Which is often exactly the information you are
>>looking for (even worse, since C++ stack isn't in the stack trace, the
>>lineno for what seems like the ' ultimate cause' is not there). Having
>>to surround statements with try/except just to pinpoint which one is
>>raising the exception would be incredibly irritating.
>>>
>>> Dag
>>
>>Oh yeah, good point. Maybe we could use these zero-cost exceptions for
>>cdef functions in Cython though, instead of error checks (if it
>>appears to make any significant difference). Basically instead of the
>>'error' argument in CEP 526. It'd need version that ABI as well...
>>
>
> Wait, one can just do
>
> lineno = 25;
> a()
> linono = 27;
> b()
>
> And wrap the block with a try catch.

Heh, I just had the same idea before going to bed yesterday :) But it
means the common case (no exception) will get some overhead. That can
probably be afforded since normal error checking branches anyway.
Maybe it's worth investigating how many instructions try/catch blocks
generate and seeing whether a) it's (significantly) better than
branching and b) whether a try/catch block can be afforded per
subexpression/lineno.

> I'm not sure if I like C++ exceptions internally in Cython myself. It would mean that C-compiled Cython modules would not be able to call C++-compiled Cython modules, which I think would create a lot of headache. And I think we want to keep the ability to compile with a C compiler if you don't call C++ code...

I don't know how often people mix the two. It's probably not worth
making a directive either... Are there any platforms where installing
a C++ compiler is more of a hassle than a C compiler? E.g. Theano
depends on a C++ compiler (and a lot more as well).

The cool thing about it would be that as you said earlier in the
thread, you could at the very least do one try/catch per line number,
so you wouldn't need N branches for N subexpressions that can fail.

> Dag
>
>>>>
>>>>>> "except *" only has a point when calling functions using the
>>CPython
>>>>API,
>>>>>> but most external C functions are pure C, not
>>>>CPython-API-using-functions.
>>>>>> OTOH, all external C++ functions are C++ :-)
>>>>>
>>>>> Fair point.
>>>>>
>>>>>> (Also, if we wrote Cython from scratch now I'm pretty sure the
>>>>"except *"
>>>>>> defaults would be a tad different.)
>>>>>
>>>>> For sure.
>>>>>
>>>>>>>> But if users are correct about their declarations, we'd end up
>>>>with the
>>>>>>>> same thing. I think it's worth a try.
>>>>>>>
>>>>>>>
>>>>>>> Most C++ code (that I've ever run into) doesn't use exceptions,
>>>>>>> because exception handling is so broken in C++ anyways.
>>>>>>
>>>>>>
>>>>>> Except for the fact that any code touching "new" could be raising
>>>>>> exceptions? That propagates.
>>>>>
>>>>> I would guess most of the time people don't bother catching these
>>and
>>>>> let the program die, as there's often no sane recovery (the same as
>>>>> MemoryErrors in Python, though I guess C++ is less often used from
>>an
>>>>> event loop).
>>>>>
>>>>>> There is a lot of C++ code out there using exceptions. I'd guess
>>>>that both
>>>>>> mathematical code and Google-written code is unlike most C++ code
>>>>out there
>>>>>> :-) Many C++ programmers go on and on about RAII and auto_ptrs and
>>>>so on,
>>>>>> and that doesn't have much point unless you throw an exception now
>>>>and then
>>>>>> (OK, there's the occasional return statement where it matters
>>well).
>>>>>
>>>>> True, I've seen a small subset of the C++ code that's out there.
>>>>Maybe
>>>>> numerical computations use it a lot?
>>>>>
>>>>> +1 to making catch-everywhere a directive at least. Lets see what
>>the
>>>>> impact is before we decide to make it the default.
>>>>>
>>>>> - Robert
>>>>> _______________________________________________
>>>>> cython-devel mailing list
>>>>> cython-devel at python.org
>>>>> http://mail.python.org/mailman/listinfo/cython-devel
>>>>_______________________________________________
>>>>cython-devel mailing list
>>>>cython-devel at python.org
>>>>http://mail.python.org/mailman/listinfo/cython-devel
>>>
>>> --
>>> Sent from my Android phone with K-9 Mail. Please excuse my brevity.
>>> _______________________________________________
>>> cython-devel mailing list
>>> cython-devel at python.org
>>> http://mail.python.org/mailman/listinfo/cython-devel
>>_______________________________________________
>>cython-devel mailing list
>>cython-devel at python.org
>>http://mail.python.org/mailman/listinfo/cython-devel
>
> --
> Sent from my Android phone with K-9 Mail. Please excuse my brevity.
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

From barry at python.org  Fri Jul  6 16:21:54 2012
From: barry at python.org (Barry Warsaw)
Date: Fri, 6 Jul 2012 10:21:54 -0400
Subject: [Cython] Odd behavior with std::string and .decode()
References: <20120705182919.5d526c85@resist.wooz.org>
	<4FF66E25.3070808@behnel.de>
Message-ID: <20120706102154.0d289e53@resist.wooz.org>

Thanks for the follow up Stefan,

On Jul 06, 2012, at 06:48 AM, Stefan Behnel wrote:

>This is very weird behaviour indeed. I wouldn't know why that should
>happen. What "return as_bytes.decode('utf-8')" does is that is calls
>strlen() to see how long the string is, then it calls the UTF-8 decode
>C-API function with that.

It seems like either the strlen() or the cast through char* is the problem.

>The string that get_description() returns is allocated internally in the
>C++ object, right? So it can't suddenly die or something?

I don't think so.

>One thing I would generally suggest is to do this:
>
>    descr = self._this.get_description()
>    return descr.data()[:descr.size()].decode('utf-8')
>
>Avoids the call to strlen() by explicitly slicing the pointer. Also avoids
>needing to make sure the C string is 0-terminated.

According to

http://www.cplusplus.com/reference/string/string/c_str/

    The returned array points to an internal location with the required
    storage space for this sequence of characters plus its terminating
    null-character, but the values in this array should not be modified in the
    program and are only guaranteed to remain unchanged until the next call to
    a non-constant member function of the string object.

I believe the const char* returned by c_str() is guaranteed to be null
terminated.  AFAICT, there are no embedded NULs.  I also don't think there are
any non-constant member function calls of the parent string object getting in
the way.

Next, I tried two different implementations:

    property description:
        def __get__(self):
            # works
            descr = self._this.get_description()
            return descr.c_str()[:descr.size()].decode('utf-8')

    property destruction:
        def __get__(self):
            # broken
            as_bytes = <char *>self._this.get_description().c_str()
            return as_bytes.decode('utf-8')

The second case requires the cast or you get an error:

xapian.cpp:1409:67: error: invalid conversion from ?const char*? to ?char*? [-fpermissive]

but I don't think that's the problem.  Looking at the generated C++ code, I
see these two different implementations:

works:

  __pyx_t_1 = ((PyObject *)PyUnicode_Decode(__pyx_v_descr.c_str(), __pyx_v_descr.size(), __pyx_k_1, NULL)); if (unlikely(!__pyx_t_1)) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 84; __pyx_clineno = __LINE__; goto __pyx_L1_error;}

broken:

  __pyx_t_1 = ((PyObject *)PyUnicode_Decode(__pyx_v_as_bytes, strlen(__pyx_v_as_bytes), __pyx_k_1, NULL)); if (unlikely(!__pyx_t_1)) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 91; __pyx_clineno = __LINE__; goto __pyx_L1_error;}

In the working case, __pyx_v_descr is a std::string, so the const char*
returned by .c_str() is passed directly to PyUnicode_Decode() without a cast.
The length is returned by std::string.size().

In the broken case, __pyx_v_as_bytes is a char* (I could not figure out how to
preserve the const char* type) and strlen() is used to find the length.

Those are the only substantive differences I could find.

>I wouldn't know any differences out of the top of my head, except that 0.17
>has generally better support for STL containers and std:string (but that's
>unrelated to this failure). I'm planning to enable direct support for
>cpp_string.decode(...) as well, but that's not implemented yet. It would
>basically generate the verbose code above automatically.
>
>> Is this a bug or am I doing something stupid?
>
>Definitely not doing something stupid, but I have no idea why this should
>go wrong.

Okay, at least I have a few workarounds :).  I'd file a bug but I don't have
permission to file new issues.

If you have any other suggestions for ways to debug this, I'm happy to give
them a try.

Cheers,
-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20120706/46390600/attachment.pgp>

From barry at python.org  Fri Jul  6 19:14:05 2012
From: barry at python.org (Barry Warsaw)
Date: Fri, 6 Jul 2012 13:14:05 -0400
Subject: [Cython] Higher fidelity translations from C++ exception to Python
 exception?
Message-ID: <20120706131405.6915a5f8@resist.wooz.org>

So if you have code like the following:

cdef class Database:
    cdef open(self, path) except +raise_py_error:
        something_that_can_throw_a_cpp_exception(path)

you can write

cdef int raise_py_error():
    raise Something

to kind of turn a C++ exception into a Python exception.  The problem appears
to be that you cannot include in the Python exception any information
contained in the C++ exception, because there's no access to the C++ exception
that was thrown.

The generated code does a `catch(...)` so you lose that useful information.
AFAICT, there's no way to find out within the catch clause (or anything called
by that clause) what C++ exception occurred.

It would sure be useful if raise_py_error() was passed the exception instance
that was caught.  Then at least our handler could extract some useful
information to relay to the Python level.  E.g. in the case above, it might
tell us that the `path` is nonexistent.

Cheers,
-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20120706/51b3066c/attachment.pgp>

From d.s.seljebotn at astro.uio.no  Fri Jul  6 23:59:51 2012
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Fri, 06 Jul 2012 23:59:51 +0200
Subject: [Cython] Higher fidelity translations from C++ exception to
	Python exception?
In-Reply-To: <20120706131405.6915a5f8@resist.wooz.org>
References: <20120706131405.6915a5f8@resist.wooz.org>
Message-ID: <62a30799-6aa6-4e6e-aa24-6db5087787da@email.android.com>



Barry Warsaw <barry at python.org> wrote:

>So if you have code like the following:
>
>cdef class Database:
>    cdef open(self, path) except +raise_py_error:
>        something_that_can_throw_a_cpp_exception(path)
>
>you can write
>
>cdef int raise_py_error():
>    raise Something
>
>to kind of turn a C++ exception into a Python exception.  The problem
>appears
>to be that you cannot include in the Python exception any information
>contained in the C++ exception, because there's no access to the C++
>exception
>that was thrown.
>
>The generated code does a `catch(...)` so you lose that useful
>information.
>AFAICT, there's no way to find out within the catch clause (or anything
>called
>by that clause) what C++ exception occurred.
>
>It would sure be useful if raise_py_error() was passed the exception
>instance
>that was caught.  Then at least our handler could extract some useful
>information to relay to the Python level.  E.g. in the case above, it
>might
>tell us that the `path` is nonexistent.

But that requires something more than a catch(...), right? There must be more syntax introduced in the Cython language to map C++ exceptions to constructor functions (taking exceptions of different types). Do you have a concrete proposal for how that could look like?

Dag 

>
>Cheers,
>-Barry
>_______________________________________________
>cython-devel mailing list
>cython-devel at python.org
>http://mail.python.org/mailman/listinfo/cython-devel

-- 
Sent from my Android phone with K-9 Mail. Please excuse my brevity.

From barry at python.org  Sat Jul  7 00:05:09 2012
From: barry at python.org (Barry Warsaw)
Date: Fri, 6 Jul 2012 18:05:09 -0400
Subject: [Cython] Higher fidelity translations from C++ exception to
 Python exception?
References: <20120706131405.6915a5f8@resist.wooz.org>
	<62a30799-6aa6-4e6e-aa24-6db5087787da@email.android.com>
Message-ID: <20120706180509.7ea3f351@resist.wooz.org>

On Jul 06, 2012, at 11:59 PM, Dag Sverre Seljebotn wrote:

>>The generated code does a `catch(...)` so you lose that useful information.
>>AFAICT, there's no way to find out within the catch clause (or anything
>>called by that clause) what C++ exception occurred.
>
>But that requires something more than a catch(...), right? There must be more
>syntax introduced in the Cython language to map C++ exceptions to constructor
>functions (taking exceptions of different types). Do you have a concrete
>proposal for how that could look like?

What about:

cdef void raise_fooexc(e) except *:
    raise MyPythonError(e.detail_1, e.detail_2)


cdef func(self) except +raise_fooexc(exception_type):
    something_that_raises_exception_type()

?

Then `exception_type` would get stuffed inside the catch() instead of ...

Cheers,
-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20120706/967f1ebd/attachment.pgp>

From stefan_ml at behnel.de  Sat Jul  7 01:18:11 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 07 Jul 2012 01:18:11 +0200
Subject: [Cython] Higher fidelity translations from C++ exception to
 Python exception?
In-Reply-To: <20120706131405.6915a5f8@resist.wooz.org>
References: <20120706131405.6915a5f8@resist.wooz.org>
Message-ID: <4FF77233.9000305@behnel.de>

Barry Warsaw, 06.07.2012 19:14:
> So if you have code like the following:
> 
> cdef class Database:
>     cdef open(self, path) except +raise_py_error:
>         something_that_can_throw_a_cpp_exception(path)
> 
> you can write
> 
> cdef int raise_py_error():
>     raise Something
> 
> to kind of turn a C++ exception into a Python exception.  The problem appears
> to be that you cannot include in the Python exception any information
> contained in the C++ exception, because there's no access to the C++ exception
> that was thrown.
> 
> The generated code does a `catch(...)` so you lose that useful information.
> AFAICT, there's no way to find out within the catch clause (or anything called
> by that clause) what C++ exception occurred.
> 
> It would sure be useful if raise_py_error() was passed the exception instance
> that was caught.  Then at least our handler could extract some useful
> information to relay to the Python level.  E.g. in the case above, it might
> tell us that the `path` is nonexistent.

What's your use case? Are you trying to raise your own custom exceptions here?

As a work-around, you could use "except +" (without a function) and then
catch and handle (or wrap) the resulting Python exceptions. Admittedly not
exactly beautiful ...

Stefan

From stefan_ml at behnel.de  Sat Jul  7 01:54:55 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 07 Jul 2012 01:54:55 +0200
Subject: [Cython] Odd behavior with std::string and .decode()
In-Reply-To: <20120706102154.0d289e53@resist.wooz.org>
References: <20120705182919.5d526c85@resist.wooz.org>
	<4FF66E25.3070808@behnel.de>
	<20120706102154.0d289e53@resist.wooz.org>
Message-ID: <4FF77ACF.6000908@behnel.de>

Barry Warsaw, 06.07.2012 16:21:
> Thanks for the follow up Stefan,
> 
> On Jul 06, 2012, at 06:48 AM, Stefan Behnel wrote:
> 
>> This is very weird behaviour indeed. I wouldn't know why that should
>> happen. What "return as_bytes.decode('utf-8')" does is that is calls
>> strlen() to see how long the string is, then it calls the UTF-8 decode
>> C-API function with that.
> 
> It seems like either the strlen() or the cast through char* is the problem.

Could you try it without the cast?

https://sage.math.washington.edu:8091/hudson/job/cython-docs/doclinks/1/src/tutorial/strings.html#dealing-with-const

In older Cython versions, you can use the declarations directly:

https://github.com/cython/cython/blob/master/Cython/Includes/libc/string.pxd#L3

I just noticed that .c_str() is incorrectly declared (without "const") in
Cython and that .data() is missing completely. I've pushed a fix for that
(note that the current master looks a bit broken, which is rather
unfortunate for testing).


>> One thing I would generally suggest is to do this:
>>
>>    descr = self._this.get_description()
>>    return descr.data()[:descr.size()].decode('utf-8')
>>
>> Avoids the call to strlen() by explicitly slicing the pointer. Also avoids
>> needing to make sure the C string is 0-terminated.
> 
> According to
> 
> http://www.cplusplus.com/reference/string/string/c_str/
> 
>     The returned array points to an internal location with the required
>     storage space for this sequence of characters plus its terminating
>     null-character, but the values in this array should not be modified in the
>     program and are only guaranteed to remain unchanged until the next call to
>     a non-constant member function of the string object.
> 
> I believe the const char* returned by c_str() is guaranteed to be null
> terminated.  AFAICT, there are no embedded NULs.  I also don't think there are
> any non-constant member function calls of the parent string object getting in
> the way.

Yes to all of the above.

What I meant was that .c_str() may be less efficient than .data() because
the internal string buffer may not be 0-terminated originally.


> Next, I tried two different implementations:
> 
>     property description:
>         def __get__(self):
>             # works
>             descr = self._this.get_description()
>             return descr.c_str()[:descr.size()].decode('utf-8')
> 
>     property destruction:
>         def __get__(self):
>             # broken
>             as_bytes = <char *>self._this.get_description().c_str()
>             return as_bytes.decode('utf-8')
> 
> The second case requires the cast or you get an error:
> 
> xapian.cpp:1409:67: error: invalid conversion from ?const char*? to ?char*? [-fpermissive]
> 
> but I don't think that's the problem.  Looking at the generated C++ code, I
> see these two different implementations:
> 
> works:
> 
>   __pyx_t_1 = ((PyObject *)PyUnicode_Decode(__pyx_v_descr.c_str(), __pyx_v_descr.size(), __pyx_k_1, NULL)); if (unlikely(!__pyx_t_1)) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 84; __pyx_clineno = __LINE__; goto __pyx_L1_error;}
> 
> broken:
> 
>   __pyx_t_1 = ((PyObject *)PyUnicode_Decode(__pyx_v_as_bytes, strlen(__pyx_v_as_bytes), __pyx_k_1, NULL)); if (unlikely(!__pyx_t_1)) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 91; __pyx_clineno = __LINE__; goto __pyx_L1_error;}
> 
> In the working case, __pyx_v_descr is a std::string, so the const char*
> returned by .c_str() is passed directly to PyUnicode_Decode() without a cast.
> The length is returned by std::string.size().
> 
> In the broken case, __pyx_v_as_bytes is a char* (I could not figure out how to
> preserve the const char* type) and strlen() is used to find the length.
> 
> Those are the only substantive differences I could find.

Maybe the C++ compiler is going mad because of the cast that kills "const"?


>> I wouldn't know any differences out of the top of my head, except that 0.17
>> has generally better support for STL containers and std:string (but that's
>> unrelated to this failure). I'm planning to enable direct support for
>> cpp_string.decode(...) as well, but that's not implemented yet. It would
>> basically generate the verbose code above automatically.
>>
>>> Is this a bug or am I doing something stupid?
>>
>> Definitely not doing something stupid, but I have no idea why this should
>> go wrong.
> 
> Okay, at least I have a few workarounds :).  I'd file a bug but I don't have
> permission to file new issues.

Please send a htpasswd entry to me or Robert.


> If you have any other suggestions for ways to debug this, I'm happy to give
> them a try.

Could you try to reproduce this without needing the Xapian library? It
would be good to have a (failing) test case.

Stefan

From robertwb at gmail.com  Sat Jul  7 07:24:58 2012
From: robertwb at gmail.com (Robert Bradshaw)
Date: Fri, 6 Jul 2012 22:24:58 -0700
Subject: [Cython] Higher fidelity translations from C++ exception to
 Python exception?
In-Reply-To: <20120706180509.7ea3f351@resist.wooz.org>
References: <20120706131405.6915a5f8@resist.wooz.org>
	<62a30799-6aa6-4e6e-aa24-6db5087787da@email.android.com>
	<20120706180509.7ea3f351@resist.wooz.org>
Message-ID: <CADiQ+QCucHSy65ucf35fgYgHt=7_AqgLoozaq1hnmgG6YUs9cg@mail.gmail.com>

On Fri, Jul 6, 2012 at 3:05 PM, Barry Warsaw <barry at python.org> wrote:
> On Jul 06, 2012, at 11:59 PM, Dag Sverre Seljebotn wrote:
>
>>>The generated code does a `catch(...)` so you lose that useful information.
>>>AFAICT, there's no way to find out within the catch clause (or anything
>>>called by that clause) what C++ exception occurred.
>>
>>But that requires something more than a catch(...), right? There must be more
>>syntax introduced in the Cython language to map C++ exceptions to constructor
>>functions (taking exceptions of different types). Do you have a concrete
>>proposal for how that could look like?
>
> What about:
>
> cdef void raise_fooexc(e) except *:
>     raise MyPythonError(e.detail_1, e.detail_2)
>
>
> cdef func(self) except +raise_fooexc(exception_type):
>     something_that_raises_exception_type()
>
> ?
>
> Then `exception_type` would get stuffed inside the catch() instead of ...

How about

cdef void handle_exception(exception_type e):
    ...

cdef extern from *:
    cdef something_that_raises_exception_type() except +handle_exception

If handle_exception takes no arguments, we behave as we do now,
otherwise we look at the single argument type and translate that into

catch (exception_type e) {
    handle_exception(e);
}

Taking things a step further, it could make sense to support C++
exceptions in standard Cython try...except blocks too. I've toyed with
the idea of putting signal handling support there as well. This would
not require any new syntax.

- Robert

From robertwb at gmail.com  Sun Jul  8 09:59:11 2012
From: robertwb at gmail.com (Robert Bradshaw)
Date: Sun, 8 Jul 2012 00:59:11 -0700
Subject: [Cython] [cython-users] C++: how to handle failures of 'new'?
In-Reply-To: <CANg26EUTjNtQz6ZX=Ve_cq9r3A3AEUJfKBPv4Yc2T-M5yVGoqA@mail.gmail.com>
References: <4FECA49B.8090404@behnel.de>
	<CADiQ+QBeBw=FYsNPpSU_OJOU48m7E1=csxMxfenMQLs2jhJgPw@mail.gmail.com>
	<4FED40E1.1040301@behnel.de>
	<CADiQ+QAFYMgycY5-RkTf7SOwDb0-qM93Y8+sEzA=+1P9DSyOyA@mail.gmail.com>
	<4FF29BE3.6030201@behnel.de> <4FF31999.6090501@astro.uio.no>
	<4FF31FF8.10507@behnel.de>
	<CADiQ+QA8nRcpCUsmMx=zb50XQAnOXLVky5pxOxQu=DQxF+BMfw@mail.gmail.com>
	<4FF335D2.3070706@behnel.de>
	<CADiQ+QATCzuJ2vNXJm-QxueG9s_DBPyCBRGhZ1cnMpEMQvMtzQ@mail.gmail.com>
	<4FF33D60.5080804@astro.uio.no>
	<CADiQ+QD0etaANiRy=03=yTjCi60xCpN+GVV5=mueWuQnBotrDQ@mail.gmail.com>
	<CANg26EVTZC0mfxbcka_v8fE3try5YkLgc8x-VyvBh0kooT97iA@mail.gmail.com>
	<23685ab8-5945-418f-b3f6-0c7a92a03bea@email.android.com>
	<CANg26EWp-fqysQeE58hVDeWtWZJ9p+sPem8MOwaEs1tUdqcKZg@mail.gmail.com>
	<84789ed7-3426-4d18-818e-e680bea9487b@email.android.com>
	<CANg26EUTjNtQz6ZX=Ve_cq9r3A3AEUJfKBPv4Yc2T-M5yVGoqA@mail.gmail.com>
Message-ID: <CADiQ+QByp=7_40pvYAAoMht9_Nt_Zw+1MBsDQUGZqe_SvNzV=Q@mail.gmail.com>

On Fri, Jul 6, 2012 at 2:33 AM, mark florisson
<markflorisson88 at gmail.com> wrote:
> On 6 July 2012 03:34, Dag Sverre Seljebotn <d.s.seljebotn at astro.uio.no> wrote:
>>
>>
>> mark florisson <markflorisson88 at gmail.com> wrote:
>>
>>>On 5 July 2012 21:46, Dag Sverre Seljebotn <d.s.seljebotn at astro.uio.no>
>>>wrote:
>>>>
>>>>
>>>> mark florisson <markflorisson88 at gmail.com> wrote:
>>>>
>>>>>On 3 July 2012 20:15, Robert Bradshaw <robertwb at gmail.com> wrote:
>>>>>> On Tue, Jul 3, 2012 at 11:43 AM, Dag Sverre Seljebotn
>>>>>> <d.s.seljebotn at astro.uio.no> wrote:
>>>>>>> On 07/03/2012 08:23 PM, Robert Bradshaw wrote:
>>>>>>>>
>>>>>>>> On Tue, Jul 3, 2012 at 11:11 AM, Stefan
>>>Behnel<stefan_ml at behnel.de>
>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> Robert Bradshaw, 03.07.2012 19:58:
>>>>>>>>>>
>>>>>>>>>> On Tue, Jul 3, 2012 at 9:38 AM, Stefan Behnel wrote:
>>>>>>>>>>>
>>>>>>>>>>> Dag Sverre Seljebotn, 03.07.2012 18:11:
>>>>>>>>>>>>
>>>>>>>>>>>> On 07/03/2012 09:14 AM, Stefan Behnel wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> I don't know what happens if a C++ exception is not being
>>>>>caught, but
>>>>>>>>>>>>> I
>>>>>>>>>>>>> guess it would simply crash the application. That's a bit
>>>more
>>>>>>>>>>>>> visible than
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Yep.
>>>>>>>>>>>>
>>>>>>>>>>>>> just printing a warning when a Python exception is being
>>>>>ignored due
>>>>>>>>>>>>> to a
>>>>>>>>>>>>> missing declaration. It's really unfortunate that our
>>>>>documentation
>>>>>>>>>>>>> didn't
>>>>>>>>>>>>> even mention the need for this, because it's not immediately
>>>>>obvious
>>>>>>>>>>>>> that
>>>>>>>>>>>>> Cython won't handle errors in "new", and testing for memory
>>>>>errors
>>>>>>>>>>>>> isn't
>>>>>>>>>>>>> quite what people commonly do in their test suites.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Apart from that, I agree, users have to take care to
>>>properly
>>>>>declare
>>>>>>>>>>>>> the
>>>>>>>>>>>>> API they are using.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Is there any time you do NOT want a "catch (...) {}" block? I
>>>>>can't
>>>>>>>>>>>> see a
>>>>>>>>>>>> C++ exception propagating to Python-land doing anything
>>>useful
>>>>>ever.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> That would have been my intuition, too.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> If it's actually embedded, with the main driver in C++, one
>>>might
>>>>>want
>>>>>>>>>> it to propagate up.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> But what kind of a propagation would that be? On the way out, it
>>>>>could
>>>>>>>>> induce anything, from side effects to resource leaks to crashes,
>>>>>>>>> depending
>>>>>>>>> on what the state of the surrounding code is. It would leave the
>>>>>whole
>>>>>>>>> system in an unpredictable state. I cannot imagine anyone really
>>>>>wanting
>>>>>>>>> this.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>>> So shouldn't we just make --cplus turn *all* external
>>>functions
>>>>>and
>>>>>>>>>>>> methods
>>>>>>>>>>>> (whether C-like or C++-like) into "except +"? (Or keep
>>>except+
>>>>>for
>>>>>>>>>>>> manual
>>>>>>>>>>>> translation, but always have a catch(...)".
>>>>>>>>>>>>
>>>>>>>>>>>> Performance overhead is the only reason I can think of to not
>>>>>do this,
>>>>>>>>>>>> although IIRC C++ catch blocks are only dealt with during
>>>stack
>>>>>>>>>>>> unwinds and
>>>>>>>>>>>> doesn't cost anything/much (?) when they're not triggered.
>>>>>>>>>>>>
>>>>>>>>>>>> "except -1" should then actually mean both; "except + except
>>>>>-1". So
>>>>>>>>>>>> it's
>>>>>>>>>>>> more a question of just adding catch(...) *everywhere*, than
>>>>>making
>>>>>>>>>>>> "except
>>>>>>>>>>>> +" the default.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> I have no idea if there is a performance impact, but if there
>>>>>isn't,
>>>>>>>>>>> always
>>>>>>>>>>> catching all exceptions sounds like a reasonable thing to do.
>>>>>After
>>>>>>>>>>> all, we
>>>>>>>>>>> have no support for catching C++ exceptions on user side.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> This is a bit like following every C call with "except *"
>>>(though
>>>>>the
>>>>>>>>>> performance ratios are unclear). It just seems a lot to wrap
>>>>>every
>>>>>>>>>> single line of a non-trivial C++ using function with try..catch
>>>>>>>>>> blocks.
>>>>>>>
>>>>>>>
>>>>>>> It seems "a lot" of just what exactly? Generated code? Binary
>>>size?
>>>>>Time
>>>>>>> spent in GCC parser?
>>>>>>
>>>>>> All of the above. And we should take a look at the runtime overhead
>>>>>> (which is hopefully nil, but who knows.)
>>>>>>
>>>>>>> Though I guess one might want to try to pull out the try-catch to
>>>at
>>>>>least
>>>>>>> only one per code line rather than one per SimpleCallNode.
>>>>>>
>>>>>> Or even higher, if possible. It's still a lot.
>>>>>
>>>>>Why would you have to do that? Can't you just insert a try/catch per
>>>>>try/except or try/finally block, or if absent, the function body.
>>>That
>>>>>will still work with the way temporaries are cleaned up. (It should
>>>>>also be implemented for parallel/prange sections).
>>>>
>>>> One disadvantage is that you don't get source code line for the .pyx
>>>file in the stack trace. Which is often exactly the information you are
>>>looking for (even worse, since C++ stack isn't in the stack trace, the
>>>lineno for what seems like the ' ultimate cause' is not there). Having
>>>to surround statements with try/except just to pinpoint which one is
>>>raising the exception would be incredibly irritating.
>>>>
>>>> Dag
>>>
>>>Oh yeah, good point. Maybe we could use these zero-cost exceptions for
>>>cdef functions in Cython though, instead of error checks (if it
>>>appears to make any significant difference). Basically instead of the
>>>'error' argument in CEP 526. It'd need version that ABI as well...
>>>
>>
>> Wait, one can just do
>>
>> lineno = 25;
>> a()
>> linono = 27;
>> b()
>>
>> And wrap the block with a try catch.
>
> Heh, I just had the same idea before going to bed yesterday :) But it
> means the common case (no exception) will get some overhead. That can
> probably be afforded since normal error checking branches anyway.
> Maybe it's worth investigating how many instructions try/catch blocks
> generate and seeing whether a) it's (significantly) better than
> branching and b) whether a try/catch block can be afforded per
> subexpression/lineno.

For sure some performance testing needs to be done to move this
significantly forward.

>> I'm not sure if I like C++ exceptions internally in Cython myself. It would mean that C-compiled Cython modules would not be able to call C++-compiled Cython modules, which I think would create a lot of headache. And I think we want to keep the ability to compile with a C compiler if you don't call C++ code...
>
> I don't know how often people mix the two. It's probably not worth
> making a directive either... Are there any platforms where installing
> a C++ compiler is more of a hassle than a C compiler? E.g. Theano
> depends on a C++ compiler (and a lot more as well).

Are you implying, for example, that one would have to re-compile NumPy
with a C++ compiler if you wanted to use any C++ extension modules?
I'd rather avoid this.

> The cool thing about it would be that as you said earlier in the
> thread, you could at the very least do one try/catch per line number,
> so you wouldn't need N branches for N subexpressions that can fail.
>
>> Dag
>>
>>>>>
>>>>>>> "except *" only has a point when calling functions using the
>>>CPython
>>>>>API,
>>>>>>> but most external C functions are pure C, not
>>>>>CPython-API-using-functions.
>>>>>>> OTOH, all external C++ functions are C++ :-)
>>>>>>
>>>>>> Fair point.
>>>>>>
>>>>>>> (Also, if we wrote Cython from scratch now I'm pretty sure the
>>>>>"except *"
>>>>>>> defaults would be a tad different.)
>>>>>>
>>>>>> For sure.
>>>>>>
>>>>>>>>> But if users are correct about their declarations, we'd end up
>>>>>with the
>>>>>>>>> same thing. I think it's worth a try.
>>>>>>>>
>>>>>>>>
>>>>>>>> Most C++ code (that I've ever run into) doesn't use exceptions,
>>>>>>>> because exception handling is so broken in C++ anyways.
>>>>>>>
>>>>>>>
>>>>>>> Except for the fact that any code touching "new" could be raising
>>>>>>> exceptions? That propagates.
>>>>>>
>>>>>> I would guess most of the time people don't bother catching these
>>>and
>>>>>> let the program die, as there's often no sane recovery (the same as
>>>>>> MemoryErrors in Python, though I guess C++ is less often used from
>>>an
>>>>>> event loop).
>>>>>>
>>>>>>> There is a lot of C++ code out there using exceptions. I'd guess
>>>>>that both
>>>>>>> mathematical code and Google-written code is unlike most C++ code
>>>>>out there
>>>>>>> :-) Many C++ programmers go on and on about RAII and auto_ptrs and
>>>>>so on,
>>>>>>> and that doesn't have much point unless you throw an exception now
>>>>>and then
>>>>>>> (OK, there's the occasional return statement where it matters
>>>well).
>>>>>>
>>>>>> True, I've seen a small subset of the C++ code that's out there.
>>>>>Maybe
>>>>>> numerical computations use it a lot?
>>>>>>
>>>>>> +1 to making catch-everywhere a directive at least. Lets see what
>>>the
>>>>>> impact is before we decide to make it the default.
>>>>>>
>>>>>> - Robert
>>>>>> _______________________________________________
>>>>>> cython-devel mailing list
>>>>>> cython-devel at python.org
>>>>>> http://mail.python.org/mailman/listinfo/cython-devel
>>>>>_______________________________________________
>>>>>cython-devel mailing list
>>>>>cython-devel at python.org
>>>>>http://mail.python.org/mailman/listinfo/cython-devel
>>>>
>>>> --
>>>> Sent from my Android phone with K-9 Mail. Please excuse my brevity.
>>>> _______________________________________________
>>>> cython-devel mailing list
>>>> cython-devel at python.org
>>>> http://mail.python.org/mailman/listinfo/cython-devel
>>>_______________________________________________
>>>cython-devel mailing list
>>>cython-devel at python.org
>>>http://mail.python.org/mailman/listinfo/cython-devel
>>
>> --
>> Sent from my Android phone with K-9 Mail. Please excuse my brevity.
>> _______________________________________________
>> cython-devel mailing list
>> cython-devel at python.org
>> http://mail.python.org/mailman/listinfo/cython-devel
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

From markflorisson88 at gmail.com  Sun Jul  8 11:34:47 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Sun, 8 Jul 2012 10:34:47 +0100
Subject: [Cython] [cython-users] C++: how to handle failures of 'new'?
In-Reply-To: <CADiQ+QByp=7_40pvYAAoMht9_Nt_Zw+1MBsDQUGZqe_SvNzV=Q@mail.gmail.com>
References: <4FECA49B.8090404@behnel.de>
	<CADiQ+QBeBw=FYsNPpSU_OJOU48m7E1=csxMxfenMQLs2jhJgPw@mail.gmail.com>
	<4FED40E1.1040301@behnel.de>
	<CADiQ+QAFYMgycY5-RkTf7SOwDb0-qM93Y8+sEzA=+1P9DSyOyA@mail.gmail.com>
	<4FF29BE3.6030201@behnel.de> <4FF31999.6090501@astro.uio.no>
	<4FF31FF8.10507@behnel.de>
	<CADiQ+QA8nRcpCUsmMx=zb50XQAnOXLVky5pxOxQu=DQxF+BMfw@mail.gmail.com>
	<4FF335D2.3070706@behnel.de>
	<CADiQ+QATCzuJ2vNXJm-QxueG9s_DBPyCBRGhZ1cnMpEMQvMtzQ@mail.gmail.com>
	<4FF33D60.5080804@astro.uio.no>
	<CADiQ+QD0etaANiRy=03=yTjCi60xCpN+GVV5=mueWuQnBotrDQ@mail.gmail.com>
	<CANg26EVTZC0mfxbcka_v8fE3try5YkLgc8x-VyvBh0kooT97iA@mail.gmail.com>
	<23685ab8-5945-418f-b3f6-0c7a92a03bea@email.android.com>
	<CANg26EWp-fqysQeE58hVDeWtWZJ9p+sPem8MOwaEs1tUdqcKZg@mail.gmail.com>
	<84789ed7-3426-4d18-818e-e680bea9487b@email.android.com>
	<CANg26EUTjNtQz6ZX=Ve_cq9r3A3AEUJfKBPv4Yc2T-M5yVGoqA@mail.gmail.com>
	<CADiQ+QByp=7_40pvYAAoMht9_Nt_Zw+1MBsDQUGZqe_SvNzV=Q@mail.gmail.com>
Message-ID: <CANg26EVU8_7ZBSTi56PUpN5ePSuPWT94inE9j+FLWzxAQBTnqw@mail.gmail.com>

On 8 July 2012 08:59, Robert Bradshaw <robertwb at gmail.com> wrote:
> On Fri, Jul 6, 2012 at 2:33 AM, mark florisson
> <markflorisson88 at gmail.com> wrote:
>> On 6 July 2012 03:34, Dag Sverre Seljebotn <d.s.seljebotn at astro.uio.no> wrote:
>>>
>>>
>>> mark florisson <markflorisson88 at gmail.com> wrote:
>>>
>>>>On 5 July 2012 21:46, Dag Sverre Seljebotn <d.s.seljebotn at astro.uio.no>
>>>>wrote:
>>>>>
>>>>>
>>>>> mark florisson <markflorisson88 at gmail.com> wrote:
>>>>>
>>>>>>On 3 July 2012 20:15, Robert Bradshaw <robertwb at gmail.com> wrote:
>>>>>>> On Tue, Jul 3, 2012 at 11:43 AM, Dag Sverre Seljebotn
>>>>>>> <d.s.seljebotn at astro.uio.no> wrote:
>>>>>>>> On 07/03/2012 08:23 PM, Robert Bradshaw wrote:
>>>>>>>>>
>>>>>>>>> On Tue, Jul 3, 2012 at 11:11 AM, Stefan
>>>>Behnel<stefan_ml at behnel.de>
>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> Robert Bradshaw, 03.07.2012 19:58:
>>>>>>>>>>>
>>>>>>>>>>> On Tue, Jul 3, 2012 at 9:38 AM, Stefan Behnel wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> Dag Sverre Seljebotn, 03.07.2012 18:11:
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 07/03/2012 09:14 AM, Stefan Behnel wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I don't know what happens if a C++ exception is not being
>>>>>>caught, but
>>>>>>>>>>>>>> I
>>>>>>>>>>>>>> guess it would simply crash the application. That's a bit
>>>>more
>>>>>>>>>>>>>> visible than
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Yep.
>>>>>>>>>>>>>
>>>>>>>>>>>>>> just printing a warning when a Python exception is being
>>>>>>ignored due
>>>>>>>>>>>>>> to a
>>>>>>>>>>>>>> missing declaration. It's really unfortunate that our
>>>>>>documentation
>>>>>>>>>>>>>> didn't
>>>>>>>>>>>>>> even mention the need for this, because it's not immediately
>>>>>>obvious
>>>>>>>>>>>>>> that
>>>>>>>>>>>>>> Cython won't handle errors in "new", and testing for memory
>>>>>>errors
>>>>>>>>>>>>>> isn't
>>>>>>>>>>>>>> quite what people commonly do in their test suites.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Apart from that, I agree, users have to take care to
>>>>properly
>>>>>>declare
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>> API they are using.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Is there any time you do NOT want a "catch (...) {}" block? I
>>>>>>can't
>>>>>>>>>>>>> see a
>>>>>>>>>>>>> C++ exception propagating to Python-land doing anything
>>>>useful
>>>>>>ever.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> That would have been my intuition, too.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> If it's actually embedded, with the main driver in C++, one
>>>>might
>>>>>>want
>>>>>>>>>>> it to propagate up.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> But what kind of a propagation would that be? On the way out, it
>>>>>>could
>>>>>>>>>> induce anything, from side effects to resource leaks to crashes,
>>>>>>>>>> depending
>>>>>>>>>> on what the state of the surrounding code is. It would leave the
>>>>>>whole
>>>>>>>>>> system in an unpredictable state. I cannot imagine anyone really
>>>>>>wanting
>>>>>>>>>> this.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>>> So shouldn't we just make --cplus turn *all* external
>>>>functions
>>>>>>and
>>>>>>>>>>>>> methods
>>>>>>>>>>>>> (whether C-like or C++-like) into "except +"? (Or keep
>>>>except+
>>>>>>for
>>>>>>>>>>>>> manual
>>>>>>>>>>>>> translation, but always have a catch(...)".
>>>>>>>>>>>>>
>>>>>>>>>>>>> Performance overhead is the only reason I can think of to not
>>>>>>do this,
>>>>>>>>>>>>> although IIRC C++ catch blocks are only dealt with during
>>>>stack
>>>>>>>>>>>>> unwinds and
>>>>>>>>>>>>> doesn't cost anything/much (?) when they're not triggered.
>>>>>>>>>>>>>
>>>>>>>>>>>>> "except -1" should then actually mean both; "except + except
>>>>>>-1". So
>>>>>>>>>>>>> it's
>>>>>>>>>>>>> more a question of just adding catch(...) *everywhere*, than
>>>>>>making
>>>>>>>>>>>>> "except
>>>>>>>>>>>>> +" the default.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> I have no idea if there is a performance impact, but if there
>>>>>>isn't,
>>>>>>>>>>>> always
>>>>>>>>>>>> catching all exceptions sounds like a reasonable thing to do.
>>>>>>After
>>>>>>>>>>>> all, we
>>>>>>>>>>>> have no support for catching C++ exceptions on user side.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> This is a bit like following every C call with "except *"
>>>>(though
>>>>>>the
>>>>>>>>>>> performance ratios are unclear). It just seems a lot to wrap
>>>>>>every
>>>>>>>>>>> single line of a non-trivial C++ using function with try..catch
>>>>>>>>>>> blocks.
>>>>>>>>
>>>>>>>>
>>>>>>>> It seems "a lot" of just what exactly? Generated code? Binary
>>>>size?
>>>>>>Time
>>>>>>>> spent in GCC parser?
>>>>>>>
>>>>>>> All of the above. And we should take a look at the runtime overhead
>>>>>>> (which is hopefully nil, but who knows.)
>>>>>>>
>>>>>>>> Though I guess one might want to try to pull out the try-catch to
>>>>at
>>>>>>least
>>>>>>>> only one per code line rather than one per SimpleCallNode.
>>>>>>>
>>>>>>> Or even higher, if possible. It's still a lot.
>>>>>>
>>>>>>Why would you have to do that? Can't you just insert a try/catch per
>>>>>>try/except or try/finally block, or if absent, the function body.
>>>>That
>>>>>>will still work with the way temporaries are cleaned up. (It should
>>>>>>also be implemented for parallel/prange sections).
>>>>>
>>>>> One disadvantage is that you don't get source code line for the .pyx
>>>>file in the stack trace. Which is often exactly the information you are
>>>>looking for (even worse, since C++ stack isn't in the stack trace, the
>>>>lineno for what seems like the ' ultimate cause' is not there). Having
>>>>to surround statements with try/except just to pinpoint which one is
>>>>raising the exception would be incredibly irritating.
>>>>>
>>>>> Dag
>>>>
>>>>Oh yeah, good point. Maybe we could use these zero-cost exceptions for
>>>>cdef functions in Cython though, instead of error checks (if it
>>>>appears to make any significant difference). Basically instead of the
>>>>'error' argument in CEP 526. It'd need version that ABI as well...
>>>>
>>>
>>> Wait, one can just do
>>>
>>> lineno = 25;
>>> a()
>>> linono = 27;
>>> b()
>>>
>>> And wrap the block with a try catch.
>>
>> Heh, I just had the same idea before going to bed yesterday :) But it
>> means the common case (no exception) will get some overhead. That can
>> probably be afforded since normal error checking branches anyway.
>> Maybe it's worth investigating how many instructions try/catch blocks
>> generate and seeing whether a) it's (significantly) better than
>> branching and b) whether a try/catch block can be afforded per
>> subexpression/lineno.
>
> For sure some performance testing needs to be done to move this
> significantly forward.
>
>>> I'm not sure if I like C++ exceptions internally in Cython myself. It would mean that C-compiled Cython modules would not be able to call C++-compiled Cython modules, which I think would create a lot of headache. And I think we want to keep the ability to compile with a C compiler if you don't call C++ code...
>>
>> I don't know how often people mix the two. It's probably not worth
>> making a directive either... Are there any platforms where installing
>> a C++ compiler is more of a hassle than a C compiler? E.g. Theano
>> depends on a C++ compiler (and a lot more as well).
>
> Are you implying, for example, that one would have to re-compile NumPy
> with a C++ compiler if you wanted to use any C++ extension modules?
> I'd rather avoid this.

No, you can still freely mix any extension modules, it's just that
when you cimport stuff from other Cython modules, they have to be ABI
compatible. If C++ is part of the ABI due to how exceptions are
propagated and handled, then those modules need to be compiled both
with C++ (or neither, to get the other semantics with slower (to be
verified) error checks).

>> The cool thing about it would be that as you said earlier in the
>> thread, you could at the very least do one try/catch per line number,
>> so you wouldn't need N branches for N subexpressions that can fail.
>>
>>> Dag
>>>
>>>>>>
>>>>>>>> "except *" only has a point when calling functions using the
>>>>CPython
>>>>>>API,
>>>>>>>> but most external C functions are pure C, not
>>>>>>CPython-API-using-functions.
>>>>>>>> OTOH, all external C++ functions are C++ :-)
>>>>>>>
>>>>>>> Fair point.
>>>>>>>
>>>>>>>> (Also, if we wrote Cython from scratch now I'm pretty sure the
>>>>>>"except *"
>>>>>>>> defaults would be a tad different.)
>>>>>>>
>>>>>>> For sure.
>>>>>>>
>>>>>>>>>> But if users are correct about their declarations, we'd end up
>>>>>>with the
>>>>>>>>>> same thing. I think it's worth a try.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Most C++ code (that I've ever run into) doesn't use exceptions,
>>>>>>>>> because exception handling is so broken in C++ anyways.
>>>>>>>>
>>>>>>>>
>>>>>>>> Except for the fact that any code touching "new" could be raising
>>>>>>>> exceptions? That propagates.
>>>>>>>
>>>>>>> I would guess most of the time people don't bother catching these
>>>>and
>>>>>>> let the program die, as there's often no sane recovery (the same as
>>>>>>> MemoryErrors in Python, though I guess C++ is less often used from
>>>>an
>>>>>>> event loop).
>>>>>>>
>>>>>>>> There is a lot of C++ code out there using exceptions. I'd guess
>>>>>>that both
>>>>>>>> mathematical code and Google-written code is unlike most C++ code
>>>>>>out there
>>>>>>>> :-) Many C++ programmers go on and on about RAII and auto_ptrs and
>>>>>>so on,
>>>>>>>> and that doesn't have much point unless you throw an exception now
>>>>>>and then
>>>>>>>> (OK, there's the occasional return statement where it matters
>>>>well).
>>>>>>>
>>>>>>> True, I've seen a small subset of the C++ code that's out there.
>>>>>>Maybe
>>>>>>> numerical computations use it a lot?
>>>>>>>
>>>>>>> +1 to making catch-everywhere a directive at least. Lets see what
>>>>the
>>>>>>> impact is before we decide to make it the default.
>>>>>>>
>>>>>>> - Robert
>>>>>>> _______________________________________________
>>>>>>> cython-devel mailing list
>>>>>>> cython-devel at python.org
>>>>>>> http://mail.python.org/mailman/listinfo/cython-devel
>>>>>>_______________________________________________
>>>>>>cython-devel mailing list
>>>>>>cython-devel at python.org
>>>>>>http://mail.python.org/mailman/listinfo/cython-devel
>>>>>
>>>>> --
>>>>> Sent from my Android phone with K-9 Mail. Please excuse my brevity.
>>>>> _______________________________________________
>>>>> cython-devel mailing list
>>>>> cython-devel at python.org
>>>>> http://mail.python.org/mailman/listinfo/cython-devel
>>>>_______________________________________________
>>>>cython-devel mailing list
>>>>cython-devel at python.org
>>>>http://mail.python.org/mailman/listinfo/cython-devel
>>>
>>> --
>>> Sent from my Android phone with K-9 Mail. Please excuse my brevity.
>>> _______________________________________________
>>> cython-devel mailing list
>>> cython-devel at python.org
>>> http://mail.python.org/mailman/listinfo/cython-devel
>> _______________________________________________
>> cython-devel mailing list
>> cython-devel at python.org
>> http://mail.python.org/mailman/listinfo/cython-devel
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

From stefan_ml at behnel.de  Sun Jul  8 12:17:35 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 08 Jul 2012 12:17:35 +0200
Subject: [Cython] [cython-users] C++: how to handle failures of 'new'?
In-Reply-To: <CANg26EVU8_7ZBSTi56PUpN5ePSuPWT94inE9j+FLWzxAQBTnqw@mail.gmail.com>
References: <4FECA49B.8090404@behnel.de> <4FED40E1.1040301@behnel.de>
	<CADiQ+QAFYMgycY5-RkTf7SOwDb0-qM93Y8+sEzA=+1P9DSyOyA@mail.gmail.com>
	<4FF29BE3.6030201@behnel.de> <4FF31999.6090501@astro.uio.no>
	<4FF31FF8.10507@behnel.de>
	<CADiQ+QA8nRcpCUsmMx=zb50XQAnOXLVky5pxOxQu=DQxF+BMfw@mail.gmail.com>
	<4FF335D2.3070706@behnel.de>
	<CADiQ+QATCzuJ2vNXJm-QxueG9s_DBPyCBRGhZ1cnMpEMQvMtzQ@mail.gmail.com>
	<4FF33D60.5080804@astro.uio.no>
	<CADiQ+QD0etaANiRy=03=yTjCi60xCpN+GVV5=mueWuQnBotrDQ@mail.gmail.com>
	<CANg26EVTZC0mfxbcka_v8fE3try5YkLgc8x-VyvBh0kooT97iA@mail.gmail.com>
	<23685ab8-5945-418f-b3f6-0c7a92a03bea@email.android.com>
	<CANg26EWp-fqysQeE58hVDeWtWZJ9p+sPem8MOwaEs1tUdqcKZg@mail.gmail.com>
	<84789ed7-3426-4d18-818e-e680bea9487b@email.android.com>
	<CANg26EUTjNtQz6ZX=Ve_cq9r3A3AEUJfKBPv4Yc2T-M5yVGoqA@mail.gmail.com>
	<CADiQ+QByp=7_40pvYAAoMht9_Nt_Zw+1MBsDQUGZqe_SvNzV=Q@mail.gmail.com>
	<CANg26EVU8_7ZBSTi56PUpN5ePSuPWT94inE9j+FLWzxAQBTnqw@mail.gmail.com>
Message-ID: <4FF95E3F.6060201@behnel.de>

mark florisson, 08.07.2012 11:34:
> On 8 July 2012 08:59, Robert Bradshaw wrote:
>> On Fri, Jul 6, 2012 at 2:33 AM, mark florisson wrote:
>>> On 6 July 2012 03:34, Dag Sverre Seljebotn wrote:
>>>> I'm not sure if I like C++ exceptions internally in Cython myself.
>>>> It would mean that C-compiled Cython modules would not be able to call
>>>> C++-compiled Cython modules, which I think would create a lot of headache.
>>>> And I think we want to keep the ability to compile with a C compiler if you
>>>> don't call C++ code...
>>> 
>>> I don't know how often people mix the two. It's probably not worth
>>> making a directive either... Are there any platforms where installing
>>> a C++ compiler is more of a hassle than a C compiler? E.g. Theano
>>> depends on a C++ compiler (and a lot more as well).
>>
>> Are you implying, for example, that one would have to re-compile NumPy
>> with a C++ compiler if you wanted to use any C++ extension modules?
>> I'd rather avoid this.
> 
> No, you can still freely mix any extension modules, it's just that
> when you cimport stuff from other Cython modules, they have to be ABI
> compatible. If C++ is part of the ABI due to how exceptions are
> propagated and handled, then those modules need to be compiled both
> with C++ (or neither, to get the other semantics with slower (to be
> verified) error checks).

-1. I don't want to prevent users of the public C-API of lxml from talking
to C++ code in their own modules.

Stefan

From stefan_ml at behnel.de  Sun Jul  8 12:33:38 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 08 Jul 2012 12:33:38 +0200
Subject: [Cython] errors in C++ tests
Message-ID: <4FF96202.9060003@behnel.de>

Hi,

I'm seeing C++ compiler errors errors in the C++ tests.

https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests/BACKEND=cpp,PYVERSION=py3k/491/testReport/

"""
cpp_operators.cpp: In function ?void __Pyx_CppExn2PyErr()?:
cpp_operators.cpp:442: error: expected unqualified-id before ?&? token
cpp_operators.cpp:442: error: expected `)' before ?&? token
"""

The failing code line is this:

"""
  } catch (const std::bad_alloc& exn) {
"""

That line looks ok to me. It started failing after I enabled C++ error
handling for default constructors, which then lead to the above function
being generated into the module code.

Does anyone know what to make of this?

Stefan

From markflorisson88 at gmail.com  Sun Jul  8 13:42:23 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Sun, 8 Jul 2012 12:42:23 +0100
Subject: [Cython] [cython-users] C++: how to handle failures of 'new'?
In-Reply-To: <4FF95E3F.6060201@behnel.de>
References: <4FECA49B.8090404@behnel.de> <4FED40E1.1040301@behnel.de>
	<CADiQ+QAFYMgycY5-RkTf7SOwDb0-qM93Y8+sEzA=+1P9DSyOyA@mail.gmail.com>
	<4FF29BE3.6030201@behnel.de> <4FF31999.6090501@astro.uio.no>
	<4FF31FF8.10507@behnel.de>
	<CADiQ+QA8nRcpCUsmMx=zb50XQAnOXLVky5pxOxQu=DQxF+BMfw@mail.gmail.com>
	<4FF335D2.3070706@behnel.de>
	<CADiQ+QATCzuJ2vNXJm-QxueG9s_DBPyCBRGhZ1cnMpEMQvMtzQ@mail.gmail.com>
	<4FF33D60.5080804@astro.uio.no>
	<CADiQ+QD0etaANiRy=03=yTjCi60xCpN+GVV5=mueWuQnBotrDQ@mail.gmail.com>
	<CANg26EVTZC0mfxbcka_v8fE3try5YkLgc8x-VyvBh0kooT97iA@mail.gmail.com>
	<23685ab8-5945-418f-b3f6-0c7a92a03bea@email.android.com>
	<CANg26EWp-fqysQeE58hVDeWtWZJ9p+sPem8MOwaEs1tUdqcKZg@mail.gmail.com>
	<84789ed7-3426-4d18-818e-e680bea9487b@email.android.com>
	<CANg26EUTjNtQz6ZX=Ve_cq9r3A3AEUJfKBPv4Yc2T-M5yVGoqA@mail.gmail.com>
	<CADiQ+QByp=7_40pvYAAoMht9_Nt_Zw+1MBsDQUGZqe_SvNzV=Q@mail.gmail.com>
	<CANg26EVU8_7ZBSTi56PUpN5ePSuPWT94inE9j+FLWzxAQBTnqw@mail.gmail.com>
	<4FF95E3F.6060201@behnel.de>
Message-ID: <CANg26EXrih--C1f386E2+pF-0B3BscHSUB9d3L4g=u+MgkKriw@mail.gmail.com>

On 8 July 2012 11:17, Stefan Behnel <stefan_ml at behnel.de> wrote:
> mark florisson, 08.07.2012 11:34:
>> On 8 July 2012 08:59, Robert Bradshaw wrote:
>>> On Fri, Jul 6, 2012 at 2:33 AM, mark florisson wrote:
>>>> On 6 July 2012 03:34, Dag Sverre Seljebotn wrote:
>>>>> I'm not sure if I like C++ exceptions internally in Cython myself.
>>>>> It would mean that C-compiled Cython modules would not be able to call
>>>>> C++-compiled Cython modules, which I think would create a lot of headache.
>>>>> And I think we want to keep the ability to compile with a C compiler if you
>>>>> don't call C++ code...
>>>>
>>>> I don't know how often people mix the two. It's probably not worth
>>>> making a directive either... Are there any platforms where installing
>>>> a C++ compiler is more of a hassle than a C compiler? E.g. Theano
>>>> depends on a C++ compiler (and a lot more as well).
>>>
>>> Are you implying, for example, that one would have to re-compile NumPy
>>> with a C++ compiler if you wanted to use any C++ extension modules?
>>> I'd rather avoid this.
>>
>> No, you can still freely mix any extension modules, it's just that
>> when you cimport stuff from other Cython modules, they have to be ABI
>> compatible. If C++ is part of the ABI due to how exceptions are
>> propagated and handled, then those modules need to be compiled both
>> with C++ (or neither, to get the other semantics with slower (to be
>> verified) error checks).
>
> -1. I don't want to prevent users of the public C-API of lxml from talking
> to C++ code in their own modules.
>
> Stefan
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

I think you misunderstand. This would only be part of the internal
Cython ABI. If you use public/api functions, or take the pointer to
the function, you get the C semantics (possibly with unraisable
exceptions). The details are in the CEP, this would only replace the
extra 'error' argument to cdef functions.

From L.J.Buitinck at uva.nl  Sun Jul  8 20:38:46 2012
From: L.J.Buitinck at uva.nl (Lars Buitinck)
Date: Sun, 8 Jul 2012 20:38:46 +0200
Subject: [Cython] errors in C++ tests
In-Reply-To: <4FF96202.9060003@behnel.de>
References: <4FF96202.9060003@behnel.de>
Message-ID: <CAKz-xUfFD-uKbiRDKNSSAawHfGAvK=kLnCYiqkj0iqtzHWDfXA@mail.gmail.com>

2012/7/8 Stefan Behnel <stefan_ml at behnel.de>:
> """
> cpp_operators.cpp: In function ?void __Pyx_CppExn2PyErr()?:
> cpp_operators.cpp:442: error: expected unqualified-id before ?&? token
> cpp_operators.cpp:442: error: expected `)' before ?&? token
> """
>
> The failing code line is this:
>
> """
>   } catch (const std::bad_alloc& exn) {
> """

Could you check whether the header <new> is included? It should be to
get the definition of bad_alloc.

-- 
Lars Buitinck
Scientific programmer, ILPS
University of Amsterdam

From stefan_ml at behnel.de  Sun Jul  8 20:54:59 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 08 Jul 2012 20:54:59 +0200
Subject: [Cython] errors in C++ tests
In-Reply-To: <CAKz-xUfFD-uKbiRDKNSSAawHfGAvK=kLnCYiqkj0iqtzHWDfXA@mail.gmail.com>
References: <4FF96202.9060003@behnel.de>
	<CAKz-xUfFD-uKbiRDKNSSAawHfGAvK=kLnCYiqkj0iqtzHWDfXA@mail.gmail.com>
Message-ID: <4FF9D783.60609@behnel.de>

Lars Buitinck, 08.07.2012 20:38:
> 2012/7/8 Stefan Behnel:
>> """
>> cpp_operators.cpp: In function ?void __Pyx_CppExn2PyErr()?:
>> cpp_operators.cpp:442: error: expected unqualified-id before ?&? token
>> cpp_operators.cpp:442: error: expected `)' before ?&? token
>> """
>>
>> The failing code line is this:
>>
>> """
>>   } catch (const std::bad_alloc& exn) {
>> """
> 
> Could you check whether the header <new> is included? It should be to
> get the definition of bad_alloc.

Ah, yes. I'm sure that's it. Are there any side-effects in that header
file, or would it be ok to always include it in C++ mode when the above
exception conversion function is used?

Stefan

From stefan_ml at behnel.de  Sun Jul  8 21:18:54 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 08 Jul 2012 21:18:54 +0200
Subject: [Cython] errors in C++ tests
In-Reply-To: <4FF9D783.60609@behnel.de>
References: <4FF96202.9060003@behnel.de>
	<CAKz-xUfFD-uKbiRDKNSSAawHfGAvK=kLnCYiqkj0iqtzHWDfXA@mail.gmail.com>
	<4FF9D783.60609@behnel.de>
Message-ID: <4FF9DD1E.1000105@behnel.de>

Stefan Behnel, 08.07.2012 20:54:
> Lars Buitinck, 08.07.2012 20:38:
>> 2012/7/8 Stefan Behnel:
>>> """
>>> cpp_operators.cpp: In function ?void __Pyx_CppExn2PyErr()?:
>>> cpp_operators.cpp:442: error: expected unqualified-id before ?&? token
>>> cpp_operators.cpp:442: error: expected `)' before ?&? token
>>> """
>>>
>>> The failing code line is this:
>>>
>>> """
>>>   } catch (const std::bad_alloc& exn) {
>>> """
>>
>> Could you check whether the header <new> is included? It should be to
>> get the definition of bad_alloc.
> 
> Ah, yes. I'm sure that's it.

... and there also were some more headers missing, so basically, this
feature never worked. Great. Here's the complete implementation with the
four header files that I had to add at the top:

"""
#include <new>
#include <typeinfo>
#include <stdexcept>
#include <iostream>

static void __Pyx_CppExn2PyErr() {
  // Catch a handful of different errors here and turn them into the
  // equivalent Python errors.
  try {
    if (PyErr_Occurred())
      ; // let the latest Python exn pass through and ignore the current one
    else
      throw;
  } catch (const std::bad_alloc& exn) {
    PyErr_SetString(PyExc_MemoryError, exn.what());
  } catch (const std::bad_cast& exn) {
    PyErr_SetString(PyExc_TypeError, exn.what());
  } catch (const std::domain_error& exn) {
    PyErr_SetString(PyExc_ValueError, exn.what());
  } catch (const std::invalid_argument& exn) {
    PyErr_SetString(PyExc_ValueError, exn.what());
  } catch (const std::ios_base::failure& exn) {
    PyErr_SetString(PyExc_IOError, exn.what());
  } catch (const std::out_of_range& exn) {
    // Change out_of_range to IndexError
    PyErr_SetString(PyExc_IndexError, exn.what());
  } catch (const std::overflow_error& exn) {
    PyErr_SetString(PyExc_OverflowError, exn.what());
  } catch (const std::range_error& exn) {
    PyErr_SetString(PyExc_ArithmeticError, exn.what());
  } catch (const std::underflow_error& exn) {
    PyErr_SetString(PyExc_ArithmeticError, exn.what());
  } catch (const std::exception& exn) {
    PyErr_SetString(PyExc_RuntimeError, exn.what());
  }
  catch (...)
  {
    PyErr_SetString(PyExc_RuntimeError, "Unknown exception");
  }
}
"""

Back to this question then:

> Are there any side-effects in that header
> file, or would it be ok to always include it in C++ mode when the above
> exception conversion function is used?

Does it make sense to always include these four headers, or should we try
to handle the exceptions conditionally?

Stefan

From L.J.Buitinck at uva.nl  Sun Jul  8 22:24:48 2012
From: L.J.Buitinck at uva.nl (Lars Buitinck)
Date: Sun, 8 Jul 2012 22:24:48 +0200
Subject: [Cython] errors in C++ tests
In-Reply-To: <4FF9DD1E.1000105@behnel.de>
References: <4FF96202.9060003@behnel.de>
	<CAKz-xUfFD-uKbiRDKNSSAawHfGAvK=kLnCYiqkj0iqtzHWDfXA@mail.gmail.com>
	<4FF9D783.60609@behnel.de> <4FF9DD1E.1000105@behnel.de>
Message-ID: <CAKz-xUcjjd7Zjex+MnfpA9Csik8gcOv3FLP1MouXb+52YMYMDw@mail.gmail.com>

2012/7/8 Stefan Behnel <stefan_ml at behnel.de>:
> ... and there also were some more headers missing, so basically, this
> feature never worked. Great. Here's the complete implementation with the
> four header files that I had to add at the top:

I'm pretty sure that at some point, it worked. I implemented this in
6291a2 and the five prior commits, but I did add the appropriate
headers to Cython/Compiler/Nodes.py.

There is a unit test for this feature in tests/run/cpp_exceptions.pyx;
that file includes a header that in turn includes <ios>, <new> and
<stdexcept>, but not <typeinfo> because it doesn't need that to raise
the exception. (It would be more reliable if the test where refactored
into C++ modules so that the headers need not be indirectly included.)

> #include <iostream>

Actually, you'll want to include <ios> where std::ios_base is
declared. <iostream> is a much "heavier" header.

> Does it make sense to always include these four headers, or should we try
> to handle the exceptions conditionally?

As I said, this should already have been done by
Cython/Compiler/Nodes.py. I'm not sure what's going wrong.

-- 
Lars Buitinck
Scientific programmer, ILPS
University of Amsterdam

From stefan_ml at behnel.de  Sun Jul  8 22:41:13 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 08 Jul 2012 22:41:13 +0200
Subject: [Cython] errors in C++ tests
In-Reply-To: <CAKz-xUcjjd7Zjex+MnfpA9Csik8gcOv3FLP1MouXb+52YMYMDw@mail.gmail.com>
References: <4FF96202.9060003@behnel.de>
	<CAKz-xUfFD-uKbiRDKNSSAawHfGAvK=kLnCYiqkj0iqtzHWDfXA@mail.gmail.com>
	<4FF9D783.60609@behnel.de> <4FF9DD1E.1000105@behnel.de>
	<CAKz-xUcjjd7Zjex+MnfpA9Csik8gcOv3FLP1MouXb+52YMYMDw@mail.gmail.com>
Message-ID: <4FF9F069.5030503@behnel.de>

Lars Buitinck, 08.07.2012 22:24:
> 2012/7/8 Stefan Behnel:
>> ... and there also were some more headers missing, so basically, this
>> feature never worked. Great. Here's the complete implementation with the
>> four header files that I had to add at the top:
> 
> I'm pretty sure that at some point, it worked. I implemented this in
> 6291a2 and the five prior commits, but I did add the appropriate
> headers to Cython/Compiler/Nodes.py.
> 
> There is a unit test for this feature in tests/run/cpp_exceptions.pyx;
> that file includes a header that in turn includes <ios>, <new> and
> <stdexcept>, but not <typeinfo> because it doesn't need that to raise
> the exception. (It would be more reliable if the test where refactored
> into C++ modules so that the headers need not be indirectly included.)
> 
>> #include <iostream>
> 
> Actually, you'll want to include <ios> where std::ios_base is
> declared. <iostream> is a much "heavier" header.
> 
>> Does it make sense to always include these four headers, or should we try
>> to handle the exceptions conditionally?
> 
> As I said, this should already have been done by
> Cython/Compiler/Nodes.py. I'm not sure what's going wrong.

Ah, I see it now. Those only work for user declared functions. This problem
arises for the default construction, i.e. the undeclared no-args
constructor that Cython creates automatically. We didn't previously
propagate exceptions for it, so the feature did work and I only broke it by
fixing the default constructor. Thanks for clearing this up.

Any reason why the env.add_include_file() calls would be better than the
header inclusion that I implemented right before the function? It's inside
of the #ifndef block now. After all, they are only needed by that function,
and when users override it by redefining the name from the outside, they do
not need to get included. If users include them themselves, they'll be
included twice - so what.

Stefan

From L.J.Buitinck at uva.nl  Sun Jul  8 23:48:37 2012
From: L.J.Buitinck at uva.nl (Lars Buitinck)
Date: Sun, 8 Jul 2012 23:48:37 +0200
Subject: [Cython] errors in C++ tests
In-Reply-To: <4FF9F069.5030503@behnel.de>
References: <4FF96202.9060003@behnel.de>
	<CAKz-xUfFD-uKbiRDKNSSAawHfGAvK=kLnCYiqkj0iqtzHWDfXA@mail.gmail.com>
	<4FF9D783.60609@behnel.de> <4FF9DD1E.1000105@behnel.de>
	<CAKz-xUcjjd7Zjex+MnfpA9Csik8gcOv3FLP1MouXb+52YMYMDw@mail.gmail.com>
	<4FF9F069.5030503@behnel.de>
Message-ID: <CAKz-xUe5aS_EZxY5JH7A8VshpCVFmVdQL-zXpq4rFtWarvfmjQ@mail.gmail.com>

2012/7/8 Stefan Behnel <stefan_ml at behnel.de>:
> Any reason why the env.add_include_file() calls would be better than the
> header inclusion that I implemented right before the function? It's inside
> of the #ifndef block now. After all, they are only needed by that function,
> and when users override it by redefining the name from the outside, they do
> not need to get included. If users include them themselves, they'll be
> included twice - so what.

I'm not really familiar with Cython internals; I just extended
Compiler/Nodes.py that previously added only <stdexcept>. I observed
that this leads to the headers being included once, at the top of the
file, which seemed like the clean solution.

Multiple inclusion of C++ standard headers should be completely safe,
but I don't know whether the C++ standard says anything about
including standard headers in the middle of a module.

-- 
Lars Buitinck
Scientific programmer, ILPS
University of Amsterdam

From stefan_ml at behnel.de  Tue Jul 10 09:24:54 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 10 Jul 2012 09:24:54 +0200
Subject: [Cython] failing memory view tests in Python 2.4/5
Message-ID: <4FFBD8C6.6090708@behnel.de>

Hi,

I've fixed some of the errors that current block the tests. The remaining
failures in the memory view tests exist in both Py2.4 and 2.5, which hints
at a problem with the buffer support in numpy.pxd.

https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests/BACKEND=c,PYVERSION=py25-ext/504/testReport/

The memslice test failure is annoying. It looks like slice() fails to
properly deep-copy in Py2.[45]. I don't know if we can do anything about
this, seems more like a CPython bug. Deep copying is used to split function
definitions for fused types and I guess slice objects are used in
combination with the memory slices. So both don't currently work together
in Py2.[45].

I'd like to release a 0.17 alpha soon, so please get these sorted out in
one way or another ASAP. I'm also fine with declaring these features
unusable in older Python releases if we can't fix them, but it would
obviously be better to get them working.

Stefan

From markflorisson88 at gmail.com  Tue Jul 10 11:20:07 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Tue, 10 Jul 2012 10:20:07 +0100
Subject: [Cython] failing memory view tests in Python 2.4/5
In-Reply-To: <4FFBD8C6.6090708@behnel.de>
References: <4FFBD8C6.6090708@behnel.de>
Message-ID: <CANg26EV3Ce3ixR=UzM9JSzRkZ6ZO7OR2FLTnjnkxMUQ8MNiV9g@mail.gmail.com>

On 10 July 2012 08:24, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Hi,
>
> I've fixed some of the errors that current block the tests. The remaining
> failures in the memory view tests exist in both Py2.4 and 2.5, which hints
> at a problem with the buffer support in numpy.pxd.
>
> https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests/BACKEND=c,PYVERSION=py25-ext/504/testReport/
>
> The memslice test failure is annoying. It looks like slice() fails to
> properly deep-copy in Py2.[45]. I don't know if we can do anything about
> this, seems more like a CPython bug. Deep copying is used to split function
> definitions for fused types and I guess slice objects are used in
> combination with the memory slices. So both don't currently work together
> in Py2.[45].

I'm pretty sure I fixed the slice copying problem through a
__deepcopy__ overload a long time ago elsewhere. I don't think these
tests should be excluded from the test run yet. Perhaps we should run
the 2x tests from py27 down to 2.4. Tests in newer python's are less
likely to fail.

Do you want _refactor_indexnode merged for this release?

> I'd like to release a 0.17 alpha soon, so please get these sorted out in
> one way or another ASAP. I'm also fine with declaring these features
> unusable in older Python releases if we can't fix them, but it would
> obviously be better to get them working.

I'll try to find some time this week.

> Stefan
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

From vitja.makarov at gmail.com  Fri Jul 13 18:53:18 2012
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Fri, 13 Jul 2012 20:53:18 +0400
Subject: [Cython] Local type inference, first success
Message-ID: <CAKGHGPTru=ZGfYVe2CHbmBvf0S+bthHabpOtZs4xyZOBmBDmRg@mail.gmail.com>

Hi!

I've been a little bit busy last months, now I'm back.

I made first steps on implementing local type inference, TI for
assignments not only for their names.
Here is sample program that works:

(_entry_split) vitja at mchome:~/work/cython-vitek-git/zzz$ cat tf.pyx
# cython: infer_types.verbose=True
from cython cimport typeof

def foo(c):
    a = "abc"
    print a, typeof(a)
    if c:
        a = 123
    else:
        a = 1.123
    print a, typeof(a)

(_entry_split) vitja at mchome:~/work/cython-vitek-git/zzz$ make tf.c
python ../cython.py  -v tf.pyx -o tf.c
Compiling /home/vitja/work/cython-vitek-git/zzz/tf.pyx
Gonna split entries:
entry: Entry(name=a, type=<unspecified>)
... group: set([NameAssignment(entry=Entry(name=a,
type=<unspecified>)), NameAssignment(entry=Entry(name=a,
type=<unspecified>))])
... ... (<FileSourceDescriptor:/home/vitja/work/cython-vitek-git/zzz/tf.pyx>,
8, 10)
... ... (<FileSourceDescriptor:/home/vitja/work/cython-vitek-git/zzz/tf.pyx>,
10, 10)
... group: set([NameAssignment(entry=Entry(name=a, type=<unspecified>))])
... ... (<FileSourceDescriptor:/home/vitja/work/cython-vitek-git/zzz/tf.pyx>,
5, 6)
note: tf.pyx:5:6: inferred 'a.#1' to be of type 'Python object'
note: tf.pyx:5:6: inferred 'a.#0' to be of type 'double'

-- 
vitja.

From vitja.makarov at gmail.com  Fri Jul 13 18:55:09 2012
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Fri, 13 Jul 2012 20:55:09 +0400
Subject: [Cython] Local type inference, first success
In-Reply-To: <CAKGHGPTru=ZGfYVe2CHbmBvf0S+bthHabpOtZs4xyZOBmBDmRg@mail.gmail.com>
References: <CAKGHGPTru=ZGfYVe2CHbmBvf0S+bthHabpOtZs4xyZOBmBDmRg@mail.gmail.com>
Message-ID: <CAKGHGPRz+QQqQV18kji78Y+EFGZCpSNZGEaXeVzTU4iaj9UVUg@mail.gmail.com>

2012/7/13 Vitja Makarov <vitja.makarov at gmail.com>:
> Hi!
>
> I've been a little bit busy last months, now I'm back.
>
> I made first steps on implementing local type inference, TI for
> assignments not only for their names.
> Here is sample program that works:
>
> (_entry_split) vitja at mchome:~/work/cython-vitek-git/zzz$ cat tf.pyx
> # cython: infer_types.verbose=True
> from cython cimport typeof
>
> def foo(c):
>     a = "abc"
>     print a, typeof(a)
>     if c:
>         a = 123
>     else:
>         a = 1.123
>     print a, typeof(a)
>
> (_entry_split) vitja at mchome:~/work/cython-vitek-git/zzz$ make tf.c
> python ../cython.py  -v tf.pyx -o tf.c
> Compiling /home/vitja/work/cython-vitek-git/zzz/tf.pyx
> Gonna split entries:
> entry: Entry(name=a, type=<unspecified>)
> ... group: set([NameAssignment(entry=Entry(name=a,
> type=<unspecified>)), NameAssignment(entry=Entry(name=a,
> type=<unspecified>))])
> ... ... (<FileSourceDescriptor:/home/vitja/work/cython-vitek-git/zzz/tf.pyx>,
> 8, 10)
> ... ... (<FileSourceDescriptor:/home/vitja/work/cython-vitek-git/zzz/tf.pyx>,
> 10, 10)
> ... group: set([NameAssignment(entry=Entry(name=a, type=<unspecified>))])
> ... ... (<FileSourceDescriptor:/home/vitja/work/cython-vitek-git/zzz/tf.pyx>,
> 5, 6)
> note: tf.pyx:5:6: inferred 'a.#1' to be of type 'Python object'
> note: tf.pyx:5:6: inferred 'a.#0' to be of type 'double'
>

Oops, sorry, here is my branch https://github.com/vitek/cython/tree/_entry_split


-- 
vitja.

From stefan_ml at behnel.de  Sun Jul 15 19:41:26 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 15 Jul 2012 19:41:26 +0200
Subject: [Cython] [cython-users] C++: how to handle failures of 'new'?
In-Reply-To: <4FED40E1.1040301@behnel.de>
References: <4FECA49B.8090404@behnel.de>
	<CADiQ+QBeBw=FYsNPpSU_OJOU48m7E1=csxMxfenMQLs2jhJgPw@mail.gmail.com>
	<4FED40E1.1040301@behnel.de>
Message-ID: <500300C6.2040302@behnel.de>

Stefan Behnel, 29.06.2012 07:45:
> [moving this to cython-devel as it's getting technical]
> 
> Robert Bradshaw, 28.06.2012 21:46:
>> On Thu, Jun 28, 2012 at 11:38 AM, Stefan Behnel wrote:
>>> currently, when I write "new CppClass()" in Cython, it generates a straight
>>> call to the "new" operator. It doesn't do any error handling. And the
>>> current documentation doesn't even mention this case.
>>>
>>> Is there a "standard" way to handle this? It seems that C++ has different
>>> ways to deal with failures here but raises an exception by default. Would
>>> you declare the constructor(s) with an "except +MemoryError"? Is there a
>>> reason Cython shouldn't be doing this automatically (if nothing else was
>>> declared) ?
>>
>> I think it certainly makes sense to declare the default constructor as
>> "except +" (and std::bad_alloc should become MemoryError),
> 
> Right. The code in the constructor can raise other exceptions that must
> also be handled properly. An explicit "except +" will handle that.

What about the declarations that we ship in libcpp.*? They currently lack
any such exception declarations. Can we safely add them where appropriate?

And, would someone care to do it?

Stefan

From stefan_ml at behnel.de  Tue Jul 17 10:58:53 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 17 Jul 2012 10:58:53 +0200
Subject: [Cython] failing memory view tests in Python 2.4/5
In-Reply-To: <CANg26EV3Ce3ixR=UzM9JSzRkZ6ZO7OR2FLTnjnkxMUQ8MNiV9g@mail.gmail.com>
References: <4FFBD8C6.6090708@behnel.de>
	<CANg26EV3Ce3ixR=UzM9JSzRkZ6ZO7OR2FLTnjnkxMUQ8MNiV9g@mail.gmail.com>
Message-ID: <5005294D.8080203@behnel.de>

mark florisson, 10.07.2012 11:20:
> On 10 July 2012 08:24, Stefan Behnel wrote:
>> I've fixed some of the errors that current block the tests. The remaining
>> failures in the memory view tests exist in both Py2.4 and 2.5, which hints
>> at a problem with the buffer support in numpy.pxd.
>>
>> https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests/BACKEND=c,PYVERSION=py25-ext/504/testReport/
>>
>> The memslice test failure is annoying. It looks like slice() fails to
>> properly deep-copy in Py2.[45]. I don't know if we can do anything about
>> this, seems more like a CPython bug. Deep copying is used to split function
>> definitions for fused types and I guess slice objects are used in
>> combination with the memory slices. So both don't currently work together
>> in Py2.[45].
> 
> I'm pretty sure I fixed the slice copying problem through a
> __deepcopy__ overload a long time ago elsewhere. I don't think these
> tests should be excluded from the test run yet.

Ok.


> Perhaps we should run
> the 2x tests from py27 down to 2.4. Tests in newer python's are less
> likely to fail.

Test failures are bad in general, though. Just because you hear about them
a couple of minutes later doesn't mean that they are less important.


>> I'd like to release a 0.17 alpha soon, so please get these sorted out in
>> one way or another ASAP. I'm also fine with declaring these features
>> unusable in older Python releases if we can't fix them, but it would
>> obviously be better to get them working.
> 
> I'll try to find some time this week.

Any news from this?

Stefan

From stefan_ml at behnel.de  Tue Jul 17 11:02:10 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 17 Jul 2012 11:02:10 +0200
Subject: [Cython] merging _refactor_indexnode branch (was: failing memory
 view tests in Python 2.4/5)
In-Reply-To: <CANg26EV3Ce3ixR=UzM9JSzRkZ6ZO7OR2FLTnjnkxMUQ8MNiV9g@mail.gmail.com>
References: <4FFBD8C6.6090708@behnel.de>
	<CANg26EV3Ce3ixR=UzM9JSzRkZ6ZO7OR2FLTnjnkxMUQ8MNiV9g@mail.gmail.com>
Message-ID: <50052A12.9020205@behnel.de>

mark florisson, 10.07.2012 11:20:
> Do you want _refactor_indexnode merged for this release?

I was ok with it when I first skipped over it. It would be good to give it
another pair of eyeballs, but apart from that, yes, I think it should go
in. I'll merge it as soon as Jenkins is happy with the test suite again
(and after re-enabling the broken tests, which are related after all), so
that we can see if everything is ok after the merge.

Stefan

From markflorisson88 at gmail.com  Tue Jul 17 12:15:05 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Tue, 17 Jul 2012 11:15:05 +0100
Subject: [Cython] failing memory view tests in Python 2.4/5
In-Reply-To: <5005294D.8080203@behnel.de>
References: <4FFBD8C6.6090708@behnel.de>
	<CANg26EV3Ce3ixR=UzM9JSzRkZ6ZO7OR2FLTnjnkxMUQ8MNiV9g@mail.gmail.com>
	<5005294D.8080203@behnel.de>
Message-ID: <CANg26EWRT6FYm9f-k44uigrS3LMfqorKOba_4y-nAym_U3bEog@mail.gmail.com>

On 17 July 2012 09:58, Stefan Behnel <stefan_ml at behnel.de> wrote:
> mark florisson, 10.07.2012 11:20:
>> On 10 July 2012 08:24, Stefan Behnel wrote:
>>> I've fixed some of the errors that current block the tests. The remaining
>>> failures in the memory view tests exist in both Py2.4 and 2.5, which hints
>>> at a problem with the buffer support in numpy.pxd.
>>>
>>> https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests/BACKEND=c,PYVERSION=py25-ext/504/testReport/
>>>
>>> The memslice test failure is annoying. It looks like slice() fails to
>>> properly deep-copy in Py2.[45]. I don't know if we can do anything about
>>> this, seems more like a CPython bug. Deep copying is used to split function
>>> definitions for fused types and I guess slice objects are used in
>>> combination with the memory slices. So both don't currently work together
>>> in Py2.[45].
>>
>> I'm pretty sure I fixed the slice copying problem through a
>> __deepcopy__ overload a long time ago elsewhere. I don't think these
>> tests should be excluded from the test run yet.
>
> Ok.
>
>
>> Perhaps we should run
>> the 2x tests from py27 down to 2.4. Tests in newer python's are less
>> likely to fail.
>
> Test failures are bad in general, though. Just because you hear about them
> a couple of minutes later doesn't mean that they are less important.
>

If tests fail in a lower version and not a higher version it often
means an easy to fix version incompatibility (although sometimes,
especially with buffers and it's special casing, this is really not
true). But the point is you know whether something is broken in
general, or it's a version related thing, and that information in
itself is useful.

>>> I'd like to release a 0.17 alpha soon, so please get these sorted out in
>>> one way or another ASAP. I'm also fine with declaring these features
>>> unusable in older Python releases if we can't fix them, but it would
>>> obviously be better to get them working.
>>
>> I'll try to find some time this week.
>
> Any news from this?

Sorry to be holding this up, I've been busy. I don't want to make
promises, but I'll try again for the next few days.

> Stefan
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

From markflorisson88 at gmail.com  Tue Jul 17 12:20:40 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Tue, 17 Jul 2012 11:20:40 +0100
Subject: [Cython] merging _refactor_indexnode branch (was: failing
 memory view tests in Python 2.4/5)
In-Reply-To: <50052A12.9020205@behnel.de>
References: <4FFBD8C6.6090708@behnel.de>
	<CANg26EV3Ce3ixR=UzM9JSzRkZ6ZO7OR2FLTnjnkxMUQ8MNiV9g@mail.gmail.com>
	<50052A12.9020205@behnel.de>
Message-ID: <CANg26EVK9JENirijG-Y1o2poHGKuLVOArSK4K8Sw_FoPRwpzZA@mail.gmail.com>

On 17 July 2012 10:02, Stefan Behnel <stefan_ml at behnel.de> wrote:
> mark florisson, 10.07.2012 11:20:
>> Do you want _refactor_indexnode merged for this release?
>
> I was ok with it when I first skipped over it. It would be good to give it
> another pair of eyeballs, but apart from that, yes, I think it should go
> in. I'll merge it as soon as Jenkins is happy with the test suite again
> (and after re-enabling the broken tests, which are related after all), so
> that we can see if everything is ok after the merge.
>
> Stefan
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

Hm, some things changed here. I rebased my current working branch
(_array_expressions) on that branch, and then I moved the buffer and
memoryview tests into their own directories in master, and I think I
rebased my devel branch along with _refactor_indexnode on master. So
I'll have to renew that pull request using the final rebase. I'll then
rebase the rest of my working branch on master.

From stefan_ml at behnel.de  Tue Jul 17 12:52:46 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 17 Jul 2012 12:52:46 +0200
Subject: [Cython] merging _refactor_indexnode branch
In-Reply-To: <CANg26EVK9JENirijG-Y1o2poHGKuLVOArSK4K8Sw_FoPRwpzZA@mail.gmail.com>
References: <4FFBD8C6.6090708@behnel.de>
	<CANg26EV3Ce3ixR=UzM9JSzRkZ6ZO7OR2FLTnjnkxMUQ8MNiV9g@mail.gmail.com>
	<50052A12.9020205@behnel.de>
	<CANg26EVK9JENirijG-Y1o2poHGKuLVOArSK4K8Sw_FoPRwpzZA@mail.gmail.com>
Message-ID: <500543FE.3070909@behnel.de>

mark florisson, 17.07.2012 12:20:
> On 17 July 2012 10:02, Stefan Behnel wrote:
>> mark florisson, 10.07.2012 11:20:
>>> Do you want _refactor_indexnode merged for this release?
>>
>> I was ok with it when I first skipped over it. It would be good to give it
>> another pair of eyeballs, but apart from that, yes, I think it should go
>> in. I'll merge it as soon as Jenkins is happy with the test suite again
>> (and after re-enabling the broken tests, which are related after all), so
>> that we can see if everything is ok after the merge.
>
> Hm, some things changed here. I rebased my current working branch
> (_array_expressions) on that branch, and then I moved the buffer and
> memoryview tests into their own directories in master, and I think I
> rebased my devel branch along with _refactor_indexnode on master. So
> I'll have to renew that pull request using the final rebase. I'll then
> rebase the rest of my working branch on master.

Ok, just take care to not overwrite it so that we keep the original comments.

Stefan

From stefan_ml at behnel.de  Tue Jul 17 13:12:27 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 17 Jul 2012 13:12:27 +0200
Subject: [Cython] failing memory view tests in Python 2.4/5
In-Reply-To: <CANg26EWRT6FYm9f-k44uigrS3LMfqorKOba_4y-nAym_U3bEog@mail.gmail.com>
References: <4FFBD8C6.6090708@behnel.de>
	<CANg26EV3Ce3ixR=UzM9JSzRkZ6ZO7OR2FLTnjnkxMUQ8MNiV9g@mail.gmail.com>
	<5005294D.8080203@behnel.de>
	<CANg26EWRT6FYm9f-k44uigrS3LMfqorKOba_4y-nAym_U3bEog@mail.gmail.com>
Message-ID: <5005489B.6020604@behnel.de>

mark florisson, 17.07.2012 12:15:
> On 17 July 2012 09:58, Stefan Behnel wrote:
>> mark florisson, 10.07.2012 11:20:
>>> Perhaps we should run
>>> the 2x tests from py27 down to 2.4. Tests in newer python's are less
>>> likely to fail.
>>
>> Test failures are bad in general, though. Just because you hear about them
>> a couple of minutes later doesn't mean that they are less important.
> 
> If tests fail in a lower version and not a higher version it often
> means an easy to fix version incompatibility

And also the other way round, but that's usually more related to Py3 bugs.


> (although sometimes,
> especially with buffers and it's special casing, this is really not
> true). But the point is you know whether something is broken in
> general, or it's a version related thing, and that information in
> itself is useful.

We currently run the tests for Py2.[47] and Py3.[23] in parallel before all
others. I think that should be enough to handle that case. However, if one
of them fails, the tests against the remaining Python versions will not be
triggered. I think it might be more helpful to always have all of them run,
so that it's clearer which of the release series are affected (just 2.4? or
2.4 and 2.5, as in this case?)

Stefan

From markflorisson88 at gmail.com  Tue Jul 17 13:18:43 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Tue, 17 Jul 2012 12:18:43 +0100
Subject: [Cython] failing memory view tests in Python 2.4/5
In-Reply-To: <5005489B.6020604@behnel.de>
References: <4FFBD8C6.6090708@behnel.de>
	<CANg26EV3Ce3ixR=UzM9JSzRkZ6ZO7OR2FLTnjnkxMUQ8MNiV9g@mail.gmail.com>
	<5005294D.8080203@behnel.de>
	<CANg26EWRT6FYm9f-k44uigrS3LMfqorKOba_4y-nAym_U3bEog@mail.gmail.com>
	<5005489B.6020604@behnel.de>
Message-ID: <CANg26EUw4Bd3GyVr5xQzFLwfBf36bK6geby7KOWy-2ZiUwkmjg@mail.gmail.com>

On 17 July 2012 12:12, Stefan Behnel <stefan_ml at behnel.de> wrote:
> mark florisson, 17.07.2012 12:15:
>> On 17 July 2012 09:58, Stefan Behnel wrote:
>>> mark florisson, 10.07.2012 11:20:
>>>> Perhaps we should run
>>>> the 2x tests from py27 down to 2.4. Tests in newer python's are less
>>>> likely to fail.
>>>
>>> Test failures are bad in general, though. Just because you hear about them
>>> a couple of minutes later doesn't mean that they are less important.
>>
>> If tests fail in a lower version and not a higher version it often
>> means an easy to fix version incompatibility
>
> And also the other way round, but that's usually more related to Py3 bugs.
>
>
>> (although sometimes,
>> especially with buffers and it's special casing, this is really not
>> true). But the point is you know whether something is broken in
>> general, or it's a version related thing, and that information in
>> itself is useful.
>
> We currently run the tests for Py2.[47] and Py3.[23] in parallel before all
> others. I think that should be enough to handle that case. However, if one
> of them fails, the tests against the remaining Python versions will not be
> triggered. I think it might be more helpful to always have all of them run,
> so that it's clearer which of the release series are affected (just 2.4? or
> 2.4 and 2.5, as in this case?)

Ah great, that's exactly what I had in mind :)

> Stefan
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

From lists at cheimes.de  Tue Jul 17 17:38:59 2012
From: lists at cheimes.de (Christian Heimes)
Date: Tue, 17 Jul 2012 17:38:59 +0200
Subject: [Cython] 2d buffer interface with aligned data
Message-ID: <50058713.9060705@cheimes.de>

Hello,

I'm the author of https://bitbucket.org/tiran/smc.freeimage , a Cython
wrapper of the FreeImage and LCMS libraries. FreeImage supports a broad
variety of image formats and pixel formats from standard RGB up to RGBAF
and complex numbers.

Now I like to add support for buffer interface to my code to support
zero-copy access of pixel data from NumPy. The wiki describes several
ways to access objects through the new and old Python buffer interface.
However I'm unable to find an example how to *implement* the buffer
interface.

Memory alignment makes the buffer interface more complicated, too.
FreeImage stores the raw pixels non-contiguously. The lines are memory
aligned (usually 32bit). For example a standard RGB image with 1 byte
per pixel and size of 2*3 pixels has a memory layout of

   BGRBGRxxBGRBGRxxBGRBGRxx

on a little-endian machine. 'xx' are the two alignment bytes.

Here is an excerpt from the FreeImage manual on how to access the raw
pixel data. The above example has width: 2, height: 3, pitch: 8 and bpp: 3.

unsigned width = FreeImage_GetWidth(dib);   // 2
unsigned height = FreeImage_GetHeight(dib); // 3
unsigned pitch = FreeImage_GetPitch(dib);   // 8
unsigned bpp = FreeImage_GetBPP(dib);       // 3

BYTE *bits = (BYTE*)FreeImage_GetBits(dib);
for(y = 0; y < height; y++) {
    BYTE *pixel = (BYTE*)bits;
    for(x = 0; x < width; x++) {
        pixel[FI_RGBA_RED] = 128;
        pixel[FI_RGBA_GREEN] = 128;
        pixel[FI_RGBA_BLUE] = 128;
        pixel += bpp;
    }
    // next line
    bits += pitch;
}

Please help,
Christian

From d.s.seljebotn at astro.uio.no  Tue Jul 17 18:56:21 2012
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Tue, 17 Jul 2012 18:56:21 +0200
Subject: [Cython] 2d buffer interface with aligned data
In-Reply-To: <5005990C.8000405@astro.uio.no>
References: <50058713.9060705@cheimes.de> <5005990C.8000405@astro.uio.no>
Message-ID: <50059935.9080902@astro.uio.no>

On 07/17/2012 06:55 PM, Dag Sverre Seljebotn wrote:
> On 07/17/2012 05:38 PM, Christian Heimes wrote:
>> Hello,
>>
>> I'm the author of https://bitbucket.org/tiran/smc.freeimage , a Cython
>> wrapper of the FreeImage and LCMS libraries. FreeImage supports a broad
>> variety of image formats and pixel formats from standard RGB up to RGBAF
>> and complex numbers.
>>
>> Now I like to add support for buffer interface to my code to support
>> zero-copy access of pixel data from NumPy. The wiki describes several
>> ways to access objects through the new and old Python buffer interface.
>> However I'm unable to find an example how to *implement* the buffer
>> interface.
>
> Read PEP 3118. Then implement __getbuffer__ and __releasebuffer__ in
> your cdef class (don't know if it's documented but you can see example
> in tests/run/buffer.pyx).
>
>
>>
>> Memory alignment makes the buffer interface more complicated, too.
>> FreeImage stores the raw pixels non-contiguously. The lines are memory
>> aligned (usually 32bit). For example a standard RGB image with 1 byte
>> per pixel and size of 2*3 pixels has a memory layout of
>>
>> BGRBGRxxBGRBGRxxBGRBGRxx
>>
>> on a little-endian machine. 'xx' are the two alignment bytes.
>
> This is easily supported; above you would let
>
> ndim = 2
> strides = [8, 1]
> shape = [2, 3]

Sorry; shape = [3, 2] to match your example.

Dag

> itemsize = 1
>
> The alignment bytes are skipped simply because shape[0] * itemsize <
> strides[0].
>
> Dag
>
>>
>> Here is an excerpt from the FreeImage manual on how to access the raw
>> pixel data. The above example has width: 2, height: 3, pitch: 8 and
>> bpp: 3.
>>
>> unsigned width = FreeImage_GetWidth(dib); // 2
>> unsigned height = FreeImage_GetHeight(dib); // 3
>> unsigned pitch = FreeImage_GetPitch(dib); // 8
>> unsigned bpp = FreeImage_GetBPP(dib); // 3
>>
>> BYTE *bits = (BYTE*)FreeImage_GetBits(dib);
>> for(y = 0; y< height; y++) {
>> BYTE *pixel = (BYTE*)bits;
>> for(x = 0; x< width; x++) {
>> pixel[FI_RGBA_RED] = 128;
>> pixel[FI_RGBA_GREEN] = 128;
>> pixel[FI_RGBA_BLUE] = 128;
>> pixel += bpp;
>> }
>> // next line
>> bits += pitch;
>> }
>>
>> Please help,
>> Christian
>> _______________________________________________
>> cython-devel mailing list
>> cython-devel at python.org
>> http://mail.python.org/mailman/listinfo/cython-devel
>


From d.s.seljebotn at astro.uio.no  Tue Jul 17 18:55:40 2012
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Tue, 17 Jul 2012 18:55:40 +0200
Subject: [Cython] 2d buffer interface with aligned data
In-Reply-To: <50058713.9060705@cheimes.de>
References: <50058713.9060705@cheimes.de>
Message-ID: <5005990C.8000405@astro.uio.no>

On 07/17/2012 05:38 PM, Christian Heimes wrote:
> Hello,
>
> I'm the author of https://bitbucket.org/tiran/smc.freeimage , a Cython
> wrapper of the FreeImage and LCMS libraries. FreeImage supports a broad
> variety of image formats and pixel formats from standard RGB up to RGBAF
> and complex numbers.
>
> Now I like to add support for buffer interface to my code to support
> zero-copy access of pixel data from NumPy. The wiki describes several
> ways to access objects through the new and old Python buffer interface.
> However I'm unable to find an example how to *implement* the buffer
> interface.

Read PEP 3118. Then implement __getbuffer__ and __releasebuffer__ in 
your cdef class (don't know if it's documented but you can see example 
in tests/run/buffer.pyx).


>
> Memory alignment makes the buffer interface more complicated, too.
> FreeImage stores the raw pixels non-contiguously. The lines are memory
> aligned (usually 32bit). For example a standard RGB image with 1 byte
> per pixel and size of 2*3 pixels has a memory layout of
>
>     BGRBGRxxBGRBGRxxBGRBGRxx
>
> on a little-endian machine. 'xx' are the two alignment bytes.

This is easily supported; above you would let

ndim = 2
strides = [8, 1]
shape = [2, 3]
itemsize = 1

The alignment bytes are skipped simply because shape[0] * itemsize < 
strides[0].

Dag

>
> Here is an excerpt from the FreeImage manual on how to access the raw
> pixel data. The above example has width: 2, height: 3, pitch: 8 and bpp: 3.
>
> unsigned width = FreeImage_GetWidth(dib);   // 2
> unsigned height = FreeImage_GetHeight(dib); // 3
> unsigned pitch = FreeImage_GetPitch(dib);   // 8
> unsigned bpp = FreeImage_GetBPP(dib);       // 3
>
> BYTE *bits = (BYTE*)FreeImage_GetBits(dib);
> for(y = 0; y<  height; y++) {
>      BYTE *pixel = (BYTE*)bits;
>      for(x = 0; x<  width; x++) {
>          pixel[FI_RGBA_RED] = 128;
>          pixel[FI_RGBA_GREEN] = 128;
>          pixel[FI_RGBA_BLUE] = 128;
>          pixel += bpp;
>      }
>      // next line
>      bits += pitch;
> }
>
> Please help,
> Christian
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel


From lists at cheimes.de  Tue Jul 17 22:55:10 2012
From: lists at cheimes.de (Christian Heimes)
Date: Tue, 17 Jul 2012 22:55:10 +0200
Subject: [Cython] 2d buffer interface with aligned data
In-Reply-To: <5005990C.8000405@astro.uio.no>
References: <50058713.9060705@cheimes.de> <5005990C.8000405@astro.uio.no>
Message-ID: <5005D12E.2080800@cheimes.de>

Am 17.07.2012 18:55, schrieb Dag Sverre Seljebotn:
> Read PEP 3118. Then implement __getbuffer__ and __releasebuffer__ in
> your cdef class (don't know if it's documented but you can see example
> in tests/run/buffer.pyx).

The new buffer interface from PEP 3118 is only available for Python 2.6
and newer. I was hoping for some abstraction layer in Cython. Well, I
don't have to support Python 2.5 and older. Thanks for the hint!


> This is easily supported; above you would let
> 
> ndim = 2
> strides = [8, 1]
> shape = [2, 3]
> itemsize = 1
> 
> The alignment bytes are skipped simply because shape[0] * itemsize <
> strides[0].

Either I'm doing something wrong or I found a Cython bug. I've attached
two files. The output is unexpected and looks like something is
accessing uninitialized memory:

format BBB
itemsize 3
ndim 2
readonly True
shape (140704200676960L, 140704199917920L)
strides (2L, 140704196619665L)
suboffsets None
len 140704200677056


Cython: 0.16
Python: 2.7.3 on 64bit Linux

Christian
-------------- next part --------------
# buffertest.pyx
cimport cpython

cdef char *s = b"BGRBGRxxBGRBGRxxBGRBGRxx"

cdef class Buffertest:
    def __getbuffer__(self, cpython.Py_buffer* buffer, int flags):

        buffer.buf = <char*>s
        buffer.obj = self
        buffer.len = len(s)
        buffer.readonly = 1
        buffer.format = "BBB"
        buffer.ndim = 2
        buffer.shape = [3, 2]
        buffer.strides = [8, 1] 
        buffer.suboffsets = NULL
        buffer.itemsize = 3
        buffer.internal = NULL

-------------- next part --------------
A non-text attachment was scrubbed...
Name: setup.py
Type: text/x-python
Size: 526 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20120717/97bc214a/attachment.py>

From lists at onerussian.com  Tue Jul 17 23:41:34 2012
From: lists at onerussian.com (Yaroslav Halchenko)
Date: Tue, 17 Jul 2012 17:41:34 -0400
Subject: [Cython] known? 0.16 on varioius platforms: ValueError: Buffer
 dtype mismatch, expected 'char' but got 'unsigned char' in ...
Message-ID: <20120717214134.GE5788@onerussian.com>

Hi everyone!

1. uploaded 0.16 to Debian experimental... many platforms fail with smth
like

,--
| ======================================================================
| FAIL: Doctest: memslice.__test__.test_padded_structs
| ----------------------------------------------------------------------
| Traceback (most recent call last):
|   File "/usr/lib/python2.6/doctest.py", line 2163, in runTest
|     raise self.failureException(self.format_failure(new.getvalue()))
| AssertionError: Failed doctest test for memslice.__test__.test_padded_structs
|   File "/build/buildd-cython_0.16-1-armhf-FpFrv7/cython-0.16/build/work-dir/run/cpp/memslice.so", line unknown line number, in test_padded_structs
|
| ----------------------------------------------------------------------
| File "/build/buildd-cython_0.16-1-armhf-FpFrv7/cython-0.16/build/work-dir/run/cpp/memslice.so", line ?, in memslice.__test__.test_padded_structs
| Failed example:
|     test_padded_structs()
| Exception raised:
|     Traceback (most recent call last):
|       File "/usr/lib/python2.6/doctest.py", line 1253, in __run
|         compileflags, 1) in test.globs
|       File "<doctest memslice.__test__.test_padded_structs[0]>", line 1, in <module>
|         test_padded_structs()
|       File "memslice.pyx", line 42, in memslice.testcase.wrapper (memslice.cpp:3729)
|       File "memslice.pyx", line 1766, in memslice.test_padded_structs (memslice.cpp:27137)
|       File "memslice.pyx", line 1779, in memslice._test_padded (memslice.cpp:27284)
|     ValueError: Buffer dtype mismatch, expected 'char' but got 'unsigned char' in 'ArrayStruct.chars'
`---

more here:
https://buildd.debian.org/status/package.php?p=cython&suite=experimental

I have tested on a sparc box and current master seems to not show this
particular problem (see below though) -- does anyone remember this
particular issue and commit which fixed it?

2. related question -- what is the way with runtests.py to run a particular
unittest? not the full suite (e.g. my specifying memslice in cmdline), but this
particular one, e.g.:

,---
| (sid)yoh at vagus:~/deb/cython/cython$ PYTHONPATH=$PWD/install/lib/python2.7/site-packages/ python runtests.py --no-refnanny -v -v --exclude="parallel" --work-dir=build/work-dir memslice.__test__.test_padded_structs
| Python 2.7.3 (default, Jun 18 2012, 19:31:31)
| [GCC 4.6.3]
|
| Running tests against Cython 0.17pre b0a428d4d438b1ce3e1f5d805b09899377d1ecb5
| Backends: c,cpp
|
|
| ----------------------------------------------------------------------
| Ran 0 tests in 0.001s
|
| OK
| ALL DONE
`---
so I could bisect reasonably fast whenever needed... ?

3. FWIW there are some similar and novel failures on a sparc box on current master:
http://www.onerussian.com/tmp/tests-cython-sparc-0.16rc1-490-gb0a428d.log

I wondered -- do you have any brief instructions on how to setup a local
build bot to contribute back to the cython's Jenkins setup?  Then I could
put a sparc to provide you with CI updates for this platform -- or do you have
a similar one already?


-- 
Yaroslav O. Halchenko
Postdoctoral Fellow,   Department of Psychological and Brain Sciences
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834                       Fax: +1 (603) 646-1419
WWW:   http://www.linkedin.com/in/yarik        

From njs at pobox.com  Wed Jul 18 01:53:42 2012
From: njs at pobox.com (Nathaniel Smith)
Date: Wed, 18 Jul 2012 00:53:42 +0100
Subject: [Cython] 2d buffer interface with aligned data
In-Reply-To: <5005D12E.2080800@cheimes.de>
References: <50058713.9060705@cheimes.de> <5005990C.8000405@astro.uio.no>
	<5005D12E.2080800@cheimes.de>
Message-ID: <CAPJVwBmhRRsk0rLgMQjzPFRVc+YmCJ_T6iRTjuRQO7H+5UDktg@mail.gmail.com>

On Tue, Jul 17, 2012 at 9:55 PM, Christian Heimes <lists at cheimes.de> wrote:
> Am 17.07.2012 18:55, schrieb Dag Sverre Seljebotn:
>> Read PEP 3118. Then implement __getbuffer__ and __releasebuffer__ in
>> your cdef class (don't know if it's documented but you can see example
>> in tests/run/buffer.pyx).
>
> The new buffer interface from PEP 3118 is only available for Python 2.6
> and newer. I was hoping for some abstraction layer in Cython. Well, I
> don't have to support Python 2.5 and older. Thanks for the hint!

If you're worried about supporting older Python versions then the
simplest thing is probably to just implement the __array_interface__
or __array_struct__ interface. They're pretty trivial, and while not
as standards-compliant and Correct as PEP3118, they'll work just fine
with numpy across any version of Python. (Of course you can also
implement both PEP3118 and __array_struct__ together.)

http://docs.scipy.org/doc/numpy/reference/arrays.interface.html

-n

From stefan_ml at behnel.de  Wed Jul 18 07:18:34 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 18 Jul 2012 07:18:34 +0200
Subject: [Cython] 2d buffer interface with aligned data
In-Reply-To: <5005D12E.2080800@cheimes.de>
References: <50058713.9060705@cheimes.de> <5005990C.8000405@astro.uio.no>
	<5005D12E.2080800@cheimes.de>
Message-ID: <5006472A.6090505@behnel.de>

Christian Heimes, 17.07.2012 22:55:
> Am 17.07.2012 18:55, schrieb Dag Sverre Seljebotn:
>> Read PEP 3118. Then implement __getbuffer__ and __releasebuffer__ in
>> your cdef class (don't know if it's documented but you can see example
>> in tests/run/buffer.pyx).
> 
> The new buffer interface from PEP 3118 is only available for Python 2.6
> and newer. I was hoping for some abstraction layer in Cython. Well, I
> don't have to support Python 2.5 and older. Thanks for the hint!

Cython supports the buffer interface also in older Python versions as long
as you implement your buffer methods in a .pxd file as part of the
extension type declaration and cimport from that in the modules where you
use it. It's a pure Cython feature though, other extensions (and Python
code, obviously) won't see it.

BTW, note that your question is better suited for the Cython users mailing
list than the core developer mailing list.

Stefan

From stefan_ml at behnel.de  Wed Jul 18 09:09:06 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 18 Jul 2012 09:09:06 +0200
Subject: [Cython] code generated for arguments of type "type"
Message-ID: <50066112.4020800@behnel.de>

Hi,

the following pull request tries to fix the fact that we currently generate
arguments of type "PyObject*" when someone spells out "type" for them.

https://github.com/cython/cython/pull/126

That's mostly a problem in CPython's C-API, but also in some NumPy
functions. The correct way to translate this would be as a "PyTypeObject*",
not a bare "PyObject*".

Does anyone see a general problem with changing this in the compiler
directly? I'm not sure that it won't break code, for example. We'll at
least have to cast the arguments on the way in as well.

Stefan

From lists at cheimes.de  Wed Jul 18 11:55:14 2012
From: lists at cheimes.de (Christian Heimes)
Date: Wed, 18 Jul 2012 11:55:14 +0200
Subject: [Cython] 2d buffer interface with aligned data
In-Reply-To: <5005D12E.2080800@cheimes.de>
References: <50058713.9060705@cheimes.de> <5005990C.8000405@astro.uio.no>
	<5005D12E.2080800@cheimes.de>
Message-ID: <50068802.2050909@cheimes.de>

Am 17.07.2012 22:55, schrieb Christian Heimes:
> Either I'm doing something wrong or I found a Cython bug. I've attached
> two files. The output is unexpected and looks like something is
> accessing uninitialized memory:

For the record:
It's my fault. The shade and stripes Py_ssize_t* arrays can't be local
variables. I've to malloc() two arrays and free() them in
__releasebuffer__(). Cython is so easy to use, it makes me forget all
this little annoying C things.

Christian


From lists at cheimes.de  Wed Jul 18 11:55:42 2012
From: lists at cheimes.de (Christian Heimes)
Date: Wed, 18 Jul 2012 11:55:42 +0200
Subject: [Cython] 2d buffer interface with aligned data
In-Reply-To: <5006472A.6090505@behnel.de>
References: <50058713.9060705@cheimes.de> <5005990C.8000405@astro.uio.no>
	<5005D12E.2080800@cheimes.de> <5006472A.6090505@behnel.de>
Message-ID: <5006881E.4030403@cheimes.de>

Am 18.07.2012 07:18, schrieb Stefan Behnel:
> BTW, note that your question is better suited for the Cython users mailing
> list than the core developer mailing list.

I'm sorry. I didn't know Cython had a user list, too. Thanks Stefan!

Christian

From d.s.seljebotn at astro.uio.no  Wed Jul 18 12:24:41 2012
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Wed, 18 Jul 2012 12:24:41 +0200
Subject: [Cython] 2d buffer interface with aligned data
In-Reply-To: <50068802.2050909@cheimes.de>
References: <50058713.9060705@cheimes.de> <5005990C.8000405@astro.uio.no>
	<5005D12E.2080800@cheimes.de> <50068802.2050909@cheimes.de>
Message-ID: <e6d8b087-751c-4455-9114-e2965bb67dc3@email.android.com>

If the image will always be 2D, you can add

cdef Py_ssize_t shape[2]

To your cdef class and assign self.shape to the Py_buffer. This is a bit more efficient.

Dag
-- 
Sent from my Android phone with K-9 Mail. Please excuse my brevity.

Christian Heimes <lists at cheimes.de> wrote:

Am 17.07.2012 22:55, schrieb Christian Heimes:
> Either I'm doing something wrong or I found a Cython bug. I've attached
> two files. The output is unexpected and looks like something is
> accessing uninitialized memory:

For the record:
It's my fault. The shade and stripes Py_ssize_t* arrays can't be local
variables. I've to malloc() two arrays and free() them in
__releasebuffer__(). Cython is so easy to use, it makes me forget all
this little annoying C things.

Christian

_____________________________________________

cython-devel mailing list
cython-devel at python.org
http://mail.python.org/mailman/listinfo/cython-devel

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20120718/a4c2a427/attachment.html>

From lists at onerussian.com  Thu Jul 19 04:26:23 2012
From: lists at onerussian.com (Yaroslav Halchenko)
Date: Wed, 18 Jul 2012 22:26:23 -0400
Subject: [Cython] known? 0.16 on varioius platforms: ValueError: Buffer
 dtype mismatch, expected 'char' but got 'unsigned char' in ...
In-Reply-To: <20120717214134.GE5788@onerussian.com>
References: <20120717214134.GE5788@onerussian.com>
Message-ID: <20120719022623.GG5788@onerussian.com>


On Tue, 17 Jul 2012, Yaroslav Halchenko wrote:
> I have tested on a sparc box and current master seems to not show this
> particular problem (see below though) -- does anyone remember this
> particular issue and commit which fixed it?

I am sorry -- I got lost among architectures and what fails where and possibly
misguided you in my original report -- current master does still have this
problem -- this time testing on s390x (it does pass on sparc):

(s390x-sid)yoh at zelenka:~/cython/cython$ python runtests.py -vv memoryview_compare_type_pointers
Python 2.7.3 (default, Jul 14 2012, 05:19:55) 
[GCC 4.6.3]

Running tests against Cython 0.17pre 0cbc0f4398cb0edbc7a128cfb2b72567f2e5e217
/home/yoh/cython/cython/BUILD/support/temp.linux-s390x-2.7/pyrex/refnanny.c: In function ?__pyx_f_8refnanny_report_unraisable?:
/home/yoh/cython/cython/BUILD/support/temp.linux-s390x-2.7/pyrex/refnanny.c:2309:7: warning: variable ?__pyx_clineno? set but not used [-Wunused-but-set-variable]
/home/yoh/cython/cython/BUILD/support/temp.linux-s390x-2.7/pyrex/refnanny.c:2308:15: warning: variable ?__pyx_filename? set but not used [-Wunused-but-set-variable]
/home/yoh/cython/cython/BUILD/support/temp.linux-s390x-2.7/pyrex/refnanny.c:2307:7: warning: variable ?__pyx_lineno? set but not used [-Wunused-but-set-variable]
Backends: c,cpp

runTest (__main__.EndToEndTest)
End-to-end memoryview_compare_type_pointers ... /usr/bin/python setup.py build_ext --inplace
Compiling test_compare_type_pointers.pyx because it changed.
Compiling other_module.pyx because it changed.
Cythonizing other_module.pyx
Cythonizing test_compare_type_pointers.pyx
running build_ext
building 'test_compare_type_pointers' extension
creating build
creating build/temp.linux-s390x-2.7
gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/python2.7 -c test_compare_type_pointers.c -o build/temp.linux-s390x-2.7/test_compare_type_pointers.o
gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-z,relro build/temp.linux-s390x-2.7/test_compare_type_pointers.o -o /home/yoh/cython/cython/BUILD/memoryview/memoryview_compare_type_pointers/test_compare_type_pointers.so
building 'other_module' extension
gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/python2.7 -c other_module.c -o build/temp.linux-s390x-2.7/other_module.o
gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-z,relro build/temp.linux-s390x-2.7/other_module.o -o /home/yoh/cython/cython/BUILD/memoryview/memoryview_compare_type_pointers/other_module.so

test_compare_type_pointers.c:1040:1: warning: useless type name in empty declaration [enabled by default]
test_compare_type_pointers.c:1042:1: warning: useless type name in empty declaration [enabled by default]
other_module.c:1038:1: warning: useless type name in empty declaration [enabled by default]
other_module.c:1040:1: warning: useless type name in empty declaration [enabled by default]
other_module.c:1044:1: warning: useless type name in empty declaration [enabled by default]
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "test_compare_type_pointers.pyx", line 3, in init test_compare_type_pointers (test_compare_type_pointers.c:13542)
    import other_module
  File "other_module.pyx", line 4, in init other_module (other_module.c:13350)
    cdef Foo[:] fooview = fooarray
ValueError: Buffer dtype mismatch, expected 'char' but got 'unsigned char' in 'Foo.c'

FAIL

======================================================================
FAIL: runTest (__main__.EndToEndTest)
End-to-end memoryview_compare_type_pointers
----------------------------------------------------------------------
Traceback (most recent call last):
  File "runtests.py", line 1100, in runTest
    self.assertEqual(0, res, "non-zero exit status")
AssertionError: non-zero exit status

----------------------------------------------------------------------
Ran 1 test in 11.070s

FAILED (failures=1)
ALL DONE

-- 
Yaroslav O. Halchenko
Postdoctoral Fellow,   Department of Psychological and Brain Sciences
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834                       Fax: +1 (603) 646-1419
WWW:   http://www.linkedin.com/in/yarik        

From stefan_ml at behnel.de  Thu Jul 19 12:33:52 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 19 Jul 2012 12:33:52 +0200
Subject: [Cython] Odd behavior with std::string and .decode()
In-Reply-To: <4FF66E25.3070808@behnel.de>
References: <20120705182919.5d526c85@resist.wooz.org>
	<4FF66E25.3070808@behnel.de>
Message-ID: <5007E290.7080203@behnel.de>

Stefan Behnel, 06.07.2012 06:48:
> One thing I would generally suggest is to do this:
> 
>     descr = self._this.get_description()
>     return descr.data()[:descr.size()].decode('utf-8')
> 
> Avoids the call to strlen() by explicitly slicing the pointer. Also avoids
> needing to make sure the C string is 0-terminated.
> 
> Barry Warsaw, 06.07.2012 00:29:
>> I looked at the generated code in the first example, but didn't really see
>> anything obvious.  There are no NULs in the char* description afaict.  I
>> haven't yet tested Cython 0.16 or 0.17 to see if this behaves differently.
> 
> I wouldn't know any differences out of the top of my head, except that 0.17
> has generally better support for STL containers and std:string (but that's
> unrelated to this failure). I'm planning to enable direct support for
> cpp_string.decode(...) as well, but that's not implemented yet. It would
> basically generate the verbose code above automatically.

Done.

https://github.com/cython/cython/commit/bf702727e1bb129e9d74548625658a224ce9e30d

https://sage.math.washington.edu:8091/hudson/job/cython-docs/doclinks/1/src/tutorial/strings.html#decoding-bytes-to-text

Stefan

From stefan_ml at behnel.de  Thu Jul 19 21:19:58 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 19 Jul 2012 21:19:58 +0200
Subject: [Cython] [cython-users] C++: how to handle failures of 'new'?
In-Reply-To: <500300C6.2040302@behnel.de>
References: <4FECA49B.8090404@behnel.de>
	<CADiQ+QBeBw=FYsNPpSU_OJOU48m7E1=csxMxfenMQLs2jhJgPw@mail.gmail.com>
	<4FED40E1.1040301@behnel.de> <500300C6.2040302@behnel.de>
Message-ID: <50085DDE.1090104@behnel.de>

Stefan Behnel, 15.07.2012 19:41:
> Stefan Behnel, 29.06.2012 07:45:
>> [moving this to cython-devel as it's getting technical]
>>
>> Robert Bradshaw, 28.06.2012 21:46:
>>> On Thu, Jun 28, 2012 at 11:38 AM, Stefan Behnel wrote:
>>>> currently, when I write "new CppClass()" in Cython, it generates a straight
>>>> call to the "new" operator. It doesn't do any error handling. And the
>>>> current documentation doesn't even mention this case.
>>>>
>>>> Is there a "standard" way to handle this? It seems that C++ has different
>>>> ways to deal with failures here but raises an exception by default. Would
>>>> you declare the constructor(s) with an "except +MemoryError"? Is there a
>>>> reason Cython shouldn't be doing this automatically (if nothing else was
>>>> declared) ?
>>>
>>> I think it certainly makes sense to declare the default constructor as
>>> "except +" (and std::bad_alloc should become MemoryError),
>>
>> Right. The code in the constructor can raise other exceptions that must
>> also be handled properly. An explicit "except +" will handle that.
> 
> What about the declarations that we ship in libcpp.*? They currently lack
> any such exception declarations. Can we safely add them where appropriate?
> 
> And, would someone care to do it?

Maybe the question at the end just scares off too many potential repliers.

May I assume that the answer to the first paragraph is "yes"? Or does
anyone see problems with changing that now? My reading is that it would
(above all) fix user code that's currently broken.

Stefan

From robertwb at gmail.com  Thu Jul 19 21:29:39 2012
From: robertwb at gmail.com (Robert Bradshaw)
Date: Thu, 19 Jul 2012 12:29:39 -0700
Subject: [Cython] [cython-users] C++: how to handle failures of 'new'?
In-Reply-To: <50085DDE.1090104@behnel.de>
References: <4FECA49B.8090404@behnel.de>
	<CADiQ+QBeBw=FYsNPpSU_OJOU48m7E1=csxMxfenMQLs2jhJgPw@mail.gmail.com>
	<4FED40E1.1040301@behnel.de> <500300C6.2040302@behnel.de>
	<50085DDE.1090104@behnel.de>
Message-ID: <CADiQ+QDoALu_4zVeiFckbCkyHPAKHs81VSzPcBA_K5_cS=5FEg@mail.gmail.com>

On Thu, Jul 19, 2012 at 12:19 PM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Stefan Behnel, 15.07.2012 19:41:
>> Stefan Behnel, 29.06.2012 07:45:
>>> [moving this to cython-devel as it's getting technical]
>>>
>>> Robert Bradshaw, 28.06.2012 21:46:
>>>> On Thu, Jun 28, 2012 at 11:38 AM, Stefan Behnel wrote:
>>>>> currently, when I write "new CppClass()" in Cython, it generates a straight
>>>>> call to the "new" operator. It doesn't do any error handling. And the
>>>>> current documentation doesn't even mention this case.
>>>>>
>>>>> Is there a "standard" way to handle this? It seems that C++ has different
>>>>> ways to deal with failures here but raises an exception by default. Would
>>>>> you declare the constructor(s) with an "except +MemoryError"? Is there a
>>>>> reason Cython shouldn't be doing this automatically (if nothing else was
>>>>> declared) ?
>>>>
>>>> I think it certainly makes sense to declare the default constructor as
>>>> "except +" (and std::bad_alloc should become MemoryError),
>>>
>>> Right. The code in the constructor can raise other exceptions that must
>>> also be handled properly. An explicit "except +" will handle that.
>>
>> What about the declarations that we ship in libcpp.*? They currently lack
>> any such exception declarations. Can we safely add them where appropriate?
>>
>> And, would someone care to do it?
>
> Maybe the question at the end just scares off too many potential repliers.
>
> May I assume that the answer to the first paragraph is "yes"? Or does
> anyone see problems with changing that now? My reading is that it would
> (above all) fix user code that's currently broken.

My thoughts are that we can (and should) safely

(1) Add these exception declarations to our shipped pxd files, and
(2) Let implicit constructors be declared as "except +MemoryError."

If no one beats me too it, I suppose I could, but it's an easy bug for
anyone to knock off.

- Robert

From stefan_ml at behnel.de  Thu Jul 19 21:43:49 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 19 Jul 2012 21:43:49 +0200
Subject: [Cython] [cython-users] C++: how to handle failures of 'new'?
In-Reply-To: <CADiQ+QDoALu_4zVeiFckbCkyHPAKHs81VSzPcBA_K5_cS=5FEg@mail.gmail.com>
References: <4FECA49B.8090404@behnel.de>
	<CADiQ+QBeBw=FYsNPpSU_OJOU48m7E1=csxMxfenMQLs2jhJgPw@mail.gmail.com>
	<4FED40E1.1040301@behnel.de> <500300C6.2040302@behnel.de>
	<50085DDE.1090104@behnel.de>
	<CADiQ+QDoALu_4zVeiFckbCkyHPAKHs81VSzPcBA_K5_cS=5FEg@mail.gmail.com>
Message-ID: <50086375.5010709@behnel.de>

Robert Bradshaw, 19.07.2012 21:29:
> On Thu, Jul 19, 2012 at 12:19 PM, Stefan Behnel wrote:
>> Stefan Behnel, 15.07.2012 19:41:
>>> Stefan Behnel, 29.06.2012 07:45:
>>>> Robert Bradshaw, 28.06.2012 21:46:
>>>>> I think it certainly makes sense to declare the default constructor as
>>>>> "except +" (and std::bad_alloc should become MemoryError),
>>>>
>>>> Right. The code in the constructor can raise other exceptions that must
>>>> also be handled properly. An explicit "except +" will handle that.
>>>
>>> What about the declarations that we ship in libcpp.*? They currently lack
>>> any such exception declarations. Can we safely add them where appropriate?
>
> My thoughts are that we can (and should) safely
> 
> (1) Add these exception declarations to our shipped pxd files, and

Ok.

> (2) Let implicit constructors be declared as "except +MemoryError."

We don't know if they are really just "implicit constructors" in the C++
code or if the user just left them out of the Cython declarations. So they
may really exist in the C++ code and may raise other exceptions than just
bad_alloc. Would you consider that a user error and ignore it? I wouldn't
mind, bad declarations give bad code. A MemoryError may be rather
misleading in cases, but it's more forgiving than a crash.

Is there an actual advantage in only declaring one exception here?

I recently implemented it as "except +".


> If no one beats me too it, I suppose I could, but it's an easy bug for
> anyone to knock off.

Absolutely, a nice way of giving back to the project for anyone who can
read the STL documentation of C++.

Stefan

From robertwb at gmail.com  Thu Jul 19 21:48:45 2012
From: robertwb at gmail.com (Robert Bradshaw)
Date: Thu, 19 Jul 2012 12:48:45 -0700
Subject: [Cython] [cython-users] C++: how to handle failures of 'new'?
In-Reply-To: <50086375.5010709@behnel.de>
References: <4FECA49B.8090404@behnel.de>
	<CADiQ+QBeBw=FYsNPpSU_OJOU48m7E1=csxMxfenMQLs2jhJgPw@mail.gmail.com>
	<4FED40E1.1040301@behnel.de> <500300C6.2040302@behnel.de>
	<50085DDE.1090104@behnel.de>
	<CADiQ+QDoALu_4zVeiFckbCkyHPAKHs81VSzPcBA_K5_cS=5FEg@mail.gmail.com>
	<50086375.5010709@behnel.de>
Message-ID: <CADiQ+QBM8nSKK=KhsM5RX-Ez0r_HRNmQo4gSytUEz2FazY-WKw@mail.gmail.com>

On Thu, Jul 19, 2012 at 12:43 PM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Robert Bradshaw, 19.07.2012 21:29:
>> On Thu, Jul 19, 2012 at 12:19 PM, Stefan Behnel wrote:
>>> Stefan Behnel, 15.07.2012 19:41:
>>>> Stefan Behnel, 29.06.2012 07:45:
>>>>> Robert Bradshaw, 28.06.2012 21:46:
>>>>>> I think it certainly makes sense to declare the default constructor as
>>>>>> "except +" (and std::bad_alloc should become MemoryError),
>>>>>
>>>>> Right. The code in the constructor can raise other exceptions that must
>>>>> also be handled properly. An explicit "except +" will handle that.
>>>>
>>>> What about the declarations that we ship in libcpp.*? They currently lack
>>>> any such exception declarations. Can we safely add them where appropriate?
>>
>> My thoughts are that we can (and should) safely
>>
>> (1) Add these exception declarations to our shipped pxd files, and
>
> Ok.
>
>> (2) Let implicit constructors be declared as "except +MemoryError."
>
> We don't know if they are really just "implicit constructors" in the C++
> code or if the user just left them out of the Cython declarations. So they
> may really exist in the C++ code and may raise other exceptions than just
> bad_alloc. Would you consider that a user error and ignore it? I wouldn't
> mind, bad declarations give bad code. A MemoryError may be rather
> misleading in cases, but it's more forgiving than a crash.
>
> Is there an actual advantage in only declaring one exception here?
>
> I recently implemented it as "except +".

+1

>> If no one beats me too it, I suppose I could, but it's an easy bug for
>> anyone to knock off.
>
> Absolutely, a nice way of giving back to the project for anyone who can
> read the STL documentation of C++.
>
> Stefan
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

From markflorisson88 at gmail.com  Sat Jul 21 20:48:36 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Sat, 21 Jul 2012 19:48:36 +0100
Subject: [Cython] planning for 0.17
In-Reply-To: <4FEA1D63.5090902@behnel.de>
References: <4FEA1D63.5090902@behnel.de>
Message-ID: <CANg26EW=sg6+0x79psRqwuwXSqAoVTqhUkOtyseEi-ecAVKQMA@mail.gmail.com>

On 26 June 2012 21:36, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Hi,
>
> I'd like to get an idea of what's still open for 0.17.
>
> Mark mentioned some open memoryview issues on his list and I know that
> there are still issues with PyPy, some of which could get fixed in a
> reasonable time frame. Also, Jenkins isn't all that happy yet.
>
> https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests/

Jenkins is blue now, sorry for the holdup. I also updated the release
notes, are all contributors already in there?

> What's the current state of the master branch for everyone? Anything that
> you're working on and/or that you think should go in but isn't yet?
>
> I would like to see 0.17 released some time next month, if possible. I
> don't currently see any real blockers, so that might be doable.
>
> The release notes look ok so far, but the bug tracker list is really short
> in comparison. Please add to both as you see fit.
>
> http://wiki.cython.org/ReleaseNotes-0.17
>
> http://trac.cython.org/cython_trac/query?status=closed&group=component&order=id&col=id&col=summary&col=milestone&col=status&col=type&col=priority&col=component&milestone=0.17&desc=1
>
> Stefan
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

From stefan_ml at behnel.de  Sun Jul 22 21:42:49 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 22 Jul 2012 21:42:49 +0200
Subject: [Cython] planning for 0.17
In-Reply-To: <CANg26EW=sg6+0x79psRqwuwXSqAoVTqhUkOtyseEi-ecAVKQMA@mail.gmail.com>
References: <4FEA1D63.5090902@behnel.de>
	<CANg26EW=sg6+0x79psRqwuwXSqAoVTqhUkOtyseEi-ecAVKQMA@mail.gmail.com>
Message-ID: <500C57B9.7050001@behnel.de>

mark florisson, 21.07.2012 20:48:
> On 26 June 2012 21:36, Stefan Behnel wrote:
>> I'd like to get an idea of what's still open for 0.17.
>>
>> Mark mentioned some open memoryview issues on his list and I know that
>> there are still issues with PyPy, some of which could get fixed in a
>> reasonable time frame. Also, Jenkins isn't all that happy yet.
>>
>> https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests/
> 
> Jenkins is blue now, sorry for the holdup.

Thanks!


> I also updated the release
> notes, are all contributors already in there?

I'm not sure they are. I haven't touched the list for a while, and I don't
think it was complete when I last did.

In any case, I'll take another look through my patch queue to make sure I
didn't forget anything and see that I can get out an alpha release next week.

Stefan


From vitja.makarov at gmail.com  Mon Jul 23 07:13:11 2012
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Mon, 23 Jul 2012 09:13:11 +0400
Subject: [Cython] Type inference and C++
Message-ID: <CAKGHGPTetohfeoMdCDcXfOMhssOEkX448fctH6pgnhaqGy1EgA@mail.gmail.com>

Here is part of original testcase:

from cython.operator cimport dereference as d
from cython.operator cimport preincrement as incr
from libcpp.vector cimport vector

def reverse_iteration_test(L):
    v = new vector[int]()
    for a in L:
        v.push_back(a)
    it = v.rbegin()
    while it != v.rend():
        a = d(it)
        incr(it)
        print(a)

It doesn't work with local type inference enabled since `a` in
for-loop is infered as pyobject and in while loop it's inferred as
reverse_iterator.

Then I tried to comment out for loop and compile it with
upstream/master, same error here:

note: cpp_stl_vector.pyx:8:6: inferred 'v' to be of type 'vector[int] *'
note: cpp_stl_vector.pyx:11:7: inferred 'it' to be of type 'reverse_iterator'
note: cpp_stl_vector.pyx:13:10: inferred 'a' to be of type 'reverse_iterator'

Error compiling Cython file:
------------------------------------------------------------
...
    v = new vector[int]()
    #for a in L:
    #    v.push_back(a)
    it = v.rbegin()
    while it != v.rend():
        a = d(it)
            ^
------------------------------------------------------------

cpp_stl_vector.pyx:13:13: Cannot assign type 'int &' to 'reverse_iterator'

Error compiling Cython file:
------------------------------------------------------------
...
    #    v.push_back(a)
    it = v.rbegin()
    while it != v.rend():
        a = d(it)
        incr(it)
        print(a)
              ^
------------------------------------------------------------

cpp_stl_vector.pyx:15:15: Cannot convert 'reverse_iterator' to Python object


I think it's not correct to infer `a` as reverse_iterator because it's
not an iterator it's vector's item.

-- 
vitja.

From stefan_ml at behnel.de  Mon Jul 23 18:43:24 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 23 Jul 2012 18:43:24 +0200
Subject: [Cython] code generated for arguments of type "type"
In-Reply-To: <50066112.4020800@behnel.de>
References: <50066112.4020800@behnel.de>
Message-ID: <500D7F2C.5060906@behnel.de>

Stefan Behnel, 18.07.2012 09:09:
> the following pull request tries to fix the fact that we currently generate
> arguments of type "PyObject*" when someone spells out "type" for them.
> 
> https://github.com/cython/cython/pull/126
> 
> That's mostly a problem in CPython's C-API, but also in some NumPy
> functions. The correct way to translate this would be as a "PyTypeObject*",
> not a bare "PyObject*".
> 
> Does anyone see a general problem with changing this in the compiler
> directly? I'm not sure that it won't break code, for example. We'll at
> least have to cast the arguments on the way in as well.

Following up on this: I came to the conclusion that PyTypeObject is really
not what one wants in normal code. Even if a "type" is declared, you still
want a plain object to work with in almost all cases that just happens to
be a type object on top.

That makes the request a pure C-API thing and I think the pull request
handles this quite ok.

Stefan


From stefan_ml at behnel.de  Mon Jul 23 19:54:49 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 23 Jul 2012 19:54:49 +0200
Subject: [Cython] merging _refactor_indexnode branch
In-Reply-To: <CANg26EVK9JENirijG-Y1o2poHGKuLVOArSK4K8Sw_FoPRwpzZA@mail.gmail.com>
References: <4FFBD8C6.6090708@behnel.de>
	<CANg26EV3Ce3ixR=UzM9JSzRkZ6ZO7OR2FLTnjnkxMUQ8MNiV9g@mail.gmail.com>
	<50052A12.9020205@behnel.de>
	<CANg26EVK9JENirijG-Y1o2poHGKuLVOArSK4K8Sw_FoPRwpzZA@mail.gmail.com>
Message-ID: <500D8FE9.4070801@behnel.de>

mark florisson, 17.07.2012 12:20:
> On 17 July 2012 10:02, Stefan Behnel wrote:
>> mark florisson, 10.07.2012 11:20:
>>> Do you want _refactor_indexnode merged for this release?
>>
>> I was ok with it when I first skipped over it. It would be good to give it
>> another pair of eyeballs, but apart from that, yes, I think it should go
>> in. I'll merge it as soon as Jenkins is happy with the test suite again
>> (and after re-enabling the broken tests, which are related after all), so
>> that we can see if everything is ok after the merge.
> 
> Hm, some things changed here. I rebased my current working branch
> (_array_expressions) on that branch, and then I moved the buffer and
> memoryview tests into their own directories in master, and I think I
> rebased my devel branch along with _refactor_indexnode on master. So
> I'll have to renew that pull request using the final rebase. I'll then
> rebase the rest of my working branch on master.

Would you have an estimate when this will be done? It would be good if this
could be merged for the alpha, to avoid too much code churn after the first
user tests.

Stefan


From markflorisson88 at gmail.com  Mon Jul 23 20:07:22 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Mon, 23 Jul 2012 19:07:22 +0100
Subject: [Cython] merging _refactor_indexnode branch
In-Reply-To: <500D8FE9.4070801@behnel.de>
References: <4FFBD8C6.6090708@behnel.de>
	<CANg26EV3Ce3ixR=UzM9JSzRkZ6ZO7OR2FLTnjnkxMUQ8MNiV9g@mail.gmail.com>
	<50052A12.9020205@behnel.de>
	<CANg26EVK9JENirijG-Y1o2poHGKuLVOArSK4K8Sw_FoPRwpzZA@mail.gmail.com>
	<500D8FE9.4070801@behnel.de>
Message-ID: <CANg26EWiaCuhfHsR=PFe8hnALgBvtwPA1h1CcbtoHdgp11eX-A@mail.gmail.com>

On 23 July 2012 18:54, Stefan Behnel <stefan_ml at behnel.de> wrote:
> mark florisson, 17.07.2012 12:20:
>> On 17 July 2012 10:02, Stefan Behnel wrote:
>>> mark florisson, 10.07.2012 11:20:
>>>> Do you want _refactor_indexnode merged for this release?
>>>
>>> I was ok with it when I first skipped over it. It would be good to give it
>>> another pair of eyeballs, but apart from that, yes, I think it should go
>>> in. I'll merge it as soon as Jenkins is happy with the test suite again
>>> (and after re-enabling the broken tests, which are related after all), so
>>> that we can see if everything is ok after the merge.
>>
>> Hm, some things changed here. I rebased my current working branch
>> (_array_expressions) on that branch, and then I moved the buffer and
>> memoryview tests into their own directories in master, and I think I
>> rebased my devel branch along with _refactor_indexnode on master. So
>> I'll have to renew that pull request using the final rebase. I'll then
>> rebase the rest of my working branch on master.
>
> Would you have an estimate when this will be done? It would be good if this
> could be merged for the alpha, to avoid too much code churn after the first
> user tests.

Sorry, I forgot to do this. Actually if you want I can do it now, but
since it doesn't really bring any features it might be better to have
it wait until next release if it happens to break things. Your call,
I'll make a new PR.

> Stefan
>
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

From markflorisson88 at gmail.com  Mon Jul 23 20:38:10 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Mon, 23 Jul 2012 19:38:10 +0100
Subject: [Cython] merging _refactor_indexnode branch
In-Reply-To: <CANg26EWiaCuhfHsR=PFe8hnALgBvtwPA1h1CcbtoHdgp11eX-A@mail.gmail.com>
References: <4FFBD8C6.6090708@behnel.de>
	<CANg26EV3Ce3ixR=UzM9JSzRkZ6ZO7OR2FLTnjnkxMUQ8MNiV9g@mail.gmail.com>
	<50052A12.9020205@behnel.de>
	<CANg26EVK9JENirijG-Y1o2poHGKuLVOArSK4K8Sw_FoPRwpzZA@mail.gmail.com>
	<500D8FE9.4070801@behnel.de>
	<CANg26EWiaCuhfHsR=PFe8hnALgBvtwPA1h1CcbtoHdgp11eX-A@mail.gmail.com>
Message-ID: <CANg26EX0ouyGQk=1Fuaf4vczaY0gDnbLBSwD03qd_77txMfQyQ@mail.gmail.com>

On 23 July 2012 19:07, mark florisson <markflorisson88 at gmail.com> wrote:
> On 23 July 2012 18:54, Stefan Behnel <stefan_ml at behnel.de> wrote:
>> mark florisson, 17.07.2012 12:20:
>>> On 17 July 2012 10:02, Stefan Behnel wrote:
>>>> mark florisson, 10.07.2012 11:20:
>>>>> Do you want _refactor_indexnode merged for this release?
>>>>
>>>> I was ok with it when I first skipped over it. It would be good to give it
>>>> another pair of eyeballs, but apart from that, yes, I think it should go
>>>> in. I'll merge it as soon as Jenkins is happy with the test suite again
>>>> (and after re-enabling the broken tests, which are related after all), so
>>>> that we can see if everything is ok after the merge.
>>>
>>> Hm, some things changed here. I rebased my current working branch
>>> (_array_expressions) on that branch, and then I moved the buffer and
>>> memoryview tests into their own directories in master, and I think I
>>> rebased my devel branch along with _refactor_indexnode on master. So
>>> I'll have to renew that pull request using the final rebase. I'll then
>>> rebase the rest of my working branch on master.
>>
>> Would you have an estimate when this will be done? It would be good if this
>> could be merged for the alpha, to avoid too much code churn after the first
>> user tests.
>
> Sorry, I forgot to do this. Actually if you want I can do it now, but
> since it doesn't really bring any features it might be better to have
> it wait until next release if it happens to break things. Your call,
> I'll make a new PR.

Ok, it's here: https://github.com/cython/cython/pull/137

Are we going for a release branch again, or a devel branch? Last time
many people assumed the master branch was the release branch.

>> Stefan
>>
>> _______________________________________________
>> cython-devel mailing list
>> cython-devel at python.org
>> http://mail.python.org/mailman/listinfo/cython-devel

From stefan_ml at behnel.de  Mon Jul 23 20:46:33 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 23 Jul 2012 20:46:33 +0200
Subject: [Cython] merging _refactor_indexnode branch
In-Reply-To: <CANg26EX0ouyGQk=1Fuaf4vczaY0gDnbLBSwD03qd_77txMfQyQ@mail.gmail.com>
References: <4FFBD8C6.6090708@behnel.de>
	<CANg26EV3Ce3ixR=UzM9JSzRkZ6ZO7OR2FLTnjnkxMUQ8MNiV9g@mail.gmail.com>
	<50052A12.9020205@behnel.de>
	<CANg26EVK9JENirijG-Y1o2poHGKuLVOArSK4K8Sw_FoPRwpzZA@mail.gmail.com>
	<500D8FE9.4070801@behnel.de>
	<CANg26EWiaCuhfHsR=PFe8hnALgBvtwPA1h1CcbtoHdgp11eX-A@mail.gmail.com>
	<CANg26EX0ouyGQk=1Fuaf4vczaY0gDnbLBSwD03qd_77txMfQyQ@mail.gmail.com>
Message-ID: <500D9C09.90100@behnel.de>

mark florisson, 23.07.2012 20:38:
> On 23 July 2012 19:07, mark florisson wrote:
>> On 23 July 2012 18:54, Stefan Behnel wrote:
>>> mark florisson, 17.07.2012 12:20:
>>>> I'll have to renew that pull request using the final rebase. I'll then
>>>> rebase the rest of my working branch on master.
>>>
>>> Would you have an estimate when this will be done? It would be good if this
>>> could be merged for the alpha, to avoid too much code churn after the first
>>> user tests.
>>
>> Sorry, I forgot to do this. Actually if you want I can do it now, but
>> since it doesn't really bring any features it might be better to have
>> it wait until next release if it happens to break things. Your call,
>> I'll make a new PR.
> 
> Ok, it's here: https://github.com/cython/cython/pull/137

Cool, I'll give it a try.


> Are we going for a release branch again, or a devel branch? Last time
> many people assumed the master branch was the release branch.

I think we can leave things in the master for now and recreate the release
branch after the release. People can work in their local branches if they
have anything for 0.18 already.

Stefan


From stefan_ml at behnel.de  Mon Jul 23 20:55:55 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 23 Jul 2012 20:55:55 +0200
Subject: [Cython] merging _refactor_indexnode branch
In-Reply-To: <CANg26EWiaCuhfHsR=PFe8hnALgBvtwPA1h1CcbtoHdgp11eX-A@mail.gmail.com>
References: <4FFBD8C6.6090708@behnel.de>
	<CANg26EV3Ce3ixR=UzM9JSzRkZ6ZO7OR2FLTnjnkxMUQ8MNiV9g@mail.gmail.com>
	<50052A12.9020205@behnel.de>
	<CANg26EVK9JENirijG-Y1o2poHGKuLVOArSK4K8Sw_FoPRwpzZA@mail.gmail.com>
	<500D8FE9.4070801@behnel.de>
	<CANg26EWiaCuhfHsR=PFe8hnALgBvtwPA1h1CcbtoHdgp11eX-A@mail.gmail.com>
Message-ID: <500D9E3B.60201@behnel.de>

mark florisson, 23.07.2012 20:07:
> On 23 July 2012 18:54, Stefan Behnel wrote:
>> mark florisson, 17.07.2012 12:20:
>>> On 17 July 2012 10:02, Stefan Behnel wrote:
>>>> mark florisson, 10.07.2012 11:20:
>>>>> Do you want _refactor_indexnode merged for this release?
>>>>
>>>> I was ok with it when I first skipped over it. It would be good to give it
>>>> another pair of eyeballs, but apart from that, yes, I think it should go
>>>> in. I'll merge it as soon as Jenkins is happy with the test suite again
>>>> (and after re-enabling the broken tests, which are related after all), so
>>>> that we can see if everything is ok after the merge.
>>>
>>> Hm, some things changed here. I rebased my current working branch
>>> (_array_expressions) on that branch, and then I moved the buffer and
>>> memoryview tests into their own directories in master, and I think I
>>> rebased my devel branch along with _refactor_indexnode on master. So
>>> I'll have to renew that pull request using the final rebase. I'll then
>>> rebase the rest of my working branch on master.
>>
>> Would you have an estimate when this will be done? It would be good if this
>> could be merged for the alpha, to avoid too much code churn after the first
>> user tests.
> 
> Sorry, I forgot to do this. Actually if you want I can do it now, but
> since it doesn't really bring any features it might be better to have
> it wait until next release if it happens to break things. Your call,
> I'll make a new PR.

Hmm, revisiting the amount of code that this changes now, I would prefer
taking a step back, playing safe and leaving this for the next release.
Sorry to have rushed you here, but it carries a considerable risk for the
release. We can merge it right when 0.17 is out, I don't expect any major
changes till then that would require yet another rebase of your pull request.

Stefan


From markflorisson88 at gmail.com  Mon Jul 23 21:02:09 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Mon, 23 Jul 2012 20:02:09 +0100
Subject: [Cython] merging _refactor_indexnode branch
In-Reply-To: <500D9E3B.60201@behnel.de>
References: <4FFBD8C6.6090708@behnel.de>
	<CANg26EV3Ce3ixR=UzM9JSzRkZ6ZO7OR2FLTnjnkxMUQ8MNiV9g@mail.gmail.com>
	<50052A12.9020205@behnel.de>
	<CANg26EVK9JENirijG-Y1o2poHGKuLVOArSK4K8Sw_FoPRwpzZA@mail.gmail.com>
	<500D8FE9.4070801@behnel.de>
	<CANg26EWiaCuhfHsR=PFe8hnALgBvtwPA1h1CcbtoHdgp11eX-A@mail.gmail.com>
	<500D9E3B.60201@behnel.de>
Message-ID: <CANg26EUHsHOvw+YqecCg7CeUV+di+j=NdyYEGgQibDBLJcAe6Q@mail.gmail.com>

On 23 July 2012 19:55, Stefan Behnel <stefan_ml at behnel.de> wrote:
> mark florisson, 23.07.2012 20:07:
>> On 23 July 2012 18:54, Stefan Behnel wrote:
>>> mark florisson, 17.07.2012 12:20:
>>>> On 17 July 2012 10:02, Stefan Behnel wrote:
>>>>> mark florisson, 10.07.2012 11:20:
>>>>>> Do you want _refactor_indexnode merged for this release?
>>>>>
>>>>> I was ok with it when I first skipped over it. It would be good to give it
>>>>> another pair of eyeballs, but apart from that, yes, I think it should go
>>>>> in. I'll merge it as soon as Jenkins is happy with the test suite again
>>>>> (and after re-enabling the broken tests, which are related after all), so
>>>>> that we can see if everything is ok after the merge.
>>>>
>>>> Hm, some things changed here. I rebased my current working branch
>>>> (_array_expressions) on that branch, and then I moved the buffer and
>>>> memoryview tests into their own directories in master, and I think I
>>>> rebased my devel branch along with _refactor_indexnode on master. So
>>>> I'll have to renew that pull request using the final rebase. I'll then
>>>> rebase the rest of my working branch on master.
>>>
>>> Would you have an estimate when this will be done? It would be good if this
>>> could be merged for the alpha, to avoid too much code churn after the first
>>> user tests.
>>
>> Sorry, I forgot to do this. Actually if you want I can do it now, but
>> since it doesn't really bring any features it might be better to have
>> it wait until next release if it happens to break things. Your call,
>> I'll make a new PR.
>
> Hmm, revisiting the amount of code that this changes now, I would prefer
> taking a step back, playing safe and leaving this for the next release.
> Sorry to have rushed you here, but it carries a considerable risk for the
> release. We can merge it right when 0.17 is out, I don't expect any major
> changes till then that would require yet another rebase of your pull request.
>
> Stefan
>
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

Great, totally understandable. Was this comment "Note that some
aspects of the memory views feature may not currently work in Python
2.4.x. This is a known bug." in the release notes about the copying of
Python slice objects?

From stefan_ml at behnel.de  Mon Jul 23 21:08:20 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 23 Jul 2012 21:08:20 +0200
Subject: [Cython] merging _refactor_indexnode branch
In-Reply-To: <CANg26EUHsHOvw+YqecCg7CeUV+di+j=NdyYEGgQibDBLJcAe6Q@mail.gmail.com>
References: <4FFBD8C6.6090708@behnel.de>
	<CANg26EV3Ce3ixR=UzM9JSzRkZ6ZO7OR2FLTnjnkxMUQ8MNiV9g@mail.gmail.com>
	<50052A12.9020205@behnel.de>
	<CANg26EVK9JENirijG-Y1o2poHGKuLVOArSK4K8Sw_FoPRwpzZA@mail.gmail.com>
	<500D8FE9.4070801@behnel.de>
	<CANg26EWiaCuhfHsR=PFe8hnALgBvtwPA1h1CcbtoHdgp11eX-A@mail.gmail.com>
	<500D9E3B.60201@behnel.de>
	<CANg26EUHsHOvw+YqecCg7CeUV+di+j=NdyYEGgQibDBLJcAe6Q@mail.gmail.com>
Message-ID: <500DA124.4060405@behnel.de>

mark florisson, 23.07.2012 21:02:
> Was this comment "Note that some
> aspects of the memory views feature may not currently work in Python
> 2.4.x. This is a known bug." in the release notes about the copying of
> Python slice objects?

Yes, about the failing tests. I removed it now.

Stefan


From vitja.makarov at gmail.com  Mon Jul 23 21:28:48 2012
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Mon, 23 Jul 2012 23:28:48 +0400
Subject: [Cython] Type inference and C++
In-Reply-To: <CAKGHGPTetohfeoMdCDcXfOMhssOEkX448fctH6pgnhaqGy1EgA@mail.gmail.com>
References: <CAKGHGPTetohfeoMdCDcXfOMhssOEkX448fctH6pgnhaqGy1EgA@mail.gmail.com>
Message-ID: <CAKGHGPRtnp195d2iw22kzKF4GwvPv9AFb5J2echQJDyu2aNvqA@mail.gmail.com>

2012/7/23 Vitja Makarov <vitja.makarov at gmail.com>:
> Here is part of original testcase:
>
> from cython.operator cimport dereference as d
> from cython.operator cimport preincrement as incr
> from libcpp.vector cimport vector
>
> def reverse_iteration_test(L):
>     v = new vector[int]()
>     for a in L:
>         v.push_back(a)
>     it = v.rbegin()
>     while it != v.rend():
>         a = d(it)
>         incr(it)
>         print(a)
>
> It doesn't work with local type inference enabled since `a` in
> for-loop is infered as pyobject and in while loop it's inferred as
> reverse_iterator.
>
> Then I tried to comment out for loop and compile it with
> upstream/master, same error here:
>
> note: cpp_stl_vector.pyx:8:6: inferred 'v' to be of type 'vector[int] *'
> note: cpp_stl_vector.pyx:11:7: inferred 'it' to be of type 'reverse_iterator'
> note: cpp_stl_vector.pyx:13:10: inferred 'a' to be of type 'reverse_iterator'
>
> Error compiling Cython file:
> ------------------------------------------------------------
> ...
>     v = new vector[int]()
>     #for a in L:
>     #    v.push_back(a)
>     it = v.rbegin()
>     while it != v.rend():
>         a = d(it)
>             ^
> ------------------------------------------------------------
>
> cpp_stl_vector.pyx:13:13: Cannot assign type 'int &' to 'reverse_iterator'
>
> Error compiling Cython file:
> ------------------------------------------------------------
> ...
>     #    v.push_back(a)
>     it = v.rbegin()
>     while it != v.rend():
>         a = d(it)
>         incr(it)
>         print(a)
>               ^
> ------------------------------------------------------------
>
> cpp_stl_vector.pyx:15:15: Cannot convert 'reverse_iterator' to Python object
>
>
> I think it's not correct to infer `a` as reverse_iterator because it's
> not an iterator it's vector's item.
>

It seems that dereference is the only way we can get iterator's value,
but UnopNode doesn't respect overloaded operator*()



-- 
vitja.

From markflorisson88 at gmail.com  Mon Jul 23 22:21:19 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Mon, 23 Jul 2012 21:21:19 +0100
Subject: [Cython] Type inference and C++
In-Reply-To: <CAKGHGPRtnp195d2iw22kzKF4GwvPv9AFb5J2echQJDyu2aNvqA@mail.gmail.com>
References: <CAKGHGPTetohfeoMdCDcXfOMhssOEkX448fctH6pgnhaqGy1EgA@mail.gmail.com>
	<CAKGHGPRtnp195d2iw22kzKF4GwvPv9AFb5J2echQJDyu2aNvqA@mail.gmail.com>
Message-ID: <CANg26EXYdnCx2m94nviKyBr1O7Q6B_+-iEGXaKEgBbbcnDjtXA@mail.gmail.com>

On 23 July 2012 20:28, Vitja Makarov <vitja.makarov at gmail.com> wrote:
> 2012/7/23 Vitja Makarov <vitja.makarov at gmail.com>:
>> Here is part of original testcase:
>>
>> from cython.operator cimport dereference as d
>> from cython.operator cimport preincrement as incr
>> from libcpp.vector cimport vector
>>
>> def reverse_iteration_test(L):
>>     v = new vector[int]()
>>     for a in L:
>>         v.push_back(a)
>>     it = v.rbegin()
>>     while it != v.rend():
>>         a = d(it)
>>         incr(it)
>>         print(a)
>>
>> It doesn't work with local type inference enabled since `a` in
>> for-loop is infered as pyobject and in while loop it's inferred as
>> reverse_iterator.
>>
>> Then I tried to comment out for loop and compile it with
>> upstream/master, same error here:
>>
>> note: cpp_stl_vector.pyx:8:6: inferred 'v' to be of type 'vector[int] *'
>> note: cpp_stl_vector.pyx:11:7: inferred 'it' to be of type 'reverse_iterator'
>> note: cpp_stl_vector.pyx:13:10: inferred 'a' to be of type 'reverse_iterator'
>>
>> Error compiling Cython file:
>> ------------------------------------------------------------
>> ...
>>     v = new vector[int]()
>>     #for a in L:
>>     #    v.push_back(a)
>>     it = v.rbegin()
>>     while it != v.rend():
>>         a = d(it)
>>             ^
>> ------------------------------------------------------------
>>
>> cpp_stl_vector.pyx:13:13: Cannot assign type 'int &' to 'reverse_iterator'
>>
>> Error compiling Cython file:
>> ------------------------------------------------------------
>> ...
>>     #    v.push_back(a)
>>     it = v.rbegin()
>>     while it != v.rend():
>>         a = d(it)
>>         incr(it)
>>         print(a)
>>               ^
>> ------------------------------------------------------------
>>
>> cpp_stl_vector.pyx:15:15: Cannot convert 'reverse_iterator' to Python object
>>
>>
>> I think it's not correct to infer `a` as reverse_iterator because it's
>> not an iterator it's vector's item.
>>
>
> It seems that dereference is the only way we can get iterator's value,
> but UnopNode doesn't respect overloaded operator*()
>

I thought unary * was only used for argument unpacking and consuming
left-over items when unpacking an iterable.

>
> --
> vitja.
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

From stefan_ml at behnel.de  Mon Jul 23 22:45:46 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 23 Jul 2012 22:45:46 +0200
Subject: [Cython] Cython 0.17 beta 1 released
Message-ID: <500DB7FA.4000409@behnel.de>

Hello everyone,

on behalf of the Cython project team, I'm proud to announce the release of
our first beta of Cython 0.17. This is another major step forward in the
development of the language that will make life easier for a lot of users,
rounds up some rough edges of the compiler and adds (preliminary) support
for CPython 3.3 and PyPy.

Download: http://cython.org/release/Cython-0.17.beta1.tar.gz

Release notes: http://wiki.cython.org/ReleaseNotes-0.17

Documentation: http://docs.cython.org/


Major features of this release include:

* vastly improved integration with the C++ STL containers

  http://docs.cython.org/src/userguide/wrapping_CPlusPlus.html#standard-library

  http://docs.cython.org/src/tutorial/strings.html#c-strings

* "yield from" delegation between generators (PEP 380)

  http://www.python.org/dev/peps/pep-0380/

* alpha quality support for PyPy (via cpyext)

  http://docs.cython.org/src/tutorial/pypy.html


Several other features and improvements are listed in the release notes:

  http://wiki.cython.org/ReleaseNotes-0.17


I'm expecting at least one release candidate to follow on this beta
version, and a final release in early August. Please give this beta release
as much testing as you can, so that we can quickly advance towards the
final release.


Have fun,

Stefan

From vitja.makarov at gmail.com  Mon Jul 23 22:58:03 2012
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Tue, 24 Jul 2012 00:58:03 +0400
Subject: [Cython] Type inference and C++
In-Reply-To: <CANg26EXYdnCx2m94nviKyBr1O7Q6B_+-iEGXaKEgBbbcnDjtXA@mail.gmail.com>
References: <CAKGHGPTetohfeoMdCDcXfOMhssOEkX448fctH6pgnhaqGy1EgA@mail.gmail.com>
	<CAKGHGPRtnp195d2iw22kzKF4GwvPv9AFb5J2echQJDyu2aNvqA@mail.gmail.com>
	<CANg26EXYdnCx2m94nviKyBr1O7Q6B_+-iEGXaKEgBbbcnDjtXA@mail.gmail.com>
Message-ID: <CAKGHGPRyoB4KUuoQqn5xqQF+wHP9jdrmLpww5ypuuNGDFDYXhA@mail.gmail.com>

2012/7/24 mark florisson <markflorisson88 at gmail.com>:
> On 23 July 2012 20:28, Vitja Makarov <vitja.makarov at gmail.com> wrote:
>> 2012/7/23 Vitja Makarov <vitja.makarov at gmail.com>:
>>> Here is part of original testcase:
>>>
>>> from cython.operator cimport dereference as d
>>> from cython.operator cimport preincrement as incr
>>> from libcpp.vector cimport vector
>>>
>>> def reverse_iteration_test(L):
>>>     v = new vector[int]()
>>>     for a in L:
>>>         v.push_back(a)
>>>     it = v.rbegin()
>>>     while it != v.rend():
>>>         a = d(it)
>>>         incr(it)
>>>         print(a)
>>>
>>> It doesn't work with local type inference enabled since `a` in
>>> for-loop is infered as pyobject and in while loop it's inferred as
>>> reverse_iterator.
>>>
>>> Then I tried to comment out for loop and compile it with
>>> upstream/master, same error here:
>>>
>>> note: cpp_stl_vector.pyx:8:6: inferred 'v' to be of type 'vector[int] *'
>>> note: cpp_stl_vector.pyx:11:7: inferred 'it' to be of type 'reverse_iterator'
>>> note: cpp_stl_vector.pyx:13:10: inferred 'a' to be of type 'reverse_iterator'
>>>
>>> Error compiling Cython file:
>>> ------------------------------------------------------------
>>> ...
>>>     v = new vector[int]()
>>>     #for a in L:
>>>     #    v.push_back(a)
>>>     it = v.rbegin()
>>>     while it != v.rend():
>>>         a = d(it)
>>>             ^
>>> ------------------------------------------------------------
>>>
>>> cpp_stl_vector.pyx:13:13: Cannot assign type 'int &' to 'reverse_iterator'
>>>
>>> Error compiling Cython file:
>>> ------------------------------------------------------------
>>> ...
>>>     #    v.push_back(a)
>>>     it = v.rbegin()
>>>     while it != v.rend():
>>>         a = d(it)
>>>         incr(it)
>>>         print(a)
>>>               ^
>>> ------------------------------------------------------------
>>>
>>> cpp_stl_vector.pyx:15:15: Cannot convert 'reverse_iterator' to Python object
>>>
>>>
>>> I think it's not correct to infer `a` as reverse_iterator because it's
>>> not an iterator it's vector's item.
>>>
>>
>> It seems that dereference is the only way we can get iterator's value,
>> but UnopNode doesn't respect overloaded operator*()
>>
>
> I thought unary * was only used for argument unpacking and consuming
> left-over items when unpacking an iterable.
>

Yes, in Python here it means C/C++ * unary operator that is called
with cython.operator.dereference()

-- 
vitja.

From stefan_ml at behnel.de  Tue Jul 24 12:43:10 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 24 Jul 2012 12:43:10 +0200
Subject: [Cython] Type inference and C++
In-Reply-To: <CAKGHGPTetohfeoMdCDcXfOMhssOEkX448fctH6pgnhaqGy1EgA@mail.gmail.com>
References: <CAKGHGPTetohfeoMdCDcXfOMhssOEkX448fctH6pgnhaqGy1EgA@mail.gmail.com>
Message-ID: <500E7C3E.7060406@behnel.de>

Vitja Makarov, 23.07.2012 07:13:
> Here is part of original testcase:
> 
> from cython.operator cimport dereference as d
> from cython.operator cimport preincrement as incr
> from libcpp.vector cimport vector
> 
> def reverse_iteration_test(L):
>     v = new vector[int]()
>     for a in L:
>         v.push_back(a)
>     it = v.rbegin()
>     while it != v.rend():
>         a = d(it)
>         incr(it)
>         print(a)
> 
> It doesn't work with local type inference enabled since `a` in
> for-loop is infered as pyobject and in while loop it's inferred as
> reverse_iterator.
> 
> Then I tried to comment out for loop and compile it with
> upstream/master, same error here:
> 
> note: cpp_stl_vector.pyx:8:6: inferred 'v' to be of type 'vector[int] *'
> note: cpp_stl_vector.pyx:11:7: inferred 'it' to be of type 'reverse_iterator'
> note: cpp_stl_vector.pyx:13:10: inferred 'a' to be of type 'reverse_iterator'
> 
> Error compiling Cython file:
> ------------------------------------------------------------
> ...
>     v = new vector[int]()
>     #for a in L:
>     #    v.push_back(a)
>     it = v.rbegin()
>     while it != v.rend():
>         a = d(it)
>             ^
> ------------------------------------------------------------
> 
> cpp_stl_vector.pyx:13:13: Cannot assign type 'int &' to 'reverse_iterator'
> 
> Error compiling Cython file:
> ------------------------------------------------------------
> ...
>     #    v.push_back(a)
>     it = v.rbegin()
>     while it != v.rend():
>         a = d(it)
>         incr(it)
>         print(a)
>               ^
> ------------------------------------------------------------
> 
> cpp_stl_vector.pyx:15:15: Cannot convert 'reverse_iterator' to Python object
> 
> 
> I think it's not correct to infer `a` as reverse_iterator because it's
> not an iterator it's vector's item.

Right. DereferenceNode should implement infer_type().

Stefan


From stefan_ml at behnel.de  Tue Jul 24 14:33:08 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 24 Jul 2012 14:33:08 +0200
Subject: [Cython] Type inference and C++
In-Reply-To: <500E7C3E.7060406@behnel.de>
References: <CAKGHGPTetohfeoMdCDcXfOMhssOEkX448fctH6pgnhaqGy1EgA@mail.gmail.com>
	<500E7C3E.7060406@behnel.de>
Message-ID: <500E9604.2040000@behnel.de>

Stefan Behnel, 24.07.2012 12:43:
> Vitja Makarov, 23.07.2012 07:13:
>> Here is part of original testcase:
>>
>> from cython.operator cimport dereference as d
>> from cython.operator cimport preincrement as incr
>> from libcpp.vector cimport vector
>>
>> def reverse_iteration_test(L):
>>     v = new vector[int]()
>>     for a in L:
>>         v.push_back(a)
>>     it = v.rbegin()
>>     while it != v.rend():
>>         a = d(it)
>>         incr(it)
>>         print(a)
>>
>> It doesn't work with local type inference enabled since `a` in
>> for-loop is infered as pyobject and in while loop it's inferred as
>> reverse_iterator.
>>
>> Then I tried to comment out for loop and compile it with
>> upstream/master, same error here:
>>
>> note: cpp_stl_vector.pyx:8:6: inferred 'v' to be of type 'vector[int] *'
>> note: cpp_stl_vector.pyx:11:7: inferred 'it' to be of type 'reverse_iterator'
>> note: cpp_stl_vector.pyx:13:10: inferred 'a' to be of type 'reverse_iterator'
>>
>> Error compiling Cython file:
>> ------------------------------------------------------------
>> ...
>>     v = new vector[int]()
>>     #for a in L:
>>     #    v.push_back(a)
>>     it = v.rbegin()
>>     while it != v.rend():
>>         a = d(it)
>>             ^
>> ------------------------------------------------------------
>>
>> cpp_stl_vector.pyx:13:13: Cannot assign type 'int &' to 'reverse_iterator'
>>
>> Error compiling Cython file:
>> ------------------------------------------------------------
>> ...
>>     #    v.push_back(a)
>>     it = v.rbegin()
>>     while it != v.rend():
>>         a = d(it)
>>         incr(it)
>>         print(a)
>>               ^
>> ------------------------------------------------------------
>>
>> cpp_stl_vector.pyx:15:15: Cannot convert 'reverse_iterator' to Python object
>>
>>
>> I think it's not correct to infer `a` as reverse_iterator because it's
>> not an iterator it's vector's item.
> 
> Right. DereferenceNode should implement infer_type().

... and not only DereferenceNode. Pretty much all C++ operators do not
properly implement type inference. I'll take a look.

Stefan


From vitja.makarov at gmail.com  Tue Jul 24 14:48:55 2012
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Tue, 24 Jul 2012 16:48:55 +0400
Subject: [Cython] Type inference and C++
In-Reply-To: <500E9604.2040000@behnel.de>
References: <CAKGHGPTetohfeoMdCDcXfOMhssOEkX448fctH6pgnhaqGy1EgA@mail.gmail.com>
	<500E7C3E.7060406@behnel.de> <500E9604.2040000@behnel.de>
Message-ID: <CAKGHGPSxgxR1NC102viXw-fShbi87ic-NgWR9uvq7CBLT2WY1Q@mail.gmail.com>

2012/7/24 Stefan Behnel <stefan_ml at behnel.de>:
> Stefan Behnel, 24.07.2012 12:43:
>> Vitja Makarov, 23.07.2012 07:13:
>>> Here is part of original testcase:
>>>
>>> from cython.operator cimport dereference as d
>>> from cython.operator cimport preincrement as incr
>>> from libcpp.vector cimport vector
>>>
>>> def reverse_iteration_test(L):
>>>     v = new vector[int]()
>>>     for a in L:
>>>         v.push_back(a)
>>>     it = v.rbegin()
>>>     while it != v.rend():
>>>         a = d(it)
>>>         incr(it)
>>>         print(a)
>>>
>>> It doesn't work with local type inference enabled since `a` in
>>> for-loop is infered as pyobject and in while loop it's inferred as
>>> reverse_iterator.
>>>
>>> Then I tried to comment out for loop and compile it with
>>> upstream/master, same error here:
>>>
>>> note: cpp_stl_vector.pyx:8:6: inferred 'v' to be of type 'vector[int] *'
>>> note: cpp_stl_vector.pyx:11:7: inferred 'it' to be of type 'reverse_iterator'
>>> note: cpp_stl_vector.pyx:13:10: inferred 'a' to be of type 'reverse_iterator'
>>>
>>> Error compiling Cython file:
>>> ------------------------------------------------------------
>>> ...
>>>     v = new vector[int]()
>>>     #for a in L:
>>>     #    v.push_back(a)
>>>     it = v.rbegin()
>>>     while it != v.rend():
>>>         a = d(it)
>>>             ^
>>> ------------------------------------------------------------
>>>
>>> cpp_stl_vector.pyx:13:13: Cannot assign type 'int &' to 'reverse_iterator'
>>>
>>> Error compiling Cython file:
>>> ------------------------------------------------------------
>>> ...
>>>     #    v.push_back(a)
>>>     it = v.rbegin()
>>>     while it != v.rend():
>>>         a = d(it)
>>>         incr(it)
>>>         print(a)
>>>               ^
>>> ------------------------------------------------------------
>>>
>>> cpp_stl_vector.pyx:15:15: Cannot convert 'reverse_iterator' to Python object
>>>
>>>
>>> I think it's not correct to infer `a` as reverse_iterator because it's
>>> not an iterator it's vector's item.
>>
>> Right. DereferenceNode should implement infer_type().
>
> ... and not only DereferenceNode. Pretty much all C++ operators do not
> properly implement type inference. I'll take a look.
>

Ok, thanks!

-- 
vitja.

From stefan_ml at behnel.de  Tue Jul 24 17:25:05 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 24 Jul 2012 17:25:05 +0200
Subject: [Cython] Type inference and C++
In-Reply-To: <CAKGHGPSxgxR1NC102viXw-fShbi87ic-NgWR9uvq7CBLT2WY1Q@mail.gmail.com>
References: <CAKGHGPTetohfeoMdCDcXfOMhssOEkX448fctH6pgnhaqGy1EgA@mail.gmail.com>
	<500E7C3E.7060406@behnel.de> <500E9604.2040000@behnel.de>
	<CAKGHGPSxgxR1NC102viXw-fShbi87ic-NgWR9uvq7CBLT2WY1Q@mail.gmail.com>
Message-ID: <500EBE51.2010800@behnel.de>

Vitja Makarov, 24.07.2012 14:48:
> 2012/7/24 Stefan Behnel:
>> Stefan Behnel, 24.07.2012 12:43:
>>> Vitja Makarov, 23.07.2012 07:13:
>>>> Here is part of original testcase:
>>>>
>>>> from cython.operator cimport dereference as d
>>>> from cython.operator cimport preincrement as incr
>>>> from libcpp.vector cimport vector
>>>>
>>>> def reverse_iteration_test(L):
>>>>     v = new vector[int]()
>>>>     for a in L:
>>>>         v.push_back(a)
>>>>     it = v.rbegin()
>>>>     while it != v.rend():
>>>>         a = d(it)
>>>>         incr(it)
>>>>         print(a)
>>>>
>>>> I think it's not correct to infer `a` as reverse_iterator because it's
>>>> not an iterator it's vector's item.
>>>
>>> Right. DereferenceNode should implement infer_type().
>>
>> ... and not only DereferenceNode. Pretty much all C++ operators do not
>> properly implement type inference. I'll take a look.
> 
> Ok, thanks!

Here's my fix.

https://github.com/cython/cython/commit/43f3d87d9760d3c7e7fa6a127d0bdaf549880621

It became a bit more involved than I had anticipated, including some
cleanups and fixes for a couple of quirks that I ran into.

It's really too bad that the Sage build is currently broken. The C++ code
in there would be a good exercise for this change (even if it won't benefit
from the type inference). At least, so far, it doesn't seem like it has
broken more than what was broken before.

Stefan


From vitja.makarov at gmail.com  Tue Jul 24 18:42:46 2012
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Tue, 24 Jul 2012 20:42:46 +0400
Subject: [Cython] Type inference and C++
In-Reply-To: <500EBE51.2010800@behnel.de>
References: <CAKGHGPTetohfeoMdCDcXfOMhssOEkX448fctH6pgnhaqGy1EgA@mail.gmail.com>
	<500E7C3E.7060406@behnel.de> <500E9604.2040000@behnel.de>
	<CAKGHGPSxgxR1NC102viXw-fShbi87ic-NgWR9uvq7CBLT2WY1Q@mail.gmail.com>
	<500EBE51.2010800@behnel.de>
Message-ID: <CAKGHGPRYmMRfmpNiBukStk517K9x1bo_rGCkkQkcQWvnmc8N3Q@mail.gmail.com>

2012/7/24 Stefan Behnel <stefan_ml at behnel.de>:
> Vitja Makarov, 24.07.2012 14:48:
>> 2012/7/24 Stefan Behnel:
>>> Stefan Behnel, 24.07.2012 12:43:
>>>> Vitja Makarov, 23.07.2012 07:13:
>>>>> Here is part of original testcase:
>>>>>
>>>>> from cython.operator cimport dereference as d
>>>>> from cython.operator cimport preincrement as incr
>>>>> from libcpp.vector cimport vector
>>>>>
>>>>> def reverse_iteration_test(L):
>>>>>     v = new vector[int]()
>>>>>     for a in L:
>>>>>         v.push_back(a)
>>>>>     it = v.rbegin()
>>>>>     while it != v.rend():
>>>>>         a = d(it)
>>>>>         incr(it)
>>>>>         print(a)
>>>>>
>>>>> I think it's not correct to infer `a` as reverse_iterator because it's
>>>>> not an iterator it's vector's item.
>>>>
>>>> Right. DereferenceNode should implement infer_type().
>>>
>>> ... and not only DereferenceNode. Pretty much all C++ operators do not
>>> properly implement type inference. I'll take a look.
>>
>> Ok, thanks!
>
> Here's my fix.
>
> https://github.com/cython/cython/commit/43f3d87d9760d3c7e7fa6a127d0bdaf549880621
>
> It became a bit more involved than I had anticipated, including some
> cleanups and fixes for a couple of quirks that I ran into.
>
> It's really too bad that the Sage build is currently broken. The C++ code
> in there would be a good exercise for this change (even if it won't benefit
> from the type inference). At least, so far, it doesn't seem like it has
> broken more than what was broken before.
>

And sorry for stupid question: do we support multiple overloaded
operators of the same kind?


-- 
vitja.

From stefan_ml at behnel.de  Tue Jul 24 18:48:26 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 24 Jul 2012 18:48:26 +0200
Subject: [Cython] Type inference and C++
In-Reply-To: <CAKGHGPRYmMRfmpNiBukStk517K9x1bo_rGCkkQkcQWvnmc8N3Q@mail.gmail.com>
References: <CAKGHGPTetohfeoMdCDcXfOMhssOEkX448fctH6pgnhaqGy1EgA@mail.gmail.com>
	<500E7C3E.7060406@behnel.de> <500E9604.2040000@behnel.de>
	<CAKGHGPSxgxR1NC102viXw-fShbi87ic-NgWR9uvq7CBLT2WY1Q@mail.gmail.com>
	<500EBE51.2010800@behnel.de>
	<CAKGHGPRYmMRfmpNiBukStk517K9x1bo_rGCkkQkcQWvnmc8N3Q@mail.gmail.com>
Message-ID: <500ED1DA.9010100@behnel.de>

Vitja Makarov, 24.07.2012 18:42:
> 2012/7/24 Stefan Behnel:
>> Vitja Makarov, 24.07.2012 14:48:
>>> 2012/7/24 Stefan Behnel:
>>>> Stefan Behnel, 24.07.2012 12:43:
>>>>> Vitja Makarov, 23.07.2012 07:13:
>>>>>> Here is part of original testcase:
>>>>>>
>>>>>> from cython.operator cimport dereference as d
>>>>>> from cython.operator cimport preincrement as incr
>>>>>> from libcpp.vector cimport vector
>>>>>>
>>>>>> def reverse_iteration_test(L):
>>>>>>     v = new vector[int]()
>>>>>>     for a in L:
>>>>>>         v.push_back(a)
>>>>>>     it = v.rbegin()
>>>>>>     while it != v.rend():
>>>>>>         a = d(it)
>>>>>>         incr(it)
>>>>>>         print(a)
>>>>>>
>>>>>> I think it's not correct to infer `a` as reverse_iterator because it's
>>>>>> not an iterator it's vector's item.
>>>>>
>>>>> Right. DereferenceNode should implement infer_type().
>>>>
>>>> ... and not only DereferenceNode. Pretty much all C++ operators do not
>>>> properly implement type inference. I'll take a look.
>>>
>>> Ok, thanks!
>>
>> Here's my fix.
>>
>> https://github.com/cython/cython/commit/43f3d87d9760d3c7e7fa6a127d0bdaf549880621
>>
>> It became a bit more involved than I had anticipated, including some
>> cleanups and fixes for a couple of quirks that I ran into.
>>
>> It's really too bad that the Sage build is currently broken. The C++ code
>> in there would be a good exercise for this change (even if it won't benefit
>> from the type inference). At least, so far, it doesn't seem like it has
>> broken more than what was broken before.
> 
> And sorry for stupid question: do we support multiple overloaded
> operators of the same kind?

I'm sure we're lacking a test for that. At least in the change above, I'm
using "lookup_operator_for_types()", which should do the right thing.

Stefan


From mikezaletel at berkeley.edu  Wed Jul 25 00:40:22 2012
From: mikezaletel at berkeley.edu (Mike Zaletel)
Date: Tue, 24 Jul 2012 18:40:22 -0400
Subject: [Cython] Bug, or changed array assignment in 0.17beta1?
Message-ID: <CBA537BF-77DF-489C-903F-EC09BD2F2235@berkeley.edu>

Hello All,
	The exact behavior of array assignment was never entirely clear to me, but I am certain the following behavior did not occur in 0.16:

--------bug.pyx----------

def foo():
       cdef int i
       cdef int* p1 = [4, 4]
       cdef int* p2 = [5, 5]

       print "p1:",
       for i in range(2):
               print p1[i],
       print "\np2:",
       for i in range(2):
               print p2[i],

-----------------------------

which in Cython 0.17beta1 gives me

>>>import bug
>>>bug.foo()
p1: 5 5
p2: 5 5


while in Cython 0.16 I get

>>>import bug
>>>bug.foo()
p1: 4 4
p2: 5 5



	Has the syntax of array assignment changed, or is this a bug? If the former, how do you assign to an array literal? Thanks for your all your efforts,

Mike

From stefan_ml at behnel.de  Wed Jul 25 07:40:31 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 25 Jul 2012 07:40:31 +0200
Subject: [Cython] Bug, or changed array assignment in 0.17beta1?
In-Reply-To: <CBA537BF-77DF-489C-903F-EC09BD2F2235@berkeley.edu>
References: <CBA537BF-77DF-489C-903F-EC09BD2F2235@berkeley.edu>
Message-ID: <500F86CF.4080702@behnel.de>

Hi,

thanks for the report.

Mike Zaletel, 25.07.2012 00:40:
> The exact behavior of array assignment was never entirely clear to me

Yes, it's not entirely obvious.


> but I am certain the following behavior did not occur in 0.16:
> 
> --------bug.pyx----------
> 
> def foo():
>        cdef int i
>        cdef int* p1 = [4, 4]
>        cdef int* p2 = [5, 5]
> 
>        print "p1:",
>        for i in range(2):
>                print p1[i],
>        print "\np2:",
>        for i in range(2):
>                print p2[i],
> 
> -----------------------------
> 
> which in Cython 0.17beta1 gives me
> 
> >>> import bug
> >>> bug.foo()
> p1: 5 5
> p2: 5 5
> 
> 
> while in Cython 0.16 I get
> 
> >>> import bug
> >>> bug.foo()
> p1: 4 4
> p2: 5 5

The problem is that the same (temporary) local array variable is used in
both cases to build the array, and then only a pointer is assigned, i.e. p1
and p2 then point to the same array, which gets overwritten with the new
values in the second assignment.

Stefan


From stefan_ml at behnel.de  Wed Jul 25 07:55:29 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 25 Jul 2012 07:55:29 +0200
Subject: [Cython] Bug, or changed array assignment in 0.17beta1?
In-Reply-To: <500F86CF.4080702@behnel.de>
References: <CBA537BF-77DF-489C-903F-EC09BD2F2235@berkeley.edu>
	<500F86CF.4080702@behnel.de>
Message-ID: <500F8A51.40807@behnel.de>

Stefan Behnel, 25.07.2012 07:40:
> Mike Zaletel, 25.07.2012 00:40:
>> --------bug.pyx----------
>> def foo():
>>        cdef int i
>>        cdef int* p1 = [4, 4]
>>        cdef int* p2 = [5, 5]
>>
>>        print "p1:",
>>        for i in range(2):
>>                print p1[i],
>>        print "\np2:",
>>        for i in range(2):
>>                print p2[i],
>> -----------------------------
>>
>> which in Cython 0.17beta1 gives me
>>
>> >>> import bug
>> >>> bug.foo()
>> p1: 5 5
>> p2: 5 5
>>
>>
>> while in Cython 0.16 I get
>>
>> >>> import bug
>> >>> bug.foo()
>> p1: 4 4
>> p2: 5 5
> 
> The problem is that the same (temporary) local array variable is used in
> both cases to build the array, and then only a pointer is assigned, i.e. p1
> and p2 then point to the same array, which gets overwritten with the new
> values in the second assignment.

Oh, just in case I wasn't clear: you've found a bug.

Stefan


From stefan_ml at behnel.de  Wed Jul 25 08:29:39 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 25 Jul 2012 08:29:39 +0200
Subject: [Cython] Bug, or changed array assignment in 0.17beta1?
In-Reply-To: <500F86CF.4080702@behnel.de>
References: <CBA537BF-77DF-489C-903F-EC09BD2F2235@berkeley.edu>
	<500F86CF.4080702@behnel.de>
Message-ID: <500F9253.4050100@behnel.de>

Stefan Behnel, 25.07.2012 07:40:
> Mike Zaletel, 25.07.2012 00:40:
>> --------bug.pyx----------
>>
>> def foo():
>>        cdef int i
>>        cdef int* p1 = [4, 4]
>>        cdef int* p2 = [5, 5]
>>
>>        print "p1:",
>>        for i in range(2):
>>                print p1[i],
>>        print "\np2:",
>>        for i in range(2):
>>                print p2[i],
>>
>> -----------------------------
>>
>> which in Cython 0.17beta1 gives me
>>
>> >>> import bug
>> >>> bug.foo()
>> p1: 5 5
>> p2: 5 5
>>
>>
>> while in Cython 0.16 I get
>>
>> >>> import bug
>> >>> bug.foo()
>> p1: 4 4
>> p2: 5 5
> 
> The problem is that the same (temporary) local array variable is used in
> both cases to build the array, and then only a pointer is assigned, i.e. p1
> and p2 then point to the same array, which gets overwritten with the new
> values in the second assignment.

Looking into this some more, the problem arises from the pointer
assignment, because the left side is a pointer variable whereas the right
side is a temp value. Temps aren't really made for an enduring life. Here,
the array temp variable is being freed after the assignment and then
reused. We could fix it by not freeing the temp, or at least by not reusing
it, but the problem is really in the syntax. The LHS should be an array,
not a pointer.

It broke (and I broke it) when I fixed pointer type comparisons and made
pointer and array types properly hashable. I think it was already subtly
broken before in Py2 and just didn't show because it's sufficiently
unlikely that two equal array types that use the default hash by object
id() end up in the same dict bucket. Only then would the __eq__()
comparison strike to reuse the same temp for both types.

Stefan


From stefan_ml at behnel.de  Wed Jul 25 12:15:41 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 25 Jul 2012 12:15:41 +0200
Subject: [Cython] Bug, or changed array assignment in 0.17beta1?
In-Reply-To: <500F9253.4050100@behnel.de>
References: <CBA537BF-77DF-489C-903F-EC09BD2F2235@berkeley.edu>
	<500F86CF.4080702@behnel.de> <500F9253.4050100@behnel.de>
Message-ID: <500FC74D.6010100@behnel.de>

Stefan Behnel, 25.07.2012 08:29:
> Stefan Behnel, 25.07.2012 07:40:
>> Mike Zaletel, 25.07.2012 00:40:
>>> --------bug.pyx----------
>>>
>>> def foo():
>>>        cdef int i
>>>        cdef int* p1 = [4, 4]
>>>        cdef int* p2 = [5, 5]
>>>
>>>        print "p1:",
>>>        for i in range(2):
>>>                print p1[i],
>>>        print "\np2:",
>>>        for i in range(2):
>>>                print p2[i],
>>>
>>> -----------------------------
>>>
>>> which in Cython 0.17beta1 gives me
>>>
>>> >>> import bug
>>> >>> bug.foo()
>>> p1: 5 5
>>> p2: 5 5
>>>
>>>
>>> while in Cython 0.16 I get
>>>
>>> >>> import bug
>>> >>> bug.foo()
>>> p1: 4 4
>>> p2: 5 5
>>
>> The problem is that the same (temporary) local array variable is used in
>> both cases to build the array, and then only a pointer is assigned, i.e. p1
>> and p2 then point to the same array, which gets overwritten with the new
>> values in the second assignment.
> 
> Looking into this some more, the problem arises from the pointer
> assignment, because the left side is a pointer variable whereas the right
> side is a temp value. Temps aren't really made for an enduring life. Here,
> the array temp variable is being freed after the assignment and then
> reused. We could fix it by not freeing the temp, or at least by not reusing
> it

Ah, found the existing hack that aimed to do that and fixed it. :)

https://github.com/cython/cython/commit/557b8ed7dfdb9155327e481bef4522a9214695e9

Stefan


From markflorisson88 at gmail.com  Wed Jul 25 12:18:57 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Wed, 25 Jul 2012 11:18:57 +0100
Subject: [Cython] Bug, or changed array assignment in 0.17beta1?
In-Reply-To: <500F9253.4050100@behnel.de>
References: <CBA537BF-77DF-489C-903F-EC09BD2F2235@berkeley.edu>
	<500F86CF.4080702@behnel.de> <500F9253.4050100@behnel.de>
Message-ID: <CANg26EUaU6rU96sSFvsVUSc+qo5xv19eRWH3c2kvXh0VVWWExg@mail.gmail.com>

On 25 July 2012 07:29, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Stefan Behnel, 25.07.2012 07:40:
>> Mike Zaletel, 25.07.2012 00:40:
>>> --------bug.pyx----------
>>>
>>> def foo():
>>>        cdef int i
>>>        cdef int* p1 = [4, 4]
>>>        cdef int* p2 = [5, 5]
>>>
>>>        print "p1:",
>>>        for i in range(2):
>>>                print p1[i],
>>>        print "\np2:",
>>>        for i in range(2):
>>>                print p2[i],
>>>
>>> -----------------------------
>>>
>>> which in Cython 0.17beta1 gives me
>>>
>>> >>> import bug
>>> >>> bug.foo()
>>> p1: 5 5
>>> p2: 5 5
>>>
>>>
>>> while in Cython 0.16 I get
>>>
>>> >>> import bug
>>> >>> bug.foo()
>>> p1: 4 4
>>> p2: 5 5
>>
>> The problem is that the same (temporary) local array variable is used in
>> both cases to build the array, and then only a pointer is assigned, i.e. p1
>> and p2 then point to the same array, which gets overwritten with the new
>> values in the second assignment.
>
> Looking into this some more, the problem arises from the pointer
> assignment, because the left side is a pointer variable whereas the right
> side is a temp value. Temps aren't really made for an enduring life. Here,
> the array temp variable is being freed after the assignment and then
> reused. We could fix it by not freeing the temp, or at least by not reusing
> it, but the problem is really in the syntax. The LHS should be an array,
> not a pointer.
>
> It broke (and I broke it) when I fixed pointer type comparisons and made
> pointer and array types properly hashable. I think it was already subtly
> broken before in Py2 and just didn't show because it's sufficiently
> unlikely that two equal array types that use the default hash by object
> id() end up in the same dict bucket. Only then would the __eq__()
> comparison strike to reuse the same temp for both types.
>
> Stefan
>
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

I'm wondering, what was the original motivation to reuse temporaries?
I think it leads to more problems than benefits in most cases, all you
really care about is clearing up references. Do we ever have C++
stack-allocated object temporaries? Or does it save some stack space
(always, or when you take a pointer to the temporary)?

From stefan_ml at behnel.de  Wed Jul 25 12:28:43 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 25 Jul 2012 12:28:43 +0200
Subject: [Cython] Bug, or changed array assignment in 0.17beta1?
In-Reply-To: <CANg26EUaU6rU96sSFvsVUSc+qo5xv19eRWH3c2kvXh0VVWWExg@mail.gmail.com>
References: <CBA537BF-77DF-489C-903F-EC09BD2F2235@berkeley.edu>
	<500F86CF.4080702@behnel.de> <500F9253.4050100@behnel.de>
	<CANg26EUaU6rU96sSFvsVUSc+qo5xv19eRWH3c2kvXh0VVWWExg@mail.gmail.com>
Message-ID: <500FCA5B.6060208@behnel.de>

mark florisson, 25.07.2012 12:18:
> I'm wondering, what was the original motivation to reuse temporaries?

Shorter C code. Then, at some point, also smaller closures.


> I think it leads to more problems than benefits in most cases, all you
> really care about is clearing up references.

It's usually fine for C types, though. Except for the very specific case of
stack allocated C array literals, for which an explicitly allocated
(invisible) local variable would have been a better idea, I think.


> Do we ever have C++ stack-allocated object temporaries?

Don't think so. They'd be pointers.


> Or does it save some stack space
> (always, or when you take a pointer to the temporary)?

C compilers do variable aliasing anyway, so I don't think this matters.

Stefan


From greg.ewing at canterbury.ac.nz  Wed Jul 25 13:24:16 2012
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Wed, 25 Jul 2012 23:24:16 +1200
Subject: [Cython] Bug, or changed array assignment in 0.17beta1?
In-Reply-To: <CANg26EUaU6rU96sSFvsVUSc+qo5xv19eRWH3c2kvXh0VVWWExg@mail.gmail.com>
References: <CBA537BF-77DF-489C-903F-EC09BD2F2235@berkeley.edu>
	<500F86CF.4080702@behnel.de> <500F9253.4050100@behnel.de>
	<CANg26EUaU6rU96sSFvsVUSc+qo5xv19eRWH3c2kvXh0VVWWExg@mail.gmail.com>
Message-ID: <500FD760.4080807@canterbury.ac.nz>

mark florisson wrote:

> I'm wondering, what was the original motivation to reuse temporaries?

It goes back to Pyrex, where I didn't really give it much
thought -- it just seemed like the tidiest thing to do.
Once you have the logic to release temp references as soon
as it's safe to do so, it's not much harder to return the
variable to a pool as well.

> Do we ever have C++
> stack-allocated object temporaries?

Theoretically, yes --  temporaries aren't necessarily
object references, they can be of any type (or they can
in Pyrex, at least).

-- 
Greg

From lists at onerussian.com  Wed Jul 25 18:50:55 2012
From: lists at onerussian.com (Yaroslav Halchenko)
Date: Wed, 25 Jul 2012 12:50:55 -0400
Subject: [Cython] Cython 0.17 beta 1 released
In-Reply-To: <500DB7FA.4000409@behnel.de>
References: <500DB7FA.4000409@behnel.de>
Message-ID: <20120725165055.GN5788@onerussian.com>

Congrats on the beta-release!  While testing an updated debian package
for cython I have ran into failures with Python 3.2.3 (default, Jul 13 2012,
21:02:37) [GCC 4.7.1] (complete log:
http://neuro.debian.net/_files/_buildlogs/cython/0.17~beta1/cython_0.17~beta1-1_amd64.build)

anything familiar? (I see PY3 fix bf7981fb37b19f08a331c704df8bf25d3b299be5 but
it doesn't look relevant for this one, or am I wrong?)

======================================================================
ERROR: test_globals (Cython.Build.Tests.TestInline.TestInline)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/buildd/cython-0.17~beta1/build/work-dir/Cy3/Cython/Build/Tests/TestInline.py", line 41, in test_globals
    self.assertEquals(inline("return global_value + 1", **self.test_kwds), global_value + 1)
  File "/tmp/buildd/cython-0.17~beta1/build/work-dir/Cy3/Cython/Shadow.py", line 38, in inline
    return cython_inline(f, *args, **kwds)
  File "/tmp/buildd/cython-0.17~beta1/build/work-dir/Cy3/Cython/Build/Inline.py", line 194, in cython_inline
    module = imp.load_dynamic(module_name, module_path)
ImportError: /tmp/cython_inline_zo_0tz/_cython_inline_d0fe156ce72658e73c2b9a9438fd8d6a.cpython-32mu-x86_64-linux-gnu.so: cannot open shared object file: No such file or directory

======================================================================
ERROR: test_locals (Cython.Build.Tests.TestInline.TestInline)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/buildd/cython-0.17~beta1/build/work-dir/Cy3/Cython/Build/Tests/TestInline.py", line 38, in test_locals
    self.assertEquals(inline("return a+b", **self.test_kwds), 3)
  File "/tmp/buildd/cython-0.17~beta1/build/work-dir/Cy3/Cython/Shadow.py", line 38, in inline
    return cython_inline(f, *args, **kwds)
  File "/tmp/buildd/cython-0.17~beta1/build/work-dir/Cy3/Cython/Build/Inline.py", line 194, in cython_inline
    module = imp.load_dynamic(module_name, module_path)
ImportError: /tmp/cython_inline_iktjoo/_cython_inline_6d5c007586530fac3ff8084f3b2d9acd.cpython-32mu-x86_64-linux-gnu.so: cannot open shared object file: No such file or directory

======================================================================
ERROR: test_numpy (Cython.Build.Tests.TestInline.TestInline)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/buildd/cython-0.17~beta1/build/work-dir/Cy3/Cython/Build/Tests/TestInline.py", line 59, in test_numpy
    self.assertEquals(inline("return a[0,0]", a=a, **self.test_kwds), 10.0)
  File "/tmp/buildd/cython-0.17~beta1/build/work-dir/Cy3/Cython/Shadow.py", line 38, in inline
    return cython_inline(f, *args, **kwds)
  File "/tmp/buildd/cython-0.17~beta1/build/work-dir/Cy3/Cython/Build/Inline.py", line 194, in cython_inline
    module = imp.load_dynamic(module_name, module_path)
ImportError: /tmp/cython_inline_ahyyvg/_cython_inline_18c0c6f9842ba6f5ea13e83da4d74e83.cpython-32mu-x86_64-linux-gnu.so: cannot open shared object file: No such file or directory

======================================================================
ERROR: test_pure (Cython.Build.Tests.TestInline.TestInline)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/buildd/cython-0.17~beta1/build/work-dir/Cy3/Cython/Build/Tests/TestInline.py", line 49, in test_pure
    """, a=3)
  File "/tmp/buildd/cython-0.17~beta1/build/work-dir/Cy3/Cython/Shadow.py", line 38, in inline
    return cython_inline(f, *args, **kwds)
  File "/tmp/buildd/cython-0.17~beta1/build/work-dir/Cy3/Cython/Build/Inline.py", line 194, in cython_inline
    module = imp.load_dynamic(module_name, module_path)
ImportError: /tmp/buildd/cython-0.17~beta1/build/.cython/inline/_cython_inline_52bec2971518a5e2af15359227f2254e.cpython-32mu-x86_64-linux-gnu.so: cannot open shared object file: No such file or directory

======================================================================
ERROR: test_simple (Cython.Build.Tests.TestInline.TestInline)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/buildd/cython-0.17~beta1/build/work-dir/Cy3/Cython/Build/Tests/TestInline.py", line 27, in test_simple
    self.assertEquals(inline("return 1+2", **self.test_kwds), 3)
  File "/tmp/buildd/cython-0.17~beta1/build/work-dir/Cy3/Cython/Shadow.py", line 38, in inline
    return cython_inline(f, *args, **kwds)
  File "/tmp/buildd/cython-0.17~beta1/build/work-dir/Cy3/Cython/Build/Inline.py", line 194, in cython_inline
    module = imp.load_dynamic(module_name, module_path)
ImportError: /tmp/cython_inline_g42y8n/_cython_inline_2fcb515f029908306e961b3837311541.cpython-32mu-x86_64-linux-gnu.so: cannot open shared object file: No such file or directory

======================================================================
ERROR: test_types (Cython.Build.Tests.TestInline.TestInline)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/buildd/cython-0.17~beta1/build/work-dir/Cy3/Cython/Build/Tests/TestInline.py", line 33, in test_types
    """, a=1.0, b=[], **self.test_kwds), ('double', 'list object'))
  File "/tmp/buildd/cython-0.17~beta1/build/work-dir/Cy3/Cython/Shadow.py", line 38, in inline
    return cython_inline(f, *args, **kwds)
  File "/tmp/buildd/cython-0.17~beta1/build/work-dir/Cy3/Cython/Build/Inline.py", line 194, in cython_inline
    module = imp.load_dynamic(module_name, module_path)
ImportError: /tmp/cython_inline_k09hl9/_cython_inline_a109c4f12b08057a17f76ac1d72d5de0.cpython-32mu-x86_64-linux-gnu.so: cannot open shared object file: No such file or directory

----------------------------------------------------------------------
Ran 6706 tests in 1703.655s

FAILED (errors=6)


On Mon, 23 Jul 2012, Stefan Behnel wrote:

> Hello everyone,

> on behalf of the Cython project team, I'm proud to announce the release of
> our first beta of Cython 0.17. This is another major step forward in the
> development of the language that will make life easier for a lot of users,
> rounds up some rough edges of the compiler and adds (preliminary) support
> for CPython 3.3 and PyPy.

> Download: http://cython.org/release/Cython-0.17.beta1.tar.gz

> Release notes: http://wiki.cython.org/ReleaseNotes-0.17

> Documentation: http://docs.cython.org/


> Major features of this release include:

> * vastly improved integration with the C++ STL containers

>   http://docs.cython.org/src/userguide/wrapping_CPlusPlus.html#standard-library

>   http://docs.cython.org/src/tutorial/strings.html#c-strings

> * "yield from" delegation between generators (PEP 380)

>   http://www.python.org/dev/peps/pep-0380/

> * alpha quality support for PyPy (via cpyext)

>   http://docs.cython.org/src/tutorial/pypy.html


> Several other features and improvements are listed in the release notes:

>   http://wiki.cython.org/ReleaseNotes-0.17


> I'm expecting at least one release candidate to follow on this beta
> version, and a final release in early August. Please give this beta release
> as much testing as you can, so that we can quickly advance towards the
> final release.


> Have fun,

> Stefan
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel


-- 
Yaroslav O. Halchenko
Postdoctoral Fellow,   Department of Psychological and Brain Sciences
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834                       Fax: +1 (603) 646-1419
WWW:   http://www.linkedin.com/in/yarik        

From stefan_ml at behnel.de  Wed Jul 25 19:41:32 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 25 Jul 2012 19:41:32 +0200
Subject: [Cython] Cython 0.17 beta 1 released
In-Reply-To: <20120725165055.GN5788@onerussian.com>
References: <500DB7FA.4000409@behnel.de> <20120725165055.GN5788@onerussian.com>
Message-ID: <50102FCC.6000002@behnel.de>

Yaroslav Halchenko, 25.07.2012 18:50:
> Congrats on the beta-release!  While testing an updated debian package
> for cython I have ran into failures with Python 3.2.3 (default, Jul 13 2012,
> 21:02:37) [GCC 4.7.1] (complete log:
> http://neuro.debian.net/_files/_buildlogs/cython/0.17~beta1/cython_0.17~beta1-1_amd64.build)

Thanks for testing it.


> anything familiar? (I see PY3 fix bf7981fb37b19f08a331c704df8bf25d3b299be5 but
> it doesn't look relevant for this one, or am I wrong?)

No, I haven't seen this before.


> ======================================================================
> ERROR: test_globals (Cython.Build.Tests.TestInline.TestInline)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File "/tmp/buildd/cython-0.17~beta1/build/work-dir/Cy3/Cython/Build/Tests/TestInline.py", line 41, in test_globals
>     self.assertEquals(inline("return global_value + 1", **self.test_kwds), global_value + 1)
>   File "/tmp/buildd/cython-0.17~beta1/build/work-dir/Cy3/Cython/Shadow.py", line 38, in inline
>     return cython_inline(f, *args, **kwds)
>   File "/tmp/buildd/cython-0.17~beta1/build/work-dir/Cy3/Cython/Build/Inline.py", line 194, in cython_inline
>     module = imp.load_dynamic(module_name, module_path)
> ImportError: /tmp/cython_inline_zo_0tz/_cython_inline_d0fe156ce72658e73c2b9a9438fd8d6a.cpython-32mu-x86_64-linux-gnu.so: cannot open shared object file: No such file or directory

That's weird, and I don't think this is Cython's fault. According to the
logs, the shared library file that it built is called

_cython_inline_d0fe156ce72658e73c2b9a9438fd8d6a.cpython-32mu.so

i.e. the "cpython-32mu-x86_64-linux-gnu" should be "cpython-32mu".

The module path that Cython tries (and fails) to load the module from is
built like this in the code:

"""
    so_ext = [ ext for ext,_,mod_type in imp.get_suffixes()
               if mod_type == imp.C_EXTENSION ][0]
    module_path = os.path.join(lib_dir, module_name+so_ext)
"""

So, apparently, imp.get_suffixes() returns an unexpected result here. Could
you figure out what exactly that function returns in your Py3.2 build
environment? The same code works for all other environments, and it also
works for our own Py3.2 test build:

https://sage.math.washington.edu:8091/hudson/job/cython-devel-build/PYVERSION=py32/572/testReport/Cython.Build.Tests.TestInline/

Stefan


From lists at onerussian.com  Wed Jul 25 21:02:27 2012
From: lists at onerussian.com (Yaroslav Halchenko)
Date: Wed, 25 Jul 2012 15:02:27 -0400
Subject: [Cython] Cython 0.17 beta 1 released
In-Reply-To: <50102FCC.6000002@behnel.de>
References: <500DB7FA.4000409@behnel.de> <20120725165055.GN5788@onerussian.com>
	<50102FCC.6000002@behnel.de>
Message-ID: <20120725190227.GO5788@onerussian.com>

NB Sorry for a lengthy reply -- more like notes for myself I guess ;)
   Short story -- imp.get_suffixes()[0] != get_config_var('SO')
   on Debian multiarch sid
   Perspective patch is at the bottom.

On Wed, 25 Jul 2012, Stefan Behnel wrote:

> > anything familiar? (I see PY3 fix bf7981fb37b19f08a331c704df8bf25d3b299be5 but
> > it doesn't look relevant for this one, or am I wrong?)

> No, I haven't seen this before.

cool -- I am glad then to bring  the freshiest stuff to the table ;)

> > ImportError: /tmp/cython_inline_zo_0tz/_cython_inline_d0fe156ce72658e73c2b9a9438fd8d6a.cpython-32mu-x86_64-linux-gnu.so: cannot open shared object file: No such file or directory

> That's weird, and I don't think this is Cython's fault. According to the
> logs, the shared library file that it built is called

> _cython_inline_d0fe156ce72658e73c2b9a9438fd8d6a.cpython-32mu.so

> i.e. the "cpython-32mu-x86_64-linux-gnu" should be "cpython-32mu".

Thanks for the analysis!  Additional suffix comes from

    $> cc --print-multiarch
    x86_64-linux-gnu

for multiarch Debian support

> The module path that Cython tries (and fails) to load the module from is
> built like this in the code:

> """
>     so_ext = [ ext for ext,_,mod_type in imp.get_suffixes()
>                if mod_type == imp.C_EXTENSION ][0]

just a note: [0] -- so only the first possible one is considered


> So, apparently, imp.get_suffixes() returns an unexpected result here. Could
> you figure out what exactly that function returns in your Py3.2 build
> environment? 

here it is in my local environment:

    $> python3 -c 'import imp; print(imp.get_suffixes())' 
    [('.cpython-32mu.so', 'rb', 3), ('module.cpython-32mu.so', 'rb', 3), ('.abi3.so', 'rb', 3), ('module.abi3.so', 'rb', 3), ('.so', 'rb', 3), ('module.so', 'rb', 3), ('.py', 'U', 1), ('.pyc', 'rb', 2)]

BUT it is with python 3.2.3-2 while sid (the build environment) was with
up-to-date python 3.2.3-3, which according to
http://packages.debian.org/changelogs/pool/main/p/python3.2/python3.2_3.2.3-3/changelog
did:

   * Lookup extension modules with a multiarch suffix too.

updated my local env to match the build env:

    $> python3 -c 'import imp; print(imp.get_suffixes())'
    [('.cpython-32mu-x86_64-linux-gnu.so', 'rb', 3), ('.cpython-32mu.so', 'rb', 3), ('module.cpython-32mu-x86_64-linux-gnu.so', 'rb', 3), ('module.cpython-32mu.so', 'rb', 3), ('.abi3.so', 'rb', 3), ('module.abi3.so', 'rb', 3), ('.so', 'rb', 3), ('module.so', 'rb', 3), ('.py', 'U', 1), ('.pyc', 'rb', 2)]

so here we get it ;):

    (gdb) py-down
    #3 Frame 0x2814760, for file /home/yoh/deb/gits/cython/build/work-dir/Cy3/Cython/Build/Inline.py, line 194, in cython_inline (code='return global_value + 1', get_type=<function at remote 0x260b4b0>, lib_dir='/home/yoh/.tmp/cython_inline_mr_ml9', cython_include_dirs=['.'], force=True, quiet=True, locals={'self': <TestInline(_outcomeForDoCleanups=<_Outcome(unexpectedSuccess=None, errors=[], success=True, expectedFailure=None, skipped=None, failures=[]) at remote 0x297d290>, _testMethodName='test_globals', test_kwds={'lib_dir': '/home/yoh/.tmp/cython_inline_mr_ml9', 'force': True, 'quiet': True}, _cleanups=[], _type_equality_funcs={<type at remote 0x9aac20>: 'assertDictEqual', <type at remote 0x8a54a0>: 'assertMultiLineEqual', <type at remote 0x89fb60>: 'assertTupleEqual', <type at remote 0x89ebc0>: 'assertSetEqual', <type at remote 0x89f0a0>: 'assertSetEqual', <type at remote 0x9a8de0>: 'assertListEqual'}, listing_file=None, _testMethodDoc=None, echo_file=None) at remote 0x260cfb0>}, globals={'CythonTest': <type a...(truncated)
        module = imp.load_dynamic(module_name, module_path)
    (gdb) py-print module_name
    local 'module_name' = '_cython_inline_7bc6bab54c5c46dc2ccbec2dc048aa42'
    (gdb) py-print module_path
    local 'module_path' = '/home/yoh/.tmp/cython_inline_mr_ml9/_cython_inline_7bc6bab54c5c46dc2ccbec2dc048aa42.cpython-32dmu-x86_64-linux-gnu.so'

while there is only:

    $> ls -l /home/yoh/.tmp/cython_inline_mr_ml9/_cython_inline_7bc6bab54c5c46dc2ccbec2dc048aa42.cpython-32dmu*.so                
    -rwx------ 1 yoh yoh 49027 Jul 25 14:22 /home/yoh/.tmp/cython_inline_mr_ml9/_cython_inline_7bc6bab54c5c46dc2ccbec2dc048aa42.cpython-32dmu.so*

So -- there is a dichotomy between what is the first in the list of
imp.get_suffixes() and what is the default extension filename from  build_ext
which is based on

    (Pdb) from distutils.sysconfig import get_config_var
    (Pdb) get_config_var('SO')
    '.cpython-32dmu.so'

used by .get_ext_filename():

    (Pdb) print build_extension.get_ext_filename(module_name)
    '_cython_inline_7bc6bab54c5c46dc2ccbec2dc048aa42.cpython-32dmu.so'


So I wonder, wouldn't it be reasonable (i.e. more robust) in cython_inline  to
instantiate first build_extension and seek full name for the resultant
extension from it?  That should eliminate any possibility to get different
names.

e.g. smth like:


$> quilt diff
--- a/Cython/Build/Inline.py
+++ b/Cython/Build/Inline.py
@@ -139,8 +139,15 @@ def cython_inline(code,
     key = orig_code, arg_sigs, sys.version_info, sys.executable, Cython.__version__
     module_name = "_cython_inline_" + hashlib.md5(str(key).encode('utf-8')).hexdigest()
 
-    so_ext = [ ext for ext,_,mod_type in imp.get_suffixes() if mod_type == imp.C_EXTENSION ][0]
-    module_path = os.path.join(lib_dir, module_name+so_ext)
+    dist = Distribution()
+    # Ensure the build respects distutils configuration by parsing
+    # the configuration files
+    config_files = dist.find_config_files()
+    dist.parse_config_files(config_files)
+    build_extension = build_ext(dist)
+    build_extension.finalize_options()
+
+    module_path = os.path.join(lib_dir, build_extension.get_ext_filename(module_name))
 
     if not os.path.exists(lib_dir):
         os.makedirs(lib_dir)
@@ -178,13 +185,6 @@ def __invoke(%(params)s):
             sources = [pyx_file],
             include_dirs = c_include_dirs,
             extra_compile_args = cflags)
-        dist = Distribution()
-        # Ensure the build respects distutils configuration by parsing
-        # the configuration files
-        config_files = dist.find_config_files()
-        dist.parse_config_files(config_files)
-        build_extension = build_ext(dist)
-        build_extension.finalize_options()
         build_extension.extensions = cythonize([extension], ctx=ctx, quiet=quiet)
         build_extension.build_temp = os.path.dirname(pyx_file)
         build_extension.build_lib  = lib_dir


-- 
Yaroslav O. Halchenko
Postdoctoral Fellow,   Department of Psychological and Brain Sciences
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834                       Fax: +1 (603) 646-1419
WWW:   http://www.linkedin.com/in/yarik        

From robertwb at gmail.com  Wed Jul 25 22:58:37 2012
From: robertwb at gmail.com (Robert Bradshaw)
Date: Wed, 25 Jul 2012 13:58:37 -0700
Subject: [Cython] Cython 0.17 beta 1 released
In-Reply-To: <20120725190227.GO5788@onerussian.com>
References: <500DB7FA.4000409@behnel.de> <20120725165055.GN5788@onerussian.com>
	<50102FCC.6000002@behnel.de> <20120725190227.GO5788@onerussian.com>
Message-ID: <CADiQ+QBk=Z4oF7_hw2dP4Hbdcavf1LVAkiM3gZWpn2Q-KrAjTQ@mail.gmail.com>

Thanks for the report!

On Wed, Jul 25, 2012 at 12:02 PM, Yaroslav Halchenko
<lists at onerussian.com> wrote:
> So I wonder, wouldn't it be reasonable (i.e. more robust) in cython_inline  to
> instantiate first build_extension and seek full name for the resultant
> extension from it?  That should eliminate any possibility to get different
> names.
>
> e.g. smth like:
>
>
> $> quilt diff
> --- a/Cython/Build/Inline.py
> +++ b/Cython/Build/Inline.py
> @@ -139,8 +139,15 @@ def cython_inline(code,
>      key = orig_code, arg_sigs, sys.version_info, sys.executable, Cython.__version__
>      module_name = "_cython_inline_" + hashlib.md5(str(key).encode('utf-8')).hexdigest()
>
> -    so_ext = [ ext for ext,_,mod_type in imp.get_suffixes() if mod_type == imp.C_EXTENSION ][0]
> -    module_path = os.path.join(lib_dir, module_name+so_ext)
> +    dist = Distribution()
> +    # Ensure the build respects distutils configuration by parsing
> +    # the configuration files
> +    config_files = dist.find_config_files()
> +    dist.parse_config_files(config_files)
> +    build_extension = build_ext(dist)
> +    build_extension.finalize_options()
> +
> +    module_path = os.path.join(lib_dir, build_extension.get_ext_filename(module_name))
>
>      if not os.path.exists(lib_dir):
>          os.makedirs(lib_dir)
> @@ -178,13 +185,6 @@ def __invoke(%(params)s):
>              sources = [pyx_file],
>              include_dirs = c_include_dirs,
>              extra_compile_args = cflags)
> -        dist = Distribution()
> -        # Ensure the build respects distutils configuration by parsing
> -        # the configuration files
> -        config_files = dist.find_config_files()
> -        dist.parse_config_files(config_files)
> -        build_extension = build_ext(dist)
> -        build_extension.finalize_options()
>          build_extension.extensions = cythonize([extension], ctx=ctx, quiet=quiet)
>          build_extension.build_temp = os.path.dirname(pyx_file)
>          build_extension.build_lib  = lib_dir

One essential feature of cython.inline(...) is that if the code has
already been compiled (and loaded) it should return very fast. This
would seem to add significant overhead. Is the extension relatively
consistant? Perhaps it could be cached at module load time.

- Robert

From lists at onerussian.com  Thu Jul 26 00:36:10 2012
From: lists at onerussian.com (Yaroslav Halchenko)
Date: Wed, 25 Jul 2012 18:36:10 -0400
Subject: [Cython] Cython 0.17 beta 1 released
In-Reply-To: <CADiQ+QBk=Z4oF7_hw2dP4Hbdcavf1LVAkiM3gZWpn2Q-KrAjTQ@mail.gmail.com>
References: <500DB7FA.4000409@behnel.de> <20120725165055.GN5788@onerussian.com>
	<50102FCC.6000002@behnel.de> <20120725190227.GO5788@onerussian.com>
	<CADiQ+QBk=Z4oF7_hw2dP4Hbdcavf1LVAkiM3gZWpn2Q-KrAjTQ@mail.gmail.com>
Message-ID: <20120725223557.GP5788@onerussian.com>


On Wed, 25 Jul 2012, Robert Bradshaw wrote:

> One essential feature of cython.inline(...) is that if the code has
> already been compiled (and loaded) it should return very fast. This
> would seem to add significant overhead. 

that is what was my concern also with such an approach...  I am not sure
if that is a significant overhead -- on my laptop (python 2.7):

In [13]: !cat test_get_ext_time.py
from distutils.core import Distribution
from distutils.command.build_ext import build_ext

def get_ext_filename(module_name):
    dist = Distribution()
    # Ensure the build respects distutils configuration by parsing
    # the configuration files
    config_files = dist.find_config_files()
    dist.parse_config_files(config_files)
    build_extension = build_ext(dist)
    build_extension.finalize_options()

    return build_extension.get_ext_filename(module_name)

In [14]: %run test_get_ext_time.py

In [15]: %timeit get_ext_filename('/asd/f.asdf/asdf/asd/fasdf/xx')
1000 loops, best of 3: 301 us per loop

of cause it is a relatively big slowdown relatively to 3ms of solution
based on imp.get_suffixes ...  and with cython 0.16 full dummy inline:

In [4]: %timeit cython_inline('i=1')
1000 loops, best of 3: 445 us per loop

so I guess indeed 300 us would be a significant overhead

> Is the extension relatively
> consistant? Perhaps it could be cached at module load time.

could indeed be especially given the code of

    def get_ext_filename(self, ext_name):
        r"""Convert the name of an extension (eg. "foo.bar") into the name
        of the file from which it will be loaded (eg. "foo/bar.so", or
        "foo\bar.pyd").
        """
        from distutils.sysconfig import get_config_var
        ext_path = ext_name.split('.')
        # OS/2 has an 8 character module (extension) limit :-(
        if os.name == "os2":
            ext_path[len(ext_path) - 1] = ext_path[len(ext_path) - 1][:8]
        # extensions in debug_mode are named 'module_d.pyd' under windows
        so_ext = get_config_var('SO')
        if os.name == 'nt' and self.debug:
            return os.path.join(*ext_path) + '_d' + so_ext
        return os.path.join(*ext_path) + so_ext

suggesting only get_config_var and 1 self.debug variable which could affect .. which are probably would not be changed at run time.  Paranoid me
though would still have added some check when extension does get built to
verify that assumption was right and then spit out a warning and adjust
module_name to match so that load_dynamic  doesn't fail... or would that be too
much?

would you like me to prep a perspective path or you would like to do that? ;)

-- 
Yaroslav O. Halchenko
Postdoctoral Fellow,   Department of Psychological and Brain Sciences
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834                       Fax: +1 (603) 646-1419
WWW:   http://www.linkedin.com/in/yarik        

From lists at onerussian.com  Thu Jul 26 03:12:37 2012
From: lists at onerussian.com (Yaroslav Halchenko)
Date: Wed, 25 Jul 2012 21:12:37 -0400
Subject: [Cython] Cython 0.17 beta 1 released
In-Reply-To: <20120725223557.GP5788@onerussian.com>
References: <500DB7FA.4000409@behnel.de> <20120725165055.GN5788@onerussian.com>
	<50102FCC.6000002@behnel.de> <20120725190227.GO5788@onerussian.com>
	<CADiQ+QBk=Z4oF7_hw2dP4Hbdcavf1LVAkiM3gZWpn2Q-KrAjTQ@mail.gmail.com>
	<20120725223557.GP5788@onerussian.com>
Message-ID: <20120726011236.GQ5788@onerussian.com>

actually I have not stated alternative variant since I thought it would
not be wise to 'waste' memory :  just store association between a
particular build and target module_name but now I have mentioned that
such code is pretty much there ... but incorrect and not used:

$> grep -e '\Wkey\W' -e '^def cython_inline' -e 'code_cache' Cython/Build/Inline.py 
_code_cache = {}
def cython_inline(code,
    key = orig_code, arg_sigs, sys.version_info, sys.executable, Cython.__version__
    module_name = "_cython_inline_" + hashlib.md5(str(key).encode('utf-8')).hexdigest()
        for key, value in literals.items():
            module_code = module_code.replace(key, value)
        _code_cache[key] = module_name


so 
1. key in for loop overrides the key tuple identifying the module_path
2. _code_cache is not used anywhere (but does waste memory although probably
   not much since there is not that many values of key I guess which
   it would get)

thus I wondered if it is ok to waste memory ( ;) ) should following patch be
used?  or should _code_cache be removed altogether (or am I missing its role?)

$> quilt diff
--- a/Cython/Build/Inline.py
+++ b/Cython/Build/Inline.py
@@ -138,13 +138,11 @@ def cython_inline(code,
     arg_sigs = tuple([(get_type(kwds[arg], ctx), arg) for arg in arg_names])
     key = orig_code, arg_sigs, sys.version_info, sys.executable, Cython.__version__
     module_name = "_cython_inline_" + hashlib.md5(str(key).encode('utf-8')).hexdigest()
-
-    so_ext = [ ext for ext,_,mod_type in imp.get_suffixes() if mod_type == imp.C_EXTENSION ][0]
-    module_path = os.path.join(lib_dir, module_name+so_ext)
+    module_path = _code_cache.get(module_name, None)
 
     if not os.path.exists(lib_dir):
         os.makedirs(lib_dir)
-    if force or not os.path.isfile(module_path):
+    if force or module_path is None or not os.path.isfile(module_path):
         cflags = []
         c_include_dirs = []
         qualified = re.compile(r'([.\w]+)[.]')
@@ -189,7 +187,9 @@ def __invoke(%(params)s):
         build_extension.build_temp = os.path.dirname(pyx_file)
         build_extension.build_lib  = lib_dir
         build_extension.run()
-        _code_cache[key] = module_name
+        # Should we check if module_path is not None either it still matches?
+        module_path = os.path.join(lib_dir, build_extension.get_ext_filename(module_name))
+        _code_cache[module_name] = module_path
 
     module = imp.load_dynamic(module_name, module_path)
     arg_list = [kwds[arg] for arg in arg_names]




On Wed, 25 Jul 2012, Yaroslav Halchenko wrote:


> On Wed, 25 Jul 2012, Robert Bradshaw wrote:

> > One essential feature of cython.inline(...) is that if the code has
> > already been compiled (and loaded) it should return very fast. This
> > would seem to add significant overhead. 

> that is what was my concern also with such an approach...  I am not sure
> if that is a significant overhead -- on my laptop (python 2.7):

> In [13]: !cat test_get_ext_time.py
> from distutils.core import Distribution
> from distutils.command.build_ext import build_ext

> def get_ext_filename(module_name):
>     dist = Distribution()
>     # Ensure the build respects distutils configuration by parsing
>     # the configuration files
>     config_files = dist.find_config_files()
>     dist.parse_config_files(config_files)
>     build_extension = build_ext(dist)
>     build_extension.finalize_options()

>     return build_extension.get_ext_filename(module_name)

> In [14]: %run test_get_ext_time.py

> In [15]: %timeit get_ext_filename('/asd/f.asdf/asdf/asd/fasdf/xx')
> 1000 loops, best of 3: 301 us per loop

> of cause it is a relatively big slowdown relatively to 3ms of solution
> based on imp.get_suffixes ...  and with cython 0.16 full dummy inline:

> In [4]: %timeit cython_inline('i=1')
> 1000 loops, best of 3: 445 us per loop

> so I guess indeed 300 us would be a significant overhead

> > Is the extension relatively
> > consistant? Perhaps it could be cached at module load time.

> could indeed be especially given the code of

>     def get_ext_filename(self, ext_name):
>         r"""Convert the name of an extension (eg. "foo.bar") into the name
>         of the file from which it will be loaded (eg. "foo/bar.so", or
>         "foo\bar.pyd").
>         """
>         from distutils.sysconfig import get_config_var
>         ext_path = ext_name.split('.')
>         # OS/2 has an 8 character module (extension) limit :-(
>         if os.name == "os2":
>             ext_path[len(ext_path) - 1] = ext_path[len(ext_path) - 1][:8]
>         # extensions in debug_mode are named 'module_d.pyd' under windows
>         so_ext = get_config_var('SO')
>         if os.name == 'nt' and self.debug:
>             return os.path.join(*ext_path) + '_d' + so_ext
>         return os.path.join(*ext_path) + so_ext

> suggesting only get_config_var and 1 self.debug variable which could affect .. which are probably would not be changed at run time.  Paranoid me
> though would still have added some check when extension does get built to
> verify that assumption was right and then spit out a warning and adjust
> module_name to match so that load_dynamic  doesn't fail... or would that be too
> much?

> would you like me to prep a perspective path or you would like to do that? ;)
-- 
Yaroslav O. Halchenko
Postdoctoral Fellow,   Department of Psychological and Brain Sciences
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834                       Fax: +1 (603) 646-1419
WWW:   http://www.linkedin.com/in/yarik        

From robertwb at gmail.com  Thu Jul 26 03:49:08 2012
From: robertwb at gmail.com (Robert Bradshaw)
Date: Wed, 25 Jul 2012 18:49:08 -0700
Subject: [Cython] Cython 0.17 beta 1 released
In-Reply-To: <20120726011236.GQ5788@onerussian.com>
References: <500DB7FA.4000409@behnel.de> <20120725165055.GN5788@onerussian.com>
	<50102FCC.6000002@behnel.de> <20120725190227.GO5788@onerussian.com>
	<CADiQ+QBk=Z4oF7_hw2dP4Hbdcavf1LVAkiM3gZWpn2Q-KrAjTQ@mail.gmail.com>
	<20120725223557.GP5788@onerussian.com>
	<20120726011236.GQ5788@onerussian.com>
Message-ID: <CADiQ+QDB0B1XOWN9ekVDjFf2gbFCq_2hSLMN6pW5mdAe163KHA@mail.gmail.com>

On Wed, Jul 25, 2012 at 6:12 PM, Yaroslav Halchenko
<lists at onerussian.com> wrote:
> actually I have not stated alternative variant since I thought it would
> not be wise to 'waste' memory :  just store association between a
> particular build and target module_name but now I have mentioned that
> such code is pretty much there ... but incorrect and not used:
>
> $> grep -e '\Wkey\W' -e '^def cython_inline' -e 'code_cache' Cython/Build/Inline.py
> _code_cache = {}
> def cython_inline(code,
>     key = orig_code, arg_sigs, sys.version_info, sys.executable, Cython.__version__
>     module_name = "_cython_inline_" + hashlib.md5(str(key).encode('utf-8')).hexdigest()
>         for key, value in literals.items():
>             module_code = module_code.replace(key, value)
>         _code_cache[key] = module_name
>
>
> so
> 1. key in for loop overrides the key tuple identifying the module_path
> 2. _code_cache is not used anywhere (but does waste memory although probably
>    not much since there is not that many values of key I guess which
>    it would get)
>
> thus I wondered if it is ok to waste memory ( ;) ) should following patch be
> used?  or should _code_cache be removed altogether (or am I missing its role?)
>
> $> quilt diff
> --- a/Cython/Build/Inline.py
> +++ b/Cython/Build/Inline.py
> @@ -138,13 +138,11 @@ def cython_inline(code,
>      arg_sigs = tuple([(get_type(kwds[arg], ctx), arg) for arg in arg_names])
>      key = orig_code, arg_sigs, sys.version_info, sys.executable, Cython.__version__
>      module_name = "_cython_inline_" + hashlib.md5(str(key).encode('utf-8')).hexdigest()
> -
> -    so_ext = [ ext for ext,_,mod_type in imp.get_suffixes() if mod_type == imp.C_EXTENSION ][0]
> -    module_path = os.path.join(lib_dir, module_name+so_ext)
> +    module_path = _code_cache.get(module_name, None)
>
>      if not os.path.exists(lib_dir):
>          os.makedirs(lib_dir)
> -    if force or not os.path.isfile(module_path):
> +    if force or module_path is None or not os.path.isfile(module_path):
>          cflags = []
>          c_include_dirs = []
>          qualified = re.compile(r'([.\w]+)[.]')
> @@ -189,7 +187,9 @@ def __invoke(%(params)s):
>          build_extension.build_temp = os.path.dirname(pyx_file)
>          build_extension.build_lib  = lib_dir
>          build_extension.run()
> -        _code_cache[key] = module_name
> +        # Should we check if module_path is not None either it still matches?
> +        module_path = os.path.join(lib_dir, build_extension.get_ext_filename(module_name))
> +        _code_cache[module_name] = module_path
>
>      module = imp.load_dynamic(module_name, module_path)
>      arg_list = [kwds[arg] for arg in arg_names]

Compiled modules can persist between sessions as well.

I like your solution of caching the module name, what if we computed
the module name iff it wasn't in the cache, then compiled the file iff
the .so file didn't exist (with a check that the module name was OK).

Alternatively, could we just rename the compiled library to be what we
expect if it wasn't already?

- Robert

From lists at onerussian.com  Thu Jul 26 04:08:41 2012
From: lists at onerussian.com (Yaroslav Halchenko)
Date: Wed, 25 Jul 2012 22:08:41 -0400
Subject: [Cython] Cython 0.17 beta 1 released
In-Reply-To: <CADiQ+QDB0B1XOWN9ekVDjFf2gbFCq_2hSLMN6pW5mdAe163KHA@mail.gmail.com>
References: <500DB7FA.4000409@behnel.de> <20120725165055.GN5788@onerussian.com>
	<50102FCC.6000002@behnel.de> <20120725190227.GO5788@onerussian.com>
	<CADiQ+QBk=Z4oF7_hw2dP4Hbdcavf1LVAkiM3gZWpn2Q-KrAjTQ@mail.gmail.com>
	<20120725223557.GP5788@onerussian.com>
	<20120726011236.GQ5788@onerussian.com>
	<CADiQ+QDB0B1XOWN9ekVDjFf2gbFCq_2hSLMN6pW5mdAe163KHA@mail.gmail.com>
Message-ID: <20120726020841.GR5788@onerussian.com>


On Wed, 25 Jul 2012, Robert Bradshaw wrote:

> >      module = imp.load_dynamic(module_name, module_path)
> >      arg_list = [kwds[arg] for arg in arg_names]

> Compiled modules can persist between sessions as well.

yeah -- figured it down also while working on another version of this
trivial patch ;)

> I like your solution of caching the module name, what if we computed
> the module name iff it wasn't in the cache, then compiled the file iff
> the .so file didn't exist (with a check that the module name was OK).

> Alternatively, could we just rename the compiled library to be what we
> expect if it wasn't already?

we could do that ... but altogether -- do you really like caching the
names?  imho it is somewhat wasteful for long-running interactive
sessions where people might try different things.   Ok -- here is my next
version (stop me eventually) which just does what you wanted -- cache the
ultimate suffix under assumption that it would not change (also removed unused
_code_cache) (it came out a bit longer simply due to me adding helper function
_get_build_extension() to avoid duplication):

--- a/Cython/Build/Inline.py
+++ b/Cython/Build/Inline.py
@@ -29,8 +29,6 @@ if sys.version_info[0] < 3:
 else:
     to_unicode = lambda x: x
 
-_code_cache = {}
-
 
 class AllSymbols(CythonTransform, SkipDeclarations):
     def __init__(self):
@@ -94,6 +92,16 @@ def safe_type(arg, context=None):
                     return '%s.%s' % (base_type.__module__, base_type.__name__)
         return 'object'
 
+def _get_build_extension():
+    dist = Distribution()
+    # Ensure the build respects distutils configuration by parsing
+    # the configuration files
+    config_files = dist.find_config_files()
+    dist.parse_config_files(config_files)
+    build_extension = build_ext(dist)
+    build_extension.finalize_options()
+    return build_extension
+
 def cython_inline(code,
                   get_type=unsafe_type,
                   lib_dir=os.path.join(get_cython_cache_dir(), 'inline'),
@@ -139,8 +147,13 @@ def cython_inline(code,
     key = orig_code, arg_sigs, sys.version_info, sys.executable, Cython.__version__
     module_name = "_cython_inline_" + hashlib.md5(str(key).encode('utf-8')).hexdigest()
 
-    so_ext = [ ext for ext,_,mod_type in imp.get_suffixes() if mod_type == imp.C_EXTENSION ][0]
-    module_path = os.path.join(lib_dir, module_name+so_ext)
+    build_extension = None
+    if cython_inline.so_ext is None:
+        # Figure out and cache current extension suffix
+        build_extension = _get_build_extension()
+        cython_inline.so_ext = build_extension.get_ext_filename('')
+
+    module_path = os.path.join(lib_dir, module_name + cython_inline.so_ext)
 
     if not os.path.exists(lib_dir):
         os.makedirs(lib_dir)
@@ -178,23 +191,21 @@ def __invoke(%(params)s):
             sources = [pyx_file],
             include_dirs = c_include_dirs,
             extra_compile_args = cflags)
-        dist = Distribution()
-        # Ensure the build respects distutils configuration by parsing
-        # the configuration files
-        config_files = dist.find_config_files()
-        dist.parse_config_files(config_files)
-        build_extension = build_ext(dist)
-        build_extension.finalize_options()
+        if build_extension is None:
+            build_extension = _get_build_extension()
         build_extension.extensions = cythonize([extension], ctx=ctx, quiet=quiet)
         build_extension.build_temp = os.path.dirname(pyx_file)
         build_extension.build_lib  = lib_dir
         build_extension.run()
-        _code_cache[key] = module_name
 
     module = imp.load_dynamic(module_name, module_path)
     arg_list = [kwds[arg] for arg in arg_names]
     return module.__invoke(*arg_list)
 
+# Cached suffix used by cython_inline above.  None should get
+# overridden with actual value upon the first cython_inline invocation
+cython_inline.so_ext = None
+
 non_space = re.compile('[^ ]')
 def strip_common_indent(code):
     min_indent = None


-- 
Yaroslav O. Halchenko
Postdoctoral Fellow,   Department of Psychological and Brain Sciences
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834                       Fax: +1 (603) 646-1419
WWW:   http://www.linkedin.com/in/yarik        

From robertwb at gmail.com  Thu Jul 26 04:23:59 2012
From: robertwb at gmail.com (Robert Bradshaw)
Date: Wed, 25 Jul 2012 19:23:59 -0700
Subject: [Cython] Cython 0.17 beta 1 released
In-Reply-To: <20120726020841.GR5788@onerussian.com>
References: <500DB7FA.4000409@behnel.de> <20120725165055.GN5788@onerussian.com>
	<50102FCC.6000002@behnel.de> <20120725190227.GO5788@onerussian.com>
	<CADiQ+QBk=Z4oF7_hw2dP4Hbdcavf1LVAkiM3gZWpn2Q-KrAjTQ@mail.gmail.com>
	<20120725223557.GP5788@onerussian.com>
	<20120726011236.GQ5788@onerussian.com>
	<CADiQ+QDB0B1XOWN9ekVDjFf2gbFCq_2hSLMN6pW5mdAe163KHA@mail.gmail.com>
	<20120726020841.GR5788@onerussian.com>
Message-ID: <CADiQ+QATnn-KEMA_kmcBiyBx++2rN44Cb3C_ZsU=bT4p3NBoXA@mail.gmail.com>

On Wed, Jul 25, 2012 at 7:08 PM, Yaroslav Halchenko
<lists at onerussian.com> wrote:
>
> On Wed, 25 Jul 2012, Robert Bradshaw wrote:
>
>> >      module = imp.load_dynamic(module_name, module_path)
>> >      arg_list = [kwds[arg] for arg in arg_names]
>
>> Compiled modules can persist between sessions as well.
>
> yeah -- figured it down also while working on another version of this
> trivial patch ;)
>
>> I like your solution of caching the module name, what if we computed
>> the module name iff it wasn't in the cache, then compiled the file iff
>> the .so file didn't exist (with a check that the module name was OK).
>
>> Alternatively, could we just rename the compiled library to be what we
>> expect if it wasn't already?
>
> we could do that ... but altogether -- do you really like caching the
> names?  imho it is somewhat wasteful for long-running interactive
> sessions where people might try different things.   Ok -- here is my next
> version (stop me eventually) which just does what you wanted -- cache the
> ultimate suffix under assumption that it would not change (also removed unused
> _code_cache) (it came out a bit longer simply due to me adding helper function
> _get_build_extension() to avoid duplication):

Looks good. Thanks. File a pull request and we'll merge it in.

> --- a/Cython/Build/Inline.py
> +++ b/Cython/Build/Inline.py
> @@ -29,8 +29,6 @@ if sys.version_info[0] < 3:
>  else:
>      to_unicode = lambda x: x
>
> -_code_cache = {}
> -
>
>  class AllSymbols(CythonTransform, SkipDeclarations):
>      def __init__(self):
> @@ -94,6 +92,16 @@ def safe_type(arg, context=None):
>                      return '%s.%s' % (base_type.__module__, base_type.__name__)
>          return 'object'
>
> +def _get_build_extension():
> +    dist = Distribution()
> +    # Ensure the build respects distutils configuration by parsing
> +    # the configuration files
> +    config_files = dist.find_config_files()
> +    dist.parse_config_files(config_files)
> +    build_extension = build_ext(dist)
> +    build_extension.finalize_options()
> +    return build_extension
> +
>  def cython_inline(code,
>                    get_type=unsafe_type,
>                    lib_dir=os.path.join(get_cython_cache_dir(), 'inline'),
> @@ -139,8 +147,13 @@ def cython_inline(code,
>      key = orig_code, arg_sigs, sys.version_info, sys.executable, Cython.__version__
>      module_name = "_cython_inline_" + hashlib.md5(str(key).encode('utf-8')).hexdigest()
>
> -    so_ext = [ ext for ext,_,mod_type in imp.get_suffixes() if mod_type == imp.C_EXTENSION ][0]
> -    module_path = os.path.join(lib_dir, module_name+so_ext)
> +    build_extension = None
> +    if cython_inline.so_ext is None:
> +        # Figure out and cache current extension suffix
> +        build_extension = _get_build_extension()
> +        cython_inline.so_ext = build_extension.get_ext_filename('')
> +
> +    module_path = os.path.join(lib_dir, module_name + cython_inline.so_ext)
>
>      if not os.path.exists(lib_dir):
>          os.makedirs(lib_dir)
> @@ -178,23 +191,21 @@ def __invoke(%(params)s):
>              sources = [pyx_file],
>              include_dirs = c_include_dirs,
>              extra_compile_args = cflags)
> -        dist = Distribution()
> -        # Ensure the build respects distutils configuration by parsing
> -        # the configuration files
> -        config_files = dist.find_config_files()
> -        dist.parse_config_files(config_files)
> -        build_extension = build_ext(dist)
> -        build_extension.finalize_options()
> +        if build_extension is None:
> +            build_extension = _get_build_extension()
>          build_extension.extensions = cythonize([extension], ctx=ctx, quiet=quiet)
>          build_extension.build_temp = os.path.dirname(pyx_file)
>          build_extension.build_lib  = lib_dir
>          build_extension.run()
> -        _code_cache[key] = module_name
>
>      module = imp.load_dynamic(module_name, module_path)
>      arg_list = [kwds[arg] for arg in arg_names]
>      return module.__invoke(*arg_list)
>
> +# Cached suffix used by cython_inline above.  None should get
> +# overridden with actual value upon the first cython_inline invocation
> +cython_inline.so_ext = None
> +
>  non_space = re.compile('[^ ]')
>  def strip_common_indent(code):
>      min_indent = None
>
>
> --
> Yaroslav O. Halchenko
> Postdoctoral Fellow,   Department of Psychological and Brain Sciences
> Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
> Phone: +1 (603) 646-9834                       Fax: +1 (603) 646-1419
> WWW:   http://www.linkedin.com/in/yarik
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

From lists at onerussian.com  Thu Jul 26 04:44:56 2012
From: lists at onerussian.com (Yaroslav Halchenko)
Date: Wed, 25 Jul 2012 22:44:56 -0400
Subject: [Cython] Cython 0.17 beta 1 released
In-Reply-To: <CADiQ+QATnn-KEMA_kmcBiyBx++2rN44Cb3C_ZsU=bT4p3NBoXA@mail.gmail.com>
References: <500DB7FA.4000409@behnel.de> <20120725165055.GN5788@onerussian.com>
	<50102FCC.6000002@behnel.de> <20120725190227.GO5788@onerussian.com>
	<CADiQ+QBk=Z4oF7_hw2dP4Hbdcavf1LVAkiM3gZWpn2Q-KrAjTQ@mail.gmail.com>
	<20120725223557.GP5788@onerussian.com>
	<20120726011236.GQ5788@onerussian.com>
	<CADiQ+QDB0B1XOWN9ekVDjFf2gbFCq_2hSLMN6pW5mdAe163KHA@mail.gmail.com>
	<20120726020841.GR5788@onerussian.com>
	<CADiQ+QATnn-KEMA_kmcBiyBx++2rN44Cb3C_ZsU=bT4p3NBoXA@mail.gmail.com>
Message-ID: <20120726024456.GS5788@onerussian.com>


On Wed, 25 Jul 2012, Robert Bradshaw wrote:
> > ultimate suffix under assumption that it would not change (also removed unused
> > _code_cache) (it came out a bit longer simply due to me adding helper function
> > _get_build_extension() to avoid duplication):

> Looks good. Thanks. File a pull request and we'll merge it in.

thanks for the blessing -- I will do whenever the full build
(+tests against all supported in Debian Python versions) of the patched
cython finishes to tell paranoid me that there is no surprises ;)

-- 
Yaroslav O. Halchenko
Postdoctoral Fellow,   Department of Psychological and Brain Sciences
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834                       Fax: +1 (603) 646-1419
WWW:   http://www.linkedin.com/in/yarik        

From robertwb at gmail.com  Thu Jul 26 06:57:22 2012
From: robertwb at gmail.com (Robert Bradshaw)
Date: Wed, 25 Jul 2012 21:57:22 -0700
Subject: [Cython] Bug, or changed array assignment in 0.17beta1?
In-Reply-To: <500FD760.4080807@canterbury.ac.nz>
References: <CBA537BF-77DF-489C-903F-EC09BD2F2235@berkeley.edu>
	<500F86CF.4080702@behnel.de> <500F9253.4050100@behnel.de>
	<CANg26EUaU6rU96sSFvsVUSc+qo5xv19eRWH3c2kvXh0VVWWExg@mail.gmail.com>
	<500FD760.4080807@canterbury.ac.nz>
Message-ID: <CADiQ+QCR47ZH_POZhL8kLMxWbB3F5OGbX0RxRPE02205Z-nacA@mail.gmail.com>

On Wed, Jul 25, 2012 at 4:24 AM, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
> mark florisson wrote:
>
>> I'm wondering, what was the original motivation to reuse temporaries?
>
> It goes back to Pyrex, where I didn't really give it much
> thought -- it just seemed like the tidiest thing to do.
> Once you have the logic to release temp references as soon
> as it's safe to do so, it's not much harder to return the
> variable to a pool as well.

Somewhat along the lines of what's been said, it also makes cleaning
up any temps (on exceptions or function exits) shorter too, as opposed
to what would be required if every temporary was its own variable.
This cleanup might interfere with the C compiler's ability to alias
the stack variables.

>> Do we ever have C++
>> stack-allocated object temporaries?
>
>
> Theoretically, yes --  temporaries aren't necessarily
> object references, they can be of any type (or they can
> in Pyrex, at least).

Same in Cython, including C++ stack-allocated objects (assuming a
zero-argument constructor), though in this latter case the scope can't
be restricted to less than the entire function body.

This actually relates to one of the annoying C++isms I've run into: an
untouched stack-allocated C++ object will be a "unused variable"
warning if and only if its constructors and destructors are
side-effect free.

- Robert

From stefan_ml at behnel.de  Thu Jul 26 18:50:27 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 26 Jul 2012 18:50:27 +0200
Subject: [Cython] [cython-users] Cython 0.17 beta 1 released
In-Reply-To: <500DB7FA.4000409@behnel.de>
References: <500DB7FA.4000409@behnel.de>
Message-ID: <50117553.6040009@behnel.de>

Stefan Behnel, 23.07.2012 22:45:
> I'm expecting at least one release candidate to follow on this beta
> version, and a final release in early August. Please give this beta release
> as much testing as you can, so that we can quickly advance towards the
> final release.

Any comments from Windows users? We've almost always had a couple of
Windows quirks with our pre-releases, so I'd be surprised if it was
different this time.

Stefan


From stefan_ml at behnel.de  Sat Jul 28 11:37:35 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 28 Jul 2012 11:37:35 +0200
Subject: [Cython] help getting NumPy into a 32-bit Jenkins build environment
Message-ID: <5013B2DF.4020309@behnel.de>

Hi,

I've been trying to set up a 32bit build environment on Jenkins. Most of it
works nicely by just switching to "gcc -m32", but NumPy fails to build with
that setup due to the lack of dependencies. It actually finds them when
looking for their header files in the wrong (64bit) places, and then fails
to build when it can't find the expected 32bit libraries.

Is there a way to tell NumPy that I really don't care about BLAS, LAPACK
and all of its friends and just want bare arrays? Or is there an easy way
to install 32 bit versions of those dependencies on sage.math? I'd be fine
with having them built and installed into a local directory (namely, in
standard directory layout under /home/scoder/jenkins/Tools/libs32/).

Given that all current 32bit problems are in NumPy related tests, it
wouldn't be all that helpful to exclude NumPy from the test runs...

Stefan

From markflorisson88 at gmail.com  Sat Jul 28 14:04:28 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Sat, 28 Jul 2012 13:04:28 +0100
Subject: [Cython] [cython-users] Re: Cython 0.17 beta 1 released
In-Reply-To: <55a9d45a-27ea-428f-9fb6-8a1531b2edac@googlegroups.com>
References: <500DB7FA.4000409@behnel.de>
	<55a9d45a-27ea-428f-9fb6-8a1531b2edac@googlegroups.com>
Message-ID: <CANg26EV3saFJPeHp5JUPbfdZZ86H4NGM12Lrc2xgxniBQmQ_ag@mail.gmail.com>

On 27 July 2012 23:30, Bradley Froehle <brad.froehle at gmail.com> wrote:
> Thanks to the work of Yaroslav Halchenko, there is an experimental Debian
> package for Cython 0.17.beta1 -- http://packages.qa.debian.org/c/cython.html
>
> However, the builds are showing a lot of test failures on non-amd64 sytems.
> See https://buildd.debian.org/status/package.php?p=cython&suite=experimental
> Many of the errors seem related to assumptions made about the sizes of
> various integers (int vs. long vs long long).  Ideally we'll be able to
> clean up these failures before the final release.
>
> -Brad

Thanks, I think it's mostly the tests that are wrong. I'll try to get
it fixed on linux 32 bit.

From stefan_ml at behnel.de  Tue Jul 31 16:24:28 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 31 Jul 2012 16:24:28 +0200
Subject: [Cython] [cython-users] Re: Cython 0.17 beta 1 released
In-Reply-To: <CANg26EV3saFJPeHp5JUPbfdZZ86H4NGM12Lrc2xgxniBQmQ_ag@mail.gmail.com>
References: <500DB7FA.4000409@behnel.de>
	<55a9d45a-27ea-428f-9fb6-8a1531b2edac@googlegroups.com>
	<CANg26EV3saFJPeHp5JUPbfdZZ86H4NGM12Lrc2xgxniBQmQ_ag@mail.gmail.com>
Message-ID: <5017EA9C.8030803@behnel.de>

mark florisson, 28.07.2012 14:04:
> On 27 July 2012 23:30, Bradley Froehle wrote:
>> Thanks to the work of Yaroslav Halchenko, there is an experimental Debian
>> package for Cython 0.17.beta1 -- http://packages.qa.debian.org/c/cython.html
>>
>> However, the builds are showing a lot of test failures on non-amd64 sytems.
>> See https://buildd.debian.org/status/package.php?p=cython&suite=experimental
>> Many of the errors seem related to assumptions made about the sizes of
>> various integers (int vs. long vs long long).  Ideally we'll be able to
>> clean up these failures before the final release.
> 
> Thanks, I think it's mostly the tests that are wrong. I'll try to get
> it fixed on linux 32 bit.

I'm not sure the it's only the tests. The "char" vs. "unsigned char" errors
hint at platform specific differences, which might trigger bugs (aka.
incorrect assumptions) in the generated memory view code. Specifically,
plain "char" is unsigned at least on ARM and PowerPC, both of which fail
with this error.

Stefan


From dalcinl at gmail.com  Tue Jul 31 18:59:24 2012
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Tue, 31 Jul 2012 13:59:24 -0300
Subject: [Cython] help getting NumPy into a 32-bit Jenkins build
	environment
In-Reply-To: <5013B2DF.4020309@behnel.de>
References: <5013B2DF.4020309@behnel.de>
Message-ID: <CAEcYPwAZOmnbG327wLUDdj=+eg9WkyPEMvR_2iXTyo+1xU1vDw@mail.gmail.com>

On 28 July 2012 06:37, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Hi,
>
> I've been trying to set up a 32bit build environment on Jenkins. Most of it
> works nicely by just switching to "gcc -m32", but NumPy fails to build with
> that setup due to the lack of dependencies. It actually finds them when
> looking for their header files in the wrong (64bit) places, and then fails
> to build when it can't find the expected 32bit libraries.
>
> Is there a way to tell NumPy that I really don't care about BLAS, LAPACK
> and all of its friends and just want bare arrays? Or is there an easy way
> to install 32 bit versions of those dependencies on sage.math? I'd be fine
> with having them built and installed into a local directory (namely, in
> standard directory layout under /home/scoder/jenkins/Tools/libs32/).
>

>From doc/source/user/install.rst in numpy-dev:


Disabling ATLAS and other accelerated libraries
-----------------------------------------------

Usage of ATLAS and other accelerated libraries in Numpy can be disabled
via::

    BLAS=None LAPACK=None ATLAS=None python setup.py build




-- 
Lisandro Dalcin
---------------
CIMEC (INTEC/CONICET-UNL)
Predio CONICET-Santa Fe
Colectora RN 168 Km 472, Paraje El Pozo
3000 Santa Fe, Argentina
Tel: +54-342-4511594 (ext 1011)
Tel/Fax: +54-342-4511169

From robertwb at gmail.com  Tue Jul 31 19:24:05 2012
From: robertwb at gmail.com (Robert Bradshaw)
Date: Tue, 31 Jul 2012 10:24:05 -0700
Subject: [Cython] [cython-users] Re: Cython 0.17 beta 1 released
In-Reply-To: <5017EA9C.8030803@behnel.de>
References: <500DB7FA.4000409@behnel.de>
	<55a9d45a-27ea-428f-9fb6-8a1531b2edac@googlegroups.com>
	<CANg26EV3saFJPeHp5JUPbfdZZ86H4NGM12Lrc2xgxniBQmQ_ag@mail.gmail.com>
	<5017EA9C.8030803@behnel.de>
Message-ID: <CADiQ+QCJmAs1ho3stYLY7gM20+gCpAOSYXaiPQb4S7U8u0vm0Q@mail.gmail.com>

On Tue, Jul 31, 2012 at 7:24 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> mark florisson, 28.07.2012 14:04:
>> On 27 July 2012 23:30, Bradley Froehle wrote:
>>> Thanks to the work of Yaroslav Halchenko, there is an experimental Debian
>>> package for Cython 0.17.beta1 -- http://packages.qa.debian.org/c/cython.html
>>>
>>> However, the builds are showing a lot of test failures on non-amd64 sytems.
>>> See https://buildd.debian.org/status/package.php?p=cython&suite=experimental
>>> Many of the errors seem related to assumptions made about the sizes of
>>> various integers (int vs. long vs long long).  Ideally we'll be able to
>>> clean up these failures before the final release.
>>
>> Thanks, I think it's mostly the tests that are wrong. I'll try to get
>> it fixed on linux 32 bit.
>
> I'm not sure the it's only the tests. The "char" vs. "unsigned char" errors
> hint at platform specific differences, which might trigger bugs (aka.
> incorrect assumptions) in the generated memory view code. Specifically,
> plain "char" is unsigned at least on ARM and PowerPC, both of which fail
> with this error.

Yes, I think somewhere we're assuming char is signed, but wasn't able
to see where in my quick investigations.

On another note, I updated jenkins to the latest release of Sage (5.2)
and all seems well.

- Robert

From stefan_ml at behnel.de  Tue Jul 31 21:14:20 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 31 Jul 2012 21:14:20 +0200
Subject: [Cython] [cython-users] Re: Cython 0.17 beta 1 released
In-Reply-To: <CADiQ+QCJmAs1ho3stYLY7gM20+gCpAOSYXaiPQb4S7U8u0vm0Q@mail.gmail.com>
References: <500DB7FA.4000409@behnel.de>
	<55a9d45a-27ea-428f-9fb6-8a1531b2edac@googlegroups.com>
	<CANg26EV3saFJPeHp5JUPbfdZZ86H4NGM12Lrc2xgxniBQmQ_ag@mail.gmail.com>
	<5017EA9C.8030803@behnel.de>
	<CADiQ+QCJmAs1ho3stYLY7gM20+gCpAOSYXaiPQb4S7U8u0vm0Q@mail.gmail.com>
Message-ID: <50182E8C.1060409@behnel.de>

Robert Bradshaw, 31.07.2012 19:24:
> On Tue, Jul 31, 2012 at 7:24 AM, Stefan Behnel wrote:
>> mark florisson, 28.07.2012 14:04:
>>> On 27 July 2012 23:30, Bradley Froehle wrote:
>>>> Thanks to the work of Yaroslav Halchenko, there is an experimental Debian
>>>> package for Cython 0.17.beta1 -- http://packages.qa.debian.org/c/cython.html
>>>>
>>>> However, the builds are showing a lot of test failures on non-amd64 sytems.
>>>> See https://buildd.debian.org/status/package.php?p=cython&suite=experimental
>>>> Many of the errors seem related to assumptions made about the sizes of
>>>> various integers (int vs. long vs long long).  Ideally we'll be able to
>>>> clean up these failures before the final release.
>>>
>>> Thanks, I think it's mostly the tests that are wrong. I'll try to get
>>> it fixed on linux 32 bit.
>>
>> I'm not sure the it's only the tests. The "char" vs. "unsigned char" errors
>> hint at platform specific differences, which might trigger bugs (aka.
>> incorrect assumptions) in the generated memory view code. Specifically,
>> plain "char" is unsigned at least on ARM and PowerPC, both of which fail
>> with this error.
> 
> Yes, I think somewhere we're assuming char is signed, but wasn't able
> to see where in my quick investigations.

Yes, it wasn't immediately obvious to me either. Here is a patch that
*might* fix the issue - obviously untested for the platforms in question.

Yaroslav, could you give it a try on the Debian build servers?

Stefan

-------------- next part --------------
A non-text attachment was scrubbed...
Name: buffer_integer_signedness.patch
Type: text/x-patch
Size: 1570 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20120731/d892dbb6/attachment.bin>