From greg.ewing at canterbury.ac.nz  Sun Jun  1 02:06:38 2014
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sun, 01 Jun 2014 12:06:38 +1200
Subject: [Cython]
 =?utf-8?q?Wrong_order_of_=5F=5FPyx=5FDECREF_when_calling?=
 =?utf-8?q?_a_function_with_an_implicit_str_=E2=86=92_char*_conversion=2E?=
In-Reply-To: <5389CD85.2070804@behnel.de>
References: <20140528112547.GA2814@yuyuko> <5388811E.3090802@behnel.de>
 <5389CD85.2070804@behnel.de>
Message-ID: <538A6E8E.4080603@canterbury.ac.nz>

Stefan Behnel wrote:
> Stefan Behnel, 30.05.2014 15:01:
> 
>>Emmanuel Gil Peyrot, 28.05.2014 13:25:
>>
>>>I was testing my cython codebase on top of pypy/cpyext, and I found a
>>>memory corruption.  After investigating it with the pypy guys (thanks
>>>arigato!), we identified it as a cython bug:
>>>
>>> cdef void test(char *string):
>>>     print(string)
>>>
>>> def run(array):
>>>     test(array[0])

I wouldn't call it a bug, more a limitation. When casting a
Python object to char *, it's the programmer's responsibility
to ensure that a reference is kept to the underlying object
for as long as necessary.

Having said that, keeping the reference for the duration of
the call would result in less surprising behaviour in cases
like this.

>>Cython has a mechanism to reject this kind of code, not sure why it
>>wouldn't strike here.

Not sure about Cython, but Pyrex only catches the most glaring
cases of this, such as adding two string and then immediately
casting the result to char *, which is almost guaranteed to
fail. Trying to catch anything more would have resulted in huge
numbers of false positives and been very annoying, so I
didn't try.

-- 
Greg

From robertwb at gmail.com  Wed Jun  4 22:37:06 2014
From: robertwb at gmail.com (Robert Bradshaw)
Date: Wed, 4 Jun 2014 13:37:06 -0700
Subject: [Cython] Cython 0.20.2 beta
In-Reply-To: <CADiQ+QDZC0zp5izk-G_pfkRe=SCFT3sTR=mM77JuX3s=9ZMyiw@mail.gmail.com>
References: <CADiQ+QDZC0zp5izk-G_pfkRe=SCFT3sTR=mM77JuX3s=9ZMyiw@mail.gmail.com>
Message-ID: <CADiQ+QAsboM0eySCRw9EtUQCw6b6zm_6WJz=ppJF+-i6iXfhcw@mail.gmail.com>

I'm going to take the lack of feedback as a good sign and release this
in the next couple of days...

On Thu, May 29, 2014 at 6:06 PM, Robert Bradshaw <robertwb at gmail.com> wrote:
> I've pushed a beta for the 0.20.2 can be found at
> http://cython.org/release/Cython-0.20rc1.tar.gz . This is a
> bugfix-only release, more development is going on for 0.21 which will
> also be released soon. Note, however, that the 0.20.x branch is the
> last to support older versions of Python (pre 2.6 or 3.2).
>
> Please try it out and report back.
>
> - Robert
>
>
> Features added
>
> Some optimisations for set/frozenset instantiation.
> Support for C++ unordered_set and unordered_map.
>
> Bugs fixed
>
> Access to attributes of optimised builtin methods (e.g.
> [].append.__name__) could fail to compile.
> Memory leak when extension subtypes add a memory view as attribute to
> those of the parent type without having Python object attributes or a
> user provided dealloc method.
> Compiler crash on readonly properties in "binding" mode.
> Auto-encoding with c_string_encoding=ascii failed in Py3.3.
> Crash when subtyping freelist enabled Cython extension types with
> Python classes that use __slots__.
> Freelist usage is restricted to CPython to avoid problems with other
> Python implementations.
> Memory leak in memory views when copying overlapping, contiguous slices.
> Format checking when requesting non-contiguous buffers from
> cython.array objects was disabled in Py3.
> C++ destructor calls in extension types could fail to compile in clang.
> Buffer format validation failed for sequences of strings in structs.
> Docstrings on extension type attributes in .pxd files were rejected.

From stefan_ml at behnel.de  Mon Jun 16 10:07:47 2014
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 16 Jun 2014 10:07:47 +0200
Subject: [Cython] any more changes for 0.21?
Message-ID: <539EA5D3.4010103@behnel.de>

Hi,

I think the master branch is good enough for at least a 0.21 alpha. Are
there any pending changes that should go in before it can be released? Any
pull requests that should be considered?

We should get this in, I think:

https://github.com/cython/cython/pull/284

And the gdb tests are still broken.

Robert, could you take a look through the Sage test failures? Most of them
might be issues in Sage rather than Cython, but it's better to make sure we
can either fix or safely ignore them.

Stefan

From stefan_ml at behnel.de  Mon Jun 16 15:58:36 2014
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 16 Jun 2014 15:58:36 +0200
Subject: [Cython] JyNI  - C-API emulation for Jython
Message-ID: <539EF80C.8040206@behnel.de>

Hi,

it looks like there is a C-API emulation for Jython on the way:

http://jyni.org/

https://github.com/Stewori/JyNI

I haven't tried it yet, and I'm sure Cython modules won't work with its
current state, but the effort I put into making them run in PyPy's cpyext
should pay off here, so we might at some point have a third backend to target.

This also reminded me of IronClad, the C-API layer for IronPython. I looked
it up, but it seems to have died some years ago, with no current effort to
improve it or even just make it work with the latest IronPython releases.

https://groups.google.com/d/msg/c-extensions-for-ironpython/SrIesUIh1Gw/cBQG1FMt0AUJ

https://code.google.com/p/ironclad/

Not sure if it's still worth trying to make that run in Cython.

Stefan

From robertwb at gmail.com  Mon Jun 16 19:44:04 2014
From: robertwb at gmail.com (Robert Bradshaw)
Date: Mon, 16 Jun 2014 10:44:04 -0700
Subject: [Cython] JyNI - C-API emulation for Jython
In-Reply-To: <539EF80C.8040206@behnel.de>
References: <539EF80C.8040206@behnel.de>
Message-ID: <CADiQ+QB9J8UhDObL2Mi=n2XUN0Z2q6=NVFbVgcJhe=5LJv8uNg@mail.gmail.com>

Cool.

I find their motivation "Since Java is rather present in industry,
while Python is more present in science, JyNI will be an important
step to lower the cost of using scientific code in industrial
environments" interesting.

On the other hand, I think IronClad is pretty dead.

On Mon, Jun 16, 2014 at 6:58 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Hi,
>
> it looks like there is a C-API emulation for Jython on the way:
>
> http://jyni.org/
>
> https://github.com/Stewori/JyNI
>
> I haven't tried it yet, and I'm sure Cython modules won't work with its
> current state, but the effort I put into making them run in PyPy's cpyext
> should pay off here, so we might at some point have a third backend to target.
>
> This also reminded me of IronClad, the C-API layer for IronPython. I looked
> it up, but it seems to have died some years ago, with no current effort to
> improve it or even just make it work with the latest IronPython releases.
>
> https://groups.google.com/d/msg/c-extensions-for-ironpython/SrIesUIh1Gw/cBQG1FMt0AUJ
>
> https://code.google.com/p/ironclad/
>
> Not sure if it's still worth trying to make that run in Cython.
>
> Stefan
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> https://mail.python.org/mailman/listinfo/cython-devel

From robertwb at gmail.com  Tue Jun 17 02:07:32 2014
From: robertwb at gmail.com (Robert Bradshaw)
Date: Mon, 16 Jun 2014 17:07:32 -0700
Subject: [Cython] any more changes for 0.21?
In-Reply-To: <539EA5D3.4010103@behnel.de>
References: <539EA5D3.4010103@behnel.de>
Message-ID: <CADiQ+QANDRmyS925YamXB0NRs9-1LR-6UUeKiWppyGxVFBs4Dg@mail.gmail.com>

On Mon, Jun 16, 2014 at 1:07 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Hi,
>
> I think the master branch is good enough for at least a 0.21 alpha. Are
> there any pending changes that should go in before it can be released? Any
> pull requests that should be considered?

I started some work on support for static methods (for c++ and cdef
classes). I'll see if I can wrap this up and create a pull request.

> We should get this in, I think:
>
> https://github.com/cython/cython/pull/284
>
> And the gdb tests are still broken.
>
> Robert, could you take a look through the Sage test failures? Most of them
> might be issues in Sage rather than Cython, but it's better to make sure we
> can either fix or safely ignore them.

Yeah, I'll take a look at those.

- Robert

From robertwb at gmail.com  Tue Jun 17 06:39:55 2014
From: robertwb at gmail.com (Robert Bradshaw)
Date: Mon, 16 Jun 2014 21:39:55 -0700
Subject: [Cython] Cython bugfix release
Message-ID: <CADiQ+QAbqG+iXHp030EBpds4qZOKojM5Ypp7r8Ks-ZdmzZwSxw@mail.gmail.com>

I just pushed another bugfix release for the 0.20.x line, available on
github, cython.org, or and pypi.

== Features added ==

* Some optimisations for set/frozenset instantiation.
* Support for C++ unordered_set and unordered_map.

== Bugs fixed ==

* Access to attributes of optimised builtin methods (e.g.
[].append.__name__) could fail to compile.
* Memory leak when extension subtypes add a memory view as attribute
to those of the parent type without having Python object attributes or
a user provided dealloc method.
* Compiler crash on readonly properties in "binding" mode.
* Auto-encoding with c_string_encoding=ascii failed in Py3.3.
* Crash when subtyping freelist enabled Cython extension types with
Python classes that use __slots__.
* Freelist usage is restricted to CPython to avoid problems with other
Python implementations.
* Memory leak in memory views when copying overlapping, contiguous slices.
* Format checking when requesting non-contiguous buffers from
cython.array objects was disabled in Py3.
* C++ destructor calls in extension types could fail to compile in clang.
* Buffer format validation failed for sequences of strings in structs.
* Docstrings on extension type attributes in .pxd files were rejected.

== Contributors ==

Andreas van Cranenburgh
Ian Bell
Lars Buitinck
Martin Quarda
Mikhail Korobov
Robert Bradshaw
Stefan Behnel

From robertwb at gmail.com  Tue Jun 17 07:14:52 2014
From: robertwb at gmail.com (Robert Bradshaw)
Date: Mon, 16 Jun 2014 22:14:52 -0700
Subject: [Cython] any more changes for 0.21?
In-Reply-To: <CADiQ+QANDRmyS925YamXB0NRs9-1LR-6UUeKiWppyGxVFBs4Dg@mail.gmail.com>
References: <539EA5D3.4010103@behnel.de>
 <CADiQ+QANDRmyS925YamXB0NRs9-1LR-6UUeKiWppyGxVFBs4Dg@mail.gmail.com>
Message-ID: <CADiQ+QA4k-=9HNQmYremzJ6picz7PrCiYZm3NTMYRKdEA2AKUQ@mail.gmail.com>

On Mon, Jun 16, 2014 at 5:07 PM, Robert Bradshaw <robertwb at gmail.com> wrote:
> On Mon, Jun 16, 2014 at 1:07 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
>> Hi,
>>
>> I think the master branch is good enough for at least a 0.21 alpha. Are
>> there any pending changes that should go in before it can be released? Any
>> pull requests that should be considered?
>
> I started some work on support for static methods (for c++ and cdef
> classes). I'll see if I can wrap this up and create a pull request.
>
>> We should get this in, I think:
>>
>> https://github.com/cython/cython/pull/284
>>
>> And the gdb tests are still broken.
>>
>> Robert, could you take a look through the Sage test failures? Most of them
>> might be issues in Sage rather than Cython, but it's better to make sure we
>> can either fix or safely ignore them.
>
> Yeah, I'll take a look at those.

Not all of them look benign. I'm updating Sage to the latest release
and will see what's goin on.

From robertwb at gmail.com  Tue Jun 17 23:14:06 2014
From: robertwb at gmail.com (Robert Bradshaw)
Date: Tue, 17 Jun 2014 14:14:06 -0700
Subject: [Cython] any more changes for 0.21?
In-Reply-To: <CADiQ+QA4k-=9HNQmYremzJ6picz7PrCiYZm3NTMYRKdEA2AKUQ@mail.gmail.com>
References: <539EA5D3.4010103@behnel.de>
 <CADiQ+QANDRmyS925YamXB0NRs9-1LR-6UUeKiWppyGxVFBs4Dg@mail.gmail.com>
 <CADiQ+QA4k-=9HNQmYremzJ6picz7PrCiYZm3NTMYRKdEA2AKUQ@mail.gmail.com>
Message-ID: <CADiQ+QBMObAKEXJPaWvn0JxV6q4XsQgffYBygk0GCdYGuoY+MA@mail.gmail.com>

On Mon, Jun 16, 2014 at 10:14 PM, Robert Bradshaw <robertwb at gmail.com> wrote:
> On Mon, Jun 16, 2014 at 5:07 PM, Robert Bradshaw <robertwb at gmail.com> wrote:
>> On Mon, Jun 16, 2014 at 1:07 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
>>> Hi,
>>>
>>> I think the master branch is good enough for at least a 0.21 alpha. Are
>>> there any pending changes that should go in before it can be released? Any
>>> pull requests that should be considered?
>>
>> I started some work on support for static methods (for c++ and cdef
>> classes). I'll see if I can wrap this up and create a pull request.
>>
>>> We should get this in, I think:
>>>
>>> https://github.com/cython/cython/pull/284
>>>
>>> And the gdb tests are still broken.
>>>
>>> Robert, could you take a look through the Sage test failures? Most of them
>>> might be issues in Sage rather than Cython, but it's better to make sure we
>>> can either fix or safely ignore them.
>>
>> Yeah, I'll take a look at those.
>
> Not all of them look benign. I'm updating Sage to the latest release
> and will see what's goin on.

The two remaining cmdline.py tests look inocuous, we should be good on
that front (though I'll try to get this clean again).

- Robert

From robertwb at gmail.com  Wed Jun 18 07:46:33 2014
From: robertwb at gmail.com (Robert Bradshaw)
Date: Tue, 17 Jun 2014 22:46:33 -0700
Subject: [Cython] any more changes for 0.21?
In-Reply-To: <CADiQ+QBMObAKEXJPaWvn0JxV6q4XsQgffYBygk0GCdYGuoY+MA@mail.gmail.com>
References: <539EA5D3.4010103@behnel.de>
 <CADiQ+QANDRmyS925YamXB0NRs9-1LR-6UUeKiWppyGxVFBs4Dg@mail.gmail.com>
 <CADiQ+QA4k-=9HNQmYremzJ6picz7PrCiYZm3NTMYRKdEA2AKUQ@mail.gmail.com>
 <CADiQ+QBMObAKEXJPaWvn0JxV6q4XsQgffYBygk0GCdYGuoY+MA@mail.gmail.com>
Message-ID: <CADiQ+QBcJ0jLeZmRDUa4_Jy19xGndMD8SnvX4=4z8K8vn_1zgA@mail.gmail.com>

On Tue, Jun 17, 2014 at 2:14 PM, Robert Bradshaw <robertwb at gmail.com> wrote:
> On Mon, Jun 16, 2014 at 10:14 PM, Robert Bradshaw <robertwb at gmail.com> wrote:
>> On Mon, Jun 16, 2014 at 5:07 PM, Robert Bradshaw <robertwb at gmail.com> wrote:
>>> On Mon, Jun 16, 2014 at 1:07 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
>>>> Hi,
>>>>
>>>> I think the master branch is good enough for at least a 0.21 alpha. Are
>>>> there any pending changes that should go in before it can be released? Any
>>>> pull requests that should be considered?
>>>
>>> I started some work on support for static methods (for c++ and cdef
>>> classes). I'll see if I can wrap this up and create a pull request.
>>>
>>>> We should get this in, I think:
>>>>
>>>> https://github.com/cython/cython/pull/284
>>>>
>>>> And the gdb tests are still broken.
>>>>
>>>> Robert, could you take a look through the Sage test failures? Most of them
>>>> might be issues in Sage rather than Cython, but it's better to make sure we
>>>> can either fix or safely ignore them.
>>>
>>> Yeah, I'll take a look at those.
>>
>> Not all of them look benign. I'm updating Sage to the latest release
>> and will see what's goin on.
>
> The two remaining cmdline.py tests look inocuous, we should be good on
> that front (though I'll try to get this clean again).

All sage tests are passing now (the weren't Cython issues).

- Robert

From lists at onerussian.com  Wed Jun 18 17:37:40 2014
From: lists at onerussian.com (Yaroslav Halchenko)
Date: Wed, 18 Jun 2014 11:37:40 -0400
Subject: [Cython] Cython bugfix release
In-Reply-To: <CADiQ+QAbqG+iXHp030EBpds4qZOKojM5Ypp7r8Ks-ZdmzZwSxw@mail.gmail.com>
References: <CADiQ+QAbqG+iXHp030EBpds4qZOKojM5Ypp7r8Ks-ZdmzZwSxw@mail.gmail.com>
Message-ID: <20140618153740.GA8748@onerussian.com>

FWIW -- 0.20.2 was just uploaded to Debian sid, thus should be available
to Debian folks soon too

while trying 0.20.1 across debian/ubuntus I ran into this failure

======================================================================
ERROR: test_all (Cython.Debugger.Tests.TestLibCython.TestAll)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/buildd/cython-0.20.2/Cython/Debugger/Tests/TestLibCython.py", line 281, in test_all
    sys.stderr.write(errmsg)
UnicodeEncodeError: 'ascii' codec can't encode characters in position 18851-18854: ordinal not in range(128)

on debian wheezy i386.  it didn't happen on amd64 and on both architectures
under debian jessie (testing) and it seemed to happen while testing with
python2.6 

On Mon, 16 Jun 2014, Robert Bradshaw wrote:

> I just pushed another bugfix release for the 0.20.x line, available on
> github, cython.org, or and pypi.

> == Features added ==

> * Some optimisations for set/frozenset instantiation.
> * Support for C++ unordered_set and unordered_map.

> == Bugs fixed ==

> * Access to attributes of optimised builtin methods (e.g.
> [].append.__name__) could fail to compile.
> * Memory leak when extension subtypes add a memory view as attribute
> to those of the parent type without having Python object attributes or
> a user provided dealloc method.
> * Compiler crash on readonly properties in "binding" mode.
> * Auto-encoding with c_string_encoding=ascii failed in Py3.3.
> * Crash when subtyping freelist enabled Cython extension types with
> Python classes that use __slots__.
> * Freelist usage is restricted to CPython to avoid problems with other
> Python implementations.
> * Memory leak in memory views when copying overlapping, contiguous slices.
> * Format checking when requesting non-contiguous buffers from
> cython.array objects was disabled in Py3.
> * C++ destructor calls in extension types could fail to compile in clang.
> * Buffer format validation failed for sequences of strings in structs.
> * Docstrings on extension type attributes in .pxd files were rejected.

> == Contributors ==

> Andreas van Cranenburgh
> Ian Bell
> Lars Buitinck
> Martin Quarda
> Mikhail Korobov
> Robert Bradshaw
> Stefan Behnel
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> https://mail.python.org/mailman/listinfo/cython-devel


-- 
Yaroslav O. Halchenko, Ph.D.
http://neuro.debian.net http://www.pymvpa.org http://www.fail2ban.org
Research Scientist,            Psychological and Brain Sciences Dept.
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834                       Fax: +1 (603) 646-1419
WWW:   http://www.linkedin.com/in/yarik        

From lists at onerussian.com  Thu Jun 19 17:28:10 2014
From: lists at onerussian.com (Yaroslav Halchenko)
Date: Thu, 19 Jun 2014 11:28:10 -0400
Subject: [Cython] Cython bugfix release
In-Reply-To: <20140618153740.GA8748@onerussian.com>
References: <CADiQ+QAbqG+iXHp030EBpds4qZOKojM5Ypp7r8Ks-ZdmzZwSxw@mail.gmail.com>
 <20140618153740.GA8748@onerussian.com>
Message-ID: <20140619152810.GC8748@onerussian.com>

and the same issue in current(ish) master:

======================================================================
ERROR: test_all (Cython.Debugger.Tests.TestLibCython.TestAll)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/buildd/cython-0.20.2+git216-ga96882e/Cython/Debugger/Tests/TestLibCython.py", line 280, in test_all
    sys.stderr.write(errmsg)
UnicodeEncodeError: 'ascii' codec can't encode characters in position 21001-21004: ordinal not in range(128)

----------------------------------------------------------------------
Ran 8348 tests in 2674.102s



On Wed, 18 Jun 2014, Yaroslav Halchenko wrote:

> FWIW -- 0.20.2 was just uploaded to Debian sid, thus should be available
> to Debian folks soon too

> while trying 0.20.1 across debian/ubuntus I ran into this failure

> ======================================================================
> ERROR: test_all (Cython.Debugger.Tests.TestLibCython.TestAll)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File "/tmp/buildd/cython-0.20.2/Cython/Debugger/Tests/TestLibCython.py", line 281, in test_all
>     sys.stderr.write(errmsg)
> UnicodeEncodeError: 'ascii' codec can't encode characters in position 18851-18854: ordinal not in range(128)

> on debian wheezy i386.  it didn't happen on amd64 and on both architectures
> under debian jessie (testing) and it seemed to happen while testing with
> python2.6 

> On Mon, 16 Jun 2014, Robert Bradshaw wrote:

> > I just pushed another bugfix release for the 0.20.x line, available on
> > github, cython.org, or and pypi.

> > == Features added ==

> > * Some optimisations for set/frozenset instantiation.
> > * Support for C++ unordered_set and unordered_map.

> > == Bugs fixed ==

> > * Access to attributes of optimised builtin methods (e.g.
> > [].append.__name__) could fail to compile.
> > * Memory leak when extension subtypes add a memory view as attribute
> > to those of the parent type without having Python object attributes or
> > a user provided dealloc method.
> > * Compiler crash on readonly properties in "binding" mode.
> > * Auto-encoding with c_string_encoding=ascii failed in Py3.3.
> > * Crash when subtyping freelist enabled Cython extension types with
> > Python classes that use __slots__.
> > * Freelist usage is restricted to CPython to avoid problems with other
> > Python implementations.
> > * Memory leak in memory views when copying overlapping, contiguous slices.
> > * Format checking when requesting non-contiguous buffers from
> > cython.array objects was disabled in Py3.
> > * C++ destructor calls in extension types could fail to compile in clang.
> > * Buffer format validation failed for sequences of strings in structs.
> > * Docstrings on extension type attributes in .pxd files were rejected.

> > == Contributors ==

> > Andreas van Cranenburgh
> > Ian Bell
> > Lars Buitinck
> > Martin Quarda
> > Mikhail Korobov
> > Robert Bradshaw
> > Stefan Behnel
> > _______________________________________________
> > cython-devel mailing list
> > cython-devel at python.org
> > https://mail.python.org/mailman/listinfo/cython-devel
-- 
Yaroslav O. Halchenko, Ph.D.
http://neuro.debian.net http://www.pymvpa.org http://www.fail2ban.org
Research Scientist,            Psychological and Brain Sciences Dept.
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834                       Fax: +1 (603) 646-1419
WWW:   http://www.linkedin.com/in/yarik        

From stefan_ml at behnel.de  Sun Jun 22 12:48:27 2014
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 22 Jun 2014 12:48:27 +0200
Subject: [Cython] Fused type signature resolution failures
In-Reply-To: <lgaium$uuj$1@ger.gmane.org>
References: <lgaium$uuj$1@ger.gmane.org>
Message-ID: <53A6B47B.7050000@behnel.de>

Hi,

it looks like no-one replied to is so far, so here's a response.

Pauli Virtanen, 18.03.2014 23:56:
> Here are the fused types + memoryview signature resolution failures I
> promised earlier (Cython 0.20.1):
> 
> 
> (A)
> 
> TypeError: No matching signature
> 
> ----------------------------- asd.pyx
> cimport numpy as cnp
> 
> ctypedef fused value_t:
>     cnp.float32_t
>     cnp.float64_t
> 
> cpdef foo(value_t x):
>     pass
> ----------------------------- quux.py
> import numpy as np
> import asd
> asd.foo(np.float32(1.0))
> -----------------------------

I don't actually know how NumPy represents its types at the Python level,
but my guess is that it would be tricky to match these two without teaching
Cython itself something about NumPy (and how it wraps basic C types here).
I'd rather like to avoid that and live with the above.


> (B)
> 
> ValueError: Buffer dtype mismatch, expected 'int64_t' but got 'double'
> 
> ----------------------------- asd.pyx
> cimport numpy as cnp
> 
> ctypedef fused idx_t:
>     cnp.int32_t
>     cnp.int64_t
> 
> ctypedef fused value_t:
>     cnp.int64_t
>     cnp.float64_t
> 
> cpdef foo(idx_t[:,:] i, idx_t[:,:] j, value_t[:,:] x):
>     pass
> ----------------------------- quux.py
> import numpy as np
> import asd
> i = np.zeros((3, 3), np.int64)
> j = np.zeros((3, 3), np.int64)
> x = np.zeros((3, 3), np.float64)
> asd.foo(i, j, x)
> -----------------------------

This looks like a bug to me at first sight.


> (C)
> 
> Then some nasty platform-dependent failures:
> 
> https://github.com/scipy/scipy/issues/3461
> 
> The relevant code is:
> 
> https://github.com/scipy/scipy/blob/master/scipy/sparse/_csparsetools.pyx#L202
> 
> The code looks nothing special. However, call to `lil_fancy_get` fails
> with "TypeError: No matching signature found" when the inputs have types
> 
> <class 'int'> <class 'int'> <class 'numpy.ndarray'>
> <class 'numpy.ndarray'> <class 'numpy.ndarray'> <class 'numpy.ndarray'>
> <class 'numpy.ndarray'> <class 'numpy.ndarray'>
> 
> with ndarray dtypes: object object object object int32 int32
> 
> The failure occurs only on Christoph Gohlke's Win64 build, but not on
> Linux/OSX/MINGW32. This sounds like some integer size combination
> issue, but I'm far from sure.
> 
> Unfortunately, I'm not easily able to isolate/understand what's going
> wrong here.

Generally speaking, I think that the signature matching algorithm has some
room for improvements, especially the one that matches Python signatures at
runtime.

We should take a look at how other implementations do this dispatch. There
are multiple "generic functions" implementations for Python that do similar
things.

Stefan


From pav at iki.fi  Sun Jun 22 13:35:24 2014
From: pav at iki.fi (Pauli Virtanen)
Date: Sun, 22 Jun 2014 14:35:24 +0300
Subject: [Cython] Fused type signature resolution failures
In-Reply-To: <53A6B47B.7050000@behnel.de>
References: <lgaium$uuj$1@ger.gmane.org> <53A6B47B.7050000@behnel.de>
Message-ID: <lo6f1s$5vh$1@ger.gmane.org>

22.06.2014 13:48, Stefan Behnel kirjoitti:
[clip]
> > (A)
[clip]
> > asd.foo(np.float32(1.0))
>
> I don't actually know how NumPy represents its types at the Python
> level, but my guess is that it would be tricky to match these two
> without teaching Cython itself something about NumPy (and how it
> wraps basic C types here). I'd rather like to avoid that and live
> with the above.

Agreed, it's probably not possible to properly deal with this without
making use of Numpy scalar type object binary layout in some form.

On the other hand, `asd.foo(np.array(1.0))` doesn't work either ---
maybe the buffer code path does not trigger for scalar values.

[clip]
>> (B)
>>
>> ValueError: Buffer dtype mismatch, expected 'int64_t' but got 'double'
[clip]
> This looks like a bug to me at first sight.

This was fixed by my PR #284 that you merged.

>> (C)
>>
>> Then some nasty platform-dependent failures:
[clip]
> Generally speaking, I think that the signature matching algorithm has some
> room for improvements, especially the one that matches Python signatures at
> runtime.
> 
> We should take a look at how other implementations do this dispatch. There
> are multiple "generic functions" implementations for Python that do similar
> things.

I agree that there probably is room for improvement, possibly also
speed-wise.

I'll try to revisit (at some point) the csparsetools Cython
implementation to see if there are low-hanging fixes that would be
useful there.

-- 
Pauli Virtanen


From stefan_ml at behnel.de  Sun Jun 22 14:01:28 2014
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 22 Jun 2014 14:01:28 +0200
Subject: [Cython] Fused type signature resolution failures
In-Reply-To: <lo6f1s$5vh$1@ger.gmane.org>
References: <lgaium$uuj$1@ger.gmane.org> <53A6B47B.7050000@behnel.de>
 <lo6f1s$5vh$1@ger.gmane.org>
Message-ID: <53A6C598.3070708@behnel.de>

Pauli Virtanen, 22.06.2014 13:35:
> 22.06.2014 13:48, Stefan Behnel kirjoitti:
>> Generally speaking, I think that the signature matching algorithm has some
>> room for improvements, especially the one that matches Python signatures at
>> runtime.
>>
>> We should take a look at how other implementations do this dispatch. There
>> are multiple "generic functions" implementations for Python that do similar
>> things.
> 
> I agree that there probably is room for improvement, possibly also
> speed-wise.

Definitely speed-wise. I was considering to get rid of the "build a key and
do a dict lookup" approach and just do sequential type checks instead,
although looking at your example with its dozen of different numeric types
in one fused type makes me worry that there might be code out there that
actually benefits from dict usage. Still, I'm sure it would be possible to
speed up the most common case of exactly one fused type in a signature,
maybe also that of two, or that of one structured (i.e. array) type.
Everything else isn't worth it, I guess.


> I'll try to revisit (at some point) the csparsetools Cython
> implementation to see if there are low-hanging fixes that would be
> useful there.

That would be great.

Stefan


From stefan_ml at behnel.de  Wed Jun 25 21:16:36 2014
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 25 Jun 2014 21:16:36 +0200
Subject: [Cython] NumPy 1.7+ warnings
Message-ID: <53AB2014.8060807@behnel.de>

Hi,

recent NumPy versions are a bit unhappy about Cython code and generate
deprecation warnings about it:

.../numpy/core/include/numpy/npy_1_7_deprecated_api.h:15:2: warning:
#warning "Using deprecated NumPy API, disable it by " "#defining
NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION"

I don't know how much work it would be to port the C code properly, but it
should be mostly about updating the implementation in Cython's
"numpy/__init__.pxd", which is dated anyway.

Given that NumPy 1.7 appeared not much more than a year ago, dropping
support for older versions sounds a bit aggressive, but there should be a
way to keep supporting them by defining some macros, similar to what we do
for CPython in ModuleSetupCode.c. Any volunteers?

The relevant bits of NumPy's C-API are described here:

http://docs.scipy.org/doc/numpy-dev/reference/c-api.array.html

My guess is that most of the work should be replacing direct access to
ndarray struct fields with C macro calls. That won't easily fix user code,
but it's the first major step on the way.

Stefan

From nouiz at nouiz.org  Wed Jun 25 21:46:39 2014
From: nouiz at nouiz.org (=?UTF-8?B?RnLDqWTDqXJpYyBCYXN0aWVu?=)
Date: Wed, 25 Jun 2014 15:46:39 -0400
Subject: [Cython] NumPy 1.7+ warnings
In-Reply-To: <53AB2014.8060807@behnel.de>
References: <53AB2014.8060807@behnel.de>
Message-ID: <CADKKbthpekNmGkTLyFRcBbLw+031grcUoowEKmxjVvnNhz9W7g@mail.gmail.com>

Just to help people working on this, here is what I added in Theano:

        # numpy 1.7 deprecated the following macro but the new one didn't
        # existed in the past
        if bool(numpy_ver < [1, 7]):
            cxxflags.append("-D NPY_ARRAY_ENSUREARRAY=NPY_ENSUREARRAY")
            cxxflags.append("-D NPY_ARRAY_ENSURECOPY=NPY_ENSURECOPY")
            cxxflags.append("-D NPY_ARRAY_ALIGNED=NPY_ALIGNED")
            cxxflags.append("-D NPY_ARRAY_WRITEABLE=NPY_WRITEABLE")
            cxxflags.append("-D NPY_ARRAY_UPDATE_ALL=NPY_UPDATE_ALL")
            cxxflags.append("-D NPY_ARRAY_C_CONTIGUOUS=NPY_C_CONTIGUOUS")
            cxxflags.append("-D NPY_ARRAY_F_CONTIGUOUS=NPY_F_CONTIGUOUS")


Some of the newer macro that are not deprecated, have been added only in
early version of NumPy. You could do an ifdef on the numpy bersion and add
those macro to be compatible with older numpy version when using the new
macro.

Fred


On Wed, Jun 25, 2014 at 3:16 PM, Stefan Behnel <stefan_ml at behnel.de> wrote:

> Hi,
>
> recent NumPy versions are a bit unhappy about Cython code and generate
> deprecation warnings about it:
>
> .../numpy/core/include/numpy/npy_1_7_deprecated_api.h:15:2: warning:
> #warning "Using deprecated NumPy API, disable it by " "#defining
> NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION"
>
> I don't know how much work it would be to port the C code properly, but it
> should be mostly about updating the implementation in Cython's
> "numpy/__init__.pxd", which is dated anyway.
>
> Given that NumPy 1.7 appeared not much more than a year ago, dropping
> support for older versions sounds a bit aggressive, but there should be a
> way to keep supporting them by defining some macros, similar to what we do
> for CPython in ModuleSetupCode.c. Any volunteers?
>
> The relevant bits of NumPy's C-API are described here:
>
> http://docs.scipy.org/doc/numpy-dev/reference/c-api.array.html
>
> My guess is that most of the work should be replacing direct access to
> ndarray struct fields with C macro calls. That won't easily fix user code,
> but it's the first major step on the way.
>
> Stefan
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> https://mail.python.org/mailman/listinfo/cython-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20140625/1c831651/attachment.html>