From barcc at gmx.de  Fri Jan  1 04:53:11 2016
From: barcc at gmx.de (B. Clausius)
Date: Fri, 1 Jan 2016 10:53:11 +0100
Subject: [Cython] strange error message
In-Reply-To: <CADiQ+QD-Zte1+K0_+1zLkuMFP-W+sAPJ+6O_QZs4dBb4TbhENg@mail.gmail.com>
References: <567E7320.8070606@gmx.de> <n5rn29$ke9$1@ger.gmane.org>
 <n5s8q3$a6m$1@ger.gmane.org>
 <CADiQ+QD-Zte1+K0_+1zLkuMFP-W+sAPJ+6O_QZs4dBb4TbhENg@mail.gmail.com>
Message-ID: <56864C87.6050907@gmx.de>

Am 31.12.2015 um 01:00 schrieb Robert Bradshaw:
> On Mon, Dec 28, 2015 at 1:15 PM, Baptiste Carvello
> <devel at baptiste-carvello.net> wrote:
>> Le 28/12/2015 17:12, Baptiste Carvello a ?crit :
>>> Hi,
>>>
>>> not a cython dev, but I believe I hit the very same problem (I'm going
>>> on vacation now, full report in January).
>>>
>>> You may fix the compilation error by adding the following unused import:
>>>>>> from gldraw import vec4
>>
>> sorry, that's a cimport, i.e.
>>
>>>>> from gldraw cimport vec4
> 
> +1, that's the solution. We should make the error better...

I added that line and replaced "cdef void sync():" by
"cpdef void sync():" to be able to test it:

$ cat glarea.pyx
cimport gldraw
from gldraw cimport vec4

cdef struct Data:
    gldraw.mat4 matrix

cdef Data data

cpdef void sync():
    data.changed = True  # <- this wrong line produces the error

It compiles without errors, but i get a exception at runtime (not
surprised):

$ python3
Python 3.4.3+ (default, Oct 14 2015, 16:03:50)
[GCC 5.2.1 20151010] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import glarea
>>> glarea.sync()
Exception ignored in: 'glarea.sync'
AttributeError: 'dict' object has no attribute 'changed'
>>>

This would be ok for me, but i wonder if cython should recognize this at
compile time. The code tries to assign a new attribute to a cdef struct
and that will always fail, or am I wrong?


From devel at baptiste-carvello.net  Tue Jan  5 04:12:57 2016
From: devel at baptiste-carvello.net (Baptiste Carvello)
Date: Tue, 05 Jan 2016 10:12:57 +0100
Subject: [Cython] strange error message
In-Reply-To: <CADiQ+QD-Zte1+K0_+1zLkuMFP-W+sAPJ+6O_QZs4dBb4TbhENg@mail.gmail.com>
References: <567E7320.8070606@gmx.de> <n5rn29$ke9$1@ger.gmane.org>
 <n5s8q3$a6m$1@ger.gmane.org>
 <CADiQ+QD-Zte1+K0_+1zLkuMFP-W+sAPJ+6O_QZs4dBb4TbhENg@mail.gmail.com>
Message-ID: <n6g1er$q50$1@ger.gmane.org>

Le 31/12/2015 01:00, Robert Bradshaw a ?crit :
> On Mon, Dec 28, 2015 at 1:15 PM, Baptiste Carvello
> <devel at baptiste-carvello.net> wrote:
>> Le 28/12/2015 17:12, Baptiste Carvello a ?crit :
>>> Hi,
>>>
>>> not a cython dev, but I believe I hit the very same problem (I'm going
>>> on vacation now, full report in January).
>>>
>>> You may fix the compilation error by adding the following unused import:
>>>>>> from gldraw import vec4
>>
>> sorry, that's a cimport, i.e.
>>
>>>>> from gldraw cimport vec4
> 
> +1, that's the solution. We should make the error better...

Hi, back from vacation :-)

I would like to know if this behaviour (not taking namespace into
account in generated Cython code) is a deliberate design choice, or a
wart that should be fixed if possible. In the latter case, I'll open an
issue about it.

FYI, the behaviour was newly introduced in 2014 when the Python<->C
conversion code was rewritten (rev. fd565517). Code like the OP's worked
fine before that.

Baptiste


From storchaka at gmail.com  Mon Jan 11 15:32:44 2016
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Mon, 11 Jan 2016 22:32:44 +0200
Subject: [Cython] Cython creates non-pickleable classes
Message-ID: <n713hc$jg5$1@ger.gmane.org>

Sorry, I'm not very familiar with Cython, but it looks to me that Cython 
creates extension classes that can't be correctly copied and unpickled. 
Since default __reduce_ex__ implementation knows nothing about Cython 
class structure, it's result doesn't contain the state of the object. 
The object could be pickled, but unpickled object doesn't contain 
original values of fields, it is just non-initialized. In recent CPython 
releases was added new restrictions for pickling objects that can't be 
correctly unpickled. This had added incompatibility with Cython classes. 
Building some packages now is failed, because Cython makes a deep copy 
that actually doesn't work.

See for original changes and discussion about the regression: 
http://bugs.python.org/issue22995 .

Besides the fact of Cython regression, it looks to me, that Cython also 
needs to be fixed. It can be done either by generating default 
pickle-related methods (__getnewargs__/__getnewargs_ex__, or 
__getstate__ + __setstate__, or __reduce__/__reduce_ex__) for all Cython 
classes, or implementing pickling or deepcopying only for NameAssignment 
and like.


From stefan_ml at behnel.de  Tue Jan 12 07:51:04 2016
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 12 Jan 2016 13:51:04 +0100
Subject: [Cython] Cython creates non-pickleable classes
In-Reply-To: <n713hc$jg5$1@ger.gmane.org>
References: <n713hc$jg5$1@ger.gmane.org>
Message-ID: <5694F6B8.2050304@behnel.de>

Hi Serhiy!

Serhiy Storchaka schrieb am 11.01.2016 um 21:32:
> Sorry, I'm not very familiar with Cython, but it looks to me that Cython
> creates extension classes that can't be correctly copied and unpickled.
> Since default __reduce_ex__ implementation knows nothing about Cython class
> structure, it's result doesn't contain the state of the object. The object
> could be pickled, but unpickled object doesn't contain original values of
> fields, it is just non-initialized. In recent CPython releases was added
> new restrictions for pickling objects that can't be correctly unpickled.
> This had added incompatibility with Cython classes. Building some packages
> now is failed, because Cython makes a deep copy that actually doesn't work.
> 
> See for original changes and discussion about the regression:
> http://bugs.python.org/issue22995 .
> 
> Besides the fact of Cython regression, it looks to me, that Cython also
> needs to be fixed. It can be done either by generating default
> pickle-related methods (__getnewargs__/__getnewargs_ex__, or __getstate__ +
> __setstate__, or __reduce__/__reduce_ex__) for all Cython classes, or
> implementing pickling or deepcopying only for NameAssignment and like.

Here's a call trace from the above ticket:

https://paste.debian.net/361351/

I can confirm that this is a bug in Cython (the compiler, not the code it
generates). There are a couple of classes (NameAssignment, its subclasses
and potentially others) that get deep-copied by the standard mechanism and
thus loose their internal object state. This should get fixed. It wasn't an
issue in practice because these objects aren't commonly used after this
copy, but there's no reason they couldn't be.

Stefan


From orion at cora.nwra.com  Wed Jan 13 12:50:30 2016
From: orion at cora.nwra.com (Orion Poplawski)
Date: Wed, 13 Jan 2016 17:50:30 +0000 (UTC)
Subject: [Cython] 
 =?utf-8?q?cython_0=2E23=2E4_test_failure=2C_async=5Fite?=
 =?utf-8?q?r=5Fpep492_=28=29=2C_asyncio=5Fgenerators=2Ctest=5Fasync=5Fdef?=
 =?utf-8?q?=5Ffuture=2Epy?=
References: <CANBHLUgyWd8EanZksNeieORsf994=1b3B8C_oE4YgUJqWcJ43Q@mail.gmail.com>
Message-ID: <loom.20160113T170129-92@post.gmane.org>

Dimitri John Ledkov <xnox at ...> writes:

> 
> Hello,
> 
> Building 0.23.4 fails on ubuntu with python 3.5.
> 
> Full build log is at:
>
https://launchpadlibrarian.net/228728126/buildlog_ubuntu-xenial
-amd64.cython_0.23.4-0ubuntu1_BUILDING.txt.gz
> 
> The what i think relevant portion of the build log is as follows:
> 
> End-to-end asyncio_generators ... FAIL
> runTest (__main__.CythonRunTestCase)
> compiling (c) and running attr ... attr ()
> Doctest: attr ... /usr/bin/python3.5 test_async_def_future.py
> 
> Traceback (most recent call last):
>   File "test_async_def_future.py", line 16, in <module>
>     runloop()
>   File "test_async_def_future.py", line 14, in runloop
>     assert events == expected, events
> AssertionError: ['setup', 'setval', None]
> 
> Any help resolving this would help. Is this a broken python3.5? or
> cython? or both?

We are seeing the same thing building cython 0.23.4 on Fedora rawhide for
python 3.5.1:

https://kojipkgs.fedoraproject.org//work/tasks/5210/12535210/build.log

- Orion


From xnox at ubuntu.com  Thu Jan 14 08:26:43 2016
From: xnox at ubuntu.com (Dimitri John Ledkov)
Date: Thu, 14 Jan 2016 13:26:43 +0000
Subject: [Cython] cython 0.23.4 test failure, async_iter_pep492 (),
 asyncio_generators, test_async_def_future.py
In-Reply-To: <loom.20160113T170129-92@post.gmane.org>
References: <CANBHLUgyWd8EanZksNeieORsf994=1b3B8C_oE4YgUJqWcJ43Q@mail.gmail.com>
 <loom.20160113T170129-92@post.gmane.org>
Message-ID: <CANBHLUinaYD2Q_DRiQdERdY4GNO7TWuMnhi1-34pL+4s5WQrsA@mail.gmail.com>

On 13 January 2016 at 17:50, Orion Poplawski <orion at cora.nwra.com> wrote:
> Dimitri John Ledkov <xnox at ...> writes:
>
>>
>> Hello,
>>
>> Building 0.23.4 fails on ubuntu with python 3.5.
>>
>> Full build log is at:
>>
> https://launchpadlibrarian.net/228728126/buildlog_ubuntu-xenial
> -amd64.cython_0.23.4-0ubuntu1_BUILDING.txt.gz
>>
>> The what i think relevant portion of the build log is as follows:
>>
>> End-to-end asyncio_generators ... FAIL
>> runTest (__main__.CythonRunTestCase)
>> compiling (c) and running attr ... attr ()
>> Doctest: attr ... /usr/bin/python3.5 test_async_def_future.py
>>
>> Traceback (most recent call last):
>>   File "test_async_def_future.py", line 16, in <module>
>>     runloop()
>>   File "test_async_def_future.py", line 14, in runloop
>>     assert events == expected, events
>> AssertionError: ['setup', 'setval', None]
>>
>> Any help resolving this would help. Is this a broken python3.5? or
>> cython? or both?
>
> We are seeing the same thing building cython 0.23.4 on Fedora rawhide for
> python 3.5.1:
>
> https://kojipkgs.fedoraproject.org//work/tasks/5210/12535210/build.log

I've also filed an issue against asyncio module on github, but i was
redirected by to cypthon.

https://github.com/python/asyncio/issues/308

On Ubuntu / Launchpad this is tracked here
https://bugs.launchpad.net/ubuntu/+source/cython/+bug/1526613

Reverting the changes between 3.5.0 and 3.5.1 of the asyncio module
alone, makes the cyphon test-suite pass again.

But I cannot reproduce the thing being tested in a stand-alone
scenario, to validate if it's cython that is depending on undefined
behaviour and/or asyncio module has something broken.

-- 
Regards,

Dimitri.

From toddrme2178 at gmail.com  Wed Jan 20 13:36:04 2016
From: toddrme2178 at gmail.com (Todd Rme)
Date: Wed, 20 Jan 2016 19:36:04 +0100
Subject: [Cython]  cython 0.23.4 test failure, async_iter_pep492 (),
 asyncio_generators, test_async_def_future.py
In-Reply-To: <CADb7s=u7wmjYG2ttifF65Fys6oy9Rk-ssPrinqmLORqotLk-Ug@mail.gmail.com>
References: <CADb7s=tLvZOzqK+PsuCKT5=Nr0M57yscPLs+G-=oENJx+gZNrQ@mail.gmail.com>
 <CADb7s=tEn2TpPFx22AJOQMC+WtzS6VLu4KsOeqmOeoc=zYuz7w@mail.gmail.com>
 <CADb7s=u7wmjYG2ttifF65Fys6oy9Rk-ssPrinqmLORqotLk-Ug@mail.gmail.com>
Message-ID: <CADb7s=taiZBkxgaC=v8S2ER04e7qYc5cdcTg0zr9z7_JdJF6TQ@mail.gmail.com>

On 13 January 2016 at 17:50, Orion Poplawski <orion at
<https://mail.python.org/mailman/listinfo/cython-devel> cora.nwra.com
<https://mail.python.org/mailman/listinfo/cython-devel>> wrote: >* Dimitri
John** Ledkov** <**xnox*
<https://mail.python.org/mailman/listinfo/cython-devel>* at ...*
<https://mail.python.org/mailman/listinfo/cython-devel>*> writes:* > >> >>*
Hello,* >> >>* Building 0.23.4 fails on** ubuntu** with python 3.5.* >> >>*
Full build log is at:* >> >* https*
<https://launchpadlibrarian.net/228728126/buildlog_ubuntu-xenial>*://*
<https://launchpadlibrarian.net/228728126/buildlog_ubuntu-xenial>
*launchpadlibrarian.net*
<https://launchpadlibrarian.net/228728126/buildlog_ubuntu-xenial>
*/228728126/*
<https://launchpadlibrarian.net/228728126/buildlog_ubuntu-xenial>*buildlog*
<https://launchpadlibrarian.net/228728126/buildlog_ubuntu-xenial>*_*
<https://launchpadlibrarian.net/228728126/buildlog_ubuntu-xenial>
*ubuntu-xenial*
<https://launchpadlibrarian.net/228728126/buildlog_ubuntu-xenial> >*
-amd64.**cython**_0.23.4-0ubuntu1_**BUILDING.txt.gz* >> >>* The what i
think relevant portion of the build log is as follows:* >> >>* End-to-end**
asyncio**_generators ... FAIL* >>* runTest** (__main__.**CythonRunTestCase*
*)* >>* compiling (c) and running** attr** ...** attr** ()* >>* Doctest**:**
attr** ... /usr/bin/**python3.5** test_**async**_def_**future.py* >> >>*
Traceback (most recent call last):* >>* File "test_**async**_def_*
*future.py**", line 16, in <module>* >>* runloop**()* >>* File "test_*
*async**_def_**future.py**", line 14, in** runloop* >>* assert events ==
expected, events* >>* AssertionError**: ['setup',** 'setval'**, None]* >> >>*
Any help resolving this would help. Is this a broken** python3.5**? or* >>*
cython**? or both?* > >* We are seeing the same thing building cython
0.23.4 on Fedora rawhide for* >* python 3.5.1:* > >* https://*
<https://kojipkgs.fedoraproject.org//work/tasks/5210/12535210/build.log>
*kojipkgs.fedoraproject.org*
<https://kojipkgs.fedoraproject.org//work/tasks/5210/12535210/build.log>
*//work/tasks/5210/12535210/*
<https://kojipkgs.fedoraproject.org//work/tasks/5210/12535210/build.log>
*build.log*
<https://kojipkgs.fedoraproject.org//work/tasks/5210/12535210/build.log>

We are seeing the same failure on openSUSE, and I think I have worked out
part of the reason why.

I have been playing around with the failing unit test using travis and my
own github fork.  If I enable testing on python 3.5.1 or 3.5-dev the
problem appears there too.  Cython doesn't currently test against these,
which is why the issue wasn't discovered earlier.

I have discovered that the change in behaviour between 3.5 and 3.5.1 occurs
in line 224 of  cython/tests/run/asyncio_generators.srctree

In this line, in python 3.5 the "await fut" statement returned the result
of "fut", 123.  In python 3.5.1+, however, it returns None instead.  The
result value is successfully getting set, as seen by checking
"fut.get_result()" after the await, but the await is not returning it.

You would think this would be an upstream Python problem, so I tried it in
a standalone python interpreter.  However, the problem didn't happen, it
worked as it did in 3.5.  So I went back to the test and changed the
extension of "async_def_future" from "pyx" to "py" and the problem
disappeared!

So somehow cython is doing something different with await than normal
python, and this is causing await to return None in python 3.5.1+.
However, I do not know specifically what this is, or why it behaves
differently in 3.5 and 3.5.1
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20160120/f23e47c6/attachment.html>

From rpf2116 at columbia.edu  Fri Jan 29 14:02:54 2016
From: rpf2116 at columbia.edu (Elizabeth Fischer)
Date: Fri, 29 Jan 2016 14:02:54 -0500
Subject: [Cython] Cython 0.23.4: Crash with Template Stuff
Message-ID: <CAC_jL3z-JZB5HS5sBcP207DfROwVcmvzdda10Un75JgZNhxh7w@mail.gmail.com>

Cython is crashing on me when I compile.  See below.

-- Elizabeth

cython --version

Cython version 0.23.4

Error compiling Cython file:
------------------------------------------------------------
...
#        cdef int flags

def test_double_blitz(a):
cdef vector[int] v
cdef cblitz.Array[double,1] a_b
a_b = cibmisc.np_to_blitz[double,1](a, b'var', [-1])
                         ^
------------------------------------------------------------

/Users/rpfische/git/ibmisc/pylib/ibmisc.pyx:38:26: Compiler crash in
AnalyseExpressionsTransform

ModuleNode.body = StatListNode(ibmisc.pyx:1:0)
StatListNode.stats[3] = StatListNode(ibmisc.pyx:35:0)
StatListNode.stats[0] = DefNode(ibmisc.pyx:35:0,
    modifiers = [...]/0,
    name = 'test_double_blitz',
    num_required_args = 1,
    py_wrapper_required = True,
    reqd_kw_flags_cname = '0',
    used = True)
File 'Nodes.py', line 430, in analyse_expressions:
StatListNode(ibmisc.pyx:36:1)
File 'Nodes.py', line 4775, in analyse_expressions:
SingleAssignmentNode(ibmisc.pyx:38:36)
File 'Nodes.py', line 4887, in analyse_types:
SingleAssignmentNode(ibmisc.pyx:38:36)
File 'ExprNodes.py', line 4640, in analyse_types:
SimpleCallNode(ibmisc.pyx:38:36,
    analysed = True,
    result_is_used = True,
    use_managed_ref = True)
File 'ExprNodes.py', line 3109, in analyse_types:
IndexNode(ibmisc.pyx:38:26,
    is_called = 1,
    is_subscript = True,
    result_is_used = True,
    use_managed_ref = True)
File 'ExprNodes.py', line 3419, in analyse_base_and_index_types:
IndexNode(ibmisc.pyx:38:26,
    is_called = 1,
    is_subscript = True,
    result_is_used = True,
    use_managed_ref = True)

Compiler crash traceback from this point on:
  File
"/Users/rpfische/macports/mpgompi-4.9.3/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/Cython/Compiler/ExprNodes.py",
line 3419, in analyse_base_and_index_types
    elif len(base_type.templates) != len(self.type_indices):
TypeError: object of type 'NoneType' has no len()
make[2]: *** [pylib/ibmisc.cpp] Error 1
make[2]: *** Deleting file `pylib/ibmisc.cpp'
make[1]: *** [pylib/CMakeFiles/ibmisc_so.dir/all] Error 2
make: *** [all] Error 2
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20160129/d177d452/attachment.html>