From markflorisson88 at gmail.com  Wed Jun  1 12:02:12 2011
From: markflorisson88 at gmail.com (mark florisson)
Date: Wed, 1 Jun 2011 12:02:12 +0200
Subject: [Cython] Bindings performance issue
In-Reply-To: <BANLkTi=yWzE3MJqq3CUwe8TCcPPkpXSuiA@mail.gmail.com>
References: <BANLkTi=yWzE3MJqq3CUwe8TCcPPkpXSuiA@mail.gmail.com>
Message-ID: <BANLkTik59bpyST=7RC6EJiLkrOPQHSujvg@mail.gmail.com>

On 31 May 2011 20:25, Vitja Makarov <vitja.makarov at gmail.com> wrote:
> Hi!
>
> Is bindings performance issue valuable?
>
> $ cat bindbench.pyx
> def wo_bindings():
> ? ?pass
>
> def outer():
> ? ?def inner():
> ? ? ? ?pass
> ? ?return inner
> with_bindings = outer()
>
> $ python
>>>> import timeit
>>>> timeit.repeat('with_bindings()', setup='from bindbench import wo_bindings, with_bindings', repeat=1, number=100000000)
> [6.169871807098389]
>>>> timeit.repeat('wo_bindings()', setup='from bindbench import wo_bindings, with_bindings', repeat=1, number=100000000)
> [4.609416961669922]
>
> PyCBindings makes it 1.3 (difference is about 15ns on my laptop) times
> slower for CPython interpreter execution.
> As CPython has some optimizations for CFunctions and PyCFunctions.
>
> Does it make sense for us? Or we can easily switch to bindings?
>
> --
> vitja.
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>

I think switching should be fine, if you'd desperately need the speed
you'd be calling c(p)def functions from Cython. In fact, when the
fused cfunction will be ready it will be even slightly slower, as it
overrides the tp_call. But perhaps that should really be made into a
subclass...

Anyway, would you use these for Python classes and module-level def
functions (and closures) only, or would you also use them in extension
classes? Because I found at least a bit of problem there, as extension
class methods get 'self' passed as the first argument to the C
function (as PyCFunctionObject.m_self), whereas methods in Python
classes get 'self' passed in the args tuple (and the m_self is
unused).

Mark

From vitja.makarov at gmail.com  Wed Jun  1 16:26:33 2011
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Wed, 1 Jun 2011 18:26:33 +0400
Subject: [Cython] Bindings performance issue
In-Reply-To: <BANLkTik59bpyST=7RC6EJiLkrOPQHSujvg@mail.gmail.com>
References: <BANLkTi=yWzE3MJqq3CUwe8TCcPPkpXSuiA@mail.gmail.com>
	<BANLkTik59bpyST=7RC6EJiLkrOPQHSujvg@mail.gmail.com>
Message-ID: <BANLkTik99WbBap5BgKyme2UBjtoiVrEATQ@mail.gmail.com>

2011/6/1 mark florisson <markflorisson88 at gmail.com>:
> On 31 May 2011 20:25, Vitja Makarov <vitja.makarov at gmail.com> wrote:
>> Hi!
>>
>> Is bindings performance issue valuable?
>>
>> $ cat bindbench.pyx
>> def wo_bindings():
>> ? ?pass
>>
>> def outer():
>> ? ?def inner():
>> ? ? ? ?pass
>> ? ?return inner
>> with_bindings = outer()
>>
>> $ python
>>>>> import timeit
>>>>> timeit.repeat('with_bindings()', setup='from bindbench import wo_bindings, with_bindings', repeat=1, number=100000000)
>> [6.169871807098389]
>>>>> timeit.repeat('wo_bindings()', setup='from bindbench import wo_bindings, with_bindings', repeat=1, number=100000000)
>> [4.609416961669922]
>>
>> PyCBindings makes it 1.3 (difference is about 15ns on my laptop) times
>> slower for CPython interpreter execution.
>> As CPython has some optimizations for CFunctions and PyCFunctions.
>>
>> Does it make sense for us? Or we can easily switch to bindings?
>>
>> --
>> vitja.
>> _______________________________________________
>> cython-devel mailing list
>> cython-devel at python.org
>> http://mail.python.org/mailman/listinfo/cython-devel
>>
>
> I think switching should be fine, if you'd desperately need the speed
> you'd be calling c(p)def functions from Cython. In fact, when the
> fused cfunction will be ready it will be even slightly slower, as it
> overrides the tp_call. But perhaps that should really be made into a
> subclass...
>
> Anyway, would you use these for Python classes and module-level def
> functions (and closures) only, or would you also use them in extension
> classes? Because I found at least a bit of problem there, as extension
> class methods get 'self' passed as the first argument to the C
> function (as PyCFunctionObject.m_self), whereas methods in Python
> classes get 'self' passed in the args tuple (and the m_self is
> unused).
>

Recently I've found a problem with static methods (__new__ for
example) of inner classes (classes created inside a function, for
instance)
I think binding version should be used for all regular def functions
(defs, staticmethods, classmethods and so on).
I also think that it's better to rename binding_PyCFunction_Type to
something like CyFunctionType and make it more like PyFunction.


-- 
vitja.

From robertwb at math.washington.edu  Thu Jun  2 07:49:02 2011
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 1 Jun 2011 22:49:02 -0700
Subject: [Cython] Bindings performance issue
In-Reply-To: <BANLkTik99WbBap5BgKyme2UBjtoiVrEATQ@mail.gmail.com>
References: <BANLkTi=yWzE3MJqq3CUwe8TCcPPkpXSuiA@mail.gmail.com>
	<BANLkTik59bpyST=7RC6EJiLkrOPQHSujvg@mail.gmail.com>
	<BANLkTik99WbBap5BgKyme2UBjtoiVrEATQ@mail.gmail.com>
Message-ID: <BANLkTikYr4qfkW-qA0-LcynQhdu9Zg6DcA@mail.gmail.com>

On Wed, Jun 1, 2011 at 7:26 AM, Vitja Makarov <vitja.makarov at gmail.com> wrote:
> 2011/6/1 mark florisson <markflorisson88 at gmail.com>:
>> On 31 May 2011 20:25, Vitja Makarov <vitja.makarov at gmail.com> wrote:
>>> Hi!
>>>
>>> Is bindings performance issue valuable?
>>>
>>> $ cat bindbench.pyx
>>> def wo_bindings():
>>> ? ?pass
>>>
>>> def outer():
>>> ? ?def inner():
>>> ? ? ? ?pass
>>> ? ?return inner
>>> with_bindings = outer()
>>>
>>> $ python
>>>>>> import timeit
>>>>>> timeit.repeat('with_bindings()', setup='from bindbench import wo_bindings, with_bindings', repeat=1, number=100000000)
>>> [6.169871807098389]
>>>>>> timeit.repeat('wo_bindings()', setup='from bindbench import wo_bindings, with_bindings', repeat=1, number=100000000)
>>> [4.609416961669922]
>>>
>>> PyCBindings makes it 1.3 (difference is about 15ns on my laptop) times
>>> slower for CPython interpreter execution.
>>> As CPython has some optimizations for CFunctions and PyCFunctions.
>>>
>>> Does it make sense for us? Or we can easily switch to bindings?
>>>
>>> --
>>> vitja.
>>> _______________________________________________
>>> cython-devel mailing list
>>> cython-devel at python.org
>>> http://mail.python.org/mailman/listinfo/cython-devel
>>>
>>
>> I think switching should be fine, if you'd desperately need the speed
>> you'd be calling c(p)def functions from Cython. In fact, when the
>> fused cfunction will be ready it will be even slightly slower, as it
>> overrides the tp_call. But perhaps that should really be made into a
>> subclass...

Yes, this should be a subclass, slowing all calls down is a bad idea.

>> Anyway, would you use these for Python classes and module-level def
>> functions (and closures) only, or would you also use them in extension
>> classes? Because I found at least a bit of problem there, as extension
>> class methods get 'self' passed as the first argument to the C
>> function (as PyCFunctionObject.m_self), whereas methods in Python
>> classes get 'self' passed in the args tuple (and the m_self is
>> unused).
>>
>
> Recently I've found a problem with static methods (__new__ for
> example) of inner classes (classes created inside a function, for
> instance)
> I think binding version should be used for all regular def functions
> (defs, staticmethods, classmethods and so on).

+1 to moving that way. (Well, static and class methods bind
differently, right?)

> I also think that it's better to rename binding_PyCFunction_Type to
> something like CyFunctionType and make it more like PyFunction.

+1

Is there any advantage to the method descriptors (other than that
they're easier for a human to write?)

I think that small speed regression for better compatibility is OK if
we add a directive to not create binding functions. (It'd be nice if
we could work around it, but that's hard as CPython has special
hooks...) The bigger issue is that the binding behavior is backwards
incompatible, and perhaps in a subtle way. Perhaps we need to have a
phase where all currently non-binding functions that will become
binding functions will raise an error (or at least a warning) to wean
people off of the old behavior before making a switch.

- Robert

From vitja.makarov at gmail.com  Thu Jun  2 10:30:01 2011
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Thu, 2 Jun 2011 12:30:01 +0400
Subject: [Cython] Bindings performance issue
In-Reply-To: <BANLkTikYr4qfkW-qA0-LcynQhdu9Zg6DcA@mail.gmail.com>
References: <BANLkTi=yWzE3MJqq3CUwe8TCcPPkpXSuiA@mail.gmail.com>
	<BANLkTik59bpyST=7RC6EJiLkrOPQHSujvg@mail.gmail.com>
	<BANLkTik99WbBap5BgKyme2UBjtoiVrEATQ@mail.gmail.com>
	<BANLkTikYr4qfkW-qA0-LcynQhdu9Zg6DcA@mail.gmail.com>
Message-ID: <BANLkTi=ps-zjHZxaodoGQ4djJQJRCU7zfw@mail.gmail.com>

2011/6/2 Robert Bradshaw <robertwb at math.washington.edu>:
> On Wed, Jun 1, 2011 at 7:26 AM, Vitja Makarov <vitja.makarov at gmail.com> wrote:
>> 2011/6/1 mark florisson <markflorisson88 at gmail.com>:
>>> On 31 May 2011 20:25, Vitja Makarov <vitja.makarov at gmail.com> wrote:
>>>> Hi!
>>>>
>>>> Is bindings performance issue valuable?
>>>>
>>>> $ cat bindbench.pyx
>>>> def wo_bindings():
>>>> ? ?pass
>>>>
>>>> def outer():
>>>> ? ?def inner():
>>>> ? ? ? ?pass
>>>> ? ?return inner
>>>> with_bindings = outer()
>>>>
>>>> $ python
>>>>>>> import timeit
>>>>>>> timeit.repeat('with_bindings()', setup='from bindbench import wo_bindings, with_bindings', repeat=1, number=100000000)
>>>> [6.169871807098389]
>>>>>>> timeit.repeat('wo_bindings()', setup='from bindbench import wo_bindings, with_bindings', repeat=1, number=100000000)
>>>> [4.609416961669922]
>>>>
>>>> PyCBindings makes it 1.3 (difference is about 15ns on my laptop) times
>>>> slower for CPython interpreter execution.
>>>> As CPython has some optimizations for CFunctions and PyCFunctions.
>>>>
>>>> Does it make sense for us? Or we can easily switch to bindings?
>>>>
>>>> --
>>>> vitja.
>>>> _______________________________________________
>>>> cython-devel mailing list
>>>> cython-devel at python.org
>>>> http://mail.python.org/mailman/listinfo/cython-devel
>>>>
>>>
>>> I think switching should be fine, if you'd desperately need the speed
>>> you'd be calling c(p)def functions from Cython. In fact, when the
>>> fused cfunction will be ready it will be even slightly slower, as it
>>> overrides the tp_call. But perhaps that should really be made into a
>>> subclass...
>
> Yes, this should be a subclass, slowing all calls down is a bad idea.
>
>>> Anyway, would you use these for Python classes and module-level def
>>> functions (and closures) only, or would you also use them in extension
>>> classes? Because I found at least a bit of problem there, as extension
>>> class methods get 'self' passed as the first argument to the C
>>> function (as PyCFunctionObject.m_self), whereas methods in Python
>>> classes get 'self' passed in the args tuple (and the m_self is
>>> unused).
>>>
>>
>> Recently I've found a problem with static methods (__new__ for
>> example) of inner classes (classes created inside a function, for
>> instance)
>> I think binding version should be used for all regular def functions
>> (defs, staticmethods, classmethods and so on).
>
> +1 to moving that way. (Well, static and class methods bind
> differently, right?)
>
>> I also think that it's better to rename binding_PyCFunction_Type to
>> something like CyFunctionType and make it more like PyFunction.
>
> +1
>
> Is there any advantage to the method descriptors (other than that
> they're easier for a human to write?)

Initially bindings was written to support bound class methods (am I right?)
So when we use it for regular functions 'binding' in the name doesn't
reflect its purpose.

One the other hand it's much more easy to write manually.

About staticmethods: I think that CyFunction type should handle it as well
because staticmethods can have custom attributes and act just like
normal def one.

The difference is insde descr_get it should return bound method for
normal and self for staticmethods.

Here I've tried to support staticmethods inside bindings type:
https://github.com/vitek/cython/commit/c0725ab340a8173d8e6724c62be3a135df58980e

>
> I think that small speed regression for better compatibility is OK if
> we add a directive to not create binding functions. (It'd be nice if
> we could work around it, but that's hard as CPython has special
> hooks...) The bigger issue is that the binding behavior is backwards
> incompatible, and perhaps in a subtle way. Perhaps we need to have a
> phase where all currently non-binding functions that will become
> binding functions will raise an error (or at least a warning) to wean
> people off of the old behavior before making a switch.
>

Is the difference so significant?

-- 
vitja.

From robertwb at math.washington.edu  Thu Jun  2 18:31:20 2011
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 2 Jun 2011 09:31:20 -0700
Subject: [Cython] Bindings performance issue
In-Reply-To: <BANLkTi=ps-zjHZxaodoGQ4djJQJRCU7zfw@mail.gmail.com>
References: <BANLkTi=yWzE3MJqq3CUwe8TCcPPkpXSuiA@mail.gmail.com>
	<BANLkTik59bpyST=7RC6EJiLkrOPQHSujvg@mail.gmail.com>
	<BANLkTik99WbBap5BgKyme2UBjtoiVrEATQ@mail.gmail.com>
	<BANLkTikYr4qfkW-qA0-LcynQhdu9Zg6DcA@mail.gmail.com>
	<BANLkTi=ps-zjHZxaodoGQ4djJQJRCU7zfw@mail.gmail.com>
Message-ID: <BANLkTindDiWJEjZGbELa-dbz591EhzLPug@mail.gmail.com>

On Thu, Jun 2, 2011 at 1:30 AM, Vitja Makarov <vitja.makarov at gmail.com> wrote:
> 2011/6/2 Robert Bradshaw <robertwb at math.washington.edu>:
>> On Wed, Jun 1, 2011 at 7:26 AM, Vitja Makarov <vitja.makarov at gmail.com> wrote:
>>> 2011/6/1 mark florisson <markflorisson88 at gmail.com>:
>>>> On 31 May 2011 20:25, Vitja Makarov <vitja.makarov at gmail.com> wrote:
>>>>> Hi!
>>>>>
>>>>> Is bindings performance issue valuable?
>>>>>
>>>>> $ cat bindbench.pyx
>>>>> def wo_bindings():
>>>>> ? ?pass
>>>>>
>>>>> def outer():
>>>>> ? ?def inner():
>>>>> ? ? ? ?pass
>>>>> ? ?return inner
>>>>> with_bindings = outer()
>>>>>
>>>>> $ python
>>>>>>>> import timeit
>>>>>>>> timeit.repeat('with_bindings()', setup='from bindbench import wo_bindings, with_bindings', repeat=1, number=100000000)
>>>>> [6.169871807098389]
>>>>>>>> timeit.repeat('wo_bindings()', setup='from bindbench import wo_bindings, with_bindings', repeat=1, number=100000000)
>>>>> [4.609416961669922]
>>>>>
>>>>> PyCBindings makes it 1.3 (difference is about 15ns on my laptop) times
>>>>> slower for CPython interpreter execution.
>>>>> As CPython has some optimizations for CFunctions and PyCFunctions.
>>>>>
>>>>> Does it make sense for us? Or we can easily switch to bindings?
>>>>>
>>>>> --
>>>>> vitja.
>>>>> _______________________________________________
>>>>> cython-devel mailing list
>>>>> cython-devel at python.org
>>>>> http://mail.python.org/mailman/listinfo/cython-devel
>>>>>
>>>>
>>>> I think switching should be fine, if you'd desperately need the speed
>>>> you'd be calling c(p)def functions from Cython. In fact, when the
>>>> fused cfunction will be ready it will be even slightly slower, as it
>>>> overrides the tp_call. But perhaps that should really be made into a
>>>> subclass...
>>
>> Yes, this should be a subclass, slowing all calls down is a bad idea.
>>
>>>> Anyway, would you use these for Python classes and module-level def
>>>> functions (and closures) only, or would you also use them in extension
>>>> classes? Because I found at least a bit of problem there, as extension
>>>> class methods get 'self' passed as the first argument to the C
>>>> function (as PyCFunctionObject.m_self), whereas methods in Python
>>>> classes get 'self' passed in the args tuple (and the m_self is
>>>> unused).
>>>>
>>>
>>> Recently I've found a problem with static methods (__new__ for
>>> example) of inner classes (classes created inside a function, for
>>> instance)
>>> I think binding version should be used for all regular def functions
>>> (defs, staticmethods, classmethods and so on).
>>
>> +1 to moving that way. (Well, static and class methods bind
>> differently, right?)
>>
>>> I also think that it's better to rename binding_PyCFunction_Type to
>>> something like CyFunctionType and make it more like PyFunction.
>>
>> +1
>>
>> Is there any advantage to the method descriptors (other than that
>> they're easier for a human to write?)
>
> Initially bindings was written to support bound class methods (am I right?)
> So when we use it for regular functions 'binding' in the name doesn't
> reflect its purpose.

There's three kinds of functions we create: PyCFunctions, binding
PyCFunctions, and extension class methods (via descriptors). I was
asking about the latter.

As for the first two, it reflects the difference in behavior. If I
take the function and assign it to a class, it will bind as a method
when I go to look it up. This is arguably the biggest difference
between built-in functions and Python functions.

> One the other hand it's much more easy to write manually.
>
> About staticmethods: I think that CyFunction type should handle it as well
> because staticmethods can have custom attributes and act just like
> normal def one.
>
> The difference is insde descr_get it should return bound method for
> normal and self for staticmethods.

Yep. Does that merit a new class, or a flag that's checked on every
function invocation (false 99.9% of the time)?

> Here I've tried to support staticmethods inside bindings type:
> https://github.com/vitek/cython/commit/c0725ab340a8173d8e6724c62be3a135df58980e
>
>>
>> I think that small speed regression for better compatibility is OK if
>> we add a directive to not create binding functions. (It'd be nice if
>> we could work around it, but that's hard as CPython has special
>> hooks...) The bigger issue is that the binding behavior is backwards
>> incompatible, and perhaps in a subtle way. Perhaps we need to have a
>> phase where all currently non-binding functions that will become
>> binding functions will raise an error (or at least a warning) to wean
>> people off of the old behavior before making a switch.
>>
>
> Is the difference so significant?

If anyone is assigning a Cython function to an object and then using
it they're counting on the current non-binding behavior, and it will
break. The speed is probably a lesser issue, which is what benchmarks
are for.

- Robert

From robertwb at math.washington.edu  Thu Jun  2 18:39:29 2011
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 2 Jun 2011 09:39:29 -0700
Subject: [Cython] Fused types syntax
Message-ID: <BANLkTikrKixnW2EfgeeMjx16G33FMai33A@mail.gmail.com>

In looking at merging fused types, it's time to nail down the syntax.
The current implementation is

    ctypedef cython.fused_type(list, dict, object) fused_t

This requires an addition to the grammer to allow the "call" syntax in
a type declaration, as well as special casing to make it allowed only
in a typedef. What about

    cython.fused_type[list, dict, object].

One advantage is that indexing is already valid in type declarations,
and its the typical syntax for parameterized types. Thoughts? Any
other ideas?

- Robert


P.S. Anyone remember buffers and C++ templated types are dissallowed
as typedefs?

From sccolbert at gmail.com  Thu Jun  2 19:14:09 2011
From: sccolbert at gmail.com (Chris Colbert)
Date: Thu, 2 Jun 2011 12:14:09 -0500
Subject: [Cython] Fused types syntax
In-Reply-To: <BANLkTikrKixnW2EfgeeMjx16G33FMai33A@mail.gmail.com>
References: <BANLkTikrKixnW2EfgeeMjx16G33FMai33A@mail.gmail.com>
Message-ID: <BANLkTinuG=7kNV64n9_D+xPu4s+LLRtk0A@mail.gmail.com>

Not that my opinion carriers any weight, but I'm +1 on this.


On Thu, Jun 2, 2011 at 11:39 AM, Robert Bradshaw <
robertwb at math.washington.edu> wrote:

> In looking at merging fused types, it's time to nail down the syntax.
> The current implementation is
>
>    ctypedef cython.fused_type(list, dict, object) fused_t
>
> This requires an addition to the grammer to allow the "call" syntax in
> a type declaration, as well as special casing to make it allowed only
> in a typedef. What about
>
>    cython.fused_type[list, dict, object].
>
> One advantage is that indexing is already valid in type declarations,
> and its the typical syntax for parameterized types. Thoughts? Any
> other ideas?
>
> - Robert
>
>
> P.S. Anyone remember buffers and C++ templated types are dissallowed
> as typedefs?
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20110602/0b696304/attachment.html>

From markflorisson88 at gmail.com  Thu Jun  2 22:00:44 2011
From: markflorisson88 at gmail.com (mark florisson)
Date: Thu, 2 Jun 2011 22:00:44 +0200
Subject: [Cython] Bindings performance issue
In-Reply-To: <BANLkTindDiWJEjZGbELa-dbz591EhzLPug@mail.gmail.com>
References: <BANLkTi=yWzE3MJqq3CUwe8TCcPPkpXSuiA@mail.gmail.com>
	<BANLkTik59bpyST=7RC6EJiLkrOPQHSujvg@mail.gmail.com>
	<BANLkTik99WbBap5BgKyme2UBjtoiVrEATQ@mail.gmail.com>
	<BANLkTikYr4qfkW-qA0-LcynQhdu9Zg6DcA@mail.gmail.com>
	<BANLkTi=ps-zjHZxaodoGQ4djJQJRCU7zfw@mail.gmail.com>
	<BANLkTindDiWJEjZGbELa-dbz591EhzLPug@mail.gmail.com>
Message-ID: <BANLkTinAwrra+ru8PC4x_68Eou1iH9QACw@mail.gmail.com>

On 2 June 2011 18:31, Robert Bradshaw <robertwb at math.washington.edu> wrote:
> On Thu, Jun 2, 2011 at 1:30 AM, Vitja Makarov <vitja.makarov at gmail.com> wrote:
>> 2011/6/2 Robert Bradshaw <robertwb at math.washington.edu>:
>>> On Wed, Jun 1, 2011 at 7:26 AM, Vitja Makarov <vitja.makarov at gmail.com> wrote:
>>>> 2011/6/1 mark florisson <markflorisson88 at gmail.com>:
>>>>> On 31 May 2011 20:25, Vitja Makarov <vitja.makarov at gmail.com> wrote:
>>>>>> Hi!
>>>>>>
>>>>>> Is bindings performance issue valuable?
>>>>>>
>>>>>> $ cat bindbench.pyx
>>>>>> def wo_bindings():
>>>>>> ? ?pass
>>>>>>
>>>>>> def outer():
>>>>>> ? ?def inner():
>>>>>> ? ? ? ?pass
>>>>>> ? ?return inner
>>>>>> with_bindings = outer()
>>>>>>
>>>>>> $ python
>>>>>>>>> import timeit
>>>>>>>>> timeit.repeat('with_bindings()', setup='from bindbench import wo_bindings, with_bindings', repeat=1, number=100000000)
>>>>>> [6.169871807098389]
>>>>>>>>> timeit.repeat('wo_bindings()', setup='from bindbench import wo_bindings, with_bindings', repeat=1, number=100000000)
>>>>>> [4.609416961669922]
>>>>>>
>>>>>> PyCBindings makes it 1.3 (difference is about 15ns on my laptop) times
>>>>>> slower for CPython interpreter execution.
>>>>>> As CPython has some optimizations for CFunctions and PyCFunctions.
>>>>>>
>>>>>> Does it make sense for us? Or we can easily switch to bindings?
>>>>>>
>>>>>> --
>>>>>> vitja.
>>>>>> _______________________________________________
>>>>>> cython-devel mailing list
>>>>>> cython-devel at python.org
>>>>>> http://mail.python.org/mailman/listinfo/cython-devel
>>>>>>
>>>>>
>>>>> I think switching should be fine, if you'd desperately need the speed
>>>>> you'd be calling c(p)def functions from Cython. In fact, when the
>>>>> fused cfunction will be ready it will be even slightly slower, as it
>>>>> overrides the tp_call. But perhaps that should really be made into a
>>>>> subclass...
>>>
>>> Yes, this should be a subclass, slowing all calls down is a bad idea.
>>>
>>>>> Anyway, would you use these for Python classes and module-level def
>>>>> functions (and closures) only, or would you also use them in extension
>>>>> classes? Because I found at least a bit of problem there, as extension
>>>>> class methods get 'self' passed as the first argument to the C
>>>>> function (as PyCFunctionObject.m_self), whereas methods in Python
>>>>> classes get 'self' passed in the args tuple (and the m_self is
>>>>> unused).
>>>>>
>>>>
>>>> Recently I've found a problem with static methods (__new__ for
>>>> example) of inner classes (classes created inside a function, for
>>>> instance)
>>>> I think binding version should be used for all regular def functions
>>>> (defs, staticmethods, classmethods and so on).
>>>
>>> +1 to moving that way. (Well, static and class methods bind
>>> differently, right?)
>>>
>>>> I also think that it's better to rename binding_PyCFunction_Type to
>>>> something like CyFunctionType and make it more like PyFunction.
>>>
>>> +1
>>>
>>> Is there any advantage to the method descriptors (other than that
>>> they're easier for a human to write?)
>>
>> Initially bindings was written to support bound class methods (am I right?)
>> So when we use it for regular functions 'binding' in the name doesn't
>> reflect its purpose.
>
> There's three kinds of functions we create: PyCFunctions, binding
> PyCFunctions, and extension class methods (via descriptors). I was
> asking about the latter.

Yeah those descriptors have no advantage at all, it's simple
convenience. So we could just use our binding descriptor. The only
difference is that those descriptors return PyCFunctionObjects (which
may have special-cased performance-related advantages, at Vitja
demonstrated), and we would return a binding function (that would have
the same dict as the original).

I still believe that you'd have to do something different for
extension classes (e.g. depending on a flag pass 'self' as m_self or
in the args tuple, so tp_call would have to be overridden). Besides,
if you're going to return a binding pycfunction from descr_get, you'll
want to do type checking for 'self' for unbound method calls. I got
all that working for normal python classes here:
https://github.com/markflorisson88/cython/tree/fusedtypes . Currently
it is intermixed with support for fused Python functions, but seeing
you'd override tp_call anyway the overhead should be negligible.

> As for the first two, it reflects the difference in behavior. If I
> take the function and assign it to a class, it will bind as a method
> when I go to look it up. This is arguably the biggest difference
> between built-in functions and Python functions.
>
>> One the other hand it's much more easy to write manually.
>>
>> About staticmethods: I think that CyFunction type should handle it as well
>> because staticmethods can have custom attributes and act just like
>> normal def one.
>>
>> The difference is insde descr_get it should return bound method for
>> normal and self for staticmethods.
>
> Yep. Does that merit a new class, or a flag that's checked on every
> function invocation (false 99.9% of the time)?

Well, it only needs to do the check for when it gets bound (i.e., in
tp_descr_get). But I wonder if that check is really necessary? Why not
simply call the builtin staticmethod() to wrap a (binding) pycfunction
and put it in the class's dict? I think that that is how it works
currently. And when you wrap something with staticmethod() you expect
it to be a staticmethod object, i.e. uncallable and without a dict.

>> Here I've tried to support staticmethods inside bindings type:
>> https://github.com/vitek/cython/commit/c0725ab340a8173d8e6724c62be3a135df58980e
>>
>>>
>>> I think that small speed regression for better compatibility is OK if
>>> we add a directive to not create binding functions. (It'd be nice if
>>> we could work around it, but that's hard as CPython has special
>>> hooks...) The bigger issue is that the binding behavior is backwards
>>> incompatible, and perhaps in a subtle way. Perhaps we need to have a
>>> phase where all currently non-binding functions that will become
>>> binding functions will raise an error (or at least a warning) to wean
>>> people off of the old behavior before making a switch.
>>>
>>
>> Is the difference so significant?
>
> If anyone is assigning a Cython function to an object and then using
> it they're counting on the current non-binding behavior, and it will
> break. The speed is probably a lesser issue, which is what benchmarks
> are for.

If you're binding functions to classes without expecting it to ever
bind, you don't really have bitching rights when stuff breaks later
on. You should have been using staticmethod() to begin with. And we
never said that our functions would never bind :)

> - Robert
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>

From vitja.makarov at gmail.com  Thu Jun  2 22:06:46 2011
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Fri, 3 Jun 2011 00:06:46 +0400
Subject: [Cython] Bindings performance issue
In-Reply-To: <BANLkTindDiWJEjZGbELa-dbz591EhzLPug@mail.gmail.com>
References: <BANLkTi=yWzE3MJqq3CUwe8TCcPPkpXSuiA@mail.gmail.com>
	<BANLkTik59bpyST=7RC6EJiLkrOPQHSujvg@mail.gmail.com>
	<BANLkTik99WbBap5BgKyme2UBjtoiVrEATQ@mail.gmail.com>
	<BANLkTikYr4qfkW-qA0-LcynQhdu9Zg6DcA@mail.gmail.com>
	<BANLkTi=ps-zjHZxaodoGQ4djJQJRCU7zfw@mail.gmail.com>
	<BANLkTindDiWJEjZGbELa-dbz591EhzLPug@mail.gmail.com>
Message-ID: <BANLkTi=CUNBnA2aFaas4-iLJfCRAPStfoA@mail.gmail.com>

2011/6/2 Robert Bradshaw <robertwb at math.washington.edu>:
>> Initially bindings was written to support bound class methods (am I right?)
>> So when we use it for regular functions 'binding' in the name doesn't
>> reflect its purpose.
>
> There's three kinds of functions we create: PyCFunctions, binding
> PyCFunctions, and extension class methods (via descriptors). I was
> asking about the latter.
>
> As for the first two, it reflects the difference in behavior. If I
> take the function and assign it to a class, it will bind as a method
> when I go to look it up. This is arguably the biggest difference
> between built-in functions and Python functions.
>
>> One the other hand it's much more easy to write manually.
>>
>> About staticmethods: I think that CyFunction type should handle it as well
>> because staticmethods can have custom attributes and act just like
>> normal def one.
>>
>> The difference is insde descr_get it should return bound method for
>> normal and self for staticmethods.
>
> Yep. Does that merit a new class, or a flag that's checked on every
> function invocation (false 99.9% of the time)?
>

tp_descr_get() is called each time function gets bound, isn't so?
If it makes sens we can create both CyFunction and CyStaticFunction types.

>> Here I've tried to support staticmethods inside bindings type:
>> https://github.com/vitek/cython/commit/c0725ab340a8173d8e6724c62be3a135df58980e
>>
>>>
>>> I think that small speed regression for better compatibility is OK if
>>> we add a directive to not create binding functions. (It'd be nice if
>>> we could work around it, but that's hard as CPython has special
>>> hooks...) The bigger issue is that the binding behavior is backwards
>>> incompatible, and perhaps in a subtle way. Perhaps we need to have a
>>> phase where all currently non-binding functions that will become
>>> binding functions will raise an error (or at least a warning) to wean
>>> people off of the old behavior before making a switch.
>>>
>>
>> Is the difference so significant?
>
> If anyone is assigning a Cython function to an object and then using
> it they're counting on the current non-binding behavior, and it will
> break. The speed is probably a lesser issue, which is what benchmarks
> are for.
>




-- 
vitja.

From markflorisson88 at gmail.com  Thu Jun  2 22:07:54 2011
From: markflorisson88 at gmail.com (mark florisson)
Date: Thu, 2 Jun 2011 22:07:54 +0200
Subject: [Cython] Fused types syntax
In-Reply-To: <BANLkTikrKixnW2EfgeeMjx16G33FMai33A@mail.gmail.com>
References: <BANLkTikrKixnW2EfgeeMjx16G33FMai33A@mail.gmail.com>
Message-ID: <BANLkTimkvXTSUogU3jHnkFqkfB5eOO_G0Q@mail.gmail.com>

On 2 June 2011 18:39, Robert Bradshaw <robertwb at math.washington.edu> wrote:
> In looking at merging fused types, it's time to nail down the syntax.
> The current implementation is
>
> ? ?ctypedef cython.fused_type(list, dict, object) fused_t
>
> This requires an addition to the grammer to allow the "call" syntax in
> a type declaration, as well as special casing to make it allowed only
> in a typedef. What about
>
> ? ?cython.fused_type[list, dict, object].
>
> One advantage is that indexing is already valid in type declarations,
> and its the typical syntax for parameterized types. Thoughts? Any
> other ideas?

I like it, so I'll give another +1. Changing wouldn't be too much work anyways.

> - Robert
>
>
> P.S. Anyone remember buffers and C++ templated types are dissallowed
> as typedefs?

Yes, there's even a test for it: tests/errors/buffertypedef_T117.pyx
(http://trac.cython.org/cython_trac/ticket/117). But it looks more
like an easy fix than a feature.

> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>

From robertwb at math.washington.edu  Thu Jun  2 22:16:54 2011
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 2 Jun 2011 13:16:54 -0700
Subject: [Cython] Bindings performance issue
In-Reply-To: <BANLkTinAwrra+ru8PC4x_68Eou1iH9QACw@mail.gmail.com>
References: <BANLkTi=yWzE3MJqq3CUwe8TCcPPkpXSuiA@mail.gmail.com>
	<BANLkTik59bpyST=7RC6EJiLkrOPQHSujvg@mail.gmail.com>
	<BANLkTik99WbBap5BgKyme2UBjtoiVrEATQ@mail.gmail.com>
	<BANLkTikYr4qfkW-qA0-LcynQhdu9Zg6DcA@mail.gmail.com>
	<BANLkTi=ps-zjHZxaodoGQ4djJQJRCU7zfw@mail.gmail.com>
	<BANLkTindDiWJEjZGbELa-dbz591EhzLPug@mail.gmail.com>
	<BANLkTinAwrra+ru8PC4x_68Eou1iH9QACw@mail.gmail.com>
Message-ID: <BANLkTi=KnePkE7Lf7jWK4tDUzX1A9ufBKQ@mail.gmail.com>

On Thu, Jun 2, 2011 at 1:00 PM, mark florisson
<markflorisson88 at gmail.com> wrote:
> On 2 June 2011 18:31, Robert Bradshaw <robertwb at math.washington.edu> wrote:
>> On Thu, Jun 2, 2011 at 1:30 AM, Vitja Makarov <vitja.makarov at gmail.com> wrote:
>>> 2011/6/2 Robert Bradshaw <robertwb at math.washington.edu>:
>>>> On Wed, Jun 1, 2011 at 7:26 AM, Vitja Makarov <vitja.makarov at gmail.com> wrote:
>>>>> 2011/6/1 mark florisson <markflorisson88 at gmail.com>:
>>>>>> On 31 May 2011 20:25, Vitja Makarov <vitja.makarov at gmail.com> wrote:
>>>>>>> Hi!
>>>>>>>
>>>>>>> Is bindings performance issue valuable?
>>>>>>>
>>>>>>> $ cat bindbench.pyx
>>>>>>> def wo_bindings():
>>>>>>> ? ?pass
>>>>>>>
>>>>>>> def outer():
>>>>>>> ? ?def inner():
>>>>>>> ? ? ? ?pass
>>>>>>> ? ?return inner
>>>>>>> with_bindings = outer()
>>>>>>>
>>>>>>> $ python
>>>>>>>>>> import timeit
>>>>>>>>>> timeit.repeat('with_bindings()', setup='from bindbench import wo_bindings, with_bindings', repeat=1, number=100000000)
>>>>>>> [6.169871807098389]
>>>>>>>>>> timeit.repeat('wo_bindings()', setup='from bindbench import wo_bindings, with_bindings', repeat=1, number=100000000)
>>>>>>> [4.609416961669922]
>>>>>>>
>>>>>>> PyCBindings makes it 1.3 (difference is about 15ns on my laptop) times
>>>>>>> slower for CPython interpreter execution.
>>>>>>> As CPython has some optimizations for CFunctions and PyCFunctions.
>>>>>>>
>>>>>>> Does it make sense for us? Or we can easily switch to bindings?
>>>>>>>
>>>>>>> --
>>>>>>> vitja.
>>>>>>> _______________________________________________
>>>>>>> cython-devel mailing list
>>>>>>> cython-devel at python.org
>>>>>>> http://mail.python.org/mailman/listinfo/cython-devel
>>>>>>>
>>>>>>
>>>>>> I think switching should be fine, if you'd desperately need the speed
>>>>>> you'd be calling c(p)def functions from Cython. In fact, when the
>>>>>> fused cfunction will be ready it will be even slightly slower, as it
>>>>>> overrides the tp_call. But perhaps that should really be made into a
>>>>>> subclass...
>>>>
>>>> Yes, this should be a subclass, slowing all calls down is a bad idea.
>>>>
>>>>>> Anyway, would you use these for Python classes and module-level def
>>>>>> functions (and closures) only, or would you also use them in extension
>>>>>> classes? Because I found at least a bit of problem there, as extension
>>>>>> class methods get 'self' passed as the first argument to the C
>>>>>> function (as PyCFunctionObject.m_self), whereas methods in Python
>>>>>> classes get 'self' passed in the args tuple (and the m_self is
>>>>>> unused).
>>>>>>
>>>>>
>>>>> Recently I've found a problem with static methods (__new__ for
>>>>> example) of inner classes (classes created inside a function, for
>>>>> instance)
>>>>> I think binding version should be used for all regular def functions
>>>>> (defs, staticmethods, classmethods and so on).
>>>>
>>>> +1 to moving that way. (Well, static and class methods bind
>>>> differently, right?)
>>>>
>>>>> I also think that it's better to rename binding_PyCFunction_Type to
>>>>> something like CyFunctionType and make it more like PyFunction.
>>>>
>>>> +1
>>>>
>>>> Is there any advantage to the method descriptors (other than that
>>>> they're easier for a human to write?)
>>>
>>> Initially bindings was written to support bound class methods (am I right?)
>>> So when we use it for regular functions 'binding' in the name doesn't
>>> reflect its purpose.
>>
>> There's three kinds of functions we create: PyCFunctions, binding
>> PyCFunctions, and extension class methods (via descriptors). I was
>> asking about the latter.
>
> Yeah those descriptors have no advantage at all, it's simple
> convenience. So we could just use our binding descriptor. The only
> difference is that those descriptors return PyCFunctionObjects (which
> may have special-cased performance-related advantages, at Vitja
> demonstrated), and we would return a binding function (that would have
> the same dict as the original).

We could do this too on binding if we wanted...

> I still believe that you'd have to do something different for
> extension classes (e.g. depending on a flag pass 'self' as m_self or
> in the args tuple, so tp_call would have to be overridden). Besides,
> if you're going to return a binding pycfunction from descr_get, you'll
> want to do type checking for 'self' for unbound method calls. I got
> all that working for normal python classes here:
> https://github.com/markflorisson88/cython/tree/fusedtypes . Currently
> it is intermixed with support for fused Python functions, but seeing
> you'd override tp_call anyway the overhead should be negligible.

I was thinking of avoiding descriptors altogether, and simply
populating the cdef class dict.

>> As for the first two, it reflects the difference in behavior. If I
>> take the function and assign it to a class, it will bind as a method
>> when I go to look it up. This is arguably the biggest difference
>> between built-in functions and Python functions.
>>
>>> One the other hand it's much more easy to write manually.
>>>
>>> About staticmethods: I think that CyFunction type should handle it as well
>>> because staticmethods can have custom attributes and act just like
>>> normal def one.
>>>
>>> The difference is insde descr_get it should return bound method for
>>> normal and self for staticmethods.
>>
>> Yep. Does that merit a new class, or a flag that's checked on every
>> function invocation (false 99.9% of the time)?
>
> Well, it only needs to do the check for when it gets bound (i.e., in
> tp_descr_get). But I wonder if that check is really necessary? Why not
> simply call the builtin staticmethod() to wrap a (binding) pycfunction
> and put it in the class's dict? I think that that is how it works
> currently. And when you wrap something with staticmethod() you expect
> it to be a staticmethod object, i.e. uncallable and without a dict.

If we don't care much about the speed of static and class methods, we
could just use Pythons stanadard wrappers (and decorators) rather than
implement this ourselves. I'm leaning towards that. Maybe eventually
we'll want to support cdef static/class methods, but that would be an
entirely different mechanism.

>>> Here I've tried to support staticmethods inside bindings type:
>>> https://github.com/vitek/cython/commit/c0725ab340a8173d8e6724c62be3a135df58980e
>>>
>>>>
>>>> I think that small speed regression for better compatibility is OK if
>>>> we add a directive to not create binding functions. (It'd be nice if
>>>> we could work around it, but that's hard as CPython has special
>>>> hooks...) The bigger issue is that the binding behavior is backwards
>>>> incompatible, and perhaps in a subtle way. Perhaps we need to have a
>>>> phase where all currently non-binding functions that will become
>>>> binding functions will raise an error (or at least a warning) to wean
>>>> people off of the old behavior before making a switch.
>>>>
>>>
>>> Is the difference so significant?
>>
>> If anyone is assigning a Cython function to an object and then using
>> it they're counting on the current non-binding behavior, and it will
>> break. The speed is probably a lesser issue, which is what benchmarks
>> are for.
>
> If you're binding functions to classes without expecting it to ever
> bind, you don't really have bitching rights when stuff breaks later
> on. You should have been using staticmethod() to begin with. And we
> never said that our functions would never bind :)

No, you're assigning it to an object, counting on being able to call
it later on. E.g. the following is legal (though contrived in this
example):

sage: class A:
....:     pass
....:
sage: a = A()
sage: a.foo = max
sage: a.foo([1,2,3])
3

If instead of len, it was one of our functions, then it would be bad
to suddenly change the semantics, because it could still run but
produce bad answers (e.g. if we had implemented max, suddenly a would
be included in the comparison). This is why I proposed raising an
explicit error as an intermediate step.

If we don't promise anything, the contract is whatever the code does.
That's the problem with not having a specification (which would be
really nice, but is a lot of work).

- Robert

From robertwb at math.washington.edu  Thu Jun  2 22:21:00 2011
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 2 Jun 2011 13:21:00 -0700
Subject: [Cython] Fused types syntax
In-Reply-To: <BANLkTimkvXTSUogU3jHnkFqkfB5eOO_G0Q@mail.gmail.com>
References: <BANLkTikrKixnW2EfgeeMjx16G33FMai33A@mail.gmail.com>
	<BANLkTimkvXTSUogU3jHnkFqkfB5eOO_G0Q@mail.gmail.com>
Message-ID: <BANLkTimdp7WqcTa4n9L5Q_bMYvHBKW=xmA@mail.gmail.com>

On Thu, Jun 2, 2011 at 1:07 PM, mark florisson
<markflorisson88 at gmail.com> wrote:
> On 2 June 2011 18:39, Robert Bradshaw <robertwb at math.washington.edu> wrote:
>> In looking at merging fused types, it's time to nail down the syntax.
>> The current implementation is
>>
>> ? ?ctypedef cython.fused_type(list, dict, object) fused_t
>>
>> This requires an addition to the grammer to allow the "call" syntax in
>> a type declaration, as well as special casing to make it allowed only
>> in a typedef. What about
>>
>> ? ?cython.fused_type[list, dict, object].
>>
>> One advantage is that indexing is already valid in type declarations,
>> and its the typical syntax for parameterized types. Thoughts? Any
>> other ideas?
>
> I like it, so I'll give another +1. Changing wouldn't be too much work anyways.
>
>> - Robert
>>
>>
>> P.S. Anyone remember buffers and C++ templated types are dissallowed
>> as typedefs?
>
> Yes, there's even a test for it: tests/errors/buffertypedef_T117.pyx
> (http://trac.cython.org/cython_trac/ticket/117). But it looks more
> like an easy fix than a feature.

I could see how for buffers it would require at bit more work, but for
C++ types that shouldn't be a problem. In particular, I'm looking at
https://github.com/cython/cython/blob/master/Cython/Compiler/Parsing.py#L2615

- Robert

From markflorisson88 at gmail.com  Thu Jun  2 23:03:38 2011
From: markflorisson88 at gmail.com (mark florisson)
Date: Thu, 2 Jun 2011 23:03:38 +0200
Subject: [Cython] Bindings performance issue
In-Reply-To: <BANLkTi=KnePkE7Lf7jWK4tDUzX1A9ufBKQ@mail.gmail.com>
References: <BANLkTi=yWzE3MJqq3CUwe8TCcPPkpXSuiA@mail.gmail.com>
	<BANLkTik59bpyST=7RC6EJiLkrOPQHSujvg@mail.gmail.com>
	<BANLkTik99WbBap5BgKyme2UBjtoiVrEATQ@mail.gmail.com>
	<BANLkTikYr4qfkW-qA0-LcynQhdu9Zg6DcA@mail.gmail.com>
	<BANLkTi=ps-zjHZxaodoGQ4djJQJRCU7zfw@mail.gmail.com>
	<BANLkTindDiWJEjZGbELa-dbz591EhzLPug@mail.gmail.com>
	<BANLkTinAwrra+ru8PC4x_68Eou1iH9QACw@mail.gmail.com>
	<BANLkTi=KnePkE7Lf7jWK4tDUzX1A9ufBKQ@mail.gmail.com>
Message-ID: <BANLkTi=8szSqJTrwwvOZwx1x=BXFcs6hdQ@mail.gmail.com>

On 2 June 2011 22:16, Robert Bradshaw <robertwb at math.washington.edu> wrote:
> On Thu, Jun 2, 2011 at 1:00 PM, mark florisson
> <markflorisson88 at gmail.com> wrote:
>> On 2 June 2011 18:31, Robert Bradshaw <robertwb at math.washington.edu> wrote:
>>> On Thu, Jun 2, 2011 at 1:30 AM, Vitja Makarov <vitja.makarov at gmail.com> wrote:
>>>> 2011/6/2 Robert Bradshaw <robertwb at math.washington.edu>:
>>>>> On Wed, Jun 1, 2011 at 7:26 AM, Vitja Makarov <vitja.makarov at gmail.com> wrote:
>>>>>> 2011/6/1 mark florisson <markflorisson88 at gmail.com>:
>>>>>>> On 31 May 2011 20:25, Vitja Makarov <vitja.makarov at gmail.com> wrote:
>>>>>>>> Hi!
>>>>>>>>
>>>>>>>> Is bindings performance issue valuable?
>>>>>>>>
>>>>>>>> $ cat bindbench.pyx
>>>>>>>> def wo_bindings():
>>>>>>>> ? ?pass
>>>>>>>>
>>>>>>>> def outer():
>>>>>>>> ? ?def inner():
>>>>>>>> ? ? ? ?pass
>>>>>>>> ? ?return inner
>>>>>>>> with_bindings = outer()
>>>>>>>>
>>>>>>>> $ python
>>>>>>>>>>> import timeit
>>>>>>>>>>> timeit.repeat('with_bindings()', setup='from bindbench import wo_bindings, with_bindings', repeat=1, number=100000000)
>>>>>>>> [6.169871807098389]
>>>>>>>>>>> timeit.repeat('wo_bindings()', setup='from bindbench import wo_bindings, with_bindings', repeat=1, number=100000000)
>>>>>>>> [4.609416961669922]
>>>>>>>>
>>>>>>>> PyCBindings makes it 1.3 (difference is about 15ns on my laptop) times
>>>>>>>> slower for CPython interpreter execution.
>>>>>>>> As CPython has some optimizations for CFunctions and PyCFunctions.
>>>>>>>>
>>>>>>>> Does it make sense for us? Or we can easily switch to bindings?
>>>>>>>>
>>>>>>>> --
>>>>>>>> vitja.
>>>>>>>> _______________________________________________
>>>>>>>> cython-devel mailing list
>>>>>>>> cython-devel at python.org
>>>>>>>> http://mail.python.org/mailman/listinfo/cython-devel
>>>>>>>>
>>>>>>>
>>>>>>> I think switching should be fine, if you'd desperately need the speed
>>>>>>> you'd be calling c(p)def functions from Cython. In fact, when the
>>>>>>> fused cfunction will be ready it will be even slightly slower, as it
>>>>>>> overrides the tp_call. But perhaps that should really be made into a
>>>>>>> subclass...
>>>>>
>>>>> Yes, this should be a subclass, slowing all calls down is a bad idea.
>>>>>
>>>>>>> Anyway, would you use these for Python classes and module-level def
>>>>>>> functions (and closures) only, or would you also use them in extension
>>>>>>> classes? Because I found at least a bit of problem there, as extension
>>>>>>> class methods get 'self' passed as the first argument to the C
>>>>>>> function (as PyCFunctionObject.m_self), whereas methods in Python
>>>>>>> classes get 'self' passed in the args tuple (and the m_self is
>>>>>>> unused).
>>>>>>>
>>>>>>
>>>>>> Recently I've found a problem with static methods (__new__ for
>>>>>> example) of inner classes (classes created inside a function, for
>>>>>> instance)
>>>>>> I think binding version should be used for all regular def functions
>>>>>> (defs, staticmethods, classmethods and so on).
>>>>>
>>>>> +1 to moving that way. (Well, static and class methods bind
>>>>> differently, right?)
>>>>>
>>>>>> I also think that it's better to rename binding_PyCFunction_Type to
>>>>>> something like CyFunctionType and make it more like PyFunction.
>>>>>
>>>>> +1
>>>>>
>>>>> Is there any advantage to the method descriptors (other than that
>>>>> they're easier for a human to write?)
>>>>
>>>> Initially bindings was written to support bound class methods (am I right?)
>>>> So when we use it for regular functions 'binding' in the name doesn't
>>>> reflect its purpose.
>>>
>>> There's three kinds of functions we create: PyCFunctions, binding
>>> PyCFunctions, and extension class methods (via descriptors). I was
>>> asking about the latter.
>>
>> Yeah those descriptors have no advantage at all, it's simple
>> convenience. So we could just use our binding descriptor. The only
>> difference is that those descriptors return PyCFunctionObjects (which
>> may have special-cased performance-related advantages, at Vitja
>> demonstrated), and we would return a binding function (that would have
>> the same dict as the original).
>
> We could do this too on binding if we wanted...

That's what I mean, this all happens when we bind the function. If we
return a bound PyCFunctionObject, we lose our dict.

>> I still believe that you'd have to do something different for
>> extension classes (e.g. depending on a flag pass 'self' as m_self or
>> in the args tuple, so tp_call would have to be overridden). Besides,
>> if you're going to return a binding pycfunction from descr_get, you'll
>> want to do type checking for 'self' for unbound method calls. I got
>> all that working for normal python classes here:
>> https://github.com/markflorisson88/cython/tree/fusedtypes . Currently
>> it is intermixed with support for fused Python functions, but seeing
>> you'd override tp_call anyway the overhead should be negligible.
>
> I was thinking of avoiding descriptors altogether, and simply
> populating the cdef class dict.

Yes, that is my proposal. Without populating the class dict, you
cannot have custom pycfunctions.

>>> As for the first two, it reflects the difference in behavior. If I
>>> take the function and assign it to a class, it will bind as a method
>>> when I go to look it up. This is arguably the biggest difference
>>> between built-in functions and Python functions.
>>>
>>>> One the other hand it's much more easy to write manually.
>>>>
>>>> About staticmethods: I think that CyFunction type should handle it as well
>>>> because staticmethods can have custom attributes and act just like
>>>> normal def one.
>>>>
>>>> The difference is insde descr_get it should return bound method for
>>>> normal and self for staticmethods.
>>>
>>> Yep. Does that merit a new class, or a flag that's checked on every
>>> function invocation (false 99.9% of the time)?
>>
>> Well, it only needs to do the check for when it gets bound (i.e., in
>> tp_descr_get). But I wonder if that check is really necessary? Why not
>> simply call the builtin staticmethod() to wrap a (binding) pycfunction
>> and put it in the class's dict? I think that that is how it works
>> currently. And when you wrap something with staticmethod() you expect
>> it to be a staticmethod object, i.e. uncallable and without a dict.
>
> If we don't care much about the speed of static and class methods, we
> could just use Pythons stanadard wrappers (and decorators) rather than
> implement this ourselves. I'm leaning towards that. Maybe eventually
> we'll want to support cdef static/class methods, but that would be an
> entirely different mechanism.

Right, it seems that that's the thing that already happens. For cdef
classes we are using PyDescr_NewClassMethod and for Python classes we
are using PyClassMethod_New (in __Pyx_Method_ClassMethod). So this
conveniently fixes the 'self' (or 'cls') problem for classmethods. Of
course for normal methods the 'self' problem remains as you'd want to
keep the dict after binding and only want to implement one class
(implementing two types, one for Python classes and one for cdef
classes would be a bit inconvenient, considering also that fused types
need to be supported in either class (or in either subclass)).

>>>> Here I've tried to support staticmethods inside bindings type:
>>>> https://github.com/vitek/cython/commit/c0725ab340a8173d8e6724c62be3a135df58980e
>>>>
>>>>>
>>>>> I think that small speed regression for better compatibility is OK if
>>>>> we add a directive to not create binding functions. (It'd be nice if
>>>>> we could work around it, but that's hard as CPython has special
>>>>> hooks...) The bigger issue is that the binding behavior is backwards
>>>>> incompatible, and perhaps in a subtle way. Perhaps we need to have a
>>>>> phase where all currently non-binding functions that will become
>>>>> binding functions will raise an error (or at least a warning) to wean
>>>>> people off of the old behavior before making a switch.
>>>>>
>>>>
>>>> Is the difference so significant?
>>>
>>> If anyone is assigning a Cython function to an object and then using
>>> it they're counting on the current non-binding behavior, and it will
>>> break. The speed is probably a lesser issue, which is what benchmarks
>>> are for.
>>
>> If you're binding functions to classes without expecting it to ever
>> bind, you don't really have bitching rights when stuff breaks later
>> on. You should have been using staticmethod() to begin with. And we
>> never said that our functions would never bind :)
>
> No, you're assigning it to an object, counting on being able to call
> it later on. E.g. the following is legal (though contrived in this
> example):
>
> sage: class A:
> ....: ? ? pass
> ....:
> sage: a = A()
> sage: a.foo = max
> sage: a.foo([1,2,3])
> 3
>
> If instead of len, it was one of our functions, then it would be bad
> to suddenly change the semantics, because it could still run but
> produce bad answers (e.g. if we had implemented max, suddenly a would
> be included in the comparison). This is why I proposed raising an
> explicit error as an intermediate step.
>
> If we don't promise anything, the contract is whatever the code does.
> That's the problem with not having a specification (which would be
> really nice, but is a lot of work).

Functions on objects never get bound, they only get bound if they are
on the class. So your code would still work with binding functions.

> - Robert
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>

From robertwb at math.washington.edu  Thu Jun  2 23:13:32 2011
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 2 Jun 2011 14:13:32 -0700
Subject: [Cython] Bindings performance issue
In-Reply-To: <BANLkTi=8szSqJTrwwvOZwx1x=BXFcs6hdQ@mail.gmail.com>
References: <BANLkTi=yWzE3MJqq3CUwe8TCcPPkpXSuiA@mail.gmail.com>
	<BANLkTik59bpyST=7RC6EJiLkrOPQHSujvg@mail.gmail.com>
	<BANLkTik99WbBap5BgKyme2UBjtoiVrEATQ@mail.gmail.com>
	<BANLkTikYr4qfkW-qA0-LcynQhdu9Zg6DcA@mail.gmail.com>
	<BANLkTi=ps-zjHZxaodoGQ4djJQJRCU7zfw@mail.gmail.com>
	<BANLkTindDiWJEjZGbELa-dbz591EhzLPug@mail.gmail.com>
	<BANLkTinAwrra+ru8PC4x_68Eou1iH9QACw@mail.gmail.com>
	<BANLkTi=KnePkE7Lf7jWK4tDUzX1A9ufBKQ@mail.gmail.com>
	<BANLkTi=8szSqJTrwwvOZwx1x=BXFcs6hdQ@mail.gmail.com>
Message-ID: <BANLkTimELr=UmYLpt=wLaxiarFVP6j_ecg@mail.gmail.com>

On Thu, Jun 2, 2011 at 2:03 PM, mark florisson
<markflorisson88 at gmail.com> wrote:

>>>> If anyone is assigning a Cython function to an object and then using
>>>> it they're counting on the current non-binding behavior, and it will
>>>> break. The speed is probably a lesser issue, which is what benchmarks
>>>> are for.
>>>
>>> If you're binding functions to classes without expecting it to ever
>>> bind, you don't really have bitching rights when stuff breaks later
>>> on. You should have been using staticmethod() to begin with. And we
>>> never said that our functions would never bind :)
>>
>> No, you're assigning it to an object, counting on being able to call
>> it later on. E.g. the following is legal (though contrived in this
>> example):
>>
>> sage: class A:
>> ....: ? ? pass
>> ....:
>> sage: a = A()
>> sage: a.foo = max
>> sage: a.foo([1,2,3])
>> 3
>>
>> If instead of len, it was one of our functions, then it would be bad
>> to suddenly change the semantics, because it could still run but
>> produce bad answers (e.g. if we had implemented max, suddenly a would
>> be included in the comparison). This is why I proposed raising an
>> explicit error as an intermediate step.
>>
>> If we don't promise anything, the contract is whatever the code does.
>> That's the problem with not having a specification (which would be
>> really nice, but is a lot of work).
>
> Functions on objects never get bound, they only get bound if they are
> on the class. So your code would still work with binding functions.

True. It would still break "A.foo = max" though. I'm not saying we
should support or encourage this, but lets break it hard before we
break it subtly.

- Robert

From d.s.seljebotn at astro.uio.no  Thu Jun  2 23:18:01 2011
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Thu, 02 Jun 2011 23:18:01 +0200
Subject: [Cython] Fused types syntax
In-Reply-To: <BANLkTikrKixnW2EfgeeMjx16G33FMai33A@mail.gmail.com>
References: <BANLkTikrKixnW2EfgeeMjx16G33FMai33A@mail.gmail.com>
Message-ID: <4DE7FE09.7040001@astro.uio.no>

On 06/02/2011 06:39 PM, Robert Bradshaw wrote:
> In looking at merging fused types, it's time to nail down the syntax.
> The current implementation is
>
>      ctypedef cython.fused_type(list, dict, object) fused_t
>
> This requires an addition to the grammer to allow the "call" syntax in
> a type declaration, as well as special casing to make it allowed only
> in a typedef. What about
>
>      cython.fused_type[list, dict, object].
>
> One advantage is that indexing is already valid in type declarations,
> and its the typical syntax for parameterized types. Thoughts? Any
> other ideas?

I don't really like overloading [] even more, and I think () (or, 
perhaps, 'fused_type([list, dict, object])').

But I don't feel very strongly about it.

If you only want this allowed in typedefs, then, being puristic, I think 
that really a "fused type" is really different from a ctypedef, and that 
it would warrant something like a new keyword.

cdef fusedtype [list, dict, object] fused_t

That's rather horrible, but you get the gist. The thing is, once you use 
a ctypeudef, you really should allow

def f(fused_type(float, double) x, fused_type(float, double) y): ...

but then, how many fused types do you have, one or two?

So this makes it seem to me that using ctypedef is a rather horrible hack.

But, like I said, I don't feel strongly about this.


>
> P.S. Anyone remember buffers and C++ templated types are dissallowed
> as typedefs?

As for buffers I just think I never got around to it...

Dag Sverre

From markflorisson88 at gmail.com  Thu Jun  2 23:21:18 2011
From: markflorisson88 at gmail.com (mark florisson)
Date: Thu, 2 Jun 2011 23:21:18 +0200
Subject: [Cython] Bindings performance issue
In-Reply-To: <BANLkTimELr=UmYLpt=wLaxiarFVP6j_ecg@mail.gmail.com>
References: <BANLkTi=yWzE3MJqq3CUwe8TCcPPkpXSuiA@mail.gmail.com>
	<BANLkTik59bpyST=7RC6EJiLkrOPQHSujvg@mail.gmail.com>
	<BANLkTik99WbBap5BgKyme2UBjtoiVrEATQ@mail.gmail.com>
	<BANLkTikYr4qfkW-qA0-LcynQhdu9Zg6DcA@mail.gmail.com>
	<BANLkTi=ps-zjHZxaodoGQ4djJQJRCU7zfw@mail.gmail.com>
	<BANLkTindDiWJEjZGbELa-dbz591EhzLPug@mail.gmail.com>
	<BANLkTinAwrra+ru8PC4x_68Eou1iH9QACw@mail.gmail.com>
	<BANLkTi=KnePkE7Lf7jWK4tDUzX1A9ufBKQ@mail.gmail.com>
	<BANLkTi=8szSqJTrwwvOZwx1x=BXFcs6hdQ@mail.gmail.com>
	<BANLkTimELr=UmYLpt=wLaxiarFVP6j_ecg@mail.gmail.com>
Message-ID: <BANLkTinzJTPx58fs+C-R0Y26O=iFpJC-YA@mail.gmail.com>

On 2 June 2011 23:13, Robert Bradshaw <robertwb at math.washington.edu> wrote:
> On Thu, Jun 2, 2011 at 2:03 PM, mark florisson
> <markflorisson88 at gmail.com> wrote:
>
>>>>> If anyone is assigning a Cython function to an object and then using
>>>>> it they're counting on the current non-binding behavior, and it will
>>>>> break. The speed is probably a lesser issue, which is what benchmarks
>>>>> are for.
>>>>
>>>> If you're binding functions to classes without expecting it to ever
>>>> bind, you don't really have bitching rights when stuff breaks later
>>>> on. You should have been using staticmethod() to begin with. And we
>>>> never said that our functions would never bind :)
>>>
>>> No, you're assigning it to an object, counting on being able to call
>>> it later on. E.g. the following is legal (though contrived in this
>>> example):
>>>
>>> sage: class A:
>>> ....: ? ? pass
>>> ....:
>>> sage: a = A()
>>> sage: a.foo = max
>>> sage: a.foo([1,2,3])
>>> 3
>>>
>>> If instead of len, it was one of our functions, then it would be bad
>>> to suddenly change the semantics, because it could still run but
>>> produce bad answers (e.g. if we had implemented max, suddenly a would
>>> be included in the comparison). This is why I proposed raising an
>>> explicit error as an intermediate step.
>>>
>>> If we don't promise anything, the contract is whatever the code does.
>>> That's the problem with not having a specification (which would be
>>> really nice, but is a lot of work).
>>
>> Functions on objects never get bound, they only get bound if they are
>> on the class. So your code would still work with binding functions.
>
> True. It would still break "A.foo = max" though. I'm not saying we
> should support or encourage this, but lets break it hard before we
> break it subtly.

Again, such code is highly fragile and frankly incorrect to begin
with, as it's based on the assumption that "Cython functions" never
get bound. Getting functions (defined outside of class bodies) to bind
in classes is a feature, I sometimes found myself to want it. So
basically an error would be fine, but it would prevent normal usage as
we have it in Python.

> - Robert
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>

From njs at vorpus.org  Thu Jun  2 23:25:39 2011
From: njs at vorpus.org (Nathaniel Smith)
Date: Thu, 2 Jun 2011 14:25:39 -0700
Subject: [Cython] Fused types syntax
In-Reply-To: <4DE7FE09.7040001@astro.uio.no>
References: <BANLkTikrKixnW2EfgeeMjx16G33FMai33A@mail.gmail.com>
	<4DE7FE09.7040001@astro.uio.no>
Message-ID: <BANLkTi=KcdbZ-w=UWuKeX40XA3-X0_rLAg@mail.gmail.com>

On Jun 2, 2011 2:18 PM, "Dag Sverre Seljebotn" <d.s.seljebotn at astro.uio.no>
wrote:
> If you only want this allowed in typedefs, then, being puristic, I think
that really a "fused type" is really different from a ctypedef, and that it
would warrant something like a new keyword.
>
> cdef fusedtype [list, dict, object] fused_t

+1 (to some variant of this, not necessarily this exactly)

I find the argument that defining the same fused type twice results in two
different, incompatible types to be very compelling. That's a fundamental
difference from typedefs.

- Nathaniel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20110602/989c6e09/attachment.html>

From robertwb at math.washington.edu  Thu Jun  2 23:31:28 2011
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 2 Jun 2011 14:31:28 -0700
Subject: [Cython] Fused types syntax
In-Reply-To: <4DE7FE09.7040001@astro.uio.no>
References: <BANLkTikrKixnW2EfgeeMjx16G33FMai33A@mail.gmail.com>
	<4DE7FE09.7040001@astro.uio.no>
Message-ID: <BANLkTi=pm1_0gtbdi6pFt2-powphB-bT0A@mail.gmail.com>

On Thu, Jun 2, 2011 at 2:18 PM, Dag Sverre Seljebotn
<d.s.seljebotn at astro.uio.no> wrote:
> On 06/02/2011 06:39 PM, Robert Bradshaw wrote:
>>
>> In looking at merging fused types, it's time to nail down the syntax.
>> The current implementation is
>>
>> ? ? ctypedef cython.fused_type(list, dict, object) fused_t
>>
>> This requires an addition to the grammer to allow the "call" syntax in
>> a type declaration, as well as special casing to make it allowed only
>> in a typedef. What about
>>
>> ? ? cython.fused_type[list, dict, object].
>>
>> One advantage is that indexing is already valid in type declarations,
>> and its the typical syntax for parameterized types. Thoughts? Any
>> other ideas?
>
> I don't really like overloading [] even more, and I think () (or, perhaps,
> 'fused_type([list, dict, object])').
>
> But I don't feel very strongly about it.
>
> If you only want this allowed in typedefs, then, being puristic, I think
> that really a "fused type" is really different from a ctypedef, and that it
> would warrant something like a new keyword.
>
> cdef fusedtype [list, dict, object] fused_t
>
> That's rather horrible, but you get the gist. The thing is, once you use a
> ctypeudef, you really should allow
>
> def f(fused_type(float, double) x, fused_type(float, double) y): ...
>
> but then, how many fused types do you have, one or two?

Two, and you can't refer to them. Such anonymous types could be handy
for one-off functions, e.g.

    def f(fused_type[list, dict, object] x): ...

but maybe that's not needed.

> So this makes it seem to me that using ctypedef is a rather horrible hack.

Yeah, this is the crux of the issue. (To be clear, Mark's
implementation is good, it's our syntax that's hacky). It's perhaps
less ugly than
introducing a new keyword. A crazy idea:

cdef fused floating_t:
    float
    double
    long double

(in analogy with how we declare types like structs and enums).

> But, like I said, I don't feel strongly about this.
>
>
>>
>> P.S. Anyone remember buffers and C++ templated types are dissallowed
>> as typedefs?
>
> As for buffers I just think I never got around to it...

And in that case you can't just punt the typedef to C :).

- Robert

From robertwb at math.washington.edu  Thu Jun  2 23:34:38 2011
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 2 Jun 2011 14:34:38 -0700
Subject: [Cython] Bindings performance issue
In-Reply-To: <BANLkTinzJTPx58fs+C-R0Y26O=iFpJC-YA@mail.gmail.com>
References: <BANLkTi=yWzE3MJqq3CUwe8TCcPPkpXSuiA@mail.gmail.com>
	<BANLkTik59bpyST=7RC6EJiLkrOPQHSujvg@mail.gmail.com>
	<BANLkTik99WbBap5BgKyme2UBjtoiVrEATQ@mail.gmail.com>
	<BANLkTikYr4qfkW-qA0-LcynQhdu9Zg6DcA@mail.gmail.com>
	<BANLkTi=ps-zjHZxaodoGQ4djJQJRCU7zfw@mail.gmail.com>
	<BANLkTindDiWJEjZGbELa-dbz591EhzLPug@mail.gmail.com>
	<BANLkTinAwrra+ru8PC4x_68Eou1iH9QACw@mail.gmail.com>
	<BANLkTi=KnePkE7Lf7jWK4tDUzX1A9ufBKQ@mail.gmail.com>
	<BANLkTi=8szSqJTrwwvOZwx1x=BXFcs6hdQ@mail.gmail.com>
	<BANLkTimELr=UmYLpt=wLaxiarFVP6j_ecg@mail.gmail.com>
	<BANLkTinzJTPx58fs+C-R0Y26O=iFpJC-YA@mail.gmail.com>
Message-ID: <BANLkTi=Kn87FiQHY1pJ=rjTFR6HiqHPPCA@mail.gmail.com>

On Thu, Jun 2, 2011 at 2:21 PM, mark florisson
<markflorisson88 at gmail.com> wrote:
> On 2 June 2011 23:13, Robert Bradshaw <robertwb at math.washington.edu> wrote:
>> On Thu, Jun 2, 2011 at 2:03 PM, mark florisson
>> <markflorisson88 at gmail.com> wrote:
>>
>>>>>> If anyone is assigning a Cython function to an object and then using
>>>>>> it they're counting on the current non-binding behavior, and it will
>>>>>> break. The speed is probably a lesser issue, which is what benchmarks
>>>>>> are for.
>>>>>
>>>>> If you're binding functions to classes without expecting it to ever
>>>>> bind, you don't really have bitching rights when stuff breaks later
>>>>> on. You should have been using staticmethod() to begin with. And we
>>>>> never said that our functions would never bind :)
>>>>
>>>> No, you're assigning it to an object, counting on being able to call
>>>> it later on. E.g. the following is legal (though contrived in this
>>>> example):
>>>>
>>>> sage: class A:
>>>> ....: ? ? pass
>>>> ....:
>>>> sage: a = A()
>>>> sage: a.foo = max
>>>> sage: a.foo([1,2,3])
>>>> 3
>>>>
>>>> If instead of len, it was one of our functions, then it would be bad
>>>> to suddenly change the semantics, because it could still run but
>>>> produce bad answers (e.g. if we had implemented max, suddenly a would
>>>> be included in the comparison). This is why I proposed raising an
>>>> explicit error as an intermediate step.
>>>>
>>>> If we don't promise anything, the contract is whatever the code does.
>>>> That's the problem with not having a specification (which would be
>>>> really nice, but is a lot of work).
>>>
>>> Functions on objects never get bound, they only get bound if they are
>>> on the class. So your code would still work with binding functions.
>>
>> True. It would still break "A.foo = max" though. I'm not saying we
>> should support or encourage this, but lets break it hard before we
>> break it subtly.
>
> Again, such code is highly fragile and frankly incorrect to begin
> with, as it's based on the assumption that "Cython functions" never
> get bound.

I agree, but I bet there's code out there depending on it, in
particular workarounds for our current broken semantics will
themselves break.

> Getting functions (defined outside of class bodies) to bind
> in classes is a feature, I sometimes found myself to want it. So
> basically an error would be fine, but it would prevent normal usage as
> we have it in Python.

The error would just be for a transition period.

- Robert

From markflorisson88 at gmail.com  Thu Jun  2 23:42:28 2011
From: markflorisson88 at gmail.com (mark florisson)
Date: Thu, 2 Jun 2011 23:42:28 +0200
Subject: [Cython] Fused types syntax
In-Reply-To: <BANLkTi=pm1_0gtbdi6pFt2-powphB-bT0A@mail.gmail.com>
References: <BANLkTikrKixnW2EfgeeMjx16G33FMai33A@mail.gmail.com>
	<4DE7FE09.7040001@astro.uio.no>
	<BANLkTi=pm1_0gtbdi6pFt2-powphB-bT0A@mail.gmail.com>
Message-ID: <BANLkTikFSOTZ0LafUUC3QMqCcrU8kaG9Qw@mail.gmail.com>

On 2 June 2011 23:31, Robert Bradshaw <robertwb at math.washington.edu> wrote:
> On Thu, Jun 2, 2011 at 2:18 PM, Dag Sverre Seljebotn
> <d.s.seljebotn at astro.uio.no> wrote:
>> On 06/02/2011 06:39 PM, Robert Bradshaw wrote:
>>>
>>> In looking at merging fused types, it's time to nail down the syntax.
>>> The current implementation is
>>>
>>> ? ? ctypedef cython.fused_type(list, dict, object) fused_t
>>>
>>> This requires an addition to the grammer to allow the "call" syntax in
>>> a type declaration, as well as special casing to make it allowed only
>>> in a typedef. What about
>>>
>>> ? ? cython.fused_type[list, dict, object].
>>>
>>> One advantage is that indexing is already valid in type declarations,
>>> and its the typical syntax for parameterized types. Thoughts? Any
>>> other ideas?
>>
>> I don't really like overloading [] even more, and I think () (or, perhaps,
>> 'fused_type([list, dict, object])').
>>
>> But I don't feel very strongly about it.
>>
>> If you only want this allowed in typedefs, then, being puristic, I think
>> that really a "fused type" is really different from a ctypedef, and that it
>> would warrant something like a new keyword.
>>
>> cdef fusedtype [list, dict, object] fused_t
>>
>> That's rather horrible, but you get the gist. The thing is, once you use a
>> ctypeudef, you really should allow
>>
>> def f(fused_type(float, double) x, fused_type(float, double) y): ...
>>
>> but then, how many fused types do you have, one or two?
>
> Two, and you can't refer to them. Such anonymous types could be handy
> for one-off functions, e.g.
>
> ? ?def f(fused_type[list, dict, object] x): ...
>
> but maybe that's not needed.
>
>> So this makes it seem to me that using ctypedef is a rather horrible hack.
>
> Yeah, this is the crux of the issue. (To be clear, Mark's
> implementation is good, it's our syntax that's hacky). It's perhaps
> less ugly than
> introducing a new keyword. A crazy idea:
>
> cdef fused floating_t:
> ? ?float
> ? ?double
> ? ?long double
>
> (in analogy with how we declare types like structs and enums).

I like that syntax, the bad thing about ctypedef for me is that
everything goes on one line, at which point you have to start breaking
them if they get too long (which can happen quite easily with lenghty
names like 'long double complex'), which looks awkward.

So much for trying to reduce grammar changes, though :)

>> But, like I said, I don't feel strongly about this.
>>
>>
>>>
>>> P.S. Anyone remember buffers and C++ templated types are dissallowed
>>> as typedefs?
>>
>> As for buffers I just think I never got around to it...
>
> And in that case you can't just punt the typedef to C :).
>
> - Robert
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>

From markflorisson88 at gmail.com  Thu Jun  2 23:45:07 2011
From: markflorisson88 at gmail.com (mark florisson)
Date: Thu, 2 Jun 2011 23:45:07 +0200
Subject: [Cython] Bindings performance issue
In-Reply-To: <BANLkTi=Kn87FiQHY1pJ=rjTFR6HiqHPPCA@mail.gmail.com>
References: <BANLkTi=yWzE3MJqq3CUwe8TCcPPkpXSuiA@mail.gmail.com>
	<BANLkTik59bpyST=7RC6EJiLkrOPQHSujvg@mail.gmail.com>
	<BANLkTik99WbBap5BgKyme2UBjtoiVrEATQ@mail.gmail.com>
	<BANLkTikYr4qfkW-qA0-LcynQhdu9Zg6DcA@mail.gmail.com>
	<BANLkTi=ps-zjHZxaodoGQ4djJQJRCU7zfw@mail.gmail.com>
	<BANLkTindDiWJEjZGbELa-dbz591EhzLPug@mail.gmail.com>
	<BANLkTinAwrra+ru8PC4x_68Eou1iH9QACw@mail.gmail.com>
	<BANLkTi=KnePkE7Lf7jWK4tDUzX1A9ufBKQ@mail.gmail.com>
	<BANLkTi=8szSqJTrwwvOZwx1x=BXFcs6hdQ@mail.gmail.com>
	<BANLkTimELr=UmYLpt=wLaxiarFVP6j_ecg@mail.gmail.com>
	<BANLkTinzJTPx58fs+C-R0Y26O=iFpJC-YA@mail.gmail.com>
	<BANLkTi=Kn87FiQHY1pJ=rjTFR6HiqHPPCA@mail.gmail.com>
Message-ID: <BANLkTi=qUHAMp6ENB+1GhkSyhSypkNazJQ@mail.gmail.com>

On 2 June 2011 23:34, Robert Bradshaw <robertwb at math.washington.edu> wrote:
> On Thu, Jun 2, 2011 at 2:21 PM, mark florisson
> <markflorisson88 at gmail.com> wrote:
>> On 2 June 2011 23:13, Robert Bradshaw <robertwb at math.washington.edu> wrote:
>>> On Thu, Jun 2, 2011 at 2:03 PM, mark florisson
>>> <markflorisson88 at gmail.com> wrote:
>>>
>>>>>>> If anyone is assigning a Cython function to an object and then using
>>>>>>> it they're counting on the current non-binding behavior, and it will
>>>>>>> break. The speed is probably a lesser issue, which is what benchmarks
>>>>>>> are for.
>>>>>>
>>>>>> If you're binding functions to classes without expecting it to ever
>>>>>> bind, you don't really have bitching rights when stuff breaks later
>>>>>> on. You should have been using staticmethod() to begin with. And we
>>>>>> never said that our functions would never bind :)
>>>>>
>>>>> No, you're assigning it to an object, counting on being able to call
>>>>> it later on. E.g. the following is legal (though contrived in this
>>>>> example):
>>>>>
>>>>> sage: class A:
>>>>> ....: ? ? pass
>>>>> ....:
>>>>> sage: a = A()
>>>>> sage: a.foo = max
>>>>> sage: a.foo([1,2,3])
>>>>> 3
>>>>>
>>>>> If instead of len, it was one of our functions, then it would be bad
>>>>> to suddenly change the semantics, because it could still run but
>>>>> produce bad answers (e.g. if we had implemented max, suddenly a would
>>>>> be included in the comparison). This is why I proposed raising an
>>>>> explicit error as an intermediate step.
>>>>>
>>>>> If we don't promise anything, the contract is whatever the code does.
>>>>> That's the problem with not having a specification (which would be
>>>>> really nice, but is a lot of work).
>>>>
>>>> Functions on objects never get bound, they only get bound if they are
>>>> on the class. So your code would still work with binding functions.
>>>
>>> True. It would still break "A.foo = max" though. I'm not saying we
>>> should support or encourage this, but lets break it hard before we
>>> break it subtly.
>>
>> Again, such code is highly fragile and frankly incorrect to begin
>> with, as it's based on the assumption that "Cython functions" never
>> get bound.
>
> I agree, but I bet there's code out there depending on it, in
> particular workarounds for our current broken semantics will
> themselves break.

Workarounds wouldn't break, as they would wrap the non-binding
function in another object, and implement __get__ to return a new
object that, when called, would call the original function with 'self'
as the first argument.

>> Getting functions (defined outside of class bodies) to bind
>> in classes is a feature, I sometimes found myself to want it. So
>> basically an error would be fine, but it would prevent normal usage as
>> we have it in Python.
>
> The error would just be for a transition period.

The transition period would be for an entire release?

> - Robert
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>

From robertwb at math.washington.edu  Thu Jun  2 23:51:32 2011
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 2 Jun 2011 14:51:32 -0700
Subject: [Cython] Fused types syntax
In-Reply-To: <BANLkTikFSOTZ0LafUUC3QMqCcrU8kaG9Qw@mail.gmail.com>
References: <BANLkTikrKixnW2EfgeeMjx16G33FMai33A@mail.gmail.com>
	<4DE7FE09.7040001@astro.uio.no>
	<BANLkTi=pm1_0gtbdi6pFt2-powphB-bT0A@mail.gmail.com>
	<BANLkTikFSOTZ0LafUUC3QMqCcrU8kaG9Qw@mail.gmail.com>
Message-ID: <BANLkTimKheZ=u-L=8GOpu+K7EqFUTOG=Ow@mail.gmail.com>

On Thu, Jun 2, 2011 at 2:42 PM, mark florisson
<markflorisson88 at gmail.com> wrote:
> On 2 June 2011 23:31, Robert Bradshaw <robertwb at math.washington.edu> wrote:
>> On Thu, Jun 2, 2011 at 2:18 PM, Dag Sverre Seljebotn
>> <d.s.seljebotn at astro.uio.no> wrote:
>>> On 06/02/2011 06:39 PM, Robert Bradshaw wrote:
>>>>
>>>> In looking at merging fused types, it's time to nail down the syntax.
>>>> The current implementation is
>>>>
>>>> ? ? ctypedef cython.fused_type(list, dict, object) fused_t
>>>>
>>>> This requires an addition to the grammer to allow the "call" syntax in
>>>> a type declaration, as well as special casing to make it allowed only
>>>> in a typedef. What about
>>>>
>>>> ? ? cython.fused_type[list, dict, object].
>>>>
>>>> One advantage is that indexing is already valid in type declarations,
>>>> and its the typical syntax for parameterized types. Thoughts? Any
>>>> other ideas?
>>>
>>> I don't really like overloading [] even more, and I think () (or, perhaps,
>>> 'fused_type([list, dict, object])').
>>>
>>> But I don't feel very strongly about it.
>>>
>>> If you only want this allowed in typedefs, then, being puristic, I think
>>> that really a "fused type" is really different from a ctypedef, and that it
>>> would warrant something like a new keyword.
>>>
>>> cdef fusedtype [list, dict, object] fused_t
>>>
>>> That's rather horrible, but you get the gist. The thing is, once you use a
>>> ctypeudef, you really should allow
>>>
>>> def f(fused_type(float, double) x, fused_type(float, double) y): ...
>>>
>>> but then, how many fused types do you have, one or two?
>>
>> Two, and you can't refer to them. Such anonymous types could be handy
>> for one-off functions, e.g.
>>
>> ? ?def f(fused_type[list, dict, object] x): ...
>>
>> but maybe that's not needed.
>>
>>> So this makes it seem to me that using ctypedef is a rather horrible hack.
>>
>> Yeah, this is the crux of the issue. (To be clear, Mark's
>> implementation is good, it's our syntax that's hacky). It's perhaps
>> less ugly than
>> introducing a new keyword. A crazy idea:
>>
>> cdef fused floating_t:
>> ? ?float
>> ? ?double
>> ? ?long double
>>
>> (in analogy with how we declare types like structs and enums).
>
> I like that syntax, the bad thing about ctypedef for me is that
> everything goes on one line, at which point you have to start breaking
> them if they get too long (which can happen quite easily with lenghty
> names like 'long double complex'), which looks awkward.
>
> So much for trying to reduce grammar changes, though :)

Well, it introduces a new kind of node, rather than the kind of funny
interaction/restriction with typedef, which got me thinking about this
in the first place. Of course then you can't create an anonymous one
for use in a function (which may be viewed as a feature or a defect).

- Robert

From robertwb at math.washington.edu  Thu Jun  2 23:59:29 2011
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 2 Jun 2011 14:59:29 -0700
Subject: [Cython] Bindings performance issue
In-Reply-To: <BANLkTi=qUHAMp6ENB+1GhkSyhSypkNazJQ@mail.gmail.com>
References: <BANLkTi=yWzE3MJqq3CUwe8TCcPPkpXSuiA@mail.gmail.com>
	<BANLkTik59bpyST=7RC6EJiLkrOPQHSujvg@mail.gmail.com>
	<BANLkTik99WbBap5BgKyme2UBjtoiVrEATQ@mail.gmail.com>
	<BANLkTikYr4qfkW-qA0-LcynQhdu9Zg6DcA@mail.gmail.com>
	<BANLkTi=ps-zjHZxaodoGQ4djJQJRCU7zfw@mail.gmail.com>
	<BANLkTindDiWJEjZGbELa-dbz591EhzLPug@mail.gmail.com>
	<BANLkTinAwrra+ru8PC4x_68Eou1iH9QACw@mail.gmail.com>
	<BANLkTi=KnePkE7Lf7jWK4tDUzX1A9ufBKQ@mail.gmail.com>
	<BANLkTi=8szSqJTrwwvOZwx1x=BXFcs6hdQ@mail.gmail.com>
	<BANLkTimELr=UmYLpt=wLaxiarFVP6j_ecg@mail.gmail.com>
	<BANLkTinzJTPx58fs+C-R0Y26O=iFpJC-YA@mail.gmail.com>
	<BANLkTi=Kn87FiQHY1pJ=rjTFR6HiqHPPCA@mail.gmail.com>
	<BANLkTi=qUHAMp6ENB+1GhkSyhSypkNazJQ@mail.gmail.com>
Message-ID: <BANLkTinxxB4rAdfVK7RwynB+bcoTy=YvzA@mail.gmail.com>

On Thu, Jun 2, 2011 at 2:45 PM, mark florisson
<markflorisson88 at gmail.com> wrote:
> On 2 June 2011 23:34, Robert Bradshaw <robertwb at math.washington.edu> wrote:
>> On Thu, Jun 2, 2011 at 2:21 PM, mark florisson
>> <markflorisson88 at gmail.com> wrote:
>>> On 2 June 2011 23:13, Robert Bradshaw <robertwb at math.washington.edu> wrote:
>>>> On Thu, Jun 2, 2011 at 2:03 PM, mark florisson
>>>> <markflorisson88 at gmail.com> wrote:
>>>>
>>>>>>>> If anyone is assigning a Cython function to an object and then using
>>>>>>>> it they're counting on the current non-binding behavior, and it will
>>>>>>>> break. The speed is probably a lesser issue, which is what benchmarks
>>>>>>>> are for.
>>>>>>>
>>>>>>> If you're binding functions to classes without expecting it to ever
>>>>>>> bind, you don't really have bitching rights when stuff breaks later
>>>>>>> on. You should have been using staticmethod() to begin with. And we
>>>>>>> never said that our functions would never bind :)
>>>>>>
>>>>>> No, you're assigning it to an object, counting on being able to call
>>>>>> it later on. E.g. the following is legal (though contrived in this
>>>>>> example):
>>>>>>
>>>>>> sage: class A:
>>>>>> ....: ? ? pass
>>>>>> ....:
>>>>>> sage: a = A()
>>>>>> sage: a.foo = max
>>>>>> sage: a.foo([1,2,3])
>>>>>> 3
>>>>>>
>>>>>> If instead of len, it was one of our functions, then it would be bad
>>>>>> to suddenly change the semantics, because it could still run but
>>>>>> produce bad answers (e.g. if we had implemented max, suddenly a would
>>>>>> be included in the comparison). This is why I proposed raising an
>>>>>> explicit error as an intermediate step.
>>>>>>
>>>>>> If we don't promise anything, the contract is whatever the code does.
>>>>>> That's the problem with not having a specification (which would be
>>>>>> really nice, but is a lot of work).
>>>>>
>>>>> Functions on objects never get bound, they only get bound if they are
>>>>> on the class. So your code would still work with binding functions.
>>>>
>>>> True. It would still break "A.foo = max" though. I'm not saying we
>>>> should support or encourage this, but lets break it hard before we
>>>> break it subtly.
>>>
>>> Again, such code is highly fragile and frankly incorrect to begin
>>> with, as it's based on the assumption that "Cython functions" never
>>> get bound.
>>
>> I agree, but I bet there's code out there depending on it, in
>> particular workarounds for our current broken semantics will
>> themselves break.
>
> Workarounds wouldn't break, as they would wrap the non-binding
> function in another object, and implement __get__ to return a new
> object that, when called, would call the original function with 'self'
> as the first argument.

Depends on the workaround. I'm thinking workarounds like "oh, self
isn't getting passed, guess I'll have to pass it manually here..."

>>> Getting functions (defined outside of class bodies) to bind
>>> in classes is a feature, I sometimes found myself to want it. So
>>> basically an error would be fine, but it would prevent normal usage as
>>> we have it in Python.
>>
>> The error would just be for a transition period.
>
> The transition period would be for an entire release?

Yes, though hopefully a much shorter release cycle than this last one.
I just haven't had time to fix those remaining failing Sage tests, and
we keep merging in more and more stuff (which is good, but we're well
overdue for a release by now.)

- Robert

From markflorisson88 at gmail.com  Fri Jun  3 00:04:37 2011
From: markflorisson88 at gmail.com (mark florisson)
Date: Fri, 3 Jun 2011 00:04:37 +0200
Subject: [Cython] Bindings performance issue
In-Reply-To: <BANLkTinxxB4rAdfVK7RwynB+bcoTy=YvzA@mail.gmail.com>
References: <BANLkTi=yWzE3MJqq3CUwe8TCcPPkpXSuiA@mail.gmail.com>
	<BANLkTik59bpyST=7RC6EJiLkrOPQHSujvg@mail.gmail.com>
	<BANLkTik99WbBap5BgKyme2UBjtoiVrEATQ@mail.gmail.com>
	<BANLkTikYr4qfkW-qA0-LcynQhdu9Zg6DcA@mail.gmail.com>
	<BANLkTi=ps-zjHZxaodoGQ4djJQJRCU7zfw@mail.gmail.com>
	<BANLkTindDiWJEjZGbELa-dbz591EhzLPug@mail.gmail.com>
	<BANLkTinAwrra+ru8PC4x_68Eou1iH9QACw@mail.gmail.com>
	<BANLkTi=KnePkE7Lf7jWK4tDUzX1A9ufBKQ@mail.gmail.com>
	<BANLkTi=8szSqJTrwwvOZwx1x=BXFcs6hdQ@mail.gmail.com>
	<BANLkTimELr=UmYLpt=wLaxiarFVP6j_ecg@mail.gmail.com>
	<BANLkTinzJTPx58fs+C-R0Y26O=iFpJC-YA@mail.gmail.com>
	<BANLkTi=Kn87FiQHY1pJ=rjTFR6HiqHPPCA@mail.gmail.com>
	<BANLkTi=qUHAMp6ENB+1GhkSyhSypkNazJQ@mail.gmail.com>
	<BANLkTinxxB4rAdfVK7RwynB+bcoTy=YvzA@mail.gmail.com>
Message-ID: <BANLkTim+TibgtVgrT-1TG+OduGegAXtj-A@mail.gmail.com>

On 2 June 2011 23:59, Robert Bradshaw <robertwb at math.washington.edu> wrote:
> On Thu, Jun 2, 2011 at 2:45 PM, mark florisson
> <markflorisson88 at gmail.com> wrote:
>> On 2 June 2011 23:34, Robert Bradshaw <robertwb at math.washington.edu> wrote:
>>> On Thu, Jun 2, 2011 at 2:21 PM, mark florisson
>>> <markflorisson88 at gmail.com> wrote:
>>>> On 2 June 2011 23:13, Robert Bradshaw <robertwb at math.washington.edu> wrote:
>>>>> On Thu, Jun 2, 2011 at 2:03 PM, mark florisson
>>>>> <markflorisson88 at gmail.com> wrote:
>>>>>
>>>>>>>>> If anyone is assigning a Cython function to an object and then using
>>>>>>>>> it they're counting on the current non-binding behavior, and it will
>>>>>>>>> break. The speed is probably a lesser issue, which is what benchmarks
>>>>>>>>> are for.
>>>>>>>>
>>>>>>>> If you're binding functions to classes without expecting it to ever
>>>>>>>> bind, you don't really have bitching rights when stuff breaks later
>>>>>>>> on. You should have been using staticmethod() to begin with. And we
>>>>>>>> never said that our functions would never bind :)
>>>>>>>
>>>>>>> No, you're assigning it to an object, counting on being able to call
>>>>>>> it later on. E.g. the following is legal (though contrived in this
>>>>>>> example):
>>>>>>>
>>>>>>> sage: class A:
>>>>>>> ....: ? ? pass
>>>>>>> ....:
>>>>>>> sage: a = A()
>>>>>>> sage: a.foo = max
>>>>>>> sage: a.foo([1,2,3])
>>>>>>> 3
>>>>>>>
>>>>>>> If instead of len, it was one of our functions, then it would be bad
>>>>>>> to suddenly change the semantics, because it could still run but
>>>>>>> produce bad answers (e.g. if we had implemented max, suddenly a would
>>>>>>> be included in the comparison). This is why I proposed raising an
>>>>>>> explicit error as an intermediate step.
>>>>>>>
>>>>>>> If we don't promise anything, the contract is whatever the code does.
>>>>>>> That's the problem with not having a specification (which would be
>>>>>>> really nice, but is a lot of work).
>>>>>>
>>>>>> Functions on objects never get bound, they only get bound if they are
>>>>>> on the class. So your code would still work with binding functions.
>>>>>
>>>>> True. It would still break "A.foo = max" though. I'm not saying we
>>>>> should support or encourage this, but lets break it hard before we
>>>>> break it subtly.
>>>>
>>>> Again, such code is highly fragile and frankly incorrect to begin
>>>> with, as it's based on the assumption that "Cython functions" never
>>>> get bound.
>>>
>>> I agree, but I bet there's code out there depending on it, in
>>> particular workarounds for our current broken semantics will
>>> themselves break.
>>
>> Workarounds wouldn't break, as they would wrap the non-binding
>> function in another object, and implement __get__ to return a new
>> object that, when called, would call the original function with 'self'
>> as the first argument.
>
> Depends on the workaround. I'm thinking workarounds like "oh, self
> isn't getting passed, guess I'll have to pass it manually here..."

Ah, something like functools.partial. Yeah, that'd break :)

>>>> Getting functions (defined outside of class bodies) to bind
>>>> in classes is a feature, I sometimes found myself to want it. So
>>>> basically an error would be fine, but it would prevent normal usage as
>>>> we have it in Python.
>>>
>>> The error would just be for a transition period.
>>
>> The transition period would be for an entire release?
>
> Yes, though hopefully a much shorter release cycle than this last one.
> I just haven't had time to fix those remaining failing Sage tests, and
> we keep merging in more and more stuff (which is good, but we're well
> overdue for a release by now.)

Ok. Then perhaps it would be better to merge the support for fused
types in the next release? I won't be available until the 6th of July
for coding, so until then we'd be stuck with cython.fused_type() in a
ctypedef.

> - Robert
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>

From markflorisson88 at gmail.com  Fri Jun  3 00:05:40 2011
From: markflorisson88 at gmail.com (mark florisson)
Date: Fri, 3 Jun 2011 00:05:40 +0200
Subject: [Cython] Bindings performance issue
In-Reply-To: <BANLkTim+TibgtVgrT-1TG+OduGegAXtj-A@mail.gmail.com>
References: <BANLkTi=yWzE3MJqq3CUwe8TCcPPkpXSuiA@mail.gmail.com>
	<BANLkTik59bpyST=7RC6EJiLkrOPQHSujvg@mail.gmail.com>
	<BANLkTik99WbBap5BgKyme2UBjtoiVrEATQ@mail.gmail.com>
	<BANLkTikYr4qfkW-qA0-LcynQhdu9Zg6DcA@mail.gmail.com>
	<BANLkTi=ps-zjHZxaodoGQ4djJQJRCU7zfw@mail.gmail.com>
	<BANLkTindDiWJEjZGbELa-dbz591EhzLPug@mail.gmail.com>
	<BANLkTinAwrra+ru8PC4x_68Eou1iH9QACw@mail.gmail.com>
	<BANLkTi=KnePkE7Lf7jWK4tDUzX1A9ufBKQ@mail.gmail.com>
	<BANLkTi=8szSqJTrwwvOZwx1x=BXFcs6hdQ@mail.gmail.com>
	<BANLkTimELr=UmYLpt=wLaxiarFVP6j_ecg@mail.gmail.com>
	<BANLkTinzJTPx58fs+C-R0Y26O=iFpJC-YA@mail.gmail.com>
	<BANLkTi=Kn87FiQHY1pJ=rjTFR6HiqHPPCA@mail.gmail.com>
	<BANLkTi=qUHAMp6ENB+1GhkSyhSypkNazJQ@mail.gmail.com>
	<BANLkTinxxB4rAdfVK7RwynB+bcoTy=YvzA@mail.gmail.com>
	<BANLkTim+TibgtVgrT-1TG+OduGegAXtj-A@mail.gmail.com>
Message-ID: <BANLkTik_iZEpqb4yDn4saj0FAUkqz4qtwA@mail.gmail.com>

On 3 June 2011 00:04, mark florisson <markflorisson88 at gmail.com> wrote:
> On 2 June 2011 23:59, Robert Bradshaw <robertwb at math.washington.edu> wrote:
>> On Thu, Jun 2, 2011 at 2:45 PM, mark florisson
>> <markflorisson88 at gmail.com> wrote:
>>> On 2 June 2011 23:34, Robert Bradshaw <robertwb at math.washington.edu> wrote:
>>>> On Thu, Jun 2, 2011 at 2:21 PM, mark florisson
>>>> <markflorisson88 at gmail.com> wrote:
>>>>> On 2 June 2011 23:13, Robert Bradshaw <robertwb at math.washington.edu> wrote:
>>>>>> On Thu, Jun 2, 2011 at 2:03 PM, mark florisson
>>>>>> <markflorisson88 at gmail.com> wrote:
>>>>>>
>>>>>>>>>> If anyone is assigning a Cython function to an object and then using
>>>>>>>>>> it they're counting on the current non-binding behavior, and it will
>>>>>>>>>> break. The speed is probably a lesser issue, which is what benchmarks
>>>>>>>>>> are for.
>>>>>>>>>
>>>>>>>>> If you're binding functions to classes without expecting it to ever
>>>>>>>>> bind, you don't really have bitching rights when stuff breaks later
>>>>>>>>> on. You should have been using staticmethod() to begin with. And we
>>>>>>>>> never said that our functions would never bind :)
>>>>>>>>
>>>>>>>> No, you're assigning it to an object, counting on being able to call
>>>>>>>> it later on. E.g. the following is legal (though contrived in this
>>>>>>>> example):
>>>>>>>>
>>>>>>>> sage: class A:
>>>>>>>> ....: ? ? pass
>>>>>>>> ....:
>>>>>>>> sage: a = A()
>>>>>>>> sage: a.foo = max
>>>>>>>> sage: a.foo([1,2,3])
>>>>>>>> 3
>>>>>>>>
>>>>>>>> If instead of len, it was one of our functions, then it would be bad
>>>>>>>> to suddenly change the semantics, because it could still run but
>>>>>>>> produce bad answers (e.g. if we had implemented max, suddenly a would
>>>>>>>> be included in the comparison). This is why I proposed raising an
>>>>>>>> explicit error as an intermediate step.
>>>>>>>>
>>>>>>>> If we don't promise anything, the contract is whatever the code does.
>>>>>>>> That's the problem with not having a specification (which would be
>>>>>>>> really nice, but is a lot of work).
>>>>>>>
>>>>>>> Functions on objects never get bound, they only get bound if they are
>>>>>>> on the class. So your code would still work with binding functions.
>>>>>>
>>>>>> True. It would still break "A.foo = max" though. I'm not saying we
>>>>>> should support or encourage this, but lets break it hard before we
>>>>>> break it subtly.
>>>>>
>>>>> Again, such code is highly fragile and frankly incorrect to begin
>>>>> with, as it's based on the assumption that "Cython functions" never
>>>>> get bound.
>>>>
>>>> I agree, but I bet there's code out there depending on it, in
>>>> particular workarounds for our current broken semantics will
>>>> themselves break.
>>>
>>> Workarounds wouldn't break, as they would wrap the non-binding
>>> function in another object, and implement __get__ to return a new
>>> object that, when called, would call the original function with 'self'
>>> as the first argument.
>>
>> Depends on the workaround. I'm thinking workarounds like "oh, self
>> isn't getting passed, guess I'll have to pass it manually here..."
>
> Ah, something like functools.partial. Yeah, that'd break :)
>
>>>>> Getting functions (defined outside of class bodies) to bind
>>>>> in classes is a feature, I sometimes found myself to want it. So
>>>>> basically an error would be fine, but it would prevent normal usage as
>>>>> we have it in Python.
>>>>
>>>> The error would just be for a transition period.
>>>
>>> The transition period would be for an entire release?
>>
>> Yes, though hopefully a much shorter release cycle than this last one.
>> I just haven't had time to fix those remaining failing Sage tests, and
>> we keep merging in more and more stuff (which is good, but we're well
>> overdue for a release by now.)
>
> Ok. Then perhaps it would be better to merge the support for fused
> types in the next release? I won't be available until the 6th of July
> for coding, so until then we'd be stuck with cython.fused_type() in a
> ctypedef.

... where 'next' means not the upcoming release, but the one after that.

>> - Robert
>> _______________________________________________
>> cython-devel mailing list
>> cython-devel at python.org
>> http://mail.python.org/mailman/listinfo/cython-devel
>>
>

From robertwb at math.washington.edu  Fri Jun  3 00:22:25 2011
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 2 Jun 2011 15:22:25 -0700
Subject: [Cython] Bindings performance issue
In-Reply-To: <BANLkTim+TibgtVgrT-1TG+OduGegAXtj-A@mail.gmail.com>
References: <BANLkTi=yWzE3MJqq3CUwe8TCcPPkpXSuiA@mail.gmail.com>
	<BANLkTik59bpyST=7RC6EJiLkrOPQHSujvg@mail.gmail.com>
	<BANLkTik99WbBap5BgKyme2UBjtoiVrEATQ@mail.gmail.com>
	<BANLkTikYr4qfkW-qA0-LcynQhdu9Zg6DcA@mail.gmail.com>
	<BANLkTi=ps-zjHZxaodoGQ4djJQJRCU7zfw@mail.gmail.com>
	<BANLkTindDiWJEjZGbELa-dbz591EhzLPug@mail.gmail.com>
	<BANLkTinAwrra+ru8PC4x_68Eou1iH9QACw@mail.gmail.com>
	<BANLkTi=KnePkE7Lf7jWK4tDUzX1A9ufBKQ@mail.gmail.com>
	<BANLkTi=8szSqJTrwwvOZwx1x=BXFcs6hdQ@mail.gmail.com>
	<BANLkTimELr=UmYLpt=wLaxiarFVP6j_ecg@mail.gmail.com>
	<BANLkTinzJTPx58fs+C-R0Y26O=iFpJC-YA@mail.gmail.com>
	<BANLkTi=Kn87FiQHY1pJ=rjTFR6HiqHPPCA@mail.gmail.com>
	<BANLkTi=qUHAMp6ENB+1GhkSyhSypkNazJQ@mail.gmail.com>
	<BANLkTinxxB4rAdfVK7RwynB+bcoTy=YvzA@mail.gmail.com>
	<BANLkTim+TibgtVgrT-1TG+OduGegAXtj-A@mail.gmail.com>
Message-ID: <BANLkTin2A7iP3ocmwqfqv1JS0gotDZztGQ@mail.gmail.com>

On Thu, Jun 2, 2011 at 3:04 PM, mark florisson
<markflorisson88 at gmail.com> wrote:
> On 2 June 2011 23:59, Robert Bradshaw <robertwb at math.washington.edu> wrote:
>> On Thu, Jun 2, 2011 at 2:45 PM, mark florisson
>> <markflorisson88 at gmail.com> wrote:
>>> On 2 June 2011 23:34, Robert Bradshaw <robertwb at math.washington.edu> wrote:
>>>> On Thu, Jun 2, 2011 at 2:21 PM, mark florisson
>>>> <markflorisson88 at gmail.com> wrote:
>>>>> On 2 June 2011 23:13, Robert Bradshaw <robertwb at math.washington.edu> wrote:
>>>>>> On Thu, Jun 2, 2011 at 2:03 PM, mark florisson
>>>>>> <markflorisson88 at gmail.com> wrote:
>>>>>>
>>>>>>>>>> If anyone is assigning a Cython function to an object and then using
>>>>>>>>>> it they're counting on the current non-binding behavior, and it will
>>>>>>>>>> break. The speed is probably a lesser issue, which is what benchmarks
>>>>>>>>>> are for.
>>>>>>>>>
>>>>>>>>> If you're binding functions to classes without expecting it to ever
>>>>>>>>> bind, you don't really have bitching rights when stuff breaks later
>>>>>>>>> on. You should have been using staticmethod() to begin with. And we
>>>>>>>>> never said that our functions would never bind :)
>>>>>>>>
>>>>>>>> No, you're assigning it to an object, counting on being able to call
>>>>>>>> it later on. E.g. the following is legal (though contrived in this
>>>>>>>> example):
>>>>>>>>
>>>>>>>> sage: class A:
>>>>>>>> ....: ? ? pass
>>>>>>>> ....:
>>>>>>>> sage: a = A()
>>>>>>>> sage: a.foo = max
>>>>>>>> sage: a.foo([1,2,3])
>>>>>>>> 3
>>>>>>>>
>>>>>>>> If instead of len, it was one of our functions, then it would be bad
>>>>>>>> to suddenly change the semantics, because it could still run but
>>>>>>>> produce bad answers (e.g. if we had implemented max, suddenly a would
>>>>>>>> be included in the comparison). This is why I proposed raising an
>>>>>>>> explicit error as an intermediate step.
>>>>>>>>
>>>>>>>> If we don't promise anything, the contract is whatever the code does.
>>>>>>>> That's the problem with not having a specification (which would be
>>>>>>>> really nice, but is a lot of work).
>>>>>>>
>>>>>>> Functions on objects never get bound, they only get bound if they are
>>>>>>> on the class. So your code would still work with binding functions.
>>>>>>
>>>>>> True. It would still break "A.foo = max" though. I'm not saying we
>>>>>> should support or encourage this, but lets break it hard before we
>>>>>> break it subtly.
>>>>>
>>>>> Again, such code is highly fragile and frankly incorrect to begin
>>>>> with, as it's based on the assumption that "Cython functions" never
>>>>> get bound.
>>>>
>>>> I agree, but I bet there's code out there depending on it, in
>>>> particular workarounds for our current broken semantics will
>>>> themselves break.
>>>
>>> Workarounds wouldn't break, as they would wrap the non-binding
>>> function in another object, and implement __get__ to return a new
>>> object that, when called, would call the original function with 'self'
>>> as the first argument.
>>
>> Depends on the workaround. I'm thinking workarounds like "oh, self
>> isn't getting passed, guess I'll have to pass it manually here..."
>
> Ah, something like functools.partial. Yeah, that'd break :)
>
>>>>> Getting functions (defined outside of class bodies) to bind
>>>>> in classes is a feature, I sometimes found myself to want it. So
>>>>> basically an error would be fine, but it would prevent normal usage as
>>>>> we have it in Python.
>>>>
>>>> The error would just be for a transition period.
>>>
>>> The transition period would be for an entire release?
>>
>> Yes, though hopefully a much shorter release cycle than this last one.
>> I just haven't had time to fix those remaining failing Sage tests, and
>> we keep merging in more and more stuff (which is good, but we're well
>> overdue for a release by now.)
>
> Ok. Then perhaps it would be better to merge the support for fused
> types in the next release? I won't be available until the 6th of July
> for coding, so until then we'd be stuck with cython.fused_type() in a
> ctypedef.

Well, it's such a cool feature :). More to the point, it's not nearly
as invasive (i.e. compiling existing code will still pretty much work
the way it always has).

- Robert

From d.s.seljebotn at astro.uio.no  Fri Jun  3 10:39:42 2011
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Fri, 03 Jun 2011 10:39:42 +0200
Subject: [Cython] Fused types syntax
In-Reply-To: <BANLkTimKheZ=u-L=8GOpu+K7EqFUTOG=Ow@mail.gmail.com>
References: <BANLkTikrKixnW2EfgeeMjx16G33FMai33A@mail.gmail.com>	<4DE7FE09.7040001@astro.uio.no>	<BANLkTi=pm1_0gtbdi6pFt2-powphB-bT0A@mail.gmail.com>	<BANLkTikFSOTZ0LafUUC3QMqCcrU8kaG9Qw@mail.gmail.com>
	<BANLkTimKheZ=u-L=8GOpu+K7EqFUTOG=Ow@mail.gmail.com>
Message-ID: <4DE89DCE.80300@astro.uio.no>

On 06/02/2011 11:51 PM, Robert Bradshaw wrote:
> On Thu, Jun 2, 2011 at 2:42 PM, mark florisson
> <markflorisson88 at gmail.com>  wrote:
>> On 2 June 2011 23:31, Robert Bradshaw<robertwb at math.washington.edu>  wrote:
>>> On Thu, Jun 2, 2011 at 2:18 PM, Dag Sverre Seljebotn
>>> <d.s.seljebotn at astro.uio.no>  wrote:
>>>> On 06/02/2011 06:39 PM, Robert Bradshaw wrote:
>>>>>
>>>>> In looking at merging fused types, it's time to nail down the syntax.
>>>>> The current implementation is
>>>>>
>>>>>      ctypedef cython.fused_type(list, dict, object) fused_t
>>>>>
>>>>> This requires an addition to the grammer to allow the "call" syntax in
>>>>> a type declaration, as well as special casing to make it allowed only
>>>>> in a typedef. What about
>>>>>
>>>>>      cython.fused_type[list, dict, object].
>>>>>
>>>>> One advantage is that indexing is already valid in type declarations,
>>>>> and its the typical syntax for parameterized types. Thoughts? Any
>>>>> other ideas?
>>>>
>>>> I don't really like overloading [] even more, and I think () (or, perhaps,
>>>> 'fused_type([list, dict, object])').
>>>>
>>>> But I don't feel very strongly about it.
>>>>
>>>> If you only want this allowed in typedefs, then, being puristic, I think
>>>> that really a "fused type" is really different from a ctypedef, and that it
>>>> would warrant something like a new keyword.
>>>>
>>>> cdef fusedtype [list, dict, object] fused_t
>>>>
>>>> That's rather horrible, but you get the gist. The thing is, once you use a
>>>> ctypeudef, you really should allow
>>>>
>>>> def f(fused_type(float, double) x, fused_type(float, double) y): ...
>>>>
>>>> but then, how many fused types do you have, one or two?
>>>
>>> Two, and you can't refer to them. Such anonymous types could be handy
>>> for one-off functions, e.g.
>>>
>>>     def f(fused_type[list, dict, object] x): ...
>>>
>>> but maybe that's not needed.
>>>
>>>> So this makes it seem to me that using ctypedef is a rather horrible hack.
>>>
>>> Yeah, this is the crux of the issue. (To be clear, Mark's
>>> implementation is good, it's our syntax that's hacky). It's perhaps
>>> less ugly than
>>> introducing a new keyword. A crazy idea:
>>>
>>> cdef fused floating_t:
>>>     float
>>>     double
>>>     long double
>>>
>>> (in analogy with how we declare types like structs and enums).
>>
>> I like that syntax, the bad thing about ctypedef for me is that
>> everything goes on one line, at which point you have to start breaking
>> them if they get too long (which can happen quite easily with lenghty
>> names like 'long double complex'), which looks awkward.
>>
>> So much for trying to reduce grammar changes, though :)
>
> Well, it introduces a new kind of node, rather than the kind of funny
> interaction/restriction with typedef, which got me thinking about this
> in the first place. Of course then you can't create an anonymous one
> for use in a function (which may be viewed as a feature or a defect).

I'm +1 on this syntax, I really like it in fact. It also leaves a lot 
more room for anything else one might need to add in eventually (the 
"paired" idea of Pauli and so on).

And I think not being able to create an anonymous one is a feature.

Dag Sverre

From vitja.makarov at gmail.com  Sat Jun  4 12:24:42 2011
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Sat, 4 Jun 2011 14:24:42 +0400
Subject: [Cython] Bindings performance issue
In-Reply-To: <BANLkTin2A7iP3ocmwqfqv1JS0gotDZztGQ@mail.gmail.com>
References: <BANLkTi=yWzE3MJqq3CUwe8TCcPPkpXSuiA@mail.gmail.com>
	<BANLkTik59bpyST=7RC6EJiLkrOPQHSujvg@mail.gmail.com>
	<BANLkTik99WbBap5BgKyme2UBjtoiVrEATQ@mail.gmail.com>
	<BANLkTikYr4qfkW-qA0-LcynQhdu9Zg6DcA@mail.gmail.com>
	<BANLkTi=ps-zjHZxaodoGQ4djJQJRCU7zfw@mail.gmail.com>
	<BANLkTindDiWJEjZGbELa-dbz591EhzLPug@mail.gmail.com>
	<BANLkTinAwrra+ru8PC4x_68Eou1iH9QACw@mail.gmail.com>
	<BANLkTi=KnePkE7Lf7jWK4tDUzX1A9ufBKQ@mail.gmail.com>
	<BANLkTi=8szSqJTrwwvOZwx1x=BXFcs6hdQ@mail.gmail.com>
	<BANLkTimELr=UmYLpt=wLaxiarFVP6j_ecg@mail.gmail.com>
	<BANLkTinzJTPx58fs+C-R0Y26O=iFpJC-YA@mail.gmail.com>
	<BANLkTi=Kn87FiQHY1pJ=rjTFR6HiqHPPCA@mail.gmail.com>
	<BANLkTi=qUHAMp6ENB+1GhkSyhSypkNazJQ@mail.gmail.com>
	<BANLkTinxxB4rAdfVK7RwynB+bcoTy=YvzA@mail.gmail.com>
	<BANLkTim+TibgtVgrT-1TG+OduGegAXtj-A@mail.gmail.com>
	<BANLkTin2A7iP3ocmwqfqv1JS0gotDZztGQ@mail.gmail.com>
Message-ID: <BANLkTin36dOg--NE2zVRpjsDi0aaCbfa8g@mail.gmail.com>

I've tried that: https://github.com/vitek/cython/compare/master..._bindings

Results are not bad: 168 failing tests for pyregr2.7 and 463 for py3

-- 
vitja.

From markflorisson88 at gmail.com  Sat Jun  4 12:58:41 2011
From: markflorisson88 at gmail.com (mark florisson)
Date: Sat, 4 Jun 2011 12:58:41 +0200
Subject: [Cython] Bindings performance issue
In-Reply-To: <BANLkTin36dOg--NE2zVRpjsDi0aaCbfa8g@mail.gmail.com>
References: <BANLkTi=yWzE3MJqq3CUwe8TCcPPkpXSuiA@mail.gmail.com>
	<BANLkTik59bpyST=7RC6EJiLkrOPQHSujvg@mail.gmail.com>
	<BANLkTik99WbBap5BgKyme2UBjtoiVrEATQ@mail.gmail.com>
	<BANLkTikYr4qfkW-qA0-LcynQhdu9Zg6DcA@mail.gmail.com>
	<BANLkTi=ps-zjHZxaodoGQ4djJQJRCU7zfw@mail.gmail.com>
	<BANLkTindDiWJEjZGbELa-dbz591EhzLPug@mail.gmail.com>
	<BANLkTinAwrra+ru8PC4x_68Eou1iH9QACw@mail.gmail.com>
	<BANLkTi=KnePkE7Lf7jWK4tDUzX1A9ufBKQ@mail.gmail.com>
	<BANLkTi=8szSqJTrwwvOZwx1x=BXFcs6hdQ@mail.gmail.com>
	<BANLkTimELr=UmYLpt=wLaxiarFVP6j_ecg@mail.gmail.com>
	<BANLkTinzJTPx58fs+C-R0Y26O=iFpJC-YA@mail.gmail.com>
	<BANLkTi=Kn87FiQHY1pJ=rjTFR6HiqHPPCA@mail.gmail.com>
	<BANLkTi=qUHAMp6ENB+1GhkSyhSypkNazJQ@mail.gmail.com>
	<BANLkTinxxB4rAdfVK7RwynB+bcoTy=YvzA@mail.gmail.com>
	<BANLkTim+TibgtVgrT-1TG+OduGegAXtj-A@mail.gmail.com>
	<BANLkTin2A7iP3ocmwqfqv1JS0gotDZztGQ@mail.gmail.com>
	<BANLkTin36dOg--NE2zVRpjsDi0aaCbfa8g@mail.gmail.com>
Message-ID: <BANLkTinM9bwbZbz=_yYok7UXUqrbZwAsBw@mail.gmail.com>

On 4 June 2011 12:24, Vitja Makarov <vitja.makarov at gmail.com> wrote:
> I've tried that: https://github.com/vitek/cython/compare/master..._bindings
>
> Results are not bad: 168 failing tests for pyregr2.7 and 463 for py3

Nice, it partly duplicates work in my fusedtypes branch but I suppose
it will have to be reworked anyway into a subclass. So this works only
for normal classes right, i.e. not for extension classes? It's also
pickle-able, cool :)

> --
> vitja.
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>

From vitja.makarov at gmail.com  Sat Jun  4 13:17:14 2011
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Sat, 4 Jun 2011 15:17:14 +0400
Subject: [Cython] Bindings performance issue
In-Reply-To: <BANLkTinM9bwbZbz=_yYok7UXUqrbZwAsBw@mail.gmail.com>
References: <BANLkTi=yWzE3MJqq3CUwe8TCcPPkpXSuiA@mail.gmail.com>
	<BANLkTik59bpyST=7RC6EJiLkrOPQHSujvg@mail.gmail.com>
	<BANLkTik99WbBap5BgKyme2UBjtoiVrEATQ@mail.gmail.com>
	<BANLkTikYr4qfkW-qA0-LcynQhdu9Zg6DcA@mail.gmail.com>
	<BANLkTi=ps-zjHZxaodoGQ4djJQJRCU7zfw@mail.gmail.com>
	<BANLkTindDiWJEjZGbELa-dbz591EhzLPug@mail.gmail.com>
	<BANLkTinAwrra+ru8PC4x_68Eou1iH9QACw@mail.gmail.com>
	<BANLkTi=KnePkE7Lf7jWK4tDUzX1A9ufBKQ@mail.gmail.com>
	<BANLkTi=8szSqJTrwwvOZwx1x=BXFcs6hdQ@mail.gmail.com>
	<BANLkTimELr=UmYLpt=wLaxiarFVP6j_ecg@mail.gmail.com>
	<BANLkTinzJTPx58fs+C-R0Y26O=iFpJC-YA@mail.gmail.com>
	<BANLkTi=Kn87FiQHY1pJ=rjTFR6HiqHPPCA@mail.gmail.com>
	<BANLkTi=qUHAMp6ENB+1GhkSyhSypkNazJQ@mail.gmail.com>
	<BANLkTinxxB4rAdfVK7RwynB+bcoTy=YvzA@mail.gmail.com>
	<BANLkTim+TibgtVgrT-1TG+OduGegAXtj-A@mail.gmail.com>
	<BANLkTin2A7iP3ocmwqfqv1JS0gotDZztGQ@mail.gmail.com>
	<BANLkTin36dOg--NE2zVRpjsDi0aaCbfa8g@mail.gmail.com>
	<BANLkTinM9bwbZbz=_yYok7UXUqrbZwAsBw@mail.gmail.com>
Message-ID: <BANLkTi=k5MqQKKEt6pr+wK0fv7tRyJGJSw@mail.gmail.com>

2011/6/4 mark florisson <markflorisson88 at gmail.com>:
> On 4 June 2011 12:24, Vitja Makarov <vitja.makarov at gmail.com> wrote:
>> I've tried that: https://github.com/vitek/cython/compare/master..._bindings
>>
>> Results are not bad: 168 failing tests for pyregr2.7 and 463 for py3
>
> Nice, it partly duplicates work in my fusedtypes branch but I suppose
> it will have to be reworked anyway into a subclass.

Sure. I've taken some code from your branch ;)

> So this works only
> for normal classes right, i.e. not for extension classes? It's also
> pickle-able, cool :)
>

Yes. It works for python classes and python def functions.
__reduce__ only returns function name, so it's possible to
pickle/unpickle it in some cases
but that was enough to fix some broken tests from pyregr.

-- 
vitja.

From sshannin at stwing.upenn.edu  Sat Jun  4 19:36:16 2011
From: sshannin at stwing.upenn.edu (Seth Shannin)
Date: Sat, 04 Jun 2011 13:36:16 -0400
Subject: [Cython] Header file bug
Message-ID: <4DEA6D10.6090704@stwing.upenn.edu>

Hello all,

First, I tried to post this earlier but am pretty sure it didn't go 
through.  If this is a double, I apologize.

This comes out of a discussion on the cython-users list.  I am getting 
some warnings when using cython to compile a python module and then 
calling into the module from c.

I was able to condense the problem down to a very succinct example.

There are three files of interest:

test.pyx:
----------------------------------
cdef class foo:
     cdef int a
     cdef int b
     def __init__(foo self, int a, int b):
         self.a = a
         self.b = b

cdef public void bar(foo x):
     print "My_foo", x.a, x.b

cdef public foo make_foo(int a, int b):
     return foo(a, b)
------------------------------------------

setup.py:
------------------------------------------
from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext

ext_modules = [Extension("test", ["test.pyx"])]

setup(
   name = 'Hello world app',
   cmdclass = {'build_ext': build_ext},
   ext_modules = ext_modules
)
------------------------------------------
and dummy.c:
-----------------------------------------
#include "Python.h"
#include "test.h"

int main(int argc, char** argv){
   Py_Initialize();
   inittest();
   bar(make_foo(1,2));
   return 0;
}
---------------------------------------

I use the following commands to compile and link:
python setup.py build_ext --inplace
gcc -I/usr/include/python2.6 -lpython2.6 -L/usr/include/python2.6/config 
-c dummy.c

The second command generates the following error messages:
In file included from dummy.c:2:
test.h:11: warning: 'struct __pyx_obj_4test_foo' declared inside 
parameter list
test.h:11: warning: its scope is only this definition or declaration, 
which is probably not what you want
dummy.c: In function 'main':
dummy.c:7: warning: passing argument 1 of 'bar' from incompatible 
pointer type
test.h:11: note: expected 'struct __pyx_obj_4test_foo *' but argument is 
of type 'struct __pyx_obj_4test_foo *'

After a bit of digging, I found the problem:
The definition for the struct in question was created in test.c, not 
test.h.
GCC is therefore treating the parameter in test.h as a completely 
separate (and empty) struct declaration.  Luckily, in this case struct 
pointers are being used, so enough mem gets allocated, etc. and 
everything actually ends up working.

That being said, the compiler errors are quite bothersome, and it may be 
that this could cause real errors in more complex programs with more 
interdependencies of types.

I would like to propose as a fix that the struct declaration be 
generated inside the header file instead of the .c file (and on a side 
note, I would personally prefer that the #includes be placed in the 
header file as well).

Thanks for all the work that has been put into this project.  Cython is 
really quite a remarkable tool!

-Seth




From romain.py at gmail.com  Tue Jun  7 04:40:11 2011
From: romain.py at gmail.com (Romain Guillebert)
Date: Tue, 7 Jun 2011 04:40:11 +0200
Subject: [Cython] [GSoC] CTypes backend for Cython aiming PyPy - Week 2
Message-ID: <20110607024011.GA4162@ubuntu>

Hi

I summarized the second week of my Summer of Code project in this blog
post:
http://rguillebert.blogspot.com/2011/06/cython-backend-aiming-pypy-week-2.html

Cheers
Romain

From robertwb at math.washington.edu  Tue Jun  7 07:42:25 2011
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 6 Jun 2011 22:42:25 -0700
Subject: [Cython] Header file bug
In-Reply-To: <4DEA6D10.6090704@stwing.upenn.edu>
References: <4DEA6D10.6090704@stwing.upenn.edu>
Message-ID: <BANLkTimA3=i8mePuPLDxSc0scKe-eQt2bQ@mail.gmail.com>

On Sat, Jun 4, 2011 at 10:36 AM, Seth Shannin <sshannin at stwing.upenn.edu> wrote:
> Hello all,
>
> First, I tried to post this earlier but am pretty sure it didn't go through.
> ?If this is a double, I apologize.
>
> This comes out of a discussion on the cython-users list. ?I am getting some
> warnings when using cython to compile a python module and then calling into
> the module from c.
>
> I was able to condense the problem down to a very succinct example.
>
> There are three files of interest:
>
> test.pyx:
> ----------------------------------
> cdef class foo:
> ? ?cdef int a
> ? ?cdef int b
> ? ?def __init__(foo self, int a, int b):
> ? ? ? ?self.a = a
> ? ? ? ?self.b = b
>
> cdef public void bar(foo x):
> ? ?print "My_foo", x.a, x.b
>
> cdef public foo make_foo(int a, int b):
> ? ?return foo(a, b)
> ------------------------------------------
>
> setup.py:
> ------------------------------------------
> from distutils.core import setup
> from distutils.extension import Extension
> from Cython.Distutils import build_ext
>
> ext_modules = [Extension("test", ["test.pyx"])]
>
> setup(
> ?name = 'Hello world app',
> ?cmdclass = {'build_ext': build_ext},
> ?ext_modules = ext_modules
> )
> ------------------------------------------
> and dummy.c:
> -----------------------------------------
> #include "Python.h"
> #include "test.h"
>
> int main(int argc, char** argv){
> ?Py_Initialize();
> ?inittest();
> ?bar(make_foo(1,2));
> ?return 0;
> }
> ---------------------------------------
>
> I use the following commands to compile and link:
> python setup.py build_ext --inplace
> gcc -I/usr/include/python2.6 -lpython2.6 -L/usr/include/python2.6/config -c
> dummy.c
>
> The second command generates the following error messages:
> In file included from dummy.c:2:
> test.h:11: warning: 'struct __pyx_obj_4test_foo' declared inside parameter
> list
> test.h:11: warning: its scope is only this definition or declaration, which
> is probably not what you want
> dummy.c: In function 'main':
> dummy.c:7: warning: passing argument 1 of 'bar' from incompatible pointer
> type
> test.h:11: note: expected 'struct __pyx_obj_4test_foo *' but argument is of
> type 'struct __pyx_obj_4test_foo *'
>
> After a bit of digging, I found the problem:
> The definition for the struct in question was created in test.c, not test.h.
> GCC is therefore treating the parameter in test.h as a completely separate
> (and empty) struct declaration. ?Luckily, in this case struct pointers are
> being used, so enough mem gets allocated, etc. and everything actually ends
> up working.
>
> That being said, the compiler errors are quite bothersome, and it may be
> that this could cause real errors in more complex programs with more
> interdependencies of types.

True. Cython doesn't usually get used this way, so thanks for the
report. BTW, for embedding the --embed option is usually easier to
use.

> I would like to propose as a fix that the struct declaration be generated
> inside the header file instead of the .c file (and on a side note, I would
> personally prefer that the #includes be placed in the header file as well).
>
> Thanks for all the work that has been put into this project. ?Cython is
> really quite a remarkable tool!
>
> -Seth
>
>
>
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>

From yury at shurup.com  Tue Jun  7 09:20:49 2011
From: yury at shurup.com (Yury V. Zaytsev)
Date: Tue, 07 Jun 2011 09:20:49 +0200
Subject: [Cython] [GSoC] CTypes backend for Cython aiming PyPy - Week 2
In-Reply-To: <20110607024011.GA4162@ubuntu>
References: <20110607024011.GA4162@ubuntu>
Message-ID: <1307431249.2648.6.camel@newpride>

Hi Romain!

On Tue, 2011-06-07 at 04:40 +0200, Romain Guillebert wrote: 
> 
> I summarized the second week of my Summer of Code project in this blog
> post:
> http://rguillebert.blogspot.com/2011/06/cython-backend-aiming-pypy-week-2.html

Great job, keep on rocking! This is just to let you know how I consider
your efforts important. 

We have Python bindings for our project done in plain Python API, I was
for long toying the idea of re-implementing them in Cython which will
for sure make it so much more efficient and easier to maintain, but the
major objection was that it's a dead-end since it's not gonna work with
PyPy, and a ctypes backend would be more preferable. Now there is a
possibility that it's no longer going to be the case...

Wow!

-- 
Sincerely yours,
Yury V. Zaytsev



From vitja.makarov at gmail.com  Tue Jun  7 20:37:04 2011
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Tue, 7 Jun 2011 22:37:04 +0400
Subject: [Cython] local variable handling in generators
In-Reply-To: <4DE4CE91.3000009@astro.uio.no>
References: <4DD90292.8070505@behnel.de>
	<BANLkTi=ABOAjFcJ6dskwPYi7Xpww975XGQ@mail.gmail.com>
	<4DDA1AD6.1070309@behnel.de>
	<BANLkTimmoRh7vBW8cbaa82W8RpiS55WVPg@mail.gmail.com>
	<4DDA259B.1000203@behnel.de> <4DDA2909.20604@behnel.de>
	<4DDA2F80.3060907@behnel.de>
	<BANLkTi=0ROZVsKy8nkgUnMEm=_xAZtvehA@mail.gmail.com>
	<4DDAA9DF.2000809@behnel.de>
	<BANLkTim1ExZ55HNjVO=E1ksVnYSKWAJcYw@mail.gmail.com>
	<4DDB3520.5020609@behnel.de>
	<BANLkTimT1ojJS60SjZ8G-9uSMU5HbxzY5A@mail.gmail.com>
	<4DE4CE91.3000009@astro.uio.no>
Message-ID: <BANLkTimVvKR11YCgSLngLVx292mMD9D1qw@mail.gmail.com>

2011/5/31 Dag Sverre Seljebotn <d.s.seljebotn at astro.uio.no>:
> On 05/31/2011 01:07 PM, Vitja Makarov wrote:
>>
>> 2011/5/24 Stefan Behnel<stefan_ml at behnel.de>:
>>>
>>> Vitja Makarov, 23.05.2011 21:33:
>>>>
>>>> 2011/5/23 Stefan Behnel:
>>>>>
>>>>> However, once we really know which values change between yield calls,
>>>>> i.e.
>>>>> which ones need to be stored away, it will actually be less expensive
>>>>> in
>>>>> most cases. We currently pay the indirection penalty for each access,
>>>>> even
>>>>> read access, whereas the C compiler can then keep important variables
>>>>> in
>>>>> registers and only write them back once per yield.
>>>>>
>>>>
>>>> I think that all not NULL variables should be saved/restored inside
>>>> yield.
>>>> I can not really track changes only assignments.
>>>
>>> I mean "assignments" when I write "changes". When a variable is being
>>> assigned to, we should assume that its value changed. Users can be
>>> expected
>>> to be smart enough to avoid unnecessary assignments in most cases,
>>> there's
>>> no need to put work into optimising them away automatically.
>>>
>>>
>>>> for i in a:
>>>> ? ? yield i ?# j is NULL here
>>>> for j in b:
>>>> ? ? yield j # a, b, i ,j should be saved/restored
>>>
>>> Right. However, in general, we can expect that most variables will have
>>> been
>>> initialised on a yield.
>>>
>>> We can still avoid storing away C typed variables that are not being used
>>> later on, because they are not reference counted.
>>>
>>> And if the user sets 'a' and 'i' to None between the loops, they won't
>>> need
>>> to be saved on the second yield either. But that's an optimisation, not a
>>> requirement.
>>>
>>
>> What about copying C++ structs and objects with constructor (and/or
>> overloaded = method)?
>> That wouldn't work well.
>
> Currently I believe C++ objects are only supported through pointers, which
> of course are copyable. If or when we support C++ objects on the stack, then
> we should definitely avoid copying them.
>
> If you declare a C++ class as a struct Cython-side then I think we should
> assume that we can freely invoke the copy constructor and assignment
> operator in this context.
>

May be it's better to declare struct with all the locals and then copy
it with memcpy()?

Btw cpp still wouldn't love it :(
So probably it's better to look forward for stack switching library...

-- 
vitja.

From greg.ewing at canterbury.ac.nz  Wed Jun  8 00:24:33 2011
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Wed, 08 Jun 2011 10:24:33 +1200
Subject: [Cython] Header file bug
In-Reply-To: <BANLkTimA3=i8mePuPLDxSc0scKe-eQt2bQ@mail.gmail.com>
References: <4DEA6D10.6090704@stwing.upenn.edu>
	<BANLkTimA3=i8mePuPLDxSc0scKe-eQt2bQ@mail.gmail.com>
Message-ID: <4DEEA521.9050901@canterbury.ac.nz>

> On Sat, Jun 4, 2011 at 10:36 AM, Seth Shannin <sshannin at stwing.upenn.edu> wrote:
> 
>>test.h:11: warning: 'struct __pyx_obj_4test_foo' declared inside parameter
>>list
>>test.h:11: warning: its scope is only this definition or declaration, which
>>is probably not what you want

Not sure about Cython, but the way to to this in Pyrex
is to declare the class 'public' as well, and then the
struct declaration for it will be put in the .h file.

You will need to supply C names for the struct and type
object as well, e.g.

    cdef public class foo[type FooType, object FooObject]:
       ...

The generated test.h file then contains

    struct FooObject {
      PyObject_HEAD
      int a;
      int b;
    };

    __PYX_EXTERN_C DL_IMPORT(void) bar(struct FooObject *);
    __PYX_EXTERN_C DL_IMPORT(struct FooObject) *make_foo(int,int);

-- 
Greg

From sshannin at stwing.upenn.edu  Wed Jun  8 03:51:46 2011
From: sshannin at stwing.upenn.edu (Seth Shannin)
Date: Tue, 07 Jun 2011 21:51:46 -0400
Subject: [Cython] Header file bug
In-Reply-To: <4DEEA521.9050901@canterbury.ac.nz>
References: <4DEA6D10.6090704@stwing.upenn.edu>	<BANLkTimA3=i8mePuPLDxSc0scKe-eQt2bQ@mail.gmail.com>
	<4DEEA521.9050901@canterbury.ac.nz>
Message-ID: <4DEED5B2.4050905@stwing.upenn.edu>

Ah, that's what it means.  I saw something similar to this somewhere in
some documentation, but the object keyword threw me off.  To me,
'object' indicates something more on the python side than on the c side.

Also, even though you can do this to ensure the struct declaration ends
up in the header file, I would still make the case that currently
something is a bit off in that the header file tries to use something
which it has no information about (and in the process actually ends up
creating its own empty struct type).

Thanks for all the help.

-Seth

On 06/07/2011 06:24 PM, Greg Ewing wrote:
>> On Sat, Jun 4, 2011 at 10:36 AM, Seth Shannin
>> <sshannin at stwing.upenn.edu> wrote:
>>
>>> test.h:11: warning: 'struct __pyx_obj_4test_foo' declared inside
>>> parameter
>>> list
>>> test.h:11: warning: its scope is only this definition or declaration,
>>> which
>>> is probably not what you want
> 
> Not sure about Cython, but the way to to this in Pyrex
> is to declare the class 'public' as well, and then the
> struct declaration for it will be put in the .h file.
> 
> You will need to supply C names for the struct and type
> object as well, e.g.
> 
>    cdef public class foo[type FooType, object FooObject]:
>       ...
> 
> The generated test.h file then contains
> 
>    struct FooObject {
>      PyObject_HEAD
>      int a;
>      int b;
>    };
> 
>    __PYX_EXTERN_C DL_IMPORT(void) bar(struct FooObject *);
>    __PYX_EXTERN_C DL_IMPORT(struct FooObject) *make_foo(int,int);
> 


From greg.ewing at canterbury.ac.nz  Wed Jun  8 14:05:16 2011
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Thu, 09 Jun 2011 00:05:16 +1200
Subject: [Cython] Header file bug
In-Reply-To: <4DEED5B2.4050905@stwing.upenn.edu>
References: <4DEA6D10.6090704@stwing.upenn.edu>
	<BANLkTimA3=i8mePuPLDxSc0scKe-eQt2bQ@mail.gmail.com>
	<4DEEA521.9050901@canterbury.ac.nz> <4DEED5B2.4050905@stwing.upenn.edu>
Message-ID: <4DEF657C.1080805@canterbury.ac.nz>

Seth Shannin wrote:
> I would still make the case that currently
> something is a bit off in that the header file tries to use something
> which it has no information about

Yes, I agree that it could do with improvement. I'm not sure
that automatically putting the struct declaration in the
header is the best idea, though, since with mangled names it's
hard to do anything with it from external C code (that's why
you're required to manually supply type and object struct names).

Maybe it should just be an error to make a declaration public
if it uses something that's non-public?

-- 
Greg

From sshannin at stwing.upenn.edu  Wed Jun  8 18:47:13 2011
From: sshannin at stwing.upenn.edu (sshannin at stwing.upenn.edu)
Date: Wed, 08 Jun 2011 12:47:13 -0400
Subject: [Cython] Header file bug
In-Reply-To: <4DEF657C.1080805@canterbury.ac.nz>
References: <4DEA6D10.6090704@stwing.upenn.edu>
	<BANLkTimA3=i8mePuPLDxSc0scKe-eQt2bQ@mail.gmail.com>
	<4DEEA521.9050901@canterbury.ac.nz> <4DEED5B2.4050905@stwing.upenn.edu>
	<4DEF657C.1080805@canterbury.ac.nz>
Message-ID: <20110608124713.18595m8n81zg6zyp@webmail.stwing.upenn.edu>

Yeah, I think just erroring would be perfectly reasonable.

-Seth

Quoting "Greg Ewing" <greg.ewing at canterbury.ac.nz>:

> Seth Shannin wrote:
>> I would still make the case that currently
>> something is a bit off in that the header file tries to use something
>> which it has no information about
>
> Yes, I agree that it could do with improvement. I'm not sure
> that automatically putting the struct declaration in the
> header is the best idea, though, since with mangled names it's
> hard to do anything with it from external C code (that's why
> you're required to manually supply type and object struct names).
>
> Maybe it should just be an error to make a declaration public
> if it uses something that's non-public?
>
> -- 
> Greg
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>



From robertwb at math.washington.edu  Thu Jun  9 07:57:37 2011
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 8 Jun 2011 22:57:37 -0700
Subject: [Cython] Header file bug
In-Reply-To: <20110608124713.18595m8n81zg6zyp@webmail.stwing.upenn.edu>
References: <4DEA6D10.6090704@stwing.upenn.edu>
	<BANLkTimA3=i8mePuPLDxSc0scKe-eQt2bQ@mail.gmail.com>
	<4DEEA521.9050901@canterbury.ac.nz> <4DEED5B2.4050905@stwing.upenn.edu>
	<4DEF657C.1080805@canterbury.ac.nz>
	<20110608124713.18595m8n81zg6zyp@webmail.stwing.upenn.edu>
Message-ID: <BANLkTikKtcnzRAZUWL+ET2iRWoSeZpmmCg@mail.gmail.com>

I agree.

On Wed, Jun 8, 2011 at 9:47 AM,  <sshannin at stwing.upenn.edu> wrote:
> Yeah, I think just erroring would be perfectly reasonable.
>
> -Seth
>
> Quoting "Greg Ewing" <greg.ewing at canterbury.ac.nz>:
>
>> Seth Shannin wrote:
>>>
>>> I would still make the case that currently
>>> something is a bit off in that the header file tries to use something
>>> which it has no information about
>>
>> Yes, I agree that it could do with improvement. I'm not sure
>> that automatically putting the struct declaration in the
>> header is the best idea, though, since with mangled names it's
>> hard to do anything with it from external C code (that's why
>> you're required to manually supply type and object struct names).
>>
>> Maybe it should just be an error to make a declaration public
>> if it uses something that's non-public?
>>
>> --
>> Greg
>> _______________________________________________
>> cython-devel mailing list
>> cython-devel at python.org
>> http://mail.python.org/mailman/listinfo/cython-devel
>>
>
>
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>

From sshannin at stwing.upenn.edu  Thu Jun  9 13:49:49 2011
From: sshannin at stwing.upenn.edu (Seth Shannin)
Date: Thu, 09 Jun 2011 07:49:49 -0400
Subject: [Cython] Header file bug
In-Reply-To: <BANLkTikKtcnzRAZUWL+ET2iRWoSeZpmmCg@mail.gmail.com>
References: <4DEA6D10.6090704@stwing.upenn.edu>	<BANLkTimA3=i8mePuPLDxSc0scKe-eQt2bQ@mail.gmail.com>	<4DEEA521.9050901@canterbury.ac.nz>
	<4DEED5B2.4050905@stwing.upenn.edu>	<4DEF657C.1080805@canterbury.ac.nz>	<20110608124713.18595m8n81zg6zyp@webmail.stwing.upenn.edu>
	<BANLkTikKtcnzRAZUWL+ET2iRWoSeZpmmCg@mail.gmail.com>
Message-ID: <4DF0B35D.40507@stwing.upenn.edu>

One note: I personally found the documentation on how to make a class
declaration public fairly hard to find.  It could be nice to make this
clearer.

-Seth

On 06/09/2011 01:57 AM, Robert Bradshaw wrote:
> I agree.
> 
> On Wed, Jun 8, 2011 at 9:47 AM,  <sshannin at stwing.upenn.edu> wrote:
>> Yeah, I think just erroring would be perfectly reasonable.
>>
>> -Seth
>>
>> Quoting "Greg Ewing" <greg.ewing at canterbury.ac.nz>:
>>
>>> Seth Shannin wrote:
>>>>
>>>> I would still make the case that currently
>>>> something is a bit off in that the header file tries to use something
>>>> which it has no information about
>>>
>>> Yes, I agree that it could do with improvement. I'm not sure
>>> that automatically putting the struct declaration in the
>>> header is the best idea, though, since with mangled names it's
>>> hard to do anything with it from external C code (that's why
>>> you're required to manually supply type and object struct names).
>>>
>>> Maybe it should just be an error to make a declaration public
>>> if it uses something that's non-public?
>>>
>>> --
>>> Greg
>>> _______________________________________________
>>> cython-devel mailing list
>>> cython-devel at python.org
>>> http://mail.python.org/mailman/listinfo/cython-devel
>>>
>>
>>
>> _______________________________________________
>> cython-devel mailing list
>> cython-devel at python.org
>> http://mail.python.org/mailman/listinfo/cython-devel
>>
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel


From robertwb at math.washington.edu  Thu Jun  9 17:35:35 2011
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 9 Jun 2011 08:35:35 -0700
Subject: [Cython] Header file bug
In-Reply-To: <4DF0B35D.40507@stwing.upenn.edu>
References: <4DEA6D10.6090704@stwing.upenn.edu>
	<BANLkTimA3=i8mePuPLDxSc0scKe-eQt2bQ@mail.gmail.com>
	<4DEEA521.9050901@canterbury.ac.nz> <4DEED5B2.4050905@stwing.upenn.edu>
	<4DEF657C.1080805@canterbury.ac.nz>
	<20110608124713.18595m8n81zg6zyp@webmail.stwing.upenn.edu>
	<BANLkTikKtcnzRAZUWL+ET2iRWoSeZpmmCg@mail.gmail.com>
	<4DF0B35D.40507@stwing.upenn.edu>
Message-ID: <BANLkTi=pUv-Hri09kWZ96CGbkpuGLnhwTQ@mail.gmail.com>

I agree. We welcome improvement to our documentation--fork and make a
push request: https://github.com/cython/cython/tree/master/docs

On Thu, Jun 9, 2011 at 4:49 AM, Seth Shannin <sshannin at stwing.upenn.edu> wrote:
> One note: I personally found the documentation on how to make a class
> declaration public fairly hard to find. ?It could be nice to make this
> clearer.
>
> -Seth
>
> On 06/09/2011 01:57 AM, Robert Bradshaw wrote:
>> I agree.
>>
>> On Wed, Jun 8, 2011 at 9:47 AM, ?<sshannin at stwing.upenn.edu> wrote:
>>> Yeah, I think just erroring would be perfectly reasonable.
>>>
>>> -Seth
>>>
>>> Quoting "Greg Ewing" <greg.ewing at canterbury.ac.nz>:
>>>
>>>> Seth Shannin wrote:
>>>>>
>>>>> I would still make the case that currently
>>>>> something is a bit off in that the header file tries to use something
>>>>> which it has no information about
>>>>
>>>> Yes, I agree that it could do with improvement. I'm not sure
>>>> that automatically putting the struct declaration in the
>>>> header is the best idea, though, since with mangled names it's
>>>> hard to do anything with it from external C code (that's why
>>>> you're required to manually supply type and object struct names).
>>>>
>>>> Maybe it should just be an error to make a declaration public
>>>> if it uses something that's non-public?
>>>>
>>>> --
>>>> Greg
>>>> _______________________________________________
>>>> cython-devel mailing list
>>>> cython-devel at python.org
>>>> http://mail.python.org/mailman/listinfo/cython-devel
>>>>
>>>
>>>
>>> _______________________________________________
>>> cython-devel mailing list
>>> cython-devel at python.org
>>> http://mail.python.org/mailman/listinfo/cython-devel
>>>
>> _______________________________________________
>> cython-devel mailing list
>> cython-devel at python.org
>> http://mail.python.org/mailman/listinfo/cython-devel
>
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>

From vitja.makarov at gmail.com  Fri Jun 10 17:22:06 2011
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Fri, 10 Jun 2011 19:22:06 +0400
Subject: [Cython] Constant folding
Message-ID: <BANLkTi=YTU-h70abrw2N901FqTO=+NCvPg@mail.gmail.com>

Hi!

When CF is there it is possible to improve constant folding.
Some NameNode references could be easily replaced by appropriate ConstNode.

Could that help GCC to produce faster code?

-- 
vitja.

From robertwb at math.washington.edu  Sat Jun 11 02:21:42 2011
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 10 Jun 2011 17:21:42 -0700
Subject: [Cython] Constant folding
In-Reply-To: <BANLkTi=YTU-h70abrw2N901FqTO=+NCvPg@mail.gmail.com>
References: <BANLkTi=YTU-h70abrw2N901FqTO=+NCvPg@mail.gmail.com>
Message-ID: <BANLkTinW1ie54HYsR0v=Gb+oiK87WUmgkUZW53nkQ9mTOjrWVw@mail.gmail.com>

On Fri, Jun 10, 2011 at 8:22 AM, Vitja Makarov <vitja.makarov at gmail.com> wrote:
> Hi!
>
> When CF is there it is possible to improve constant folding.

Do you have some examples? I can see that

    a = 1.0
    b = sin(a)

could now be optimized, but I don't see that being a common pattern.
(Also, if a is typed, gcc is likely to already be able to do this.)

> Some NameNode references could be easily replaced by appropriate ConstNode.
>
> Could that help GCC to produce faster code?

Possibly a first (and more fruitful) step to investigate would be more
local type inference (e.g a variable might be a double within a block,
and only need to be used as/store an object elsewhere.

- Robert

From stefan_ml at behnel.de  Sun Jun 12 19:04:16 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 12 Jun 2011 19:04:16 +0200
Subject: [Cython] local variable handling in generators
In-Reply-To: <BANLkTimT1ojJS60SjZ8G-9uSMU5HbxzY5A@mail.gmail.com>
References: <4DD90292.8070505@behnel.de>	<BANLkTi=ABOAjFcJ6dskwPYi7Xpww975XGQ@mail.gmail.com>	<4DDA1AD6.1070309@behnel.de>	<BANLkTimmoRh7vBW8cbaa82W8RpiS55WVPg@mail.gmail.com>	<4DDA259B.1000203@behnel.de>
	<4DDA2909.20604@behnel.de>	<4DDA2F80.3060907@behnel.de>	<BANLkTi=0ROZVsKy8nkgUnMEm=_xAZtvehA@mail.gmail.com>	<4DDAA9DF.2000809@behnel.de>	<BANLkTim1ExZ55HNjVO=E1ksVnYSKWAJcYw@mail.gmail.com>	<4DDB3520.5020609@behnel.de>
	<BANLkTimT1ojJS60SjZ8G-9uSMU5HbxzY5A@mail.gmail.com>
Message-ID: <4DF4F190.7010508@behnel.de>

Vitja Makarov, 31.05.2011 13:07:
> 2011/5/24 Stefan Behnel:
>> Vitja Makarov, 23.05.2011 21:33:
>>>
>>> 2011/5/23 Stefan Behnel:
>>>>
>>>> However, once we really know which values change between yield calls,
>>>> i.e.
>>>> which ones need to be stored away, it will actually be less expensive in
>>>> most cases. We currently pay the indirection penalty for each access,
>>>> even
>>>> read access, whereas the C compiler can then keep important variables in
>>>> registers and only write them back once per yield.
>>>>
>>>
>>> I think that all not NULL variables should be saved/restored inside yield.
>>> I can not really track changes only assignments.
>>
>> I mean "assignments" when I write "changes". When a variable is being
>> assigned to, we should assume that its value changed. Users can be expected
>> to be smart enough to avoid unnecessary assignments in most cases, there's
>> no need to put work into optimising them away automatically.
>>
>>
>>> for i in a:
>>>      yield i  # j is NULL here
>>> for j in b:
>>>      yield j # a, b, i ,j should be saved/restored
>>
>> Right. However, in general, we can expect that most variables will have been
>> initialised on a yield.
>>
>> We can still avoid storing away C typed variables that are not being used
>> later on, because they are not reference counted.
>>
>> And if the user sets 'a' and 'i' to None between the loops, they won't need
>> to be saved on the second yield either. But that's an optimisation, not a
>> requirement.
>
> What about copying C++ structs and objects with constructor (and/or
> overloaded = method)?
> That wouldn't work well.

I'd just make that an "unsupported feature" error for now. Remember, we are 
talking about C++ types used inside of generators here, which can only 
appear in newly written code. It's best to prevent users from doing that, 
at least until someone comes up with an actual requirement.

Stefan

From dalcinl at gmail.com  Tue Jun 14 21:39:05 2011
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Tue, 14 Jun 2011 16:39:05 -0300
Subject: [Cython] setup.py refusing to run cython
In-Reply-To: <BANLkTimxe1pq8ro6+=ABgwo7MriLiwNGiQ@mail.gmail.com>
References: <BANLkTimxe1pq8ro6+=ABgwo7MriLiwNGiQ@mail.gmail.com>
Message-ID: <BANLkTim3pn9biUZGe_Z8gnnY235poEDGUA@mail.gmail.com>

On 14 June 2011 16:20, Nathaniel Smith <njs at pobox.com> wrote:
> Hello Cython folks,
>
> This message (see below) is the second report I've gotten of a very
> strange build problem with a cython module. I'm just using the
> standard 'from Cython.Distutils import build_ext', 'cmdclass =
> {"build_ext": build_ext}' incantation, but for some reason setup.py is
> attempting to just run gcc directly without calling cython first to
> generate the .c file.
>
> You can see the setup.py here:
> ?https://code.google.com/p/scikits-sparse/source/browse/setup.py?name=scikits.sparse-0.1
>
> Anyone have any ideas?
>

Your setup.py is mixing setuptools and Cython.Distutils. Both are
monkeypatching distutils, and in general that's a very bad idea.


-- 
Lisandro Dalcin
---------------
CIMEC (INTEC/CONICET-UNL)
Predio CONICET-Santa Fe
Colectora RN 168 Km 472, Paraje El Pozo
3000 Santa Fe, Argentina
Tel: +54-342-4511594 (ext 1011)
Tel/Fax: +54-342-4511169

From robertwb at math.washington.edu  Tue Jun 14 21:54:10 2011
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 14 Jun 2011 12:54:10 -0700
Subject: [Cython] setup.py refusing to run cython
In-Reply-To: <BANLkTim3pn9biUZGe_Z8gnnY235poEDGUA@mail.gmail.com>
References: <BANLkTimxe1pq8ro6+=ABgwo7MriLiwNGiQ@mail.gmail.com>
	<BANLkTim3pn9biUZGe_Z8gnnY235poEDGUA@mail.gmail.com>
Message-ID: <BANLkTinkBqwWv89s23rCJZCtTgRxq7AhKLPoqioDfwHZ8epQVQ@mail.gmail.com>

On Tue, Jun 14, 2011 at 12:39 PM, Lisandro Dalcin <dalcinl at gmail.com> wrote:
> On 14 June 2011 16:20, Nathaniel Smith <njs at pobox.com> wrote:
>> Hello Cython folks,
>>
>> This message (see below) is the second report I've gotten of a very
>> strange build problem with a cython module. I'm just using the
>> standard 'from Cython.Distutils import build_ext', 'cmdclass =
>> {"build_ext": build_ext}' incantation, but for some reason setup.py is
>> attempting to just run gcc directly without calling cython first to
>> generate the .c file.
>>
>> You can see the setup.py here:
>> ?https://code.google.com/p/scikits-sparse/source/browse/setup.py?name=scikits.sparse-0.1
>>
>> Anyone have any ideas?
>>
>
> Your setup.py is mixing setuptools and Cython.Distutils. Both are
> monkeypatching distutils, and in general that's a very bad idea.

One way to get around this is to use
http://wiki.cython.org/enhancements/distutils_preprocessing

- Robert

From stefan_ml at behnel.de  Wed Jun 15 09:18:45 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 15 Jun 2011 09:18:45 +0200
Subject: [Cython] setup.py refusing to run cython
In-Reply-To: <BANLkTim3pn9biUZGe_Z8gnnY235poEDGUA@mail.gmail.com>
References: <BANLkTimxe1pq8ro6+=ABgwo7MriLiwNGiQ@mail.gmail.com>
	<BANLkTim3pn9biUZGe_Z8gnnY235poEDGUA@mail.gmail.com>
Message-ID: <4DF85CD5.2090809@behnel.de>

Lisandro Dalcin, 14.06.2011 21:39:
> On 14 June 2011 16:20, Nathaniel Smith wrote:
>> Hello Cython folks,
>>
>> This message (see below) is the second report I've gotten of a very
>> strange build problem with a cython module. I'm just using the
>> standard 'from Cython.Distutils import build_ext', 'cmdclass =
>> {"build_ext": build_ext}' incantation, but for some reason setup.py is
>> attempting to just run gcc directly without calling cython first to
>> generate the .c file.
>>
>> You can see the setup.py here:
>>   https://code.google.com/p/scikits-sparse/source/browse/setup.py?name=scikits.sparse-0.1
>>
>> Anyone have any ideas?
>
> Your setup.py is mixing setuptools and Cython.Distutils. Both are
> monkeypatching distutils, and in general that's a very bad idea.

Specifically, "setuptools" checks the list of source files and when it sees 
a .pyx extension, it checks if Pyrex (!) is installed. If not, which is 
rather unlikely for Cython users, it changes the file extension to ".c", 
thus preventing Cython from compiling it.

There are two ways around this. The ugly way is to put a fake Pyrex into 
sys.path during installation, e.g.

https://github.com/lxml/lxml/tree/master/fake_pyrex

A better way is to switch from "setuptools" to "distribute".

Stefan

From njs at pobox.com  Wed Jun 15 17:59:17 2011
From: njs at pobox.com (Nathaniel Smith)
Date: Wed, 15 Jun 2011 08:59:17 -0700
Subject: [Cython] setup.py refusing to run cython
In-Reply-To: <4DF85CD5.2090809@behnel.de>
References: <BANLkTimxe1pq8ro6+=ABgwo7MriLiwNGiQ@mail.gmail.com>
	<BANLkTim3pn9biUZGe_Z8gnnY235poEDGUA@mail.gmail.com>
	<4DF85CD5.2090809@behnel.de>
Message-ID: <BANLkTi=wmT1jXK3WEjjRb8_7zsQ__cm68Q@mail.gmail.com>

On Wed, Jun 15, 2011 at 12:18 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Lisandro Dalcin, 14.06.2011 21:39:
>>
>> On 14 June 2011 16:20, Nathaniel Smith wrote:
>>>
>>> Hello Cython folks,
>>>
>>> This message (see below) is the second report I've gotten of a very
>>> strange build problem with a cython module. I'm just using the
>>> standard 'from Cython.Distutils import build_ext', 'cmdclass =
>>> {"build_ext": build_ext}' incantation, but for some reason setup.py is
>>> attempting to just run gcc directly without calling cython first to
>>> generate the .c file.
>>>
>>> You can see the setup.py here:
>>>
>>> ?https://code.google.com/p/scikits-sparse/source/browse/setup.py?name=scikits.sparse-0.1
>>>
>>> Anyone have any ideas?
>>
>> Your setup.py is mixing setuptools and Cython.Distutils. Both are
>> monkeypatching distutils, and in general that's a very bad idea.
>
> Specifically, "setuptools" checks the list of source files and when it sees
> a .pyx extension, it checks if Pyrex (!) is installed. If not, which is
> rather unlikely for Cython users, it changes the file extension to ".c",
> thus preventing Cython from compiling it.
>
> There are two ways around this. The ugly way is to put a fake Pyrex into
> sys.path during installation, e.g.
>
> https://github.com/lxml/lxml/tree/master/fake_pyrex

Right, I'm already doing that, and it works for me...but not for all
my users, I guess?

> A better way is to switch from "setuptools" to "distribute".

Right, I'd just switch to plain distutils except this package uses the
'scikits' namespace package, and IIUC distutils can't handle namespace
packages.

I'm finding the docs for 'distribute' a bit confusing, though. Is the
way that I switch from 'setuptools' to 'distribute' to just leave my
code the same, and tell my users to install distribute? That seems
kind of error prone...

-- Nathaniel

From andrew.collette at gmail.com  Thu Jun 16 04:51:44 2011
From: andrew.collette at gmail.com (Andrew Collette)
Date: Wed, 15 Jun 2011 20:51:44 -0600
Subject: [Cython] Possible bug related to multiple assignment
Message-ID: <BANLkTimNKs7s5yfJFynr_xMNj-zJPNTw6A@mail.gmail.com>

Hi,

I ran into some odd behavior when working on my cython-based project
(h5py).  The following cython code snippet is the culprit ("priv" is a
function argument of type void**):

    cdef conv_size_t *sizes
    priv[0] = sizes = <conv_size_t*>malloc(sizeof(conv_size_t))

gets turned into this (with Cython 0.14.1):

    (__pyx_v_priv[0]) = ((__pyx_t_4h5py_5_conv_conv_size_t
*)malloc((sizeof(__pyx_t_4h5py_5_conv_conv_size_t))));
    __pyx_v_sizes = ((__pyx_t_4h5py_5_conv_conv_size_t
*)malloc((sizeof(__pyx_t_4h5py_5_conv_conv_size_t))));

which leads to much head-scratching when one initializes "sizes", and
then later tries to recover that value via "priv".  Interestingly,
Cython 0.13 correctly initializes both priv[0] and sizes to the same
value:

  __pyx_t_5 = ((__pyx_t_4h5py_5_conv_conv_size_t
*)malloc((sizeof(__pyx_t_4h5py_5_conv_conv_size_t))));
  (__pyx_v_priv[0]) = __pyx_t_5;
  __pyx_v_sizes = __pyx_t_5;

I wasn't able to find anything on this in the list archives or issue
tracker.  Right now I'm working around it by splitting the statement
into multiple lines:

   sizes =  <conv_size_t*>malloc(sizeof(conv_size_t))
   priv[0] = sizes

Andrew

From stefan_ml at behnel.de  Thu Jun 16 09:46:07 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 16 Jun 2011 09:46:07 +0200
Subject: [Cython] Possible bug related to multiple assignment
In-Reply-To: <BANLkTimNKs7s5yfJFynr_xMNj-zJPNTw6A@mail.gmail.com>
References: <BANLkTimNKs7s5yfJFynr_xMNj-zJPNTw6A@mail.gmail.com>
Message-ID: <4DF9B4BF.9020604@behnel.de>

Andrew Collette, 16.06.2011 04:51:
> I ran into some odd behavior when working on my cython-based project
> (h5py).  The following cython code snippet is the culprit ("priv" is a
> function argument of type void**):
>
>      cdef conv_size_t *sizes
>      priv[0] = sizes =<conv_size_t*>malloc(sizeof(conv_size_t))
>
> gets turned into this (with Cython 0.14.1):
>
>      (__pyx_v_priv[0]) = ((__pyx_t_4h5py_5_conv_conv_size_t
> *)malloc((sizeof(__pyx_t_4h5py_5_conv_conv_size_t))));
>      __pyx_v_sizes = ((__pyx_t_4h5py_5_conv_conv_size_t
> *)malloc((sizeof(__pyx_t_4h5py_5_conv_conv_size_t))));

This is fixed in the latest master branch.

And, yes, we're long overdue for a release.

Stefan

From stefan_ml at behnel.de  Thu Jun 16 10:35:13 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 16 Jun 2011 10:35:13 +0200
Subject: [Cython] Possible bug related to multiple assignment
In-Reply-To: <4DF9B4BF.9020604@behnel.de>
References: <BANLkTimNKs7s5yfJFynr_xMNj-zJPNTw6A@mail.gmail.com>
	<4DF9B4BF.9020604@behnel.de>
Message-ID: <4DF9C041.8070603@behnel.de>

Stefan Behnel, 16.06.2011 09:46:
> Andrew Collette, 16.06.2011 04:51:
>> I ran into some odd behavior when working on my cython-based project
>> (h5py). The following cython code snippet is the culprit ("priv" is a
>> function argument of type void**):
>>
>> cdef conv_size_t *sizes
>> priv[0] = sizes =<conv_size_t*>malloc(sizeof(conv_size_t))
>>
>> gets turned into this (with Cython 0.14.1):
>>
>> (__pyx_v_priv[0]) = ((__pyx_t_4h5py_5_conv_conv_size_t
>> *)malloc((sizeof(__pyx_t_4h5py_5_conv_conv_size_t))));
>> __pyx_v_sizes = ((__pyx_t_4h5py_5_conv_conv_size_t
>> *)malloc((sizeof(__pyx_t_4h5py_5_conv_conv_size_t))));
>
> This is fixed in the latest master branch.

Ah, sorry. It was actually broken, even though my test run initially told 
me otherwise. Strange ...

The problem was the C type cast, which was incorrectly considered a 
"simple" expression that didn't need a temporary assignment. Malloc is the 
most obvious case where this hits, as it's almost always used through a cast.

It's fixed now. Thanks for report and example.

Stefan

From stefan_ml at behnel.de  Fri Jun 17 08:07:41 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 17 Jun 2011 08:07:41 +0200
Subject: [Cython] Test runner changes for Python/PyPy backend
Message-ID: <4DFAEF2D.3070404@behnel.de>

Hi,

Romain wants to start writing tests for his Python/ctypes backend branch, 
so we need to extend the test runner.

We'll eventually want to merge his branch back into normal mainline, so I 
suggest actually just running the complete test suite in a new, separate 
configuration, just like we do with C and C++ now. So we'd have a switch 
"--no-ctypes" that disables these tests, and (I assume) a new test class 
(inheriting from the normal compile test case) that properly runs them in 
plain CPython. If the implementation becomes PyPy specific, "--no-pypy" 
will make more sense, but it currently appears to target ctypes in general.

I'm also considering making the backend selection a positive opt-in, i.e. 
switch from "--no-cpp" and friends to "--backends=c,cpp,ctypes". That would 
have avoided the future configuration change in all existing Jenkins test 
jobs (which will then need to disable the new backend tests).

Does that fit everyone's expectations?

There's one problem I see: the ctypes backend will likely have most 
optimisations disabled, so many (most?) of the test tree assertions will 
make no sense. I think it's best to just keep that out of the backend 
specific pipeline completely for now. We have the C/C++ test modes for that.

And, obviously, switching from duplicating to triplicating the test cases 
will substantially increase the time for a complete test run, although a 
lot of the running time is currently spent in the C compiler more than the 
actual test code runs.

Stefan

From robertwb at math.washington.edu  Fri Jun 17 17:45:19 2011
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 17 Jun 2011 08:45:19 -0700
Subject: [Cython] Test runner changes for Python/PyPy backend
In-Reply-To: <4DFAEF2D.3070404@behnel.de>
References: <4DFAEF2D.3070404@behnel.de>
Message-ID: <BANLkTikf-EN1wbz+Skp=n1pBvMAg4m0CLg@mail.gmail.com>

On Thu, Jun 16, 2011 at 11:07 PM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Hi,
>
> Romain wants to start writing tests for his Python/ctypes backend branch, so
> we need to extend the test runner.
>
> We'll eventually want to merge his branch back into normal mainline, so I
> suggest actually just running the complete test suite in a new, separate
> configuration, just like we do with C and C++ now.

Makes sense.

> So we'd have a switch
> "--no-ctypes" that disables these tests, and (I assume) a new test class
> (inheriting from the normal compile test case) that properly runs them in
> plain CPython. If the implementation becomes PyPy specific, "--no-pypy" will
> make more sense, but it currently appears to target ctypes in general.
>
> I'm also considering making the backend selection a positive opt-in, i.e.
> switch from "--no-cpp" and friends to "--backends=c,cpp,ctypes". That would
> have avoided the future configuration change in all existing Jenkins test
> jobs (which will then need to disable the new backend tests).
>
> Does that fit everyone's expectations?

+1, though it might makes sense for no specification -> all options.

> There's one problem I see: the ctypes backend will likely have most
> optimisations disabled, so many (most?) of the test tree assertions will
> make no sense. I think it's best to just keep that out of the backend
> specific pipeline completely for now. We have the C/C++ test modes for that.

Probably makes more sense to disable the tree assertions in these
cases, but probably even more important to test them, as the code
paths will be different.

> And, obviously, switching from duplicating to triplicating the test cases
> will substantially increase the time for a complete test run, although a lot
> of the running time is currently spent in the C compiler more than the
> actual test code runs.

I've noticed that running the test suite already takes a while--any
ideas what we can do to improve here? As you mentioned, it's the pyx
-> c and c -> so that's taking up the bulk of the time.

- Robert

From vitja.makarov at gmail.com  Sun Jun 19 21:37:55 2011
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Sun, 19 Jun 2011 23:37:55 +0400
Subject: [Cython] Some bugs found while testing cython on django
Message-ID: <BANLkTinrx6deoyLwd4yUtATM79KCuHPMMw@mail.gmail.com>

Hi!

Trying to compile django I've found some problems:

1. It seems that cython currently doesn't support tuples inside args definition:

def foo((a, b), c):

Currently this gives missing argument name and crash inside
CreateControlFlow graph

cython -X binding=True django/contrib/gis/forms/fields.py -o
django/contrib/gis/forms/fields.c

2. Concatenating unicode and str isn't supported:

I guess that this is mostly django problem

Error compiling Cython file:
------------------------------------------------------------
...
    default_error_messages = {
        'no_geom' : _(u'No geometry value provided.'),
        'invalid_geom' : _(u'Invalid geometry value.'),
        'invalid_geom_type' : _(u'Invalid geometry type.'),
        'transform_error' : _(u'An error occurred when transforming
the geometry '
                              'to the SRID of the geometry form field.'),
                             ^
------------------------------------------------------------

django/contrib/gis/forms/fields.py:21:30: Cannot mix string literals
of different types, expected u'', got ''

3.Reraise not inside except clause, here is simple example

def foo():
    print 'catched'
    raise
try:
    raise IndexError
except IndexError:
    foo()


-- 
vitja.

From jonovik at gmail.com  Mon Jun 20 07:28:05 2011
From: jonovik at gmail.com (Jon Olav Vik)
Date: Mon, 20 Jun 2011 05:28:05 +0000 (UTC)
Subject: [Cython] Some bugs found while testing cython on django
References: <BANLkTinrx6deoyLwd4yUtATM79KCuHPMMw@mail.gmail.com>
Message-ID: <loom.20110620T072216-896@post.gmane.org>

Vitja Makarov <vitja.makarov at ...> writes:

> 1. It seems that cython currently doesn't support tuples inside args 
definition:
> def foo((a, b), c):

Note that this feature is removed in Python 3.
http://www.python.org/dev/peps/pep-3113/



From stefan_ml at behnel.de  Mon Jun 20 08:58:37 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 20 Jun 2011 08:58:37 +0200
Subject: [Cython] Some bugs found while testing cython on django
In-Reply-To: <loom.20110620T072216-896@post.gmane.org>
References: <BANLkTinrx6deoyLwd4yUtATM79KCuHPMMw@mail.gmail.com>
	<loom.20110620T072216-896@post.gmane.org>
Message-ID: <4DFEEF9D.8000802@behnel.de>

Jon Olav Vik, 20.06.2011 07:28:
> Vitja Makarov writes:
>
>> 1. It seems that cython currently doesn't support tuples inside args
> definition:
>> def foo((a, b), c):
>
> Note that this feature is removed in Python 3.
> http://www.python.org/dev/peps/pep-3113/

... as already noted in ticket 692.

http://trac.cython.org/cython_trac/ticket/692

Personally, I'm against supporting this as it adds substantial overhead to 
the parser and the argument handling code for what is considered a dead 
feature, but full Py2 syntax support requires it.

Given that the Django developers are planning to move forward towards 
eventual Py3 support in their code base anyway, I wouldn't be surprised if 
they accepted a patch that fixed this in their code.

Stefan

From stefan_ml at behnel.de  Mon Jun 20 09:14:39 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 20 Jun 2011 09:14:39 +0200
Subject: [Cython] Some bugs found while testing cython on django
In-Reply-To: <BANLkTinrx6deoyLwd4yUtATM79KCuHPMMw@mail.gmail.com>
References: <BANLkTinrx6deoyLwd4yUtATM79KCuHPMMw@mail.gmail.com>
Message-ID: <4DFEF35F.9060801@behnel.de>

Vitja Makarov, 19.06.2011 21:37:
> Trying to compile django I've found some problems:
>
> 1. It seems that cython currently doesn't support tuples inside args definition:
>
> def foo((a, b), c):
>
> Currently this gives missing argument name and crash inside
> CreateControlFlow graph
>
> cython -X binding=True django/contrib/gis/forms/fields.py -o
> django/contrib/gis/forms/fields.c

It may be valid Py2-only syntax, but I consider its usage a bug in Django.


> 2. Concatenating unicode and str isn't supported:
>
> I guess that this is mostly django problem
>
> Error compiling Cython file:
> ------------------------------------------------------------
> ...
>      default_error_messages = {
>          'no_geom' : _(u'No geometry value provided.'),
>          'invalid_geom' : _(u'Invalid geometry value.'),
>          'invalid_geom_type' : _(u'Invalid geometry type.'),
>          'transform_error' : _(u'An error occurred when transforming
> the geometry '
>                                'to the SRID of the geometry form field.'),
>                               ^
> ------------------------------------------------------------
>
> django/contrib/gis/forms/fields.py:21:30: Cannot mix string literals
> of different types, expected u'', got ''

That's a bug in Django. Worth submitting a patch to their bug tracker. It 
works in this case, but it's not what they want in their code:

     >>> u'abc\u1234'
     u'abc\u1234'
     >>> u'abc' '\u1234'
     u'abc\\u1234'


> 3.Reraise not inside except clause, here is simple example
>
> def foo():
>      print 'catched'
>      raise
> try:
>      raise IndexError
> except IndexError:
>      foo()

It would be possible to support this, given that we are compatible with 
CPython regarding the place where exceptions are stored while being 
handled. Despite of what I initially thought, this is a feature of both Py2 
and Py3, even though I expect it to be rarely used and somewhat error prone.

Stefan

From vitja.makarov at gmail.com  Mon Jun 20 10:00:01 2011
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Mon, 20 Jun 2011 12:00:01 +0400
Subject: [Cython] Some bugs found while testing cython on django
In-Reply-To: <4DFEF35F.9060801@behnel.de>
References: <BANLkTinrx6deoyLwd4yUtATM79KCuHPMMw@mail.gmail.com>
	<4DFEF35F.9060801@behnel.de>
Message-ID: <BANLkTincCvydmrJoyWmW+PqhTGpWDBjHsw@mail.gmail.com>

2011/6/20 Stefan Behnel <stefan_ml at behnel.de>:
> Vitja Makarov, 19.06.2011 21:37:
>>
>> Trying to compile django I've found some problems:
>>
>> 1. It seems that cython currently doesn't support tuples inside args
>> definition:
>>
>> def foo((a, b), c):
>>
>> Currently this gives missing argument name and crash inside
>> CreateControlFlow graph
>>
>> cython -X binding=True django/contrib/gis/forms/fields.py -o
>> django/contrib/gis/forms/fields.c
>
> It may be valid Py2-only syntax, but I consider its usage a bug in Django.
>

Ok. Any way I should fix crash in FlowControl.

>
>> 2. Concatenating unicode and str isn't supported:
>>
>> I guess that this is mostly django problem
>>
>> Error compiling Cython file:
>> ------------------------------------------------------------
>> ...
>> ? ? default_error_messages = {
>> ? ? ? ? 'no_geom' : _(u'No geometry value provided.'),
>> ? ? ? ? 'invalid_geom' : _(u'Invalid geometry value.'),
>> ? ? ? ? 'invalid_geom_type' : _(u'Invalid geometry type.'),
>> ? ? ? ? 'transform_error' : _(u'An error occurred when transforming
>> the geometry '
>> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 'to the SRID of the geometry form field.'),
>> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?^
>> ------------------------------------------------------------
>>
>> django/contrib/gis/forms/fields.py:21:30: Cannot mix string literals
>> of different types, expected u'', got ''
>
> That's a bug in Django. Worth submitting a patch to their bug tracker. It
> works in this case, but it's not what they want in their code:
>
> ? ?>>> u'abc\u1234'
> ? ?u'abc\u1234'
> ? ?>>> u'abc' '\u1234'
> ? ?u'abc\\u1234'
>

Ok.

-- 
vitja.

From vitja.makarov at gmail.com  Mon Jun 20 22:23:54 2011
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Tue, 21 Jun 2011 00:23:54 +0400
Subject: [Cython] Bindings performance issue
In-Reply-To: <BANLkTi=k5MqQKKEt6pr+wK0fv7tRyJGJSw@mail.gmail.com>
References: <BANLkTi=yWzE3MJqq3CUwe8TCcPPkpXSuiA@mail.gmail.com>
	<BANLkTik59bpyST=7RC6EJiLkrOPQHSujvg@mail.gmail.com>
	<BANLkTik99WbBap5BgKyme2UBjtoiVrEATQ@mail.gmail.com>
	<BANLkTikYr4qfkW-qA0-LcynQhdu9Zg6DcA@mail.gmail.com>
	<BANLkTi=ps-zjHZxaodoGQ4djJQJRCU7zfw@mail.gmail.com>
	<BANLkTindDiWJEjZGbELa-dbz591EhzLPug@mail.gmail.com>
	<BANLkTinAwrra+ru8PC4x_68Eou1iH9QACw@mail.gmail.com>
	<BANLkTi=KnePkE7Lf7jWK4tDUzX1A9ufBKQ@mail.gmail.com>
	<BANLkTi=8szSqJTrwwvOZwx1x=BXFcs6hdQ@mail.gmail.com>
	<BANLkTimELr=UmYLpt=wLaxiarFVP6j_ecg@mail.gmail.com>
	<BANLkTinzJTPx58fs+C-R0Y26O=iFpJC-YA@mail.gmail.com>
	<BANLkTi=Kn87FiQHY1pJ=rjTFR6HiqHPPCA@mail.gmail.com>
	<BANLkTi=qUHAMp6ENB+1GhkSyhSypkNazJQ@mail.gmail.com>
	<BANLkTinxxB4rAdfVK7RwynB+bcoTy=YvzA@mail.gmail.com>
	<BANLkTim+TibgtVgrT-1TG+OduGegAXtj-A@mail.gmail.com>
	<BANLkTin2A7iP3ocmwqfqv1JS0gotDZztGQ@mail.gmail.com>
	<BANLkTin36dOg--NE2zVRpjsDi0aaCbfa8g@mail.gmail.com>
	<BANLkTinM9bwbZbz=_yYok7UXUqrbZwAsBw@mail.gmail.com>
	<BANLkTi=k5MqQKKEt6pr+wK0fv7tRyJGJSw@mail.gmail.com>
Message-ID: <BANLkTi=Zy7QNwtAsQsWQKeMh4OxV_V7cDA@mail.gmail.com>

Wow now we have about 11K tests with 171 errors!

https://sage.math.washington.edu:8091/hudson/view/cython-vitek/job/cython-vitek-tests-pyregr-py27-c/

-- 
vitja.

From robertwb at math.washington.edu  Tue Jun 21 01:14:50 2011
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 20 Jun 2011 16:14:50 -0700
Subject: [Cython] Some bugs found while testing cython on django
In-Reply-To: <4DFEF35F.9060801@behnel.de>
References: <BANLkTinrx6deoyLwd4yUtATM79KCuHPMMw@mail.gmail.com>
	<4DFEF35F.9060801@behnel.de>
Message-ID: <BANLkTimNwg2zgmyxuFr+gL3udrzgaCXpf6zWmhzKu_LZR7612g@mail.gmail.com>

On Mon, Jun 20, 2011 at 12:14 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Vitja Makarov, 19.06.2011 21:37:
>>
>> Trying to compile django I've found some problems:
>>
>> 1. It seems that cython currently doesn't support tuples inside args
>> definition:
>>
>> def foo((a, b), c):
>>
>> Currently this gives missing argument name and crash inside
>> CreateControlFlow graph
>>
>> cython -X binding=True django/contrib/gis/forms/fields.py -o
>> django/contrib/gis/forms/fields.c
>
> It may be valid Py2-only syntax, but I consider its usage a bug in Django.

I don't know about the parser side of things, but for argument
handling, I'd be OK with falling back to the generic
parse-tuple-and-keywords for this case rather than complicating our
argument unpacking code. Assuming it simplifies things, I'm OK with
not supporting typed arguments of this form.

>> 2. Concatenating unicode and str isn't supported:
>>
>> I guess that this is mostly django problem
>>
>> Error compiling Cython file:
>> ------------------------------------------------------------
>> ...
>> ? ? default_error_messages = {
>> ? ? ? ? 'no_geom' : _(u'No geometry value provided.'),
>> ? ? ? ? 'invalid_geom' : _(u'Invalid geometry value.'),
>> ? ? ? ? 'invalid_geom_type' : _(u'Invalid geometry type.'),
>> ? ? ? ? 'transform_error' : _(u'An error occurred when transforming
>> the geometry '
>> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 'to the SRID of the geometry form field.'),
>> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?^
>> ------------------------------------------------------------
>>
>> django/contrib/gis/forms/fields.py:21:30: Cannot mix string literals
>> of different types, expected u'', got ''
>
> That's a bug in Django. Worth submitting a patch to their bug tracker. It
> works in this case, but it's not what they want in their code:
>
> ? ?>>> u'abc\u1234'
> ? ?u'abc\u1234'
> ? ?>>> u'abc' '\u1234'
> ? ?u'abc\\u1234'

This error is only for literals (or otherwise typed value, right?) I
might say it's not a bug in their code--the second unicode marker is
redundant here if you know how things work in Py2.

>> 3.Reraise not inside except clause, here is simple example
>>
>> def foo():
>> ? ? print 'catched'
>> ? ? raise
>> try:
>> ? ? raise IndexError
>> except IndexError:
>> ? ? foo()
>
> It would be possible to support this, given that we are compatible with
> CPython regarding the place where exceptions are stored while being handled.
> Despite of what I initially thought, this is a feature of both Py2 and Py3,
> even though I expect it to be rarely used and somewhat error prone.

It's a bit odd, but should still be supported.

- Robert

From stefan_ml at behnel.de  Tue Jun 21 08:50:15 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 21 Jun 2011 08:50:15 +0200
Subject: [Cython] Bindings performance issue
In-Reply-To: <BANLkTi=Zy7QNwtAsQsWQKeMh4OxV_V7cDA@mail.gmail.com>
References: <BANLkTi=yWzE3MJqq3CUwe8TCcPPkpXSuiA@mail.gmail.com>	<BANLkTikYr4qfkW-qA0-LcynQhdu9Zg6DcA@mail.gmail.com>	<BANLkTi=ps-zjHZxaodoGQ4djJQJRCU7zfw@mail.gmail.com>	<BANLkTindDiWJEjZGbELa-dbz591EhzLPug@mail.gmail.com>	<BANLkTinAwrra+ru8PC4x_68Eou1iH9QACw@mail.gmail.com>	<BANLkTi=KnePkE7Lf7jWK4tDUzX1A9ufBKQ@mail.gmail.com>	<BANLkTi=8szSqJTrwwvOZwx1x=BXFcs6hdQ@mail.gmail.com>	<BANLkTimELr=UmYLpt=wLaxiarFVP6j_ecg@mail.gmail.com>	<BANLkTinzJTPx58fs+C-R0Y26O=iFpJC-YA@mail.gmail.com>	<BANLkTi=Kn87FiQHY1pJ=rjTFR6HiqHPPCA@mail.gmail.com>	<BANLkTi=qUHAMp6ENB+1GhkSyhSypkNazJQ@mail.gmail.com>	<BANLkTinxxB4rAdfVK7RwynB+bcoTy=YvzA@mail.gmail.com>	<BANLkTim+TibgtVgrT-1TG+OduGegAXtj-A@mail.gmail.com>	<BANLkTin2A7iP3ocmwqfqv1JS0gotDZztGQ@mail.gmail.com>	<BANLkTin36dOg--NE2zVRpjsDi0aaCbfa8g@mail.gmail.com>	<BANLkTinM9bwbZbz=_yYok7UXUqrbZwAsBw@mail.gmail.com>	<BANLkTi=k5MqQKKEt6pr+wK0fv7tRyJGJSw@mail.gmail.com>
	<BANLkTi=Zy7QNwtAsQsWQKeMh4OxV_V7cDA@mail.gmail.com>
Message-ID: <4E003F27.5030808@behnel.de>

Vitja Makarov, 20.06.2011 22:23:
> Wow now we have about 11K tests with 171 errors!
>
> https://sage.math.washington.edu:8091/hudson/view/cython-vitek/job/cython-vitek-tests-pyregr-py27-c/

Yes, that was a low hanging, high value bug.

"""
Make cyfunction __name__ attribute writable
"""

Could you provide a pull request with that change?

Thanks!

Stefan

From vitja.makarov at gmail.com  Tue Jun 21 09:07:07 2011
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Tue, 21 Jun 2011 11:07:07 +0400
Subject: [Cython] Bindings performance issue
In-Reply-To: <4E003F27.5030808@behnel.de>
References: <BANLkTi=yWzE3MJqq3CUwe8TCcPPkpXSuiA@mail.gmail.com>
	<BANLkTikYr4qfkW-qA0-LcynQhdu9Zg6DcA@mail.gmail.com>
	<BANLkTi=ps-zjHZxaodoGQ4djJQJRCU7zfw@mail.gmail.com>
	<BANLkTindDiWJEjZGbELa-dbz591EhzLPug@mail.gmail.com>
	<BANLkTinAwrra+ru8PC4x_68Eou1iH9QACw@mail.gmail.com>
	<BANLkTi=KnePkE7Lf7jWK4tDUzX1A9ufBKQ@mail.gmail.com>
	<BANLkTi=8szSqJTrwwvOZwx1x=BXFcs6hdQ@mail.gmail.com>
	<BANLkTimELr=UmYLpt=wLaxiarFVP6j_ecg@mail.gmail.com>
	<BANLkTinzJTPx58fs+C-R0Y26O=iFpJC-YA@mail.gmail.com>
	<BANLkTi=Kn87FiQHY1pJ=rjTFR6HiqHPPCA@mail.gmail.com>
	<BANLkTi=qUHAMp6ENB+1GhkSyhSypkNazJQ@mail.gmail.com>
	<BANLkTinxxB4rAdfVK7RwynB+bcoTy=YvzA@mail.gmail.com>
	<BANLkTim+TibgtVgrT-1TG+OduGegAXtj-A@mail.gmail.com>
	<BANLkTin2A7iP3ocmwqfqv1JS0gotDZztGQ@mail.gmail.com>
	<BANLkTin36dOg--NE2zVRpjsDi0aaCbfa8g@mail.gmail.com>
	<BANLkTinM9bwbZbz=_yYok7UXUqrbZwAsBw@mail.gmail.com>
	<BANLkTi=k5MqQKKEt6pr+wK0fv7tRyJGJSw@mail.gmail.com>
	<BANLkTi=Zy7QNwtAsQsWQKeMh4OxV_V7cDA@mail.gmail.com>
	<4E003F27.5030808@behnel.de>
Message-ID: <BANLkTik85HNOQLHWSyn_XetC-z4ioUNUbA@mail.gmail.com>

2011/6/21 Stefan Behnel <stefan_ml at behnel.de>:
> Vitja Makarov, 20.06.2011 22:23:
>>
>> Wow now we have about 11K tests with 171 errors!
>>
>>
>> https://sage.math.washington.edu:8091/hudson/view/cython-vitek/job/cython-vitek-tests-pyregr-py27-c/
>
> Yes, that was a low hanging, high value bug.
>
> """
> Make cyfunction __name__ attribute writable
> """
>
> Could you provide a pull request with that change?
>

Do you mean single change or whole branch?

-- 
vitja.

From vitja.makarov at gmail.com  Tue Jun 21 09:11:24 2011
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Tue, 21 Jun 2011 11:11:24 +0400
Subject: [Cython] Bindings performance issue
In-Reply-To: <BANLkTik85HNOQLHWSyn_XetC-z4ioUNUbA@mail.gmail.com>
References: <BANLkTi=yWzE3MJqq3CUwe8TCcPPkpXSuiA@mail.gmail.com>
	<BANLkTikYr4qfkW-qA0-LcynQhdu9Zg6DcA@mail.gmail.com>
	<BANLkTi=ps-zjHZxaodoGQ4djJQJRCU7zfw@mail.gmail.com>
	<BANLkTindDiWJEjZGbELa-dbz591EhzLPug@mail.gmail.com>
	<BANLkTinAwrra+ru8PC4x_68Eou1iH9QACw@mail.gmail.com>
	<BANLkTi=KnePkE7Lf7jWK4tDUzX1A9ufBKQ@mail.gmail.com>
	<BANLkTi=8szSqJTrwwvOZwx1x=BXFcs6hdQ@mail.gmail.com>
	<BANLkTimELr=UmYLpt=wLaxiarFVP6j_ecg@mail.gmail.com>
	<BANLkTinzJTPx58fs+C-R0Y26O=iFpJC-YA@mail.gmail.com>
	<BANLkTi=Kn87FiQHY1pJ=rjTFR6HiqHPPCA@mail.gmail.com>
	<BANLkTi=qUHAMp6ENB+1GhkSyhSypkNazJQ@mail.gmail.com>
	<BANLkTinxxB4rAdfVK7RwynB+bcoTy=YvzA@mail.gmail.com>
	<BANLkTim+TibgtVgrT-1TG+OduGegAXtj-A@mail.gmail.com>
	<BANLkTin2A7iP3ocmwqfqv1JS0gotDZztGQ@mail.gmail.com>
	<BANLkTin36dOg--NE2zVRpjsDi0aaCbfa8g@mail.gmail.com>
	<BANLkTinM9bwbZbz=_yYok7UXUqrbZwAsBw@mail.gmail.com>
	<BANLkTi=k5MqQKKEt6pr+wK0fv7tRyJGJSw@mail.gmail.com>
	<BANLkTi=Zy7QNwtAsQsWQKeMh4OxV_V7cDA@mail.gmail.com>
	<4E003F27.5030808@behnel.de>
	<BANLkTik85HNOQLHWSyn_XetC-z4ioUNUbA@mail.gmail.com>
Message-ID: <BANLkTi=+dVOGiTXX+h-mLKfy9Keg_unZnw@mail.gmail.com>

2011/6/21 Vitja Makarov <vitja.makarov at gmail.com>:
> 2011/6/21 Stefan Behnel <stefan_ml at behnel.de>:
>> Vitja Makarov, 20.06.2011 22:23:
>>>
>>> Wow now we have about 11K tests with 171 errors!
>>>
>>>
>>> https://sage.math.washington.edu:8091/hudson/view/cython-vitek/job/cython-vitek-tests-pyregr-py27-c/
>>
>> Yes, that was a low hanging, high value bug.
>>
>> """
>> Make cyfunction __name__ attribute writable
>> """
>>
>> Could you provide a pull request with that change?
>>
>
> Do you mean single change or whole branch?
>

This commit
https://github.com/vitek/cython/commit/7b4471bcc5eeb09b2e4851d3a1c2c3d2a5bd085c

disables bindings for inner functions and breaks closure_decorators_T478 test.

I can remove that or fix testcase.


-- 
vitja.

From stefan_ml at behnel.de  Tue Jun 21 09:19:43 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 21 Jun 2011 09:19:43 +0200
Subject: [Cython] Some bugs found while testing cython on django
In-Reply-To: <BANLkTimNwg2zgmyxuFr+gL3udrzgaCXpf6zWmhzKu_LZR7612g@mail.gmail.com>
References: <BANLkTinrx6deoyLwd4yUtATM79KCuHPMMw@mail.gmail.com>	<4DFEF35F.9060801@behnel.de>
	<BANLkTimNwg2zgmyxuFr+gL3udrzgaCXpf6zWmhzKu_LZR7612g@mail.gmail.com>
Message-ID: <4E00460F.8090104@behnel.de>

Robert Bradshaw, 21.06.2011 01:14:
> On Mon, Jun 20, 2011 at 12:14 AM, Stefan Behnel wrote:
>> Vitja Makarov, 19.06.2011 21:37:
>>>
>>> Trying to compile django I've found some problems:
>>>
>>> 1. It seems that cython currently doesn't support tuples inside args
>>> definition:
>>>
>>> def foo((a, b), c):
>>>
>>> Currently this gives missing argument name and crash inside
>>> CreateControlFlow graph
>>>
>>> cython -X binding=True django/contrib/gis/forms/fields.py -o
>>> django/contrib/gis/forms/fields.c
>>
>> It may be valid Py2-only syntax, but I consider its usage a bug in Django.
>
> I don't know about the parser side of things

Sadly, that's the bigger part of the problem. The parser code for function 
arguments is pretty complex because it basically has to parse C syntax at 
this point. I tried to implement the parser part before I opened ticket 692 
and it turned out to be a major ugliness. Currently, the parser even reads 
this without complaining. I don't recall the exact details, but I think it 
was  because it resembles a part of a C function type declaration or something.

Once we get this through the parser correctly, it won't be all that hard to 
implement anymore. However, IMHO, it's a lot of work overall for a dead 
feature.


>>> 2. Concatenating unicode and str isn't supported:
>>>
>>> I guess that this is mostly django problem
>>>
>>> Error compiling Cython file:
>>> ------------------------------------------------------------
>>> ...
>>>      default_error_messages = {
>>>          'no_geom' : _(u'No geometry value provided.'),
>>>          'invalid_geom' : _(u'Invalid geometry value.'),
>>>          'invalid_geom_type' : _(u'Invalid geometry type.'),
>>>          'transform_error' : _(u'An error occurred when transforming
>>> the geometry '
>>>                                'to the SRID of the geometry form field.'),
>>>                               ^
>>> ------------------------------------------------------------
>>>
>>> django/contrib/gis/forms/fields.py:21:30: Cannot mix string literals
>>> of different types, expected u'', got ''
>>
>> That's a bug in Django. Worth submitting a patch to their bug tracker. It
>> works in this case, but it's not what they want in their code:
>>
>>     >>>  u'abc\u1234'
>>     u'abc\u1234'
>>     >>>  u'abc' '\u1234'
>>     u'abc\\u1234'
>
> This error is only for literals (or otherwise typed value, right?)

This is *only* about string literals and their auto-concatenation by the 
parser.


> I might say it's not a bug in their code--the second unicode marker is
> redundant here if you know how things work in Py2.

Right, *if* you know it. So the above will fail to provide the correct 
result if someone who happens to be not aware of these details changes the 
literal to something that contains a non-ASCII character. And this problem 
may or may not show up in a test, as the above is only a text message, even 
just an (unlikely?) error message.

Rather error prone, if you ask me. Which lets me suspect that the code was 
written by someone who was not aware of the details in the first place.

I must say, I like the error message that Cython outputs here. Would have 
caught that problem earlier.  :)


>>> 3.Reraise not inside except clause, here is simple example
>>>
>>> def foo():
>>>      print 'catched'
>>>      raise
>>> try:
>>>      raise IndexError
>>> except IndexError:
>>>      foo()
>>
>> It would be possible to support this, given that we are compatible with
>> CPython regarding the place where exceptions are stored while being handled.
>> Despite of what I initially thought, this is a feature of both Py2 and Py3,
>> even though I expect it to be rarely used and somewhat error prone.
>
> It's a bit odd, but should still be supported.

Oh, absolutely. I should have stated that clearer.

Stefan

From stefan_ml at behnel.de  Tue Jun 21 09:27:59 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 21 Jun 2011 09:27:59 +0200
Subject: [Cython] Bindings performance issue
In-Reply-To: <BANLkTik85HNOQLHWSyn_XetC-z4ioUNUbA@mail.gmail.com>
References: <BANLkTi=yWzE3MJqq3CUwe8TCcPPkpXSuiA@mail.gmail.com>	<BANLkTi=ps-zjHZxaodoGQ4djJQJRCU7zfw@mail.gmail.com>	<BANLkTindDiWJEjZGbELa-dbz591EhzLPug@mail.gmail.com>	<BANLkTinAwrra+ru8PC4x_68Eou1iH9QACw@mail.gmail.com>	<BANLkTi=KnePkE7Lf7jWK4tDUzX1A9ufBKQ@mail.gmail.com>	<BANLkTi=8szSqJTrwwvOZwx1x=BXFcs6hdQ@mail.gmail.com>	<BANLkTimELr=UmYLpt=wLaxiarFVP6j_ecg@mail.gmail.com>	<BANLkTinzJTPx58fs+C-R0Y26O=iFpJC-YA@mail.gmail.com>	<BANLkTi=Kn87FiQHY1pJ=rjTFR6HiqHPPCA@mail.gmail.com>	<BANLkTi=qUHAMp6ENB+1GhkSyhSypkNazJQ@mail.gmail.com>	<BANLkTinxxB4rAdfVK7RwynB+bcoTy=YvzA@mail.gmail.com>	<BANLkTim+TibgtVgrT-1TG+OduGegAXtj-A@mail.gmail.com>	<BANLkTin2A7iP3ocmwqfqv1JS0gotDZztGQ@mail.gmail.com>	<BANLkTin36dOg--NE2zVRpjsDi0aaCbfa8g@mail.gmail.com>	<BANLkTinM9bwbZbz=_yYok7UXUqrbZwAsBw@mail.gmail.com>	<BANLkTi=k5MqQKKEt6pr+wK0fv7tRyJGJSw@mail.gmail.com>	<BANLkTi=Zy7QNwtAsQsWQKeMh4OxV_V7cDA@mail.gmail.com>	<4E003F27.5030808@behnel.de>
	<BANLkTik85HNOQLHWSyn_XetC-z4ioUNUbA@mail.gmail.com>
Message-ID: <4E0047FF.10007@behnel.de>

Vitja Makarov, 21.06.2011 09:07:
> 2011/6/21 Stefan Behnel:
>> Vitja Makarov, 20.06.2011 22:23:
>>>
>>> Wow now we have about 11K tests with 171 errors!
>>>
>>> https://sage.math.washington.edu:8091/hudson/view/cython-vitek/job/cython-vitek-tests-pyregr-py27-c/
>>
>> Yes, that was a low hanging, high value bug.
>>
>> """
>> Make cyfunction __name__ attribute writable
>> """
>>
>> Could you provide a pull request with that change?
>
> Do you mean single change or whole branch?

Hmm, I didn't look at the other changes in the branch yet. Actually, I 
followed the link in Jenkins and the changeset page didn't even show me 
which branch it was. This one, I guess:

https://github.com/vitek/cython/commits/_bindings

I'll give it a review when I get to it.

Do you think this branch is ready for a merge, or is there more to be done?

Stefan

From vitja.makarov at gmail.com  Tue Jun 21 09:34:11 2011
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Tue, 21 Jun 2011 11:34:11 +0400
Subject: [Cython] Bindings performance issue
In-Reply-To: <4E0047FF.10007@behnel.de>
References: <BANLkTi=yWzE3MJqq3CUwe8TCcPPkpXSuiA@mail.gmail.com>
	<BANLkTi=ps-zjHZxaodoGQ4djJQJRCU7zfw@mail.gmail.com>
	<BANLkTindDiWJEjZGbELa-dbz591EhzLPug@mail.gmail.com>
	<BANLkTinAwrra+ru8PC4x_68Eou1iH9QACw@mail.gmail.com>
	<BANLkTi=KnePkE7Lf7jWK4tDUzX1A9ufBKQ@mail.gmail.com>
	<BANLkTi=8szSqJTrwwvOZwx1x=BXFcs6hdQ@mail.gmail.com>
	<BANLkTimELr=UmYLpt=wLaxiarFVP6j_ecg@mail.gmail.com>
	<BANLkTinzJTPx58fs+C-R0Y26O=iFpJC-YA@mail.gmail.com>
	<BANLkTi=Kn87FiQHY1pJ=rjTFR6HiqHPPCA@mail.gmail.com>
	<BANLkTi=qUHAMp6ENB+1GhkSyhSypkNazJQ@mail.gmail.com>
	<BANLkTinxxB4rAdfVK7RwynB+bcoTy=YvzA@mail.gmail.com>
	<BANLkTim+TibgtVgrT-1TG+OduGegAXtj-A@mail.gmail.com>
	<BANLkTin2A7iP3ocmwqfqv1JS0gotDZztGQ@mail.gmail.com>
	<BANLkTin36dOg--NE2zVRpjsDi0aaCbfa8g@mail.gmail.com>
	<BANLkTinM9bwbZbz=_yYok7UXUqrbZwAsBw@mail.gmail.com>
	<BANLkTi=k5MqQKKEt6pr+wK0fv7tRyJGJSw@mail.gmail.com>
	<BANLkTi=Zy7QNwtAsQsWQKeMh4OxV_V7cDA@mail.gmail.com>
	<4E003F27.5030808@behnel.de>
	<BANLkTik85HNOQLHWSyn_XetC-z4ioUNUbA@mail.gmail.com>
	<4E0047FF.10007@behnel.de>
Message-ID: <BANLkTino-eO=Sew=+wfnZBhU0QfWxrD4+w@mail.gmail.com>

2011/6/21 Stefan Behnel <stefan_ml at behnel.de>:
> Vitja Makarov, 21.06.2011 09:07:
>>
>> 2011/6/21 Stefan Behnel:
>>>
>>> Vitja Makarov, 20.06.2011 22:23:
>>>>
>>>> Wow now we have about 11K tests with 171 errors!
>>>>
>>>>
>>>> https://sage.math.washington.edu:8091/hudson/view/cython-vitek/job/cython-vitek-tests-pyregr-py27-c/
>>>
>>> Yes, that was a low hanging, high value bug.
>>>
>>> """
>>> Make cyfunction __name__ attribute writable
>>> """
>>>
>>> Could you provide a pull request with that change?
>>
>> Do you mean single change or whole branch?
>
> Hmm, I didn't look at the other changes in the branch yet. Actually, I
> followed the link in Jenkins and the changeset page didn't even show me
> which branch it was. This one, I guess:
>
> https://github.com/vitek/cython/commits/_bindings
>
> I'll give it a review when I get to it.
>
> Do you think this branch is ready for a merge, or is there more to be done?
>

I think it's almost done. But it's better to review it because lots of
new C code there.
So I can create pull request to make code review easy.

-- 
vitja.

From vitja.makarov at gmail.com  Wed Jun 22 09:37:12 2011
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Wed, 22 Jun 2011 11:37:12 +0400
Subject: [Cython] strange bug in parallel assignment
Message-ID: <BANLkTiknjtDKPy+jUxgL3yaqpfYGn3CsHg@mail.gmail.com>

Hi!

Looking at hudson pyregr output I've found this issue:

(a, b), (c,) = (1, 2), {1: 2}

Traceback (most recent call last):
  File "cython.py", line 17, in <module>
    main(command_line = 1)
  File "/home/vitja/work/cython-vitek/Cython/Compiler/Main.py", line
821, in main
    result = compile(sources, options)
  File "/home/vitja/work/cython-vitek/Cython/Compiler/Main.py", line
796, in compile
    return compile_multiple(source, options)
  File "/home/vitja/work/cython-vitek/Cython/Compiler/Main.py", line
768, in compile_multiple
    result = run_pipeline(source, options)
  File "/home/vitja/work/cython-vitek/Cython/Compiler/Main.py", line
632, in run_pipeline
    err, enddata = context.run_pipeline(pipeline, source)
  File "/home/vitja/work/cython-vitek/Cython/Compiler/Main.py", line
251, in run_pipeline
    data = phase(data)
  File "/home/vitja/work/cython-vitek/Cython/Compiler/Main.py", line
173, in generate_pyx_code
    module_node.process_implementation(options, result)
  File "/home/vitja/work/cython-vitek/Cython/Compiler/ModuleNode.py",
line 73, in process_implementation
    self.generate_c_code(env, options, result)
  File "/home/vitja/work/cython-vitek/Cython/Compiler/ModuleNode.py",
line 310, in generate_c_code
    self.generate_module_init_func(modules[:-1], env,
globalstate['init_module'])
  File "/home/vitja/work/cython-vitek/Cython/Compiler/ModuleNode.py",
line 1924, in generate_module_init_func
    self.body.generate_execution_code(code)
  File "/home/vitja/work/cython-vitek/Cython/Compiler/Nodes.py", line
347, in generate_execution_code
    stat.generate_execution_code(code)
  File "/home/vitja/work/cython-vitek/Cython/Compiler/Nodes.py", line
3811, in generate_execution_code
    stat.generate_assignment_code(code)
  File "/home/vitja/work/cython-vitek/Cython/Compiler/Nodes.py", line
3709, in generate_assignment_code
    self.lhs.generate_assignment_code(self.rhs, code)
  File "/home/vitja/work/cython-vitek/Cython/Compiler/ExprNodes.py",
line 4036, in generate_assignment_code
    self.generate_parallel_assignment_code(rhs, code)
  File "/home/vitja/work/cython-vitek/Cython/Compiler/ExprNodes.py",
line 4088, in generate_parallel_assignment_code
    self.generate_generic_parallel_unpacking_code(code, rhs)
  File "/home/vitja/work/cython-vitek/Cython/Compiler/ExprNodes.py",
line 4137, in generate_generic_parallel_unpacking_code
    item.result(),
  File "/home/vitja/work/cython-vitek/Cython/Compiler/ExprNodes.py",
line 2047, in result
    print self.pos, self.type, self.temp_cname
AttributeError: 'PyTempNode' object has no attribute 'temp_cname'



-- 
vitja.

From vitja.makarov at gmail.com  Wed Jun 22 09:40:10 2011
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Wed, 22 Jun 2011 11:40:10 +0400
Subject: [Cython] strange bug in parallel assignment
In-Reply-To: <BANLkTiknjtDKPy+jUxgL3yaqpfYGn3CsHg@mail.gmail.com>
References: <BANLkTiknjtDKPy+jUxgL3yaqpfYGn3CsHg@mail.gmail.com>
Message-ID: <BANLkTikMnAgLFpubZ8oBiVHVzQPRK0cV0g@mail.gmail.com>

2011/6/22 Vitja Makarov <vitja.makarov at gmail.com>:
> Hi!
>
> Looking at hudson pyregr output I've found this issue:
>
> (a, b), (c,) = (1, 2), {1: 2}
>
> Traceback (most recent call last):
> ?File "cython.py", line 17, in <module>
> ? ?main(command_line = 1)
> ?File "/home/vitja/work/cython-vitek/Cython/Compiler/Main.py", line
> 821, in main
> ? ?result = compile(sources, options)
> ?File "/home/vitja/work/cython-vitek/Cython/Compiler/Main.py", line
> 796, in compile
> ? ?return compile_multiple(source, options)
> ?File "/home/vitja/work/cython-vitek/Cython/Compiler/Main.py", line
> 768, in compile_multiple
> ? ?result = run_pipeline(source, options)
> ?File "/home/vitja/work/cython-vitek/Cython/Compiler/Main.py", line
> 632, in run_pipeline
> ? ?err, enddata = context.run_pipeline(pipeline, source)
> ?File "/home/vitja/work/cython-vitek/Cython/Compiler/Main.py", line
> 251, in run_pipeline
> ? ?data = phase(data)
> ?File "/home/vitja/work/cython-vitek/Cython/Compiler/Main.py", line
> 173, in generate_pyx_code
> ? ?module_node.process_implementation(options, result)
> ?File "/home/vitja/work/cython-vitek/Cython/Compiler/ModuleNode.py",
> line 73, in process_implementation
> ? ?self.generate_c_code(env, options, result)
> ?File "/home/vitja/work/cython-vitek/Cython/Compiler/ModuleNode.py",
> line 310, in generate_c_code
> ? ?self.generate_module_init_func(modules[:-1], env,
> globalstate['init_module'])
> ?File "/home/vitja/work/cython-vitek/Cython/Compiler/ModuleNode.py",
> line 1924, in generate_module_init_func
> ? ?self.body.generate_execution_code(code)
> ?File "/home/vitja/work/cython-vitek/Cython/Compiler/Nodes.py", line
> 347, in generate_execution_code
> ? ?stat.generate_execution_code(code)
> ?File "/home/vitja/work/cython-vitek/Cython/Compiler/Nodes.py", line
> 3811, in generate_execution_code
> ? ?stat.generate_assignment_code(code)
> ?File "/home/vitja/work/cython-vitek/Cython/Compiler/Nodes.py", line
> 3709, in generate_assignment_code
> ? ?self.lhs.generate_assignment_code(self.rhs, code)
> ?File "/home/vitja/work/cython-vitek/Cython/Compiler/ExprNodes.py",
> line 4036, in generate_assignment_code
> ? ?self.generate_parallel_assignment_code(rhs, code)
> ?File "/home/vitja/work/cython-vitek/Cython/Compiler/ExprNodes.py",
> line 4088, in generate_parallel_assignment_code
> ? ?self.generate_generic_parallel_unpacking_code(code, rhs)
> ?File "/home/vitja/work/cython-vitek/Cython/Compiler/ExprNodes.py",
> line 4137, in generate_generic_parallel_unpacking_code
> ? ?item.result(),
> ?File "/home/vitja/work/cython-vitek/Cython/Compiler/ExprNodes.py",
> line 2047, in result
> ? ?print self.pos, self.type, self.temp_cname
> AttributeError: 'PyTempNode' object has no attribute 'temp_cname'
>
>

Oops. last line is incorrect it should be something like this:

  File "/home/vitja/work/cython-vitek/Cython/Compiler/ExprNodes.py",
line 2050, in result
    assert False, "Remember to call allocate/release on TempNode"
AssertionError: Remember to call allocate/release on TempNode

Btw this code leads to the same error:
(a, b), (c,) = (1, 2), set([1])



-- 
vitja.

From stefan_ml at behnel.de  Mon Jun 27 10:08:18 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 27 Jun 2011 10:08:18 +0200
Subject: [Cython] PEP 380 approved: syntax for generator delegation
Message-ID: <4E083A72.3050309@behnel.de>

Hi,

PEP 380 ("yield from") recently went through the approval gate.

http://www.python.org/dev/peps/pep-0380/

The implementation in CPython isn't rounded up yet and lots of tests are 
missing from the test suite. But it's agreed that it'll make it into 3.3, 
and we should try to support it as well. It's a really nice feature.

Stefan

From vitja.makarov at gmail.com  Mon Jun 27 22:31:00 2011
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Tue, 28 Jun 2011 00:31:00 +0400
Subject: [Cython] PEP 380 approved: syntax for generator delegation
In-Reply-To: <4E083A72.3050309@behnel.de>
References: <4E083A72.3050309@behnel.de>
Message-ID: <BANLkTi=zjaS43x8F_UvGk_2ecCtgwp9Y9Q@mail.gmail.com>

2011/6/27 Stefan Behnel <stefan_ml at behnel.de>:
> Hi,
>
> PEP 380 ("yield from") recently went through the approval gate.
>
> http://www.python.org/dev/peps/pep-0380/
>
> The implementation in CPython isn't rounded up yet and lots of tests are
> missing from the test suite. But it's agreed that it'll make it into 3.3,
> and we should try to support it as well. It's a really nice feature.
>

Wow! Interesting thing.
I think that could be implemented but I think we should try to
optimize cython generators first.


-- 
vitja.