From robertwb at gmail.com  Fri Aug  1 06:34:22 2014
From: robertwb at gmail.com (Robert Bradshaw)
Date: Thu, 31 Jul 2014 21:34:22 -0700
Subject: [Cython] Automatic conversion with fixed-size C arrays
In-Reply-To: <CALpTaNZS9127ti_PPui4gJpA8tK6AHH2LpLKqXLR9HQ-uP-O1g@mail.gmail.com>
References: <CALpTaNYskf94RzTExj4JEn5sO7b6ryYBhVFrFYAzxZ96R7onRg@mail.gmail.com>
 <CADiQ+QBJ_dSCXV-hVRmyqMAmhApQfVDE4B7H9DVZzP8L8N+S+A@mail.gmail.com>
 <CALpTaNb+nGeOfPgaLjsLxrWP8ug=y_2f7+uTKs5_hUQnGaWx5Q@mail.gmail.com>
 <CADiQ+QDC5SF2xRhzZCH_ufPjn_59VuJX+gVyMJpzizK+S9-zTA@mail.gmail.com>
 <CADiQ+QBwn7=_5bXAO+-3-m31LG13ASo5Df8-QWum6FhTsDpS=A@mail.gmail.com>
 <CALpTaNbF4UypSmVAATcGRiJnno7A642Nd8uAPVz25rpAELD2Ew@mail.gmail.com>
 <CALpTaNYj9QhVwNk1JEZgn5Sq7YdApXYZJR_6=Fuj2umfk6xPfQ@mail.gmail.com>
 <CADiQ+QDWe_mJPPFQV0FJO3zrRuDDNLTcPmWAv7Z-NTK72XjGoQ@mail.gmail.com>
 <CALpTaNZ3nd878=okszUj36E8JJazc19fnFny8NW-ZCQBfJoMMw@mail.gmail.com>
 <CALpTaNZS9127ti_PPui4gJpA8tK6AHH2LpLKqXLR9HQ-uP-O1g@mail.gmail.com>
Message-ID: <CADiQ+QDpPBzLg9N617JLL7TbzMwfWtpsRifzOg=E5Y4YBkYwxw@mail.gmail.com>

We can move to the PR, makes it easier to do line commenting.

On Tue, Jul 29, 2014 at 9:54 PM, Kurt Smith <kwmsmith at gmail.com> wrote:
> On Mon, Jul 28, 2014 at 11:29 PM, Kurt Smith <kwmsmith at gmail.com> wrote:
>>
>> On Mon, Jul 28, 2014 at 11:02 AM, Robert Bradshaw <robertwb at gmail.com>
>> wrote:
>>>
>>> On Sun, Jul 27, 2014 at 9:34 PM, Kurt Smith <kwmsmith at gmail.com> wrote:
>>>
>>> > I have sorted out all the code
>>> > generation pieces and nearly have a minimal example working, and will
>>> > have
>>> > something for you to review soon.  From what I can tell, Cython has
>>> > some
>>> > baked-in assumptions when it comes to assignment statements.  Telling
>>> > Cython
>>> > that in this case, no, don't assign to something at the C level,
>>> > instead
>>> > generate a function call and pass in the lvalue by reference takes some
>>> > doing.
>>>
>>> Yeah, I'm not too surprised about that.
>>>
>>
>> I have an initial version working for assigning a Python object to a
>> fixed-size array.
>
>
> https://github.com/cython/cython/pull/308
>
> Do you prefer that we continue discussion on this thread, or on the PR?
>
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> https://mail.python.org/mailman/listinfo/cython-devel
>

From ian.h.bell at gmail.com  Sat Aug  2 12:56:46 2014
From: ian.h.bell at gmail.com (Ian Bell)
Date: Sat, 2 Aug 2014 12:56:46 +0200
Subject: [Cython] Test/example for cpdef enum
Message-ID: <CAJQnXJf6HbKOb9mvZ1OASH6zfx+BuJfiXp=zT9+YLc8RoKuMZg@mail.gmail.com>

Are there any tests/docs/examples showing how to use the cpdef enum?  This
is a feature I have been looking for for a long time and I am hoping to use
it very soon in my own project.

I have a windows box that I can run the alpha test suite on with the full
complement of python versions.  Let me know if there is interest and I can
run the tests.  Alternatively, if you are still doing a buildbot (the link
at https://github.com/cython/cython/wiki/HackerGuide#buildbot is broken) ,
we could configure a nightly slave to run the tests.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20140802/7cd1025c/attachment.html>

From robertwb at gmail.com  Sat Aug  2 19:08:55 2014
From: robertwb at gmail.com (Robert Bradshaw)
Date: Sat, 2 Aug 2014 10:08:55 -0700
Subject: [Cython] Test/example for cpdef enum
In-Reply-To: <CAJQnXJf6HbKOb9mvZ1OASH6zfx+BuJfiXp=zT9+YLc8RoKuMZg@mail.gmail.com>
References: <CAJQnXJf6HbKOb9mvZ1OASH6zfx+BuJfiXp=zT9+YLc8RoKuMZg@mail.gmail.com>
Message-ID: <CADiQ+QCzQB2-UZNbN7mAdq0M7PcOwFb1CQn6KTYNvGZ6fFvwhA@mail.gmail.com>

On Sat, Aug 2, 2014 at 3:56 AM, Ian Bell <ian.h.bell at gmail.com> wrote:
> Are there any tests/docs/examples showing how to use the cpdef enum?  This
> is a feature I have been looking for for a long time and I am hoping to use
> it very soon in my own project.

https://github.com/cython/cython/blob/master/tests/run/cpdef_enums.pyx
https://github.com/cython/cython/blob/master/tests/run/cpdef_enums.pxd

> I have a windows box that I can run the alpha test suite on with the full
> complement of python versions.  Let me know if there is interest and I can
> run the tests.  Alternatively, if you are still doing a buildbot (the link
> at https://github.com/cython/cython/wiki/HackerGuide#buildbot is broken) ,
> we could configure a nightly slave to run the tests.

That would be nice; windows is woefully undertested for Cython.

- Robert

From matthew.brett at gmail.com  Sat Aug  2 19:42:07 2014
From: matthew.brett at gmail.com (Matthew Brett)
Date: Sat, 02 Aug 2014 10:42:07 -0700
Subject: [Cython] Test/example for cpdef enum
References: <CADiQ+QCzQB2-UZNbN7mAdq0M7PcOwFb1CQn6KTYNvGZ6fFvwhA@mail.gmail.com>
Message-ID: <53dd22ef22972_48b93fd8b385949c98953@shanghai.local.mail>

On Sat, Aug 02, 2014 at 10:08 AM, Robert Bradshaw <robertwb at gmail.com> wrote:

> from: Robert Bradshaw <robertwb at gmail.com>
> date: Sat, Aug 02 10:08 AM -07:00 2014
> to: Core developer mailing list of the Cython compiler <cython-devel at python.org>
> reply-to: Core developer mailing list of the Cython compiler <cython-devel at python.org>
> subject: Re: [Cython] Test/example for cpdef enum
> 
> On Sat, Aug 2, 2014 at 3:56 AM, Ian Bell <ian.h.bell at gmail.com> wrote:
>> Are there any tests/docs/examples showing how to use the cpdef enum?  This
>> is a feature I have been looking for for a long time and I am hoping to use
>> it very soon in my own project.
> 
> https://github.com/cython/cython/blob/master/tests/run/cpdef_enums.pyx
> https://github.com/cython/cython/blob/master/tests/run/cpdef_enums.pxd
> 
>> I have a windows box that I can run the alpha test suite on with the full
>> complement of python versions.  Let me know if there is interest and I can
>> run the tests.  Alternatively, if you are still doing a buildbot (the link
>> at https://github.com/cython/cython/wiki/HackerGuide#buildbot is broken) ,
>> we could configure a nightly slave to run the tests.
> 
> That would be nice; windows is woefully undertested for Cython.

Another big 'that would be nice' from me.

We (the berkeley / nipy team) have a buildbot set up you could hook into
if you need one.  The configuration is all on github at
https://github.com/nipy/nibotmi , and you are welcome to access to the
build master to play with configuration if you'd like.

Cheers,

Matthew

--
Sent from Vmail

From ian.h.bell at gmail.com  Tue Aug  5 01:11:52 2014
From: ian.h.bell at gmail.com (Ian Bell)
Date: Tue, 5 Aug 2014 01:11:52 +0200
Subject: [Cython] Test/example for cpdef enum
In-Reply-To: <53dd22ef22972_48b93fd8b385949c98953@shanghai.local.mail>
References: <CADiQ+QCzQB2-UZNbN7mAdq0M7PcOwFb1CQn6KTYNvGZ6fFvwhA@mail.gmail.com>
 <53dd22ef22972_48b93fd8b385949c98953@shanghai.local.mail>
Message-ID: <CAJQnXJf1NjNvXstE1Njrd=TEzhVhDY4y1rszdTpyKKuMZXRSmQ@mail.gmail.com>

On Sat, Aug 2, 2014 at 7:42 PM, Matthew Brett <matthew.brett at gmail.com>
wrote:

> On Sat, Aug 02, 2014 at 10:08 AM, Robert Bradshaw <robertwb at gmail.com>
> wrote:
>
> > from: Robert Bradshaw <robertwb at gmail.com>
> > date: Sat, Aug 02 10:08 AM -07:00 2014
> > to: Core developer mailing list of the Cython compiler <
> cython-devel at python.org>
> > reply-to: Core developer mailing list of the Cython compiler <
> cython-devel at python.org>
> > subject: Re: [Cython] Test/example for cpdef enum
> >
> > On Sat, Aug 2, 2014 at 3:56 AM, Ian Bell <ian.h.bell at gmail.com> wrote:
> >> Are there any tests/docs/examples showing how to use the cpdef enum?
>  This
> >> is a feature I have been looking for for a long time and I am hoping to
> use
> >> it very soon in my own project.
> >
> > https://github.com/cython/cython/blob/master/tests/run/cpdef_enums.pyx
> > https://github.com/cython/cython/blob/master/tests/run/cpdef_enums.pxd
> >
> >> I have a windows box that I can run the alpha test suite on with the
> full
> >> complement of python versions.  Let me know if there is interest and I
> can
> >> run the tests.  Alternatively, if you are still doing a buildbot (the
> link
> >> at https://github.com/cython/cython/wiki/HackerGuide#buildbot is
> broken) ,
> >> we could configure a nightly slave to run the tests.
> >
> > That would be nice; windows is woefully undertested for Cython.
>
> Another big 'that would be nice' from me.
>
> We (the berkeley / nipy team) have a buildbot set up you could hook into
> if you need one.  The configuration is all on github at
> https://github.com/nipy/nibotmi , and you are welcome to access to the
> build master to play with configuration if you'd like.
>
> Can you set me up a nightly build (once per 24 hours) and we can then
iterate on getting things functional on my box to the level that is
needed.  I use anaconda for instance which might cause issues - I briefly
saw you use virtualenv.

Ian


> Cheers,
>
> Matthew
>
> --
> Sent from Vmail
>
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> https://mail.python.org/mailman/listinfo/cython-devel
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20140805/30e429dd/attachment.html>

From ian.h.bell at gmail.com  Tue Aug  5 01:14:03 2014
From: ian.h.bell at gmail.com (Ian Bell)
Date: Tue, 5 Aug 2014 01:14:03 +0200
Subject: [Cython] Test/example for cpdef enum
In-Reply-To: <CADiQ+QCzQB2-UZNbN7mAdq0M7PcOwFb1CQn6KTYNvGZ6fFvwhA@mail.gmail.com>
References: <CAJQnXJf6HbKOb9mvZ1OASH6zfx+BuJfiXp=zT9+YLc8RoKuMZg@mail.gmail.com>
 <CADiQ+QCzQB2-UZNbN7mAdq0M7PcOwFb1CQn6KTYNvGZ6fFvwhA@mail.gmail.com>
Message-ID: <CAJQnXJfcmVb0KscnZPdu6VR+xktiJVHygnnwe5_PT_WbfY+-sw@mail.gmail.com>

On Sat, Aug 2, 2014 at 7:08 PM, Robert Bradshaw <robertwb at gmail.com> wrote:

> On Sat, Aug 2, 2014 at 3:56 AM, Ian Bell <ian.h.bell at gmail.com> wrote:
> > Are there any tests/docs/examples showing how to use the cpdef enum?
>  This
> > is a feature I have been looking for for a long time and I am hoping to
> use
> > it very soon in my own project.
>
> https://github.com/cython/cython/blob/master/tests/run/cpdef_enums.pyx
> https://github.com/cython/cython/blob/master/tests/run/cpdef_enums.pxd
>

I see.  Stupidly I had thought that it would be automatic in the sense that
you don't have to manually declare which parameters will be output in the
python module.  Evidently that is not the case, which I guess is ok.  I had
hoped to avoid my script which parses my header and make a constants
module.  It works fine, its not the prettiest, but I guess I won't be
saying goodbye to it any time soon.


>
> > I have a windows box that I can run the alpha test suite on with the full
> > complement of python versions.  Let me know if there is interest and I
> can
> > run the tests.  Alternatively, if you are still doing a buildbot (the
> link
> > at https://github.com/cython/cython/wiki/HackerGuide#buildbot is
> broken) ,
> > we could configure a nightly slave to run the tests.
>
> That would be nice; windows is woefully undertested for Cython.
>
> - Robert
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> https://mail.python.org/mailman/listinfo/cython-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20140805/15aae4ee/attachment.html>

From matthew.brett at gmail.com  Tue Aug  5 01:16:50 2014
From: matthew.brett at gmail.com (Matthew Brett)
Date: Mon, 4 Aug 2014 16:16:50 -0700
Subject: [Cython] Test/example for cpdef enum
In-Reply-To: <CAJQnXJf1NjNvXstE1Njrd=TEzhVhDY4y1rszdTpyKKuMZXRSmQ@mail.gmail.com>
References: <CADiQ+QCzQB2-UZNbN7mAdq0M7PcOwFb1CQn6KTYNvGZ6fFvwhA@mail.gmail.com>
 <53dd22ef22972_48b93fd8b385949c98953@shanghai.local.mail>
 <CAJQnXJf1NjNvXstE1Njrd=TEzhVhDY4y1rszdTpyKKuMZXRSmQ@mail.gmail.com>
Message-ID: <CAH6Pt5oXtZYUtt1rL7HWN7=v83vCanQ774-FToO=uLmvOpoEpA@mail.gmail.com>

Hi,

On Mon, Aug 4, 2014 at 4:11 PM, Ian Bell <ian.h.bell at gmail.com> wrote:
>
>
>
> On Sat, Aug 2, 2014 at 7:42 PM, Matthew Brett <matthew.brett at gmail.com>
> wrote:
>>
>> On Sat, Aug 02, 2014 at 10:08 AM, Robert Bradshaw <robertwb at gmail.com>
>> wrote:
>>
>> > from: Robert Bradshaw <robertwb at gmail.com>
>> > date: Sat, Aug 02 10:08 AM -07:00 2014
>> > to: Core developer mailing list of the Cython compiler
>> > <cython-devel at python.org>
>> > reply-to: Core developer mailing list of the Cython compiler
>> > <cython-devel at python.org>
>> > subject: Re: [Cython] Test/example for cpdef enum
>> >
>> > On Sat, Aug 2, 2014 at 3:56 AM, Ian Bell <ian.h.bell at gmail.com> wrote:
>> >> Are there any tests/docs/examples showing how to use the cpdef enum?
>> >> This
>> >> is a feature I have been looking for for a long time and I am hoping to
>> >> use
>> >> it very soon in my own project.
>> >
>> > https://github.com/cython/cython/blob/master/tests/run/cpdef_enums.pyx
>> > https://github.com/cython/cython/blob/master/tests/run/cpdef_enums.pxd
>> >
>> >> I have a windows box that I can run the alpha test suite on with the
>> >> full
>> >> complement of python versions.  Let me know if there is interest and I
>> >> can
>> >> run the tests.  Alternatively, if you are still doing a buildbot (the
>> >> link
>> >> at https://github.com/cython/cython/wiki/HackerGuide#buildbot is
>> >> broken) ,
>> >> we could configure a nightly slave to run the tests.
>> >
>> > That would be nice; windows is woefully undertested for Cython.
>>
>> Another big 'that would be nice' from me.
>>
>> We (the berkeley / nipy team) have a buildbot set up you could hook into
>> if you need one.  The configuration is all on github at
>> https://github.com/nipy/nibotmi , and you are welcome to access to the
>> build master to play with configuration if you'd like.
>>
> Can you set me up a nightly build (once per 24 hours) and we can then
> iterate on getting things functional on my box to the level that is needed.
> I use anaconda for instance which might cause issues - I briefly saw you use
> virtualenv.

Yes, the classes I wrote to make making a build factor a little more
automated use virtualenvs; if you don't want to use those classes, you
can use the bare-metal buildbot build steps with Ananconda on the
slave.  But - do you need Anaconda to set up a Cython build test?
Maybe you can use virtualenv with the Anaconda Python?

Cheers,

Matthew

From ian.h.bell at gmail.com  Tue Aug  5 01:19:53 2014
From: ian.h.bell at gmail.com (Ian Bell)
Date: Tue, 5 Aug 2014 01:19:53 +0200
Subject: [Cython] Test/example for cpdef enum
In-Reply-To: <CAH6Pt5oXtZYUtt1rL7HWN7=v83vCanQ774-FToO=uLmvOpoEpA@mail.gmail.com>
References: <CADiQ+QCzQB2-UZNbN7mAdq0M7PcOwFb1CQn6KTYNvGZ6fFvwhA@mail.gmail.com>
 <53dd22ef22972_48b93fd8b385949c98953@shanghai.local.mail>
 <CAJQnXJf1NjNvXstE1Njrd=TEzhVhDY4y1rszdTpyKKuMZXRSmQ@mail.gmail.com>
 <CAH6Pt5oXtZYUtt1rL7HWN7=v83vCanQ774-FToO=uLmvOpoEpA@mail.gmail.com>
Message-ID: <CAJQnXJdcfQ5GbaUY6uWpCQn=40YqU3QN1dJ=c_6fMf=3DnX=Rg@mail.gmail.com>

On Tue, Aug 5, 2014 at 1:16 AM, Matthew Brett <matthew.brett at gmail.com>
wrote:

> Hi,
>
> On Mon, Aug 4, 2014 at 4:11 PM, Ian Bell <ian.h.bell at gmail.com> wrote:
> >
> >
> >
> > On Sat, Aug 2, 2014 at 7:42 PM, Matthew Brett <matthew.brett at gmail.com>
> > wrote:
> >>
> >> On Sat, Aug 02, 2014 at 10:08 AM, Robert Bradshaw <robertwb at gmail.com>
> >> wrote:
> >>
> >> > from: Robert Bradshaw <robertwb at gmail.com>
> >> > date: Sat, Aug 02 10:08 AM -07:00 2014
> >> > to: Core developer mailing list of the Cython compiler
> >> > <cython-devel at python.org>
> >> > reply-to: Core developer mailing list of the Cython compiler
> >> > <cython-devel at python.org>
> >> > subject: Re: [Cython] Test/example for cpdef enum
> >> >
> >> > On Sat, Aug 2, 2014 at 3:56 AM, Ian Bell <ian.h.bell at gmail.com>
> wrote:
> >> >> Are there any tests/docs/examples showing how to use the cpdef enum?
> >> >> This
> >> >> is a feature I have been looking for for a long time and I am hoping
> to
> >> >> use
> >> >> it very soon in my own project.
> >> >
> >> >
> https://github.com/cython/cython/blob/master/tests/run/cpdef_enums.pyx
> >> >
> https://github.com/cython/cython/blob/master/tests/run/cpdef_enums.pxd
> >> >
> >> >> I have a windows box that I can run the alpha test suite on with the
> >> >> full
> >> >> complement of python versions.  Let me know if there is interest and
> I
> >> >> can
> >> >> run the tests.  Alternatively, if you are still doing a buildbot (the
> >> >> link
> >> >> at https://github.com/cython/cython/wiki/HackerGuide#buildbot is
> >> >> broken) ,
> >> >> we could configure a nightly slave to run the tests.
> >> >
> >> > That would be nice; windows is woefully undertested for Cython.
> >>
> >> Another big 'that would be nice' from me.
> >>
> >> We (the berkeley / nipy team) have a buildbot set up you could hook into
> >> if you need one.  The configuration is all on github at
> >> https://github.com/nipy/nibotmi , and you are welcome to access to the
> >> build master to play with configuration if you'd like.
> >>
> > Can you set me up a nightly build (once per 24 hours) and we can then
> > iterate on getting things functional on my box to the level that is
> needed.
> > I use anaconda for instance which might cause issues - I briefly saw you
> use
> > virtualenv.
>
> Yes, the classes I wrote to make making a build factor a little more
> automated use virtualenvs; if you don't want to use those classes, you
> can use the bare-metal buildbot build steps with Ananconda on the
> slave.  But - do you need Anaconda to set up a Cython build test?
> Maybe you can use virtualenv with the Anaconda Python?
>
> So far as I can tell, anaconda and virtualenv don't play too well
together.  Which is too bad, because I recall having some issues with
something (buildbot?) not running happily in an anaconda env


> Cheers,
>
> Matthew
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> https://mail.python.org/mailman/listinfo/cython-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20140805/eb447f5e/attachment-0001.html>

From matthew.brett at gmail.com  Tue Aug  5 01:28:58 2014
From: matthew.brett at gmail.com (Matthew Brett)
Date: Mon, 4 Aug 2014 16:28:58 -0700
Subject: [Cython] Test/example for cpdef enum
In-Reply-To: <CAJQnXJdcfQ5GbaUY6uWpCQn=40YqU3QN1dJ=c_6fMf=3DnX=Rg@mail.gmail.com>
References: <CADiQ+QCzQB2-UZNbN7mAdq0M7PcOwFb1CQn6KTYNvGZ6fFvwhA@mail.gmail.com>
 <53dd22ef22972_48b93fd8b385949c98953@shanghai.local.mail>
 <CAJQnXJf1NjNvXstE1Njrd=TEzhVhDY4y1rszdTpyKKuMZXRSmQ@mail.gmail.com>
 <CAH6Pt5oXtZYUtt1rL7HWN7=v83vCanQ774-FToO=uLmvOpoEpA@mail.gmail.com>
 <CAJQnXJdcfQ5GbaUY6uWpCQn=40YqU3QN1dJ=c_6fMf=3DnX=Rg@mail.gmail.com>
Message-ID: <CAH6Pt5pbBHBPWVcrFF6qt=5NMHSSnMj+45khrhih0OQc_aGjAw@mail.gmail.com>

On Mon, Aug 4, 2014 at 4:19 PM, Ian Bell <ian.h.bell at gmail.com> wrote:
>
>
>
> On Tue, Aug 5, 2014 at 1:16 AM, Matthew Brett <matthew.brett at gmail.com>
> wrote:
>>
>> Hi,
>>
>> On Mon, Aug 4, 2014 at 4:11 PM, Ian Bell <ian.h.bell at gmail.com> wrote:
>> >
>> >
>> >
>> > On Sat, Aug 2, 2014 at 7:42 PM, Matthew Brett <matthew.brett at gmail.com>
>> > wrote:
>> >>
>> >> On Sat, Aug 02, 2014 at 10:08 AM, Robert Bradshaw <robertwb at gmail.com>
>> >> wrote:
>> >>
>> >> > from: Robert Bradshaw <robertwb at gmail.com>
>> >> > date: Sat, Aug 02 10:08 AM -07:00 2014
>> >> > to: Core developer mailing list of the Cython compiler
>> >> > <cython-devel at python.org>
>> >> > reply-to: Core developer mailing list of the Cython compiler
>> >> > <cython-devel at python.org>
>> >> > subject: Re: [Cython] Test/example for cpdef enum
>> >> >
>> >> > On Sat, Aug 2, 2014 at 3:56 AM, Ian Bell <ian.h.bell at gmail.com>
>> >> > wrote:
>> >> >> Are there any tests/docs/examples showing how to use the cpdef enum?
>> >> >> This
>> >> >> is a feature I have been looking for for a long time and I am hoping
>> >> >> to
>> >> >> use
>> >> >> it very soon in my own project.
>> >> >
>> >> >
>> >> > https://github.com/cython/cython/blob/master/tests/run/cpdef_enums.pyx
>> >> >
>> >> > https://github.com/cython/cython/blob/master/tests/run/cpdef_enums.pxd
>> >> >
>> >> >> I have a windows box that I can run the alpha test suite on with the
>> >> >> full
>> >> >> complement of python versions.  Let me know if there is interest and
>> >> >> I
>> >> >> can
>> >> >> run the tests.  Alternatively, if you are still doing a buildbot
>> >> >> (the
>> >> >> link
>> >> >> at https://github.com/cython/cython/wiki/HackerGuide#buildbot is
>> >> >> broken) ,
>> >> >> we could configure a nightly slave to run the tests.
>> >> >
>> >> > That would be nice; windows is woefully undertested for Cython.
>> >>
>> >> Another big 'that would be nice' from me.
>> >>
>> >> We (the berkeley / nipy team) have a buildbot set up you could hook
>> >> into
>> >> if you need one.  The configuration is all on github at
>> >> https://github.com/nipy/nibotmi , and you are welcome to access to the
>> >> build master to play with configuration if you'd like.
>> >>
>> > Can you set me up a nightly build (once per 24 hours) and we can then
>> > iterate on getting things functional on my box to the level that is
>> > needed.
>> > I use anaconda for instance which might cause issues - I briefly saw you
>> > use
>> > virtualenv.
>>
>> Yes, the classes I wrote to make making a build factor a little more
>> automated use virtualenvs; if you don't want to use those classes, you
>> can use the bare-metal buildbot build steps with Ananconda on the
>> slave.  But - do you need Anaconda to set up a Cython build test?
>> Maybe you can use virtualenv with the Anaconda Python?
>>
> So far as I can tell, anaconda and virtualenv don't play too well together.
> Which is too bad, because I recall having some issues with something
> (buildbot?) not running happily in an anaconda env

Maybe it's worth giving it a shot and see if problems arise?

Am I right in thinking you can install Python.org Python alongside
Anaconda?   Maybe you could do that?

Cheers,

Matthew

From ian.h.bell at gmail.com  Tue Aug  5 01:41:45 2014
From: ian.h.bell at gmail.com (Ian Bell)
Date: Tue, 5 Aug 2014 01:41:45 +0200
Subject: [Cython] Test/example for cpdef enum
In-Reply-To: <CAH6Pt5pbBHBPWVcrFF6qt=5NMHSSnMj+45khrhih0OQc_aGjAw@mail.gmail.com>
References: <CADiQ+QCzQB2-UZNbN7mAdq0M7PcOwFb1CQn6KTYNvGZ6fFvwhA@mail.gmail.com>
 <53dd22ef22972_48b93fd8b385949c98953@shanghai.local.mail>
 <CAJQnXJf1NjNvXstE1Njrd=TEzhVhDY4y1rszdTpyKKuMZXRSmQ@mail.gmail.com>
 <CAH6Pt5oXtZYUtt1rL7HWN7=v83vCanQ774-FToO=uLmvOpoEpA@mail.gmail.com>
 <CAJQnXJdcfQ5GbaUY6uWpCQn=40YqU3QN1dJ=c_6fMf=3DnX=Rg@mail.gmail.com>
 <CAH6Pt5pbBHBPWVcrFF6qt=5NMHSSnMj+45khrhih0OQc_aGjAw@mail.gmail.com>
Message-ID: <CAJQnXJd1KPpOFjx7Vf1K3svQBDjNknX8DioAq9OS=aMFrXwcjg@mail.gmail.com>

On Tue, Aug 5, 2014 at 1:28 AM, Matthew Brett <matthew.brett at gmail.com>
wrote:

> On Mon, Aug 4, 2014 at 4:19 PM, Ian Bell <ian.h.bell at gmail.com> wrote:
> >
> >
> >
> > On Tue, Aug 5, 2014 at 1:16 AM, Matthew Brett <matthew.brett at gmail.com>
> > wrote:
> >>
> >> Hi,
> >>
> >> On Mon, Aug 4, 2014 at 4:11 PM, Ian Bell <ian.h.bell at gmail.com> wrote:
> >> >
> >> >
> >> >
> >> > On Sat, Aug 2, 2014 at 7:42 PM, Matthew Brett <
> matthew.brett at gmail.com>
> >> > wrote:
> >> >>
> >> >> On Sat, Aug 02, 2014 at 10:08 AM, Robert Bradshaw <
> robertwb at gmail.com>
> >> >> wrote:
> >> >>
> >> >> > from: Robert Bradshaw <robertwb at gmail.com>
> >> >> > date: Sat, Aug 02 10:08 AM -07:00 2014
> >> >> > to: Core developer mailing list of the Cython compiler
> >> >> > <cython-devel at python.org>
> >> >> > reply-to: Core developer mailing list of the Cython compiler
> >> >> > <cython-devel at python.org>
> >> >> > subject: Re: [Cython] Test/example for cpdef enum
> >> >> >
> >> >> > On Sat, Aug 2, 2014 at 3:56 AM, Ian Bell <ian.h.bell at gmail.com>
> >> >> > wrote:
> >> >> >> Are there any tests/docs/examples showing how to use the cpdef
> enum?
> >> >> >> This
> >> >> >> is a feature I have been looking for for a long time and I am
> hoping
> >> >> >> to
> >> >> >> use
> >> >> >> it very soon in my own project.
> >> >> >
> >> >> >
> >> >> >
> https://github.com/cython/cython/blob/master/tests/run/cpdef_enums.pyx
> >> >> >
> >> >> >
> https://github.com/cython/cython/blob/master/tests/run/cpdef_enums.pxd
> >> >> >
> >> >> >> I have a windows box that I can run the alpha test suite on with
> the
> >> >> >> full
> >> >> >> complement of python versions.  Let me know if there is interest
> and
> >> >> >> I
> >> >> >> can
> >> >> >> run the tests.  Alternatively, if you are still doing a buildbot
> >> >> >> (the
> >> >> >> link
> >> >> >> at https://github.com/cython/cython/wiki/HackerGuide#buildbot is
> >> >> >> broken) ,
> >> >> >> we could configure a nightly slave to run the tests.
> >> >> >
> >> >> > That would be nice; windows is woefully undertested for Cython.
> >> >>
> >> >> Another big 'that would be nice' from me.
> >> >>
> >> >> We (the berkeley / nipy team) have a buildbot set up you could hook
> >> >> into
> >> >> if you need one.  The configuration is all on github at
> >> >> https://github.com/nipy/nibotmi , and you are welcome to access to
> the
> >> >> build master to play with configuration if you'd like.
> >> >>
> >> > Can you set me up a nightly build (once per 24 hours) and we can then
> >> > iterate on getting things functional on my box to the level that is
> >> > needed.
> >> > I use anaconda for instance which might cause issues - I briefly saw
> you
> >> > use
> >> > virtualenv.
> >>
> >> Yes, the classes I wrote to make making a build factor a little more
> >> automated use virtualenvs; if you don't want to use those classes, you
> >> can use the bare-metal buildbot build steps with Ananconda on the
> >> slave.  But - do you need Anaconda to set up a Cython build test?
> >> Maybe you can use virtualenv with the Anaconda Python?
> >>
> > So far as I can tell, anaconda and virtualenv don't play too well
> together.
> > Which is too bad, because I recall having some issues with something
> > (buildbot?) not running happily in an anaconda env
>
> Maybe it's worth giving it a shot and see if problems arise?
>
> Am I right in thinking you can install Python.org Python alongside
> Anaconda?   Maybe you could do that?
>
> That's right - my current config has two minicondas (32/64 bit) defaulting
to py27 along with python.org 27 32 bit which I need to keep around due to
the delicate interdependencies of wxpython, python, and cx_freeze for a
couple of my projects

Cheers,
>
> Matthew
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> https://mail.python.org/mailman/listinfo/cython-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20140805/8edd5fa7/attachment.html>

From robertwb at gmail.com  Tue Aug  5 01:48:43 2014
From: robertwb at gmail.com (Robert Bradshaw)
Date: Mon, 4 Aug 2014 16:48:43 -0700
Subject: [Cython] Test/example for cpdef enum
In-Reply-To: <CAJQnXJfcmVb0KscnZPdu6VR+xktiJVHygnnwe5_PT_WbfY+-sw@mail.gmail.com>
References: <CAJQnXJf6HbKOb9mvZ1OASH6zfx+BuJfiXp=zT9+YLc8RoKuMZg@mail.gmail.com>
 <CADiQ+QCzQB2-UZNbN7mAdq0M7PcOwFb1CQn6KTYNvGZ6fFvwhA@mail.gmail.com>
 <CAJQnXJfcmVb0KscnZPdu6VR+xktiJVHygnnwe5_PT_WbfY+-sw@mail.gmail.com>
Message-ID: <CADiQ+QCd4XKu=71+QW6vEiBoU5HeKoKKc-vNOP=H+A4FDihMPw@mail.gmail.com>

On Mon, Aug 4, 2014 at 4:14 PM, Ian Bell <ian.h.bell at gmail.com> wrote:
>
> On Sat, Aug 2, 2014 at 7:08 PM, Robert Bradshaw <robertwb at gmail.com> wrote:
>>
>> On Sat, Aug 2, 2014 at 3:56 AM, Ian Bell <ian.h.bell at gmail.com> wrote:
>> > Are there any tests/docs/examples showing how to use the cpdef enum?
>> > This
>> > is a feature I have been looking for for a long time and I am hoping to
>> > use
>> > it very soon in my own project.
>>
>> https://github.com/cython/cython/blob/master/tests/run/cpdef_enums.pyx
>> https://github.com/cython/cython/blob/master/tests/run/cpdef_enums.pxd
>
>
> I see.  Stupidly I had thought that it would be automatic in the sense that
> you don't have to manually declare which parameters will be output in the
> python module.  Evidently that is not the case, which I guess is ok.  I had
> hoped to avoid my script which parses my header and make a constants module.
> It works fine, its not the prettiest, but I guess I won't be saying goodbye
> to it any time soon.

Yeah, we don't have anything that looks at the actual .h/.c files yet.

From ian.h.bell at gmail.com  Tue Aug  5 18:15:44 2014
From: ian.h.bell at gmail.com (Ian Bell)
Date: Tue, 5 Aug 2014 18:15:44 +0200
Subject: [Cython] Test/example for cpdef enum
In-Reply-To: <CADiQ+QCd4XKu=71+QW6vEiBoU5HeKoKKc-vNOP=H+A4FDihMPw@mail.gmail.com>
References: <CAJQnXJf6HbKOb9mvZ1OASH6zfx+BuJfiXp=zT9+YLc8RoKuMZg@mail.gmail.com>
 <CADiQ+QCzQB2-UZNbN7mAdq0M7PcOwFb1CQn6KTYNvGZ6fFvwhA@mail.gmail.com>
 <CAJQnXJfcmVb0KscnZPdu6VR+xktiJVHygnnwe5_PT_WbfY+-sw@mail.gmail.com>
 <CADiQ+QCd4XKu=71+QW6vEiBoU5HeKoKKc-vNOP=H+A4FDihMPw@mail.gmail.com>
Message-ID: <CAJQnXJe5s696GD6QyiqVmVJV-dMxPpF5Dcen6u=KS5wYwnSy1Q@mail.gmail.com>

On Tue, Aug 5, 2014 at 1:48 AM, Robert Bradshaw <robertwb at gmail.com> wrote:

> On Mon, Aug 4, 2014 at 4:14 PM, Ian Bell <ian.h.bell at gmail.com> wrote:
> >
> > On Sat, Aug 2, 2014 at 7:08 PM, Robert Bradshaw <robertwb at gmail.com>
> wrote:
> >>
> >> On Sat, Aug 2, 2014 at 3:56 AM, Ian Bell <ian.h.bell at gmail.com> wrote:
> >> > Are there any tests/docs/examples showing how to use the cpdef enum?
> >> > This
> >> > is a feature I have been looking for for a long time and I am hoping
> to
> >> > use
> >> > it very soon in my own project.
> >>
> >> https://github.com/cython/cython/blob/master/tests/run/cpdef_enums.pyx
> >> https://github.com/cython/cython/blob/master/tests/run/cpdef_enums.pxd
> >
> >
> > I see.  Stupidly I had thought that it would be automatic in the sense
> that
> > you don't have to manually declare which parameters will be output in the
> > python module.  Evidently that is not the case, which I guess is ok.  I
> had
> > hoped to avoid my script which parses my header and make a constants
> module.
> > It works fine, its not the prettiest, but I guess I won't be saying
> goodbye
> > to it any time soon.
>
> Yeah, we don't have anything that looks at the actual .h/.c files yet.
>
Too bad - I guess that's why projects like xdress/SWIG exist - though for
the life of me I can't get xdress to work at all.  SWIG is great, but not
so much from the standpoint of interfacing with other cython code.

> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> https://mail.python.org/mailman/listinfo/cython-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20140805/a1823415/attachment.html>

From insertinterestingnamehere at gmail.com  Mon Aug  4 21:11:13 2014
From: insertinterestingnamehere at gmail.com (Ian Henriksen)
Date: Mon, 4 Aug 2014 13:11:13 -0600
Subject: [Cython] aritmetic with arrays in Cython
Message-ID: <CAPWbfpSGTwxiVY2TUd_juEHJ4a_CCukvTS793RSFmBHWNfxyRg@mail.gmail.com>

Hi all,
I noticed that some time ago there was a pull request (
https://github.com/cython/cython/pull/144) open that was trying to
implement basic arithmetic operations with arrays. This seems to have also
been proposed in CEP 518 (
https://github.com/cython/cython/wiki/enhancements-simd). What is the
status on this?

This is a feature I'd really like to see in Cython. I'd be happy to help
with it, though I don't currently know enough about the internals of Cython
to do it without some guidance. What would be the best way to do this? I've
had three ideas thus far that might allow something like this to work, but
I'd really like to get some second opinions before I get too far with any
of them.

1. Make these array operations available as a part of Cython's memory view
objects.

2. Write a Cython wrapper around a NumPy-like C++ library.
https://github.com/ContinuumIO/libdynd seems to be a good candidate, though
I'm not as familiar with it. (This doesn't strike me as something that
would be a part of the main Cython project, but it could be a good way to
make this sort of functionality available. It seems like a good way to
avoid a lot of duplicate work though.)

3. Write a Fortran backend for Cython that uses Fortran's arrays wherever
memory views are used.
(This could be a bit crazy, but with the iso_c_binding module, it might be
possible. It could also allow better optimizations for arrays to be applied
by the compiler.)

What would be the best way to go about this?

Thanks!

-Ian Henriksen
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20140804/0e634981/attachment.html>

From jtaylor.debian at googlemail.com  Tue Aug  5 21:38:38 2014
From: jtaylor.debian at googlemail.com (Julian Taylor)
Date: Tue, 05 Aug 2014 21:38:38 +0200
Subject: [Cython] aritmetic with arrays in Cython
In-Reply-To: <CAPWbfpSGTwxiVY2TUd_juEHJ4a_CCukvTS793RSFmBHWNfxyRg@mail.gmail.com>
References: <CAPWbfpSGTwxiVY2TUd_juEHJ4a_CCukvTS793RSFmBHWNfxyRg@mail.gmail.com>
Message-ID: <53E132BE.3070207@googlemail.com>

On 04.08.2014 21:11, Ian Henriksen wrote:
> Hi all,
> I noticed that some time ago there was a pull request
> (https://github.com/cython/cython/pull/144) open that was trying to
> implement basic arithmetic operations with arrays. This seems to have
> also been proposed in CEP 518
> (https://github.com/cython/cython/wiki/enhancements-simd). What is the
> status on this?
> 
> This is a feature I'd really like to see in Cython. I'd be happy to help
> with it, though I don't currently know enough about the internals of
> Cython to do it without some guidance. What would be the best way to do
> this? I've had three ideas thus far that might allow something like this
> to work, but I'd really like to get some second opinions before I get
> too far with any of them.
> 
> 1. Make these array operations available as a part of Cython's memory
> view objects.
> 
> 2. Write a Cython wrapper around a NumPy-like C++
> library. https://github.com/ContinuumIO/libdynd seems to be a good
> candidate, though I'm not as familiar with it. (This doesn't strike me
> as something that would be a part of the main Cython project, but it
> could be a good way to make this sort of functionality available. It
> seems like a good way to avoid a lot of duplicate work though.)
> 
> 3. Write a Fortran backend for Cython that uses Fortran's arrays
> wherever memory views are used.
> (This could be a bit crazy, but with the iso_c_binding module, it might
> be possible. It could also allow better optimizations for arrays to be
> applied by the compiler.)
> 
> What would be the best way to go about this?
> 

4. Use numpy itself. numpy since 1.8 has vectorized base math functions,
though for large arrays you have to apply manual blocking and inplace
operations to see significant gains. That is where cython can help.
Possibly numpy is also open to expose the raw data buffer functions in
its api to make blocking more efficient.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20140805/6cf7f05d/attachment.sig>

From insertinterestingnamehere at gmail.com  Thu Aug  7 01:30:21 2014
From: insertinterestingnamehere at gmail.com (Ian Henriksen)
Date: Wed, 6 Aug 2014 17:30:21 -0600
Subject: [Cython] aritmetic with arrays in Cython
Message-ID: <CAPWbfpTX8zwn1iSxvSciJz6bVrNadhfiDASsaxwcA_mat6vc-A@mail.gmail.com>

Message: 3
Date: Tue, 05 Aug 2014 21:38:38 +0200
From: Julian Taylor <jtaylor.debian at googlemail.com>
To: Core developer mailing list of the Cython compiler
        <cython-devel at python.org>
Subject: Re: [Cython] aritmetic with arrays in Cython
Message-ID: <53E132BE.3070207 at googlemail.com>
Content-Type: text/plain; charset="windows-1252"

>On 04.08.2014 21:11, Ian Henriksen wrote:
>> Hi all,
>> I noticed that some time ago there was a pull request
>> (https://github.com/cython/cython/pull/144) open that was trying to
>> implement basic arithmetic operations with arrays. This seems to have
>> also been proposed in CEP 518
>> (https://github.com/cython/cython/wiki/enhancements-simd). What is the
>> status on this?
>>
>> This is a feature I'd really like to see in Cython. I'd be happy to help
>> with it, though I don't currently know enough about the internals of
>> Cython to do it without some guidance. What would be the best way to do
>> this? I've had three ideas thus far that might allow something like this
>> to work, but I'd really like to get some second opinions before I get
>> too far with any of them.
>>
>> 1. Make these array operations available as a part of Cython's memory
>> view objects.
>>
>> 2. Write a Cython wrapper around a NumPy-like C++
>> library. https://github.com/ContinuumIO/libdynd seems to be a good
>> candidate, though I'm not as familiar with it. (This doesn't strike me
>> as something that would be a part of the main Cython project, but it
>> could be a good way to make this sort of functionality available. It
>> seems like a good way to avoid a lot of duplicate work though.)
>>
>> 3. Write a Fortran backend for Cython that uses Fortran's arrays
>> wherever memory views are used.
>> (This could be a bit crazy, but with the iso_c_binding module, it might
>> be possible. It could also allow better optimizations for arrays to be
>> applied by the compiler.)
>>
>> What would be the best way to go about this?
>>
>
>4. Use numpy itself. numpy since 1.8 has vectorized base math functions,
>though for large arrays you have to apply manual blocking and inplace
>operations to see significant gains. That is where cython can help.
>Possibly numpy is also open to expose the raw data buffer functions in
>its api to make blocking more efficient.

Thank you for pointing that out. Using the machinery that is already built
in to numpy would be ideal. I hadn't realized that was a part of numpy's
API. If I understand correctly, you're referring to
https://github.com/numpy/numpy/blob/master/doc/neps/generalized-ufuncs.rst.
Is that the right portion of the API? I'll try and figure out how to use
it. Any working examples would be greatly appreciated. Other suggestions
are also still welcome
Thanks!
-Ian Henriksen
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20140806/e4c09031/attachment.html>

From matej at laitl.cz  Thu Aug  7 09:58:03 2014
From: matej at laitl.cz (=?utf-8?B?TWF0xJtq?= Laitl)
Date: Thu, 07 Aug 2014 09:58:03 +0200
Subject: [Cython] aritmetic with arrays in Cython
In-Reply-To: <CAPWbfpSGTwxiVY2TUd_juEHJ4a_CCukvTS793RSFmBHWNfxyRg@mail.gmail.com>
References: <CAPWbfpSGTwxiVY2TUd_juEHJ4a_CCukvTS793RSFmBHWNfxyRg@mail.gmail.com>
Message-ID: <22700580.GCF8m5KeTp@sleeky>

On Monday 04 of August 2014 13:11:13 Ian Henriksen wrote:
> Hi all,
> I noticed that some time ago there was a pull request (
> https://github.com/cython/cython/pull/144) open that was trying to
> implement basic arithmetic operations with arrays. This seems to have also
> been proposed in CEP 518 (
> https://github.com/cython/cython/wiki/enhancements-simd). What is the
> status on this?
> 
> This is a feature I'd really like to see in Cython. I'd be happy to help
> with it, though I don't currently know enough about the internals of Cython
> to do it without some guidance. What would be the best way to do this? I've
> had three ideas thus far that might allow something like this to work, but
> I'd really like to get some second opinions before I get too far with any
> of them.

Hi,
you may also check out https://github.com/strohel/Ceygen if it would suit your 
needs.

Cheers,
	Mat?j

From insertinterestingnamehere at gmail.com  Fri Aug  8 00:58:26 2014
From: insertinterestingnamehere at gmail.com (Ian Henriksen)
Date: Thu, 7 Aug 2014 16:58:26 -0600
Subject: [Cython] aritmetic with arrays in Cython
In-Reply-To: <22700580.GCF8m5KeTp@sleeky>
References: <CAPWbfpSGTwxiVY2TUd_juEHJ4a_CCukvTS793RSFmBHWNfxyRg@mail.gmail.com>
 <22700580.GCF8m5KeTp@sleeky>
Message-ID: <CAPWbfpSLSQZUw_CYYjXUm=Ma5kGGpm8yAP4kmyeRjaLprqfaSQ@mail.gmail.com>

Nice module! That seems to implement most of the basic functionality I've
seen lacking. Ideally, I'm still trying to find a way to be able to do
something syntactically like this:

cpdef whateverfunction(double[:] a, float[:] b):
    return a + a * b + .5

It would be ideal if that sort of thing were already a part of Cython. All
things considered though, most of what I'm looking for seems to be there.

Thanks!

-Ian Henriksen


On Thu, Aug 7, 2014 at 1:58 AM, Mat?j Laitl <matej at laitl.cz> wrote:

> On Monday 04 of August 2014 13:11:13 Ian Henriksen wrote:
> > Hi all,
> > I noticed that some time ago there was a pull request (
> > https://github.com/cython/cython/pull/144) open that was trying to
> > implement basic arithmetic operations with arrays. This seems to have
> also
> > been proposed in CEP 518 (
> > https://github.com/cython/cython/wiki/enhancements-simd). What is the
> > status on this?
> >
> > This is a feature I'd really like to see in Cython. I'd be happy to help
> > with it, though I don't currently know enough about the internals of
> Cython
> > to do it without some guidance. What would be the best way to do this?
> I've
> > had three ideas thus far that might allow something like this to work,
> but
> > I'd really like to get some second opinions before I get too far with any
> > of them.
>
> Hi,
> you may also check out https://github.com/strohel/Ceygen if it would suit
> your
> needs.
>
> Cheers,
>         Mat?j
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20140807/8b598b34/attachment.html>

From stefan_ml at behnel.de  Fri Aug  8 07:05:08 2014
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 08 Aug 2014 07:05:08 +0200
Subject: [Cython] aritmetic with arrays in Cython
In-Reply-To: <CAPWbfpSLSQZUw_CYYjXUm=Ma5kGGpm8yAP4kmyeRjaLprqfaSQ@mail.gmail.com>
References: <CAPWbfpSGTwxiVY2TUd_juEHJ4a_CCukvTS793RSFmBHWNfxyRg@mail.gmail.com>
 <22700580.GCF8m5KeTp@sleeky>
 <CAPWbfpSLSQZUw_CYYjXUm=Ma5kGGpm8yAP4kmyeRjaLprqfaSQ@mail.gmail.com>
Message-ID: <53E45A84.2070308@behnel.de>

Ian Henriksen schrieb am 08.08.2014 um 00:58:
> On Thu, Aug 7, 2014 at 1:58 AM, Mat?j Laitl wrote:
>> you may also check out https://github.com/strohel/Ceygen if it would suit
>> your needs.

Interesting project. Thanks for sharing.


> Ideally, I'm still trying to find a way to be able to do
> something syntactically like this:
> 
> cpdef whateverfunction(double[:] a, float[:] b):
>     return a + a * b + .5
> 
> It would be ideal if that sort of thing were already a part of Cython. All
> things considered though, most of what I'm looking for seems to be there.

Eigen can apparently do these things already:

http://eigen.tuxfamily.org/dox/TopicLazyEvaluation.html

It might not be all that much work to integrate this with Cython to enable
syntax support. Or have some kind of preprocessor, but I guess that's not
going to be simpler than have it in Cython as an optional AST transform.
Basically, whenever an arithmetic expression consists of a mixture of
memory views and (optionally) scalars, it could be mapped to the kind of
code that Ceygen uses, just at an arbitrarily complex expression length.

Specifically, this could be bound to the Matrix multiplication operator
(which Cython now supports anyway) and limited to C++ compilation mode
(which Eigen requires).

http://legacy.python.org/dev/peps/pep-0465/

Stefan


From matej at laitl.cz  Fri Aug  8 11:07:27 2014
From: matej at laitl.cz (=?utf-8?B?TWF0xJtq?= Laitl)
Date: Fri, 08 Aug 2014 11:07:27 +0200
Subject: [Cython] aritmetic with arrays in Cython
In-Reply-To: <53E45A84.2070308@behnel.de>
References: <CAPWbfpSGTwxiVY2TUd_juEHJ4a_CCukvTS793RSFmBHWNfxyRg@mail.gmail.com>
 <CAPWbfpSLSQZUw_CYYjXUm=Ma5kGGpm8yAP4kmyeRjaLprqfaSQ@mail.gmail.com>
 <53E45A84.2070308@behnel.de>
Message-ID: <5698943.UtWdqSqk1C@sleeky>

On Friday 08 of August 2014 07:05:08 Stefan Behnel wrote:
> Ian Henriksen schrieb am 08.08.2014 um 00:58:
> > On Thu, Aug 7, 2014 at 1:58 AM, Mat?j Laitl wrote:
> >> you may also check out https://github.com/strohel/Ceygen if it would suit
> >> your needs.
> 
> Interesting project. Thanks for sharing.
> 
> > Ideally, I'm still trying to find a way to be able to do
> > something syntactically like this:
> > 
> > cpdef whateverfunction(double[:] a, float[:] b):
> >     return a + a * b + .5
> > 
> > It would be ideal if that sort of thing were already a part of Cython. All
> > things considered though, most of what I'm looking for seems to be there.
> 
> Eigen can apparently do these things already:
> 
> http://eigen.tuxfamily.org/dox/TopicLazyEvaluation.html
> 
> It might not be all that much work to integrate this with Cython to enable
> syntax support. Or have some kind of preprocessor, but I guess that's not
> going to be simpler than have it in Cython as an optional AST transform.
> Basically, whenever an arithmetic expression consists of a mixture of
> memory views and (optionally) scalars, it could be mapped to the kind of
> code that Ceygen uses, just at an arbitrarily complex expression length.

Exactly! That was one of my ideas when I decided to create Ceygen, but I 
resorted to take the easier route and made uglier static API because I didn't 
(and still don't) have enough knowledge of Cython internals and Mark's "array 
expressions" still looked like they are going to be merged.

What you suggest above would be definitely possible IMO, and it could easily 
cherry-pick some work done on Ceygen, for example how memoryviews are wrapped 
as Eigen arrays: 
https://github.com/strohel/Ceygen/blob/master/ceygen/eigen_cpp.h

Unfortunately, I no longer work on the project (my master's thesis) that 
triggered work on Ceygen, and now I can invest only little maintenance work to 
it.

*However*, if such project is found worth to tacking by Cython devs, I can 
imagine in my wild dreams that there may be a student at my former university 
interested in doing it as their project. (this is rather optimistic statement, 
please don't get too excited yet)

Regards,
	Mat?j

From stefan_ml at behnel.de  Fri Aug  8 12:28:09 2014
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 08 Aug 2014 12:28:09 +0200
Subject: [Cython] aritmetic with arrays in Cython
In-Reply-To: <5698943.UtWdqSqk1C@sleeky>
References: <CAPWbfpSGTwxiVY2TUd_juEHJ4a_CCukvTS793RSFmBHWNfxyRg@mail.gmail.com>
 <CAPWbfpSLSQZUw_CYYjXUm=Ma5kGGpm8yAP4kmyeRjaLprqfaSQ@mail.gmail.com>
 <53E45A84.2070308@behnel.de> <5698943.UtWdqSqk1C@sleeky>
Message-ID: <53E4A639.4000507@behnel.de>

Mat?j Laitl schrieb am 08.08.2014 um 11:07:
> On Friday 08 of August 2014 07:05:08 Stefan Behnel wrote:
>> Eigen can apparently do these things already:
>>
>> http://eigen.tuxfamily.org/dox/TopicLazyEvaluation.html
>>
>> It might not be all that much work to integrate this with Cython to enable
>> syntax support. Or have some kind of preprocessor, but I guess that's not
>> going to be simpler than have it in Cython as an optional AST transform.
>> Basically, whenever an arithmetic expression consists of a mixture of
>> memory views and (optionally) scalars, it could be mapped to the kind of
>> code that Ceygen uses, just at an arbitrarily complex expression length.
> 
> Exactly! That was one of my ideas when I decided to create Ceygen, but I 
> resorted to take the easier route and made uglier static API because I didn't 
> (and still don't) have enough knowledge of Cython internals and Mark's "array 
> expressions" still looked like they are going to be merged.

There are a couple of points in favour of Eigen/Ceygen. It's a way simpler
approach than moving the entire vectorised code generator into Cython. And
it lets Eigen do the hard work, leaving only the interfacing part to
Cython. Basically, both tools can do what they're good at. We're always
happy that there's a C compiler cleaning up after us to which we can
delegate the low level optimisation and tuning stuff. The same would apply
to Eigen. It's a much more cythonic approach than what's implemented in the
pie-in-the-sky array expressions branch.

It requires a C++ compiler, but I think that's an acceptable dependency
these days in places where array computation performance matters. People
even accept a dependency on huge tool stacks like Numba/LLVM for the same
kind of tasks. An array expression syntax is just so much simpler than
writing dumb nested loops.


> Unfortunately, I no longer work on the project (my master's thesis) that 
> triggered work on Ceygen, and now I can invest only little maintenance work to 
> it.

I don't see me being the one to implement this, either. :)


> *However*, if such project is found worth to tacking by Cython devs, I can 
> imagine in my wild dreams that there may be a student at my former university 
> interested in doing it as their project. (this is rather optimistic statement, 
> please don't get too excited yet)

I'm sure there are many people who would benefit from this. Some of them
might even be able to provide funding, if someone wants to pick up this
idea. But past experience shows that we shouldn't count on this either.
It's definitely a nice project to tackle for someone who likes coding.

Stefan


From insertinterestingnamehere at gmail.com  Fri Aug  8 18:47:56 2014
From: insertinterestingnamehere at gmail.com (Ian Henriksen)
Date: Fri, 8 Aug 2014 10:47:56 -0600
Subject: [Cython] aritmetic with arrays in Cython
In-Reply-To: <53E4A639.4000507@behnel.de>
References: <CAPWbfpSGTwxiVY2TUd_juEHJ4a_CCukvTS793RSFmBHWNfxyRg@mail.gmail.com>
 <CAPWbfpSLSQZUw_CYYjXUm=Ma5kGGpm8yAP4kmyeRjaLprqfaSQ@mail.gmail.com>
 <53E45A84.2070308@behnel.de> <5698943.UtWdqSqk1C@sleeky>
 <53E4A639.4000507@behnel.de>
Message-ID: <CAPWbfpTFXVq5KwWWFCkag9rEb=hBJU5=AmcFdMUd5w74eMeeWw@mail.gmail.com>

I'd like to work on it. That was why I asked. It's been bothering me for
some time that we don't have this feature yet. This will take me some time
though since I'm not at all familiar with Cython's internals. I've used it
in relatively small settings for speeding up array operations and wrapping
external functions, but I've never had occasion to dig through much of what
it does internally. Some help navigating Cython's internals would be
greatly appreciated.

I'll also have to take some time to become more familiar with eigen itself.
Most of my work has used numpy arrays.

Eigen seems to be ideal for this since it allows us to offload the
expression analysis to another package where it is already
well-implemented. It looks like they have an array class for n-dimensional
array operations as well. My guess is that that would allow a fairly
natural translation from memory views to eigen arrays.

For now, I'll work on learning some of the details of how to use eigen.
Where would be a good place to start from there?

Would it be more helpful for me to start working on Cython bindings for
eigen, or to work on adding this directly as a part of Cython? Where should
I look in Cython's source code to do that? What kinds of modifications
would be necessary?

Thanks!

-Ian


On Fri, Aug 8, 2014 at 4:28 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:

> Mat?j Laitl schrieb am 08.08.2014 um 11:07:
> > On Friday 08 of August 2014 07:05:08 Stefan Behnel wrote:
> >> Eigen can apparently do these things already:
> >>
> >> http://eigen.tuxfamily.org/dox/TopicLazyEvaluation.html
> >>
> >> It might not be all that much work to integrate this with Cython to
> enable
> >> syntax support. Or have some kind of preprocessor, but I guess that's
> not
> >> going to be simpler than have it in Cython as an optional AST transform.
> >> Basically, whenever an arithmetic expression consists of a mixture of
> >> memory views and (optionally) scalars, it could be mapped to the kind of
> >> code that Ceygen uses, just at an arbitrarily complex expression length.
> >
> > Exactly! That was one of my ideas when I decided to create Ceygen, but I
> > resorted to take the easier route and made uglier static API because I
> didn't
> > (and still don't) have enough knowledge of Cython internals and Mark's
> "array
> > expressions" still looked like they are going to be merged.
>
> There are a couple of points in favour of Eigen/Ceygen. It's a way simpler
> approach than moving the entire vectorised code generator into Cython. And
> it lets Eigen do the hard work, leaving only the interfacing part to
> Cython. Basically, both tools can do what they're good at. We're always
> happy that there's a C compiler cleaning up after us to which we can
> delegate the low level optimisation and tuning stuff. The same would apply
> to Eigen. It's a much more cythonic approach than what's implemented in the
> pie-in-the-sky array expressions branch.
>
> It requires a C++ compiler, but I think that's an acceptable dependency
> these days in places where array computation performance matters. People
> even accept a dependency on huge tool stacks like Numba/LLVM for the same
> kind of tasks. An array expression syntax is just so much simpler than
> writing dumb nested loops.
>
>
> > Unfortunately, I no longer work on the project (my master's thesis) that
> > triggered work on Ceygen, and now I can invest only little maintenance
> work to
> > it.
>
> I don't see me being the one to implement this, either. :)
>
>
> > *However*, if such project is found worth to tacking by Cython devs, I
> can
> > imagine in my wild dreams that there may be a student at my former
> university
> > interested in doing it as their project. (this is rather optimistic
> statement,
> > please don't get too excited yet)
>
> I'm sure there are many people who would benefit from this. Some of them
> might even be able to provide funding, if someone wants to pick up this
> idea. But past experience shows that we shouldn't count on this either.
> It's definitely a nice project to tackle for someone who likes coding.
>
> Stefan
>
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> https://mail.python.org/mailman/listinfo/cython-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20140808/e5eef7c5/attachment.html>

From stefan_ml at behnel.de  Fri Aug  8 19:36:26 2014
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 08 Aug 2014 19:36:26 +0200
Subject: [Cython] aritmetic with arrays in Cython
In-Reply-To: <CAPWbfpTFXVq5KwWWFCkag9rEb=hBJU5=AmcFdMUd5w74eMeeWw@mail.gmail.com>
References: <CAPWbfpSGTwxiVY2TUd_juEHJ4a_CCukvTS793RSFmBHWNfxyRg@mail.gmail.com>
 <CAPWbfpSLSQZUw_CYYjXUm=Ma5kGGpm8yAP4kmyeRjaLprqfaSQ@mail.gmail.com>
 <53E45A84.2070308@behnel.de> <5698943.UtWdqSqk1C@sleeky>
 <53E4A639.4000507@behnel.de>
 <CAPWbfpTFXVq5KwWWFCkag9rEb=hBJU5=AmcFdMUd5w74eMeeWw@mail.gmail.com>
Message-ID: <53E50A9A.5050805@behnel.de>

Hi,

please don't top-post.

Ian Henriksen schrieb am 08.08.2014 um 18:47:
> I'd like to work on it. That was why I asked. It's been bothering me for
> some time that we don't have this feature yet. This will take me some time
> though since I'm not at all familiar with Cython's internals. I've used it
> in relatively small settings for speeding up array operations and wrapping
> external functions, but I've never had occasion to dig through much of what
> it does internally. Some help navigating Cython's internals would be
> greatly appreciated.

Here's a little intro.

https://github.com/cython/cython/wiki/HackerGuide#getting-started

I recommend to start by writing a simple test that does some integer
arithmetic and enable code generation traces for AST nodes in the generated
C code. "cython -a" should make this quite navigable.


> I'll also have to take some time to become more familiar with eigen itself.
> Most of my work has used numpy arrays.
> 
> Eigen seems to be ideal for this since it allows us to offload the
> expression analysis to another package where it is already
> well-implemented. It looks like they have an array class for n-dimensional
> array operations as well. My guess is that that would allow a fairly
> natural translation from memory views to eigen arrays.

You should dig into the code that Ceygen uses for the mapping. That's most
likely the best start you can get.


> Would it be more helpful for me to start working on Cython bindings for
> eigen, or to work on adding this directly as a part of Cython? Where should
> I look in Cython's source code to do that? What kinds of modifications
> would be necessary?

My guess is that this is best implemented with a set of matrix specific AST
nodes for the arithmetic operators. There is a recursive tree processing
phase in Cython's pipeline called "AnalyseExpressionsTransform" or type
analysis, where it figures out what variables reference memory views, what
the result type of an arithmetic expression is, etc. This phase can easily
replace nodes by returning new nodes from the analyse_types() methods of
AST nodes. Currently, arithmetic with memory views is not allowed, so this
needs to be enabled in the corresponding NumBinopNode AST subclasses
(AddNode, MatMulNode, etc. in ExprNodes.py). Write a test and debug it to
see where it fails.

I'm not sure yet how the code generation would look like. In simple cases,
you may be able to implement this into the existing AST arithmetic operator
nodes. But my guess is that you want to have a new set of nodes (probably
in a new module in Cython/Compiler/, e.g. EigenNodes.py) that the
analyse_types() methods of the generic arithmetic AST nodes return instead
of themselves when they determine that they are operating on memory views.
Those would then know how to generate matrix specific arithmetic C++/Eigen
code, potentially even inherit from the general arithmetic nodes to
override only the code generation method. There is a
NewNodeClass.from_node() class method to create a new node from an existing
one.

These specialised AST nodes will be optional in the end, Eigen should only
be needed if in C++ mode and this feature is enabled and used, otherwise,
array arithmetic will continue to be a compile time error.

Try to get something simple like binary matrix multiplication working for
now, or even just matrix x scalar. Map them to Ceygen's API to get a quick
start. Once that's done, try building up the code generation for more
complex nested expressions.

Stefan


From jtaylor.debian at googlemail.com  Fri Aug  8 20:02:47 2014
From: jtaylor.debian at googlemail.com (Julian Taylor)
Date: Fri, 08 Aug 2014 20:02:47 +0200
Subject: [Cython] aritmetic with arrays in Cython
In-Reply-To: <53E50A9A.5050805@behnel.de>
References: <CAPWbfpSGTwxiVY2TUd_juEHJ4a_CCukvTS793RSFmBHWNfxyRg@mail.gmail.com>
 <CAPWbfpSLSQZUw_CYYjXUm=Ma5kGGpm8yAP4kmyeRjaLprqfaSQ@mail.gmail.com>
 <53E45A84.2070308@behnel.de> <5698943.UtWdqSqk1C@sleeky>
 <53E4A639.4000507@behnel.de>
 <CAPWbfpTFXVq5KwWWFCkag9rEb=hBJU5=AmcFdMUd5w74eMeeWw@mail.gmail.com>
 <53E50A9A.5050805@behnel.de>
Message-ID: <53E510C7.4090903@googlemail.com>

On 08.08.2014 19:36, Stefan Behnel wrote:
> Hi,
> 
> please don't top-post.
> 
> Ian Henriksen schrieb am 08.08.2014 um 18:47:
>> I'd like to work on it. That was why I asked. It's been bothering me for
>> some time that we don't have this feature yet. This will take me some time
>> though since I'm not at all familiar with Cython's internals. I've used it
>> in relatively small settings for speeding up array operations and wrapping
>> external functions, but I've never had occasion to dig through much of what
>> it does internally. Some help navigating Cython's internals would be
>> greatly appreciated.
> 
> Here's a little intro.
> 
> https://github.com/cython/cython/wiki/HackerGuide#getting-started
> 
> I recommend to start by writing a simple test that does some integer
> arithmetic and enable code generation traces for AST nodes in the generated
> C code. "cython -a" should make this quite navigable.
> 
> 
>> I'll also have to take some time to become more familiar with eigen itself.
>> Most of my work has used numpy arrays.
>>
>> Eigen seems to be ideal for this since it allows us to offload the
>> expression analysis to another package where it is already
>> well-implemented. It looks like they have an array class for n-dimensional
>> array operations as well. My guess is that that would allow a fairly
>> natural translation from memory views to eigen arrays.
> 
> You should dig into the code that Ceygen uses for the mapping. That's most
> likely the best start you can get.
> 
> 
>> Would it be more helpful for me to start working on Cython bindings for
>> eigen, or to work on adding this directly as a part of Cython? Where should
>> I look in Cython's source code to do that? What kinds of modifications
>> would be necessary?
> 
> My guess is that this is best implemented with a set of matrix specific AST
> nodes for the arithmetic operators. There is a recursive tree processing
> phase in Cython's pipeline called "AnalyseExpressionsTransform" or type
> analysis, where it figures out what variables reference memory views, what
> the result type of an arithmetic expression is, etc. This phase can easily
> replace nodes by returning new nodes from the analyse_types() methods of
> AST nodes. Currently, arithmetic with memory views is not allowed, so this
> needs to be enabled in the corresponding NumBinopNode AST subclasses
> (AddNode, MatMulNode, etc. in ExprNodes.py). Write a test and debug it to
> see where it fails.

If someone wants to work on an ast transformer for array expressions I
would not restrict its output to something ceygen can parse but also numpy.
E.g. transforming:

cpdef whateverfunction(double[:] a, float[:] b):
    return a + a * b + .5

to

cpdef whateverfunction(double[:] a, float[:] b):
    bs = 10000
    r = empty_like(a)
    for i in range(0, a.size, bs):
        u = min(i + bs, a.size)
        r[i:u] = a[i:u] * b[i:u]
        r[i:u] += a[i:u]
        r[i:u] += 0.5
    return r

this is already very efficient in numpy.

From insertinterestingnamehere at gmail.com  Fri Aug  8 23:09:05 2014
From: insertinterestingnamehere at gmail.com (Ian Henriksen)
Date: Fri, 8 Aug 2014 15:09:05 -0600
Subject: [Cython] aritmetic with arrays in Cython
In-Reply-To: <53E510C7.4090903@googlemail.com>
References: <CAPWbfpSGTwxiVY2TUd_juEHJ4a_CCukvTS793RSFmBHWNfxyRg@mail.gmail.com>
 <CAPWbfpSLSQZUw_CYYjXUm=Ma5kGGpm8yAP4kmyeRjaLprqfaSQ@mail.gmail.com>
 <53E45A84.2070308@behnel.de> <5698943.UtWdqSqk1C@sleeky>
 <53E4A639.4000507@behnel.de>
 <CAPWbfpTFXVq5KwWWFCkag9rEb=hBJU5=AmcFdMUd5w74eMeeWw@mail.gmail.com>
 <53E50A9A.5050805@behnel.de> <53E510C7.4090903@googlemail.com>
Message-ID: <CAPWbfpS1fFWcgU=Z_B8teHwcChyJRLOWYqaRvqLe2VZU4exWLQ@mail.gmail.com>

On Fri, Aug 8, 2014 at 12:02 PM, Julian Taylor <
jtaylor.debian at googlemail.com> wrote:

> On 08.08.2014 19:36, Stefan Behnel wrote:
> > Hi,
> >
> > please don't top-post.
> >
> > Ian Henriksen schrieb am 08.08.2014 um 18:47:
> >> I'd like to work on it. That was why I asked. It's been bothering me for
> >> some time that we don't have this feature yet. This will take me some
> time
> >> though since I'm not at all familiar with Cython's internals. I've used
> it
> >> in relatively small settings for speeding up array operations and
> wrapping
> >> external functions, but I've never had occasion to dig through much of
> what
> >> it does internally. Some help navigating Cython's internals would be
> >> greatly appreciated.
> >
> > Here's a little intro.
> >
> > https://github.com/cython/cython/wiki/HackerGuide#getting-started
> >
> > I recommend to start by writing a simple test that does some integer
> > arithmetic and enable code generation traces for AST nodes in the
> generated
> > C code. "cython -a" should make this quite navigable.
> >
> >
> >> I'll also have to take some time to become more familiar with eigen
> itself.
> >> Most of my work has used numpy arrays.
> >>
> >> Eigen seems to be ideal for this since it allows us to offload the
> >> expression analysis to another package where it is already
> >> well-implemented. It looks like they have an array class for
> n-dimensional
> >> array operations as well. My guess is that that would allow a fairly
> >> natural translation from memory views to eigen arrays.
> >
> > You should dig into the code that Ceygen uses for the mapping. That's
> most
> > likely the best start you can get.
> >
> >
> >> Would it be more helpful for me to start working on Cython bindings for
> >> eigen, or to work on adding this directly as a part of Cython? Where
> should
> >> I look in Cython's source code to do that? What kinds of modifications
> >> would be necessary?
> >
> > My guess is that this is best implemented with a set of matrix specific
> AST
> > nodes for the arithmetic operators. There is a recursive tree processing
> > phase in Cython's pipeline called "AnalyseExpressionsTransform" or type
> > analysis, where it figures out what variables reference memory views,
> what
> > the result type of an arithmetic expression is, etc. This phase can
> easily
> > replace nodes by returning new nodes from the analyse_types() methods of
> > AST nodes. Currently, arithmetic with memory views is not allowed, so
> this
> > needs to be enabled in the corresponding NumBinopNode AST subclasses
> > (AddNode, MatMulNode, etc. in ExprNodes.py). Write a test and debug it to
> > see where it fails.
>
> If someone wants to work on an ast transformer for array expressions I
> would not restrict its output to something ceygen can parse but also numpy.
> E.g. transforming:
>
> cpdef whateverfunction(double[:] a, float[:] b):
>     return a + a * b + .5
>
> to
>
> cpdef whateverfunction(double[:] a, float[:] b):
>     bs = 10000
>     r = empty_like(a)
>     for i in range(0, a.size, bs):
>         u = min(i + bs, a.size)
>         r[i:u] = a[i:u] * b[i:u]
>         r[i:u] += a[i:u]
>         r[i:u] += 0.5
>     return r
>
> this is already very efficient in numpy.
>
>
Sorry about the top post.

Stefan, thanks for the direction. I think I know how to proceed now. It'll
probably take me some time to learn how to do all this, but I'll start
reading/working on it now.

Julian, from a dependency standpoint I'd really prefer to use numpy. As
near as I can tell, the people most likely to use this feature already use
numpy. That would also remove the C++ dependency for this. I think the main
reason for using eigen is that it will be easy to interface with and will
eliminate temporaries automatically. In the long term, it could be good to
offer multiple backends for these sorts of operations. Numpy would be a
primary candidate for something like that. I'll spend some time working
with numpy's ufunc api to see if I can figure out a good way to use that
instead.
Thanks!
-Ian
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20140808/6ac7d23a/attachment.html>

From insertinterestingnamehere at gmail.com  Sat Aug  9 00:15:29 2014
From: insertinterestingnamehere at gmail.com (Ian Henriksen)
Date: Fri, 8 Aug 2014 16:15:29 -0600
Subject: [Cython] aritmetic with arrays in Cython
In-Reply-To: <CAPWbfpS1fFWcgU=Z_B8teHwcChyJRLOWYqaRvqLe2VZU4exWLQ@mail.gmail.com>
References: <CAPWbfpSGTwxiVY2TUd_juEHJ4a_CCukvTS793RSFmBHWNfxyRg@mail.gmail.com>
 <CAPWbfpSLSQZUw_CYYjXUm=Ma5kGGpm8yAP4kmyeRjaLprqfaSQ@mail.gmail.com>
 <53E45A84.2070308@behnel.de> <5698943.UtWdqSqk1C@sleeky>
 <53E4A639.4000507@behnel.de>
 <CAPWbfpTFXVq5KwWWFCkag9rEb=hBJU5=AmcFdMUd5w74eMeeWw@mail.gmail.com>
 <53E50A9A.5050805@behnel.de> <53E510C7.4090903@googlemail.com>
 <CAPWbfpS1fFWcgU=Z_B8teHwcChyJRLOWYqaRvqLe2VZU4exWLQ@mail.gmail.com>
Message-ID: <CAPWbfpRPQi5-oyWGkR8JVsoeh128XhgVwjS1Dht_1nvujO6pDw@mail.gmail.com>

On Fri, Aug 8, 2014 at 3:09 PM, Ian Henriksen <
insertinterestingnamehere at gmail.com> wrote:

> On Fri, Aug 8, 2014 at 12:02 PM, Julian Taylor <
> jtaylor.debian at googlemail.com> wrote:
>
>> On 08.08.2014 19:36, Stefan Behnel wrote:
>> > Hi,
>> >
>> > please don't top-post.
>> >
>> > Ian Henriksen schrieb am 08.08.2014 um 18:47:
>> >> I'd like to work on it. That was why I asked. It's been bothering me
>> for
>> >> some time that we don't have this feature yet. This will take me some
>> time
>> >> though since I'm not at all familiar with Cython's internals. I've
>> used it
>> >> in relatively small settings for speeding up array operations and
>> wrapping
>> >> external functions, but I've never had occasion to dig through much of
>> what
>> >> it does internally. Some help navigating Cython's internals would be
>> >> greatly appreciated.
>> >
>> > Here's a little intro.
>> >
>> > https://github.com/cython/cython/wiki/HackerGuide#getting-started
>> >
>> > I recommend to start by writing a simple test that does some integer
>> > arithmetic and enable code generation traces for AST nodes in the
>> generated
>> > C code. "cython -a" should make this quite navigable.
>> >
>> >
>> >> I'll also have to take some time to become more familiar with eigen
>> itself.
>> >> Most of my work has used numpy arrays.
>> >>
>> >> Eigen seems to be ideal for this since it allows us to offload the
>> >> expression analysis to another package where it is already
>> >> well-implemented. It looks like they have an array class for
>> n-dimensional
>> >> array operations as well. My guess is that that would allow a fairly
>> >> natural translation from memory views to eigen arrays.
>> >
>> > You should dig into the code that Ceygen uses for the mapping. That's
>> most
>> > likely the best start you can get.
>> >
>> >
>> >> Would it be more helpful for me to start working on Cython bindings for
>> >> eigen, or to work on adding this directly as a part of Cython? Where
>> should
>> >> I look in Cython's source code to do that? What kinds of modifications
>> >> would be necessary?
>> >
>> > My guess is that this is best implemented with a set of matrix specific
>> AST
>> > nodes for the arithmetic operators. There is a recursive tree processing
>> > phase in Cython's pipeline called "AnalyseExpressionsTransform" or type
>> > analysis, where it figures out what variables reference memory views,
>> what
>> > the result type of an arithmetic expression is, etc. This phase can
>> easily
>> > replace nodes by returning new nodes from the analyse_types() methods of
>> > AST nodes. Currently, arithmetic with memory views is not allowed, so
>> this
>> > needs to be enabled in the corresponding NumBinopNode AST subclasses
>> > (AddNode, MatMulNode, etc. in ExprNodes.py). Write a test and debug it
>> to
>> > see where it fails.
>>
>> If someone wants to work on an ast transformer for array expressions I
>> would not restrict its output to something ceygen can parse but also
>> numpy.
>> E.g. transforming:
>>
>> cpdef whateverfunction(double[:] a, float[:] b):
>>     return a + a * b + .5
>>
>> to
>>
>> cpdef whateverfunction(double[:] a, float[:] b):
>>     bs = 10000
>>     r = empty_like(a)
>>     for i in range(0, a.size, bs):
>>         u = min(i + bs, a.size)
>>         r[i:u] = a[i:u] * b[i:u]
>>         r[i:u] += a[i:u]
>>         r[i:u] += 0.5
>>     return r
>>
>> this is already very efficient in numpy.
>>
>>
> Sorry about the top post.
>
> Stefan, thanks for the direction. I think I know how to proceed now. It'll
> probably take me some time to learn how to do all this, but I'll start
> reading/working on it now.
>
> Julian, from a dependency standpoint I'd really prefer to use numpy. As
> near as I can tell, the people most likely to use this feature already use
> numpy. That would also remove the C++ dependency for this. I think the main
> reason for using eigen is that it will be easy to interface with and will
> eliminate temporaries automatically. In the long term, it could be good to
> offer multiple backends for these sorts of operations. Numpy would be a
> primary candidate for something like that. I'll spend some time working
> with numpy's ufunc api to see if I can figure out a good way to use that
> instead.
> Thanks!
> -Ian
>

Maybe I should clarify a little about why eigen is a good place to start.
According to http://eigen.tuxfamily.org/dox/TopicLazyEvaluation.html it
already takes care of things like the elimination of temporary variables
and common subexpression reduction at compile time. This should make it
possible to compile array expressions in Cython without having to
re-implement those sorts of optimizations. Ideally we would just have to
map memory view operations to corresponding equivalents from eigen. It's
not yet clear to me how to do things with arbitrary-dimensional arrays or
broadcasting, but, given some more time, a solution may present itself.
-Ian
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20140808/3066a46d/attachment-0001.html>

From jrobertray at gmail.com  Sat Aug  9 04:04:31 2014
From: jrobertray at gmail.com (J Robert Ray)
Date: Fri, 8 Aug 2014 19:04:31 -0700
Subject: [Cython] Bug: operator[] does not respect except +
Message-ID: <CADb3U=53qr=9YOztdmSgVk28-+KkBb8LRP6ZPCwqaf7GQvwGEQ@mail.gmail.com>

I apologize for not having a working demonstration, but I have found this
bug and verified it is still an issue in Cython 0.20.2. No try/catch block
is generated for operator[]. In contrast, a try/catch block does get
generated for operator().

If it is any use, here is a paraphrasing of the pyx code and the generated
cpp code.

cdef extern from "blah.h" namespace "blah":
    cdef cppclass Blah:
        int operator[](int) except +

...

cdef class PyBlah:
    ...
    def demo(self, int index):
        return deref(self.thisptr)[index]

...

/* "Blah.pyx":107
 *     def demo(self, int index):
 *         return deref(self.thisptr)[index]             # <<<<<<<<<<<<<<
 */
  __Pyx_XDECREF(__pyx_r);
  __pyx_t_1 =
__Pyx_PyInt_From_int(((*__pyx_v_self->thisptr)[__pyx_v_index])); if
(unlikely(!__pyx_t_1)) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 107;
__pyx_clineno = __LINE__; goto __pyx_L1_error;}
  __Pyx_GOTREF(__pyx_t_1);
  __pyx_r = __pyx_t_1;
  __pyx_t_1 = 0;
  goto __pyx_L0;


Please let me know if I'm doing anything wrong. Thanks!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20140808/cb2b34df/attachment.html>

From sturla.molden at gmail.com  Sun Aug 10 20:41:31 2014
From: sturla.molden at gmail.com (Sturla Molden)
Date: Sun, 10 Aug 2014 18:41:31 +0000 (UTC)
Subject: [Cython] aritmetic with arrays in Cython
References: <CAPWbfpSGTwxiVY2TUd_juEHJ4a_CCukvTS793RSFmBHWNfxyRg@mail.gmail.com>
 <22700580.GCF8m5KeTp@sleeky>
 <CAPWbfpSLSQZUw_CYYjXUm=Ma5kGGpm8yAP4kmyeRjaLprqfaSQ@mail.gmail.com>
 <53E45A84.2070308@behnel.de> <5698943.UtWdqSqk1C@sleeky>
 <53E4A639.4000507@behnel.de>
 <CAPWbfpTFXVq5KwWWFCkag9rEb=hBJU5=AmcFdMUd5w74eMeeWw@mail.gmail.com>
 <53E50A9A.5050805@behnel.de> <53E510C7.4090903@googlemail.com>
 <CAPWbfpS1fFWcgU=Z_B8teHwcChyJRLOWYqaRvqLe2VZU4exWLQ@mail.gmail.com>
 <CAPWbfpRPQi5-oyWGkR8JVsoeh128XhgVwjS1Dht_1nvujO6pDw@mail.gmail.com>
Message-ID: <1984369421429388619.771407sturla.molden-gmail.com@news.gmane.org>

Ian Henriksen
<insertinterestingnamehere at gmail.com> wrote:

> Maybe I should clarify a little about why eigen is a good place to start.
> According to <a
> href="http://eigen.tuxfamily.org/dox/TopicLazyEvaluation.html">http://eigen.tuxfamily.org/dox/TopicLazyEvaluation.html</a>
> it
> already takes care of things like the elimination of temporary variables
> and common subexpression reduction at compile time. This should make it
> possible to compile array expressions in Cython without having to
> re-implement those sorts of optimizations. Ideally we would just have to
> map memory view operations to corresponding equivalents from eigen. It's
> not yet clear to me how to do things with arbitrary-dimensional arrays or
> broadcasting, but, given some more time, a solution may present itself.
> -Ian

cilkplus is what you want, not Eigen.

But if you are serious about number crunching, learn Fortran 95.


Sturla


From leon at bottou.org  Mon Aug 11 03:20:16 2014
From: leon at bottou.org (Leon Bottou)
Date: Sun, 10 Aug 2014 21:20:16 -0400
Subject: [Cython] OpenMP thread private variable not recognized (bug report
	+ discussion)
Message-ID: <2940318.xX4Y2fIxDu@fuji>


The attached cython program uses an extension class to represent a unit of 
work. The with parallel block temporarily gets the gil to allocate the object, 
then release the gil and performs the task with a for i in prange(...) 
statement. My expectation was to have w recognized as a thread private 
variable.

    with nogil, parallel():
        with gil: 
            w = Worker(n)                 # should be thread private
            with nogil: 
                for i in prange(0,m): 
                    r += w.run()          # should be reduction
            w = None                      # is this needed?


Cythonize (0.20.2) works without error but produces an incorrect C file.

hello.c: In function ?__pyx_pf_5hello_run?:
hello.c:2193:42: error: expected identifier before ?)? token
hello.c: At top level:

The erroneous line is:

#pragma omp parallel private() reduction(+:__pyx_v_r) private(__pyx_t_5, 
__pyx_t_4, __pyx_t_3) firstprivate(__pyx_t_1, __pyx_t_2) 
private(__pyx_filename, __pyx_lineno, __pyx_clineno) 
shared(__pyx_parallel_why, __pyx_parallel_exc_type, __pyx_parallel_exc_value, 
__pyx_parallel_exc_tb)

where you can see that the first private() clause has no argument. The 
variable __pyx_v_w is not declared as private either as I would expect.

I believe that the problem comes from line 7720 in Cython/Compiler/Node.py

        if self.privates:
            privates = [e.cname for e in self.privates
                                    if not e.type.is_pyobject]
            code.put('private(%s)' % ', '.join(privates))

And I further believe that the clause "if not e.type.is_pyobject" has been 
added because nothing would decrements the reference count of the thread 
private worker object when leaving the parallel block. 

My quick fix would be to remove this clause and make sure that my program 
contains the line "w = None" before leaving the thread. But I realize that 
this is not sufficient for you.

Note that the temporary python objects generated by the call to the Worker 
construction are correctly recognized as thread private and their reference 
count is correctly decremented when they are no longer needed.  The problem 
here is the clash between the python scoping rules and the semantics of thread 
private variables. This is one of these cases where I would have liked to be 
able to write

    with nogil, parallel():
        with gil: 
            cdef w = Worker(n)            # block-scoped cdef
            with nogil: 
                for i in prange(0,m): 
                    r += w.run()

with an understanding that the scope of the cdef variable is limited to the 
block where the cdef appears. But when you try this, cython tells you that 
cdefs are not legal there. 


-------------- next part --------------
# -*- Python -*-

import numpy as np
cimport numpy as np
from cython.parallel import parallel, prange


cdef class Worker:
    cdef double[::1] v
    def __init__(self, int n):
        self.v = np.random.randn(n)
    cdef double run(self) nogil:
        cdef int i
        cdef int n = self.v.shape[0]
        cdef double s = 0
        for i in range(0,n): 
            s += self.v[i]
        return s / n

def run(int n, int m):
    cdef Worker w
    cdef double r
    cdef int i
    with nogil, parallel():
        with gil: 
            w = Worker(n)
            with nogil: 
                for i in prange(0,m): 
                    r += w.run()
            w = None
    return r / m

-------------- next part --------------
A non-text attachment was scrubbed...
Name: setup.py
Type: text/x-python
Size: 349 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20140810/932f28a5/attachment.py>

From insertinterestingnamehere at gmail.com  Tue Aug 12 04:34:36 2014
From: insertinterestingnamehere at gmail.com (Ian Henriksen)
Date: Mon, 11 Aug 2014 20:34:36 -0600
Subject: [Cython] aritmetic with arrays in Cython
In-Reply-To: <1984369421429388619.771407sturla.molden-gmail.com@news.gmane.org>
References: <CAPWbfpSGTwxiVY2TUd_juEHJ4a_CCukvTS793RSFmBHWNfxyRg@mail.gmail.com>
 <22700580.GCF8m5KeTp@sleeky>
 <CAPWbfpSLSQZUw_CYYjXUm=Ma5kGGpm8yAP4kmyeRjaLprqfaSQ@mail.gmail.com>
 <53E45A84.2070308@behnel.de> <5698943.UtWdqSqk1C@sleeky>
 <53E4A639.4000507@behnel.de>
 <CAPWbfpTFXVq5KwWWFCkag9rEb=hBJU5=AmcFdMUd5w74eMeeWw@mail.gmail.com>
 <53E50A9A.5050805@behnel.de> <53E510C7.4090903@googlemail.com>
 <CAPWbfpS1fFWcgU=Z_B8teHwcChyJRLOWYqaRvqLe2VZU4exWLQ@mail.gmail.com>
 <CAPWbfpRPQi5-oyWGkR8JVsoeh128XhgVwjS1Dht_1nvujO6pDw@mail.gmail.com>
 <1984369421429388619.771407sturla.molden-gmail.com@news.gmane.org>
Message-ID: <CAPWbfpQ+Ym1N+U436T1YxOOixBmMh0tyG=mMQWyuwnW2+w_U2A@mail.gmail.com>

On Sun, Aug 10, 2014 at 12:41 PM, Sturla Molden <sturla.molden at gmail.com>
wrote:

> Ian Henriksen
> <insertinterestingnamehere at gmail.com> wrote:
>
> > Maybe I should clarify a little about why eigen is a good place to start.
> > According to <a
> > href="http://eigen.tuxfamily.org/dox/TopicLazyEvaluation.html">
> http://eigen.tuxfamily.org/dox/TopicLazyEvaluation.html</a>
> > it
> > already takes care of things like the elimination of temporary variables
> > and common subexpression reduction at compile time. This should make it
> > possible to compile array expressions in Cython without having to
> > re-implement those sorts of optimizations. Ideally we would just have to
> > map memory view operations to corresponding equivalents from eigen. It's
> > not yet clear to me how to do things with arbitrary-dimensional arrays or
> > broadcasting, but, given some more time, a solution may present itself.
> > -Ian
>
> cilkplus is what you want, not Eigen.
>
> But if you are serious about number crunching, learn Fortran 95.
>
>
> Sturla
>
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> https://mail.python.org/mailman/listinfo/cython-devel
>

Cilk Plus would also work really nicely for this. Thanks for the suggestion.
Fortran is a really great language for this sort of thing, but I don't
think I'm ready to tackle the difficulties of using it as a backend for
array arithmetic in Cython. It would be a really great feature to have
later on though.
Thanks!
-Ian
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20140811/7d497529/attachment.html>

From marcus.brinkmann at ruhr-uni-bochum.de  Tue Aug 12 04:57:00 2014
From: marcus.brinkmann at ruhr-uni-bochum.de (Marcus Brinkmann)
Date: Tue, 12 Aug 2014 04:57:00 +0200
Subject: [Cython] bug: constructor declarations not working
Message-ID: <53E9827C.2070609@ruhr-uni-bochum.de>

Hi,

I want to declare and define C++ classes from Cython.  Sure, I could 
write them in native C++, but it's just small glue code, and I prefer to 
let Cython handle the GIL and all the other good stuff.

First, for reference, the following works (c.f. 
tests/run/cpp_classes_def.pyx):

== cython --cplus example1.pyx ============
cdef cppclass Foo "Foo":
   int _foo
   __init__(int foo) nogil:
     this._foo = foo
   __dealloc__() nogil:
     pass
===========================================

This will emit a declaration for Foo and the implementations of the 
constructor and destructor (output is truncated to show relevant parts 
only):

struct Foo {
   int _foo;
    Foo(int);
   virtual  ~Foo(void);
};
  Foo::Foo(int __pyx_v_foo) {
   this->_foo = __pyx_v_foo;
}
  Foo::~Foo(void) {
}

Now, I want to export the class to other cython files.  The following 
does not work, and I think it's a bug:

== example2.pxd ===========================
cdef extern cppclass Foo:
   int _foo
   __init__(int foo)
   __dealloc__()
===========================================

== cython --cplus example2.pyx ============
cdef cppclass Foo:
   __init__(int foo) nogil:
     this._foo = foo
   __dealloc__() nogil:
     pass
===========================================

== cython --cplus example3.pyx ============
from example2 cimport Foo

cdef Foo* foo_p = new Foo(2)
===========================================

This fails for example2 with an obscure error message:

$ cython --cplus example2.pyx
example2.pyx:3:8: undeclared name not builtin: this
...more spurious errors...

For example3, Cython runs through but trying to compile shows the actual 
problem (which you can also see in example2 if you shift things around a 
bit):

$ g++ -fPIC -shared -o example3.so -I /usr/include/python2.7 example3.cpp
example3.cpp: In function ?void initexample3()?:
example3.cpp:775:38: error: no matching function for call to ?Foo::Foo(int)?
    __pyx_v_8example3_foo_p = new Foo(2);

This is in the generated code:

/*--- Type declarations ---*/
struct Foo;
struct Foo {
   int _foo;
   virtual  ~Foo(void);
};

It's missing the constructor declaration!

I traced this a bit down the Compiler/Symtab.py and found this part in 
CppClassScope::declare_var:

4e07fc52 (Robert Bradshaw      2012-08-21 00:46:00 -0700 2112) 
if name != "this" and (defining or name != "<init>"):
4e07fc52 (Robert Bradshaw      2012-08-21 00:46:00 -0700 2113) 
    self.var_entries.append(entry)

Here defining is 0, so the declaration is skipped.  I don't know Cython 
internals, so I am hoping someone who does can fix this easily given the 
above information or point me in the right direction.

Thanks a lot for Cython, it's very useful!

Marcus Brinkmann

From stefan_ml at behnel.de  Tue Aug 12 07:46:33 2014
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 12 Aug 2014 07:46:33 +0200
Subject: [Cython] aritmetic with arrays in Cython
In-Reply-To: <CAPWbfpQ+Ym1N+U436T1YxOOixBmMh0tyG=mMQWyuwnW2+w_U2A@mail.gmail.com>
References: <CAPWbfpSGTwxiVY2TUd_juEHJ4a_CCukvTS793RSFmBHWNfxyRg@mail.gmail.com>
 <22700580.GCF8m5KeTp@sleeky>
 <CAPWbfpSLSQZUw_CYYjXUm=Ma5kGGpm8yAP4kmyeRjaLprqfaSQ@mail.gmail.com>
 <53E45A84.2070308@behnel.de> <5698943.UtWdqSqk1C@sleeky>
 <53E4A639.4000507@behnel.de>
 <CAPWbfpTFXVq5KwWWFCkag9rEb=hBJU5=AmcFdMUd5w74eMeeWw@mail.gmail.com>
 <53E50A9A.5050805@behnel.de> <53E510C7.4090903@googlemail.com>
 <CAPWbfpS1fFWcgU=Z_B8teHwcChyJRLOWYqaRvqLe2VZU4exWLQ@mail.gmail.com>
 <CAPWbfpRPQi5-oyWGkR8JVsoeh128XhgVwjS1Dht_1nvujO6pDw@mail.gmail.com>
 <1984369421429388619.771407sturla.molden-gmail.com@news.gmane.org>
 <CAPWbfpQ+Ym1N+U436T1YxOOixBmMh0tyG=mMQWyuwnW2+w_U2A@mail.gmail.com>
Message-ID: <53E9AA39.8050805@behnel.de>

Ian Henriksen schrieb am 12.08.2014 um 04:34:
> On Sun, Aug 10, 2014 at 12:41 PM, Sturla Molden wrote:
>> Ian Henriksen wrote:
>>> Maybe I should clarify a little about why eigen is a good place to start.
>>> According to <a
>>> href="http://eigen.tuxfamily.org/dox/TopicLazyEvaluation.html">
>> http://eigen.tuxfamily.org/dox/TopicLazyEvaluation.html</a>
>>> it
>>> already takes care of things like the elimination of temporary variables
>>> and common subexpression reduction at compile time. This should make it
>>> possible to compile array expressions in Cython without having to
>>> re-implement those sorts of optimizations. Ideally we would just have to
>>> map memory view operations to corresponding equivalents from eigen. It's
>>> not yet clear to me how to do things with arbitrary-dimensional arrays or
>>> broadcasting, but, given some more time, a solution may present itself.
>>> -Ian
>>
>> cilkplus is what you want, not Eigen.
>>
>> But if you are serious about number crunching, learn Fortran 95.
> 
> Cilk Plus would also work really nicely for this. Thanks for the suggestion.
> Fortran is a really great language for this sort of thing, but I don't
> think I'm ready to tackle the difficulties of using it as a backend for
> array arithmetic in Cython. It would be a really great feature to have
> later on though.

That clarifies a bit of the design then: The syntax support should be
somewhat generic, with specialised (sets of) node implementations as
backends that generate code for different libraries/compilers/languages.

It's ok to start only with Eigen, though. We have working example code for
it and everything else has either a much higher entry level for the
implementation or a much lower general availability of the required tools.

For the syntax/type support, a look at the array expressions branch might
also be helpful, although I doubt that there really is all that much to do
on that front.

Stefan


From sturla.molden at gmail.com  Tue Aug 12 16:26:31 2014
From: sturla.molden at gmail.com (Sturla Molden)
Date: Tue, 12 Aug 2014 14:26:31 +0000 (UTC)
Subject: [Cython] OpenMP thread private variable not recognized (bug
	report + discussion)
References: <2940318.xX4Y2fIxDu@fuji>
Message-ID: <1538670702429545235.785094sturla.molden-gmail.com@news.gmane.org>

Cython does not do an error here:

- i is recognized as private
- r is recognized as reduction
- w is (correctly) recognized as shared

If you need thread local storage, use threading.local()

I agree that scoped cdefs would be an advantage.

Personally I prefer to avoid OpenMP and just use Python threads and an
internal function (closure) or an internal class. If you start to use
OpenMP, Apple's libdispatch ("GCD"), Intel TBB, or Intel clikplus, you will
soon discover that they are all variations over the same theme: a thread
pool and a closure. Whether you call it a parallel block in OpenMP or an
anonymous block in GCD, it is fundamentally a closure. That's all there is.
You can easily do this with Python threads: Python, unlike C, supports
closures or internal classes directly in the language, and does not need
special extensions like C. Python threads and OpenMP threads will scale
equally well (they are all native OS threads, scheduled in the same way),
and there will be no scoping problems. The sooner you discover you do not
need Cython's prange, the less pain it will cause.

Sturla


From brett.calcott at gmail.com  Tue Aug 12 17:10:43 2014
From: brett.calcott at gmail.com (Brett Calcott)
Date: Tue, 12 Aug 2014 11:10:43 -0400
Subject: [Cython] OpenMP thread private variable not recognized (bug
 report + discussion)
In-Reply-To: <1538670702429545235.785094sturla.molden-gmail.com@news.gmane.org>
References: <2940318.xX4Y2fIxDu@fuji>
 <1538670702429545235.785094sturla.molden-gmail.com@news.gmane.org>
Message-ID: <CAAvcK2b4ZgfVXkYGebx2377hZghSb1XsU-L5poFH3p05WN-Bbg@mail.gmail.com>

On Tue, Aug 12, 2014 at 10:26 AM, Sturla Molden <sturla.molden at gmail.com>
wrote:

> The sooner you discover you do not
> need Cython's prange, the less pain it will cause.
>
>
For someone who has bumbled around trying to use prange & openmp on the mac
(but successfully used python threading), this sounds great. Is there an
example of this somewhere that you can point us to?

Thanks
Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20140812/b394a615/attachment.html>

From sturla.molden at gmail.com  Tue Aug 12 17:18:12 2014
From: sturla.molden at gmail.com (Sturla Molden)
Date: Tue, 12 Aug 2014 15:18:12 +0000 (UTC)
Subject: [Cython] aritmetic with arrays in Cython
References: <53E45A84.2070308@behnel.de> <5698943.UtWdqSqk1C@sleeky>
 <53E4A639.4000507@behnel.de>
 <CAPWbfpTFXVq5KwWWFCkag9rEb=hBJU5=AmcFdMUd5w74eMeeWw@mail.gmail.com>
 <53E50A9A.5050805@behnel.de> <53E510C7.4090903@googlemail.com>
 <CAPWbfpS1fFWcgU=Z_B8teHwcChyJRLOWYqaRvqLe2VZU4exWLQ@mail.gmail.com>
 <CAPWbfpRPQi5-oyWGkR8JVsoeh128XhgVwjS1Dht_1nvujO6pDw@mail.gmail.com>
 <1984369421429388619.771407sturla.molden-gmail.com@news.gmane.org>
 <CAPWbfpQ+Ym1N+U436T1YxOOixBmMh0tyG=mMQWyuwnW2+w_U2A@mail.gmail.com>
 <53E9AA39.8050805@behnel.de>
Message-ID: <224570377429548214.833522sturla.molden-gmail.com@news.gmane.org>

But using Eigen will taint the output with Eigen's license, since the Eigen
library is statically linked. OTOH, Cilkplus is just a compiler extension
for C and C++. 

AFAIK, it is currently available for Intel C++ and Clang (also by Intel)
and GCC 4.9. On MSVC I believe it requires Intel Parallel Studio. The main
obstacle to its adoption today is MSVC and GCC 4.8. Cilkplus is also being
evaluated for becoming parts of the future C and C++ standards.

Another thing to observe is that Eigen depends on the C++ compiler to elide
temporary arrays. Currently it only/mostly happens for fixed size arrays.
Clikplus arrays does not suffer from this. You just index pointers like
typed memory views (though the indexing syntax is slightly different), and
it compiles to efficient machine code just like Fortran 90 array code. 

Sturla

Stefan Behnel <stefan_ml at behnel.de> wrote:
> Ian Henriksen schrieb am 12.08.2014 um 04:34:
>> On Sun, Aug 10, 2014 at 12:41 PM, Sturla Molden wrote:
>>> Ian Henriksen wrote:
>>>> Maybe I should clarify a little about why eigen is a good place to start.
>>>> According to <a
>>>> href="http://eigen.tuxfamily.org/dox/TopicLazyEvaluation.html">
>>> http://eigen.tuxfamily.org/dox/TopicLazyEvaluation.html</a>
>>>> it
>>>> already takes care of things like the elimination of temporary variables
>>>> and common subexpression reduction at compile time. This should make it
>>>> possible to compile array expressions in Cython without having to
>>>> re-implement those sorts of optimizations. Ideally we would just have to
>>>> map memory view operations to corresponding equivalents from eigen. It's
>>>> not yet clear to me how to do things with arbitrary-dimensional arrays or
>>>> broadcasting, but, given some more time, a solution may present itself.
>>>> -Ian
>>> 
>>> cilkplus is what you want, not Eigen.
>>> 
>>> But if you are serious about number crunching, learn Fortran 95.
>> 
>> Cilk Plus would also work really nicely for this. Thanks for the suggestion.
>> Fortran is a really great language for this sort of thing, but I don't
>> think I'm ready to tackle the difficulties of using it as a backend for
>> array arithmetic in Cython. It would be a really great feature to have
>> later on though.
> 
> That clarifies a bit of the design then: The syntax support should be
> somewhat generic, with specialised (sets of) node implementations as
> backends that generate code for different libraries/compilers/languages.
> 
> It's ok to start only with Eigen, though. We have working example code for
> it and everything else has either a much higher entry level for the
> implementation or a much lower general availability of the required tools.
> 
> For the syntax/type support, a look at the array expressions branch might
> also be helpful, although I doubt that there really is all that much to do
> on that front.
> 
> Stefan


From sturla.molden at gmail.com  Tue Aug 12 17:20:44 2014
From: sturla.molden at gmail.com (Sturla Molden)
Date: Tue, 12 Aug 2014 15:20:44 +0000 (UTC)
Subject: [Cython] OpenMP thread private variable not recognized (bug
	report + discussion)
References: <2940318.xX4Y2fIxDu@fuji>
 <1538670702429545235.785094sturla.molden-gmail.com@news.gmane.org>
 <CAAvcK2b4ZgfVXkYGebx2377hZghSb1XsU-L5poFH3p05WN-Bbg@mail.gmail.com>
Message-ID: <2126213584429549523.407019sturla.molden-gmail.com@news.gmane.org>

Brett Calcott <brett.calcott at gmail.com>
wrote:

> For someone who has bumbled around trying to use prange &amp; openmp on the mac
> (but successfully used python threading), this sounds great. Is there an
> example of this somewhere that you can point us to?

No, but I could make one :)

ipython notebook?


Sturla


From brett.calcott at gmail.com  Tue Aug 12 17:31:34 2014
From: brett.calcott at gmail.com (Brett Calcott)
Date: Tue, 12 Aug 2014 11:31:34 -0400
Subject: [Cython] OpenMP thread private variable not recognized (bug
 report + discussion)
In-Reply-To: <2126213584429549523.407019sturla.molden-gmail.com@news.gmane.org>
References: <2940318.xX4Y2fIxDu@fuji>
 <1538670702429545235.785094sturla.molden-gmail.com@news.gmane.org>
 <CAAvcK2b4ZgfVXkYGebx2377hZghSb1XsU-L5poFH3p05WN-Bbg@mail.gmail.com>
 <2126213584429549523.407019sturla.molden-gmail.com@news.gmane.org>
Message-ID: <CAAvcK2b7MuJxWunuCFq0ve0NLjF+CMKS3y1RStLX0YxUkcByNA@mail.gmail.com>

That would be awesome (I love ipython notebook).


On Tue, Aug 12, 2014 at 11:20 AM, Sturla Molden <sturla.molden at gmail.com>
wrote:

> Brett Calcott <brett.calcott at gmail.com>
> wrote:
>
> > For someone who has bumbled around trying to use prange &amp; openmp on
> the mac
> > (but successfully used python threading), this sounds great. Is there an
> > example of this somewhere that you can point us to?
>
> No, but I could make one :)
>
> ipython notebook?
>
>
> Sturla
>
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> https://mail.python.org/mailman/listinfo/cython-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20140812/40b350de/attachment.html>

From stefan_ml at behnel.de  Tue Aug 12 17:32:26 2014
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 12 Aug 2014 17:32:26 +0200
Subject: [Cython] aritmetic with arrays in Cython
In-Reply-To: <224570377429548214.833522sturla.molden-gmail.com@news.gmane.org>
References: <53E45A84.2070308@behnel.de> <5698943.UtWdqSqk1C@sleeky>
 <53E4A639.4000507@behnel.de>
 <CAPWbfpTFXVq5KwWWFCkag9rEb=hBJU5=AmcFdMUd5w74eMeeWw@mail.gmail.com>
 <53E50A9A.5050805@behnel.de> <53E510C7.4090903@googlemail.com>
 <CAPWbfpS1fFWcgU=Z_B8teHwcChyJRLOWYqaRvqLe2VZU4exWLQ@mail.gmail.com>
 <CAPWbfpRPQi5-oyWGkR8JVsoeh128XhgVwjS1Dht_1nvujO6pDw@mail.gmail.com>
 <1984369421429388619.771407sturla.molden-gmail.com@news.gmane.org>
 <CAPWbfpQ+Ym1N+U436T1YxOOixBmMh0tyG=mMQWyuwnW2+w_U2A@mail.gmail.com>
 <53E9AA39.8050805@behnel.de>
 <224570377429548214.833522sturla.molden-gmail.com@news.gmane.org>
Message-ID: <53EA338A.7080702@behnel.de>

Sturla Molden schrieb am 12.08.2014 um 17:18:
> But using Eigen will taint the output with Eigen's license

Which is ok for many users, definitely those who only run their own code
locally. And the others can eventually use a different backend.


> OTOH, Cilkplus is just a compiler extension for C and C++. 

Which isn't available in all compilers. But anyway, if you write the code,
I have no problem with Cilkplus becoming the first backend to support.

Stefan


From leon at bottou.org  Tue Aug 12 18:06:23 2014
From: leon at bottou.org (Leon Bottou)
Date: Tue, 12 Aug 2014 12:06:23 -0400
Subject: [Cython] [Re] OpenMP thread private variable not recognized (bug
	report + discussion)
Message-ID: <034201cfb647$5ec00710$1c401530$@bottou.org>

On Tue, 12 Aug 2014 14:26:31, Sturla Molden wrote:
> Cython does not do an error here:[...
> - i is recognized as private
> - r is recognized as reduction
> - w is (correctly) recognized as shared

Not according to the documentation.
http://docs.cython.org/src/userguide/parallelism.html documentation for
cython.parallel.parallel says "A contained prange will be a worksharing loop
that is not parallel, so any variable assigned to in the parallel section is
also private to the prange. Variables that are private in the parallel block
are unavailable after the parallel block.".  Variable w is such a variable.

Furthermore, if cython is correct, why does GCC report an error on the
cython generated C code?  

My point here is that there is a bug because (a) cython does not behave as
documented, and (b) it generates invalid C code despite not reporting an
error.

> Personally I prefer to avoid OpenMP and just use Python threads and an
> internal function (closure) or an internal class. If you start to use
OpenMP,
> Apple's libdispatch ("GCD"), Intel TBB, or Intel clikplus, you will soon
discover
> that they are all variations over the same theme: a thread pool and a
closure.

I am making heavy uses of OpenBlas which also uses OpenMP.
Using the same queue manager prevents lots of CPU provisioning problem.
Using multiple queue managers in the same code does not work as well because
they are not aware of what the other one is doing.


- L.





From matej at laitl.cz  Tue Aug 12 18:13:35 2014
From: matej at laitl.cz (=?utf-8?B?TWF0xJtq?= Laitl)
Date: Tue, 12 Aug 2014 18:13:35 +0200
Subject: [Cython] aritmetic with arrays in Cython
In-Reply-To: <224570377429548214.833522sturla.molden-gmail.com@news.gmane.org>
References: <53E45A84.2070308@behnel.de> <53E9AA39.8050805@behnel.de>
 <224570377429548214.833522sturla.molden-gmail.com@news.gmane.org>
Message-ID: <2444809.Z1WVDFPHXM@sleeky>

On Tuesday 12 of August 2014 15:18:12 Sturla Molden wrote:
> But using Eigen will taint the output with Eigen's license, since the Eigen
> library is statically linked.

There is no such thing as "Eigen library". Eigen is fully implemented in 
header files. Cython would just generate C++ code that uses Eigen API.

Also, Eigen is now licensed under MPL2, which is file-level and permissive one 
[1]. I don't understand what you mean by 'taint' above and how this could 
restrict someone license-wise.

Regards,
	Mat?j

[1] http://eigen.tuxfamily.org/index.php?title=Licensing_FAQ

From matej at laitl.cz  Tue Aug 12 18:22:46 2014
From: matej at laitl.cz (=?utf-8?B?TWF0xJtq?= Laitl)
Date: Tue, 12 Aug 2014 18:22:46 +0200
Subject: [Cython] aritmetic with arrays in Cython
In-Reply-To: <224570377429548214.833522sturla.molden-gmail.com@news.gmane.org>
References: <53E45A84.2070308@behnel.de> <53E9AA39.8050805@behnel.de>
 <224570377429548214.833522sturla.molden-gmail.com@news.gmane.org>
Message-ID: <54149091.WtWzdJ8d3W@sleeky>

On Tuesday 12 of August 2014 15:18:12 Sturla Molden wrote:
> Another thing to observe is that Eigen depends on the C++ compiler to elide
> temporary arrays.

Either I don't understand you, or you don't understand Eigen. Eigen overloads 
operator=() to circumvent need for temporary arrays. It is *not* an 
optimisation on the compiler part, the code is explicitly written to do so, 
where it is beneficial for performance [1].

	Mat?j

[1] http://eigen.tuxfamily.org/dox/TopicLazyEvaluation.html

From stefan_ml at behnel.de  Tue Aug 12 18:34:54 2014
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 12 Aug 2014 18:34:54 +0200
Subject: [Cython] Cython 0.21 beta 1 released
In-Reply-To: <53DA81B2.10805@behnel.de>
References: <53DA81B2.10805@behnel.de>
Message-ID: <53EA422E.8040003@behnel.de>

Hi all,

given the overwhelming lack of negative feedback on the latest alpha
release, and the recent changes that made it into the master branch, I
think it's time for a first beta release.

http://www.cython.org/release/Cython-0.21b1.tar.gz

http://www.cython.org/release/Cython-0.21b1.tar.gz.asc

This release features faster calls to C implemented Python functions and
methods in simple (non-kwargs) cases, so you might see a net win if your
code calls a lot into previously unoptimised builtins or other native
extension modules (including Cython modules). There's also a bit of a
speedup for Python method calls in general, in addition to the long list of
improvements that were already in the last alpha.

Complete changelog:

https://github.com/cython/cython/blob/ebafd54916e1fe976a282e77b4d59577c793bfca/CHANGES.rst

Please give it a try with your code. Any regressions that you find now have
a chance of being fixed before the final release. Positive success stories
are also appreciated.

Have fun,

Stefan


From sturla.molden at gmail.com  Tue Aug 12 18:42:51 2014
From: sturla.molden at gmail.com (Sturla Molden)
Date: Tue, 12 Aug 2014 16:42:51 +0000 (UTC)
Subject: [Cython] [Re] OpenMP thread private variable not recognized
	(bug report + discussion)
References: <034201cfb647$5ec00710$1c401530$@bottou.org>
Message-ID: <1959653602429554276.599389sturla.molden-gmail.com@news.gmane.org>

"Leon Bottou" <leon at bottou.org> wrote:

> I am making heavy uses of OpenBlas which also uses OpenMP.
> Using the same queue manager prevents lots of CPU provisioning problem.
> Using multiple queue managers in the same code does not work as well because
> they are not aware of what the other one is doing.

Normally OpenBLAS is built without OpenMP. Also, OpenMP is not fork safe
(cf. multiprocessing) but OpenBLAS' own threadpool is. So it is recommended
to build OpenBLAS without OpenMP dependency.

That is: If you build OpenBLAS with OpenMP, numpy.dot will hang if used
together with multiprocessing.


Sturla


From dave.hirschfeld at gmail.com  Wed Aug 13 08:53:01 2014
From: dave.hirschfeld at gmail.com (Dave Hirschfeld)
Date: Wed, 13 Aug 2014 06:53:01 +0000 (UTC)
Subject: [Cython]
	=?utf-8?q?=5BRe=5D_OpenMP_thread_private_variable_not_re?=
	=?utf-8?q?cognized=09=28bug_report_+_discussion=29?=
References: <034201cfb647$5ec00710$1c401530$@bottou.org>
 <1959653602429554276.599389sturla.molden-gmail.com@news.gmane.org>
Message-ID: <loom.20140813T084727-208@post.gmane.org>

Sturla Molden <sturla.molden at ...> writes:

> 
> "Leon Bottou" <leon at ...> wrote:
> 
> > I am making heavy uses of OpenBlas which also uses OpenMP.
> > Using the same queue manager prevents lots of CPU provisioning problem.
> > Using multiple queue managers in the same code does not work as well 
because
> > they are not aware of what the other one is doing.
> 
> Normally OpenBLAS is built without OpenMP. Also, OpenMP is not fork safe
> (cf. multiprocessing) but OpenBLAS' own threadpool is. So it is recommended
> to build OpenBLAS without OpenMP dependency.
> 
> That is: If you build OpenBLAS with OpenMP, numpy.dot will hang if used
> together with multiprocessing.
> 
> Sturla
> 
> 

Just wanting to clarify that it's only the GNU OpenMP implementation that 
isn't fork-safe? AFAIK the intel OpenMP runtime is and will at some stage be 
available in the master branch of clang.

-Dave


From sturla.molden at gmail.com  Wed Aug 13 10:17:35 2014
From: sturla.molden at gmail.com (Sturla Molden)
Date: Wed, 13 Aug 2014 08:17:35 +0000 (UTC)
Subject: [Cython] [Re] OpenMP thread private variable not re
	cognized	(bug report + discussion)
References: <034201cfb647$5ec00710$1c401530$@bottou.org>
 <1959653602429554276.599389sturla.molden-gmail.com@news.gmane.org>
 <loom.20140813T084727-208@post.gmane.org>
Message-ID: <26249161429610111.944699sturla.molden-gmail.com@news.gmane.org>

Dave Hirschfeld <dave.hirschfeld at gmail.com>
wrote:

> Just wanting to clarify that it's only the GNU OpenMP implementation that 
> isn't fork-safe? AFAIK the intel OpenMP runtime is and will at some stage be 
> available in the master branch of clang.

That is correct.


From leon at bottou.org  Wed Aug 13 20:21:02 2014
From: leon at bottou.org (Leon Bottou)
Date: Wed, 13 Aug 2014 14:21:02 -0400
Subject: [Cython] [Re] OpenMP thread private variable not recognized (bug
	report + discussion)
Message-ID: <005301cfb723$5878bdc0$096a3940$@bottou.org>

> > I am making heavy uses of OpenBlas which also uses OpenMP.
> > Using the same queue manager prevents lots of CPU provisioning problem.
> > Using multiple queue managers in the same code does not work as well
> > because they are not aware of what the other one is doing.
> 
> Normally OpenBLAS is built without OpenMP. Also, OpenMP is not fork safe
> (cf. multiprocessing) but OpenBLAS' own threadpool is. So it is
recommended
> to build OpenBLAS without OpenMP dependency.
> 
> That is: If you build OpenBLAS with OpenMP, numpy.dot will hang if used
> together with multiprocessing.

I am effectively using a version of openblas built with openmp because
Debian used to compile openblas this way. They seem to have reverted now.
Note than I cannot use python multiprocessing because my threads work on a
very large state vector.  My current solution is to use python threading and
nogil cython compiled routines but this sometimes lead to weird effects
provisioning threads. 

This is why I wanted to try the pure openmp solution and found the
aforementioned bug in cython.parallel.

Is there somebody actively trying to make cython.parallel work correctly?
- If yes, then my bug report should be of interest to this person.
- If no, then one should avoid (and possibly deprecate) cython.parallel and
find other ways to do things. 

Thanks to the replies.

- L.



From andriy.kornatskyy at live.com  Thu Aug 14 07:20:03 2014
From: andriy.kornatskyy at live.com (Andriy Kornatskyy)
Date: Thu, 14 Aug 2014 08:20:03 +0300
Subject: [Cython] error LNK2001: unresolved external symbol PyInit_init
Message-ID: <BLU436-SMTP18793784E39798639D0225B91E80@phx.gbl>

Here is an issue when trying to install wheezy.template (it uses cythonize) into environment with cython.

Host: windows 7 64bit Python version: 3.4.1 32 bit wheezy.template version: 0.1.151

When installing either from pip3 or downloading the source with python3 setup.py install

c:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\BIN\link.exe /DLL /nologo /INCREMENTAL:NO /LIBPATH:C:\Python34\libs /LIBPATH:C:\Python34\PCbuild /EXPORT: PyInit_init build\temp.win32-3.4\Release\src\wheezy\template__init.obj /O UT:build\lib.win32-3.4\wheezy\template__init.pyd /IMPLIB:build\temp.win32-3.4 \Release\src\wheezy\template__init.lib /MANIFESTFILE:build\temp.win32-3.4\Rel ease\src\wheezy\template__init.pyd.manifest
LINK : error LNK2001: unresolved external symbol PyInit_init
build\temp.win32-3.4\Release\src\wheezy\template__init__.lib : fatal error LNK1120: 1 unresolved externals
error: command 'c:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\BIN\link.exe' failed with exit status 1120

Some details are here:
https://bitbucket.org/akorn/wheezy.template/issue/10/installation-on-windows-fails

Thanks.

Andriy Kornatskyy


From arfrever.fta at gmail.com  Thu Aug 14 10:18:17 2014
From: arfrever.fta at gmail.com (Arfrever Frehtes Taifersar Arahesis)
Date: Thu, 14 Aug 2014 10:18:17 +0200
Subject: [Cython] Cython 0.21 beta 1 released
In-Reply-To: <53EA422E.8040003@behnel.de>
References: <53DA81B2.10805@behnel.de> <53EA422E.8040003@behnel.de>
Message-ID: <201408141018.19490.Arfrever.FTA@gmail.com>

Results of test suite of Cython 0.21 beta 1 with different versions of Python:
  Python 2.6: 1 error, 2 failures
  Python 2.7: 1 error, 0 failures
  Python 3.2: 1 error, 0 failures
  Python 3.3: 1 error, 0 failures
  Python 3.4: 1 error, 0 failures

The 1 error is the same with all versions of Python.
Output with Python 2.6:

======================================================================
ERROR: compiling (c) and running bufaccess
----------------------------------------------------------------------
Traceback (most recent call last):
  File "runtests.py", line 1022, in run
    check_thread_termination()
  File "runtests.py", line 1633, in check_thread_termination
    raise PendingThreadsError("left-over threads found after running test")
PendingThreadsError: left-over threads found after running test

======================================================================
FAIL: Doctest: double_dealloc_T796
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib64/python2.6/doctest.py", line 2163, in runTest
    raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for double_dealloc_T796
  File "/tmp/Cython-0.21b1/tests-2.6/run/c/double_dealloc_T796/double_dealloc_T796.so", line unknown line number, in double_dealloc_T796

----------------------------------------------------------------------
File "/tmp/Cython-0.21b1/tests-2.6/run/c/double_dealloc_T796/double_dealloc_T796.so", line ?, in double_dealloc_T796
Failed example:
    del x
Expected:
    SimpleGarbage(1) __dealloc__
    Collector.__dealloc__
    collect 0
Got:
    SimpleGarbage(1) __dealloc__
    Collector.__dealloc__
    collect 128


======================================================================
FAIL: Doctest: double_dealloc_T796
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib64/python2.6/doctest.py", line 2163, in runTest
    raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for double_dealloc_T796
  File "/tmp/Cython-0.21b1/tests-2.6/run/cpp/double_dealloc_T796/double_dealloc_T796.so", line unknown line number, in double_dealloc_T796

----------------------------------------------------------------------
File "/tmp/Cython-0.21b1/tests-2.6/run/cpp/double_dealloc_T796/double_dealloc_T796.so", line ?, in double_dealloc_T796
Failed example:
    del x
Expected:
    SimpleGarbage(1) __dealloc__
    Collector.__dealloc__
    collect 0
Got:
    SimpleGarbage(1) __dealloc__
    Collector.__dealloc__
    collect 128


----------------------------------------------------------------------
Ran 8832 tests in 6785.081s

FAILED (failures=2, errors=1)
ALL DONE


--
Arfrever Frehtes Taifersar Arahesis
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: This is a digitally signed message part.
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20140814/ccca11e3/attachment.sig>

From stefan_ml at behnel.de  Thu Aug 14 18:29:42 2014
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 14 Aug 2014 18:29:42 +0200
Subject: [Cython] Guido on type annotations in Py3
Message-ID: <53ECE3F6.1000207@behnel.de>

He's picked up the idea from a recent keynote talk at EuroPython that
there's this thing called mypy that provides type annotations.

http://thread.gmane.org/gmane.comp.python.ideas/28619

Stefan

From stefan_ml at behnel.de  Sat Aug 16 08:40:04 2014
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 16 Aug 2014 08:40:04 +0200
Subject: [Cython] python-ideas discussion on "multiple dispatch",
	also operator overloading
Message-ID: <53EEFCC4.5050805@behnel.de>

Hi,

there's a discussion going on on python-ideas about generic functions and
multiple dispatch. Might be interesting for the fused functions
implementation in Cython.

http://thread.gmane.org/gmane.comp.python.ideas/28790/focus=28848

Cython can definitely improve its own dispatch implementation.

The reply I linked to is by Nick Coghlan, who also makes an interesting
comment on operator overloading methods and NotImplemented:

"""
A *lot* of folks make the mistake
of raising TypeError or NotImplementedError directly in their operator
overload implementations, rather than returning the NotImplemented
singleton that tells the interpreter to try the other type. Even some
of the CPython builtins get that wrong, since the sq_concat and
sq_repeat slots in C don't properly support the type coercion dance,
so you *have* to raise the exception yourself if you're only
implementing those without implementing nb_add and nb_mul (types
defined in Python automatically populate both sets of C level slots if
you define __add__ or __mul__).
"""

I think Cython could help its users here to a certain extent, if only by
warning about raising TypeError explicitly in these methods and encouraging
better behaviour.

Stefan

From kwmsmith at gmail.com  Sat Aug 16 19:51:30 2014
From: kwmsmith at gmail.com (Kurt Smith)
Date: Sat, 16 Aug 2014 12:51:30 -0500
Subject: [Cython] Automatic conversion with fixed-size C arrays
In-Reply-To: <CADiQ+QDpPBzLg9N617JLL7TbzMwfWtpsRifzOg=E5Y4YBkYwxw@mail.gmail.com>
References: <CALpTaNYskf94RzTExj4JEn5sO7b6ryYBhVFrFYAzxZ96R7onRg@mail.gmail.com>
 <CADiQ+QBJ_dSCXV-hVRmyqMAmhApQfVDE4B7H9DVZzP8L8N+S+A@mail.gmail.com>
 <CALpTaNb+nGeOfPgaLjsLxrWP8ug=y_2f7+uTKs5_hUQnGaWx5Q@mail.gmail.com>
 <CADiQ+QDC5SF2xRhzZCH_ufPjn_59VuJX+gVyMJpzizK+S9-zTA@mail.gmail.com>
 <CADiQ+QBwn7=_5bXAO+-3-m31LG13ASo5Df8-QWum6FhTsDpS=A@mail.gmail.com>
 <CALpTaNbF4UypSmVAATcGRiJnno7A642Nd8uAPVz25rpAELD2Ew@mail.gmail.com>
 <CALpTaNYj9QhVwNk1JEZgn5Sq7YdApXYZJR_6=Fuj2umfk6xPfQ@mail.gmail.com>
 <CADiQ+QDWe_mJPPFQV0FJO3zrRuDDNLTcPmWAv7Z-NTK72XjGoQ@mail.gmail.com>
 <CALpTaNZ3nd878=okszUj36E8JJazc19fnFny8NW-ZCQBfJoMMw@mail.gmail.com>
 <CALpTaNZS9127ti_PPui4gJpA8tK6AHH2LpLKqXLR9HQ-uP-O1g@mail.gmail.com>
 <CADiQ+QDpPBzLg9N617JLL7TbzMwfWtpsRifzOg=E5Y4YBkYwxw@mail.gmail.com>
Message-ID: <CALpTaNa_ScHQPuih4wHn=dE=MPZ-6rAtKQqgVwnQye85137vsg@mail.gmail.com>

Apologies for the radio silence: the original project that motivated this
work has a deadline, so I've had to set this aside for now.  I fully intend
to return to this once that's done.

On Thu, Jul 31, 2014 at 11:34 PM, Robert Bradshaw <robertwb at gmail.com>
wrote:

> We can move to the PR, makes it easier to do line commenting.
>
> On Tue, Jul 29, 2014 at 9:54 PM, Kurt Smith <kwmsmith at gmail.com> wrote:
> > On Mon, Jul 28, 2014 at 11:29 PM, Kurt Smith <kwmsmith at gmail.com> wrote:
> >>
> >> On Mon, Jul 28, 2014 at 11:02 AM, Robert Bradshaw <robertwb at gmail.com>
> >> wrote:
> >>>
> >>> On Sun, Jul 27, 2014 at 9:34 PM, Kurt Smith <kwmsmith at gmail.com>
> wrote:
> >>>
> >>> > I have sorted out all the code
> >>> > generation pieces and nearly have a minimal example working, and will
> >>> > have
> >>> > something for you to review soon.  From what I can tell, Cython has
> >>> > some
> >>> > baked-in assumptions when it comes to assignment statements.  Telling
> >>> > Cython
> >>> > that in this case, no, don't assign to something at the C level,
> >>> > instead
> >>> > generate a function call and pass in the lvalue by reference takes
> some
> >>> > doing.
> >>>
> >>> Yeah, I'm not too surprised about that.
> >>>
> >>
> >> I have an initial version working for assigning a Python object to a
> >> fixed-size array.
> >
> >
> > https://github.com/cython/cython/pull/308
> >
> > Do you prefer that we continue discussion on this thread, or on the PR?
> >
> > _______________________________________________
> > cython-devel mailing list
> > cython-devel at python.org
> > https://mail.python.org/mailman/listinfo/cython-devel
> >
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> https://mail.python.org/mailman/listinfo/cython-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20140816/f7ca15a8/attachment.html>

From marcus.brinkmann at ruhr-uni-bochum.de  Sun Aug 17 06:07:54 2014
From: marcus.brinkmann at ruhr-uni-bochum.de (Marcus Brinkmann)
Date: Sun, 17 Aug 2014 06:07:54 +0200
Subject: [Cython] bug: constructor declarations not working
In-Reply-To: <53E9827C.2070609@ruhr-uni-bochum.de>
References: <53E9827C.2070609@ruhr-uni-bochum.de>
Message-ID: <53F02A9A.4000408@ruhr-uni-bochum.de>

Hi,

ok, this is not as easy as I originally thought.  I fixed a couple of 
things that prevented the suggested syntax from working, but the linking 
stage has additional challenges, as there is currently no support for 
cimporting a native cppclass from one module to another (akin to the 
"api" feature or the "public" symbol import function feature).  It's 
also not sure how this should work exactly, so I am giving up.

I settled on a work around using "cdef public" C functions and making 
use of cimport and the automatic symbol import feature.  This does not 
give access to the full type, but it's sufficient for my use case.

The following two changes were somewhat useful in my explorations:

1. Fix the "this" pointer misreference.  This is an actual bug, so I 
made a pull request:

https://github.com/cython/cython/pull/316

2. The missing constructor declaration generated when cimporting the pxd 
file is easy to fix by not testing for "defining or name != "<init>") in 
CppClassScope.  However, this breaks a test case, because 
NewExprNode.inferType adds the implicit default constructor under the 
name "<init>" without a proper cname.  So I made a larger change that 
fixes this, too:

https://github.com/lambdafu/cython/tree/lambdafu/cpp_declare_constructor

I didn't make a pull request, because currently cimport'ing an external 
cppclass declaration is not useful without some way to add the runtime 
dependency as well...

Thanks and sorry for the noise.  It wasn't entirely clear to me what the 
state of cppclass support in Cython is until I read the source code.

Marcus


On 08/12/2014 04:57 AM, Marcus Brinkmann wrote:
> Hi,
>
> I want to declare and define C++ classes from Cython.  Sure, I could
> write them in native C++, but it's just small glue code, and I prefer to
> let Cython handle the GIL and all the other good stuff.
>
> First, for reference, the following works (c.f.
> tests/run/cpp_classes_def.pyx):
>
> == cython --cplus example1.pyx ============
> cdef cppclass Foo "Foo":
>    int _foo
>    __init__(int foo) nogil:
>      this._foo = foo
>    __dealloc__() nogil:
>      pass
> ===========================================
>
> This will emit a declaration for Foo and the implementations of the
> constructor and destructor (output is truncated to show relevant parts
> only):
>
> struct Foo {
>    int _foo;
>     Foo(int);
>    virtual  ~Foo(void);
> };
>   Foo::Foo(int __pyx_v_foo) {
>    this->_foo = __pyx_v_foo;
> }
>   Foo::~Foo(void) {
> }
>
> Now, I want to export the class to other cython files.  The following
> does not work, and I think it's a bug:
>
> == example2.pxd ===========================
> cdef extern cppclass Foo:
>    int _foo
>    __init__(int foo)
>    __dealloc__()
> ===========================================
>
> == cython --cplus example2.pyx ============
> cdef cppclass Foo:
>    __init__(int foo) nogil:
>      this._foo = foo
>    __dealloc__() nogil:
>      pass
> ===========================================
>
> == cython --cplus example3.pyx ============
> from example2 cimport Foo
>
> cdef Foo* foo_p = new Foo(2)
> ===========================================
>
> This fails for example2 with an obscure error message:
>
> $ cython --cplus example2.pyx
> example2.pyx:3:8: undeclared name not builtin: this
> ...more spurious errors...
>
> For example3, Cython runs through but trying to compile shows the actual
> problem (which you can also see in example2 if you shift things around a
> bit):
>
> $ g++ -fPIC -shared -o example3.so -I /usr/include/python2.7 example3.cpp
> example3.cpp: In function ?void initexample3()?:
> example3.cpp:775:38: error: no matching function for call to
> ?Foo::Foo(int)?
>     __pyx_v_8example3_foo_p = new Foo(2);
>
> This is in the generated code:
>
> /*--- Type declarations ---*/
> struct Foo;
> struct Foo {
>    int _foo;
>    virtual  ~Foo(void);
> };
>
> It's missing the constructor declaration!
>
> I traced this a bit down the Compiler/Symtab.py and found this part in
> CppClassScope::declare_var:
>
> 4e07fc52 (Robert Bradshaw      2012-08-21 00:46:00 -0700 2112) if name
> != "this" and (defining or name != "<init>"):
> 4e07fc52 (Robert Bradshaw      2012-08-21 00:46:00 -0700 2113)
> self.var_entries.append(entry)
>
> Here defining is 0, so the declaration is skipped.  I don't know Cython
> internals, so I am hoping someone who does can fix this easily given the
> above information or point me in the right direction.
>
> Thanks a lot for Cython, it's very useful!
>
> Marcus Brinkmann
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> https://mail.python.org/mailman/listinfo/cython-devel


From robertwb at gmail.com  Sun Aug 17 07:12:08 2014
From: robertwb at gmail.com (Robert Bradshaw)
Date: Sat, 16 Aug 2014 22:12:08 -0700
Subject: [Cython] bug: constructor declarations not working
In-Reply-To: <53F02A9A.4000408@ruhr-uni-bochum.de>
References: <53E9827C.2070609@ruhr-uni-bochum.de>
 <53F02A9A.4000408@ruhr-uni-bochum.de>
Message-ID: <CADiQ+QDL41gqu3vpzytJ5Rh2QVRiu09+MgBYC8Ce6kMGsr2PrQ@mail.gmail.com>

On Sat, Aug 16, 2014 at 9:07 PM, Marcus Brinkmann
<marcus.brinkmann at ruhr-uni-bochum.de> wrote:
> Hi,
>
> ok, this is not as easy as I originally thought.  I fixed a couple of things
> that prevented the suggested syntax from working, but the linking stage has
> additional challenges, as there is currently no support for cimporting a
> native cppclass from one module to another (akin to the "api" feature or the
> "public" symbol import function feature).  It's also not sure how this
> should work exactly, so I am giving up.
>
> I settled on a work around using "cdef public" C functions and making use of
> cimport and the automatic symbol import feature.  This does not give access
> to the full type, but it's sufficient for my use case.
>
> The following two changes were somewhat useful in my explorations:
>
> 1. Fix the "this" pointer misreference.  This is an actual bug, so I made a
> pull request:
>
> https://github.com/cython/cython/pull/316
>
> 2. The missing constructor declaration generated when cimporting the pxd
> file is easy to fix by not testing for "defining or name != "<init>") in
> CppClassScope.  However, this breaks a test case, because
> NewExprNode.inferType adds the implicit default constructor under the name
> "<init>" without a proper cname.  So I made a larger change that fixes this,
> too:
>
> https://github.com/lambdafu/cython/tree/lambdafu/cpp_declare_constructor
>
> I didn't make a pull request, because currently cimport'ing an external
> cppclass declaration is not useful without some way to add the runtime
> dependency as well...
>
> Thanks and sorry for the noise.  It wasn't entirely clear to me what the
> state of cppclass support in Cython is until I read the source code.

Thanks.

As you observed, defining C++ classes in Cython is still somewhat of a
beta feature.

> On 08/12/2014 04:57 AM, Marcus Brinkmann wrote:
>>
>> Hi,
>>
>> I want to declare and define C++ classes from Cython.  Sure, I could
>> write them in native C++, but it's just small glue code, and I prefer to
>> let Cython handle the GIL and all the other good stuff.
>>
>> First, for reference, the following works (c.f.
>> tests/run/cpp_classes_def.pyx):
>>
>> == cython --cplus example1.pyx ============
>> cdef cppclass Foo "Foo":
>>    int _foo
>>    __init__(int foo) nogil:
>>      this._foo = foo
>>    __dealloc__() nogil:
>>      pass
>> ===========================================
>>
>> This will emit a declaration for Foo and the implementations of the
>> constructor and destructor (output is truncated to show relevant parts
>> only):
>>
>> struct Foo {
>>    int _foo;
>>     Foo(int);
>>    virtual  ~Foo(void);
>> };
>>   Foo::Foo(int __pyx_v_foo) {
>>    this->_foo = __pyx_v_foo;
>> }
>>   Foo::~Foo(void) {
>> }
>>
>> Now, I want to export the class to other cython files.  The following
>> does not work, and I think it's a bug:
>>
>> == example2.pxd ===========================
>> cdef extern cppclass Foo:
>>    int _foo
>>    __init__(int foo)
>>    __dealloc__()
>> ===========================================
>>
>> == cython --cplus example2.pyx ============
>> cdef cppclass Foo:
>>    __init__(int foo) nogil:
>>      this._foo = foo
>>    __dealloc__() nogil:
>>      pass
>> ===========================================
>>
>> == cython --cplus example3.pyx ============
>> from example2 cimport Foo
>>
>> cdef Foo* foo_p = new Foo(2)
>> ===========================================
>>
>> This fails for example2 with an obscure error message:
>>
>> $ cython --cplus example2.pyx
>> example2.pyx:3:8: undeclared name not builtin: this
>> ...more spurious errors...
>>
>> For example3, Cython runs through but trying to compile shows the actual
>> problem (which you can also see in example2 if you shift things around a
>> bit):
>>
>> $ g++ -fPIC -shared -o example3.so -I /usr/include/python2.7 example3.cpp
>> example3.cpp: In function ?void initexample3()?:
>> example3.cpp:775:38: error: no matching function for call to
>> ?Foo::Foo(int)?
>>     __pyx_v_8example3_foo_p = new Foo(2);
>>
>> This is in the generated code:
>>
>> /*--- Type declarations ---*/
>> struct Foo;
>> struct Foo {
>>    int _foo;
>>    virtual  ~Foo(void);
>> };
>>
>> It's missing the constructor declaration!
>>
>> I traced this a bit down the Compiler/Symtab.py and found this part in
>> CppClassScope::declare_var:
>>
>> 4e07fc52 (Robert Bradshaw      2012-08-21 00:46:00 -0700 2112) if name
>> != "this" and (defining or name != "<init>"):
>> 4e07fc52 (Robert Bradshaw      2012-08-21 00:46:00 -0700 2113)
>> self.var_entries.append(entry)
>>
>> Here defining is 0, so the declaration is skipped.  I don't know Cython
>> internals, so I am hoping someone who does can fix this easily given the
>> above information or point me in the right direction.
>>
>> Thanks a lot for Cython, it's very useful!
>>
>> Marcus Brinkmann
>> _______________________________________________
>> cython-devel mailing list
>> cython-devel at python.org
>> https://mail.python.org/mailman/listinfo/cython-devel
>
>
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> https://mail.python.org/mailman/listinfo/cython-devel

From sturla.molden at gmail.com  Wed Aug 20 19:25:22 2014
From: sturla.molden at gmail.com (Sturla Molden)
Date: Wed, 20 Aug 2014 17:25:22 +0000 (UTC)
Subject: [Cython] error LNK2001: unresolved external symbol PyInit_init
References: <BLU436-SMTP18793784E39798639D0225B91E80@phx.gbl>
Message-ID: <958940216430248213.197885sturla.molden-gmail.com@news.gmane.org>

To build a Python extension for Win64 you must define the symbol MS_WIN64,
typically -DMS_WIN64.

Sturla


Andriy Kornatskyy <andriy.kornatskyy at live.com> wrote:
> Here is an issue when trying to install wheezy.template (it uses
> cythonize) into environment with cython.
> 
> Host: windows 7 64bit Python version: 3.4.1 32 bit wheezy.template version: 0.1.151
> 
> When installing either from pip3 or downloading the source with python3 setup.py install
> 
> c:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\BIN\link.exe /DLL
> /nologo /INCREMENTAL:NO /LIBPATH:C:\Python34\libs
> /LIBPATH:C:\Python34\PCbuild /EXPORT: PyInit_init
> build\temp.win32-3.4\Release\src\wheezy\template__init.obj /O
> UT:build\lib.win32-3.4\wheezy\template__init.pyd
> /IMPLIB:build\temp.win32-3.4 \Release\src\wheezy\template__init.lib
> /MANIFESTFILE:build\temp.win32-3.4\Rel ease\src\wheezy\template__init.pyd.manifest
> LINK : error LNK2001: unresolved external symbol PyInit_init
> build\temp.win32-3.4\Release\src\wheezy\template__init__.lib : fatal
> error LNK1120: 1 unresolved externals
> error: command 'c:\Program Files (x86)\Microsoft Visual Studio
> 10.0\VC\BIN\link.exe' failed with exit status 1120
> 
> Some details are here:
> https://bitbucket.org/akorn/wheezy.template/issue/10/installation-on-windows-fails
> 
> Thanks.
> 
> Andriy Kornatskyy


From ian.h.bell at gmail.com  Thu Aug 21 09:00:12 2014
From: ian.h.bell at gmail.com (Ian Bell)
Date: Thu, 21 Aug 2014 09:00:12 +0200
Subject: [Cython] build failure on windows with 0.21b1 windows py27 x64
Message-ID: <CAJQnXJd=qxoRmwn4FBWnHpfR+=j4Y3fYzWPrF8GFBOgbfaHBXw@mail.gmail.com>

 A snippet of my failure:

lex\Scanners.c(6876) : error C2121: '#' : invalid character : possibly the
result of a macro expansion
lex\Scanners.c(6876) : error C2146: syntax error : missing ')' before
identifier 'ifdef'
lex\Scanners.c(6876) : error C2121: '#' : invalid character : possibly the
result of a macro expansion
lex\Scanners.c(6878) : error C2059: syntax error : 'else'
lex\Scanners.c(6888) : error C2059: syntax error : '}'
lex\Scanners.c(6902) : error C2121: '#' : invalid character : possibly the
result of a macro expansion
lex\Scanners.c(6902) : error C2121: '#' : invalid character : possibly the
result of a macro expansion
soft Visual Studio 9.0\VC\BIN\amd64\cl.exe"' failed with exit status 2

Anyone run into similar problems?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20140821/7868458a/attachment.html>

From dalcinl at gmail.com  Thu Aug 21 13:30:10 2014
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 21 Aug 2014 14:30:10 +0300
Subject: [Cython] error LNK2001: unresolved external symbol PyInit_init
In-Reply-To: <958940216430248213.197885sturla.molden-gmail.com@news.gmane.org>
References: <BLU436-SMTP18793784E39798639D0225B91E80@phx.gbl>
 <958940216430248213.197885sturla.molden-gmail.com@news.gmane.org>
Message-ID: <CAEcYPwDv4WOdU9P2GEHoDbGcpLNpBNcNUAUJ1-MZZjvR6-vqoQ@mail.gmail.com>

On 20 August 2014 20:25, Sturla Molden <sturla.molden at gmail.com> wrote:
> To build a Python extension for Win64 you must define the symbol MS_WIN64,
> typically -DMS_WIN64.
>

Sturla, isn't this supposed to be handled in pyconfig.h (at least when
using MSVC) ? I see these lines in PC/pyconfig.h (from Python
sources):

/* MSVC defines _WINxx to differentiate the windows platform types

   Note that for compatibility reasons _WIN32 is defined on Win32
   *and* on Win64. For the same reasons, in Python, MS_WIN32 is
   defined on Win32 *and* Win64. Win32 only code must therefore be
   guarded as follows:
    #if defined(MS_WIN32) && !defined(MS_WIN64)
   Some modules are disabled on Itanium processors, therefore we
   have MS_WINI64 set for those targets, otherwise MS_WINX64
*/
#ifdef _WIN64
#define MS_WIN64
#endif


-- 
Lisandro Dalcin
---------------
CIMEC (UNL/CONICET)
Predio CONICET-Santa Fe
Colectora RN 168 Km 472, Paraje El Pozo
3000 Santa Fe, Argentina
Tel: +54-342-4511594 (ext 1016)
Tel/Fax: +54-342-4511169

From sturla.molden at gmail.com  Sat Aug 23 20:17:26 2014
From: sturla.molden at gmail.com (Sturla Molden)
Date: Sat, 23 Aug 2014 20:17:26 +0200
Subject: [Cython] error LNK2001: unresolved external symbol PyInit_init
In-Reply-To: <CAEcYPwDv4WOdU9P2GEHoDbGcpLNpBNcNUAUJ1-MZZjvR6-vqoQ@mail.gmail.com>
References: <BLU436-SMTP18793784E39798639D0225B91E80@phx.gbl>
 <958940216430248213.197885sturla.molden-gmail.com@news.gmane.org>
 <CAEcYPwDv4WOdU9P2GEHoDbGcpLNpBNcNUAUJ1-MZZjvR6-vqoQ@mail.gmail.com>
Message-ID: <ltalrn$hel$1@ger.gmane.org>

On 21/08/14 13:30, Lisandro Dalcin wrote:

> Sturla, isn't this supposed to be handled in pyconfig.h (at least when
> using MSVC) ? I see these lines in PC/pyconfig.h (from Python
> sources):

Yes, it is supposed to, but the error message suggest it was not and the 
compile line does not have it.

Sturla


From robertwb at gmail.com  Sun Aug 24 07:46:37 2014
From: robertwb at gmail.com (Robert Bradshaw)
Date: Sat, 23 Aug 2014 22:46:37 -0700
Subject: [Cython] Towards a formal Cython grammar
Message-ID: <CADiQ+QDi2NOJ4_pj0tViefG7UOEUO+X=5GUCa08RLy9rkrNQFw@mail.gmail.com>

I've started playing around with writing a grammar for Cython. As well
as formally defining the language (in particular, with respect to
Python) this should allow us to eventually move to using
parser-generators rather than ad-hoc hand written code and be useful
to  external tools (e.g. IDEs, linters, syntax highlighting, etc.)

I've posted what I have at
https://github.com/robertwb/cython/tree/grammar , in particular

https://github.com/robertwb/cython/blob/grammar/Cython/Parser/Grammar
https://github.com/robertwb/cython/compare/robertwb:3aa9056943f83a68cc9d9335f8a9c81e9a6f3f91...363bc162fd626203f832a33b6c736ff8b10f6086#diff-8b69afcfc588fde3d763f2ec670e42c2L1

Nothing beyond generating the raw parse tree is hooked up yet, and
even that requires an extra directive (formal_grammar=True). Building
and using the parser requires a source-built Python (it looks for the
pgen artifact Python uses to compile the grammar). We may or may not
want to stick with this approach long term (though if we do we might
ship the generated .c files). This parse tree is still pretty
low-level, we might want to also create something like Python.asdl to
give us a (closer) AST.

The grammar isn't complete, but should cover a most of the language
(over 3/4 of the test suite passes, and that explores a lot of the
corner cases). The most notable omissions are that it's using Python's
lexer, so doesn't have a token for '?' or the additional literal
string prefixes/int literal suffixes. Also, as the lexer clearly
doesn't understand includes, these are handled by inlining in a
preparsing step (which messes with line numbers).

The grammar could be tightened up as well. For example, this grammar
doesn't distinguish between valid pxd vs. pyx constructs, and allows
cdef statements within if statements (or even normal class
declarations).

I tried to restrict the existing grammar as little as possible, in
particular the only new illegal identifiers are 'cdef', 'ctypedef' and
(for ambiguity reasons new' and 'sizeof'). Also, the "cython" keywords
may not be used for identifiers that might be typed (e.g. "def
foo(int): ..." is not allowed).

The most significant departure from the existing "grammar" is that
rather than using C-style declarators, cdef declarations are of the
form "cdef [type] name." Thus one write the (already legal)

    cdef double[3] loc

and

    cdef double* a, b

declares two pointers. How to handle function pointers is still up in
the air, but I wouldn't be opposed to moving to a new syntax (e.g.
"(double, void*) -> double" inspired by Py3) for those. It disallows
empty "declarators" for parameters of function declarations (though we
could consider adding this back).

i think this would also be a nice chance to simplify the grammar, so
there are some intentional ommisions. Most notably, there are several
modifiers (e.g the __cdecl, __stdcall, __fastcall callspecs, maybe
inline, maybe even "with nogil", and the "cdef class Foo [object
object_struct_name, type type_object_name ]" spec for external
classes) that would make more sense as decorators. This would be
backwards incompatible, but these are not commonly used features and
fair warning (or even translators, I included a sed script to deal
with the common case of declarators mentioned above) could be given
and I think could be worth the simplification.

Thoughts?

- Robert

From dalcinl at gmail.com  Sun Aug 24 10:57:48 2014
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Sun, 24 Aug 2014 11:57:48 +0300
Subject: [Cython] error LNK2001: unresolved external symbol PyInit_init
In-Reply-To: <BLU436-SMTP18793784E39798639D0225B91E80@phx.gbl>
References: <BLU436-SMTP18793784E39798639D0225B91E80@phx.gbl>
Message-ID: <CAEcYPwA5MmbWBVuoi3ywTJ1gJjz8P71rt7K2U1x8j-6VmqhMBQ@mail.gmail.com>

On 14 August 2014 08:20, Andriy Kornatskyy <andriy.kornatskyy at live.com> wrote:
> When installing either from pip3 or downloading the source with python3 setup.py install

Could you please show us the full output of "python3 setup.py build" ?


-- 
Lisandro Dalcin
============
Research Scientist
Computer, Electrical and Mathematical Sciences & Engineering (CEMSE)
Numerical Porous Media Center (NumPor)
King Abdullah University of Science and Technology (KAUST)
http://numpor.kaust.edu.sa/

4700 King Abdullah University of Science and Technology
al-Khawarizmi Bldg (Bldg 1), Office # 4332
Thuwal 23955-6900, Kingdom of Saudi Arabia
http://www.kaust.edu.sa

Office Phone: +966 12 808-0459

From andriy.kornatskyy at live.com  Wed Aug 27 07:08:14 2014
From: andriy.kornatskyy at live.com (Andriy Kornatskyy)
Date: Wed, 27 Aug 2014 08:08:14 +0300
Subject: [Cython] error LNK2001: unresolved external symbol PyInit_init
In-Reply-To: <CAEcYPwA5MmbWBVuoi3ywTJ1gJjz8P71rt7K2U1x8j-6VmqhMBQ@mail.gmail.com>
References: <BLU436-SMTP18793784E39798639D0225B91E80@phx.gbl>
 <CAEcYPwA5MmbWBVuoi3ywTJ1gJjz8P71rt7K2U1x8j-6VmqhMBQ@mail.gmail.com>
Message-ID: <BLU437-SMTP94EDA37AA287367668B3F91DD0@phx.gbl>

Here is a link to complete output with python3.4:

https://bitbucket.org/akorn/wheezy.template/issue/10/installation-on-windows-fails#comment-12006152

Thanks.

Andriy Kornatskyy

On Aug 24, 2014, at 11:57 AM, Lisandro Dalcin <dalcinl at gmail.com> wrote:

> On 14 August 2014 08:20, Andriy Kornatskyy <andriy.kornatskyy at live.com> wrote:
>> When installing either from pip3 or downloading the source with python3 setup.py install
> 
> Could you please show us the full output of "python3 setup.py build" ?
> 
> 
> -- 
> Lisandro Dalcin
> ============
> Research Scientist
> Computer, Electrical and Mathematical Sciences & Engineering (CEMSE)
> Numerical Porous Media Center (NumPor)
> King Abdullah University of Science and Technology (KAUST)
> http://numpor.kaust.edu.sa/
> 
> 4700 King Abdullah University of Science and Technology
> al-Khawarizmi Bldg (Bldg 1), Office # 4332
> Thuwal 23955-6900, Kingdom of Saudi Arabia
> http://www.kaust.edu.sa
> 
> Office Phone: +966 12 808-0459
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> https://mail.python.org/mailman/listinfo/cython-devel


From sturla.molden at gmail.com  Wed Aug 27 10:47:01 2014
From: sturla.molden at gmail.com (Sturla Molden)
Date: Wed, 27 Aug 2014 08:47:01 +0000 (UTC)
Subject: [Cython] error LNK2001: unresolved external symbol PyInit_init
References: <BLU436-SMTP18793784E39798639D0225B91E80@phx.gbl>
 <CAEcYPwA5MmbWBVuoi3ywTJ1gJjz8P71rt7K2U1x8j-6VmqhMBQ@mail.gmail.com>
 <BLU437-SMTP94EDA37AA287367668B3F91DD0@phx.gbl>
Message-ID: <1044647977430821724.053344sturla.molden-gmail.com@news.gmane.org>

64-bit Python cannot be used to build 32-bit extensions. Yes, you can build
extensions for 32-bit Python on 64-bit Windows, but not with 64-bit Python.

Sturla



Andriy Kornatskyy <andriy.kornatskyy at live.com> wrote:
> Here is a link to complete output with python3.4:
> 
> https://bitbucket.org/akorn/wheezy.template/issue/10/installation-on-windows-fails#comment-12006152
> 
> Thanks.
> 
> Andriy Kornatskyy
> 
> On Aug 24, 2014, at 11:57 AM, Lisandro Dalcin
> <dalcinl at gmail.com> wrote:
> 
>> On 14 August 2014 08:20, Andriy Kornatskyy
>> <andriy.kornatskyy at live.com> wrote:
>>> When installing either from pip3 or downloading the source with python3 setup.py install
>> 
>> Could you please show us the full output of "python3 setup.py build" ?
>> 
>> 
>> -- 
>> Lisandro Dalcin
>> ============
>> Research Scientist
>> Computer, Electrical and Mathematical Sciences & Engineering (CEMSE)
>> Numerical Porous Media Center (NumPor)
>> King Abdullah University of Science and Technology (KAUST)
>> http://numpor.kaust.edu.sa/
>> 
>> 4700 King Abdullah University of Science and Technology
>> al-Khawarizmi Bldg (Bldg 1), Office # 4332
>> Thuwal 23955-6900, Kingdom of Saudi Arabia
>> http://www.kaust.edu.sa
>> 
>> Office Phone: +966 12 808-0459
>> _______________________________________________
>> cython-devel mailing list
>> cython-devel at python.org
>> https://mail.python.org/mailman/listinfo/cython-devel


From andriy.kornatskyy at live.com  Wed Aug 27 16:20:06 2014
From: andriy.kornatskyy at live.com (Andriy Kornatskyy)
Date: Wed, 27 Aug 2014 17:20:06 +0300
Subject: [Cython] error LNK2001: unresolved external symbol PyInit_init
In-Reply-To: <1044647977430821724.053344sturla.molden-gmail.com@news.gmane.org>
References: <BLU436-SMTP18793784E39798639D0225B91E80@phx.gbl>
 <CAEcYPwA5MmbWBVuoi3ywTJ1gJjz8P71rt7K2U1x8j-6VmqhMBQ@mail.gmail.com>
 <BLU437-SMTP94EDA37AA287367668B3F91DD0@phx.gbl>
 <1044647977430821724.053344sturla.molden-gmail.com@news.gmane.org>
Message-ID: <BLU436-SMTP105C1DB8A66BEC040F9F9A791DD0@phx.gbl>

Sturla,

Per further details:
https://bitbucket.org/akorn/wheezy.template/issue/10/installation-on-windows-fails#comment-12018243

It is python 32 bit on 64 bit machine.

Python 3.4.1 (v3.4.1:c0e311e010fc, May 18 2014, 10:38:22) [MSC v.1600 32 bit (In
tel)] on win32

Thanks.

Andriy Kornatskyy

On Aug 27, 2014, at 11:47 AM, Sturla Molden <sturla.molden at gmail.com> wrote:

> 64-bit Python cannot be used to build 32-bit extensions. Yes, you can build
> extensions for 32-bit Python on 64-bit Windows, but not with 64-bit Python.
> 
> Sturla
> 
> 
> 
> Andriy Kornatskyy <andriy.kornatskyy at live.com> wrote:
>> Here is a link to complete output with python3.4:
>> 
>> https://bitbucket.org/akorn/wheezy.template/issue/10/installation-on-windows-fails#comment-12006152
>> 
>> Thanks.
>> 
>> Andriy Kornatskyy
>> 
>> On Aug 24, 2014, at 11:57 AM, Lisandro Dalcin
>> <dalcinl at gmail.com> wrote:
>> 
>>> On 14 August 2014 08:20, Andriy Kornatskyy
>>> <andriy.kornatskyy at live.com> wrote:
>>>> When installing either from pip3 or downloading the source with python3 setup.py install
>>> 
>>> Could you please show us the full output of "python3 setup.py build" ?
>>> 
>>> 
>>> -- 
>>> Lisandro Dalcin
>>> ============
>>> Research Scientist
>>> Computer, Electrical and Mathematical Sciences & Engineering (CEMSE)
>>> Numerical Porous Media Center (NumPor)
>>> King Abdullah University of Science and Technology (KAUST)
>>> http://numpor.kaust.edu.sa/
>>> 
>>> 4700 King Abdullah University of Science and Technology
>>> al-Khawarizmi Bldg (Bldg 1), Office # 4332
>>> Thuwal 23955-6900, Kingdom of Saudi Arabia
>>> http://www.kaust.edu.sa
>>> 
>>> Office Phone: +966 12 808-0459
>>> _______________________________________________
>>> cython-devel mailing list
>>> cython-devel at python.org
>>> https://mail.python.org/mailman/listinfo/cython-devel
> 
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> https://mail.python.org/mailman/listinfo/cython-devel


From gramaalexandruionut at gmail.com  Tue Aug 26 17:45:41 2014
From: gramaalexandruionut at gmail.com (Alexandru Ionut Grama)
Date: Tue, 26 Aug 2014 17:45:41 +0200
Subject: [Cython] [Desired feature] Process .pxd before .py
Message-ID: <CALeHuxbvxr4utt040BX2vpOxyfxEFovDkeVj6+7rjSJvAS42KA@mail.gmail.com>

Hello to all

I've start to use cython in order to use call some API modules written in
python from C/C++. Those modules written in python must remain untouched,
and all .py files should be "decorated" with their counterpart .pxd files.
I would like to mark as public some API functions from .py files, following
the indications from http://docs.cython.org/src/tutorial/pxd_files.html.
I've started to write a little example:

suma_alex.py:
def suma_Alex(a,b):
        return a+b

suma_alex.pxd:
cdef public int suma_Alex(int a,int b)


The combination of those two files should turn into something like this for
the compiler:

cdef public int suma_Alex(int a,int b):
      return a+b

On generated .h file I should obtain a function prototype like this:
__PYX_EXTERN_C DL_IMPORT(int) suma_Alex(int, int);

Instead of that, I obtain a prototype like this:
__PYX_EXTERN_C DL_IMPORT(int) __pyx_f_10suma_alex_suma_Alex(int, int);

Digging into code and executing cythonize with pdb, I followed that is
executed a pipeline for .py file before .pxd file's pipeline. Correct me if
I'm wrong, but this makes the name of .c function be generated according to
py file instead of pxd file. That makes an add of string
"__pyx_f_10suma_alex_" before function name, because the compiler doesn't
know about "public" key (on python doesn't exists). Including public key on
a pxd file for this function makes the compiler remove "static" key and
generate a .h file, but doesn't change the name prototype into a simple
"suma_Alex".

If I use cython on a pyx file containing the code below, it works perfectly
as expected.
cdef public int suma_Alex(int a,int b):
      return a+b

The questions are:
- I'm doing what I want to do on a right way?
- Is this feature included on cython development version? I've tried on
0.20.2
- If is not implemented, I want to colaborate implementing it. Could you
please provide me some guidelines with the methods/modules that I should
modify?

King regards,
Alexandru.

-- 


*---------------------------------------------------------------Alexandru
Ionut Grama**email: gramaalexandruionut at gmail.com
<gramaalexandruionut at gmail.com>*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20140826/a1e147dd/attachment.html>

From stefan_ml at behnel.de  Thu Aug 28 09:29:04 2014
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 28 Aug 2014 09:29:04 +0200
Subject: [Cython] error LNK2001: unresolved external symbol PyInit_init
In-Reply-To: <BLU437-SMTP94EDA37AA287367668B3F91DD0@phx.gbl>
References: <BLU436-SMTP18793784E39798639D0225B91E80@phx.gbl>
 <CAEcYPwA5MmbWBVuoi3ywTJ1gJjz8P71rt7K2U1x8j-6VmqhMBQ@mail.gmail.com>
 <BLU437-SMTP94EDA37AA287367668B3F91DD0@phx.gbl>
Message-ID: <53FEDA40.6070405@behnel.de>

Andriy Kornatskyy schrieb am 27.08.2014 um 07:08:
> Here is a link to complete output with python3.4:
> https://bitbucket.org/akorn/wheezy.template/issue/10/installation-on-windows-fails#comment-12006152

What Lisandro meant with "complete" is "from entering the build command to
the point where it fails".

Stefan


From andriy.kornatskyy at live.com  Thu Aug 28 10:08:37 2014
From: andriy.kornatskyy at live.com (Andriy Kornatskyy)
Date: Thu, 28 Aug 2014 11:08:37 +0300
Subject: [Cython] error LNK2001: unresolved external symbol PyInit_init
In-Reply-To: <53FEDA40.6070405@behnel.de>
References: <BLU436-SMTP18793784E39798639D0225B91E80@phx.gbl>
 <CAEcYPwA5MmbWBVuoi3ywTJ1gJjz8P71rt7K2U1x8j-6VmqhMBQ@mail.gmail.com>
 <BLU437-SMTP94EDA37AA287367668B3F91DD0@phx.gbl> <53FEDA40.6070405@behnel.de>
Message-ID: <BLU436-SMTP125D6C03F5E76D4DECE5CDC91DA0@phx.gbl>

Stefan,

Hmm? you have a complete build log from the entering the build command up to the point where it fails which you can still find here:

https://bitbucket.org/akorn/wheezy.template/issue/10/installation-on-windows-fails#comment-12006152

Or am I missing something?

Thanks.

Andriy Kornatskyy

On Aug 28, 2014, at 10:29 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:

> Andriy Kornatskyy schrieb am 27.08.2014 um 07:08:
>> Here is a link to complete output with python3.4:
>> https://bitbucket.org/akorn/wheezy.template/issue/10/installation-on-windows-fails#comment-12006152
> 
> What Lisandro meant with "complete" is "from entering the build command to
> the point where it fails".
> 
> Stefan
> 
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> https://mail.python.org/mailman/listinfo/cython-devel


From stefan_ml at behnel.de  Thu Aug 28 12:46:46 2014
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 28 Aug 2014 12:46:46 +0200
Subject: [Cython] error LNK2001: unresolved external symbol PyInit_init
In-Reply-To: <BLU436-SMTP125D6C03F5E76D4DECE5CDC91DA0@phx.gbl>
References: <BLU436-SMTP18793784E39798639D0225B91E80@phx.gbl>
 <CAEcYPwA5MmbWBVuoi3ywTJ1gJjz8P71rt7K2U1x8j-6VmqhMBQ@mail.gmail.com>
 <BLU437-SMTP94EDA37AA287367668B3F91DD0@phx.gbl> <53FEDA40.6070405@behnel.de>
 <BLU436-SMTP125D6C03F5E76D4DECE5CDC91DA0@phx.gbl>
Message-ID: <53FF0896.4050509@behnel.de>

Andriy Kornatskyy schrieb am 28.08.2014 um 10:08:
> Hmm? you have a complete build log from the entering the build command up to the point where it fails which you can still find here:
> 
> https://bitbucket.org/akorn/wheezy.template/issue/10/installation-on-windows-fails#comment-12006152

Sorry, where? All I see is the command and then the linker command, no
Cython build call, no C compiler build call, no distutils output.

Stefan


From andriy.kornatskyy at live.com  Thu Aug 28 12:59:03 2014
From: andriy.kornatskyy at live.com (Andriy Kornatskyy)
Date: Thu, 28 Aug 2014 13:59:03 +0300
Subject: [Cython] error LNK2001: unresolved external symbol PyInit_init
In-Reply-To: <53FF0896.4050509@behnel.de>
References: <BLU436-SMTP18793784E39798639D0225B91E80@phx.gbl>
 <CAEcYPwA5MmbWBVuoi3ywTJ1gJjz8P71rt7K2U1x8j-6VmqhMBQ@mail.gmail.com>
 <BLU437-SMTP94EDA37AA287367668B3F91DD0@phx.gbl> <53FEDA40.6070405@behnel.de>
 <BLU436-SMTP125D6C03F5E76D4DECE5CDC91DA0@phx.gbl>
 <53FF0896.4050509@behnel.de>
Message-ID: <BLU436-SMTP24045D6940B1D3237765B1A91DA0@phx.gbl>

Stefan,

That is an output of build:

python setup.py build.

There is nothing else produced unless you tell how to make it more verbose to provide you what you expect to see. How else you want this to be built?

Thanks.

Andriy Kornatskyy

On Aug 28, 2014, at 1:46 PM, Stefan Behnel <stefan_ml at behnel.de> wrote:

> Andriy Kornatskyy schrieb am 28.08.2014 um 10:08:
>> Hmm? you have a complete build log from the entering the build command up to the point where it fails which you can still find here:
>> 
>> https://bitbucket.org/akorn/wheezy.template/issue/10/installation-on-windows-fails#comment-12006152
> 
> Sorry, where? All I see is the command and then the linker command, no
> Cython build call, no C compiler build call, no distutils output.
> 
> Stefan
> 
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> https://mail.python.org/mailman/listinfo/cython-devel


From stefan_ml at behnel.de  Thu Aug 28 16:57:44 2014
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 28 Aug 2014 16:57:44 +0200
Subject: [Cython] [Desired feature] Process .pxd before .py
In-Reply-To: <CALeHuxbvxr4utt040BX2vpOxyfxEFovDkeVj6+7rjSJvAS42KA@mail.gmail.com>
References: <CALeHuxbvxr4utt040BX2vpOxyfxEFovDkeVj6+7rjSJvAS42KA@mail.gmail.com>
Message-ID: <53FF4368.1020709@behnel.de>

Alexandru Ionut Grama schrieb am 26.08.2014 um 17:45:
> I've start to use cython in order to use call some API modules written in
> python from C/C++. Those modules written in python must remain untouched,
> and all .py files should be "decorated" with their counterpart .pxd files.
> I would like to mark as public some API functions from .py files, following
> the indications from http://docs.cython.org/src/tutorial/pxd_files.html.
> I've started to write a little example:
> 
> suma_alex.py:
> def suma_Alex(a,b):
>         return a+b
> 
> suma_alex.pxd:
> cdef public int suma_Alex(int a,int b)
> 
> 
> The combination of those two files should turn into something like this for
> the compiler:
> 
> cdef public int suma_Alex(int a,int b):
>       return a+b
> 
> On generated .h file I should obtain a function prototype like this:
> __PYX_EXTERN_C DL_IMPORT(int) suma_Alex(int, int);
> 
> Instead of that, I obtain a prototype like this:
> __PYX_EXTERN_C DL_IMPORT(int) __pyx_f_10suma_alex_suma_Alex(int, int);

Looks like a bug to me. There is a step in the pipeline that merges the
.pxd file declarations into those found in the main source file. Either the
"public" modifier is not properly copied over or the cname needs to be
updated when merging in the override declarations.

Stefan


From stefan_ml at behnel.de  Thu Aug 28 19:13:02 2014
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 28 Aug 2014 19:13:02 +0200
Subject: [Cython] error LNK2001: unresolved external symbol PyInit_init
In-Reply-To: <CAEcYPwA5MmbWBVuoi3ywTJ1gJjz8P71rt7K2U1x8j-6VmqhMBQ@mail.gmail.com>
References: <BLU436-SMTP18793784E39798639D0225B91E80@phx.gbl>
 <CAEcYPwA5MmbWBVuoi3ywTJ1gJjz8P71rt7K2U1x8j-6VmqhMBQ@mail.gmail.com>
Message-ID: <53FF631E.20105@behnel.de>

Lisandro Dalcin schrieb am 24.08.2014 um 10:57:
> On 14 August 2014 08:20, Andriy Kornatskyy wrote:
>> When installing either from pip3 or downloading the source with python3 setup.py install
> 
> Could you please show us the full output of "python3 setup.py build" ?

It's here:

http://pastebin.com/vZb2gAvL

The other modules build fine, only the package script fails to link. The
problem is the "/EXPORT:PyInit___init__" flag, whereas the real name of the
module init function is (or should be) "PyInit_template", which is the name
of the package.

Does anyone have a good idea how to fix this? Looks like a distutils issue.
I don't think package compilation was originally envisioned as something
extension module authors would want to do.

Does the "cythonize_script_package" test work on MSWindows? It does the
same thing.

Stefan


From stefan_ml at behnel.de  Thu Aug 28 20:11:53 2014
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 28 Aug 2014 20:11:53 +0200
Subject: [Cython] timeframe for subtype support in freelists? (was: Cython
 0.21 beta 1 released)
In-Reply-To: <e3bc357f-0f4c-4000-b3eb-861cbf3af304@googlegroups.com>
References: <53DA81B2.10805@behnel.de> <53EA422E.8040003@behnel.de>
 <e3bc357f-0f4c-4000-b3eb-861cbf3af304@googlegroups.com>
Message-ID: <53FF70E9.2020108@behnel.de>

Peter Schay schrieb am 19.08.2014 um 00:46:
> One question: Is there any expected time-frame for freelists to support 
> cdef subclasses?  Soon? Never?  Somewhere in between?

Hard to say. This isn't easy as supporting subtypes means that we need full
control over what the parent types are doing in their
allocation/deallocation chain, and that may not be entirely trivial to
determine. The current call chain isn't made for that.

In any case, no-one's currently working on this AFAIK.

Stefan


From stefan_ml at behnel.de  Thu Aug 28 20:34:58 2014
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 28 Aug 2014 20:34:58 +0200
Subject: [Cython] [cython-users] Cython 0.21 alpha 1 released
In-Reply-To: <53DA81B2.10805@behnel.de>
References: <53DA81B2.10805@behnel.de>
Message-ID: <53FF7652.6010503@behnel.de>

Hi,

here is a second beta:

http://www.cython.org/release/Cython-0.21b2.tar.gz

http://www.cython.org/release/Cython-0.21b2.tar.gz.asc

Since the last beta, there were a couple of improvements for the new method
call optimisation, as well as some fixes (usually noted in the relevant
mailing list threads already).

We might have forgotten to mention previously that inner functions are
supported inside of cdef functions in 0.21, a feature that I consider worth
noting (and testing :).

Complete changelog:

https://github.com/cython/cython/blob/d4b87c6d818406275905c87cf89479bebff53c6f/CHANGES.rst

This would be a good time to bump any pending bug reports that you think
should be considered for this release.

Stefan


From gramaalexandruionut at gmail.com  Thu Aug 28 20:35:04 2014
From: gramaalexandruionut at gmail.com (Alexandru Ionut Grama)
Date: Thu, 28 Aug 2014 20:35:04 +0200
Subject: [Cython] [Desired feature] Process .pxd before .py
In-Reply-To: <53FF4368.1020709@behnel.de>
References: <CALeHuxbvxr4utt040BX2vpOxyfxEFovDkeVj6+7rjSJvAS42KA@mail.gmail.com>
 <53FF4368.1020709@behnel.de>
Message-ID: <CALeHuxbH=yGmcX707nyyokvBM0D-Qp3KtUT1SZgie2+qv3SdnA@mail.gmail.com>

2014-08-28 16:57 GMT+02:00 Stefan Behnel <stefan_ml at behnel.de>:

> Alexandru Ionut Grama schrieb am 26.08.2014 um 17:45:
> > I've start to use cython in order to use call some API modules written in
> > python from C/C++. Those modules written in python must remain untouched,
> > and all .py files should be "decorated" with their counterpart .pxd
> files.
> > I would like to mark as public some API functions from .py files,
> following
> > the indications from http://docs.cython.org/src/tutorial/pxd_files.html.
> > I've started to write a little example:
> >
> > suma_alex.py:
> > def suma_Alex(a,b):
> >         return a+b
> >
> > suma_alex.pxd:
> > cdef public int suma_Alex(int a,int b)
> >
> >
> > The combination of those two files should turn into something like this
> for
> > the compiler:
> >
> > cdef public int suma_Alex(int a,int b):
> >       return a+b
> >
> > On generated .h file I should obtain a function prototype like this:
> > __PYX_EXTERN_C DL_IMPORT(int) suma_Alex(int, int);
> >
> > Instead of that, I obtain a prototype like this:
> > __PYX_EXTERN_C DL_IMPORT(int) __pyx_f_10suma_alex_suma_Alex(int, int);
>
> Looks like a bug to me.

There is a step in the pipeline that merges the
> .pxd file declarations into those found in the main source file. Either the
> "public" modifier is not properly copied over or the cname needs to be
> updated when merging in the override declarations.
>

Hi Stefan,
Thanks for the answer at first.
If is a bug, do you need more information in order to check it? Could you
tell me the methods that are used to merge the override declarations for
investigate by myself and create a patch to correct the bug?

Is there a workaround that maybe solve this?

King regards,
Alexandru



> Stefan
>
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> https://mail.python.org/mailman/listinfo/cython-devel
>



-- 


*---------------------------------------------------------------Alexandru
Ionut Grama**email: gramaalexandruionut at gmail.com
<gramaalexandruionut at gmail.com>*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20140828/9e64b5af/attachment-0001.html>

From stefan_ml at behnel.de  Thu Aug 28 20:39:54 2014
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 28 Aug 2014 20:39:54 +0200
Subject: [Cython] Cython 0.21 beta 1 released
In-Reply-To: <201408141018.19490.Arfrever.FTA@gmail.com>
References: <53DA81B2.10805@behnel.de> <53EA422E.8040003@behnel.de>
 <201408141018.19490.Arfrever.FTA@gmail.com>
Message-ID: <53FF777A.4030301@behnel.de>

Hi,

thanks for testing.

Arfrever Frehtes Taifersar Arahesis schrieb am 14.08.2014 um 10:18:
> Results of test suite of Cython 0.21 beta 1 with different versions of Python:
>   Python 2.6: 1 error, 2 failures
>   Python 2.7: 1 error, 0 failures
>   Python 3.2: 1 error, 0 failures
>   Python 3.3: 1 error, 0 failures
>   Python 3.4: 1 error, 0 failures
> 
> The 1 error is the same with all versions of Python.
> Output with Python 2.6:
> 
> ======================================================================
> ERROR: compiling (c) and running bufaccess
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File "runtests.py", line 1022, in run
>     check_thread_termination()
>   File "runtests.py", line 1633, in check_thread_termination
>     raise PendingThreadsError("left-over threads found after running test")
> PendingThreadsError: left-over threads found after running test
> 
> ======================================================================

There should be some output in the log where the test is run that shows
what threads are still running here. Could you look that up?


> ======================================================================
> FAIL: Doctest: double_dealloc_T796
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File "/usr/lib64/python2.6/doctest.py", line 2163, in runTest
>     raise self.failureException(self.format_failure(new.getvalue()))
> AssertionError: Failed doctest test for double_dealloc_T796
>   File "/tmp/Cython-0.21b1/tests-2.6/run/cpp/double_dealloc_T796/double_dealloc_T796.so", line unknown line number, in double_dealloc_T796
> 
> ----------------------------------------------------------------------
> File "/tmp/Cython-0.21b1/tests-2.6/run/cpp/double_dealloc_T796/double_dealloc_T796.so", line ?, in double_dealloc_T796
> Failed example:
>     del x
> Expected:
>     SimpleGarbage(1) __dealloc__
>     Collector.__dealloc__
>     collect 0
> Got:
>     SimpleGarbage(1) __dealloc__
>     Collector.__dealloc__
>     collect 128

This might just be a problem with the test. Does this patch help?

=========================================
diff -r f7b5d79a04e6 -r e2f2cf8efcc7 tests/run/double_dealloc_T796.pyx
--- a/tests/run/double_dealloc_T796.pyx Thu Aug 28 20:21:14 2014 +0200
+++ b/tests/run/double_dealloc_T796.pyx Thu Aug 28 20:31:06 2014 +0200
@@ -1,4 +1,5 @@
 """
+>>> gc.collect()
 >>> x = SimpleGarbage()
 SimpleGarbage(1) __cinit__
 >>> del x
=========================================

Stefan


From stefan_ml at behnel.de  Fri Aug 29 19:38:51 2014
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 29 Aug 2014 19:38:51 +0200
Subject: [Cython] [cython-users] Cython 0.21 beta 2 released
In-Reply-To: <53FF7652.6010503@behnel.de>
References: <53DA81B2.10805@behnel.de> <53FF7652.6010503@behnel.de>
Message-ID: <5400BAAB.9080004@behnel.de>

[ forgot to change the subject to attract attention :) ]

Stefan Behnel schrieb am 28.08.2014 um 20:34:
> Hi,
> 
> here is a second beta:
> 
> http://www.cython.org/release/Cython-0.21b2.tar.gz
> 
> http://www.cython.org/release/Cython-0.21b2.tar.gz.asc
> 
> Since the last beta, there were a couple of improvements for the new method
> call optimisation, as well as some fixes (usually noted in the relevant
> mailing list threads already).
> 
> We might have forgotten to mention previously that inner functions are
> supported inside of cdef functions in 0.21, a feature that I consider worth
> noting (and testing :).
> 
> Complete changelog:
> 
> https://github.com/cython/cython/blob/d4b87c6d818406275905c87cf89479bebff53c6f/CHANGES.rst
> 
> This would be a good time to bump any pending bug reports that you think
> should be considered for this release.
> 
> Stefan


From arfrever.fta at gmail.com  Sun Aug 31 10:29:05 2014
From: arfrever.fta at gmail.com (Arfrever Frehtes Taifersar Arahesis)
Date: Sun, 31 Aug 2014 10:29:05 +0200
Subject: [Cython] Cython 0.21 beta 1 released
In-Reply-To: <53FF777A.4030301@behnel.de>
References: <53DA81B2.10805@behnel.de>
 <201408141018.19490.Arfrever.FTA@gmail.com> <53FF777A.4030301@behnel.de>
Message-ID: <201408311029.06233.Arfrever.FTA@gmail.com>

2014-08-28 20:39 Stefan Behnel napisa?(a):
> Arfrever Frehtes Taifersar Arahesis schrieb am 14.08.2014 um 10:18:
> > Results of test suite of Cython 0.21 beta 1 with different versions of Python:
> >   Python 2.6: 1 error, 2 failures
> >   Python 2.7: 1 error, 0 failures
> >   Python 3.2: 1 error, 0 failures
> >   Python 3.3: 1 error, 0 failures
> >   Python 3.4: 1 error, 0 failures
> > 
> > The 1 error is the same with all versions of Python.
> > Output with Python 2.6:
> > 
> > ======================================================================
> > ERROR: compiling (c) and running bufaccess
> > ----------------------------------------------------------------------
> > Traceback (most recent call last):
> >   File "runtests.py", line 1022, in run
> >     check_thread_termination()
> >   File "runtests.py", line 1633, in check_thread_termination
> >     raise PendingThreadsError("left-over threads found after running test")
> > PendingThreadsError: left-over threads found after running test
> > 
> > ======================================================================
> 
> There should be some output in the log where the test is run that shows
> what threads are still running here. Could you look that up?

Output for Cython 0.21 beta 2:

compiling (c) and running bufaccess ... Doctest: bufaccess.__test__.acquire_failure1 ... ok
Doctest: bufaccess.__test__.acquire_failure2 ... ok
Doctest: bufaccess.__test__.acquire_failure3 ... ok
Doctest: bufaccess.__test__.acquire_failure4 ... ok
Doctest: bufaccess.__test__.acquire_failure5 ... ok
Doctest: bufaccess.__test__.acquire_nonbuffer1 ... ok
Doctest: bufaccess.__test__.acquire_nonbuffer2 ... ok
Doctest: bufaccess.__test__.acquire_raise ... ok
Doctest: bufaccess.__test__.acquire_release ... ok
Doctest: bufaccess.__test__.as_argument ... ok
Doctest: bufaccess.__test__.as_argument_defval ... ok
Doctest: bufaccess.__test__.as_argument_not_none ... ok
Doctest: bufaccess.__test__.assign_temporary_to_object ... ok
Doctest: bufaccess.__test__.assign_to_object ... ok
Doctest: bufaccess.__test__.basic_struct ... ok
Doctest: bufaccess.__test__.bufdefaults1 ... ok
Doctest: bufaccess.__test__.buffer_cast ... ok
Doctest: bufaccess.__test__.buffer_cast_fails ... ok
Doctest: bufaccess.__test__.buffer_nogil ... ok
Doctest: bufaccess.__test__.buffer_nogil_oob ... ok
Doctest: bufaccess.__test__.c_contig ... ok
Doctest: bufaccess.__test__.c_contig_2d ... ok
Doctest: bufaccess.__test__.cascaded_buffer_assignment ... ok
Doctest: bufaccess.__test__.cdef_assignment ... ok
Doctest: bufaccess.__test__.complex_dtype ... ok
Doctest: bufaccess.__test__.complex_inplace ... ok
Doctest: bufaccess.__test__.complex_struct_dtype ... ok
Doctest: bufaccess.__test__.complex_struct_inplace ... ok
Doctest: bufaccess.__test__.explicitly_release_buffer ... ok
Doctest: bufaccess.__test__.f_contig ... ok
Doctest: bufaccess.__test__.f_contig_2d ... ok
Doctest: bufaccess.__test__.forin_assignment ... ok
Doctest: bufaccess.__test__.get_int_2d ... ok
Doctest: bufaccess.__test__.get_int_2d_uintindex ... ok
Doctest: bufaccess.__test__.inplace_operators ... ok
Doctest: bufaccess.__test__.list_comprehension ... ok
Doctest: bufaccess.__test__.mixed_get ... ok
Doctest: bufaccess.__test__.nested_packed_struct ... ok
Doctest: bufaccess.__test__.nested_struct ... ok
Doctest: bufaccess.__test__.no_negative_indices ... ok
Doctest: bufaccess.__test__.packed_struct ... ok
Doctest: bufaccess.__test__.printbuf_float ... ok
Doctest: bufaccess.__test__.printbuf_int_2d ... ok
Doctest: bufaccess.__test__.printbuf_object ... ok
Doctest: bufaccess.__test__.printbuf_td_cy_int ... ok
Doctest: bufaccess.__test__.printbuf_td_h_cy_short ... ok
Doctest: bufaccess.__test__.printbuf_td_h_double ... ok
Doctest: bufaccess.__test__.printbuf_td_h_short ... ok
Doctest: bufaccess.__test__.printbuf_td_h_ushort ... ok
Doctest: bufaccess.__test__.readonly ... ok
Doctest: bufaccess.__test__.safe_get ... ok
Doctest: bufaccess.__test__.set_int_2d ... ok
Doctest: bufaccess.__test__.strided ... ok
Doctest: bufaccess.__test__.test_inplace_assignment ... ok
Doctest: bufaccess.__test__.tuple_buffer_assignment1 ... ok
Doctest: bufaccess.__test__.tuple_buffer_assignment2 ... ok
Doctest: bufaccess.__test__.typedbuffer1 ... ok
Doctest: bufaccess.__test__.typedbuffer2 ... ok
Doctest: bufaccess.__test__.unsafe_get ... ok
Doctest: bufaccess.__test__.unsafe_get_nonegative ... ok
Doctest: bufaccess.__test__.wraparound_directive ... ok
Doctest: bufaccess.__test__.writable ... ok
warning: left-over threads found after running test:
...<HistorySavingThread(Thread-1, started 140020768708352)>
ERROR



> > ======================================================================
> > FAIL: Doctest: double_dealloc_T796
> > ----------------------------------------------------------------------
> > Traceback (most recent call last):
> >   File "/usr/lib64/python2.6/doctest.py", line 2163, in runTest
> >     raise self.failureException(self.format_failure(new.getvalue()))
> > AssertionError: Failed doctest test for double_dealloc_T796
> >   File "/tmp/Cython-0.21b1/tests-2.6/run/cpp/double_dealloc_T796/double_dealloc_T796.so", line unknown line number, in double_dealloc_T796
> > 
> > ----------------------------------------------------------------------
> > File "/tmp/Cython-0.21b1/tests-2.6/run/cpp/double_dealloc_T796/double_dealloc_T796.so", line ?, in double_dealloc_T796
> > Failed example:
> >     del x
> > Expected:
> >     SimpleGarbage(1) __dealloc__
> >     Collector.__dealloc__
> >     collect 0
> > Got:
> >     SimpleGarbage(1) __dealloc__
> >     Collector.__dealloc__
> >     collect 128
> 
> This might just be a problem with the test. Does this patch help?
> 
> =========================================
> diff -r f7b5d79a04e6 -r e2f2cf8efcc7 tests/run/double_dealloc_T796.pyx
> --- a/tests/run/double_dealloc_T796.pyx Thu Aug 28 20:21:14 2014 +0200
> +++ b/tests/run/double_dealloc_T796.pyx Thu Aug 28 20:31:06 2014 +0200
> @@ -1,4 +1,5 @@
>  """
> +>>> gc.collect()
>  >>> x = SimpleGarbage()
>  SimpleGarbage(1) __cinit__
>  >>> del x
> =========================================

This patch does not help (for Cython 0.21 beta 2).

--
Arfrever Frehtes Taifersar Arahesis
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: This is a digitally signed message part.
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20140831/29c87e83/attachment.sig>