[Cython] some pull requests for 0.16?

[mark florisson]

On 20 March 2012 18:51, Stefan Behnel <stefan_ml at behnel.de> wrote:
> mark florisson, 20.03.2012 17:40:
>> On 18 March 2012 11:58, Stefan Behnel wrote:
>>> I put up two new pull requests on github:
>>>
>>> Implementation of PEP 380 (yield from):
>>> https://github.com/cython/cython/pull/96
>>>
>>> Rewrite of dict iteration:
>>> https://github.com/cython/cython/pull/95
>>>
>>> Given that the release of 0.16 has currently slowed down a bit, and given
>>> that these are really nice features, could someone (and especially Mark, as
>>> the responsible release manager) take a look at them and give an opinion
>>> regarding their suitability for 0.16?
>>
>> Sorry about the slowdown, my BT internet connection at home got closed
>> off, so working on a release is rather hard, hopefully that will be
>> fixed within the next few days. Anyway, during the sprints Dag worked
>> on the numpy attributes accesses to generate non-deprecated numpy
>> code, I'm not sure how far along that is and if we can get that in as
>> well, or have it wait until 0.16.1.
>
> I wouldn't mind if my changes waited for 0.16.1 as well, as long as that's
> not too long from now. There will certainly be things to fix after the
> release anyway.
>
>
>> I worked on enhancing fused types runtime dispatching (e.g. select the
>> right specialization at runtime from the argument types), especially
>> on matching buffers. It's also a bit of a rewrite as the complexity
>> grew quite a bit and ndarray and fused types were broken in subtle
>> ways (I would do the pull request now if I hadn't left the code at
>> home :).
>
> I took a tiny look at that when I wrote a test case and was surprised to
> see that it imports "sys" inside of the generated function. Looks like
> there's a bit left to optimise.
>
> I faintly remember a blog post (or more than one) by PJ Eby when he wrote a
> "generic functions" module. It uses type dispatching as well, and he wrote
> about the ways he found to tune that. Might be an interesting read.
>

Thanks, I'll look into that. Maybe the modules could be bound as
locals through default arguments.

>> Finally, there is a serious bug in the reversed(range()) optimization
>> that Mike Graham discovered when working on the range() optimization
>> with a runtime step, which is basically wrong for any step that is not
>> 1 or -1 (it simply swaps the bounds with an offset by 1, but it
>> actually has to figure out what the actual last value was in the range
>> and use that as the start instead). I personally consider this a
>> blocker, so I'll try contacting Mike and see if he is still
>> (interested in) working on that.
>
> Interesting. Yes, I can well imagine that being wrong. These things are
> exceedingly tricky.
>
> However, it's easy to disable the optimisation for a specific case - likely
> easier than fixing it.
>
> Also, I don't consider this a blocker. It's not like it works in the
> current release, so it's not a regression compared to that.
>

Yes, it's not a regression, but I find it hard to release something
that is so broken, and I feel it kind of represents a lack of care
which may make people believe Cython can't be taken seriously.

>> As for the two pull requests, both are quite large, but the dict
>> iteration rewrite is more like an enhancement whereas the 'yield from'
>> is really a new (somewhat big) feature. I would personally prefer to
>> merge only enhancements and bug fixes, but not new features.
>
> Funny. I would have proposed the opposite, because I consider the "yield
> from" implementation (as a new language feature) safer than the dict
> iteration (which impacts existing code). As I said, I wouldn't mind
> postponing both to 0.16.1, but I think it would be better to get them out
> and have people use them, so that we can also fix the bugs earlier.
>

A good point. My reasoning was that we want less new functionality per
release, and keeping 'yield from' out of 0.16 makes us push out the
next release earlier as we will have pending functionality we want to
get out there. If the dict iteration is tested well enough (and it
looked to me like that), then hopefully code in sage or lxml will
break if there are leftover bugs, or our next beta might catch them
(or some). If you do want it in now, then that's fine with me as well,
you have a point.

>> If you
>> feel confident enough that the dict iteration pull request is ready to
>> be merged, then please do so.
>
> I'll take another look over it and decide. Thanks.
>
>
>> We can do an 0.16.1 release with the
>> 'yield from', the 'lnotab' line number mapping from code objects to
>> Cython source
>
> Note that that's not useful yet - it requires changes to the code that
> *uses* the code objects as well, to a) actually reuse the code objects (iff
> C line numbers are disabled in traces) and b) write the proper relative
> line into the byte code offset field. I just put it up to have a place
> where we can discuss further changes.
>
>
>> and probably the boundscheck and wraparound
>> optimizations I started working on (and other things if and when they
>> come up). We can release 0.16.1 somewhat shortly after releasing 0.16.
>> I think a shorter release cycle with less changes would be beneficial
>> and give us swifter feedback about bugs, missing features, wanted
>> features, etc.
>
> +1
>
>
>> So I propose that when we merged dict iteration, fused types
>> enhancements and the range bug fix, we do another beta release. We (I)
>> also need to look into the test failures on Gentoo.
>
> Whatever you choose.
>
> In case the PyPy developers manage to get one change in that I'd like to
> see on their side (two new C-API functions for exception handling), I'll
> also push some more Cython changes based on it that I have lying around.
> That would hugely improve the compatibility for both.
>
> In general, I started using the "CYTHON_COMPILING_IN_..." #defines wherever
> they made sense, and specifically where we rely on CPython internals and
> also in some places where we use borrowed references, especially long-lived
> ones. PyPy can't handle those very well (and certainly not efficiently) due
> to its garbage collector and its emulated ref-counting. I can well imagine
> that that's similar for IronPython.
>
> Stefan
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel