[Cython] [cython-users] Problems with "private" functions/methods in Cython 3.0a2

[da-woods]

Hi,

(Sent to both cython-devel and cython-users. However, I'm not subscribed 
to users I think, so please forward if it doesn't reach there).

On closer inspection I believe both are following Python behaviour (and 
thus behaving correctly). In Python any 
`__name_preceded_by_double_underscores` used within a class ends up 
looking up `_Classname__name_preceded_by_double_underscores`. Therefore 
in case 1 it's looking up `_Bar__foo` and not finding it.

Therefore, I think it is the "correct" thing to do.

This proved a pain with the `__pyx_unpickle` functions that Cython 
generates. In the end I just disabled it for `__pyx_` names, which I 
guess could be called hypocrisy .

David


On 23/04/2020 18:47, Stefan Behnel wrote:
> Hi,
>
> thanks for the very detailed report.
>
> I'm CC-ing cython-devel, but let's keep the discussion for this on the
> cython-users list.
>
> The background here is Github issue 1382 and PR 3123
>
> https://github.com/cython/cython/issues/1382
>
> https://github.com/cython/cython/pull/3123
>
> This was newly merged for alpha 2, and I consider it somewhat provisional.
> It makes "private" Python names follow Python semantics, but if that poses
> excessive breakage for existing code, then we'll have to find a pragmatic
> way to deal with it.
>
>
> 'Toni Barth' via cython-users schrieb am 23.04.20 um 19:07:
>> I just updated Cython from 3.0a1 to 3.0a2 in one of my projects and
>> noticed that it seems to introduce issues with private
>> functions/methods, means methods with leading __.
>>
>> I noticed two cases which now fail in my project. I'm however unsure if
>> thats supposed to happen or if its a problem with the new Cython build.
>>
>>
>> Case 1: private function not declared
>>
>>
>> crash.pxd:
>>
>>
>> cpdef __foo()
>>
>> cdef class Bar:
>>    cpdef run(Bar self)
>>
>>
>> crash.pyx:
>>
>>
>> cpdef __foo():
>>    print("foo")
>>
>> cdef class Bar:
>>    cpdef run(Bar self):
>>      __foo()
>>
>>      print("bar")
>>
>>
>> Running this with the following command:
>>
>> py -3 -c "import pyximport;pyximport.install();import
>> crash;crash.Bar().run()"
>>
>>
>> Result:
>>
>> crash.c
>> C:\Program Files (x86)\Microsoft Visual
>> Studio\Shared\Python36_64\lib\site-packages\Cython\Compiler\Main.py:344:
>> FutureWarning: Cython directive 'language_level' not set, using '3str'
>> for now (Py3). This has changed from earlier releases! File:
>> D:\crash\crash.pxd
>>    tree = Parsing.p_module(s, pxd, full_module_name)
>>
>> Error compiling Cython file:
>> ------------------------------------------------------------
>> ...
>> cpdef __foo():
>>    print("foo")
>>
>> cdef class Bar:
>>    cpdef run(Bar self):
>>      __foo()
>>     ^
>> ------------------------------------------------------------
>>
>> crash.pyx:6:4: undeclared name not builtin: __foo
> At least this looks like a bug to me. Global module names should not be
> impacted by this change.
>
>
>> Error compiling Cython file:
>> ------------------------------------------------------------
>> ...
>> cpdef __foo():
>>    print("foo")
>>
>> cdef class Bar:
>>    cpdef run(Bar self):
>>      __foo()
>>     ^
>> ------------------------------------------------------------
>>
>> crash.pyx:6:4: '__foo' redeclared
>>
>> Error compiling Cython file:
>> ------------------------------------------------------------
>> ...
>> cpdef __foo()
>>             ^
>> ------------------------------------------------------------
>>
>> crash.pxd:1:12: Previous declaration is here
> Lovely :)
>
>
>
>> Another problem, probably related, occurs when trying to override a
>> private method in a derived class:
>>
>>
>> Case 2: overriding private class methods
>>
>>
>> base.pxd:
>>
>>
>> cdef class Base:
>>    cdef void __foo(Base self)
>>    cpdef run(Base self)
>>
>>
>> base.pyx:
>>
>>
>> cdef class Base:
>>
>>    cdef void __foo(Base self):
>>      print("foo!")
>>   
>>    cpdef run(Base self):
>>      self.__foo()
>>
>>
>> extend.pxd:
>>
>>
>> from base cimport Base
>>
>> cdef class Extend(Base):
>>    pass
>>
>>
>> extend.pyx:
>>
>>
>> cdef class Extend(Base):
>>
>>    cdef void __foo(Extend self):
>>      print("bar")
>>
>> Running with the following command:
>>
>> py -3 -c "import pyximport;pyximport.install();import
>> extend;extend.Extend().run()"
>>
>>
>> Results in the following output:
>>
>>
>> extend.c
>> C:\Program Files (x86)\Microsoft Visual
>> Studio\Shared\Python36_64\lib\site-packages\Cython\Compiler\Main.py:344:
>> FutureWarning: Cython directive 'language_level' not set, using '3str'
>> for now (Py3). This has changed from earlier releases! File:
>> D:\crash\extend.pxd
>>    tree = Parsing.p_module(s, pxd, full_module_name)
>>
>> Error compiling Cython file:
>> ------------------------------------------------------------
>> ...
>> cdef class Extend(Base):
>>
>>    cdef void __foo(Extend self):
>>        ^
>> ------------------------------------------------------------
>>
>> extend.pyx:3:7: C method '_Extend__foo' not previously declared in
>> definition part of extension type 'Extend'
> This, OTOH, is intended. Private methods are not meant to be inherited and
> overridden. That's what makes them "private" :) That's consistent with
> Python semantics.
>
> I think it's somewhat up for debate if this should apply in the same way to
> cdef classes as for Python classes. But from a language design point of
> view, having this kind of difference between the two seems a bad idea.
>
> Do you feel like argueing against this? :)
>
> Stefan
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> https://mail.python.org/mailman/listinfo/cython-devel