[Cython] (Possible) bug: module level global vars end up in upper scope?

Stefan Behnel stefan_ml at behnel.de
Wed Jul 27 14:42:06 EDT 2016


Yury V. Zaytsev schrieb am 25.07.2016 um 15:27:
> I've hit the following deviation from Python 2: when a Cython module has a
> global variable, somehow, upon importing the module, the global variables
> end up in the *current* (importing) module scope, rather than *imported*
> module scope, and, while normally relatively harmless, in the case of
> namedtuples, for instance, this has caused pickling errors in our
> production code.
> 
> test1.py / test2.pyx
> 
>     import collections
> 
>     Test = collections.namedtuple("Test", ["test"])
> 
> $ ipython
> Python 2.7.6 (default, Jun 22 2015, 17:58:13)
> IPython 1.2.1 -- An enhanced Interactive Python.
> 
> In [1]: import test1
> 
> In [2]: test1.Test
> Out[2]: test1.Test
> 
> In [3]: import test2
> 
> In [4]: test2.Test
> Out[4]: __main__.Test
> 
> I would appreciate any hints in the case that I'm deeply confused and doing
> something obviously wrong...

namedtuple has an impressively complicated and fragile implementation.
Amongst other things, it does this at the end:

    result.__module__ = _sys._getframe(1).f_globals.get(
                                             '__name__', '__main__')

Since Cython modules do not have frames (by default, for performance
reasons), the module it finds then points to the caller instead.

I wonder if we shouldn't consider the module init function a special
(enough) case here that is never performance critical, and just always
generate a frame for it. Later frame lookups would then still fail (so we'd
create somewhat of an inconsistency), but the case above looks like a
legitimate use case, and namedtuples are often (I guess in *most* cases)
created at module init time.

I created a ticket.

https://github.com/cython/cython/issues/536

As a work-around, I could only come up with a hack. You could create a
Python module, import and call into it from your Cython module, create the
namedtuple in Python, and then fix the __module__ reference of the
namedtuple class after the fact. Although I wonder when the insertion into
the module namespace happens. I couldn't find it on a quick look.

Stefan



More information about the cython-devel mailing list