From jdemeyer at cage.ugent.be Mon Nov 13 06:32:58 2017 From: jdemeyer at cage.ugent.be (Jeroen Demeyer) Date: Mon, 13 Nov 2017 12:32:58 +0100 Subject: [Cython] Cython master emits tons of 'foo' redeclared warnings Message-ID: <5A0982EA.3030404@cage.ugent.be> When compiling SageMath with Cython master, I get literally thousands of warnings like warning: sage/rings/polynomial/laurent_polynomial.pyx:177:10: 'number_of_terms' redeclared What is the meaning of this warning? As far as I know, Sage is not doing anything wrong here, so I feel like Cython is giving this warning erroneously. I regularly compile Sage with Cython master and it is the first time that I am seeing this. From jdemeyer at cage.ugent.be Mon Nov 13 06:41:38 2017 From: jdemeyer at cage.ugent.be (Jeroen Demeyer) Date: Mon, 13 Nov 2017 12:41:38 +0100 Subject: [Cython] Cython master emits tons of 'foo' redeclared warnings In-Reply-To: <5A0982EA.3030404@cage.ugent.be> References: <5A0982EA.3030404@cage.ugent.be> Message-ID: <5A0984F2.5080703@cage.ugent.be> Even Cython itself triggers this warning many times: warning: Cython/Compiler/Code.py:380:4: 'format_code' redeclared warning: Cython/Compiler/Code.py:442:4: 'none_or_sub' redeclared warning: Cython/Compiler/Code.py:693:4: 'new_label' redeclared warning: Cython/Compiler/Code.py:712:4: 'get_loop_labels' redeclared warning: Cython/Compiler/Code.py:717:4: 'set_loop_labels' redeclared warning: Cython/Compiler/Code.py:728:4: 'get_all_labels' redeclared warning: Cython/Compiler/Code.py:735:4: 'set_all_labels' redeclared warning: Cython/Compiler/Code.py:826:4: 'temps_in_use' redeclared warning: Cython/Compiler/Code.py:863:4: 'start_collecting_temps' redeclared warning: Cython/Compiler/Code.py:869:4: 'stop_collecting_temps' redeclared warning: Cython/Compiler/Code.py:930:4: 'get_py_string_const' redeclared warning: Cython/Compiler/FlowControl.py:74:4: 'empty' redeclared warning: Cython/Compiler/FlowControl.py:77:4: 'detach' redeclared warning: Cython/Compiler/FlowControl.py:86:4: 'add_child' redeclared warning: Cython/Compiler/FlowControl.py:94:4: 'empty' redeclared warning: Cython/Compiler/FlowControl.py:126:4: 'newblock' redeclared warning: Cython/Compiler/FlowControl.py:137:4: 'nextblock' redeclared warning: Cython/Compiler/FlowControl.py:151:4: 'is_tracked' redeclared warning: Cython/Compiler/FlowControl.py:158:4: 'is_statically_assigned' redeclared warning: Cython/Compiler/FlowControl.py:168:4: 'mark_position' redeclared warning: Cython/Compiler/FlowControl.py:173:4: 'mark_assignment' redeclared warning: Cython/Compiler/FlowControl.py:180:4: 'mark_argument' redeclared warning: Cython/Compiler/FlowControl.py:187:4: 'mark_deletion' redeclared warning: Cython/Compiler/FlowControl.py:194:4: 'mark_reference' redeclared warning: Cython/Compiler/FlowControl.py:204:4: 'normalize' redeclared warning: Cython/Compiler/FlowControl.py:227:4: 'initialize' redeclared warning: Cython/Compiler/FlowControl.py:263:4: 'map_one' redeclared warning: Cython/Compiler/FlowControl.py:763:4: 'mark_assignment' redeclared warning: Cython/Compiler/FlowControl.py:796:4: 'mark_position' redeclared warning: Cython/Compiler/Parsing.py:646:0: 'make_slice_node' redeclared warning: Cython/Compiler/Parsing.py:2422:0: 'p_c_base_type' redeclared warning: Cython/Compiler/Parsing.py:2751:0: 'p_c_declarator' redeclared warning: Cython/Compiler/Parsing.py:2943:0: 'p_c_arg_list' redeclared warning: Cython/Compiler/Parsing.py:3586:0: 'p_code' redeclared warning: Cython/Compiler/Parsing.py:3610:0: 'p_module' redeclared warning: Cython/Compiler/Parsing.py:3733:0: 'print_parse_tree' redeclared warning: Cython/Compiler/Pythran.py:21:0: 'is_pythran_supported_dtype' redeclared warning: Cython/Compiler/Pythran.py:121:0: 'to_pythran' redeclared warning: Cython/Compiler/Pythran.py:150:0: 'is_pythran_supported_type' redeclared warning: Cython/Compiler/Pythran.py:163:0: 'is_pythran_expr' redeclared warning: Cython/Compiler/Scanning.py:93:4: 'lookup' redeclared warning: Cython/Compiler/Scanning.py:400:4: 'indentation_action' redeclared warning: Cython/Compiler/Visitor.py:169:4: 'visit' redeclared warning: Cython/Compiler/Visitor.py:197:4: 'visitchildren' redeclared warning: Cython/Compiler/Visitor.py:247:4: 'visitchildren' redeclared warning: Cython/Compiler/Visitor.py:264:4: 'recurse_to_children' redeclared warning: Cython/Plex/Actions.py:13:4: 'same_as' redeclared warning: Cython/Plex/Actions.py:29:4: 'same_as' redeclared warning: Cython/Plex/Actions.py:50:4: 'same_as' redeclared warning: Cython/Plex/Actions.py:70:4: 'same_as' redeclared warning: Cython/Plex/Scanners.py:112:4: 'read' redeclared warning: Cython/StringIOTree.py:55:4: 'getvalue' redeclared warning: Cython/StringIOTree.py:60:4: 'copyto' redeclared warning: Cython/StringIOTree.py:69:4: 'commit' redeclared warning: Cython/StringIOTree.py:104:4: 'allmarkers' redeclared So I am going to assume that this warning is incorrect. From jdemeyer at cage.ugent.be Mon Nov 13 06:42:38 2017 From: jdemeyer at cage.ugent.be (Jeroen Demeyer) Date: Mon, 13 Nov 2017 12:42:38 +0100 Subject: [Cython] Cython master emits tons of 'foo' redeclared warnings In-Reply-To: <5A0982EA.3030404@cage.ugent.be> References: <5A0982EA.3030404@cage.ugent.be> Message-ID: <5A09852E.6060508@cage.ugent.be> See https://github.com/cython/cython/issues/1874 From matthew.brett at gmail.com Mon Nov 20 19:08:31 2017 From: matthew.brett at gmail.com (Matthew Brett) Date: Tue, 21 Nov 2017 00:08:31 +0000 Subject: [Cython] Drop Cython wheels for 2.6? Message-ID: Hi, The manylinux1 docker image has just stopped supporting Python 2.6, so we can no longer build Python 2.6 wheels without putting some hacks on top: https://github.com/pypa/manylinux/pull/125#issuecomment-345770870 Do y'all care about that? Cheers, Matthew From njs at vorpus.org Mon Nov 20 20:57:27 2017 From: njs at vorpus.org (Nathaniel Smith) Date: Mon, 20 Nov 2017 17:57:27 -0800 Subject: [Cython] Drop Cython wheels for 2.6? In-Reply-To: References: Message-ID: On Mon, Nov 20, 2017 at 4:08 PM, Matthew Brett wrote: > Hi, > > The manylinux1 docker image has just stopped supporting Python 2.6, so > we can no longer build Python 2.6 wheels without putting some hacks on > top: > > https://github.com/pypa/manylinux/pull/125#issuecomment-345770870 > > Do y'all care about that? To clarify, it's not so much that the docker image officially dropped support, as that pip dropped support and the last rebuild happened to pick that up, so we should figure out whether to drop it officially or what: https://github.com/pypa/manylinux/issues/126 -n -- Nathaniel J. Smith -- https://vorpus.org From robertwb at math.washington.edu Mon Nov 20 21:09:31 2017 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 20 Nov 2017 18:09:31 -0800 Subject: [Cython] Drop Cython wheels for 2.6? In-Reply-To: References: Message-ID: Wheels are a convenience; I'm OK with not going through extra hoops to support them for a version of Python retired years ago. Until we actually drop support for 2.6 entirely one can still install from source. On Mon, Nov 20, 2017 at 5:57 PM, Nathaniel Smith wrote: > On Mon, Nov 20, 2017 at 4:08 PM, Matthew Brett wrote: >> Hi, >> >> The manylinux1 docker image has just stopped supporting Python 2.6, so >> we can no longer build Python 2.6 wheels without putting some hacks on >> top: >> >> https://github.com/pypa/manylinux/pull/125#issuecomment-345770870 >> >> Do y'all care about that? > > To clarify, it's not so much that the docker image officially dropped > support, as that pip dropped support and the last rebuild happened to > pick that up, so we should figure out whether to drop it officially or > what: https://github.com/pypa/manylinux/issues/126 > > -n > > -- > Nathaniel J. Smith -- https://vorpus.org > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > https://mail.python.org/mailman/listinfo/cython-devel From jdemeyer at cage.ugent.be Thu Nov 23 09:32:13 2017 From: jdemeyer at cage.ugent.be (Jeroen Demeyer) Date: Thu, 23 Nov 2017 15:32:13 +0100 Subject: [Cython] Return types of declarations in Includes/cpython Message-ID: <5A16DBED.2000704@cage.ugent.be> Many Python/C API functions create objects of a known type. For example, PyUnicode_Decode() always returns a "unicode" object. In some cases, this return type is reflected in the declaration in Cython/Includes/cpython, in other cases (such as PyUnicodeDecode() that I just mentioned) not. What is Cython's policy here? In particular, what should be done when adding new declarations? From stefan_ml at behnel.de Thu Nov 23 12:19:43 2017 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 23 Nov 2017 18:19:43 +0100 Subject: [Cython] Return types of declarations in Includes/cpython In-Reply-To: <5A16DBED.2000704@cage.ugent.be> References: <5A16DBED.2000704@cage.ugent.be> Message-ID: Am 23. November 2017 15:32:13 MEZ schrieb Jeroen Demeyer: >Many Python/C API functions create objects of a known type. For >example, >PyUnicode_Decode() always returns a "unicode" object. In some cases, >this return type is reflected in the declaration in >Cython/Includes/cpython, in other cases (such as PyUnicodeDecode() that > >I just mentioned) not. > >What is Cython's policy here? In particular, what should be done when >adding new declarations? It's not about policy. There probably wasn't any advantage yet back when they were added, certainly no type inference, and in some cases, there still isn't much to gain from them. But it's certainly a good idea to provide as much type information as possible here, so feel free to add it wherever it may fit. Stefan From erik.m.bray at gmail.com Fri Nov 24 07:44:04 2017 From: erik.m.bray at gmail.com (Erik Bray) Date: Fri, 24 Nov 2017 13:44:04 +0100 Subject: [Cython] Bug with inlined functions that access globals Message-ID: Hi, I think maybe I've seen this brought up once or twice before in the past, but with no real discussion. There is a slightly unpythonic problem with using cpdef inline functions defined in a .pxd file that happen to access global variables. For example: $ cat foo.pxd FOO_A = 1 cpdef inline foo(): return FOO_A $ cat bar.pyx from foo cimport foo print(foo()) Running this like python -c 'import bar' results in: Traceback (most recent call last): File "", line 1, in File "bar.pyx", line 3, in init bar print(foo()) File "foo.pxd", line 4, in foo.foo return FOO_A NameError: name 'FOO_A' is not defined Of course, if the function "foo" were not inlined this would work fine. But the inlining really breaks things, I think, in an unpythonic way. I could be misunderstanding how Cython's "inline" should be interpreted but I feel like it's an implementation detail that should not fundamentally change how the code in that function works. There are a couple issues here. First of all, if a .pxd file exists but not a .pyx, no actual module will be created. This is by design of course, but in this case either, inlined functions in a .pxd module should be prohibited from accessing globals, or a trivial module should be created to store the globals for that module. The second problem boils down to the global variable lookup being compiled to something like: __pyx_t_1 = __Pyx_GetModuleGlobalName(__pyx_n_s_FOO_A) Perhaps, when a function from one module gets inlined in another module, at least in the case where it needs to access globals, the originating module should be imported first and __Pyx_GetModuleGlobalName replaced with an equivalent that uses the originating module's globals. So have a global variable like static PyObject* __pyx_d_foo and in the module init code something like: _pyx_t_1 = __Pyx_Import("foo") __pyx_d_foo = PyModule_GetDict(__pyx_t_1) then in the inlined function, replace the call to __Pyx_GetModuleGlobalName to an equivalent that uses __pyx_d_foo instead of __pyx_d. This all assumes, of course, that "foo" is an importable module. I'm not sure this makes sense otherwise... If not then maybe inlining a function should also "inline" any global variables it uses. Thoughts? Erik From stefan_ml at behnel.de Fri Nov 24 08:16:27 2017 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 24 Nov 2017 14:16:27 +0100 Subject: [Cython] Bug with inlined functions that access globals In-Reply-To: References: Message-ID: <911feb6f-c8be-c597-0575-37fabd96d970@behnel.de> Erik Bray schrieb am 24.11.2017 um 13:44: > I think maybe I've seen this brought up once or twice before in the > past, but with no real discussion. There is a slightly unpythonic > problem with using cpdef inline functions defined in a .pxd file that > happen to access global variables. For example: > > $ cat foo.pxd > FOO_A = 1 > > cpdef inline foo(): > return FOO_A Hmm, I might be missing something, but it feels like globals in a .pxd file should not be allowed in the first place - unless there is a .pyx file for them. But that's difficult to prohibit, given that the .pyx file is not needed at compile time, and might not be there at all, just the readily compiled module (and maybe not even that). And I don't see why Python globals should be allowed in .pxd files at all. So - it seems probably difficult to improve the first case. Regarding your actual example, however, if there is really no use case for Python globals in .pxd files, then we should better disallow them. Stefan From erik.m.bray at gmail.com Fri Nov 24 08:48:43 2017 From: erik.m.bray at gmail.com (Erik Bray) Date: Fri, 24 Nov 2017 14:48:43 +0100 Subject: [Cython] Bug with inlined functions that access globals In-Reply-To: <911feb6f-c8be-c597-0575-37fabd96d970@behnel.de> References: <911feb6f-c8be-c597-0575-37fabd96d970@behnel.de> Message-ID: On Fri, Nov 24, 2017 at 2:16 PM, Stefan Behnel wrote: > Erik Bray schrieb am 24.11.2017 um 13:44: >> I think maybe I've seen this brought up once or twice before in the >> past, but with no real discussion. There is a slightly unpythonic >> problem with using cpdef inline functions defined in a .pxd file that >> happen to access global variables. For example: >> >> $ cat foo.pxd >> FOO_A = 1 >> >> cpdef inline foo(): >> return FOO_A > > Hmm, I might be missing something, but it feels like globals in a .pxd file > should not be allowed in the first place - unless there is a .pyx file for > them. But that's difficult to prohibit, given that the .pyx file is not > needed at compile time, and might not be there at all, just the readily > compiled module (and maybe not even that). > > And I don't see why Python globals should be allowed in .pxd files at all. > > So - it seems probably difficult to improve the first case. Regarding your > actual example, however, if there is really no use case for Python globals > in .pxd files, then we should better disallow them. That would be an acceptable resolution, though I believe there is a use case for it. If you take some function that was originally defined in a .pyx module, but you want to be able to inline it, then one is forced to rewrite it in some way if it in any way relied on some module-level globals (which is a normal thing to do). So it's a problem of the implementation details driving what language features one is allowed to use. This is less than ideal... Anyways there's really two issues here: 1) Python globals in .pxd files, which you're saying should be disallowed--I have to wonder if there isn't some way to make that make sense, but I don't feel strongly about it. 2) Non-local variables in functions defined in .pxd files--again these should either be allowed by some mechanism, or entirely disallowed since otherwise the result is either the function breaks with a NameError, or behaves wildly unexpectedly if it gets dropped into a module that happens to have a global of the same name From robertwb at gmail.com Fri Nov 24 11:52:13 2017 From: robertwb at gmail.com (Robert Bradshaw) Date: Fri, 24 Nov 2017 08:52:13 -0800 Subject: [Cython] Bug with inlined functions that access globals In-Reply-To: References: <911feb6f-c8be-c597-0575-37fabd96d970@behnel.de> Message-ID: On Fri, Nov 24, 2017 at 5:48 AM, Erik Bray wrote: > On Fri, Nov 24, 2017 at 2:16 PM, Stefan Behnel wrote: >> Erik Bray schrieb am 24.11.2017 um 13:44: >>> I think maybe I've seen this brought up once or twice before in the >>> past, but with no real discussion. There is a slightly unpythonic >>> problem with using cpdef inline functions defined in a .pxd file that >>> happen to access global variables. For example: >>> >>> $ cat foo.pxd >>> FOO_A = 1 >>> >>> cpdef inline foo(): >>> return FOO_A >> >> Hmm, I might be missing something, but it feels like globals in a .pxd file >> should not be allowed in the first place - unless there is a .pyx file for >> them. But that's difficult to prohibit, given that the .pyx file is not >> needed at compile time, and might not be there at all, just the readily >> compiled module (and maybe not even that). >> >> And I don't see why Python globals should be allowed in .pxd files at all. >> >> So - it seems probably difficult to improve the first case. Regarding your >> actual example, however, if there is really no use case for Python globals >> in .pxd files, then we should better disallow them. > > That would be an acceptable resolution, though I believe there is a > use case for it. If you take some function that was originally > defined in a .pyx module, but you want to be able to inline it, then > one is forced to rewrite it in some way if it in any way relied on > some module-level globals (which is a normal thing to do). So it's a > problem of the implementation details driving what language features > one is allowed to use. This is less than ideal... > > Anyways there's really two issues here: > > 1) Python globals in .pxd files, which you're saying should be > disallowed--I have to wonder if there isn't some way to make that make > sense, but I don't feel strongly about it. > 2) Non-local variables in functions defined in .pxd files--again these > should either be allowed by some mechanism, or entirely disallowed > since otherwise the result is either the function breaks with a > NameError, or behaves wildly unexpectedly if it gets dropped into a > module that happens to have a global of the same name +1 to disallowing python globals altogether in .pxd files. One can cdef any global one needs.