Re: [Cython] [cython-users] Cython .pxd introspection: listing defined constants
Forgot reply-all... didin't we have this discussion before about making that the default for this list as it is by-far the most common desired behavior? On Thu, Feb 17, 2011 at 3:53 PM, Robert Bradshaw <robertwb@math.washington.edu> wrote:
On Thu, Feb 17, 2011 at 3:12 PM, W. Trevor King <wking@drexel.edu> wrote:
On Thu, Feb 17, 2011 at 01:25:10PM -0800, Robert Bradshaw wrote:
On Thu, Feb 17, 2011 at 5:29 AM, W. Trevor King <wking@drexel.edu> wrote:
On Wed, Feb 16, 2011 at 03:55:19PM -0800, Robert Bradshaw wrote:
On Wed, Feb 16, 2011 at 8:17 AM, W. Trevor King <wking@drexel.edu> wrote:
What I'm missing is a way to bind the ModuleScope namespace to a name in expose.pyx so that commands like `dir(mylib)` and `getattr(mylib, name)` will work in expose.pyx.
You have also hit into the thorny issue that .pxd files are used for many things. They may be pure C library declarations with no Python module backing, they may be declarations of (externally implemented) Python modules (such as numpy.pxd), or they may be declarations for Cython-implemented modules.
Here's another idea, what if extern blocks could contain cpdef declarations, which would automatically generate a Python-level wrappers for the declared members (if possible, otherwise an error)?
Ah, this sounds good! Of the three .pxd roles you list above, external Python modules (e.g. numpy) and Cython-implemented modules (e.g. matched .pxd/.pyx) both already have a presence in Python-space. What's missing is a way to give (where possible) declarations of external C libraries a Python presence. cpdef fills this hole nicely, since its whole purpose is to expose Python interfaces to C-based elements.
In the case of external Python modules, I'm not so sure we want to monkey-patch our stuff in
I don't think any of the changes we are suggesting would require changes to existing code, so .pxd-s with external implementations wouldn't be affected unless they brough the changes upon themselves.
Say, in numpy.pxd, I have
cdef extern from "...": cpdef struct obscure_internal_struct: ...
Do we add an "obscure_internal_struct" onto the (global) numpy module? What if it conflicts with a (runtime) name? This is the issue I'm bringing up.
(and where would we do it--on the first import of a cimporting module?)
Compilation is an issue. I think that .pxd files should be able to be cythoned directly, since then they Cython can build any wrappers they request. If the file has a matching .pyx file, cythoning either one should compile both together, since they'll produce a single Python .so module.
In this case, I think it may make more sense to consider cimporting stuff from .pyx files if no .pxd file is present. In any case, this is a big change. I don't like the idea of always creating such a module (it may be empty, or have name conflicts) nor the idea of conditionally compiling it (does one have to look at the contents and really understand Cython to see if a Python shadow will be created?)
A side effect of this cpdef change would be that now even bare .pxd files (no matching .pyx) would have a Python presence,
Where would it live? Would we just create this module (in essence, acting as if there was an empty .pyx file sitting there as well)? On this note, it may be worth pursuing the idea of a "cython helper" module where common code and objects could live.
I'm not sure exactly what you mean by "cython helper", but this sounds like my 'bare .pyx can create a Python .so module idea above.
I'm thinking of a place to put, e.g. the generator and bind-able function classes, which are now re-implemented in every module that uses them. I think there will be more cases like this in the future rather than less. C-level code could be #included and linked from "global" stores as well. However, that's somewhat tangential.
so You could do something like
cimport mylib as mylib_c import mylib as mylib_py import sys
# Access through Python for name in dir(mylib_py): setattr(sys.modules[__name__], name, getattr(mylib_py, name))
I think this smells worse than "import *"
Aha, thanks ;). I was stuck in my old .pxd-files-don't-create-modules-by-themselves mindset. Obviously, once they do, any Python code can access the contents directly and I can throw out all this indirection.
However, from Parsing.py:2369:
error(pos, "C struct/union/enum cannot be declared cpdef")
From pyrex_differences.rst:
If a function is declared :keyword:`cpdef` it can be called from and overridden by both extension and normal python subclasses.
I believe the reason that cpdef-ed enums and similar are currently illegal is confusion between "can be called from Python" and "can be overridden from Python".
The reason that error statement is there is because it had no meaning, so an error was better than just ignoring it.
Why does it have no meaning? I understand that it's not implemented yet, but a cpdef-ed enum or struct seems just as valid an idea as a cpdef-ed method.
We're only deciding what the meaning is now. I agree it's a valid idea, its just that no one had even considered it yet. (I'll also concede that it's a much more natural idea for someone who came to the language once cpdef was already implemented.)
Unions don't really have a Python parallel,
They can be a cdef class wrapping the union type.
But I would think coercion would be difficult. Unions are usually (in my limited experience) for "don't worry about the type, just make sure it fits in X bytes". How would union->Python conversion work?
There would be a wrapping type, e.g.
cdef class MyUnion: cdef union_type value
with a bunch of setters/getters for the values, just like there are for structs. (In fact the same code would handle structs and unions).
This is getting into the wrapper-generator territory, but I'm starting to think for simple things that might be worth it.
cpdef struct Foo: cpdef public int intA cpdef readonly int intB cdef void *ptr
We would both declare the important members of the C struct (as we can already do in Cython) and also have Cython automatically generate a Python class wrapping the struct (because of `cpdef struct`). The Python class would have:
* Cython-generated getter/setter for intA (because of `cpdef public`) using the standard Python<->int coercion. * Similar Cython-generated getter for int B (because of `cpdef readonly`). * No Python access to ptr (standard C-access still possible through Cython).
Doing something crazy like `cdef public void *ptr` would raise a compile-time error.
Yes, all of the above was exactly what I was proposing.
I'm definately willing to help out with this (if someone will point me in the right direction),
That would be great.
Ok, I think we're pretty much agreed ;). I think that the next step is to start working on implementations of:
* Stand alone .pxd -> Python module
I'm not sure we're agreed on this one.
* Extending class cdef/cdpef/public/readonly handling to cover enums, stucts, and possibly unions.
This seems like the best first step.
Problems with me getting started now:
* I don't know where I should start mucking about in the source.
I would start by (1) creating/declaring cdef classes for cdef structs/unions/enums (not textually, within, e.g. CStructOrUnionDefNode.analyse_declarations(...) in Nodes.py) and then making sure they get added to the module's namespace (look at ModuleNode.py where the classes get added). The other files to look at would be Parsing.py, PyrexTypes.py and Symtab.py. Could either try to patch up parsing first, but I think it'd be more effective to make this the default during development, as then all the tests would exercise your code, and it's probably both easier to dive in there and would give you a better idea of how best to handle the parsing.
Also, the creating of cpdef functions is a bit hackish (for historical reasons) and something we want to clean up, so I wouldn't go to great lengths to try and emulate that.
* I don't know how to handle things like dummy enums (perhaps by requiring all cdef-ed enums to be named).
All enums in C are named.
* I'm going to go watch a movie ;).
:)
It's good to be moving forward!
Yep. You might want to consider creating a github fork.
- Robert
On Thu, Feb 17, 2011 at 3:53 PM, Robert Bradshaw wrote:
On Thu, Feb 17, 2011 at 3:12 PM, W. Trevor King wrote:
On Thu, Feb 17, 2011 at 01:25:10PM -0800, Robert Bradshaw wrote:
On Thu, Feb 17, 2011 at 5:29 AM, W. Trevor King wrote:
On Wed, Feb 16, 2011 at 03:55:19PM -0800, Robert Bradshaw wrote:
On Wed, Feb 16, 2011 at 8:17 AM, W. Trevor King wrote:
What I'm missing is a way to bind the ModuleScope namespace to a name in expose.pyx so that commands like `dir(mylib)` and `getattr(mylib, name)` will work in expose.pyx.
You have also hit into the thorny issue that .pxd files are used for many things. They may be pure C library declarations with no Python module backing, they may be declarations of (externally implemented) Python modules (such as numpy.pxd), or they may be declarations for Cython-implemented modules.
Here's another idea, what if extern blocks could contain cpdef declarations, which would automatically generate a Python-level wrappers for the declared members (if possible, otherwise an error)?
Ah, this sounds good! Of the three .pxd roles you list above, external Python modules (e.g. numpy) and Cython-implemented modules (e.g. matched .pxd/.pyx) both already have a presence in Python-space. What's missing is a way to give (where possible) declarations of external C libraries a Python presence. cpdef fills this hole nicely, since its whole purpose is to expose Python interfaces to C-based elements.
In the case of external Python modules, I'm not so sure we want to monkey-patch our stuff in
I don't think any of the changes we are suggesting would require changes to existing code, so .pxd-s with external implementations wouldn't be affected unless they brough the changes upon themselves.
Say, in numpy.pxd, I have
cdef extern from "...": cpdef struct obscure_internal_struct: ...
Do we add an "obscure_internal_struct" onto the (global) numpy module? What if it conflicts with a (runtime) name? This is the issue I'm bringing up.
Defining a cpdef *and* a non-matching external implementation should raise a compile-time error. I agree that there is a useful distinction between external-C-library and external-Python-module .pxd wrappers. Perhaps your matching blank .py or .pyx file could serve as a marker that the .pxd file should be inflated into its own full fledged python module. I'm not even sure how you would go about adding attributes to the numpy module. When/how would the Cython-created attributes get added? In the external-C-library case, if you define something incompatible in the matching .pyx or .py file, Cython will be able to see it and die with an appropriate error, so you can resolve your programming mistake. If you try to override anything in a .so compiled module at runtime, you'd get the same kind of error you currently do trying to rebind a compiled class' method.
(and where would we do it--on the first import of a cimporting module?)
Compilation is an issue. I think that .pxd files should be able to be cythoned directly, since then they Cython can build any wrappers they request. If the file has a matching .pyx file, cythoning either one should compile both together, since they'll produce a single Python .so module.
In this case, I think it may make more sense to consider cimporting stuff from .pyx files if no .pxd file is present.
Can you cimport .pyx files that lack matching .pxd files?
In any case, this is a big change. I don't like the idea of always creating such a module (it may be empty, or have name conflicts)
It shouldn't take to long to compile an empty module ;). And odds are noone will waste time importing it either.
nor the idea of conditionally compiling it (does one have to look at the contents and really understand Cython to see if a Python shadow will be created?)
I agree here. Under the mantra "explicit is better than implicit", we could have users add something like cdef module "modname" to any .pxd files that should be inflated into Python modules. .pxd files without such a tag would receive the current treatment, error on any cpdef, etc. The drawback of this approach is that it makes Cython more complicated, but if both behaviors are reasonable, there's probably no getting around that.
A side effect of this cpdef change would be that now even bare .pxd files (no matching .pyx) would have a Python presence,
Where would it live? Would we just create this module (in essence, acting as if there was an empty .pyx file sitting there as well)? On this note, it may be worth pursuing the idea of a "cython helper" module where common code and objects could live.
I'm not sure exactly what you mean by "cython helper", but this sounds like my 'bare .pyx can create a Python .so module idea above.
I'm thinking of a place to put, e.g. the generator and bind-able function classes, which are now re-implemented in every module that uses them. I think there will be more cases like this in the future rather than less. C-level code could be #included and linked from "global" stores as well. However, that's somewhat tangential.
That sounds more and more like per-pxd .so modules ;).
Unions don't really have a Python parallel,
They can be a cdef class wrapping the union type.
But I would think coercion would be difficult. Unions are usually (in my limited experience) for "don't worry about the type, just make sure it fits in X bytes". How would union->Python conversion work?
There would be a wrapping type, e.g.
cdef class MyUnion: cdef union_type value
with a bunch of setters/getters for the values, just like there are for structs. (In fact the same code would handle structs and unions).
This is getting into the wrapper-generator territory, but I'm starting to think for simple things that might be worth it.
I think that if Cython will automatically generate a wrapper for cdef public int x it should generate a wrapper for cdef struct X: cdef public int x There really aren't that metatypes in C, so it doesn't seem like a slippery slope to me. Maybe I'm just missing something...
Ok, I think we're pretty much agreed ;). I think that the next step is to start working on implementations of:
* Stand alone .pxd -> Python module
I'm not sure we're agreed on this one.
Ok, it can wait while we hash out the best approach.
* Extending class cdef/cdpef/public/readonly handling to cover enums, stucts, and possibly unions.
This seems like the best first step.
Problems with me getting started now:
* I don't know where I should start mucking about in the source.
I would start by...
Excellent. Thanks.
* I don't know how to handle things like dummy enums (perhaps by requiring all cdef-ed enums to be named).
All enums in C are named.
But my Cython declaration (exposing a C `#define CONST_A 1`): cdef extern from 'mylib.h': enum: CONST_A is not a named enum.
Yep. You might want to consider creating a github fork.
I've started a branch 'cdef-enums-stucts-and-unions' (no activity yet). Repo: http://www.physics.drexel.edu/~wking/code/git/cython.git Gitweb interface: http://www.physics.drexel.edu/~wking/code/git/gitweb.cgi?p=cython.git I can put this up on GitHub if you want, but it's easier for me to self-host, and `git fetch` works just as well either way ;). -- This email may be signed or encrypted with GPG (http://www.gnupg.org). The GPG signature (if present) will be attached as 'signature.asc'. For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy My public key is at http://www.physics.drexel.edu/~wking/pubkey.txt
On Thu, Feb 17, 2011 at 8:38 PM, W. Trevor King <wking@drexel.edu> wrote:
On Thu, Feb 17, 2011 at 3:53 PM, Robert Bradshaw wrote:
On Thu, Feb 17, 2011 at 3:12 PM, W. Trevor King wrote:
On Thu, Feb 17, 2011 at 01:25:10PM -0800, Robert Bradshaw wrote:
On Thu, Feb 17, 2011 at 5:29 AM, W. Trevor King wrote:
On Wed, Feb 16, 2011 at 03:55:19PM -0800, Robert Bradshaw wrote:
On Wed, Feb 16, 2011 at 8:17 AM, W. Trevor King wrote: > What I'm missing is a way to bind the ModuleScope namespace to a name > in expose.pyx so that commands like `dir(mylib)` and `getattr(mylib, > name)` will work in expose.pyx.
You have also hit into the thorny issue that .pxd files are used for many things. They may be pure C library declarations with no Python module backing, they may be declarations of (externally implemented) Python modules (such as numpy.pxd), or they may be declarations for Cython-implemented modules.
Here's another idea, what if extern blocks could contain cpdef declarations, which would automatically generate a Python-level wrappers for the declared members (if possible, otherwise an error)?
Ah, this sounds good! Of the three .pxd roles you list above, external Python modules (e.g. numpy) and Cython-implemented modules (e.g. matched .pxd/.pyx) both already have a presence in Python-space. What's missing is a way to give (where possible) declarations of external C libraries a Python presence. cpdef fills this hole nicely, since its whole purpose is to expose Python interfaces to C-based elements.
In the case of external Python modules, I'm not so sure we want to monkey-patch our stuff in
I don't think any of the changes we are suggesting would require changes to existing code, so .pxd-s with external implementations wouldn't be affected unless they brough the changes upon themselves.
Say, in numpy.pxd, I have
cdef extern from "...": cpdef struct obscure_internal_struct: ...
Do we add an "obscure_internal_struct" onto the (global) numpy module? What if it conflicts with a (runtime) name? This is the issue I'm bringing up.
Defining a cpdef *and* a non-matching external implementation should raise a compile-time error. I agree that there is a useful distinction between external-C-library and external-Python-module .pxd wrappers. Perhaps your matching blank .py or .pyx file could serve as a marker that the .pxd file should be inflated into its own full fledged python module. I'm not even sure how you would go about adding attributes to the numpy module. When/how would the Cython-created attributes get added?
Yes, this is exactly the issue.
In the external-C-library case, if you define something incompatible in the matching .pyx or .py file, Cython will be able to see it and die with an appropriate error, so you can resolve your programming mistake.
That is only if you have a matching .pyx of .py file. Of course, there could be a module by that name in the user's runtime environment that we don't know about, but that currently is not in conflict with a .pxd only file.
If you try to override anything in a .so compiled module at runtime, you'd get the same kind of error you currently do trying to rebind a compiled class' method.
That's the desired behavior for statically-bound globals, but implementing it is not so trivial.
(and where would we do it--on the first import of a cimporting module?)
Compilation is an issue. I think that .pxd files should be able to be cythoned directly, since then they Cython can build any wrappers they request. If the file has a matching .pyx file, cythoning either one should compile both together, since they'll produce a single Python .so module.
In this case, I think it may make more sense to consider cimporting stuff from .pyx files if no .pxd file is present.
Can you cimport .pyx files that lack matching .pxd files?
In any case, this is a big change. I don't like the idea of always creating such a module (it may be empty, or have name conflicts)
It shouldn't take to long to compile an empty module ;). And odds are noone will waste time importing it either.
nor the idea of conditionally compiling it (does one have to look at the contents and really understand Cython to see if a Python shadow will be created?)
I agree here.
Under the mantra "explicit is better than implicit", we could have users add something like
cdef module "modname"
to any .pxd files that should be inflated into Python modules. .pxd files without such a tag would receive the current treatment, error on any cpdef, etc. The drawback of this approach is that it makes Cython more complicated, but if both behaviors are reasonable, there's probably no getting around that.
The other drawback is that it subverts the usual filename <-> module name convention that one usually expects.
A side effect of this cpdef change would be that now even bare .pxd files (no matching .pyx) would have a Python presence,
Where would it live? Would we just create this module (in essence, acting as if there was an empty .pyx file sitting there as well)? On this note, it may be worth pursuing the idea of a "cython helper" module where common code and objects could live.
I'm not sure exactly what you mean by "cython helper", but this sounds like my 'bare .pyx can create a Python .so module idea above.
I'm thinking of a place to put, e.g. the generator and bind-able function classes, which are now re-implemented in every module that uses them. I think there will be more cases like this in the future rather than less. C-level code could be #included and linked from "global" stores as well. However, that's somewhat tangential.
That sounds more and more like per-pxd .so modules ;).
Unions don't really have a Python parallel,
They can be a cdef class wrapping the union type.
But I would think coercion would be difficult. Unions are usually (in my limited experience) for "don't worry about the type, just make sure it fits in X bytes". How would union->Python conversion work?
There would be a wrapping type, e.g.
cdef class MyUnion: cdef union_type value
with a bunch of setters/getters for the values, just like there are for structs. (In fact the same code would handle structs and unions).
This is getting into the wrapper-generator territory, but I'm starting to think for simple things that might be worth it.
I think that if Cython will automatically generate a wrapper for
cdef public int x
it should generate a wrapper for
cdef struct X: cdef public int x
Or cdef public struct X: int x readonly int z private int z I would perhaps say that non-Pythonable non-private members in public structs would be a compile error.
There really aren't that metatypes in C, so it doesn't seem like a slippery slope to me. Maybe I'm just missing something...
Ok, I think we're pretty much agreed ;). I think that the next step is to start working on implementations of:
* Stand alone .pxd -> Python module
I'm not sure we're agreed on this one.
Ok, it can wait while we hash out the best approach.
* Extending class cdef/cdpef/public/readonly handling to cover enums, stucts, and possibly unions.
This seems like the best first step.
Problems with me getting started now:
* I don't know where I should start mucking about in the source.
I would start by...
Excellent. Thanks.
* I don't know how to handle things like dummy enums (perhaps by requiring all cdef-ed enums to be named).
All enums in C are named.
But my Cython declaration (exposing a C `#define CONST_A 1`):
cdef extern from 'mylib.h': enum: CONST_A
is not a named enum.
Ah, yes. Maybe we require a name (that would only be used in Python space).
Yep. You might want to consider creating a github fork.
I've started a branch 'cdef-enums-stucts-and-unions' (no activity yet).
Repo: http://www.physics.drexel.edu/~wking/code/git/cython.git
Gitweb interface: http://www.physics.drexel.edu/~wking/code/git/gitweb.cgi?p=cython.git
I can put this up on GitHub if you want, but it's easier for me to self-host, and `git fetch` works just as well either way ;).
Code review is much easier if you fork off the public branch and do pull-requests, but being a DVCS you of course don't have to "choose" where your master repo lives. - Robert
Robert Bradshaw, 18.02.2011 23:08:
On Thu, Feb 17, 2011 at 8:38 PM, W. Trevor King wrote:
On Thu, Feb 17, 2011 at 3:53 PM, Robert Bradshaw wrote:
On Thu, Feb 17, 2011 at 3:12 PM, W. Trevor King wrote:
Compilation is an issue. I think that .pxd files should be able to be cythoned directly, since then they Cython can build any wrappers they request. If the file has a matching .pyx file, cythoning either one should compile both together, since they'll produce a single Python .so module.
In this case, I think it may make more sense to consider cimporting stuff from .pyx files if no .pxd file is present.
Can you cimport .pyx files that lack matching .pxd files?
In any case, this is a big change. I don't like the idea of always creating such a module (it may be empty, or have name conflicts)
It shouldn't take to long to compile an empty module ;). And odds are noone will waste time importing it either.
nor the idea of conditionally compiling it (does one have to look at the contents and really understand Cython to see if a Python shadow will be created?)
I agree here.
Under the mantra "explicit is better than implicit", we could have users add something like
cdef module "modname"
to any .pxd files that should be inflated into Python modules. .pxd files without such a tag would receive the current treatment, error on any cpdef, etc. The drawback of this approach is that it makes Cython more complicated, but if both behaviors are reasonable, there's probably no getting around that.
The other drawback is that it subverts the usual filename<-> module name convention that one usually expects.
+1
A side effect of this cpdef change would be that now even bare .pxd files (no matching .pyx) would have a Python presence,
Where would it live? Would we just create this module (in essence, acting as if there was an empty .pyx file sitting there as well)? On this note, it may be worth pursuing the idea of a "cython helper" module where common code and objects could live.
I'm not sure exactly what you mean by "cython helper", but this sounds like my 'bare .pyx can create a Python .so module idea above.
I'm thinking of a place to put, e.g. the generator and bind-able function classes, which are now re-implemented in every module that uses them. I think there will be more cases like this in the future rather than less. C-level code could be #included and linked from "global" stores as well. However, that's somewhat tangential.
If you generate more than one file from a .pyx, including files that are shared between compiler runs (or even readily built as .so files), you'd quickly end up in dependency hell. When to regenerate that file? With what compiler version and config? C/C++? With/out cleanup code? And where to put it? Are you sure that place will be writable for all compiler runs? And will the result be guaranteed to be reproducible, regardless of how many compiler runs there were in between?
Unions don't really have a Python parallel,
They can be a cdef class wrapping the union type.
But I would think coercion would be difficult. Unions are usually (in my limited experience) for "don't worry about the type, just make sure it fits in X bytes". How would union->Python conversion work?
There would be a wrapping type, e.g.
cdef class MyUnion: cdef union_type value
Wouldn't that have to be a pointer to the real thing instead?
with a bunch of setters/getters for the values, just like there are for structs. (In fact the same code would handle structs and unions).
This is getting into the wrapper-generator territory, but I'm starting to think for simple things that might be worth it.
I think that if Cython will automatically generate a wrapper for
cdef public int x
it should generate a wrapper for
cdef struct X: cdef public int x
Or
cdef public struct X: int x readonly int z private int z
I would perhaps say that non-Pythonable non-private members in public structs would be a compile error.
+1, keep it safe at the beginning.
There really aren't that metatypes in C, so it doesn't seem like a slippery slope to me. Maybe I'm just missing something...
Ok, I think we're pretty much agreed ;). I think that the next step is to start working on implementations of:
* Stand alone .pxd -> Python module
I'm not sure we're agreed on this one.
Same from here. To me, that doesn't make much sense for code that wraps a library. And if it doesn't wrap a library, there isn't much benefit in writing a stand-alone .pxd in the first place. A .pyx is much more explicit and obvious in this case. Especially having some .pxd files that generate .so files and others that don't will make this very ugly. I'd prefer adding support for cimporting from .pyx files instead, potentially with an automated caching generation of corresponding .pxd files (maybe as ".pxdg" files to make them easier to handle for users). However, cyclic dependencies would be tricky to handle automatically then.
* Extending class cdef/cdpef/public/readonly handling to cover enums, stucts, and possibly unions.
This seems like the best first step.
+1
* I don't know how to handle things like dummy enums (perhaps by requiring all cdef-ed enums to be named).
All enums in C are named.
But my Cython declaration (exposing a C `#define CONST_A 1`):
cdef extern from 'mylib.h': enum: CONST_A
is not a named enum.
Ah, yes. Maybe we require a name (that would only be used in Python space).
... require it for cpdef enums, you mean? OTOH, the "enum: NAME" scheme is ugly by itself. There should be a way to declare external constants correctly. After all, we loose all type information that way. I just saw that in math.pxd things like "M_PI" are declared as plain "enum" instead of "const double" or something. The type inferencer can't do anything with that. It might even draw the completely wrong conclusions. Stefan
On Sat, Feb 19, 2011 at 1:24 AM, Stefan Behnel <stefan_ml@behnel.de> wrote:
> A side effect of this cpdef change would be that now even bare .pxd > files (no matching .pyx) would have a Python presence,
Where would it live? Would we just create this module (in essence, acting as if there was an empty .pyx file sitting there as well)? On this note, it may be worth pursuing the idea of a "cython helper" module where common code and objects could live.
I'm not sure exactly what you mean by "cython helper", but this sounds like my 'bare .pyx can create a Python .so module idea above.
I'm thinking of a place to put, e.g. the generator and bind-able function classes, which are now re-implemented in every module that uses them. I think there will be more cases like this in the future rather than less. C-level code could be #included and linked from "global" stores as well. However, that's somewhat tangential.
If you generate more than one file from a .pyx, including files that are shared between compiler runs (or even readily built as .so files), you'd quickly end up in dependency hell. When to regenerate that file? With what compiler version and config? C/C++? With/out cleanup code? And where to put it? Are you sure that place will be writable for all compiler runs? And will the result be guaranteed to be reproducible, regardless of how many compiler runs there were in between?
I'd probably make it an option to the cythonize(...) command, so all the regeneration/dependency information would be computed based on the whole fileset rather than trying to figure that out for each individual module compilation. I'm thinking this would be an optimization, not a requirement.
> Unions don't really have a Python parallel,
They can be a cdef class wrapping the union type.
But I would think coercion would be difficult. Unions are usually (in my limited experience) for "don't worry about the type, just make sure it fits in X bytes". How would union->Python conversion work?
There would be a wrapping type, e.g.
cdef class MyUnion: cdef union_type value
Wouldn't that have to be a pointer to the real thing instead?
Why? This would be a lot messier to get right, and structs and unions already have pass-by-value semantics.
with a bunch of setters/getters for the values, just like there are for structs. (In fact the same code would handle structs and unions).
This is getting into the wrapper-generator territory, but I'm starting to think for simple things that might be worth it.
I think that if Cython will automatically generate a wrapper for
cdef public int x
it should generate a wrapper for
cdef struct X: cdef public int x
Or
cdef public struct X: int x readonly int z private int z
I would perhaps say that non-Pythonable non-private members in public structs would be a compile error.
+1, keep it safe at the beginning.
There really aren't that metatypes in C, so it doesn't seem like a slippery slope to me. Maybe I'm just missing something...
Ok, I think we're pretty much agreed ;). I think that the next step is to start working on implementations of:
* Stand alone .pxd -> Python module
I'm not sure we're agreed on this one.
Same from here. To me, that doesn't make much sense for code that wraps a library. And if it doesn't wrap a library, there isn't much benefit in writing a stand-alone .pxd in the first place. A .pyx is much more explicit and obvious in this case. Especially having some .pxd files that generate .so files and others that don't will make this very ugly.
I'd prefer adding support for cimporting from .pyx files instead, potentially with an automated caching generation of corresponding .pxd files (maybe as ".pxdg" files to make them easier to handle for users). However, cyclic dependencies would be tricky to handle automatically then.
We could put the in the new __pycache__.
* Extending class cdef/cdpef/public/readonly handling to cover enums, stucts, and possibly unions.
This seems like the best first step.
+1
* I don't know how to handle things like dummy enums (perhaps by requiring all cdef-ed enums to be named).
All enums in C are named.
But my Cython declaration (exposing a C `#define CONST_A 1`):
cdef extern from 'mylib.h': enum: CONST_A
is not a named enum.
Ah, yes. Maybe we require a name (that would only be used in Python space).
... require it for cpdef enums, you mean?
OTOH, the "enum: NAME" scheme is ugly by itself. There should be a way to declare external constants correctly. After all, we loose all type information that way. I just saw that in math.pxd things like "M_PI" are declared as plain "enum" instead of "const double" or something. The type inferencer can't do anything with that. It might even draw the completely wrong conclusions.
+1 Const support would be really nice to have, especially given how much more C++ uses it. - Robert
On Fri, Feb 18, 2011 at 02:08:04PM -0800, Robert Bradshaw wrote:
On Thu, Feb 17, 2011 at 8:38 PM, W. Trevor King <wking@drexel.edu> wrote:
On Thu, Feb 17, 2011 at 3:53 PM, Robert Bradshaw wrote:
On Thu, Feb 17, 2011 at 3:12 PM, W. Trevor King wrote:
On Thu, Feb 17, 2011 at 01:25:10PM -0800, Robert Bradshaw wrote:
On Thu, Feb 17, 2011 at 5:29 AM, W. Trevor King wrote:
On Wed, Feb 16, 2011 at 03:55:19PM -0800, Robert Bradshaw wrote: > On Wed, Feb 16, 2011 at 8:17 AM, W. Trevor King wrote: > > What I'm missing is a way to bind the ModuleScope namespace to a name > > in expose.pyx so that commands like `dir(mylib)` and `getattr(mylib, > > name)` will work in expose.pyx. > > You have also hit into the thorny issue that .pxd files are used for > many things. They may be pure C library declarations with no Python > module backing, they may be declarations of (externally implemented) > Python modules (such as numpy.pxd), or they may be declarations for > Cython-implemented modules. > > Here's another idea, what if extern blocks could contain cpdef > declarations, which would automatically generate a Python-level > wrappers for the declared members (if possible, otherwise an error)?
Ah, this sounds good! Of the three .pxd roles you list above, external Python modules (e.g. numpy) and Cython-implemented modules (e.g. matched .pxd/.pyx) both already have a presence in Python-space. What's missing is a way to give (where possible) declarations of external C libraries a Python presence. cpdef fills this hole nicely, since its whole purpose is to expose Python interfaces to C-based elements.
In the case of external Python modules, I'm not so sure we want to monkey-patch our stuff in
I don't think any of the changes we are suggesting would require changes to existing code, so .pxd-s with external implementations wouldn't be affected unless they brough the changes upon themselves.
Say, in numpy.pxd, I have
cdef extern from "...": cpdef struct obscure_internal_struct: ...
Do we add an "obscure_internal_struct" onto the (global) numpy module? What if it conflicts with a (runtime) name? This is the issue I'm bringing up.
Defining a cpdef *and* a non-matching external implementation should raise a compile-time error. I agree that there is a useful distinction between external-C-library and external-Python-module .pxd wrappers. Perhaps your matching blank .py or .pyx file could serve as a marker that the .pxd file should be inflated into its own full fledged python module. I'm not even sure how you would go about adding attributes to the numpy module. When/how would the Cython-created attributes get added?
Yes, this is exactly the issue.
Ah, I'm retracting my agreement on the external-C-library and external-Python-module .pxd wrappers. There is no difference in how their .pxd files should be treated, and I now agree that .pxd files should not generate .so modules unless they have a paried .py/.pyx file.
If you try to override anything in a .so compiled module at runtime, you'd get the same kind of error you currently do trying to rebind a compiled class' method.
That's the desired behavior for statically-bound globals, but implementing it is not so trivial.
It's currently implemented for classes. Are modules that different from classes? >>> import types >>> types.ModuleType.__class__.__mro__ (<type 'type'>, <type 'object'>) So you've got your standard __setattr__ to override with an error-message generator. What is the implementation difficulty? I am also unclear about the distinction between cdef and cpdef (perhaps this should be a new thread?). cpdef means "I'm declaring something with C and Python interfaces. The Python interface is a thin wrapper which can be rebound to an object of any type, leaving the static C interface inaccessible from Python." cdef [private] means "I'm declaring something that only has a C interface." cdef public means "I'm declaring something with C and Python interfaces backed by C data. Python code can alter the C data." cdef readonly means "I'm declaring something with C and Python interfaces backed by C data. Python code cannot alter the C data." This seems to be broken in Cython at the module level, since I can rebind a cdef-ed class but not a cpdef-ed method: $ cat square.pyx cdef class A (object): cdef public int value cpdef square(self): return self.value**2 $ python -c 'import pyximport; pyximport.install(); import xx; a = xx.A(); a.value = 3; print a.square(); a.square = lambda self: self.value' Traceback (most recent call last): File "<string>", line 3, in <module> AttributeError: 'square.A' object attribute 'square' is read-only $ python -c 'import pyximport; pyximport.install(); import square; square.A = object; print square.A; a = square.A(); a.value = 3' <type 'object'> Traceback (most recent call last): File "<string>", line 2, in <module> AttributeError: 'object' object has no attribute 'value' So the cdef-ed A currently has a rebindable Python interface (acting like a hypothetical cpdef-ed class), but it's square method is not rebindable (acting like a hypothetical `cdef readonly`-ed method).
(and where would we do it--on the first import of a cimporting module?)
Compilation is an issue. I think that .pxd files should be able to be cythoned directly, since then they Cython can build any wrappers they request. If the file has a matching .pyx file, cythoning either one should compile both together, since they'll produce a single Python .so module.
...
Under the mantra "explicit is better than implicit", we could have users add something like
cdef module "modname"
to any .pxd files that should be inflated into Python modules. .pxd files without such a tag would receive the current treatment, error on any cpdef, etc. The drawback of this approach is that it makes Cython more complicated, but if both behaviors are reasonable, there's probably no getting around that.
The other drawback is that it subverts the usual filename <-> module name convention that one usually expects.
I've been convinced that the `cimport .pyx file` route is a better way to go. However, the filename <-> module mapping is troublesome for backing externally-implemented Python modules (e.g. numpy). If you wanted to write a .pxd file backing numpy.random, how would you go about getting your module installed in Cython/Includes/numpy/random.pxd or another path that cython would successfully match with `cimport numpy.random`? On Sat, Feb 19, 2011 at 10:24:05AM +0100, Stefan Behnel wrote:
Robert Bradshaw, 18.02.2011 23:08:
On Thu, Feb 17, 2011 at 8:38 PM, W. Trevor King wrote:
On Thu, Feb 17, 2011 at 3:53 PM, Robert Bradshaw wrote:
On Thu, Feb 17, 2011 at 3:12 PM, W. Trevor King wrote:
> A side effect of this cpdef change would be that now even bare .pxd > files (no matching .pyx) would have a Python presence,
Where would it live? Would we just create this module (in essence, acting as if there was an empty .pyx file sitting there as well)? On this note, it may be worth pursuing the idea of a "cython helper" module where common code and objects could live.
I'm not sure exactly what you mean by "cython helper", but this sounds like my 'bare .pyx can create a Python .so module idea above.
I'm thinking of a place to put, e.g. the generator and bind-able function classes, which are now re-implemented in every module that uses them. I think there will be more cases like this in the future rather than less. C-level code could be #included and linked from "global" stores as well. However, that's somewhat tangential.
If you generate more than one file from a .pyx, including files that are shared between compiler runs (or even readily built as .so files), you'd quickly end up in dependency hell.
I disagree here, but I like your cimportable .pyx better, so it doesn't matter ;).
> Unions don't really have a Python parallel,
They can be a cdef class wrapping the union type.
But I would think coercion would be difficult. Unions are usually (in my limited experience) for "don't worry about the type, just make sure it fits in X bytes". How would union->Python conversion work?
There would be a wrapping type, e.g.
cdef class MyUnion: cdef union_type value
Wouldn't that have to be a pointer to the real thing instead?
Do you mean `cdef union_type *value`? Why would the above version not work? The union type has a well defined size and a number of well defined interpretations, so I don't see the problem.
with a bunch of setters/getters for the values, just like there are for structs. (In fact the same code would handle structs and unions).
This is getting into the wrapper-generator territory, but I'm starting to think for simple things that might be worth it.
I think that if Cython will automatically generate a wrapper for
cdef public int x
it should generate a wrapper for
cdef struct X: cdef public int x
Or
cdef public struct X: int x readonly int z private int z
I would perhaps say that non-Pythonable non-private members in public structs would be a compile error.
+1, keep it safe at the beginning.
-1, keep the code clean and the interface consistent ;). I think the struct syntax should be identical to the class syntax, with the exception that you can't bind methods to structs. That's the only real difference between structs and classes, isn't it? If safety with a new feature is a concern, a warning like "EXPERIMENTAL FEATURE" in the associated docs and compiler output should be sufficient.
There really aren't that metatypes in C, so it doesn't seem like a slippery slope to me. Maybe I'm just missing something...
Ok, I think we're pretty much agreed ;). I think that the next step is to start working on implementations of:
* Stand alone .pxd -> Python module
I'm not sure we're agreed on this one.
Same from here. To me, that doesn't make much sense for code that wraps a library. And if it doesn't wrap a library, there isn't much benefit in writing a stand-alone .pxd in the first place. A .pyx is much more explicit and obvious in this case. Especially having some .pxd files that generate .so files and others that don't will make this very ugly.
I'd prefer adding support for cimporting from .pyx files instead, potentially with an automated caching generation of corresponding .pxd files (maybe as ".pxdg" files to make them easier to handle for users). However, cyclic dependencies would be tricky to handle automatically then.
That's fine with me, since I just checked and you can do cdef extern from 'somelib.h': in .pyx files. Why would cyclic dependencies be difficult? The contents of .pxdg files should only depend on the paired .pyx file, so there is no need to parse another .pyx/.pxd file when generating a .pxdg. After you've created a .pxdg file, you're in the same situation that Cython presumably already handles well, with a .pyx/.pxd(g) pair.
* Extending class cdef/cdpef/public/readonly handling to cover enums, stucts, and possibly unions.
This seems like the best first step.
+1
Working on it...
* I don't know how to handle things like dummy enums (perhaps by requiring all cdef-ed enums to be named).
All enums in C are named.
But my Cython declaration (exposing a C `#define CONST_A 1`):
cdef extern from 'mylib.h': enum: CONST_A
is not a named enum.
Ah, yes. Maybe we require a name (that would only be used in Python space).
... require it for cpdef enums, you mean?
OTOH, the "enum: NAME" scheme is ugly by itself. There should be a way to declare external constants correctly. After all, we loose all type information that way. I just saw that in math.pxd things like "M_PI" are declared as plain "enum" instead of "const double" or something. The type inferencer can't do anything with that. It might even draw the completely wrong conclusions.
Something like: [cdef|cpdef] extern [public|readonly] <type> <name> For example: cdef extern readonly double M_PI That would be nice, since the C compiler would (I think) raise an error when you try to use an invalid <type> for macro value. [1]: http://projects.scipy.org/numpy/ticket/1686 -- This email may be signed or encrypted with GPG (http://www.gnupg.org). The GPG signature (if present) will be attached as 'signature.asc'. For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy My public key is at http://www.physics.drexel.edu/~wking/pubkey.txt
On Sat, Feb 19, 2011 at 11:22 AM, W. Trevor King <wking@drexel.edu> wrote:
If you try to override anything in a .so compiled module at runtime, you'd get the same kind of error you currently do trying to rebind a compiled class' method.
That's the desired behavior for statically-bound globals, but implementing it is not so trivial.
It's currently implemented for classes. Are modules that different from classes?
>>> import types >>> types.ModuleType.__class__.__mro__ (<type 'type'>, <type 'object'>)
So you've got your standard __setattr__ to override with an error-message generator. What is the implementation difficulty?
You can't just add a __setattr__ to a module and have it work, it's a class-level slot. And you don't want to modify all module classes. To do this you have to subclass module itself and insert that in place during import.
I am also unclear about the distinction between cdef and cpdef (perhaps this should be a new thread?).
cpdef means "I'm declaring something with C and Python interfaces. The Python interface is a thin wrapper which can be rebound to an object of any type, leaving the static C interface inaccessible from Python."
No, you can't re-bind cdef class methods. Despite Cython's attempt to homogenize things for the user, extension classes are quite different than "normal" Python classes. This is a Python C/API issue.
cdef [private] means "I'm declaring something that only has a C interface." cdef public means "I'm declaring something with C and Python interfaces backed by C data. Python code can alter the C data." cdef readonly means "I'm declaring something with C and Python interfaces backed by C data. Python code cannot alter the C data."
cdef means "back this by a C variable"
This seems to be broken in Cython at the module level, since I can rebind a cdef-ed class but not a cpdef-ed method:
Yes, we don't currently control the module's set/getattr. And classes are "public" by default.
$ cat square.pyx cdef class A (object): cdef public int value
cpdef square(self): return self.value**2 $ python -c 'import pyximport; pyximport.install(); import xx; a = xx.A(); a.value = 3; print a.square(); a.square = lambda self: self.value' Traceback (most recent call last): File "<string>", line 3, in <module> AttributeError: 'square.A' object attribute 'square' is read-only $ python -c 'import pyximport; pyximport.install(); import square; square.A = object; print square.A; a = square.A(); a.value = 3' <type 'object'> Traceback (most recent call last): File "<string>", line 2, in <module> AttributeError: 'object' object has no attribute 'value'
So the cdef-ed A currently has a rebindable Python interface (acting like a hypothetical cpdef-ed class), but it's square method is not rebindable (acting like a hypothetical `cdef readonly`-ed method).
(and where would we do it--on the first import of a cimporting module?)
Compilation is an issue. I think that .pxd files should be able to be cythoned directly, since then they Cython can build any wrappers they request. If the file has a matching .pyx file, cythoning either one should compile both together, since they'll produce a single Python .so module.
...
Under the mantra "explicit is better than implicit", we could have users add something like
cdef module "modname"
to any .pxd files that should be inflated into Python modules. .pxd files without such a tag would receive the current treatment, error on any cpdef, etc. The drawback of this approach is that it makes Cython more complicated, but if both behaviors are reasonable, there's probably no getting around that.
The other drawback is that it subverts the usual filename <-> module name convention that one usually expects.
I've been convinced that the `cimport .pyx file` route is a better way to go.
However, the filename <-> module mapping is troublesome for backing externally-implemented Python modules (e.g. numpy). If you wanted to write a .pxd file backing numpy.random, how would you go about getting your module installed in Cython/Includes/numpy/random.pxd or another path that cython would successfully match with `cimport numpy.random`?
Note that extern blocks (by definition) declare where things come from.
On Sat, Feb 19, 2011 at 10:24:05AM +0100, Stefan Behnel wrote:
Robert Bradshaw, 18.02.2011 23:08:
On Thu, Feb 17, 2011 at 8:38 PM, W. Trevor King wrote:
On Thu, Feb 17, 2011 at 3:53 PM, Robert Bradshaw wrote:
On Thu, Feb 17, 2011 at 3:12 PM, W. Trevor King wrote:
>> A side effect of this cpdef change would be that now even bare .pxd >> files (no matching .pyx) would have a Python presence, > > Where would it live? Would we just create this module (in essence, > acting as if there was an empty .pyx file sitting there as well)? On > this note, it may be worth pursuing the idea of a "cython helper" > module where common code and objects could live.
I'm not sure exactly what you mean by "cython helper", but this sounds like my 'bare .pyx can create a Python .so module idea above.
I'm thinking of a place to put, e.g. the generator and bind-able function classes, which are now re-implemented in every module that uses them. I think there will be more cases like this in the future rather than less. C-level code could be #included and linked from "global" stores as well. However, that's somewhat tangential.
If you generate more than one file from a .pyx, including files that are shared between compiler runs (or even readily built as .so files), you'd quickly end up in dependency hell.
I disagree here, but I like your cimportable .pyx better, so it doesn't matter ;).
>> Unions don't really have a Python parallel, > > They can be a cdef class wrapping the union type.
But I would think coercion would be difficult. Unions are usually (in my limited experience) for "don't worry about the type, just make sure it fits in X bytes". How would union->Python conversion work?
There would be a wrapping type, e.g.
cdef class MyUnion: cdef union_type value
Wouldn't that have to be a pointer to the real thing instead?
Do you mean `cdef union_type *value`? Why would the above version not work? The union type has a well defined size and a number of well defined interpretations, so I don't see the problem.
with a bunch of setters/getters for the values, just like there are for structs. (In fact the same code would handle structs and unions).
This is getting into the wrapper-generator territory, but I'm starting to think for simple things that might be worth it.
I think that if Cython will automatically generate a wrapper for
cdef public int x
it should generate a wrapper for
cdef struct X: cdef public int x
Or
cdef public struct X: int x readonly int z private int z
I would perhaps say that non-Pythonable non-private members in public structs would be a compile error.
+1, keep it safe at the beginning.
-1, keep the code clean and the interface consistent ;). I think the struct syntax should be identical to the class syntax, with the exception that you can't bind methods to structs. That's the only real difference between structs and classes, isn't it?
In C++, the only difference between structs and classes is that struct members are public by default. (Not saying that C++ is always the model to follow, but it gives precedent). And structs can have function members, that's how to do OOP in C.
If safety with a new feature is a concern, a warning like "EXPERIMENTAL FEATURE" in the associated docs and compiler output should be sufficient.
I think the point of "safe" is to start out with a compiler error, and we can change our minds later, which is better than trying to make legal statements illegal in the future.
There really aren't that metatypes in C, so it doesn't seem like a slippery slope to me. Maybe I'm just missing something...
Ok, I think we're pretty much agreed ;). I think that the next step is to start working on implementations of:
* Stand alone .pxd -> Python module
I'm not sure we're agreed on this one.
Same from here. To me, that doesn't make much sense for code that wraps a library. And if it doesn't wrap a library, there isn't much benefit in writing a stand-alone .pxd in the first place. A .pyx is much more explicit and obvious in this case. Especially having some .pxd files that generate .so files and others that don't will make this very ugly.
I'd prefer adding support for cimporting from .pyx files instead, potentially with an automated caching generation of corresponding .pxd files (maybe as ".pxdg" files to make them easier to handle for users). However, cyclic dependencies would be tricky to handle automatically then.
That's fine with me, since I just checked and you can do cdef extern from 'somelib.h': in .pyx files.
Why would cyclic dependencies be difficult? The contents of .pxdg files should only depend on the paired .pyx file, so there is no need to parse another .pyx/.pxd file when generating a .pxdg. After you've created a .pxdg file, you're in the same situation that Cython presumably already handles well, with a .pyx/.pxd(g) pair.
Cyclic dependancies might require some special work, but they already do.
* Extending class cdef/cdpef/public/readonly handling to cover enums, stucts, and possibly unions.
This seems like the best first step.
+1
Working on it...
* I don't know how to handle things like dummy enums (perhaps by requiring all cdef-ed enums to be named).
All enums in C are named.
But my Cython declaration (exposing a C `#define CONST_A 1`):
cdef extern from 'mylib.h': enum: CONST_A
is not a named enum.
Ah, yes. Maybe we require a name (that would only be used in Python space).
... require it for cpdef enums, you mean?
OTOH, the "enum: NAME" scheme is ugly by itself. There should be a way to declare external constants correctly. After all, we loose all type information that way. I just saw that in math.pxd things like "M_PI" are declared as plain "enum" instead of "const double" or something. The type inferencer can't do anything with that. It might even draw the completely wrong conclusions.
Something like:
[cdef|cpdef] extern [public|readonly] <type> <name>
For example:
cdef extern readonly double M_PI
That would be nice, since the C compiler would (I think) raise an error when you try to use an invalid <type> for macro value.
Const is different than readonly, as readonly specifies the python-level accessibility. - Robert
On Sat, Feb 19, 2011 at 12:47:41PM -0800, Robert Bradshaw wrote:
On Sat, Feb 19, 2011 at 11:22 AM, W. Trevor King <wking@drexel.edu> wrote:
If you try to override anything in a .so compiled module at runtime, you'd get the same kind of error you currently do trying to rebind a compiled class' method.
That's the desired behavior for statically-bound globals, but implementing it is not so trivial.
It's currently implemented for classes. Are modules that different from classes?
import types types.ModuleType.__class__.__mro__ (<type 'type'>, <type 'object'>)
So you've got your standard __setattr__ to override with an error-message generator. What is the implementation difficulty?
You can't just add a __setattr__ to a module and have it work, it's a class-level slot. And you don't want to modify all module classes. To do this you have to subclass module itself and insert that in place during import.
So annoying but possible? Not particularly critical either way, though, since you could always say "don't rebind module-level stuff" in a project's docs, even if you don't enforce that at the Cython level.
I am also unclear about the distinction between cdef and cpdef (perhaps this should be a new thread?).
cpdef means "I'm declaring something with C and Python interfaces. The Python interface is a thin wrapper which can be rebound to an object of any type, leaving the static C interface inaccessible from Python."
No, you can't re-bind cdef class methods. Despite Cython's attempt to homogenize things for the user, extension classes are quite different than "normal" Python classes. This is a Python C/API issue.
Do you mean `cpdef class methods`? If so, you're right: $ cat square.pyx cdef class A (object): cdef public int value cpdef square(self): return self.value**2 $ python -c 'import pyximport; pyximport.install(); import square; square.A.square = lambda self: self.value' Traceback (most recent call last): File "<string>", line 1, in <module> TypeError: can't set attributes of built-in/extension type 'square.A' You can't override them in instances either: $ python -c 'import pyximport; pyximport.install(); import square; a = square.A(); a.square = lambda self: self.value' Traceback (most recent call last): File "<string>", line 1, in <module> AttributeError: 'square.A' object attribute 'square' is read-only But you can override them in subclasses, which is, I suppose, the point of cpdef for methods.
cdef [private] means "I'm declaring something that only has a C interface." cdef public means "I'm declaring something with C and Python interfaces backed by C data. Python code can alter the C data." cdef readonly means "I'm declaring something with C and Python interfaces backed by C data. Python code cannot alter the C data."
cdef means "back this by a C variable"
Ah, ok. That makes more sense.
However, the filename <-> module mapping is troublesome for backing externally-implemented Python modules (e.g. numpy). If you wanted to write a .pxd file backing numpy.random, how would you go about getting your module installed in Cython/Includes/numpy/random.pxd or another path that cython would successfully match with `cimport numpy.random`?
Note that extern blocks (by definition) declare where things come from.
They declare where the .pxd file looks for .h files, but not where .pyx files look for the .pxd file.
cdef public struct X: int x readonly int z private int z
I would perhaps say that non-Pythonable non-private members in public structs would be a compile error.
+1, keep it safe at the beginning.
-1, keep the code clean and the interface consistent ;). I think the struct syntax should be identical to the class syntax, with the exception that you can't bind methods to structs. That's the only real difference between structs and classes, isn't it?
In C++, the only difference between structs and classes is that struct members are public by default. (Not saying that C++ is always the model to follow, but it gives precedent). And structs can have function members, that's how to do OOP in C.
Oh. Even more reason to have identical struct and class handling in Cython ;). It is unclear to me what `cdef public struct` means. I think it should mean "Python bindings can alter this struct's definition", which doesn't make sense. Shouldn't the syntax for public members be cdef struct X: cdef public: int x readonly int y private int z
If safety with a new feature is a concern, a warning like "EXPERIMENTAL FEATURE" in the associated docs and compiler output should be sufficient.
I think the point of "safe" is to start out with a compiler error, and we can change our minds later, which is better than trying to make legal statements illegal in the future.
Ok, but I still don't understand why the cdefs were removed from the proposed structure members, when they are required for class definitions.
That would be nice, since the C compiler would (I think) raise an error when you try to use an invalid <type> for macro value.
Const is different than readonly, as readonly specifies the python-level accessibility.
Ah. Sorry for all the c(p)def/qualifier confusion, but I'm trying to consolidate the way these are handled in Parsing/Nodes/Symtab and I want to make sure I don't implement the wrong interpretation. Can you clarify how one knows if "public" means "expose a read/write Python interface to this object" or "expose this symbol to external C code"? -- This email may be signed or encrypted with GPG (http://www.gnupg.org). The GPG signature (if present) will be attached as 'signature.asc'. For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy My public key is at http://www.physics.drexel.edu/~wking/pubkey.txt
On Sat, Feb 19, 2011 at 1:45 PM, W. Trevor King <wking@drexel.edu> wrote:
On Sat, Feb 19, 2011 at 12:47:41PM -0800, Robert Bradshaw wrote:
On Sat, Feb 19, 2011 at 11:22 AM, W. Trevor King <wking@drexel.edu> wrote:
If you try to override anything in a .so compiled module at runtime, you'd get the same kind of error you currently do trying to rebind a compiled class' method.
That's the desired behavior for statically-bound globals, but implementing it is not so trivial.
It's currently implemented for classes. Are modules that different from classes?
>>> import types >>> types.ModuleType.__class__.__mro__ (<type 'type'>, <type 'object'>)
So you've got your standard __setattr__ to override with an error-message generator. What is the implementation difficulty?
You can't just add a __setattr__ to a module and have it work, it's a class-level slot. And you don't want to modify all module classes. To do this you have to subclass module itself and insert that in place during import.
So annoying but possible? Not particularly critical either way, though, since you could always say "don't rebind module-level stuff" in a project's docs, even if you don't enforce that at the Cython level.
It's still something I'd like to do. This could also be used to allow fast access to module globals and easier sharing of declarations.
I am also unclear about the distinction between cdef and cpdef (perhaps this should be a new thread?).
cpdef means "I'm declaring something with C and Python interfaces. The Python interface is a thin wrapper which can be rebound to an object of any type, leaving the static C interface inaccessible from Python."
No, you can't re-bind cdef class methods. Despite Cython's attempt to homogenize things for the user, extension classes are quite different than "normal" Python classes. This is a Python C/API issue.
Do you mean `cpdef class methods`? If so, you're right:
$ cat square.pyx cdef class A (object): cdef public int value
cpdef square(self): return self.value**2 $ python -c 'import pyximport; pyximport.install(); import square; square.A.square = lambda self: self.value' Traceback (most recent call last): File "<string>", line 1, in <module> TypeError: can't set attributes of built-in/extension type 'square.A'
You can't override them in instances either:
$ python -c 'import pyximport; pyximport.install(); import square; a = square.A(); a.square = lambda self: self.value' Traceback (most recent call last): File "<string>", line 1, in <module> AttributeError: 'square.A' object attribute 'square' is read-only
But you can override them in subclasses, which is, I suppose, the point of cpdef for methods.
Yes, otherwise they would have been pretty easy to implement :).
cdef [private] means "I'm declaring something that only has a C interface." cdef public means "I'm declaring something with C and Python interfaces backed by C data. Python code can alter the C data." cdef readonly means "I'm declaring something with C and Python interfaces backed by C data. Python code cannot alter the C data."
cdef means "back this by a C variable"
Ah, ok. That makes more sense.
However, the filename <-> module mapping is troublesome for backing externally-implemented Python modules (e.g. numpy). If you wanted to write a .pxd file backing numpy.random, how would you go about getting your module installed in Cython/Includes/numpy/random.pxd or another path that cython would successfully match with `cimport numpy.random`?
Note that extern blocks (by definition) declare where things come from.
They declare where the .pxd file looks for .h files, but not where .pyx files look for the .pxd file.
Sorry, I should have said extern blocks that make cdef class declarations (such as our numpy.pxd).
cdef public struct X: int x readonly int z private int z
I would perhaps say that non-Pythonable non-private members in public structs would be a compile error.
+1, keep it safe at the beginning.
-1, keep the code clean and the interface consistent ;). I think the struct syntax should be identical to the class syntax, with the exception that you can't bind methods to structs. That's the only real difference between structs and classes, isn't it?
In C++, the only difference between structs and classes is that struct members are public by default. (Not saying that C++ is always the model to follow, but it gives precedent). And structs can have function members, that's how to do OOP in C.
Oh. Even more reason to have identical struct and class handling in Cython ;).
It is unclear to me what `cdef public struct` means. I think it should mean "Python bindings can alter this struct's definition", which doesn't make sense.
I think it should mean "this struct is accessible from Python (as X)"
Shouldn't the syntax for public members be
cdef struct X: cdef public: int x readonly int y private int z
-1 on nesting things like this. Rather than make a struct visible from Python iff any of its members are, I think it makes more sense to put the declaration on the struct itself. We could support cdef public struct X: int x # public cdef readonly struct Y: int y # readonly cdef [private] struct Z: int z # private, as we don't even have Z in the Python namespace, and no wrapper is created.
If safety with a new feature is a concern, a warning like "EXPERIMENTAL FEATURE" in the associated docs and compiler output should be sufficient.
I think the point of "safe" is to start out with a compiler error, and we can change our minds later, which is better than trying to make legal statements illegal in the future.
Ok, but I still don't understand why the cdefs were removed from the proposed structure members, when they are required for class definitions.
Because structs can only have c members, so the cdef was entirely redundant. They weren't really removed per say, it's just that with the exception of cdef classes, "cdef ..." meant "a c declaration follows."
That would be nice, since the C compiler would (I think) raise an error when you try to use an invalid <type> for macro value.
Const is different than readonly, as readonly specifies the python-level accessibility.
Ah. Sorry for all the c(p)def/qualifier confusion, but I'm trying to consolidate the way these are handled in Parsing/Nodes/Symtab and I want to make sure I don't implement the wrong interpretation. Can you clarify how one knows if "public" means "expose a read/write Python interface to this object" or "expose this symbol to external C code"?
Public has had several different meanings. I wish there were a spec and full grammer for Cython, but there's not (yet?). The meaning is implicit in the code, and there's enough users out there that we should stay backwards compatible. It may be worth doing some backwards-incompatible normalization before we hit 1.0, but compiling the entire Python grammar is higher priority than that. - Robert
On Sat, Feb 19, 2011 at 02:04:16PM -0800, Robert Bradshaw wrote:
On Sat, Feb 19, 2011 at 1:45 PM, W. Trevor King <wking@drexel.edu> wrote:
On Sat, Feb 19, 2011 at 12:47:41PM -0800, Robert Bradshaw wrote:
On Sat, Feb 19, 2011 at 11:22 AM, W. Trevor King <wking@drexel.edu> wrote:
However, the filename <-> module mapping is troublesome for backing externally-implemented Python modules (e.g. numpy). If you wanted to write a .pxd file backing numpy.random, how would you go about getting your module installed in Cython/Includes/numpy/random.pxd or another path that cython would successfully match with `cimport numpy.random`?
Note that extern blocks (by definition) declare where things come from.
They declare where the .pxd file looks for .h files, but not where .pyx files look for the .pxd file.
Sorry, I should have said extern blocks that make cdef class declarations (such as our numpy.pxd).
It doesn't look like there are cdef class declarations in numpy.pxd: cython $ grep class Cython/Includes/numpy.pxd ctypedef class numpy.dtype [object PyArray_Descr]: ctypedef extern class numpy.flatiter [object PyArrayIterObject]: ctypedef extern class numpy.broadcast [object PyArrayMultiIterObject]: ctypedef class numpy.ndarray [object PyArrayObject]: ctypedef extern class numpy.ufunc [object PyUFuncObject]: This still doesn't explain how .pxd files specify which external implemented Python modules they correspond to.
cdef public struct X: int x readonly int z private int z
I would perhaps say that non-Pythonable non-private members in public structs would be a compile error.
+1, keep it safe at the beginning.
-1, keep the code clean and the interface consistent ;). I think the struct syntax should be identical to the class syntax, with the exception that you can't bind methods to structs. That's the only real difference between structs and classes, isn't it?
In C++, the only difference between structs and classes is that struct members are public by default. (Not saying that C++ is always the model to follow, but it gives precedent). And structs can have function members, that's how to do OOP in C.
Oh. Even more reason to have identical struct and class handling in Cython ;).
It is unclear to me what `cdef public struct` means. I think it should mean "Python bindings can alter this struct's definition", which doesn't make sense.
I think it should mean "this struct is accessible from Python (as X)"
Wouldn't that be "cdef readonly struct X"?
Shouldn't the syntax for public members be
cdef struct X: cdef public: int x readonly int y private int z
-1 on nesting things like this. Rather than make a struct visible from Python iff any of its members are,
A struct is visible from python iff it is declared public or readonly: cdef public struct X: ... or cdef readonly struct X: ... I don't think the visibility of the struct as a whole should have any effect over the visibility of its members, so you should be able to specify member visibility explicitly with per-member granularity (as you currently can for classes). I was assuming that structs would be public by default (like classes), but that is obviously configurable.
I think it makes more sense to put the declaration on the struct itself. We could support
cdef public struct X: int x # public
cdef readonly struct Y: int y # readonly
cdef [private] struct Z: int z # private, as we don't even have Z in the Python namespace, and no wrapper is created.
The problems with this are: * It's differnent from how we handle the almost identical class case. * It makes it impossible to define, for example a public struct with C-only attributes: cdef public struct X: cdef public int a cdef private void* ptr Obviously, public attributes of private structs should raise compile-time Cython errors.
If safety with a new feature is a concern, a warning like "EXPERIMENTAL FEATURE" in the associated docs and compiler output should be sufficient.
I think the point of "safe" is to start out with a compiler error, and we can change our minds later, which is better than trying to make legal statements illegal in the future.
Ok, but I still don't understand why the cdefs were removed from the proposed structure members, when they are required for class definitions.
Because structs can only have c members, so the cdef was entirely redundant. They weren't really removed per say, it's just that with the exception of cdef classes, "cdef ..." meant "a c declaration follows."
But cdef classes can also only have cdef members. I think it's better to keep cdef meaning "backed by C data", not necessarily "written using C syntax", since you're trying to do more with Cython, so it doesn't make sense to force C syntax.
That would be nice, since the C compiler would (I think) raise an error when you try to use an invalid <type> for macro value.
Const is different than readonly, as readonly specifies the python-level accessibility.
Ah. Sorry for all the c(p)def/qualifier confusion, but I'm trying to consolidate the way these are handled in Parsing/Nodes/Symtab and I want to make sure I don't implement the wrong interpretation. Can you clarify how one knows if "public" means "expose a read/write Python interface to this object" or "expose this symbol to external C code"?
Public has had several different meanings. I wish there were a spec and full grammer for Cython, but there's not (yet?). The meaning is implicit in the code, and there's enough users out there that we should stay backwards compatible. It may be worth doing some backwards-incompatible normalization before we hit 1.0, but compiling the entire Python grammar is higher priority than that.
Since I'm going to have lots of similar stuff (classes, enums, structs, unions) all with the same (hopefully) cdef/cpdef/visibility stuff for members, I'd like to consolidate now. I will of course, add special-case code as necessary to support the current syntax, which can then be removed whenever you think it is appropriate, but writing separate, near-identical handlers for each type seems like a recipe for disaster ;). I'll look to the code for guidance on public, and try to work out the appropriate meaning during the parse phase. -- This email may be signed or encrypted with GPG (http://www.gnupg.org). The GPG signature (if present) will be attached as 'signature.asc'. For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy My public key is at http://www.physics.drexel.edu/~wking/pubkey.txt
On Sat, Feb 19, 2011 at 3:31 PM, W. Trevor King <wking@drexel.edu> wrote:
On Sat, Feb 19, 2011 at 02:04:16PM -0800, Robert Bradshaw wrote:
On Sat, Feb 19, 2011 at 1:45 PM, W. Trevor King <wking@drexel.edu> wrote:
On Sat, Feb 19, 2011 at 12:47:41PM -0800, Robert Bradshaw wrote:
On Sat, Feb 19, 2011 at 11:22 AM, W. Trevor King <wking@drexel.edu> wrote:
However, the filename <-> module mapping is troublesome for backing externally-implemented Python modules (e.g. numpy). If you wanted to write a .pxd file backing numpy.random, how would you go about getting your module installed in Cython/Includes/numpy/random.pxd or another path that cython would successfully match with `cimport numpy.random`?
Note that extern blocks (by definition) declare where things come from.
They declare where the .pxd file looks for .h files, but not where .pyx files look for the .pxd file.
Sorry, I should have said extern blocks that make cdef class declarations (such as our numpy.pxd).
It doesn't look like there are cdef class declarations in numpy.pxd:
cython $ grep class Cython/Includes/numpy.pxd ctypedef class numpy.dtype [object PyArray_Descr]: ctypedef extern class numpy.flatiter [object PyArrayIterObject]: ctypedef extern class numpy.broadcast [object PyArrayMultiIterObject]: ctypedef class numpy.ndarray [object PyArrayObject]: ctypedef extern class numpy.ufunc [object PyUFuncObject]:
This still doesn't explain how .pxd files specify which external implemented Python modules they correspond to.
"numpy.dtype" is the fully qualified name of the class it's declaring--the module is "numpy."
> cdef public struct X: > int x > readonly int z > private int z > > I would perhaps say that non-Pythonable non-private members in public > structs would be a compile error.
+1, keep it safe at the beginning.
-1, keep the code clean and the interface consistent ;). I think the struct syntax should be identical to the class syntax, with the exception that you can't bind methods to structs. That's the only real difference between structs and classes, isn't it?
In C++, the only difference between structs and classes is that struct members are public by default. (Not saying that C++ is always the model to follow, but it gives precedent). And structs can have function members, that's how to do OOP in C.
Oh. Even more reason to have identical struct and class handling in Cython ;).
It is unclear to me what `cdef public struct` means. I think it should mean "Python bindings can alter this struct's definition", which doesn't make sense.
I think it should mean "this struct is accessible from Python (as X)"
Wouldn't that be "cdef readonly struct X"?
Shouldn't the syntax for public members be
cdef struct X: cdef public: int x readonly int y private int z
-1 on nesting things like this. Rather than make a struct visible from Python iff any of its members are,
A struct is visible from python iff it is declared public or readonly:
cdef public struct X: ...
or
cdef readonly struct X: ...
I don't think the visibility of the struct as a whole should have any effect over the visibility of its members, so you should be able to specify member visibility explicitly with per-member granularity (as you currently can for classes).
I was assuming that structs would be public by default (like classes), but that is obviously configurable.
I think it makes more sense to put the declaration on the struct itself. We could support
cdef public struct X: int x # public
cdef readonly struct Y: int y # readonly
cdef [private] struct Z: int z # private, as we don't even have Z in the Python namespace, and no wrapper is created.
The problems with this are:
* It's differnent from how we handle the almost identical class case.
True it's different, but there are significant differences between classes and structs in Cython, and this is the way things are now.
* It makes it impossible to define, for example a public struct with C-only attributes:
cdef public struct X: cdef public int a cdef private void* ptr
? The above would work just fine. I was proposing that it would be semantically equivalent to cdef public struct X: cdef int a cdef private void* ptr Perhaps that was not clear. Consider the currently valid cdef struct X: int a void* ptr We can't make everything public by default, as this would break valid code. (I don't think it's a good idea to make the default visibility of a member be a function of its type if we can help it.) The original proposal was to allow cpdef struct X: ... but making all fields private by default would be a bit useless, which was why I was suggesting they be public. We could allow the the modifier "cpdef readonly struct X" which would be the (probably less common) case where all members were by default readonly, though exposed to Python, rather than public. Within an exposed struct, of course, any member could be declared as private/readonly/public individually. BTW, the "public" keyword is the wrong thing to use here, as that actually controls name mangling and (c-level) symbol exporting. The fact that means a different thing for members than for top-level symbols isn't ideal, but at least it's unambiguous as members need not be mangled.
Obviously, public attributes of private structs should raise compile-time Cython errors.
If safety with a new feature is a concern, a warning like "EXPERIMENTAL FEATURE" in the associated docs and compiler output should be sufficient.
I think the point of "safe" is to start out with a compiler error, and we can change our minds later, which is better than trying to make legal statements illegal in the future.
Ok, but I still don't understand why the cdefs were removed from the proposed structure members, when they are required for class definitions.
Because structs can only have c members, so the cdef was entirely redundant. They weren't really removed per say, it's just that with the exception of cdef classes, "cdef ..." meant "a c declaration follows."
But cdef classes can also only have cdef members.
I think it's better to keep cdef meaning "backed by C data", not necessarily "written using C syntax", since you're trying to do more with Cython, so it doesn't make sense to force C syntax.
It means both. Were I to start over, I would make "cdef int* a, b, c" declare three pointers, but we're stuck with the C syntax we have. In a struct "cdef" on members is entirely redundant (though I am not strongly opposed to allowing it).
That would be nice, since the C compiler would (I think) raise an error when you try to use an invalid <type> for macro value.
Const is different than readonly, as readonly specifies the python-level accessibility.
Ah. Sorry for all the c(p)def/qualifier confusion, but I'm trying to consolidate the way these are handled in Parsing/Nodes/Symtab and I want to make sure I don't implement the wrong interpretation. Can you clarify how one knows if "public" means "expose a read/write Python interface to this object" or "expose this symbol to external C code"?
Public has had several different meanings. I wish there were a spec and full grammer for Cython, but there's not (yet?). The meaning is implicit in the code, and there's enough users out there that we should stay backwards compatible. It may be worth doing some backwards-incompatible normalization before we hit 1.0, but compiling the entire Python grammar is higher priority than that.
Since I'm going to have lots of similar stuff (classes, enums, structs, unions) all with the same (hopefully) cdef/cpdef/visibility stuff for members, I'd like to consolidate now. I will of course, add special-case code as necessary to support the current syntax, which can then be removed whenever you think it is appropriate, but writing separate, near-identical handlers for each type seems like a recipe for disaster ;).
Yes. For structs/unions, it's already almost always a StructOrUnion object anyways. Classes are somewhat special.
I'll look to the code for guidance on public, and try to work out the appropriate meaning during the parse phase.
Sounds good. - Robert
On Sat, Feb 19, 2011 at 04:41:27PM -0800, Robert Bradshaw wrote:
On Sat, Feb 19, 2011 at 3:31 PM, W. Trevor King <wking@drexel.edu> wrote:
On Sat, Feb 19, 2011 at 02:04:16PM -0800, Robert Bradshaw wrote:
On Sat, Feb 19, 2011 at 1:45 PM, W. Trevor King <wking@drexel.edu> wrote:
On Sat, Feb 19, 2011 at 12:47:41PM -0800, Robert Bradshaw wrote:
On Sat, Feb 19, 2011 at 11:22 AM, W. Trevor King <wking@drexel.edu> wrote:
However, the filename <-> module mapping is troublesome for backing externally-implemented Python modules (e.g. numpy). If you wanted to write a .pxd file backing numpy.random, how would you go about getting your module installed in Cython/Includes/numpy/random.pxd or another path that cython would successfully match with `cimport numpy.random`?
Note that extern blocks (by definition) declare where things come from.
They declare where the .pxd file looks for .h files, but not where .pyx files look for the .pxd file.
Sorry, I should have said extern blocks that make cdef class declarations (such as our numpy.pxd).
It doesn't look like there are cdef class declarations in numpy.pxd:
cython $ grep class Cython/Includes/numpy.pxd ctypedef class numpy.dtype [object PyArray_Descr]: ctypedef extern class numpy.flatiter [object PyArrayIterObject]: ctypedef extern class numpy.broadcast [object PyArrayMultiIterObject]: ctypedef class numpy.ndarray [object PyArrayObject]: ctypedef extern class numpy.ufunc [object PyUFuncObject]:
This still doesn't explain how .pxd files specify which external implemented Python modules they correspond to.
"numpy.dtype" is the fully qualified name of the class it's declaring--the module is "numpy."
Hmm, that means that cimport numpy a = numpy.numpy.ndarray also compiles. Wouldn't it make more sense to mirror numpy's module structure when backing it? You do have nested .pxd modules for cpython, libc, etc.
> > cdef public struct X: > > int x > > readonly int z > > private int z > > > > I would perhaps say that non-Pythonable non-private members in public > > structs would be a compile error. > > +1, keep it safe at the beginning.
-1, keep the code clean and the interface consistent ;). I think the struct syntax should be identical to the class syntax, with the exception that you can't bind methods to structs. That's the only real difference between structs and classes, isn't it?
In C++, the only difference between structs and classes is that struct members are public by default. (Not saying that C++ is always the model to follow, but it gives precedent). And structs can have function members, that's how to do OOP in C.
Oh. Even more reason to have identical struct and class handling in Cython ;).
It is unclear to me what `cdef public struct` means. I think it should mean "Python bindings can alter this struct's definition", which doesn't make sense.
I think it should mean "this struct is accessible from Python (as X)"
Wouldn't that be "cdef readonly struct X"?
Shouldn't the syntax for public members be
cdef struct X: cdef public: int x readonly int y private int z
-1 on nesting things like this. Rather than make a struct visible from Python iff any of its members are,
A struct is visible from python iff it is declared public or readonly:
cdef public struct X: ...
or
cdef readonly struct X: ...
I don't think the visibility of the struct as a whole should have any effect over the visibility of its members, so you should be able to specify member visibility explicitly with per-member granularity (as you currently can for classes).
I was assuming that structs would be public by default (like classes), but that is obviously configurable.
I think it makes more sense to put the declaration on the struct itself. We could support
cdef public struct X: int x # public
cdef readonly struct Y: int y # readonly
cdef [private] struct Z: int z # private, as we don't even have Z in the Python namespace, and no wrapper is created.
The problems with this are:
* It's differnent from how we handle the almost identical class case.
True it's different, but there are significant differences between classes and structs in Cython, and this is the way things are now.
Ah, well. I suppose I'll leave the Parser pretty much alone. Classes and structs will end up being much closer on the Python interface side, perhaps the syntax will grow closer in future Cythons...
* It makes it impossible to define, for example a public struct with C-only attributes:
cdef public struct X: cdef public int a cdef private void* ptr
?
The above would work just fine. I was proposing that it would be semantically equivalent to
cdef public struct X: cdef int a cdef private void* ptr
Perhaps that was not clear. Consider the currently valid
cdef struct X: int a void* ptr
We can't make everything public by default, as this would break valid code. (I don't think it's a good idea to make the default visibility of a member be a function of its type if we can help it.)
Oh, I though (for some reason) that you were ruling out member-level visibility adjustments. Using the struct visibility to set member-level defaults is fine.
BTW, the "public" keyword is the wrong thing to use here, as that actually controls name mangling and (c-level) symbol exporting. The fact that means a different thing for members than for top-level symbols isn't ideal, but at least it's unambiguous as members need not be mangled.
"public" means "read/write Python interface" for struct attributes, I was assuming that would also apply to struct members acording to tutorial/cdef_classes. Should I use a different visibility name in structs and unions?
If safety with a new feature is a concern, a warning like "EXPERIMENTAL FEATURE" in the associated docs and compiler output should be sufficient.
I think the point of "safe" is to start out with a compiler error, and we can change our minds later, which is better than trying to make legal statements illegal in the future.
Ok, but I still don't understand why the cdefs were removed from the proposed structure members, when they are required for class definitions.
Because structs can only have c members, so the cdef was entirely redundant. They weren't really removed per say, it's just that with the exception of cdef classes, "cdef ..." meant "a c declaration follows."
But cdef classes can also only have cdef members.
I think it's better to keep cdef meaning "backed by C data", not necessarily "written using C syntax", since you're trying to do more with Cython, so it doesn't make sense to force C syntax.
It means both. Were I to start over, I would make "cdef int* a, b, c" declare three pointers, but we're stuck with the C syntax we have...
In a struct "cdef" on members is entirely redundant
As it is in a cdef class. If you don't want it there for struct/union members, though, then I'll do it that way. -- This email may be signed or encrypted with GPG (http://www.gnupg.org). The GPG signature (if present) will be attached as 'signature.asc'. For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy My public key is at http://www.physics.drexel.edu/~wking/pubkey.txt
On Sat, Feb 19, 2011 at 6:32 PM, W. Trevor King <wking@drexel.edu> wrote:
On Sat, Feb 19, 2011 at 04:41:27PM -0800, Robert Bradshaw wrote:
On Sat, Feb 19, 2011 at 3:31 PM, W. Trevor King <wking@drexel.edu> wrote:
On Sat, Feb 19, 2011 at 02:04:16PM -0800, Robert Bradshaw wrote:
On Sat, Feb 19, 2011 at 1:45 PM, W. Trevor King <wking@drexel.edu> wrote:
On Sat, Feb 19, 2011 at 12:47:41PM -0800, Robert Bradshaw wrote:
On Sat, Feb 19, 2011 at 11:22 AM, W. Trevor King <wking@drexel.edu> wrote: > However, the filename <-> module mapping is troublesome for backing > externally-implemented Python modules (e.g. numpy). If you wanted to > write a .pxd file backing numpy.random, how would you go about getting > your module installed in Cython/Includes/numpy/random.pxd or another > path that cython would successfully match with `cimport numpy.random`?
Note that extern blocks (by definition) declare where things come from.
They declare where the .pxd file looks for .h files, but not where .pyx files look for the .pxd file.
Sorry, I should have said extern blocks that make cdef class declarations (such as our numpy.pxd).
It doesn't look like there are cdef class declarations in numpy.pxd:
cython $ grep class Cython/Includes/numpy.pxd ctypedef class numpy.dtype [object PyArray_Descr]: ctypedef extern class numpy.flatiter [object PyArrayIterObject]: ctypedef extern class numpy.broadcast [object PyArrayMultiIterObject]: ctypedef class numpy.ndarray [object PyArrayObject]: ctypedef extern class numpy.ufunc [object PyUFuncObject]:
This still doesn't explain how .pxd files specify which external implemented Python modules they correspond to.
"numpy.dtype" is the fully qualified name of the class it's declaring--the module is "numpy."
Hmm, that means that
cimport numpy a = numpy.numpy.ndarray
also compiles.
As does a = numpy.foo.x.y.z though perhaps that should be a compile-time error. To get the type, you have to write cimport numpy a = numpy.ndarray.
Wouldn't it make more sense to mirror numpy's module structure when backing it? You do have nested .pxd modules for cpython, libc, etc.
Yes, in fact that's what we do, but that's not required (e.g. you can have these extern declarations in .pyx files)
>> > cdef public struct X: >> > int x >> > readonly int z >> > private int z >> > >> > I would perhaps say that non-Pythonable non-private members in public >> > structs would be a compile error. >> >> +1, keep it safe at the beginning. > > -1, keep the code clean and the interface consistent ;). I think the > struct syntax should be identical to the class syntax, with the > exception that you can't bind methods to structs. That's the only > real difference between structs and classes, isn't it?
In C++, the only difference between structs and classes is that struct members are public by default. (Not saying that C++ is always the model to follow, but it gives precedent). And structs can have function members, that's how to do OOP in C.
Oh. Even more reason to have identical struct and class handling in Cython ;).
It is unclear to me what `cdef public struct` means. I think it should mean "Python bindings can alter this struct's definition", which doesn't make sense.
I think it should mean "this struct is accessible from Python (as X)"
Wouldn't that be "cdef readonly struct X"?
Shouldn't the syntax for public members be
cdef struct X: cdef public: int x readonly int y private int z
-1 on nesting things like this. Rather than make a struct visible from Python iff any of its members are,
A struct is visible from python iff it is declared public or readonly:
cdef public struct X: ...
or
cdef readonly struct X: ...
I don't think the visibility of the struct as a whole should have any effect over the visibility of its members, so you should be able to specify member visibility explicitly with per-member granularity (as you currently can for classes).
I was assuming that structs would be public by default (like classes), but that is obviously configurable.
I think it makes more sense to put the declaration on the struct itself. We could support
cdef public struct X: int x # public
cdef readonly struct Y: int y # readonly
cdef [private] struct Z: int z # private, as we don't even have Z in the Python namespace, and no wrapper is created.
The problems with this are:
* It's differnent from how we handle the almost identical class case.
True it's different, but there are significant differences between classes and structs in Cython, and this is the way things are now.
Ah, well. I suppose I'll leave the Parser pretty much alone. Classes and structs will end up being much closer on the Python interface side, perhaps the syntax will grow closer in future Cythons...
Well, we certainly can't get rid of the old syntax without a lot of thought and justification.
* It makes it impossible to define, for example a public struct with C-only attributes:
cdef public struct X: cdef public int a cdef private void* ptr
?
The above would work just fine. I was proposing that it would be semantically equivalent to
cdef public struct X: cdef int a cdef private void* ptr
Perhaps that was not clear. Consider the currently valid
cdef struct X: int a void* ptr
We can't make everything public by default, as this would break valid code. (I don't think it's a good idea to make the default visibility of a member be a function of its type if we can help it.)
Oh, I though (for some reason) that you were ruling out member-level visibility adjustments. Using the struct visibility to set member-level defaults is fine.
BTW, the "public" keyword is the wrong thing to use here, as that actually controls name mangling and (c-level) symbol exporting. The fact that means a different thing for members than for top-level symbols isn't ideal, but at least it's unambiguous as members need not be mangled.
"public" means "read/write Python interface" for struct attributes,
"public" means "C-visible" for global names, "read/write Python interface" for (cdef class) members.
I was assuming that would also apply to struct members acording to tutorial/cdef_classes. Should I use a different visibility name in structs and unions?
No, I think public/readonliy/private is a perfectly fine thing for struct members. It's just that declaring the struct itself as public already has a different meaning.
> If safety with a new feature is a concern, a warning like > "EXPERIMENTAL FEATURE" in the associated docs and compiler output > should be sufficient.
I think the point of "safe" is to start out with a compiler error, and we can change our minds later, which is better than trying to make legal statements illegal in the future.
Ok, but I still don't understand why the cdefs were removed from the proposed structure members, when they are required for class definitions.
Because structs can only have c members, so the cdef was entirely redundant. They weren't really removed per say, it's just that with the exception of cdef classes, "cdef ..." meant "a c declaration follows."
But cdef classes can also only have cdef members.
I think it's better to keep cdef meaning "backed by C data", not necessarily "written using C syntax", since you're trying to do more with Cython, so it doesn't make sense to force C syntax.
It means both. Were I to start over, I would make "cdef int* a, b, c" declare three pointers, but we're stuck with the C syntax we have...
In a struct "cdef" on members is entirely redundant
As it is in a cdef class. If you don't want it there for struct/union members, though, then I'll do it that way.
We're sticking with backwards compatibility. - Robert
W. Trevor King, 20.02.2011 03:32:
On Sat, Feb 19, 2011 at 04:41:27PM -0800, Robert Bradshaw wrote:
On Sat, Feb 19, 2011 at 3:31 PM, W. Trevor King wrote:
I think it's better to keep cdef meaning "backed by C data", not necessarily "written using C syntax", since you're trying to do more with Cython, so it doesn't make sense to force C syntax.
It means both. Were I to start over, I would make "cdef int* a, b, c" declare three pointers
+100
but we're stuck with the C syntax we have...
Sadly, yes. This will be impossible to change in the future without breaking all sorts of code in a hard-to-fix-automatically way.
In a struct "cdef" on members is entirely redundant
As it is in a cdef class.
Not quite. The class body of (cdef) classes can potentially contain code, so it's important for the parser to know what is a declaration and what is an executable statement. Stefan
W. Trevor King wrote:
On Sat, Feb 19, 2011 at 04:41:27PM -0800, Robert Bradshaw wrote:
On Sat, Feb 19, 2011 at 3:31 PM, W. Trevor King <wking@drexel.edu> wrote:
cython $ grep class Cython/Includes/numpy.pxd ctypedef class numpy.dtype [object PyArray_Descr]: ctypedef extern class numpy.flatiter [object PyArrayIterObject]: ctypedef extern class numpy.broadcast [object PyArrayMultiIterObject]: ctypedef class numpy.ndarray [object PyArrayObject]: ctypedef extern class numpy.ufunc [object PyUFuncObject]:
"numpy.dtype" is the fully qualified name of the class it's declaring--the module is "numpy."
Aren't the module names in the class declarations redundant here? Since they're in a file called numpy.pxd, declarations will go into the numpy module namespace by default. You should be able to write these as just ctypedef class dtype [object PyArray_Descr]: ctypedef class flatiter [object PyArrayIterObject]: ctypedef class broadcast [object PyArrayMultiIterObject]: ctypedef class ndarray [object PyArrayObject]: ctypedef class ufunc [object PyUFuncObject]: The 'extern' syntax for extension classes, including a module name, is really an obsolete feature. Before Pyrex had .pxd files, it was the only way to declare an externally implemented extension type to Pyrex. But now that .pxd files exist, there should be no need for it.
Hmm, that means that
cimport numpy a = numpy.numpy.ndarray
also compiles.
Compiles and runs, or just compiles? If this works as a way of getting hold of the ndarray type, then it's a bug -- it's not supposed to work that way. -- Greg
On Mon, Feb 21, 2011 at 10:09:42AM +1300, Greg Ewing wrote:
W. Trevor King wrote:
Hmm, that means that
cimport numpy a = numpy.numpy.ndarray
also compiles.
Compiles and runs, or just compiles?
You're right. Cython compilation worked, but it didn't run: $ export CFLAGS="-I/usr/lib/python2.6/site-packages/numpy/core/include/" $ python -c 'import pyximport; pyximport.install(); import np;' Traceback (most recent call last): ... File "np.pyx", line 3, in init np (/.../np.c:2846) a = numpy.numpy.ndarray ImportError: Building module failed: ['NameError: numpy\n'] Sorry for muddying the waters with that claim. -- This email may be signed or encrypted with GPG (http://www.gnupg.org). The GPG signature (if present) will be attached as 'signature.asc'. For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy My public key is at http://www.physics.drexel.edu/~wking/pubkey.txt
Robert Bradshaw wrote:
BTW, the "public" keyword is the wrong thing to use here, as that actually controls name mangling and (c-level) symbol exporting. The fact that means a different thing for members than for top-level symbols isn't ideal, but at least it's unambiguous as members need not be mangled.
The overloading of 'public' is really a bit of a mess. I've been thinking for a while that there really ought to be a different keyword such as "exposed" for declaring that things are to be exposed to Python. It would be useful in lots of ways: cdef class Foo: cdef exposed int i # formerly 'public' cdef exposed enum E: a, b, c # creates Python bindings for these names cdef exposed struct S: # Exposed but mangled as usual ... cdef public exposed struct S: # Exposed and unmangled ... -- Greg
Greg Ewing, 20.02.2011 21:13:
Robert Bradshaw wrote:
BTW, the "public" keyword is the wrong thing to use here, as that actually controls name mangling and (c-level) symbol exporting. The fact that means a different thing for members than for top-level symbols isn't ideal, but at least it's unambiguous as members need not be mangled.
The overloading of 'public' is really a bit of a mess. I've been thinking for a while that there really ought to be a different keyword such as "exposed" for declaring that things are to be exposed to Python. It would be useful in lots of ways:
cdef class Foo: cdef exposed int i # formerly 'public'
cdef exposed enum E: a, b, c # creates Python bindings for these names
cdef exposed struct S: # Exposed but mangled as usual ...
cdef public exposed struct S: # Exposed and unmangled ...
Given that Cython has "cpdef" already, why not just use that? Stefan
W. Trevor King, 20.02.2011 00:31:
On Sat, Feb 19, 2011 at 02:04:16PM -0800, Robert Bradshaw wrote:
On Sat, Feb 19, 2011 at 1:45 PM, W. Trevor King wrote:
It is unclear to me what `cdef public struct` means. I think it should mean "Python bindings can alter this struct's definition", which doesn't make sense.
I think it should mean "this struct is accessible from Python (as X)"
Wouldn't that be "cdef readonly struct X"?
The problem here is that "public" is used differently in different contexts - usually "exported at the C level" in this kind of context, with the quirk of meaning "modifiable at the Python level" for cdef class attributes. The potentially clearer "cpdef" means "overridable at the Python level" as well as "visible at the Python level", so it doesn't quite match the second meaning by itself. Structs won't be overridable at the Python level, I guess, so cpdef isn't quite right. The intention here isn't to export them at the C level either, so "public" isn't right. We could tweak the "cpdef" meaning into "cdef thing mapped to the Python level, and overridable if supported in the given context", which would lead to broader applicability. Then we could allow cpdef readonly struct ... I think that's a lot clearer than adding to the double meaning of "public". I would also prefer if Python accessible cdef class attributes were defined using "cpdef". We could potentially make that the preferred way in the future? Stefan
On Sun, Feb 20, 2011 at 1:26 AM, Stefan Behnel <stefan_ml@behnel.de> wrote:
W. Trevor King, 20.02.2011 00:31:
On Sat, Feb 19, 2011 at 02:04:16PM -0800, Robert Bradshaw wrote:
On Sat, Feb 19, 2011 at 1:45 PM, W. Trevor King wrote:
It is unclear to me what `cdef public struct` means. I think it should mean "Python bindings can alter this struct's definition", which doesn't make sense.
I think it should mean "this struct is accessible from Python (as X)"
Wouldn't that be "cdef readonly struct X"?
The problem here is that "public" is used differently in different contexts - usually "exported at the C level" in this kind of context, with the quirk of meaning "modifiable at the Python level" for cdef class attributes.
The potentially clearer "cpdef" means "overridable at the Python level" as well as "visible at the Python level", so it doesn't quite match the second meaning by itself.
Structs won't be overridable at the Python level, I guess, so cpdef isn't quite right. The intention here isn't to export them at the C level either, so "public" isn't right.
We could tweak the "cpdef" meaning into "cdef thing mapped to the Python level, and overridable if supported in the given context", which would lead to broader applicability. Then we could allow
That's what I was thinking.
cpdef readonly struct ...
I think that's a lot clearer than adding to the double meaning of "public". I would also prefer if Python accessible cdef class attributes were defined using "cpdef". We could potentially make that the preferred way in the future?
We could allow it, but -1 to disallowing "cdef class" - Robert
Robert Bradshaw, 21.02.2011 19:11:
On Sun, Feb 20, 2011 at 1:26 AM, Stefan Behnel wrote:
W. Trevor King, 20.02.2011 00:31:
On Sat, Feb 19, 2011 at 02:04:16PM -0800, Robert Bradshaw wrote:
On Sat, Feb 19, 2011 at 1:45 PM, W. Trevor King wrote:
It is unclear to me what `cdef public struct` means. I think it should mean "Python bindings can alter this struct's definition", which doesn't make sense.
I think it should mean "this struct is accessible from Python (as X)"
Wouldn't that be "cdef readonly struct X"?
The problem here is that "public" is used differently in different contexts - usually "exported at the C level" in this kind of context, with the quirk of meaning "modifiable at the Python level" for cdef class attributes.
The potentially clearer "cpdef" means "overridable at the Python level" as well as "visible at the Python level", so it doesn't quite match the second meaning by itself.
Structs won't be overridable at the Python level, I guess, so cpdef isn't quite right. The intention here isn't to export them at the C level either, so "public" isn't right.
We could tweak the "cpdef" meaning into "cdef thing mapped to the Python level, and overridable if supported in the given context", which would lead to broader applicability. Then we could allow
That's what I was thinking.
cpdef readonly struct ...
I think that's a lot clearer than adding to the double meaning of "public". I would also prefer if Python accessible cdef class attributes were defined using "cpdef". We could potentially make that the preferred way in the future?
We could allow it, but -1 to disallowing "cdef class"
With "preferred way", I was suggesting that we could *deprecate* cdef public int x cdef readonly object y for cdef class properties in favour of cpdef int x cpdef readonly object y and change the documentation accordingly etc., so that at least new users get used to the new way. The old way would likely continue to be supported until Cython 2.0 or so, for the holy cow's sake... Stefan
On 21 February 2011 15:26, Stefan Behnel <stefan_ml@behnel.de> wrote:
Robert Bradshaw, 21.02.2011 19:11:
On Sun, Feb 20, 2011 at 1:26 AM, Stefan Behnel wrote:
W. Trevor King, 20.02.2011 00:31:
On Sat, Feb 19, 2011 at 02:04:16PM -0800, Robert Bradshaw wrote:
On Sat, Feb 19, 2011 at 1:45 PM, W. Trevor King wrote:
It is unclear to me what `cdef public struct` means. I think it should mean "Python bindings can alter this struct's definition", which doesn't make sense.
I think it should mean "this struct is accessible from Python (as X)"
Wouldn't that be "cdef readonly struct X"?
The problem here is that "public" is used differently in different contexts - usually "exported at the C level" in this kind of context, with the quirk of meaning "modifiable at the Python level" for cdef class attributes.
The potentially clearer "cpdef" means "overridable at the Python level" as well as "visible at the Python level", so it doesn't quite match the second meaning by itself.
Structs won't be overridable at the Python level, I guess, so cpdef isn't quite right. The intention here isn't to export them at the C level either, so "public" isn't right.
We could tweak the "cpdef" meaning into "cdef thing mapped to the Python level, and overridable if supported in the given context", which would lead to broader applicability. Then we could allow
That's what I was thinking.
cpdef readonly struct ...
I think that's a lot clearer than adding to the double meaning of "public". I would also prefer if Python accessible cdef class attributes were defined using "cpdef". We could potentially make that the preferred way in the future?
We could allow it, but -1 to disallowing "cdef class"
With "preferred way", I was suggesting that we could *deprecate*
cdef public int x cdef readonly object y
for cdef class properties in favour of
cpdef int x cpdef readonly object y
and change the documentation accordingly etc., so that at least new users get used to the new way.
+1
The old way would likely continue to be supported until Cython 2.0 or so, for the holy cow's sake...
But with a deprecation warning, right? -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo 3000 Santa Fe, Argentina Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169
On Mon, Feb 21, 2011 at 10:26 AM, Stefan Behnel <stefan_ml@behnel.de> wrote:
Robert Bradshaw, 21.02.2011 19:11:
On Sun, Feb 20, 2011 at 1:26 AM, Stefan Behnel wrote:
W. Trevor King, 20.02.2011 00:31:
On Sat, Feb 19, 2011 at 02:04:16PM -0800, Robert Bradshaw wrote:
On Sat, Feb 19, 2011 at 1:45 PM, W. Trevor King wrote:
It is unclear to me what `cdef public struct` means. I think it should mean "Python bindings can alter this struct's definition", which doesn't make sense.
I think it should mean "this struct is accessible from Python (as X)"
Wouldn't that be "cdef readonly struct X"?
The problem here is that "public" is used differently in different contexts - usually "exported at the C level" in this kind of context, with the quirk of meaning "modifiable at the Python level" for cdef class attributes.
The potentially clearer "cpdef" means "overridable at the Python level" as well as "visible at the Python level", so it doesn't quite match the second meaning by itself.
Structs won't be overridable at the Python level, I guess, so cpdef isn't quite right. The intention here isn't to export them at the C level either, so "public" isn't right.
We could tweak the "cpdef" meaning into "cdef thing mapped to the Python level, and overridable if supported in the given context", which would lead to broader applicability. Then we could allow
That's what I was thinking.
cpdef readonly struct ...
I think that's a lot clearer than adding to the double meaning of "public". I would also prefer if Python accessible cdef class attributes were defined using "cpdef". We could potentially make that the preferred way in the future?
We could allow it, but -1 to disallowing "cdef class"
With "preferred way", I was suggesting that we could *deprecate*
cdef public int x cdef readonly object y
for cdef class properties in favour of
cpdef int x cpdef readonly object y
Oh, you were talking about members. For sure.
and change the documentation accordingly etc., so that at least new users get used to the new way. The old way would likely continue to be supported until Cython 2.0 or so, for the holy cow's sake...
:) This kind of thing is much easier to fix than, say, fallout from http://trac.cython.org/cython_trac/ticket/654 - Robert
Stefan Behnel wrote:
With "preferred way", I was suggesting that we could *deprecate*
cdef public int x cdef readonly object y
for cdef class properties in favour of
cpdef int x cpdef readonly object y
I think I've just realised one of the reasons for my gut dislike of the "cpdef" keyword -- it looks too similar to "def". At first glance, it's hard to spot the difference between the cdef and cpdef versions of two otherwise identical declarations. I think this is too subtle for something that makes such a big difference to semantics. It's often important that Python code is not allowed to mess with an object's internal state, so making it possible should require something more obvious. -- Greg
Greg Ewing, 21.02.2011 22:12:
Stefan Behnel wrote:
With "preferred way", I was suggesting that we could *deprecate*
cdef public int x cdef readonly object y
for cdef class properties in favour of
cpdef int x cpdef readonly object y
I think I've just realised one of the reasons for my gut dislike of the "cpdef" keyword -- it looks too similar to "def". At first glance, it's hard to spot the difference between the cdef and cpdef versions of two otherwise identical declarations.
I think this is too subtle for something that makes such a big difference to semantics. It's often important that Python code is not allowed to mess with an object's internal state, so making it possible should require something more obvious.
Hmm, I don't know. Maybe I'm just used to it already, but I don't find it hard to spot at all. The same argument could be brought up against "cdef" vs. "def" (between which the semantic difference is *huge*), and I hope you don't find that hard to spot. Stefan
Stefan Behnel wrote:
The same argument could be brought up against "cdef" vs. "def" (between which the semantic difference is *huge*)
There are a couple of differences: - 'cdef' and 'def' look very different (at least to me) because they *start* with a different letter. Whereas 'cdef' and 'cpdef' both start and end with the same letter, maknig tehm mcuh eazier to conufse. - There is much less semantic overlap betwen 'def' and 'cdef'. If you use the wrong one, you usually find out about it fairly quickly one way or another -- you get a syntax error (e.g. 'def struct'), or something doesn't work the way you expected (e.g. a function you thought you had exposed doesn't show up in Python). It's possible to accidentally expose a function, but only if you declare it with an implicit return type, which I never do when writing what I intend to be a C function (this could perhaps be made illegal to further reduce the chances of such an error). Mistaken use of 'cpdef' on a C attribute, on the other hand, would very often pass silently. -- Greg
Previous discussions in this thread [1,2] have discussed the issues associated with overloading the 'public' keyword. For an example of the difficulties this causes, see the ugly workaround [3] in my recent commit [4]. Definately worth fixing this syntax. How do syntax changes with deprecations work? The Parsing module doesn't seem to be setup to handle things like that. [1] http://mail.python.org/pipermail/cython-devel/2011-February/000081.html [2] http://mail.python.org/pipermail/cython-devel/2011-February/000086.html [3] https://github.com/wking/cython/commit/a918d1466928ea712817df18c3bc66d9c4c5c... [4] https://github.com/wking/cython/commit/a918d1466928ea712817df18c3bc66d9c4c5c... -- This email may be signed or encrypted with GPG (http://www.gnupg.org). The GPG signature (if present) will be attached as 'signature.asc'. For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy My public key is at http://www.physics.drexel.edu/~wking/pubkey.txt
On Wed, Mar 2, 2011 at 5:54 PM, W. Trevor King <wking@drexel.edu> wrote:
Previous discussions in this thread [1,2] have discussed the issues associated with overloading the 'public' keyword. For an example of the difficulties this causes, see the ugly workaround [3] in my recent commit [4]. Definately worth fixing this syntax.
How do syntax changes with deprecations work? The Parsing module doesn't seem to be setup to handle things like that.
It's not. We rarely deprecate syntax, and we should at least give people a chance to move away from the old syntax first to the new first, so we'll have to support both for a bit at least. When we do deprecate the old way of doing things, I think we'll just use Errors.warning(...). In the long run it'd be nice to have a formal grammar for Cython and use a generated parser rather than maintaining this all by hand, but Parsing.py does more than just parsing at the moment. - Robert
[1] http://mail.python.org/pipermail/cython-devel/2011-February/000081.html [2] http://mail.python.org/pipermail/cython-devel/2011-February/000086.html [3] https://github.com/wking/cython/commit/a918d1466928ea712817df18c3bc66d9c4c5c... [4] https://github.com/wking/cython/commit/a918d1466928ea712817df18c3bc66d9c4c5c...
-- This email may be signed or encrypted with GPG (http://www.gnupg.org). The GPG signature (if present) will be attached as 'signature.asc'. For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy
My public key is at http://www.physics.drexel.edu/~wking/pubkey.txt
_______________________________________________ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
On Wed, Mar 02, 2011 at 06:08:12PM -0800, Robert Bradshaw wrote:
On Wed, Mar 2, 2011 at 5:54 PM, W. Trevor King <wking@drexel.edu> wrote:
Previous discussions in this thread [1,2] have discussed the issues associated with overloading the 'public' keyword. For an example of the difficulties this causes, see the ugly workaround [3] in my recent commit [4]. Definately worth fixing this syntax.
How do syntax changes with deprecations work? The Parsing module doesn't seem to be setup to handle things like that.
It's not.
We rarely deprecate syntax, and we should at least give people a chance to move away from the old syntax first to the new first, so we'll have to support both for a bit at least. When we do deprecate the old way of doing things, I think we'll just use Errors.warning(...).
But how would you know which syntax version the source file was aiming for? There should probably be a way to explicitly specify a Cython syntax version, probably from the command line and/or setup.py file, e.g. pyrex_syntax_version = (<major version>, <minor version>, <sytax bugfix>) as an argument to Cython.Distutils.extension.Extension, which would select the appropriate parser. The default (if the version was not explicitly set), would be to use the current parser. With this setup, projects would only be forced to upgrade if Cython removed support for their syntax entirely, and they could choose to upgrade whenever was most convienient for them. I think the syntax version's major/minor number should probably match the Cython version in which they were introduced. Parser fixes and compatible extensions would bump the <syntax bugfix> number, and you'd probably want to deprecate buggy versions more quickly. -- This email may be signed or encrypted with GPG (http://www.gnupg.org). The GPG signature (if present) will be attached as 'signature.asc'. For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy My public key is at http://www.physics.drexel.edu/~wking/pubkey.txt
On Thu, Mar 3, 2011 at 4:13 AM, W. Trevor King <wking@drexel.edu> wrote:
On Wed, Mar 02, 2011 at 06:08:12PM -0800, Robert Bradshaw wrote:
On Wed, Mar 2, 2011 at 5:54 PM, W. Trevor King <wking@drexel.edu> wrote:
Previous discussions in this thread [1,2] have discussed the issues associated with overloading the 'public' keyword. For an example of the difficulties this causes, see the ugly workaround [3] in my recent commit [4]. Definately worth fixing this syntax.
How do syntax changes with deprecations work? The Parsing module doesn't seem to be setup to handle things like that.
It's not.
We rarely deprecate syntax, and we should at least give people a chance to move away from the old syntax first to the new first, so we'll have to support both for a bit at least. When we do deprecate the old way of doing things, I think we'll just use Errors.warning(...).
But how would you know which syntax version the source file was aiming for? There should probably be a way to explicitly specify a Cython syntax version, probably from the command line and/or setup.py file, e.g.
pyrex_syntax_version = (<major version>, <minor version>, <sytax bugfix>)
as an argument to Cython.Distutils.extension.Extension, which would select the appropriate parser. The default (if the version was not explicitly set), would be to use the current parser. With this setup, projects would only be forced to upgrade if Cython removed support for their syntax entirely, and they could choose to upgrade whenever was most convienient for them.
I think the syntax version's major/minor number should probably match the Cython version in which they were introduced. Parser fixes and compatible extensions would bump the <syntax bugfix> number, and you'd probably want to deprecate buggy versions more quickly.
With the single exception of introducing the cpdef keyword, I can't think of a single instance where Cython's syntax has ever been changed in a backwards incompatible way, so we've never needed to worry about syntax versioning. I don't see anything with the new syntax that's forcing us to be backwards incompatible, and even the local ugliness of supporting both for the time being is better than going down the route of supporting multiple parsers. - Robert
On Sat, Feb 19, 2011 at 06:31:26PM -0500, W. Trevor King wrote:
On Sat, Feb 19, 2011 at 02:04:16PM -0800, Robert Bradshaw wrote:
On Sat, Feb 19, 2011 at 1:45 PM, W. Trevor King <wking@drexel.edu> wrote:
Ah. Sorry for all the c(p)def/qualifier confusion, but I'm trying to consolidate the way these are handled in Parsing/Nodes/Symtab and I want to make sure I don't implement the wrong interpretation. Can you clarify how one knows if "public" means "expose a read/write Python interface to this object" or "expose this symbol to external C code"?
Public has had several different meanings. I wish there were a spec and full grammer for Cython, but there's not (yet?). The meaning is implicit in the code, and there's enough users out there that we should stay backwards compatible. It may be worth doing some backwards-incompatible normalization before we hit 1.0, but compiling the entire Python grammar is higher priority than that.
Since I'm going to have lots of similar stuff (classes, enums, structs, unions) all with the same (hopefully) cdef/cpdef/visibility stuff for members, I'd like to consolidate now. I will of course, add special-case code as necessary to support the current syntax, which can then be removed whenever you think it is appropriate, but writing separate, near-identical handlers for each type seems like a recipe for disaster ;).
I'll look to the code for guidance on public, and try to work out the appropriate meaning during the parse phase.
I've been working on a more explicit parser that removes the ambiguity behind the various visibilities. This will help me ensure proper impolementation of my cdef-ed enums/structs/..., and make it easier to update visibility syntax in the future. Take a look and let me know if you think this is a useful direction to take: git: http://www.physics.drexel.edu/~wking/code/git/cython.git branch: cdef-enums-stucts-and-unions gitweb: http://www.physics.drexel.edu/~wking/code/git/gitweb.cgi?p=cython.git;a=log;... -- This email may be signed or encrypted with GPG (http://www.gnupg.org). The GPG signature (if present) will be attached as 'signature.asc'. For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy My public key is at http://www.physics.drexel.edu/~wking/pubkey.txt
W. Trevor King, 22.02.2011 18:55:
I've been working on a more explicit parser that removes the ambiguity behind the various visibilities. This will help me ensure proper impolementation of my cdef-ed enums/structs/..., and make it easier to update visibility syntax in the future. Take a look and let me know if you think this is a useful direction to take:
git: http://www.physics.drexel.edu/~wking/code/git/cython.git branch: cdef-enums-stucts-and-unions gitweb: http://www.physics.drexel.edu/~wking/code/git/gitweb.cgi?p=cython.git;a=log;...
It doesn't seem like I can leave comments in the gitweb version, so I'll comment here. First thing that caught my eyes: I hope you do not intend to leave the logging usage in the parser. This is seriously performance critical code that Cython compiles down to pretty fast C code. Note that you can use a debugger and Python's profiling/tracing interface to find out what's happening, without code impact. Some of the log statements span more than one line, which makes it trickier to strip them out with sed&friends (but backing out the initial changeset would likely make it simple enough to remove the rest manually). Also note that it's best to write runnable tests ("tests/run/"). The tests in "tests/compile/" are only compiled and imported. See the hacking guide in the wiki. I know you're not there yet with your implementation, I'm just mentioning it. Most important point, however: I think it's a good idea to clean up the parsing context the way you did it. The semantic distinction of the three new classes you added makes sense to me. CtxAttribute is the wrong name, though. And the class copy implementation gets even more generic than it already was in the Ctx. I'm not a big fan of that, especially not in the parser. For one, it prevents changing the classes into cdef classes, which had long been on my list for Ctx. CSource: doesn't sound like quite the right name - it does not describe a C source file but information that Cython has about non-Cython things. I also doubt that Cython allows you to call an attribute "cdef", you'll need to change that. Stefan
On Tue, Feb 22, 2011 at 08:18:21PM +0100, Stefan Behnel wrote:
W. Trevor King, 22.02.2011 18:55:
I've been working on a more explicit parser that removes the ambiguity behind the various visibilities. This will help me ensure proper impolementation of my cdef-ed enums/structs/..., and make it easier to update visibility syntax in the future. Take a look and let me know if you think this is a useful direction to take:
First thing that caught my eyes: I hope you do not intend to leave the logging usage in the parser. This is seriously performance critical code that Cython compiles down to pretty fast C code. Note that you can use a debugger and Python's profiling/tracing interface to find out what's happening, without code impact.
I can strip them out afterwards, but it helps me figure out what I've broken if I shift too much around at the same time. I don't know enough about Python's trace module to know if I can turn on tracing only for functions defined in a single module or not, since otherwise its hard for me to separate signal from noise.
Some of the log statements span more than one line, which makes it trickier to strip them out with sed&friends (but backing out the initial changeset would likely make it simple enough to remove the rest manually).
Hmm, perhaps I'll condense the logging statements down onto one (long) line a piece, that will make it easy to comment/uncomment them with sed/emacs/etc. I suppose once Cython can compile the logging module we could leave them in with reduced overhead ;).
Also note that it's best to write runnable tests ("tests/run/"). The tests in "tests/compile/" are only compiled and imported. See the hacking guide in the wiki. I know you're not there yet with your implementation, I'm just mentioning it.
Thanks for the tip.
CtxAttribute is the wrong name, though. And the class copy implementation gets even more generic than it already was in the Ctx. I'm not a big fan of that, especially not in the parser. For one, it prevents changing the classes into cdef classes, which had long been on my list for Ctx.
An easy, if uglier, workaround would be to prepend attributes with the class name, e.g. CBinding.visibility -> CBinding.c_binding_visiblity. Then the Ctx class could subclass the current CtxAttribute classes instead of binding instances of each of them. That way Ctx would keep its traditional flat attribute namespace and easy deepcopy, but eveyone in Nodes, etc. that will use the attributes would become class-name dependent. The CtxAttribute class is, as its docstring says, just a hook for its deepcopy method. With an alternative deepcopy implementation, CtxAttribute could be replaced with the standard `object`, so don't worry too much about its name at this point ;).
CSource: doesn't sound like quite the right name - it does not describe a C source file but information that Cython has about non-Cython things.
It's a container for attributes that describe the presence and location of backing C definitions. * cdef: "Will there be a backing C defintion? * extern: "Has someone else already written it?" * name/namespace: "What did they call it?" If you'd rather I called the class something else, I'm certainly willing to change it.
I also doubt that Cython allows you to call an attribute "cdef", you'll need to change that.
It seems to work for me:
import Cython.Compiler.Parsing as P P.__file__ 'Cython/Compiler/Parsing.so' c = P.CSource() dir(c) [..., 'cdef', 'deepcopy', 'extern', 'name', 'namespace'] c.cdef 0 c.cdef = 1 c.cdef 1
However, I agree that it's generally a bad idea to play around with keywords. I'll revert it to the wordier-but-less-confusing `cdef_flag`. -- This email may be signed or encrypted with GPG (http://www.gnupg.org). The GPG signature (if present) will be attached as 'signature.asc'. For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy My public key is at http://www.physics.drexel.edu/~wking/pubkey.txt
W. Trevor King, 22.02.2011 18:55:
I've been working on a more explicit parser that removes the ambiguity behind the various visibilities. This will help me ensure proper impolementation of my cdef-ed enums/structs/..., and make it easier to update visibility syntax in the future. Take a look and let me know if you think this is a useful direction to take:
The refactoring continues as I'm moving my new binding classes into Symtab. This is, of course, leading to lots of changes, but I've been running the test suite before each commit to make sure I don't go to far astray. Anything I miss will eventually lead to a better test suite ;). I'm currently writing up new versions of most Scope methods that use my classes, and I'll replace the original methods once I've updated all the code that calls them. Next on the list will be the Nodes themselves, at which point I think I'll be positioned to put in the `cdef struct` and whatnot that got this whole thing started. Since there has been a fair amount of churn, I though I'd ask for some more feedback on the general direction I'm headed: http://www.physics.drexel.edu/~wking/code/git/gitweb.cgi?p=cython.git;a=log;... As I said in my previous response to Stefan, I don't really care what the attributes in my classes are called, or how the attributes are partitioned between them. With the current Scope work, I'm leaning towards a single Bindings class that subclasses all of my current bindings, since I have found very few cases where an entire class of nodes only needs one of the bindings, and if we have to pass them all around together, they might as well be a single class. The current names only appear in my new code, so sed & friends will make it easy to tweak things later. -- This email may be signed or encrypted with GPG (http://www.gnupg.org). The GPG signature (if present) will be attached as 'signature.asc'. For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy My public key is at http://www.physics.drexel.edu/~wking/pubkey.txt
On Thu, Feb 24, 2011 at 5:42 PM, W. Trevor King <wking@drexel.edu> wrote:
W. Trevor King, 22.02.2011 18:55:
I've been working on a more explicit parser that removes the ambiguity behind the various visibilities. This will help me ensure proper impolementation of my cdef-ed enums/structs/..., and make it easier to update visibility syntax in the future. Take a look and let me know if you think this is a useful direction to take:
The refactoring continues as I'm moving my new binding classes into Symtab. This is, of course, leading to lots of changes, but I've been running the test suite before each commit to make sure I don't go to far astray. Anything I miss will eventually lead to a better test suite ;).
I'm currently writing up new versions of most Scope methods that use my classes, and I'll replace the original methods once I've updated all the code that calls them. Next on the list will be the Nodes themselves, at which point I think I'll be positioned to put in the `cdef struct` and whatnot that got this whole thing started.
Since there has been a fair amount of churn, I though I'd ask for some more feedback on the general direction I'm headed:
http://www.physics.drexel.edu/~wking/code/git/gitweb.cgi?p=cython.git;a=log;...
On of my primary motivations to moving to github (or similar) was nicely annotated diffs between branches and the ability to do line-by-line comments. If you could also push to your fork there, that'd be great (otherwise I will ;). - Robert
On Thu, Feb 24, 2011 at 05:57:21PM -0800, Robert Bradshaw wrote:
On of my primary motivations to moving to github (or similar) was nicely annotated diffs between branches and the ability to do line-by-line comments. If you could also push to your fork there, that'd be great (otherwise I will ;).
Ah, Stephan's comment about needing to comment via the mailing list makes more sense now ;). Pushed: https://github.com/wking/cython/tree/cdef-enums-stucts-and-unions -- This email may be signed or encrypted with GPG (http://www.gnupg.org). The GPG signature (if present) will be attached as 'signature.asc'. For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy My public key is at http://www.physics.drexel.edu/~wking/pubkey.txt
Done: * Cleaned all overloaded visibilities out of Parsing and Symtab. * Merged current trunk. Todo: * Remove older Symtab interface so I can remove the WTK_* methods. * Activate `shadow` functionality (via AnalyseDeclarationsTransform.visit_CStructOrUnionDefNode()). Feel free to comment via github: https://github.com/wking/cython/commits/cdef-enums-stucts-and-unions I'm not quite sure how this shadow thing is supposed to work, but I'll take a stab at getting it going if a more seasoned Cythonista is not available for comment ;). -- This email may be signed or encrypted with GPG (http://www.gnupg.org). The GPG signature (if present) will be attached as 'signature.asc'. For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy My public key is at http://www.physics.drexel.edu/~wking/pubkey.txt
On Tue, Feb 22, 2011 at 12:02 PM, W. Trevor King <wking@drexel.edu> wrote:
On Tue, Feb 22, 2011 at 08:18:21PM +0100, Stefan Behnel wrote:
W. Trevor King, 22.02.2011 18:55:
I've been working on a more explicit parser that removes the ambiguity behind the various visibilities. This will help me ensure proper impolementation of my cdef-ed enums/structs/..., and make it easier to update visibility syntax in the future. Take a look and let me know if you think this is a useful direction to take:
First thing that caught my eyes: I hope you do not intend to leave the logging usage in the parser. This is seriously performance critical code that Cython compiles down to pretty fast C code. Note that you can use a debugger and Python's profiling/tracing interface to find out what's happening, without code impact.
I can strip them out afterwards, but it helps me figure out what I've broken if I shift too much around at the same time.
I don't know enough about Python's trace module to know if I can turn on tracing only for functions defined in a single module or not, since otherwise its hard for me to separate signal from noise.
I think you can filter things after the fact. It would also be pretty easy to write a utility that (conditionally) decorates all methods if a flag is set, which we could leave in. (Wouldn't normally be such a big deal, but this is one of the bottlenecks of compilation.)
Some of the log statements span more than one line, which makes it trickier to strip them out with sed&friends (but backing out the initial changeset would likely make it simple enough to remove the rest manually).
Hmm, perhaps I'll condense the logging statements down onto one (long) line a piece, that will make it easy to comment/uncomment them with sed/emacs/etc. I suppose once Cython can compile the logging module we could leave them in with reduced overhead ;).
Also note that it's best to write runnable tests ("tests/run/"). The tests in "tests/compile/" are only compiled and imported. See the hacking guide in the wiki. I know you're not there yet with your implementation, I'm just mentioning it.
Thanks for the tip.
CtxAttribute is the wrong name, though. And the class copy implementation gets even more generic than it already was in the Ctx. I'm not a big fan of that, especially not in the parser. For one, it prevents changing the classes into cdef classes, which had long been on my list for Ctx.
An easy, if uglier, workaround would be to prepend attributes with the class name, e.g. CBinding.visibility -> CBinding.c_binding_visiblity. Then the Ctx class could subclass the current CtxAttribute classes instead of binding instances of each of them. That way Ctx would keep its traditional flat attribute namespace and easy deepcopy, but eveyone in Nodes, etc. that will use the attributes would become class-name dependent.
I'd be up for flattening this. In particular, changing every "entry.name" to "entry.python_binding.name" seems to be a lot of churn and extra verbiage for not much benefit. The only overlap I see is name and visibility, and keeping name/cname and adding cvisibility would be preferable to me.
The CtxAttribute class is, as its docstring says, just a hook for its deepcopy method. With an alternative deepcopy implementation, CtxAttribute could be replaced with the standard `object`, so don't worry too much about its name at this point ;).
You mean shallow copy?
CSource: doesn't sound like quite the right name - it does not describe a C source file but information that Cython has about non-Cython things.
It's a container for attributes that describe the presence and location of backing C definitions.
* cdef: "Will there be a backing C defintion? * extern: "Has someone else already written it?" * name/namespace: "What did they call it?"
If you'd rather I called the class something else, I'm certainly willing to change it.
It seems a bit odd to me, but if need be we can rename it later. However, csource and c_binding seem rather redundant to me, but as mentioned above I think it's better just to flatten it all. The changes to parsing look decent to me, but admittedly there's a lot of renaming churn, so I could have missed something. - Robert
On Fri, Feb 25, 2011 at 11:11:03PM -0800, Robert Bradshaw wrote:
On Tue, Feb 22, 2011 at 12:02 PM, W. Trevor King <wking@drexel.edu> wrote:
An easy, if uglier, workaround would be to prepend attributes with the class name, e.g. CBinding.visibility -> CBinding.c_binding_visiblity. Then the Ctx class could subclass the current CtxAttribute classes instead of binding instances of each of them. That way Ctx would keep its traditional flat attribute namespace and easy deepcopy, but eveyone in Nodes, etc. that will use the attributes would become class-name dependent.
I'd be up for flattening this. In particular, changing every "entry.name" to "entry.python_binding.name" seems to be a lot of churn and extra verbiage for not much benefit. The only overlap I see is name and visibility, and keeping name/cname and adding cvisibility would be preferable to me.
That works for me, but I see possible ambiguity in cname. From the "external C code" docs: The other way is to use a C name specification to give different Cython and C names to the C function. Suppose, for example, that you want to wrap an external function called eject_tomato(). If you declare it as: cdef extern void c_eject_tomato "eject_tomato" (float speed) then its name inside the Cython module will be c_eject_tomato, whereas its name in C will be eject_tomato. In this case I was eventually going to use c_source.name = eject_tomato c_source.namespace = ... c_binding.name = c_eject_tomato c_binding.namespace = ... python_binding.name = eject_tomato I'm not sure how Cython handles name/cname with this case at the moment, but it seems like there are two cnames to keep track of.
The CtxAttribute class is, as its docstring says, just a hook for its deepcopy method. With an alternative deepcopy implementation, CtxAttribute could be replaced with the standard `object`, so don't worry too much about its name at this point ;).
You mean shallow copy?
I meant deepcopy, since that seems to be the point of Ctx cloning. At the moment, however, Ctx deep copies only require Binding shallow copies. If someone crazy wanted to add mutable attrubites to binding classes, the Binding deepcopy code would have had to be adjusted. None of this matters though, if we move back to a flat Ctx attribute space.
CSource: doesn't sound like quite the right name - it does not describe a C source file but information that Cython has about non-Cython things.
It's a container for attributes that describe the presence and location of backing C definitions.
* cdef: "Will there be a backing C defintion? * extern: "Has someone else already written it?" * name/namespace: "What did they call it?"
If you'd rather I called the class something else, I'm certainly willing to change it.
It seems a bit odd to me, but if need be we can rename it later. However, csource and c_binding seem rather redundant to me, but as mentioned above I think it's better just to flatten it all.
The changes to parsing look decent to me, but admittedly there's a lot of renaming churn, so I could have missed something.
I'll go back and revert out the name changes if you'll confirm that there is only one meaning for cname. -- This email may be signed or encrypted with GPG (http://www.gnupg.org). The GPG signature (if present) will be attached as 'signature.asc'. For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy My public key is at http://www.physics.drexel.edu/~wking/pubkey.txt
On Sat, Feb 26, 2011 at 3:48 AM, W. Trevor King <wking@drexel.edu> wrote:
On Fri, Feb 25, 2011 at 11:11:03PM -0800, Robert Bradshaw wrote:
On Tue, Feb 22, 2011 at 12:02 PM, W. Trevor King <wking@drexel.edu> wrote:
An easy, if uglier, workaround would be to prepend attributes with the class name, e.g. CBinding.visibility -> CBinding.c_binding_visiblity. Then the Ctx class could subclass the current CtxAttribute classes instead of binding instances of each of them. That way Ctx would keep its traditional flat attribute namespace and easy deepcopy, but eveyone in Nodes, etc. that will use the attributes would become class-name dependent.
I'd be up for flattening this. In particular, changing every "entry.name" to "entry.python_binding.name" seems to be a lot of churn and extra verbiage for not much benefit. The only overlap I see is name and visibility, and keeping name/cname and adding cvisibility would be preferable to me.
That works for me, but I see possible ambiguity in cname. From the "external C code" docs:
The other way is to use a C name specification to give different Cython and C names to the C function. Suppose, for example, that you want to wrap an external function called eject_tomato(). If you declare it as:
cdef extern void c_eject_tomato "eject_tomato" (float speed)
then its name inside the Cython module will be c_eject_tomato, whereas its name in C will be eject_tomato.
In this case I was eventually going to use
c_source.name = eject_tomato c_source.namespace = ... c_binding.name = c_eject_tomato c_binding.namespace = ... python_binding.name = eject_tomato
I'm not sure how Cython handles name/cname with this case at the moment, but it seems like there are two cnames to keep track of.
In this case, cname is "eject_tomato" (which actually gets emitted in the C source file to use it) and name (in both Python and Cython namespace) is "c_eject_tomato." You can think of it as name being what the user sees, and cname being what the C compiler sees.
The CtxAttribute class is, as its docstring says, just a hook for its deepcopy method. With an alternative deepcopy implementation, CtxAttribute could be replaced with the standard `object`, so don't worry too much about its name at this point ;).
You mean shallow copy?
I meant deepcopy, since that seems to be the point of Ctx cloning. At the moment, however, Ctx deep copies only require Binding shallow copies. If someone crazy wanted to add mutable attrubites to binding classes, the Binding deepcopy code would have had to be adjusted. None of this matters though, if we move back to a flat Ctx attribute space.
OK. As an aside, do you know about the copy.deepcopy() method?
CSource: doesn't sound like quite the right name - it does not describe a C source file but information that Cython has about non-Cython things.
It's a container for attributes that describe the presence and location of backing C definitions.
* cdef: "Will there be a backing C defintion? * extern: "Has someone else already written it?" * name/namespace: "What did they call it?"
If you'd rather I called the class something else, I'm certainly willing to change it.
It seems a bit odd to me, but if need be we can rename it later. However, csource and c_binding seem rather redundant to me, but as mentioned above I think it's better just to flatten it all.
The changes to parsing look decent to me, but admittedly there's a lot of renaming churn, so I could have missed something.
I'll go back and revert out the name changes if you'll confirm that there is only one meaning for cname.
Thanks. BTW, I pushed https://github.com/cython/cython/commit/3552114e1b2a21bf2f81f451d50cd48934ea... which starts to do some of the back-end implementation. - Robert
On Sat, Feb 26, 2011 at 10:01:43AM -0800, Robert Bradshaw wrote:
On Sat, Feb 26, 2011 at 3:48 AM, W. Trevor King <wking@drexel.edu> wrote:
On Fri, Feb 25, 2011 at 11:11:03PM -0800, Robert Bradshaw wrote:
On Tue, Feb 22, 2011 at 12:02 PM, W. Trevor King <wking@drexel.edu> wrote:
An easy, if uglier, workaround would be to prepend attributes with the class name, e.g. CBinding.visibility -> CBinding.c_binding_visiblity. Then the Ctx class could subclass the current CtxAttribute classes instead of binding instances of each of them. That way Ctx would keep its traditional flat attribute namespace and easy deepcopy, but eveyone in Nodes, etc. that will use the attributes would become class-name dependent.
I'd be up for flattening this. In particular, changing every "entry.name" to "entry.python_binding.name" seems to be a lot of churn and extra verbiage for not much benefit. The only overlap I see is name and visibility, and keeping name/cname and adding cvisibility would be preferable to me.
That works for me, but I see possible ambiguity in cname. From the "external C code" docs:
The other way is to use a C name specification to give different Cython and C names to the C function. Suppose, for example, that you want to wrap an external function called eject_tomato(). If you declare it as:
cdef extern void c_eject_tomato "eject_tomato" (float speed)
then its name inside the Cython module will be c_eject_tomato, whereas its name in C will be eject_tomato.
In this case I was eventually going to use
c_source.name = eject_tomato c_source.namespace = ... c_binding.name = c_eject_tomato c_binding.namespace = ... python_binding.name = eject_tomato
I'm not sure how Cython handles name/cname with this case at the moment, but it seems like there are two cnames to keep track of.
In this case, cname is "eject_tomato" (which actually gets emitted in the C source file to use it) and name (in both Python and Cython namespace) is "c_eject_tomato."
What is the "Cython namespace"? Is that what you get when you cimport something (vs. the Python namespace being what you get when you import something). If so, then that sounds like just two names, so name/cname it is.
The CtxAttribute class is, as its docstring says, just a hook for its deepcopy method. With an alternative deepcopy implementation, CtxAttribute could be replaced with the standard `object`, so don't worry too much about its name at this point ;).
You mean shallow copy?
I meant deepcopy, since that seems to be the point of Ctx cloning. At the moment, however, Ctx deep copies only require Binding shallow copies. If someone crazy wanted to add mutable attrubites to binding classes, the Binding deepcopy code would have had to be adjusted. None of this matters though, if we move back to a flat Ctx attribute space.
OK. As an aside, do you know about the copy.deepcopy() method?
Yup, but I was trying to avoid pulling in non-Cython dependencies since you guys seem to have gone through a lot of trouble to make the Parsing module fast in C.
CSource: doesn't sound like quite the right name - it does not describe a C source file but information that Cython has about non-Cython things.
It's a container for attributes that describe the presence and location of backing C definitions.
* cdef: "Will there be a backing C defintion? * extern: "Has someone else already written it?" * name/namespace: "What did they call it?"
If you'd rather I called the class something else, I'm certainly willing to change it.
It seems a bit odd to me, but if need be we can rename it later. However, csource and c_binding seem rather redundant to me, but as mentioned above I think it's better just to flatten it all.
The changes to parsing look decent to me, but admittedly there's a lot of renaming churn, so I could have missed something.
I'll go back and revert out the name changes if you'll confirm that there is only one meaning for cname.
Thanks.
BTW, I pushed https://github.com/cython/cython/commit/3552114e1b2a21bf2f81f451d50cd48934ea... which starts to do some of the back-end implementation.
Ok, I'll merge those once I get the visibility kinks worked out of Symtab. -- This email may be signed or encrypted with GPG (http://www.gnupg.org). The GPG signature (if present) will be attached as 'signature.asc'. For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy My public key is at http://www.physics.drexel.edu/~wking/pubkey.txt
On Sat, Feb 26, 2011 at 6:14 PM, W. Trevor King <wking@drexel.edu> wrote:
On Sat, Feb 26, 2011 at 10:01:43AM -0800, Robert Bradshaw wrote:
On Sat, Feb 26, 2011 at 3:48 AM, W. Trevor King <wking@drexel.edu> wrote:
On Fri, Feb 25, 2011 at 11:11:03PM -0800, Robert Bradshaw wrote:
On Tue, Feb 22, 2011 at 12:02 PM, W. Trevor King <wking@drexel.edu> wrote:
An easy, if uglier, workaround would be to prepend attributes with the class name, e.g. CBinding.visibility -> CBinding.c_binding_visiblity. Then the Ctx class could subclass the current CtxAttribute classes instead of binding instances of each of them. That way Ctx would keep its traditional flat attribute namespace and easy deepcopy, but eveyone in Nodes, etc. that will use the attributes would become class-name dependent.
I'd be up for flattening this. In particular, changing every "entry.name" to "entry.python_binding.name" seems to be a lot of churn and extra verbiage for not much benefit. The only overlap I see is name and visibility, and keeping name/cname and adding cvisibility would be preferable to me.
That works for me, but I see possible ambiguity in cname. From the "external C code" docs:
The other way is to use a C name specification to give different Cython and C names to the C function. Suppose, for example, that you want to wrap an external function called eject_tomato(). If you declare it as:
cdef extern void c_eject_tomato "eject_tomato" (float speed)
then its name inside the Cython module will be c_eject_tomato, whereas its name in C will be eject_tomato.
In this case I was eventually going to use
c_source.name = eject_tomato c_source.namespace = ... c_binding.name = c_eject_tomato c_binding.namespace = ... python_binding.name = eject_tomato
I'm not sure how Cython handles name/cname with this case at the moment, but it seems like there are two cnames to keep track of.
In this case, cname is "eject_tomato" (which actually gets emitted in the C source file to use it) and name (in both Python and Cython namespace) is "c_eject_tomato."
What is the "Cython namespace"? Is that what you get when you cimport something (vs. the Python namespace being what you get when you import something).
Yes, that would be a good way to understand it.
If so, then that sounds like just two names, so name/cname it is.
Anything that exists in the intersection of the Cython and Python namespace has the same name in both places--the user-visible one, thus name suffices for both. (Also, the Cython namespace is a superset of the Python namespace, possibly with a semantically equivalent but more optimized backing for some objects, e.g. dispatching to C methods directly for builtins.)
The CtxAttribute class is, as its docstring says, just a hook for its deepcopy method. With an alternative deepcopy implementation, CtxAttribute could be replaced with the standard `object`, so don't worry too much about its name at this point ;).
You mean shallow copy?
I meant deepcopy, since that seems to be the point of Ctx cloning. At the moment, however, Ctx deep copies only require Binding shallow copies. If someone crazy wanted to add mutable attrubites to binding classes, the Binding deepcopy code would have had to be adjusted. None of this matters though, if we move back to a flat Ctx attribute space.
OK. As an aside, do you know about the copy.deepcopy() method?
Yup, but I was trying to avoid pulling in non-Cython dependencies since you guys seem to have gone through a lot of trouble to make the Parsing module fast in C.
Ah, OK. It's not always obvious without looking what from the standard library is fast and what is not, but this doesn't look particularly optimized. - Robert
On Sat, Feb 26, 2011 at 07:11:48PM -0800, Robert Bradshaw wrote:
On Sat, Feb 26, 2011 at 6:14 PM, W. Trevor King <wking@drexel.edu> wrote:
On Sat, Feb 26, 2011 at 10:01:43AM -0800, Robert Bradshaw wrote:
On Sat, Feb 26, 2011 at 3:48 AM, W. Trevor King <wking@drexel.edu> wrote:
On Fri, Feb 25, 2011 at 11:11:03PM -0800, Robert Bradshaw wrote:
On Tue, Feb 22, 2011 at 12:02 PM, W. Trevor King <wking@drexel.edu> wrote:
An easy, if uglier, workaround would be to prepend attributes with the class name, e.g. CBinding.visibility -> CBinding.c_binding_visiblity. Then the Ctx class could subclass the current CtxAttribute classes instead of binding instances of each of them. That way Ctx would keep its traditional flat attribute namespace and easy deepcopy, but eveyone in Nodes, etc. that will use the attributes would become class-name dependent.
I'd be up for flattening this. In particular, changing every "entry.name" to "entry.python_binding.name" seems to be a lot of churn and extra verbiage for not much benefit. The only overlap I see is name and visibility, and keeping name/cname and adding cvisibility would be preferable to me.
That works for me, but I see possible ambiguity in cname. From the "external C code" docs:
The other way is to use a C name specification to give different Cython and C names to the C function. Suppose, for example, that you want to wrap an external function called eject_tomato(). If you declare it as:
cdef extern void c_eject_tomato "eject_tomato" (float speed)
then its name inside the Cython module will be c_eject_tomato, whereas its name in C will be eject_tomato.
In this case I was eventually going to use
c_source.name = eject_tomato c_source.namespace = ... c_binding.name = c_eject_tomato c_binding.namespace = ... python_binding.name = eject_tomato
I'm not sure how Cython handles name/cname with this case at the moment, but it seems like there are two cnames to keep track of.
In this case, cname is "eject_tomato" (which actually gets emitted in the C source file to use it) and name (in both Python and Cython namespace) is "c_eject_tomato."
What is the "Cython namespace"? Is that what you get when you cimport something (vs. the Python namespace being what you get when you import something).
Yes, that would be a good way to understand it.
Okay, I'm back to a flat attribute namespace in my branch, and I'm moving through Nodes removing the places where I currently collapse back to an overloaded visibility. -- This email may be signed or encrypted with GPG (http://www.gnupg.org). The GPG signature (if present) will be attached as 'signature.asc'. For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy My public key is at http://www.physics.drexel.edu/~wking/pubkey.txt
W. Trevor King, 22.02.2011 21:02:
On Tue, Feb 22, 2011 at 08:18:21PM +0100, Stefan Behnel wrote:
I also doubt that Cython allows you to call an attribute "cdef", you'll need to change that.
It seems to work for me:
import Cython.Compiler.Parsing as P P.__file__ 'Cython/Compiler/Parsing.so' c = P.CSource() dir(c) [..., 'cdef', 'deepcopy', 'extern', 'name', 'namespace'] c.cdef 0 c.cdef = 1 c.cdef 1
Ah, right. It's actually compiled as .py file, so Cython's syntax doesn't apply here. However, it may apply to other parts of the compiler at some point, so it's better not to use keywords in parts of the source that become globally visible. Stefan
Robert Bradshaw, 18.02.2011 00:54:
Forgot reply-all... didin't we have this discussion before about making that the default for this list as it is by-far the most common desired behavior?
Yes we did. And I guess it would be "the default" for mailing lists if it was just that: a default, not something that breaks replying. However, given that this has been discussed and decided, I'll just go and break replying for this list again. Stefan
participants (5)
-
Greg Ewing -
Lisandro Dalcin -
Robert Bradshaw -
Stefan Behnel -
W. Trevor King