Python's C interface for types
I have a fair amount of my binary floating-point model written, though even of what I have done only some is debugged (and none has been rigorously tested). But I have hit some things that I can't work out, and one query reduced comp.lang.python to a stunned silence :-) Note that I am not intending to do all the following, at least for now, but I have had to restructure half a dozen times to match my implementation requirements to the C interface (as I have learnt more about Python!) and designing to avoid that is always good. Any pointers appreciated. I can't find any detailed description of the methods that I need to provide. Specifically: Does Python use classic division (nb_divide) and inversion (nb_invert) or are they entirely historical? Note that I can very easily provide the latter. Is there any documentation on the coercion function (nb_coerce)? It seems to have unusual properties. How critical is the 'numeric' property of the nb_hash function? I can certainly honour it, but is it worth it? I assume that Python will call nb_richcompare if defined and nb_compare if not. Is that right? Are the inplace methods used and, if so, what is their specification? I assume that I can ignore all of the allocation, deallocation and attribute handling functions, as the default for a VAR object is fine. That seems to work. Except for one thing! My base type is static, but I create some space for every derivation (and it can ONLY be used in derived form). The space creation is donein C but the derivation in Python. I assume that I need a class (not instance) destructor, but what should it do to free the space? Call C to Py_DECREF it? I assume that a class structure will never go away until after all instances have gone away (unless I use Py_DECREF), so a C pointer from an instance to something owned by the class is OK. Is there any documentation on how to support marshalling/pickling and the converse from C types? I would quite like to provide some attributes. They are 'simple' but need code executing to return them. I assume that means that they aren't simple enough, and have to be provided as methods (like conjugate). That's what I have done, anyway. Is there any obvious place for a reduction method to be hooked in? That is a method that takes a sequence, all members of which must be convertible to a single class, and returns a member of that class. Note that it specifically does NOT make sense on a single value of that class. Sorry about the length of this! Regards, Nick Maclaren, University of Cambridge Computing Service, New Museums Site, Pembroke Street, Cambridge CB2 3QH, England. Email: nmm1@cam.ac.uk Tel.: +44 1223 334761 Fax: +44 1223 334679
On 1/26/07, Nick Maclaren
Does Python use classic division (nb_divide) and inversion (nb_invert) or are they entirely historical? Note that I can very easily provide the latter.
nb_divide is used for the division operation (/) when not using 'from __future__ import division', and PyNumber_Divide(). It doesn't fall back to foor_divide or true_divide. nb_invert is used for bitwise inversion (~) and PyNumber_Invert(). It's not historical, it's actual. Is there any documentation on the coercion function (nb_coerce)? It
seems to have unusual properties.
I don't recall ever seeing useful documentation on coerce() and nb_coerce. I suggest not to use it; it's gone in Python 3.0 anyway. How critical is the 'numeric' property of the nb_hash function? I
can certainly honour it, but is it worth it?
Which numeric property? the fact that it returns a C long? Or that, for natural numbers, it *seems* to return self? The former is quite important, the latter not very. The important bit is that an object's hash() be equal to the hash() of objects that it is considered equal to. That is to say, if you have 'd = {5: "five"}' and you want 'f = yourtype(5); d[f]' to result in "five", hash(f) must be equal to hash(5). The builtin floats do that. There's no strict requirement that equal objects must have equal hashes, but people using sets and dicts really appreciate it. ('f == 5 and f not in set([5])' to be True would be confusing.) I assume that Python will call nb_richcompare if defined and
nb_compare if not. Is that right?
Yes. Are the inplace methods used and, if so, what is their specification? (Hah, my specialty.) Inplace methods are used for the augmented-assignment statements: '+=' and the like. They are free to modify 'self' or return a new object. Python falls back to the normal, non-inplace operations if the inplace methods are not defined. I assume your floating-point type is immutable, so you won't have to implement them. (If the type is to be mutable, don't forget to make them unhashable, by not defining nb_hash.) I assume that I can ignore all of the allocation, deallocation and
attribute handling functions, as the default for a VAR object is fine. That seems to work.
Except for one thing! My base type is static, but I create some space for every derivation (and it can ONLY be used in derived form). The space creation is donein C but the derivation in Python. I assume that I need a class (not instance) destructor, but what should it do to free the space? Call C to Py_DECREF it?
Where do you allocate this space, and how do you allocate it? If it's space you malloc() and store somewhere in the type struct, yecchh. You should not just allocate stuff at the end of the type struct, as the type struct's layout is not under your control (we actually extend the type struct as needed, which is why newer features end up in less logical places at the end of the struct ;) I would suggest using attributes of the type instead, with the normal Python refcounting. That means the 'extra space' has to be an actual Python object, though. I assume that a class structure will never go away until after all
instances have gone away (unless I use Py_DECREF), so a C pointer from an instance to something owned by the class is OK.
Correct. Is there any documentation on how to support marshalling/pickling
and the converse from C types?
I don't you can make your own type marshallable. For pickle it's more or less the same as for Python types. The pickle docs (and maybe http://www.python.org/dev/peps/pep-0307/) probably cover what you want to know. You can also look at one of the complexer builtin types that support pickling, like the datetime types. I would quite like to provide some attributes. They are 'simple'
but need code executing to return them. I assume that means that they aren't simple enough, and have to be provided as methods (like conjugate). That's what I have done, anyway.
You can use PyGetSetDef to get 'easy' attributes with getters and setters. http://docs.python.org/api/type-structs.html#l2h-1020 Is there any obvious place for a reduction method to be hooked in?
That is a method that takes a sequence, all members of which must be convertible to a single class, and returns a member of that class. Note that it specifically does NOT make sense on a single value of that class.
There's nothing I can think of that is a natural match for that in standard
Python methods. I would suggest just making it a classmethod.
(dict.fromkeysis a good example of a classmethod in C.)
As a final note: Python's source itself is a good source of answers and
examples. Almost all of the features of the Python API are used in a builtin
type or stdlib module, and for C source, Python's source is remarkably
readable ;-)
--
Thomas Wouters
Thanks very much! That answers most things. Yes, I had got many of my answers from searching the source, but there is clearly some history there, and it isn't always clear what is current. Here are a few responses to the areas of confusion:
nb_invert is used for bitwise inversion (~) and PyNumber_Invert(). It's not historical, it's actual.
Ah! So it's NOT 1/x! No relevant to floating-point, then.
I don't recall ever seeing useful documentation on coerce() and nb_coerce. I suggest not to use it; it's gone in Python 3.0 anyway.
Excellent! Task completed :-)
Which numeric property? the fact that it returns a C long? Or that, for natural numbers, it *seems* to return self?
The latter. hash(123) == hash(123.0) for example. It is a real pain for advanced formats. Making it the same for things that compare equal isn't a problem.
[inplace ] I assume your floating-point type is immutable, so you won't have to implement them.
I haven't done anything special to flag it as such, but it is.
Where do you allocate this space, and how do you allocate it? If it's space you malloc() and store somewhere in the type struct, yecchh. You should not just allocate stuff at the end of the type struct, as the type struct's layout is not under your control (we actually extend the type struct as needed, which is why newer features end up in less logical places at the end of the struct ;) I would suggest using attributes of the type instead, with the normal Python refcounting. That means the 'extra space' has to be an actual Python object, though.
PyMem_Malloc. I can certainly make it an attribute, as the overhead isn't large for a per-class object. It is just a block of mutable memory, opaque to the Python layer, and NOT containing any pointers!
I don't you can make your own type marshallable. For pickle it's more or less the same as for Python types. The pickle docs (and maybe http://www.python.org/dev/peps/pep-0307/) probably cover what you want to know. You can also look at one of the complexer builtin types that support pickling, like the datetime types.
The only documentation I have found is how to do it in Python. Is that what you mean? I will look at the datetime types.
You can use PyGetSetDef to get 'easy' attributes with getters and setters. http://docs.python.org/api/type-structs.html#l2h-1020
I was put off by some of the warnings. I will revisit it.
There's nothing I can think of that is a natural match for that in standard Python methods. I would suggest just making it a classmethod. (dict.fromkeysis a good example of a classmethod in C.)
Thanks. That is a useful reference. Reductions are a problem in many languages. Regards, Nick Maclaren, University of Cambridge Computing Service, New Museums Site, Pembroke Street, Cambridge CB2 3QH, England. Email: nmm1@cam.ac.uk Tel.: +44 1223 334761 Fax: +44 1223 334679
On 26/01/2007 17.03, Thomas Wouters wrote:
How critical is the 'numeric' property of the nb_hash function? I can certainly honour it, but is it worth it?
[...] There's no strict requirement that equal objects must have equal hashes,
Uh? I thought that was the *only* strict requirement of hash. In fact the docs agree: ==================================================== __hash__( self) Called for the key object for dictionary operations, and by the built-in function hash(). Should return a 32-bit integer usable as a hash value for dictionary operations. The only required property is that objects which compare equal have the same hash value; [...] ==================================================== I personally consider *very* important that hash(5.0) == hash(5) (and that 5.0 == 5, of course). -- Giovanni Bajo
On Fri, Jan 26, 2007, Giovanni Bajo wrote:
On 26/01/2007 17.03, Thomas Wouters wrote:
There's no strict requirement that equal objects must have equal hashes,
Uh? I thought that was the *only* strict requirement of hash. In fact the docs agree:
==================================================== __hash__( self)
Called for the key object for dictionary operations, and by the built-in function hash(). Should return a 32-bit integer usable as a hash value for dictionary operations. The only required property is that objects which compare equal have the same hash value; [...] ====================================================
Possibly the docs should be updated, but this is only intended to apply to objects of the same type.
I personally consider *very* important that hash(5.0) == hash(5) (and that 5.0 == 5, of course).
Well, sure, but That's Different. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ "I disrespectfully agree." --SJM
participants (4)
-
Aahz
-
Giovanni Bajo
-
Nick Maclaren
-
Thomas Wouters