[Python-bugs-list] [ python-Bugs-540965 ] PyType_GenericNew broken

noreply@sourceforge.net noreply@sourceforge.net
Mon, 15 Apr 2002 07:32:35 -0700


Bugs item #540965, was opened at 2002-04-08 12:24
You can respond by visiting: 
http://sourceforge.net/tracker/?func=detail&atid=105470&aid=540965&group_id=5470

Category: Type/class unification
Group: Python 2.3
Status: Closed
Resolution: Rejected
Priority: 5
Submitted By: Ralf Juengling (rjuengling)
Assigned to: Guido van Rossum (gvanrossum)
Summary: PyType_GenericNew broken

Initial Comment:
A new Python type/class _A is implemented in C.
It needs a special allocator function, thus
tp_alloc=_A_alloc for this type. There is no need
for a special new-method, however, thus
tp_new=PyType_GenericNew for this type.

The class A inherits from _A and is implemented
in Python:

class A(_A):
	pass

When an instance of _A is created, its allocator
function _A_alloc gets invoked. When an instance 
of A is created, the _A_alloc does not get invoked.


PyType_GenericNew should also invoke all allocator 
functions of the argument type's superclasses (in 
reverse mro, I think).



----------------------------------------------------------------------

>Comment By: Ralf Juengling (rjuengling)
Date: 2002-04-15 14:32

Message:
Logged In: YES 
user_id=495820

> ...The right approach is to have a fixed-size object
> that contains a pointer to separately allocated memory.

This is how it works. I admit, I expressed myself 
poorly here.

Do you affirm at the end, that types with dynamic memory
requirements cannot be subclassed at the Python level,
unless one also implements a special function for the
tp_new slot -- to make sure that the base class 
allocator gets invoked...


----------------------------------------------------------------------

Comment By: Guido van Rossum (gvanrossum)
Date: 2002-04-15 13:59

Message:
Logged In: YES 
user_id=6380

I'm closing this now, because I don't see the bug in Python.
I do see the bug in your approach. You can't dynamically
grow the memory allocated for an object! That would move the
object in memory and would invalidate all pointers to the
object. The right approach is to have a fixed-size object
that contains a pointer to separately allocated memory.
(Like the list and dict object do.) BTW the tp_alloc slot is
needed because some types have a faster allocator, not to
improve upon the initial allocation size calculation.

----------------------------------------------------------------------

Comment By: Ralf Juengling (rjuengling)
Date: 2002-04-15 13:46

Message:
Logged In: YES 
user_id=495820

The data structures used internally by class _A cannot
by cast into the 'basicsize\itemsize' scheme (i.e. a 
sparse matrix type). So tp_itemsize actually is set to 
zero for type _A and its allocator invokes malloc to
request memory for the initial data structure (which
is going to grow dynamically when a sparse matrix
is build up in a Python program).

Why would we need the tp_alloc slot at all, if the 
'memory demands' of any class (its instances 
respectively) would fit into the 'basicsize\itemsize'-
scheme?

I just looked into the source for the 'int' object.
The tp_alloc slot is set to zero, allocation is
done in 'int_new'. (so it didn't enlighten me...)




----------------------------------------------------------------------

Comment By: Guido van Rossum (gvanrossum)
Date: 2002-04-15 13:01

Message:
Logged In: YES 
user_id=6380

Your understanding of instance allocation doesn't match
reality, *or* you're expressing yourself poorly. Allocation
does not include initialization; allocation simply requests
the memory for the object from malloc (or some other memory
allocator) and fills it with zeros, except for the reference
count, which is set to 1, and the ob_type field, which is
set to the type argument to the allocator. The amount of
memory allocated for the object is calculated by taking the
tp_basicsize of the type object, and adding nitems times the
tp_itemsize from the type object.

If you want to initialize your object, that should be done
in the tp_new slot, not in tp_alloc.

Have you tried to read the source code (or go through an
allocation in a C debugger) to find out how it really works?

----------------------------------------------------------------------

Comment By: Ralf Juengling (rjuengling)
Date: 2002-04-15 08:06

Message:
Logged In: YES 
user_id=495820

The class A does not know how to allocate the 'base part' of instances of itself
(i.e. what makes an instance of class _A). Thus sometime during instantiation 
of class A objects, the allocater of the base class must be called!
This will not happen, however, if the tp_new-slot function merely calls the 
allocator of the argument type.

You pointed out, that e.g. subclasses of 'int' won't inherit int's special allocator.
Does this mean, int's allocator will not be invoked on instantiation of the subclass?
Then, I don't understand why this works; I can only imagine, that PyType_GenericAlloc
is designed to deal with Python built-in types (and probably compostitions of these).




----------------------------------------------------------------------

Comment By: Guido van Rossum (gvanrossum)
Date: 2002-04-15 00:09

Message:
Logged In: YES 
user_id=6380

> PyType_GenericNew should also invoke all allocator 
> functions of the argument type's superclasses (in 
> reverse mro, I think).

No, this doesn't make sense; you can't invoke
more than one allocator. (Each allocator would
create a whole new object.)

The cause of your problem is in type_new(): it
always overrides tp_alloc with PyType_GenericAlloc.
You'd have to write a new metaclass (inheriting
from type) in C to change the tp_alloc (and tp_free)
settings.

But why do you want this? Why do you want
A instances to use the same allocator as _A?
This is not how subclassing built-in types works
elsewhere; e.g. int uses a special allocator,
but subclasses of int don't inherit that.

Before I do anything about this I'd like to
understand what you are trying to accomplish.

----------------------------------------------------------------------

You can respond by visiting: 
http://sourceforge.net/tracker/?func=detail&atid=105470&aid=540965&group_id=5470