[Python-checkins] r63520 - in python/branches/okkoto-sizeof: Doc/c-api/object.rst Doc/c-api/typeobj.rst Doc/includes/typestruct.h Doc/library/functions.rst Include/abstract.h Include/object.h Lib/test/test_builtin.py Objects/abstract.c Objects/di
Neal Norwitz
nnorwitz at gmail.com
Fri May 23 06:01:09 CEST 2008
Robert,
I noticed a couple of things in this change. See my comments below.
On Wed, May 21, 2008 at 8:01 AM, robert.schuppenies
<python-checkins at python.org> wrote:
> Author: robert.schuppenies
> Date: Wed May 21 17:01:20 2008
> New Revision: 63520
>
> Log:
> first patch proposed
>
>
> Modified:
> python/branches/okkoto-sizeof/Doc/c-api/object.rst
> python/branches/okkoto-sizeof/Doc/c-api/typeobj.rst
> python/branches/okkoto-sizeof/Doc/includes/typestruct.h
> python/branches/okkoto-sizeof/Doc/library/functions.rst
> python/branches/okkoto-sizeof/Include/abstract.h
> python/branches/okkoto-sizeof/Include/object.h
> python/branches/okkoto-sizeof/Lib/test/test_builtin.py
> python/branches/okkoto-sizeof/Objects/abstract.c
> python/branches/okkoto-sizeof/Objects/dictobject.c
> python/branches/okkoto-sizeof/Python/bltinmodule.c
>
> Modified: python/branches/okkoto-sizeof/Doc/c-api/object.rst
> ==============================================================================
> --- python/branches/okkoto-sizeof/Doc/c-api/object.rst (original)
> +++ python/branches/okkoto-sizeof/Doc/c-api/object.rst Wed May 21 17:01:20 2008
> @@ -304,6 +304,13 @@
> .. versionadded:: 2.2
>
>
> +.. cfunction:: int PyObject_Footprint(PyObject *o)
> +
> + Return the memory footprint (i.e. size in bytes) of object o or NULL on failure.
"or NULL" -> or -1 ? (this function returns an integer not a pointer)
> +
> + .. versionadded:: 2.6
> +
> +
> .. cfunction:: Py_ssize_t PyObject_Length(PyObject *o)
> Py_ssize_t PyObject_Size(PyObject *o)
>
>
> Modified: python/branches/okkoto-sizeof/Doc/c-api/typeobj.rst
> ==============================================================================
> --- python/branches/okkoto-sizeof/Doc/c-api/typeobj.rst (original)
> +++ python/branches/okkoto-sizeof/Doc/c-api/typeobj.rst Wed May 21 17:01:20 2008
> @@ -1092,6 +1092,21 @@
> Weak reference list head, for weak references to this type object. Not
> inherited. Internal use only.
>
> +
> +.. cmember:: footprintfunc PyTypeObject.tp_footprint
> +
> + An optional function pointer that returns the memory footprint of the
> + object. This function should be used if the memory usage of the object
> + cannot be determined with tp_basicsize plus, if applicable,
> + tp_itemsize * length of the object. If the field is not set, the memory
> + footprint is computed as described. Note that this formula does not
> + include any referenced objects, e.g. attributes.
> +
> + The value -1 should not be returned as a normal return value; when an
> + error occurs during the computation of the hash value, the function
> + should set an exception and return -1.
> +
> +
> The remaining fields are only defined if the feature test macro
> :const:`COUNT_ALLOCS` is defined, and are for internal use only. They are
> documented here for completeness. None of these fields are inherited by
>
> Modified: python/branches/okkoto-sizeof/Doc/includes/typestruct.h
> ==============================================================================
> --- python/branches/okkoto-sizeof/Doc/includes/typestruct.h (original)
> +++ python/branches/okkoto-sizeof/Doc/includes/typestruct.h Wed May 21 17:01:20 2008
> @@ -72,5 +72,12 @@
> PyObject *tp_cache;
> PyObject *tp_subclasses;
> PyObject *tp_weaklist;
> + destructor tp_del;
> +
> + /* Type attribute cache version tag. Added in version 2.6 */
> + unsigned int tp_version_tag;
> +
> + /* Memory usage. Added in version 2.6 */
> + footprintfunc tp_footprint;
>
> } PyTypeObject;
>
> Modified: python/branches/okkoto-sizeof/Doc/library/functions.rst
> ==============================================================================
> --- python/branches/okkoto-sizeof/Doc/library/functions.rst (original)
> +++ python/branches/okkoto-sizeof/Doc/library/functions.rst Wed May 21 17:01:20 2008
> @@ -472,6 +472,16 @@
>
> The float type is described in :ref:`typesnumeric`.
>
> +
> +.. function:: footprint(x)
> +
> + Return the size of an object in bytes. The object can be any type of
> + object. All built-in objects will return correct results, but this
> + must not hold true for third-party extensions.
You should probably define if this includes contained objects or not.
For example, would these return the same size:
[None]
[[1, 2, 3]]
> +
> + .. versionadded:: 2.6
> +
> +
> .. function:: frozenset([iterable])
> :noindex:
>
>
> Modified: python/branches/okkoto-sizeof/Include/abstract.h
> ==============================================================================
> --- python/branches/okkoto-sizeof/Include/abstract.h (original)
> +++ python/branches/okkoto-sizeof/Include/abstract.h Wed May 21 17:01:20 2008
> @@ -418,6 +418,13 @@
> equivalent to the Python expression: type(o).
> */
>
> + PyAPI_FUNC(Py_ssize_t) PyObject_Footprint(PyObject *o);
> +
> + /*
> + Return the memory footprint (i.e. size in bytes) of object o or NULL
or NULL -> or -1
> + on failure.
> + */
> +
> PyAPI_FUNC(Py_ssize_t) PyObject_Size(PyObject *o);
>
> /*
>
> Modified: python/branches/okkoto-sizeof/Include/object.h
> ==============================================================================
> --- python/branches/okkoto-sizeof/Include/object.h (original)
> +++ python/branches/okkoto-sizeof/Include/object.h Wed May 21 17:01:20 2008
> @@ -328,6 +328,7 @@
> typedef int (*initproc)(PyObject *, PyObject *, PyObject *);
> typedef PyObject *(*newfunc)(struct _typeobject *, PyObject *, PyObject *);
> typedef PyObject *(*allocfunc)(struct _typeobject *, Py_ssize_t);
> +typedef Py_ssize_t (*footprintfunc)(PyObject *);
>
> typedef struct _typeobject {
> PyObject_VAR_HEAD
> @@ -408,6 +409,9 @@
> /* Type attribute cache version tag. Added in version 2.6 */
> unsigned int tp_version_tag;
>
> + /* Memory footprint. Added in version 2.6 */
> + footprintfunc tp_footprint;
> +
> #ifdef COUNT_ALLOCS
> /* these must be last and never explicitly initialized */
> Py_ssize_t tp_allocs;
>
> Modified: python/branches/okkoto-sizeof/Lib/test/test_builtin.py
> ==============================================================================
> --- python/branches/okkoto-sizeof/Lib/test/test_builtin.py (original)
> +++ python/branches/okkoto-sizeof/Lib/test/test_builtin.py Wed May 21 17:01:20 2008
> @@ -1467,6 +1467,135 @@
> self.assertEqual(bin(-(2**65)), '-0b1' + '0' * 65)
> self.assertEqual(bin(-(2**65-1)), '-0b' + '1' * 65)
>
> +class FootprintTest(unittest.TestCase):
> +
> +
> + def setUp(self):
> + import struct
The imports should probably be moved to the top of the module.
> + self.i = len(struct.pack('i', 0))
> + self.l = len(struct.pack('l', 0))
> + self.p = len(struct.pack('P', 0))
> + self.headersize = self.l + self.p
> + if hasattr(sys, "gettotalrefcount"):
> + self.headersize += 2 * self.p
> + self.f = open(TESTFN, 'wb')
> +
> + def tearDown(self):
> + import os
> + if self.f:
> + self.f.close()
> + os.remove(TESTFN)
Should probably be test_support.unlink() in case the file doesn't
exist or can't be removed.
> +
> + def check_footprint(self, o, size, ):
> + size += self.headersize
> + msg = 'wrong size for ' + str(type(o)) + ': '+\
> + str(footprint(o)) + ' != ' + str(size)
> + self.assertEqual(footprint(o), size, msg)
> +
> + def align(self, value):
> + mod = value % self.p
> + if mod != 0:
> + return value - mod + self.p
> + else:
> + return value
> +
> + def test_align(self):
> + self.assertTrue( (self.align(0) % self.p) == 0 )
> + self.assertTrue( (self.align(1) % self.p) == 0 )
> + self.assertTrue( (self.align(3) % self.p) == 0 )
> + self.assertTrue( (self.align(4) % self.p) == 0 )
> + self.assertTrue( (self.align(7) % self.p) == 0 )
> + self.assertTrue( (self.align(8) % self.p) == 0 )
> + self.assertTrue( (self.align(9) % self.p) == 0 )
> +
> + def test_standardobjects(self):
> + import inspect
> + i = self.i
> + l = self.l
> + p = self.p
> + # bool
> + self.check_footprint(True, self.l)
> + # buffer
> + self.check_footprint(buffer(''), 2*p + 2*l + self.align(i) +l)
> + # bytearray
> + self.check_footprint(bytes(), self.align(i) + l + p)
> + # cell
> + def get_cell():
> + x = 42
> + def inner():
> + return x
> + return inner
> + self.check_footprint(get_cell().func_closure[0], p)
> + # class
> + class clazz():
> + def method():
> + pass
> + self.check_footprint(clazz, 6*p)
> + # instance
> + self.check_footprint(clazz(), 3*p)
> + # method
> + self.check_footprint(clazz().method, 4*p)
> + # code
> + self.check_footprint(get_cell().func_code, self.align(4*i) + 8*p +\
> + self.align(i) + 2*p)
> + # complex
> + self.check_footprint(complex(0,1), 2*8)
> + # enumerate
> + self.check_footprint(enumerate([]), l + 3*p)
> + # reverse
> + self.check_footprint(reversed(''), l + p )
> + # file
> + self.check_footprint(self.f, 4*p + self.align(2*i) + 4*p +\
> + self.align(3*i) + 2*p + self.align(i))
> + # float
> + self.check_footprint(float(0), 8)
> + # function
> + def func(): pass
> + self.check_footprint(func, 9 * l)
> + class c():
> + @staticmethod
> + def foo():
> + pass
> + @classmethod
> + def bar(cls):
> + pass
> + # staticmethod
> + self.check_footprint(foo, l)
> + # classmethod
> + self.check_footprint(bar, l)
> + # generator
> + def get_gen(): yield 1
> + self.check_footprint(get_gen(), p + self.align(i) + 2*p)
> + # integer
> + self.check_footprint(1, l)
> + # builtin_function_or_method
> + self.check_footprint(abs, 3*p)
> + # module
> + self.check_footprint(unittest, p)
> + # xange
> + self.check_footprint(xrange(1), 3*p)
> + # slice
> + self.check_footprint(slice(0), 3*p)
> +
> + def test_variable_size(self):
> + i = self.i
> + l = self.l
> + p = self.p
> + self.headersize += l
> + # list
> + self.check_footprint([], p + l)
> + self.check_footprint([1, 2, 3], p + l)
> + # string
> + self.check_footprint('', l + self.align(i + 1))
> + self.check_footprint('abc', l + self.align(i + 1) + 3)
> +
> + def test_special_types(self):
> + i = self.i
> + l = self.l
> + p = self.p
> + # dict
> + self.check_footprint({}, 3*l + 3*p + 8*(l + 2*p))
> +
> class TestSorted(unittest.TestCase):
>
> def test_basic(self):
> @@ -1507,7 +1636,7 @@
> self.assertRaises(TypeError, sorted, data, None, lambda x,y: 0)
>
> def test_main(verbose=None):
> - test_classes = (BuiltinTest, TestSorted)
> + test_classes = (BuiltinTest, TestSorted, FootprintTest)
>
> run_unittest(*test_classes)
>
>
> Modified: python/branches/okkoto-sizeof/Objects/abstract.c
> ==============================================================================
> --- python/branches/okkoto-sizeof/Objects/abstract.c (original)
> +++ python/branches/okkoto-sizeof/Objects/abstract.c Wed May 21 17:01:20 2008
> @@ -58,6 +58,36 @@
> }
>
> Py_ssize_t
> +PyObject_Footprint(PyObject *o)
> +{
> + Py_ssize_t res;
> + Py_ssize_t size;
> + footprintfunc m;
> +
> + if (o == NULL) {
> + null_error();
> + return -1;
> + }
> +
> + m = o->ob_type->tp_footprint;
> + if (m)
> + return o->ob_type->tp_footprint(o);
> +
> + res = 0;
> + size = o->ob_type->tp_itemsize;
> + if (size > 0) {
> + Py_ssize_t len;
> + len = PyObject_Size(o);
> + if (len == -1 && PyErr_Occurred())
> + return -1;
> + if (len)
There's no need to check for len here.
> + res += len * size;
> + }
> + res += o->ob_type->tp_basicsize;
You could also initialize res to tp_basicsize above.
> + return res;
> +}
> +
> +Py_ssize_t
> PyObject_Size(PyObject *o)
> {
> PySequenceMethods *m;
>
> Modified: python/branches/okkoto-sizeof/Objects/dictobject.c
> ==============================================================================
> --- python/branches/okkoto-sizeof/Objects/dictobject.c (original)
> +++ python/branches/okkoto-sizeof/Objects/dictobject.c Wed May 21 17:01:20 2008
> @@ -2194,6 +2194,18 @@
> return dict_update_common(self, args, kwds, "dict");
> }
>
> +static Py_ssize_t
> +dict_footprint(PyDictObject *mp)
> +{
> + Py_ssize_t res;
> +
> + res = sizeof(PyDictObject) + sizeof(mp->ma_table);
> + if (mp->ma_table == mp->ma_smalltable)
> + return res;
> + else
> + return res + (mp->ma_mask + 1) * sizeof(PyDictEntry);
> +}
> +
> static PyObject *
> dict_iter(PyDictObject *dict)
> {
> @@ -2252,6 +2264,15 @@
> PyType_GenericAlloc, /* tp_alloc */
> dict_new, /* tp_new */
> PyObject_GC_Del, /* tp_free */
> + 0, /* tp_is_gc */
> + 0, /* tp_bases */
> + 0, /* tp_mro */
> + 0, /* tp_cache */
> + 0, /* tp_subclasses */
> + 0, /* tp_weaklist */
> + 0, /* tp_del */
> + 0, /* tp_version_tag */
> + (footprintfunc)dict_footprint, /* tp_footprint */
> };
>
> /* For backward compatibility with old dictionary interface */
>
> Modified: python/branches/okkoto-sizeof/Python/bltinmodule.c
> ==============================================================================
> --- python/branches/okkoto-sizeof/Python/bltinmodule.c (original)
> +++ python/branches/okkoto-sizeof/Python/bltinmodule.c Wed May 21 17:01:20 2008
> @@ -1310,6 +1310,21 @@
> \n\
> Return the number of items of a sequence or mapping.");
>
> +static PyObject *
> +builtin_footprint(PyObject *self, PyObject *o)
> +{
> + Py_ssize_t res;
> +
> + res = PyObject_Footprint(o);
> + if (res < 0 && PyErr_Occurred())
> + return NULL;
Would it be worthwhile to check the value is non-negative here rather
than returning a bogus value to the user?
Cheers,
n
More information about the Python-checkins
mailing list