Mailman 3 April 2000 - Python-Dev

{}.supplement() -- poll results so far
by pf＠artcom-gmbh.de April 4, 2000

April 4, 2000

Look's like I should better forget my proposal to add a new method '.supplement()' to dictionaries, which should do the opposite of the already available method '.update()'. I summarize in cronological order: Ka-Ping Yee: +1 Fred Drake: +0 Greg Stein: -1 Fredrik Lundh: -1 Jeremy Hylton: -1 Barry Warsaw: -0 Are there other opinions which may change the picture? <0.1 wink> Regards, Peter

4 4

unicode: strange exception
by Fredrik Lundh April 3, 2000

April 3, 2000

>>> "!" in ("a", None) 0 >>> u"!" in ("a", None) Traceback (innermost last): File "<stdin>", line 1, in ? TypeError: expected a character buffer object </F>

2 1

Re: [Patches] [1.6] dictionary objects: new method 'supplement'
by Greg Stein April 3, 2000

April 3, 2000

I don't recall the termination of the discussion, but I don't know that consensus was ever reached. Personally, I find this of little value over the similar (not exact) code: def supplement(dict, extra): d = extra.copy() d.update(dict) return d If the dictionary needs to be modified in place, then the loop from your UserDict.supplement would be used. Another view: why keep adding methods to service all possible needs? Cheers, -g On Mon, 3 Apr 2000, Peter Funk wrote: > Dear Python patcher! > > Please consider to apply the patch appended below and commit into the CVS tree. > It applies to: Python 1.6a1 as released on april 1st. > --=-- argument: --=--=--=--=--=--=--=--=--=--=-->8--=- > This patch adds a new method to dictionary and UserDict objects: > '.supplement()' is a "sibling" of '.update()', but it add only > those items that are not already there instead of replacing them. > > This idea has been discussed on python-dev last month. > --=-- obligatory disclaimer: -=--=--=--=--=--=-->8--=- > I confirm that, to the best of my knowledge and belief, this > contribution is free of any claims of third parties under > copyright, patent or other rights or interests ("claims"). To > the extent that I have any such claims, I hereby grant to CNRI a > nonexclusive, irrevocable, royalty-free, worldwide license to > reproduce, distribute, perform and/or display publicly, prepare > derivative versions, and otherwise use this contribution as part > of the Python software and its related documentation, or any > derivative versions thereof, at no cost to CNRI or its licensed > users, and to authorize others to do so. > > I acknowledge that CNRI may, at its sole discretion, decide > whether or not to incorporate this contribution in the Python > software and its related documentation. I further grant CNRI > permission to use my name and other identifying information > provided to CNRI by me for use in connection with the Python > software and its related documentation. > --=-- dry signature: =--=--=--=--=--=--=--=--=-->8--=- > Regards, Peter > -- > Peter Funk, Oldenburger Str.86, D-27777 Ganderkesee, Germany, Fax:+49 4222950260 > office: +49 421 20419-0 (ArtCom GmbH, Grazer Str.8, D-28359 Bremen) > --=-- patch: --=--=--=--=--=--=--=--=--=--=--=-->8--=- > *** ../../cvs_01_04_00_orig/dist/src/Objects/dictobject.c Fri Mar 31 11:45:02 2000 > --- src/Objects/dictobject.c Mon Apr 3 10:30:11 2000 > *************** > *** 734,739 **** > --- 734,781 ---- > } > > static PyObject * > + dict_supplement(mp, args) > + register dictobject *mp; > + PyObject *args; > + { > + register int i; > + dictobject *other; > + dictentry *entry, *oldentry; > + if (!PyArg_Parse(args, "O!", &PyDict_Type, &other)) > + return NULL; > + if (other == mp) > + goto done; /* a.supplement(a); nothing to do */ > + /* Do one big resize at the start, rather than incrementally > + resizing as we insert new items. Expect that there will be > + no (or few) overlapping keys. */ > + if ((mp->ma_fill + other->ma_used)*3 >= mp->ma_size*2) { > + if (dictresize(mp, (mp->ma_used + other->ma_used)*3/2) != 0) > + return NULL; > + } > + for (i = 0; i < other->ma_size; i++) { > + entry = &other->ma_table[i]; > + if (entry->me_value != NULL) { > + oldentry = lookdict(mp, entry->me_key, entry->me_hash); > + if (oldentry->me_value == NULL) { > + /* TODO: optimize: > + 'insertdict' does another call to 'lookdict'. > + But for sake of readability and symmetry with > + 'dict_update' I didn't tried to avoid this. > + At least not now as we go into 1.6 alpha. */ > + Py_INCREF(entry->me_key); > + Py_INCREF(entry->me_value); > + insertdict(mp, entry->me_key, entry->me_hash, > + entry->me_value); > + } > + } > + } > + done: > + Py_INCREF(Py_None); > + return Py_None; > + } > + > + > + static PyObject * > dict_copy(mp, args) > register dictobject *mp; > PyObject *args; > *************** > *** 1045,1050 **** > --- 1087,1093 ---- > {"clear", (PyCFunction)dict_clear}, > {"copy", (PyCFunction)dict_copy}, > {"get", (PyCFunction)dict_get, METH_VARARGS}, > + {"supplement", (PyCFunction)dict_supplement}, > {NULL, NULL} /* sentinel */ > }; > > *** ../../cvs_01_04_00_orig/dist/src/Lib/test/test_types.py Wed Feb 23 23:23:17 2000 > --- src/Lib/test/test_types.py Mon Apr 3 10:41:53 2000 > *************** > *** 242,247 **** > --- 242,250 ---- > d.update({2:20}) > d.update({1:1, 2:2, 3:3}) > if d != {1:1, 2:2, 3:3}: raise TestFailed, 'dict update' > + d.supplement({1:"not", 2:"neither", 4:4}) > + if d != {1:1, 2:2, 3:3, 4:4}: raise TestFailed, 'dict supplement' > + del d[4] > if d.copy() != {1:1, 2:2, 3:3}: raise TestFailed, 'dict copy' > if {}.copy() != {}: raise TestFailed, 'empty dict copy' > # dict.get() > *** ../../cvs_01_04_00_orig/dist/src/Lib/UserDict.py Wed Feb 2 16:10:14 2000 > --- src/Lib/UserDict.py Mon Apr 3 10:45:17 2000 > *************** > *** 32,36 **** > --- 32,45 ---- > else: > for k, v in dict.items(): > self.data[k] = v > + def supplement(self, dict): > + if isinstance(dict, UserDict): > + self.data.supplement(dict.data) > + elif isinstance(dict, type(self.data)): > + self.data.supplement(dict) > + else: > + for k, v in dict.items(): > + if not self.data.has_key(k): > + self.data[k] = v > def get(self, key, failobj=None): > return self.data.get(key, failobj) > *** ../../cvs_01_04_00_orig/dist/src/Lib/test/test_userdict.py Fri Mar 26 16:32:02 1999 > --- src/Lib/test/test_userdict.py Mon Apr 3 10:50:29 2000 > *************** > *** 93,101 **** > --- 93,109 ---- > t.update(u2) > assert t == u2 > > + # Test supplement > + > + t = UserDict(d1) > + t.supplement(u2) > + assert t == u2 > + > # Test get > > for i in u2.keys(): > assert u2.get(i) == u2[i] > assert u1.get(i) == d1.get(i) > assert u0.get(i) == d0.get(i) > + > + # TODO: Add a test using dir({}) to test for unimplemented methods > > _______________________________________________ > Patches mailing list > Patches(a)python.org > http://www.python.org/mailman/listinfo/patches > -- Greg Stein, http://www.lyra.org/

5 4

Unicode and numerics
by M.-A. Lemburg April 3, 2000

April 3, 2000

I've just posted a new patch set to the patches list which contains better support for Unicode in the int(), long(), float() and complex() builtins. There are some new APIs now which can be used by extension writer to convert from Unicode to integers, floats and longs. These APIs are fully Unicode aware, meaning that you can also pass them any Unicode characters with decimal mappings, not only the standard ASCII '0'-'9' ones. One thing I noticed, which needs some discussion: There are two separate APIs which convert long string literals to long objects: PyNumber_Long() and PyLong_FromString(). The first applies the same error checking as does the PyInt_FromString() API, while the latter does not apply this check... Question is: shouldn't the check for truncated data ("9.5" -> 9L) be moved into PyLong_FromString() ? BTW, should I also post patches to string.py which use the simplified versions for string.ato?() I posted a few days ago ? -- Marc-Andre Lemburg ______________________________________________________________________ Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/

1 1

Suggested PyMem & PyObject_NEW includes (fwd)
by Vladimir.Marangozov＠inrialpes.fr April 3, 2000

April 3, 2000

Vladimir Marangozov wrote:

1 0

Suggested PyMem & PyObject_NEW includes
by Vladimir.Marangozov＠inrialpes.fr April 3, 2000

April 3, 2000

Sorry for the delay -- I simply could't progress on this as I wanted to. Here's the includes I suggest for PyMem and PyObject_New cs. PyMem is okay. Some questions arise with PyObject_NEW. 1) I'm willing to unify the implementation on Windows and Unix (so I'm retaining the Windows variant of _PyObject_New reincarnated by PyObject_FromType -- see the comments in pyobjimpl.h). 2) For the user, there's the principle to use functions if binary compatibility is desired, and macros if she needs to trade compatibility for speed. But there's the issue of allocating the user objects with PyMem, or, allocate the objects with a custom allocator. After scratching my head on how to preserve bin compatibility with old libraries and offer the freedom to the user, I ended with the following (subject to discussion): - Use the functions for bin compat (but have also an exception _PyObject_Del(), with a leading underscore, for the core...) Objects in this case are allocated with PyMem. - Use the macros for allocating the objects with the potentially custom allocator (through malloc, realloc, free -- see below) What do you think? -----------------------------[ mymalloc.h ]--------------------------- ... /* * Core memory allocator * ===================== */ /* To make sure the interpreter is user-malloc friendly, all memory and object APIs are implemented on top of this one. The PyCore_* macros can be changed to make the interpreter use a custom allocator. Note that they are for internal use only. Both the core and extension modules should use the PyMem_* API. */ #define PyCore_MALLOC_FUNC malloc #define PyCore_REALLOC_FUNC realloc #define PyCore_FREE_FUNC free #define PyCore_MALLOC_PROTO Py_PROTO((size_t)) #define PyCore_REALLOC_PROTO Py_PROTO((ANY *, size_t)) #define PyCore_FREE_PROTO Py_PROTO((ANY *)) #define PyCore_MALLOC(n) PyCore_MALLOC_FUNC(n) #define PyCore_REALLOC(p, n) PyCore_REALLOC_FUNC((p), (n)) #define PyCore_FREE(p) PyCore_FREE_FUNC(p) /* The following should never be necessary */ #ifdef NEED_TO_DECLARE_MALLOC_AND_FRIEND extern ANY *PyCore_MALLOC_FUNC PyCore_MALLOC_PROTO; extern ANY *PyCore_REALLOC_FUNC PyCore_REALLOC_PROTO; extern void PyCore_FREE_FUNC PyCore_FREE_PROTO; #endif /* BEWARE: Each interface exports both functions and macros. Extension modules should normally use the functions for ensuring binary compatibility of the user's code across Python versions. Subsequently, if Python switches to its own malloc (different from standard malloc), no recompilation is required for the extensions. The macro versions trade compatibility for speed. They can be used whenever there is a performance problem, but their use implies recompilation of the code for each new Python release. The Python core uses the macros because it *is* compiled on every upgrade. This might not be the case with 3rd party extensions in a custom setup (for example, a customer does not always have access to the source of 3rd party deliverables). You have been warned! */ /* * Raw memory interface * ==================== */ /* Functions */ /* Two sets of function wrappers around malloc and friends; useful if you need to be sure that you are using the same memory allocator as Python. Note that the wrappers make sure that allocating 0 bytes returns a non-NULL pointer, even if the underlying malloc doesn't. */ /* These wrappers around malloc call PyErr_NoMemory() on failure */ extern DL_IMPORT(ANY *) Py_Malloc Py_PROTO((size_t)); extern DL_IMPORT(ANY *) Py_Realloc Py_PROTO((ANY *, size_t)); extern DL_IMPORT(void) Py_Free Py_PROTO((ANY *)); /* These wrappers around malloc *don't* call anything on failure */ extern DL_IMPORT(ANY *) PyMem_Malloc Py_PROTO((size_t)); extern DL_IMPORT(ANY *) PyMem_Realloc Py_PROTO((ANY *, size_t)); extern DL_IMPORT(void) PyMem_Free Py_PROTO((ANY *)); /* Macros */ #define PyMem_MALLOC(n) PyCore_MALLOC(n) #define PyMem_REALLOC(p, n) PyCore_REALLOC((ANY *)(p), (n)) #define PyMem_FREE(p) PyCore_FREE((ANY *)(p)) /* * Type-oriented memory interface * ============================== */ /* Functions */ #define PyMem_New(type, n) \ ( (type *) PyMem_Malloc((n) * sizeof(type)) ) #define PyMem_Resize(p, type, n) \ ( (p) = (type *) PyMem_Realloc((n) * sizeof(type)) ) #define PyMem_Del(p) PyMem_Free(p) /* Macros */ #define PyMem_NEW(type, n) \ ( (type *) PyMem_MALLOC(_PyMem_EXTRA + (n) * sizeof(type)) ) #define PyMem_RESIZE(p, type, n) \ if ((p) == NULL) \ (p) = (type *) PyMem_MALLOC( \ _PyMem_EXTRA + (n) * sizeof(type)); \ else \ (p) = (type *) PyMem_REALLOC((p), \ _PyMem_EXTRA + (n) * sizeof(type)) #define PyMem_DEL(p) PyMem_FREE(p) /* PyMem_XDEL is deprecated. To avoid the call when p is NULL, it's recommended to write the test explicitely in the code. Note that according to ANSI C, free(NULL) has no effect. */ #define PyMem_XDEL(p) if ((p) == NULL) ; else PyMem_DEL(p) ... -----------------------------[ mymalloc.h ]--------------------------- ... /* Functions and macros for modules that implement new object types. You must first include "object.h". PyObject_New(type, typeobj) allocates memory for a new object of the given type; here 'type' must be the C structure type used to represent the object and 'typeobj' the address of the corresponding type object. Reference count and type pointer are filled in; the rest of the bytes of the object are *undefined*! The resulting expression type is 'type *'. The size of the object is actually determined by the tp_basicsize field of the type object. PyObject_NewVar(type, typeobj, n) is similar but allocates a variable-size object with n extra items. The size is computed as tp_basicsize plus n * tp_itemsize. This fills in the ob_size field as well. PyObject_Del(op) releases the memory allocated for an object. Two versions of the object constructors/destructors are provided: 1) PyObject_{New, NewVar, Del} delegate the allocation of the objects to the Python allocator which places them within the bounds of the Python heap. This way, Python keeps control on the user's objects regarding their memory management; for instance, they may be subject to automatic garbage collection, once their reference count drops to zero. Binary compatibility is preserved and there's no need to recompile the extension every time a new Python release comes out. 2) PyObject_{NEW, NEW_VAR, DEL} use the allocator of the extension module which *may* differ from the one used by the Python library. Typically, in a C++ module one may wish to redefine the default allocation strategy by overloading the operators new and del. In this case, however, the extension does not cooperate with the Python memory manager. The latter has no control on the user's objects as they won't be allocated within the Python heap. Therefore, automatic garbage collection may not be performed, binary compatibility is not guaranteed and recompilation is required on every new Python release. Unless a specific memory management is needed, it's recommended to use 1). */ /* In pre-Python-1.6 times, only the PyObject_{NEW, NEW_VAR} macros were defined in terms of internal functions _PyObject_{New, NewVar}, the implementation of which used to differ for Windows and non-Windows platforms (see object.c -- these functions are left for backwards compatibility with old libraries). Starting from 1.6, an unified interface was introduced for both 1) & 2) */ extern DL_IMPORT(PyObject *) PyObject_FromType Py_PROTO((PyTypeObject *, PyObject *)); extern DL_IMPORT(PyVarObject *) PyObject_VarFromType Py_PROTO((PyTypeObject *, int, PyVarObject *)); extern DL_IMPORT(void) PyObject_Del Py_PROTO((PyObject *)); /* Functions */ #define PyObject_New(type, typeobj) \ ((type *) PyObject_FromType(typeobj, NULL)) #define PyObject_NewVar(type, typeobj, n) \ ((type *) PyObject_VarFromType((typeobj), (n), NULL)) #define PyObject_Del(op) PyObject_Del((PyObject *)(op)) /* XXX This trades binary compatibility for speed. */ #include "mymalloc.h" #define _PyObject_Del(op) PyMem_FREE((PyObject *)(op)) /* Macros */ #define PyObject_NEW(type, typeobj) \ ((type *) PyObject_FromType(typeobj, \ (PyObject *) malloc((typeobj)->tp_basicsize))) #define PyObject_NEW_VAR(type, typeobj, n) \ ((type *) PyObject_VarFromType(typeobj, \ (PyVarObject *) malloc((typeobj)->tp_basicsize + \ n * (typeobj)->tp_itemsize))) #define PyObject_DEL(op) free(op) ---------------------------------------------------------------------- So with this, I'm planning to "give the example" by renaming everywhere in the distrib PyObject_NEW with PyObject_New, but use for the core _PyObject_Del instead of PyObject_Del. I'll use PyObject_Del for the objects defined in extension modules. The point is that I don't want to define PyObject_Del in terms of PyMem_FREE (or define PyObject_New in terms of PyMem_MALLOC) as this would break the principle of binary compatibility when the Python allocator is changed to a custom malloc from one build to another. OTOH, I don't like the underscore... Do you have a better suggestion? -- Vladimir MARANGOZOV | Vladimir.Marangozov(a)inrialpes.fr http://sirac.inrialpes.fr/~marangoz | tel:(+33-4)76615277 fax:76615252 -- Vladimir MARANGOZOV | Vladimir.Marangozov(a)inrialpes.fr http://sirac.inrialpes.fr/~marangoz | tel:(+33-4)76615277 fax:76615252

1 0

Suggested PyMem & PyObject_NEW includes
by Vladimir.Marangozov＠inrialpes.fr April 3, 2000

April 3, 2000

Sorry for the delay -- I simply could't progress on this as I wanted to. Here's the includes I suggest for PyMem and PyObject_New cs. PyMem is okay. Some questions arise with PyObject_NEW. 1) I'm willing to unify the implementation on Windows and Unix (so I'm retaining the Windows variant of _PyObject_New reincarnated by PyObject_FromType -- see the comments in pyobjimpl.h). 2) For the user, there's the principle to use functions if binary compatibility is desired, and macros if she needs to trade compatibility for speed. But there's the issue of allocating the user objects with PyMem, or, allocate the objects with a custom allocator. After scratching my head on how to preserve bin compatibility with old libraries and offer the freedom to the user, I ended with the following (subject to discussion): - Use the functions for bin compat (but have also an exception _PyObject_Del(), with a leading underscore, for the core...) Objects in this case are allocated with PyMem. - Use the macros for allocating the objects with the potentially custom allocator (through malloc, realloc, free -- see below) What do you think? -----------------------------[ mymalloc.h ]--------------------------- ... /* * Core memory allocator * ===================== */ /* To make sure the interpreter is user-malloc friendly, all memory and object APIs are implemented on top of this one. The PyCore_* macros can be changed to make the interpreter use a custom allocator. Note that they are for internal use only. Both the core and extension modules should use the PyMem_* API. */ #define PyCore_MALLOC_FUNC malloc #define PyCore_REALLOC_FUNC realloc #define PyCore_FREE_FUNC free #define PyCore_MALLOC_PROTO Py_PROTO((size_t)) #define PyCore_REALLOC_PROTO Py_PROTO((ANY *, size_t)) #define PyCore_FREE_PROTO Py_PROTO((ANY *)) #define PyCore_MALLOC(n) PyCore_MALLOC_FUNC(n) #define PyCore_REALLOC(p, n) PyCore_REALLOC_FUNC((p), (n)) #define PyCore_FREE(p) PyCore_FREE_FUNC(p) /* The following should never be necessary */ #ifdef NEED_TO_DECLARE_MALLOC_AND_FRIEND extern ANY *PyCore_MALLOC_FUNC PyCore_MALLOC_PROTO; extern ANY *PyCore_REALLOC_FUNC PyCore_REALLOC_PROTO; extern void PyCore_FREE_FUNC PyCore_FREE_PROTO; #endif /* BEWARE: Each interface exports both functions and macros. Extension modules should normally use the functions for ensuring binary compatibility of the user's code across Python versions. Subsequently, if Python switches to its own malloc (different from standard malloc), no recompilation is required for the extensions. The macro versions trade compatibility for speed. They can be used whenever there is a performance problem, but their use implies recompilation of the code for each new Python release. The Python core uses the macros because it *is* compiled on every upgrade. This might not be the case with 3rd party extensions in a custom setup (for example, a customer does not always have access to the source of 3rd party deliverables). You have been warned! */ /* * Raw memory interface * ==================== */ /* Functions */ /* Two sets of function wrappers around malloc and friends; useful if you need to be sure that you are using the same memory allocator as Python. Note that the wrappers make sure that allocating 0 bytes returns a non-NULL pointer, even if the underlying malloc doesn't. */ /* These wrappers around malloc call PyErr_NoMemory() on failure */ extern DL_IMPORT(ANY *) Py_Malloc Py_PROTO((size_t)); extern DL_IMPORT(ANY *) Py_Realloc Py_PROTO((ANY *, size_t)); extern DL_IMPORT(void) Py_Free Py_PROTO((ANY *)); /* These wrappers around malloc *don't* call anything on failure */ extern DL_IMPORT(ANY *) PyMem_Malloc Py_PROTO((size_t)); extern DL_IMPORT(ANY *) PyMem_Realloc Py_PROTO((ANY *, size_t)); extern DL_IMPORT(void) PyMem_Free Py_PROTO((ANY *)); /* Macros */ #define PyMem_MALLOC(n) PyCore_MALLOC(n) #define PyMem_REALLOC(p, n) PyCore_REALLOC((ANY *)(p), (n)) #define PyMem_FREE(p) PyCore_FREE((ANY *)(p)) /* * Type-oriented memory interface * ============================== */ /* Functions */ #define PyMem_New(type, n) \ ( (type *) PyMem_Malloc((n) * sizeof(type)) ) #define PyMem_Resize(p, type, n) \ ( (p) = (type *) PyMem_Realloc((n) * sizeof(type)) ) #define PyMem_Del(p) PyMem_Free(p) /* Macros */ #define PyMem_NEW(type, n) \ ( (type *) PyMem_MALLOC(_PyMem_EXTRA + (n) * sizeof(type)) ) #define PyMem_RESIZE(p, type, n) \ if ((p) == NULL) \ (p) = (type *) PyMem_MALLOC( \ _PyMem_EXTRA + (n) * sizeof(type)); \ else \ (p) = (type *) PyMem_REALLOC((p), \ _PyMem_EXTRA + (n) * sizeof(type)) #define PyMem_DEL(p) PyMem_FREE(p) /* PyMem_XDEL is deprecated. To avoid the call when p is NULL, it's recommended to write the test explicitely in the code. Note that according to ANSI C, free(NULL) has no effect. */ #define PyMem_XDEL(p) if ((p) == NULL) ; else PyMem_DEL(p) ... -----------------------------[ mymalloc.h ]--------------------------- ... /* Functions and macros for modules that implement new object types. You must first include "object.h". PyObject_New(type, typeobj) allocates memory for a new object of the given type; here 'type' must be the C structure type used to represent the object and 'typeobj' the address of the corresponding type object. Reference count and type pointer are filled in; the rest of the bytes of the object are *undefined*! The resulting expression type is 'type *'. The size of the object is actually determined by the tp_basicsize field of the type object. PyObject_NewVar(type, typeobj, n) is similar but allocates a variable-size object with n extra items. The size is computed as tp_basicsize plus n * tp_itemsize. This fills in the ob_size field as well. PyObject_Del(op) releases the memory allocated for an object. Two versions of the object constructors/destructors are provided: 1) PyObject_{New, NewVar, Del} delegate the allocation of the objects to the Python allocator which places them within the bounds of the Python heap. This way, Python keeps control on the user's objects regarding their memory management; for instance, they may be subject to automatic garbage collection, once their reference count drops to zero. Binary compatibility is preserved and there's no need to recompile the extension every time a new Python release comes out. 2) PyObject_{NEW, NEW_VAR, DEL} use the allocator of the extension module which *may* differ from the one used by the Python library. Typically, in a C++ module one may wish to redefine the default allocation strategy by overloading the operators new and del. In this case, however, the extension does not cooperate with the Python memory manager. The latter has no control on the user's objects as they won't be allocated within the Python heap. Therefore, automatic garbage collection may not be performed, binary compatibility is not guaranteed and recompilation is required on every new Python release. Unless a specific memory management is needed, it's recommended to use 1). */ /* In pre-Python-1.6 times, only the PyObject_{NEW, NEW_VAR} macros were defined in terms of internal functions _PyObject_{New, NewVar}, the implementation of which used to differ for Windows and non-Windows platforms (see object.c -- these functions are left for backwards compatibility with old libraries). Starting from 1.6, an unified interface was introduced for both 1) & 2) */ extern DL_IMPORT(PyObject *) PyObject_FromType Py_PROTO((PyTypeObject *, PyObject *)); extern DL_IMPORT(PyVarObject *) PyObject_VarFromType Py_PROTO((PyTypeObject *, int, PyVarObject *)); extern DL_IMPORT(void) PyObject_Del Py_PROTO((PyObject *)); /* Functions */ #define PyObject_New(type, typeobj) \ ((type *) PyObject_FromType(typeobj, NULL)) #define PyObject_NewVar(type, typeobj, n) \ ((type *) PyObject_VarFromType((typeobj), (n), NULL)) #define PyObject_Del(op) PyObject_Del((PyObject *)(op)) /* XXX This trades binary compatibility for speed. */ #include "mymalloc.h" #define _PyObject_Del(op) PyMem_FREE((PyObject *)(op)) /* Macros */ #define PyObject_NEW(type, typeobj) \ ((type *) PyObject_FromType(typeobj, \ (PyObject *) malloc((typeobj)->tp_basicsize))) #define PyObject_NEW_VAR(type, typeobj, n) \ ((type *) PyObject_VarFromType(typeobj, \ (PyVarObject *) malloc((typeobj)->tp_basicsize + \ n * (typeobj)->tp_itemsize))) #define PyObject_DEL(op) free(op) ---------------------------------------------------------------------- So with this, I'm planning to "give the example" by renaming everywhere in the distrib PyObject_NEW with PyObject_New, but use for the core _PyObject_Del instead of PyObject_Del. I'll use PyObject_Del for the objects defined in extension modules. The point is that I don't want to define PyObject_Del in terms of PyMem_FREE (or define PyObject_New in terms of PyMem_MALLOC) as this would break the principle of binary compatibility when the Python allocator is changed to a custom malloc from one build to another. OTOH, I don't like the underscore... Do you have a better suggestion? -- Vladimir MARANGOZOV | Vladimir.Marangozov(a)inrialpes.fr http://sirac.inrialpes.fr/~marangoz | tel:(+33-4)76615277 fax:76615252 -- Vladimir MARANGOZOV | Vladimir.Marangozov(a)inrialpes.fr http://sirac.inrialpes.fr/~marangoz | tel:(+33-4)76615277 fax:76615252

1 0

Suggested PyMem & PyObject_NEW includes
by Vladimir.Marangozov＠inrialpes.fr April 3, 2000

April 3, 2000

Sorry for the delay -- I simply could't progress on this as I wanted to. Here's the includes I suggest for PyMem and PyObject_New cs. PyMem is okay. Some questions arise with PyObject_NEW. 1) I'm willing to unify the implementation on Windows and Unix (so I'm retaining the Windows variant of _PyObject_New reincarnated by PyObject_FromType -- see the comments in pyobjimpl.h). 2) For the user, there's the principle to use functions if binary compatibility is desired, and macros if she needs to trade compatibility for speed. But there's the issue of allocating the user objects with PyMem, or, allocate the objects with a custom allocator. After scratching my head on how to preserve bin compatibility with old libraries and offer the freedom to the user, I ended with the following (subject to discussion): - Use the functions for bin compat (but have also an exception _PyObject_Del(), with a leading underscore, for the core...) Objects in this case are allocated with PyMem. - Use the macros for allocating the objects with the potentially custom allocator (through malloc, realloc, free -- see below) What do you think? -----------------------------[ mymalloc.h ]--------------------------- ... /* * Core memory allocator * ===================== */ /* To make sure the interpreter is user-malloc friendly, all memory and object APIs are implemented on top of this one. The PyCore_* macros can be changed to make the interpreter use a custom allocator. Note that they are for internal use only. Both the core and extension modules should use the PyMem_* API. */ #define PyCore_MALLOC_FUNC malloc #define PyCore_REALLOC_FUNC realloc #define PyCore_FREE_FUNC free #define PyCore_MALLOC_PROTO Py_PROTO((size_t)) #define PyCore_REALLOC_PROTO Py_PROTO((ANY *, size_t)) #define PyCore_FREE_PROTO Py_PROTO((ANY *)) #define PyCore_MALLOC(n) PyCore_MALLOC_FUNC(n) #define PyCore_REALLOC(p, n) PyCore_REALLOC_FUNC((p), (n)) #define PyCore_FREE(p) PyCore_FREE_FUNC(p) /* The following should never be necessary */ #ifdef NEED_TO_DECLARE_MALLOC_AND_FRIEND extern ANY *PyCore_MALLOC_FUNC PyCore_MALLOC_PROTO; extern ANY *PyCore_REALLOC_FUNC PyCore_REALLOC_PROTO; extern void PyCore_FREE_FUNC PyCore_FREE_PROTO; #endif /* BEWARE: Each interface exports both functions and macros. Extension modules should normally use the functions for ensuring binary compatibility of the user's code across Python versions. Subsequently, if Python switches to its own malloc (different from standard malloc), no recompilation is required for the extensions. The macro versions trade compatibility for speed. They can be used whenever there is a performance problem, but their use implies recompilation of the code for each new Python release. The Python core uses the macros because it *is* compiled on every upgrade. This might not be the case with 3rd party extensions in a custom setup (for example, a customer does not always have access to the source of 3rd party deliverables). You have been warned! */ /* * Raw memory interface * ==================== */ /* Functions */ /* Two sets of function wrappers around malloc and friends; useful if you need to be sure that you are using the same memory allocator as Python. Note that the wrappers make sure that allocating 0 bytes returns a non-NULL pointer, even if the underlying malloc doesn't. */ /* These wrappers around malloc call PyErr_NoMemory() on failure */ extern DL_IMPORT(ANY *) Py_Malloc Py_PROTO((size_t)); extern DL_IMPORT(ANY *) Py_Realloc Py_PROTO((ANY *, size_t)); extern DL_IMPORT(void) Py_Free Py_PROTO((ANY *)); /* These wrappers around malloc *don't* call anything on failure */ extern DL_IMPORT(ANY *) PyMem_Malloc Py_PROTO((size_t)); extern DL_IMPORT(ANY *) PyMem_Realloc Py_PROTO((ANY *, size_t)); extern DL_IMPORT(void) PyMem_Free Py_PROTO((ANY *)); /* Macros */ #define PyMem_MALLOC(n) PyCore_MALLOC(n) #define PyMem_REALLOC(p, n) PyCore_REALLOC((ANY *)(p), (n)) #define PyMem_FREE(p) PyCore_FREE((ANY *)(p)) /* * Type-oriented memory interface * ============================== */ /* Functions */ #define PyMem_New(type, n) \ ( (type *) PyMem_Malloc((n) * sizeof(type)) ) #define PyMem_Resize(p, type, n) \ ( (p) = (type *) PyMem_Realloc((n) * sizeof(type)) ) #define PyMem_Del(p) PyMem_Free(p) /* Macros */ #define PyMem_NEW(type, n) \ ( (type *) PyMem_MALLOC(_PyMem_EXTRA + (n) * sizeof(type)) ) #define PyMem_RESIZE(p, type, n) \ if ((p) == NULL) \ (p) = (type *) PyMem_MALLOC( \ _PyMem_EXTRA + (n) * sizeof(type)); \ else \ (p) = (type *) PyMem_REALLOC((p), \ _PyMem_EXTRA + (n) * sizeof(type)) #define PyMem_DEL(p) PyMem_FREE(p) /* PyMem_XDEL is deprecated. To avoid the call when p is NULL, it's recommended to write the test explicitely in the code. Note that according to ANSI C, free(NULL) has no effect. */ #define PyMem_XDEL(p) if ((p) == NULL) ; else PyMem_DEL(p) ... -----------------------------[ mymalloc.h ]--------------------------- ... /* Functions and macros for modules that implement new object types. You must first include "object.h". PyObject_New(type, typeobj) allocates memory for a new object of the given type; here 'type' must be the C structure type used to represent the object and 'typeobj' the address of the corresponding type object. Reference count and type pointer are filled in; the rest of the bytes of the object are *undefined*! The resulting expression type is 'type *'. The size of the object is actually determined by the tp_basicsize field of the type object. PyObject_NewVar(type, typeobj, n) is similar but allocates a variable-size object with n extra items. The size is computed as tp_basicsize plus n * tp_itemsize. This fills in the ob_size field as well. PyObject_Del(op) releases the memory allocated for an object. Two versions of the object constructors/destructors are provided: 1) PyObject_{New, NewVar, Del} delegate the allocation of the objects to the Python allocator which places them within the bounds of the Python heap. This way, Python keeps control on the user's objects regarding their memory management; for instance, they may be subject to automatic garbage collection, once their reference count drops to zero. Binary compatibility is preserved and there's no need to recompile the extension every time a new Python release comes out. 2) PyObject_{NEW, NEW_VAR, DEL} use the allocator of the extension module which *may* differ from the one used by the Python library. Typically, in a C++ module one may wish to redefine the default allocation strategy by overloading the operators new and del. In this case, however, the extension does not cooperate with the Python memory manager. The latter has no control on the user's objects as they won't be allocated within the Python heap. Therefore, automatic garbage collection may not be performed, binary compatibility is not guaranteed and recompilation is required on every new Python release. Unless a specific memory management is needed, it's recommended to use 1). */ /* In pre-Python-1.6 times, only the PyObject_{NEW, NEW_VAR} macros were defined in terms of internal functions _PyObject_{New, NewVar}, the implementation of which used to differ for Windows and non-Windows platforms (see object.c -- these functions are left for backwards compatibility with old libraries). Starting from 1.6, an unified interface was introduced for both 1) & 2) */ extern DL_IMPORT(PyObject *) PyObject_FromType Py_PROTO((PyTypeObject *, PyObject *)); extern DL_IMPORT(PyVarObject *) PyObject_VarFromType Py_PROTO((PyTypeObject *, int, PyVarObject *)); extern DL_IMPORT(void) PyObject_Del Py_PROTO((PyObject *)); /* Functions */ #define PyObject_New(type, typeobj) \ ((type *) PyObject_FromType(typeobj, NULL)) #define PyObject_NewVar(type, typeobj, n) \ ((type *) PyObject_VarFromType((typeobj), (n), NULL)) #define PyObject_Del(op) PyObject_Del((PyObject *)(op)) /* XXX This trades binary compatibility for speed. */ #include "mymalloc.h" #define _PyObject_Del(op) PyMem_FREE((PyObject *)(op)) /* Macros */ #define PyObject_NEW(type, typeobj) \ ((type *) PyObject_FromType(typeobj, \ (PyObject *) malloc((typeobj)->tp_basicsize))) #define PyObject_NEW_VAR(type, typeobj, n) \ ((type *) PyObject_VarFromType(typeobj, \ (PyVarObject *) malloc((typeobj)->tp_basicsize + \ n * (typeobj)->tp_itemsize))) #define PyObject_DEL(op) free(op) ---------------------------------------------------------------------- So with this, I'm planning to "give the example" by renaming everywhere in the distrib PyObject_NEW with PyObject_New, but use for the core _PyObject_Del instead of PyObject_Del. I'll use PyObject_Del for the objects defined in extension modules. The point is that I don't want to define PyObject_Del in terms of PyMem_FREE (or define PyObject_New in terms of PyMem_MALLOC) as this would break the principle of binary compatibility when the Python allocator is changed to a custom malloc from one build to another. OTOH, I don't like the underscore... Do you have a better suggestion? -- Vladimir MARANGOZOV | Vladimir.Marangozov(a)inrialpes.fr http://sirac.inrialpes.fr/~marangoz | tel:(+33-4)76615277 fax:76615252 -- Vladimir MARANGOZOV | Vladimir.Marangozov(a)inrialpes.fr http://sirac.inrialpes.fr/~marangoz | tel:(+33-4)76615277 fax:76615252

1 0

string.ato? and Unicode
by Mark Hammond April 2, 2000

April 2, 2000

Is this an over-sight, or by design? >>> string.atoi(u"1") ... TypeError: argument 1: expected string, unicode found It appears easy to support Unicode - there is already an explicit StringType check in these functions, and it simply delegates to int(), which already _does_ work for Unicode A patch would leave the following behaviour: >>> string.atio(u"1") 1 >>> string.atio(u"1", 16) ... TypeError: can't convert non-string with explicit base IMO, this is better than what we have now. I'll put together a patch if one is wanted... Mark.

2 1

Re: [Python-checkins] CVS: python/dist/src/Modules mmapmodule.c,2.1,2.2
by Barry A. Warsaw April 2, 2000

April 2, 2000

>>>>> "Guido" == Guido van Rossum <guido(a)cnri.reston.va.us> writes: Guido> Modified Files: mmapmodule.c Log Message: Hacked for Win32 Guido> by Mark Hammond. Reformatted for 8-space tabs and fitted Guido> into 80-char lines by GvR. Can we change the 8-space-tab rule for all new C code that goes in? I know that we can't practically change existing code right now, but for new C code, I propose we use no tab characters, and we use a 4-space block indentation. -Barry

9 14