Mailman 3 PyObject_New vs PyObject_NEW - Python-Dev

newer
Re: [Python-Dev] PyObject_New vs...

PyObject_New vs PyObject_NEW

David Abrahams

12 Mar 2003 12 Mar '03

12:24 p.m.

Someone I work with recently caused a test to start asserting in VC7's instrumented free() call, using a pydebug build. He explained the change this way: "I switched from PyObject_New to PyObject_NEW, which according to it's documentation omits the check for type_object != 0 and consequently should run a little bit faster" [he doesn't ever pass 0 as the typeobject] Did he miss some other important fact about PyObject_NEW? Does the doc need to be fixed? -- Dave Abrahams Boost Consulting www.boost-consulting.com

Show replies by date

M.-A. Lemburg

12 Mar 12 Mar

12:57 p.m.

David Abrahams wrote:

...

Someone I work with recently caused a test to start asserting in VC7's instrumented free() call, using a pydebug build. He explained the change this way:

"I switched from PyObject_New to PyObject_NEW, which according to it's documentation omits the check for type_object != 0 and consequently should run a little bit faster"

[he doesn't ever pass 0 as the typeobject]

Did he miss some other important fact about PyObject_NEW? Does the doc need to be fixed?

Does he use PyObject_DEL() to free the object ? -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Mar 12 2003)

...

...
...
Python/Zope Products & Consulting ... http://www.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/

Python UK 2003, Oxford: 20 days left EuroPython 2003, Charleroi, Belgium: 104 days left

Guido van Rossum

1:08 p.m.

...

Someone I work with recently caused a test to start asserting in VC7's instrumented free() call, using a pydebug build. He explained the change this way:

"I switched from PyObject_New to PyObject_NEW, which according to it's documentation omits the check for type_object != 0 and consequently should run a little bit faster"

[he doesn't ever pass 0 as the typeobject]

Did he miss some other important fact about PyObject_NEW? Does the doc need to be fixed?

You can read the source code as well as I can. --Guido van Rossum (home page: http://www.python.org/~guido/)

Tim Peters

5:15 p.m.

[David Abrahams]

...

Someone I work with recently caused a test to start asserting in VC7's instrumented free() call, using a pydebug build. He explained the change this way:

"I switched from PyObject_New to PyObject_NEW, which according to it's documentation omits the check for type_object != 0 and consequently should run a little bit faster"

[he doesn't ever pass 0 as the typeobject]

...

Did he miss some other important fact about PyObject_NEW? Does the doc need to be fixed?

[Guido]

...

You can read the source code as well as I can.

Possibly, but not as well as I can <wink> -- the memory API's implementation is monumentally convoluted, especially before 2.3. Speaking of which, David, which version of Python was "someone" using? Did they enable pymalloc? Did they give you a traceback (showing from where free() was called)? Was it even freeing a Python object at the time? In what code base did someone make this substitution (e.g., Python core, Boost sources, someone's own extension module, someone else's extension module)? The straight answer to your question is no. A nastier answer is that many memory mgmt screwups are shy, and can be triggered by seemingly irrelevant changes.

David Abrahams

6:03 p.m.

Tim Peters writes:

...

[Guido]

...
You can read the source code as well as I can.

Possibly, but not as well as I can <wink> -- the memory API's implementation is monumentally convoluted, especially before 2.3. Speaking of which, David, which version of Python was "someone" using?

I was the one who discovered the problem, using Python 2.2.2. Curiously, "someone" missed it because he was using vc6 instead of vc7.

...

Did they enable pymalloc?

I don't think I did that. I don't exactly know what pymalloc is.

...

Did they give you a traceback (showing from where free() was called)?

I can get one for you. Here:

...

MSVCRTD.DLL!_free_dbg_lk(void * pUserData=0x00c46338, int nBlockUse=1) Line 1044 + 0x30 C MSVCRTD.DLL!_free_dbg(void * pUserData=0x00c46338, int nBlockUse=1) Line 1001 + 0xd C MSVCRTD.DLL!free(void * pUserData=0x00c46338) Line 956 + 0xb C python22_d.dll!_PyObject_Del(_object * op=0x00c46338) Line 146 + 0xa C opaque_ext_d.pyd!dealloc(_object * self=0x00c46338) Line 12 + 0xa C++ python22_d.dll!_Py_Dealloc(_object * op=0x00c46338) Line 1837 + 0x7 C python22_d.dll!tupledealloc(PyTupleObject * op=0x0093c9a8) Line 147 + 0x70 C python22_d.dll!_Py_Dealloc(_object * op=0x0093c9a8) Line 1837 + 0x7 C python22_d.dll!do_call(_object * func=0x00c45e00, _object * * * pp_stack=0x0012edf8, int na=1, int nk=0) Line 3273 + 0x43 C python22_d.dll!eval_frame(_frame * f=0x008c2e68) Line 2038 + 0x1e C python22_d.dll!PyEval_EvalCodeEx(PyCodeObject * co=0x00963068, _object * globals=0x008e65a8, _object * locals=0x008e65a8, _object * * args=0x00000000, int argcount=0, _object * * kws=0x00000000, int kwcount=0, _object * * defs=0x00000000, int defcount=0, _object * closure=0x00000000) Line 2595 + 0x9 C python22_d.dll!PyEval_EvalCode(PyCodeObject * co=0x00963068, _object * globals=0x008e65a8, _object * locals=0x008e65a8) Line 486 + 0x1f C python22_d.dll!exec_statement(_frame * f=0x008efbf0, _object * prog=0x00963068, _object * globals=0x008e65a8, _object * locals=0x008e65a8) Line 3668 + 0x11 C python22_d.dll!eval_frame(_frame * f=0x008efbf0) Line 1482 + 0x15 C python22_d.dll!PyEval_EvalCodeEx(PyCodeObject * co=0x008c79a8, _object * globals=0x008e5940, _object * locals=0x00000000, _object * * args=0x00965a7c, int argcount=7, _object * * kws=0x00965a98, int kwcount=0, _object * * defs=0x00000000, int defcount=0, _object * closure=0x00000000) Line 2595 + 0x9 C python22_d.dll!fast_function(_object * func=0x0095cc98, _object * * * pp_stack=0x0012f268, int n=7, int na=7, int nk=0) Line 3173 + 0x41 C python22_d.dll!eval_frame(_frame * f=0x00965900) Line 2035 + 0x25 C python22_d.dll!PyEval_EvalCodeEx(PyCodeObject * co=0x008c6fd8, _object * globals=0x008e5940, _object * locals=0x00000000, _object * * args=0x008d8c64, int argcount=5, _object * * kws=0x008d8c78, int kwcount=0, _object * * defs=0x00000000, int defcount=0, _object * closure=0x00000000) Line 2595 + 0x9 C python22_d.dll!fast_function(_object * func=0x009551a8, _object * * * pp_stack=0x0012f494, int n=5, int na=5, int nk=0) Line 3173 + 0x41 C python22_d.dll!eval_frame(_frame * f=0x008d8af0) Line 2035 + 0x25 C python22_d.dll!PyEval_EvalCodeEx(PyCodeObject * co=0x008cee50, _object * globals=0x008e5940, _object * locals=0x00000000, _object * * args=0x008c75d8, int argcount=5, _object * * kws=0x008c75ec, int kwcount=0, _object * * defs=0x0092d7a4, int defcount=3, _object * closure=0x00000000) Line 2595 + 0x9 C python22_d.dll!fast_function(_object * func=0x00955220, _object * * * pp_stack=0x0012f6c0, int n=5, int na=5, int nk=0) Line 3173 + 0x41 C python22_d.dll!eval_frame(_frame * f=0x008c7450) Line 2035 + 0x25 C python22_d.dll!PyEval_EvalCodeEx(PyCodeObject * co=0x008ddba0, _object * globals=0x008e5940, _object * locals=0x00000000, _object * * args=0x0089582c, int argcount=3, _object * * kws=0x00895838, int kwcount=0, _object * * defs=0x0092e2cc, int defcount=1, _object * closure=0x00000000) Line 2595 + 0x9 C python22_d.dll!fast_function(_object * func=0x0095a850, _object * * * pp_stack=0x0012f8ec, int n=3, int na=3, int nk=0) Line 3173 + 0x41 C python22_d.dll!eval_frame(_frame * f=0x008956a8) Line 2035 + 0x25 C python22_d.dll!PyEval_EvalCodeEx(PyCodeObject * co=0x008e3848, _object * globals=0x008e5940, _object * locals=0x00000000, _object * * args=0x008787f4, int argcount=1, _object * * kws=0x008787f8, int kwcount=0, _object * * defs=0x0092eaa4, int defcount=5, _object * closure=0x00000000) Line 2595 + 0x9 C python22_d.dll!fast_function(_object * func=0x0095e620, _object * * * pp_stack=0x0012fb18, int n=1, int na=1, int nk=0) Line 3173 + 0x41 C python22_d.dll!eval_frame(_frame * f=0x00878690) Line 2035 + 0x25 C python22_d.dll!PyEval_EvalCodeEx(PyCodeObject * co=0x008ce618, _object * globals=0x0086f780, _object * locals=0x00000000, _object * * args=0x008777bc, int argcount=0, _object * * kws=0x008777bc, int kwcount=0, _object * * defs=0x0084e56c, int defcount=1, _object * closure=0x00000000) Line 2595 + 0x9 C python22_d.dll!fast_function(_object * func=0x008736e8, _object * * * pp_stack=0x0012fd44, int n=0, int na=0, int nk=0) Line 3173 + 0x41 C python22_d.dll!eval_frame(_frame * f=0x00877660) Line 2035 + 0x25 C python22_d.dll!PyEval_EvalCodeEx(PyCodeObject * co=0x0084d8a8, _object * globals=0x0086f780, _object * locals=0x0086f780, _object * * args=0x00000000, int argcount=0, _object * * kws=0x00000000, int kwcount=0, _object * * defs=0x00000000, int defcount=0, _object * closure=0x00000000) Line 2595 + 0x9 C python22_d.dll!PyEval_EvalCode(PyCodeObject * co=0x0084d8a8, _object * globals=0x0086f780, _object * locals=0x0086f780) Line 486 + 0x1f C python22_d.dll!run_node(_node * n=0x0088e730, char * filename=0x00842def, _object * globals=0x0086f780, _object * locals=0x0086f780, PyCompilerFlags * flags=0x0012ff38) Line 1079 + 0x11 C python22_d.dll!run_err_node(_node * n=0x0088e730, char * filename=0x00842def, _object * globals=0x0086f780, _object * locals=0x0086f780, PyCompilerFlags * flags=0x0012ff38) Line 1066 + 0x19 C python22_d.dll!PyRun_FileExFlags(_iobuf * fp=0x10261888, char * filename=0x00842def, int start=257, _object * globals=0x0086f780, _object * locals=0x0086f780, int closeit=1, PyCompilerFlags * flags=0x0012ff38) Line 1057 + 0x19 C python22_d.dll!PyRun_SimpleFileExFlags(_iobuf * fp=0x10261888, char * filename=0x00842def, int closeit=1, PyCompilerFlags * flags=0x0012ff38) Line 686 + 0x22 C python22_d.dll!PyRun_AnyFileExFlags(_iobuf * fp=0x10261888, char * filename=0x00842def, int closeit=1, PyCompilerFlags * flags=0x0012ff38) Line 495 + 0x15 C python22_d.dll!Py_Main(int argc=2, char * * argv=0x00842db8) Line 367 + 0x30 C python_d.exe!main(int argc=2, char * * argv=0x00842db8) Line 10 + 0xd C python_d.exe!mainCRTStartup() Line 338 + 0x11 C kernel32.dll!77e814c7()

...

Was it even freeing a Python object at the time?

Yup.

...

In what code base did someone make this substitution (e.g., Python core, Boost sources, someone's own extension module, someone else's extension module)?

Boost sources

...

The straight answer to your question is no. A nastier answer is that many memory mgmt screwups are shy, and can be triggered by seemingly irrelevant changes.

Both answers seem to amount to "'someone' must have a bug in his code". Am I reading that correctly? -- Dave Abrahams Boost Consulting www.boost-consulting.com

Tim Peters

6:18 p.m.

[David Abrahams]

...

I was the one who discovered the problem, using Python 2.2.2. Curiously, "someone" missed it because he was using vc6 instead of vc7.

So you were using VC7. If so, using it for what? Every stick of code in question, or were you mixing VC7-compiled code with VC6-compiled code? If the latter, talk to Microsoft (by most accounts their runtime support libraries aren't compatible with each other). [traceback freeing a tuple]

...

Both answers seem to amount to "'someone' must have a bug in his code". Am I reading that correctly?

Yes, for the right meaning of "someone". Possibilities beyond you include Python and Microsoft. Best guess I can make based on what you haven't told us yet is that you were mixing the released Python 2.2.2 Windows core DLL (built with MSVC6) with extension code using MSVC7 C runtime libraries. Right or wrong?

David Abrahams

6:42 p.m.

Tim Peters writes:

...

[David Abrahams]

...
I was the one who discovered the problem, using Python 2.2.2. Curiously, "someone" missed it because he was using vc6 instead of vc7.

So you were using VC7. If so, using it for what? Every stick of code in question, or were you mixing VC7-compiled code with VC6-compiled code?

Python was compiled with vc6, the rest with vc7. I test this combination regularly and have never seen a problem.

...

If the latter, talk to Microsoft (by most accounts their runtime support libraries aren't compatible with each other).

Sure, but that's only an issue if you are allocating resources in one runtime lib and deallocating in another AFAIK. There's nothing beyond memory allocation going on here, and the type object in question has a custom deallocator which goes to the same runtime that allocated it.

...

...
Both answers seem to amount to "'someone' must have a bug in his code". Am I reading that correctly?

Yes, for the right meaning of "someone". Possibilities beyond you include Python and Microsoft. Best guess I can make based on what you haven't told us yet is that you were mixing the released Python 2.2.2 Windows core DLL (built with MSVC6) with extension code using MSVC7 C runtime libraries. Right or wrong?

Totally and absolutely right. -- Dave Abrahams Boost Consulting www.boost-consulting.com

Tim Peters

6:57 p.m.

[David Abrahams]

...

Python was compiled with vc6, the rest with vc7. I test this combination regularly and have never seen a problem.

You have now <wink>.

...

Sure, but that's only an issue if you are allocating resources in one runtime lib and deallocating in another AFAIK. There's nothing beyond memory allocation going on here, and the type object in question has a custom deallocator which goes to the same runtime that allocated it.

See my later msg -- returning memory to a heap it wasn't obtained from is fatal enough. The object memory itself is in question here, not memory allocated *by* the object. Look at the traceback you sent if the distinction isn't clear.

David Abrahams

7:09 p.m.

Tim Peters writes:

...

...
Sure, but that's only an issue if you are allocating resources in one runtime lib and deallocating in another AFAIK. There's nothing beyond memory allocation going on here, and the type object in question has a custom deallocator which goes to the same runtime that allocated it.

See my later msg -- returning memory to a heap it wasn't obtained from is fatal enough.

I think that's exactly what I said.

...

The object memory itself is in question here, not memory allocated *by* the object.

I think that's also exactly what I thought. -- Dave Abrahams Boost Consulting www.boost-consulting.com

martin＠v.loewis.de

9:33 p.m.

David Abrahams writes:

...

Sure, but that's only an issue if you are allocating resources in one runtime lib and deallocating in another AFAIK.

No. You also cannot pass struct FILE* from one C library to the other; file locking will then crash. Regards, Martin

David Abrahams

10:15 p.m.

martin@v.loewis.de (Martin v. Löwis) writes:

...

David Abrahams writes:

...
Sure, but that's only an issue if you are allocating resources in one runtime lib and deallocating in another AFAIK.

No. You also cannot pass struct FILE* from one C library to the other; file locking will then crash.

A file is a resource. -- Dave Abrahams Boost Consulting www.boost-consulting.com

martin＠v.loewis.de

13 Mar 13 Mar

7:46 a.m.

David Abrahams writes:

...

martin@v.loewis.de (Martin v. Löwis) writes:

...
David Abrahams writes:

...
Sure, but that's only an issue if you are allocating resources in one runtime lib and deallocating in another AFAIK.

No. You also cannot pass struct FILE* from one C library to the other; file locking will then crash.

A file is a resource.

Yes, but printf is neither allocation nor deallocation. Regards, Martin

David Abrahams

12:26 p.m.

martin@v.loewis.de (Martin v. Löwis) writes:

...

David Abrahams writes:

...
...
No. You also cannot pass struct FILE* from one C library to the other; file locking will then crash.

A file is a resource.

Yes, but printf is neither allocation nor deallocation.

fprintf, but point taken. -- Dave Abrahams Boost Consulting www.boost-consulting.com

7714

Age (days ago)

7715

Last active (days ago)

List overview

Download

12 comments

5 participants

participants (5)

David Abrahams
Guido van Rossum
M.-A. Lemburg
martin＠v.loewis.de
Tim Peters

PyObject_New vs PyObject_NEW

M.-A. Lemburg

tags

participants (5)