Mailman 3 July 2007 - capi-sig

Re: [capi-sig] C/API error handling
by Campbell Barton 29 Jul '07

29 Jul '07

Hi Joe, was thinking of messing with the memory allocator yes. maybe there are other areas also, but the memory allocator would be a good place to start. This could be a compile time option for python, maybe a module thats only built in debug mode, or a seperate option. --- import CAPIForceError CAPIForceError.setmemfail(0.01) ...test the apis error stability... --- So python could initialize etc without failing, and the test code would have to set the error probability before it runs a script that tests the api. Joe Eagar wrote: > One thing you could do is hack the python allocator to fail every once > in a while. However, this is likely to cause problems within python > itself, along with problems with a python api codebase. > > The allocator is in Objects/obmalloc.c in the python source, iirc. > > Joe > > Campbell Barton wrote: >> In the Python C api is that your supposed to check every operation >> succeeds. - Append, a new list, a new float etc. >> >> This makes sense, since somebody could append to a list until the >> system is out of memory.. or whatever. >> >> The thing that bothers me is theres no good way to test the error cases. >> >> For example in Blender3D and PyGame (the only 2 C API's I'v looked at) >> - many checks for failed operations are missing. so its possible >> somebody could run out of ram and crash the API with python. >> >> for Blender3D or PyGame it probably dosnt matter a great deal, but in >> other cases, you'd want to make sure that doing crazy stuff within an >> exception (for example) wont crash the application. >> Not just to look at the code and think it should work but actually run >> the error case. >> >> Is there any way to do this? >> >> I was thinking there could be a debug mode where python/C API >> functions like PyObject_New, PyList_Append etc randomly fail a >> percentage of the time. >> >> Then you could run unit tests in a debugger and any crashes could be >> traced. >> This isnt an exact approach but if the unit tests run enough times you >> could be fairly sure its well tested (assuming the unit tests cover >> the API). >> >> Anyone tried this before or does something like this already exist? >> >> > > -- Campbell J Barton (ideasman42)

1 0

C/API error handling
by Campbell Barton 28 Jul '07

28 Jul '07

In the Python C api is that your supposed to check every operation succeeds. - Append, a new list, a new float etc. This makes sense, since somebody could append to a list until the system is out of memory.. or whatever. The thing that bothers me is theres no good way to test the error cases. For example in Blender3D and PyGame (the only 2 C API's I'v looked at) - many checks for failed operations are missing. so its possible somebody could run out of ram and crash the API with python. for Blender3D or PyGame it probably dosnt matter a great deal, but in other cases, you'd want to make sure that doing crazy stuff within an exception (for example) wont crash the application. Not just to look at the code and think it should work but actually run the error case. Is there any way to do this? I was thinking there could be a debug mode where python/C API functions like PyObject_New, PyList_Append etc randomly fail a percentage of the time. Then you could run unit tests in a debugger and any crashes could be traced. This isnt an exact approach but if the unit tests run enough times you could be fairly sure its well tested (assuming the unit tests cover the API). Anyone tried this before or does something like this already exist? -- Campbell J Barton (ideasman42)

1 0

[ANN] The Cython project
by Stefan Behnel 28 Jul '07

28 Jul '07

Hi all, I just wanted to announce the availability of the first official Cython release (0.9) by the SAGE maintainers William Stein and Robert Bradshaw (and a bit by myself). http://www.cython.org/ Cython is based on the well-known Pyrex translator by Greg Ewing, but supports more cutting edge functionality and optimizations. It was formerly known as the SageX fork of Pyrex. Some of the highlights compared to the latest Pyrex version: 1. Conditional expressions (a if blah else b) 2. List comprehensions 3. Optimized looping (for whatever in blah: is much faster in Cython) 4. Better support for module imports: import a.b.cython_module 5. 64-bit Python 2.5 support and there's a lot more in the queue ... Cython has a site up on launchpad, so if you have any questions or suggestions, found any bugs or want to contribute patches, please visit the project there: https://launchpad.net/cython/ Have fun, Stefan

1 0

Re: [capi-sig] Getting Numbers in C
by Hrvoje Niksic 21 Jul '07

21 Jul '07

> double PyNumber_AsDouble(PyObject * value) > { > if (PyFloat_Check(value) || PyInt_Check(value)) { > return PyFloat_AsDouble(value); > } else { > PyObject *pyfloat; > double d; > > pyfloat = PyNumber_Float(value); > d = PyNumber_AsDouble(pyfloat); I assume this should be PyFloat_AsDouble? > Py_XDECREF(pyfloat); > return d; > } > } What happens if PyNumber_Float raises an exception? As far as I can tell, PyNumber_AsDouble will result in a "bad argument" exception being raised and will return -1, losing track of the original exception in the process. Come to think of it, error checking problems could very well be the reason why the API doesn't provide a PyNumber_AsDouble.

2 6

is deinitialisation of embedded python equal to full restart?
by Alexey Nezhdanov 16 Jul '07

16 Jul '07

Hello. Sorry if posting to wrong maillist, I have read full list and decided that this one is the most appropriate. I have a quick question: am I able to do multiple initialisations/deinitialisations of embedded python interpreter? I mean - I'd like to init python, run a script in it for some time (say, couple of hours) and then just deinitialise python so everything, including the python program that didn't expected such brutal end to be freed and deinitialised. Then, after some more time, call Py_Initialize() again and again load and start using my python program. The most essential for me is freeing all sorts of RAM that python allocated for objects or whatever else. IOW - I want python to be returned in virgin state as it was on my program load w/o exiting my program and closing all files/tcp connections that it have opened/or resetting any of other my C data structures that were created. Is that possible? -- Respectfully Alexey Nezhdanov

4 5

Getting Numbers in C
by Campbell Barton 11 Jul '07

11 Jul '07

Iv noticed that theres no one liner way to get a number (double) from a PyNumber from the C api. PyFloat_AsDouble - http://www.python.org/doc/api/floatObjects.html - Does what I want, but only works on PyFloat (and PyInts), not user implementations of the PyNumber protocol. PyNumber_Float - http://www.python.org/doc/api/number.html - does what I need but, returnd a PyFloat that needs to then be passed to PyFloat_AsDouble, PyNumber_Float Id like somthing like PyNumber_AsDouble, thinking of writing a utility function for this that could read... /* assumes PyNumber_Check has been tested, should not segfault either way */ double PyNumber_AsDouble(PyObject * value) { if (PyFloat_Check(value) || PyInt_Check(value)) { return PyFloat_AsDouble(value); } else { PyObject *pyfloat; double d; pyfloat = PyNumber_Float(value); d = PyNumber_AsDouble(pyfloat); Py_XDECREF(pyfloat); return d; } } Using PyNumber_Float for all values would be a bit slower since it needs to make a new pyfloat each time which is why Id prefer to use PyFloat_AsDouble where possible. Wondering if other people would fine something like this useful in the Py/C api - or if Im missing something. Thanks -- Campbell J Barton (ideasman42)

1 0

Documenting a C/Py API
by Campbell Barton 03 Jul '07

03 Jul '07

Hi Guys, first post on the list, dont be shy ;) In Blender3D were doing an API refactor and Im looking at better ways to write the Python C/API documentation. At the moment we use epydoc, Its all Iv used so I don't have much to compare it to. If you havnt used it, its basically a way to add formatting to your normal .py docstrings so they can be converted into HTML or PDF. This works well with python only projects but is not somthing you can do in C without some manual docstring extraction from the C/API. for blender3d we have C doctrings as well as python files that just contain functions with docstrings (no actual code) for epydoc to write them to html. The C/API docstings often end up being brief and not that helpful, since the epydocs are what most people use so more effort is put there. Here are the current epydocs. http://www.blender.org/documentation/244PythonDoc/index.html Ideally Id like to only maintain 1 set of docs... so possibilities are 1) dont write epydocs, only use docstrings (we need a way we can browse them as HTML still) 2) dont write C API/docstrings, just have empty strings where all the docstrings would go. 3) include epydoc formatting in C API/docstrings and pass these docstrings to epydoc for it to generate HTML. Iv got 3) working as a testcase, the message attached details how this works with the pro's and con's of adding epydoc tags into C/API docstrings. However Id be interested to know how other projects deal with the issue of maintaining docstrings and online docs. Cheers

6 12

capi-sig July 2007