Hi Joe, was thinking of messing with the memory allocator yes. maybe
there are other areas also, but the memory allocator would be a good
place to start.
This could be a compile time option for python, maybe a module thats
only built in debug mode, or a seperate option.
---
import CAPIForceError
CAPIForceError.setmemfail(0.01)
...test the apis error stability...
---
So python could initialize etc without failing, and the test code would
have to set the error probability before it runs a script that tests the
api.
Joe Eagar wrote:
> One thing you could do is hack the python allocator to fail every once
> in a while. However, this is likely to cause problems within python
> itself, along with problems with a python api codebase.
>
> The allocator is in Objects/obmalloc.c in the python source, iirc.
>
> Joe
>
> Campbell Barton wrote:
>> In the Python C api is that your supposed to check every operation
>> succeeds. - Append, a new list, a new float etc.
>>
>> This makes sense, since somebody could append to a list until the
>> system is out of memory.. or whatever.
>>
>> The thing that bothers me is theres no good way to test the error cases.
>>
>> For example in Blender3D and PyGame (the only 2 C API's I'v looked at)
>> - many checks for failed operations are missing. so its possible
>> somebody could run out of ram and crash the API with python.
>>
>> for Blender3D or PyGame it probably dosnt matter a great deal, but in
>> other cases, you'd want to make sure that doing crazy stuff within an
>> exception (for example) wont crash the application.
>> Not just to look at the code and think it should work but actually run
>> the error case.
>>
>> Is there any way to do this?
>>
>> I was thinking there could be a debug mode where python/C API
>> functions like PyObject_New, PyList_Append etc randomly fail a
>> percentage of the time.
>>
>> Then you could run unit tests in a debugger and any crashes could be
>> traced.
>> This isnt an exact approach but if the unit tests run enough times you
>> could be fairly sure its well tested (assuming the unit tests cover
>> the API).
>>
>> Anyone tried this before or does something like this already exist?
>>
>>
>
>
--
Campbell J Barton (ideasman42)
In the Python C api is that your supposed to check every operation
succeeds. - Append, a new list, a new float etc.
This makes sense, since somebody could append to a list until the system
is out of memory.. or whatever.
The thing that bothers me is theres no good way to test the error cases.
For example in Blender3D and PyGame (the only 2 C API's I'v looked at) -
many checks for failed operations are missing. so its possible somebody
could run out of ram and crash the API with python.
for Blender3D or PyGame it probably dosnt matter a great deal, but in
other cases, you'd want to make sure that doing crazy stuff within an
exception (for example) wont crash the application.
Not just to look at the code and think it should work but actually run
the error case.
Is there any way to do this?
I was thinking there could be a debug mode where python/C API functions
like PyObject_New, PyList_Append etc randomly fail a percentage of the time.
Then you could run unit tests in a debugger and any crashes could be traced.
This isnt an exact approach but if the unit tests run enough times you
could be fairly sure its well tested (assuming the unit tests cover the
API).
Anyone tried this before or does something like this already exist?
--
Campbell J Barton (ideasman42)
Hi all,
I just wanted to announce the availability of the first official Cython
release (0.9) by the SAGE maintainers William Stein and Robert Bradshaw (and a
bit by myself).
http://www.cython.org/
Cython is based on the well-known Pyrex translator by Greg Ewing, but supports
more cutting edge functionality and optimizations. It was formerly known as
the SageX fork of Pyrex.
Some of the highlights compared to the latest Pyrex version:
1. Conditional expressions (a if blah else b)
2. List comprehensions
3. Optimized looping (for whatever in blah: is much faster in Cython)
4. Better support for module imports: import a.b.cython_module
5. 64-bit Python 2.5 support
and there's a lot more in the queue ...
Cython has a site up on launchpad, so if you have any questions or
suggestions, found any bugs or want to contribute patches, please visit the
project there:
https://launchpad.net/cython/
Have fun,
Stefan
> double PyNumber_AsDouble(PyObject * value)
> {
> if (PyFloat_Check(value) || PyInt_Check(value)) {
> return PyFloat_AsDouble(value);
> } else {
> PyObject *pyfloat;
> double d;
>
> pyfloat = PyNumber_Float(value);
> d = PyNumber_AsDouble(pyfloat);
I assume this should be PyFloat_AsDouble?
> Py_XDECREF(pyfloat);
> return d;
> }
> }
What happens if PyNumber_Float raises an exception? As far as I can
tell, PyNumber_AsDouble will result in a "bad argument" exception
being raised and will return -1, losing track of the original
exception in the process.
Come to think of it, error checking problems could very well be the
reason why the API doesn't provide a PyNumber_AsDouble.
Hello.
Sorry if posting to wrong maillist, I have read full list and decided that
this one is the most appropriate.
I have a quick question: am I able to do multiple
initialisations/deinitialisations of embedded python interpreter?
I mean - I'd like to init python, run a script in it for some time (say,
couple of hours) and then just deinitialise python so everything, including
the python program that didn't expected such brutal end to be freed and
deinitialised. Then, after some more time, call Py_Initialize() again and
again load and start using my python program.
The most essential for me is freeing all sorts of RAM that python allocated
for objects or whatever else.
IOW - I want python to be returned in virgin state as it was on my program
load w/o exiting my program and closing all files/tcp connections that it
have opened/or resetting any of other my C data structures that were created.
Is that possible?
--
Respectfully
Alexey Nezhdanov
Iv noticed that theres no one liner way to get a number (double) from a
PyNumber from the C api.
PyFloat_AsDouble - http://www.python.org/doc/api/floatObjects.html
- Does what I want, but only works on PyFloat (and PyInts), not user
implementations of the PyNumber protocol.
PyNumber_Float - http://www.python.org/doc/api/number.html
- does what I need but, returnd a PyFloat that needs to then be passed
to PyFloat_AsDouble,
PyNumber_Float
Id like somthing like PyNumber_AsDouble,
thinking of writing a utility function for this that could read...
/* assumes PyNumber_Check has been tested, should not segfault either way */
double PyNumber_AsDouble(PyObject * value)
{
if (PyFloat_Check(value) || PyInt_Check(value)) {
return PyFloat_AsDouble(value);
} else {
PyObject *pyfloat;
double d;
pyfloat = PyNumber_Float(value);
d = PyNumber_AsDouble(pyfloat);
Py_XDECREF(pyfloat);
return d;
}
}
Using PyNumber_Float for all values would be a bit slower since it needs
to make a new pyfloat each time which is why Id prefer to use
PyFloat_AsDouble where possible.
Wondering if other people would fine something like this useful in the
Py/C api - or if Im missing something.
Thanks
--
Campbell J Barton (ideasman42)
Hi Guys,
first post on the list, dont be shy ;)
In Blender3D were doing an API refactor and Im looking at better ways to
write the Python C/API documentation.
At the moment we use epydoc, Its all Iv used so I don't have much to
compare it to.
If you havnt used it, its basically a way to add formatting to your
normal .py docstrings so they can be converted into HTML or PDF.
This works well with python only projects but is not somthing you can do
in C without some manual docstring extraction from the C/API.
for blender3d we have C doctrings as well as python files that just
contain functions with docstrings (no actual code) for epydoc to write
them to html.
The C/API docstings often end up being brief and not that helpful, since
the epydocs are what most people use so more effort is put there.
Here are the current epydocs.
http://www.blender.org/documentation/244PythonDoc/index.html
Ideally Id like to only maintain 1 set of docs... so possibilities are
1) dont write epydocs, only use docstrings (we need a way we can browse
them as HTML still)
2) dont write C API/docstrings, just have empty strings where all the
docstrings would go.
3) include epydoc formatting in C API/docstrings and pass these
docstrings to epydoc for it to generate HTML.
Iv got 3) working as a testcase, the message attached details how this
works with the pro's and con's of adding epydoc tags into C/API docstrings.
However Id be interested to know how other projects deal with the issue
of maintaining docstrings and online docs.
Cheers
