Re: [SciPy-Dev] Multivariate ctypes integration PR

Thank you to everyone involved in assisting to move this forward. It is definitely a confusing project for me but something I am committed to finishing and getting in the library as soon as possible. I appreciate the help and I will do my best to address the issues Pauli presented. Thanks, Brian

From what we can tell (see e.g. http://stackoverflow.com/questions/18775389/is-there-a-way-to-load-the-const...),
We had a question about how to implement one of the suggested improvements: "- The code should more carefully check that the whole ctypes signature matches" there is no way to automatically generate the ctypes function signature directly from a .h header file or c function prototype. This means that any ctypes library function must be assumed to have the function signature that the user claims it does (by specifying argtypes & restype). Unfortunately, the ctypes library load & specification commands are executed by the user, not by the library, so there's no way to enforce the function signature. Maybe there's a way in Cython or f2py, but we think that ctypes should probably be supported anyway, even if it is with an "use at own risk" disclaimer. Nathan On Fri, Mar 14, 2014 at 12:08 PM, Brian Lee Newsom < Brian.Newsom@colorado.edu> wrote:
Thank you to everyone involved in assisting to move this forward. It is definitely a confusing project for me but something I am committed to finishing and getting in the library as soon as possible. I appreciate the help and I will do my best to address the issues Pauli presented.
Thanks, Brian
_______________________________________________ SciPy-Dev mailing list SciPy-Dev@scipy.org http://mail.scipy.org/mailman/listinfo/scipy-dev

Hey everyone, Sorry to be the squeaky wheel again, but this needs to see some movement pretty soon. We only have another few weeks where we can work on this enhancement before the school year ends. The big challenge that we believe needs to be fixed is with the tests. We need a way to compile (or distribute) a C function from source to a library with a given name, generally and at install time, in order to include the C route tests in the SciPy test suite. Is there a way to use distutils for this? If there's anything else that we can do on our end to speed things up, we'd be more than happy to help however we can. N On Mon, Mar 31, 2014 at 1:27 PM, Nathan Woods <charlesnwoods@gmail.com>wrote:
We had a question about how to implement one of the suggested improvements:
"- The code should more carefully check that the whole ctypes signature matches"
From what we can tell (see e.g. http://stackoverflow.com/questions/18775389/is-there-a-way-to-load-the-const...), there is no way to automatically generate the ctypes function signature directly from a .h header file or c function prototype. This means that any ctypes library function must be assumed to have the function signature that the user claims it does (by specifying argtypes & restype). Unfortunately, the ctypes library load & specification commands are executed by the user, not by the library, so there's no way to enforce the function signature.
Maybe there's a way in Cython or f2py, but we think that ctypes should probably be supported anyway, even if it is with an "use at own risk" disclaimer.
Nathan
On Fri, Mar 14, 2014 at 12:08 PM, Brian Lee Newsom < Brian.Newsom@colorado.edu> wrote:
Thank you to everyone involved in assisting to move this forward. It is definitely a confusing project for me but something I am committed to finishing and getting in the library as soon as possible. I appreciate the help and I will do my best to address the issues Pauli presented.
Thanks, Brian
_______________________________________________ SciPy-Dev mailing list SciPy-Dev@scipy.org http://mail.scipy.org/mailman/listinfo/scipy-dev

From the lack of response to my email two weeks ago, I guess that there isn't really a good way to distribute a library function for testing. When ctypes functionality was initially included, the test function used was
Anyone? Anyone? part of the standard library, to avoid this same problem. I propose that we therefore do one of two things. One, we incorporate the changes as a user-beware feature, and strongly suggest that users who are willing to do the work necessary to use the functionality (a minority, presumably) do their own testing. In order to use the speed improvements here, a user has to code their function in C, compile it to a library, and load that library using ctypes. Then, that ctypes function can be passed to scipy. I think we can agree that anyone who is willing to put in that work for performance can be expected to run a test or two to make sure everything is working. Two, we shelve the project until a better solution is available. This will probably require some alternative to ctypes, which is just too clunky to distribute well. Unfortunately, I think that shelving this till later will essentially kill it at this point. Nathan On Tue, Apr 15, 2014 at 3:38 PM, Nathan Woods <charlesnwoods@gmail.com>wrote:
Hey everyone,
Sorry to be the squeaky wheel again, but this needs to see some movement pretty soon. We only have another few weeks where we can work on this enhancement before the school year ends.
The big challenge that we believe needs to be fixed is with the tests. We need a way to compile (or distribute) a C function from source to a library with a given name, generally and at install time, in order to include the C route tests in the SciPy test suite. Is there a way to use distutils for this?
If there's anything else that we can do on our end to speed things up, we'd be more than happy to help however we can.
N
On Mon, Mar 31, 2014 at 1:27 PM, Nathan Woods <charlesnwoods@gmail.com>wrote:
We had a question about how to implement one of the suggested improvements:
"- The code should more carefully check that the whole ctypes signature matches"
From what we can tell (see e.g. http://stackoverflow.com/questions/18775389/is-there-a-way-to-load-the-const...), there is no way to automatically generate the ctypes function signature directly from a .h header file or c function prototype. This means that any ctypes library function must be assumed to have the function signature that the user claims it does (by specifying argtypes & restype). Unfortunately, the ctypes library load & specification commands are executed by the user, not by the library, so there's no way to enforce the function signature.
Maybe there's a way in Cython or f2py, but we think that ctypes should probably be supported anyway, even if it is with an "use at own risk" disclaimer.
Nathan
On Fri, Mar 14, 2014 at 12:08 PM, Brian Lee Newsom < Brian.Newsom@colorado.edu> wrote:
Thank you to everyone involved in assisting to move this forward. It is definitely a confusing project for me but something I am committed to finishing and getting in the library as soon as possible. I appreciate the help and I will do my best to address the issues Pauli presented.
Thanks, Brian
_______________________________________________ SciPy-Dev mailing list SciPy-Dev@scipy.org http://mail.scipy.org/mailman/listinfo/scipy-dev

On Mon, Apr 28, 2014 at 4:46 PM, Nathan Woods <charlesnwoods@gmail.com> wrote:
Anyone? Anyone?
From the lack of response to my email two weeks ago, I guess that there isn't really a good way to distribute a library function for testing. When ctypes functionality was initially included, the test function used was part of the standard library, to avoid this same problem. I propose that we therefore do one of two things.
The stdlib tests ctypes by building a Python extension module `_ctypes_test` along with all of the other stdlib extension modules. It is careful to `EXPORT` all of the symbols that will be needed. We could do the same here. We will just have an extra _private extension module built along with all of the others while scipy is building. It will not be compiled during the unit test. http://hg.python.org/cpython/file/a14012352f65/Modules/_ctypes/_ctypes_test.... http://hg.python.org/cpython/file/a14012352f65/Lib/ctypes/test/test_cfuncs.p... -- Robert Kern

This sounds promising, but I will have to have a much better understanding of distutils than I do just at the moment to try it. I’ll see what I can figure out. N On Apr 28, 2014, at 9:53 AM, Robert Kern <robert.kern@gmail.com> wrote:
On Mon, Apr 28, 2014 at 4:46 PM, Nathan Woods <charlesnwoods@gmail.com> wrote:
Anyone? Anyone?
From the lack of response to my email two weeks ago, I guess that there isn't really a good way to distribute a library function for testing. When ctypes functionality was initially included, the test function used was part of the standard library, to avoid this same problem. I propose that we therefore do one of two things.
The stdlib tests ctypes by building a Python extension module `_ctypes_test` along with all of the other stdlib extension modules. It is careful to `EXPORT` all of the symbols that will be needed. We could do the same here. We will just have an extra _private extension module built along with all of the others while scipy is building. It will not be compiled during the unit test.
http://hg.python.org/cpython/file/a14012352f65/Modules/_ctypes/_ctypes_test.... http://hg.python.org/cpython/file/a14012352f65/Lib/ctypes/test/test_cfuncs.p...
-- Robert Kern _______________________________________________ SciPy-Dev mailing list SciPy-Dev@scipy.org http://mail.scipy.org/mailman/listinfo/scipy-dev

On Tue, Apr 29, 2014 at 4:59 AM, Nathan Woods <charlesnwoods@gmail.com> wrote:
This sounds promising, but I will have to have a much better understanding of distutils than I do just at the moment to try it. I’ll see what I can figure out.
It's really easy. Let's say you put the test module as scipy/integrate/_ctypes_test.c. Just take a look at scipy/integrate/setup.py. Add a new line: config.add_extension('_ctypes_test', sources=['_ctypes_test.c']) And in the scipy/integrate/bento.info: Extension: Sources: _ctypes_test.c -- Robert Kern

All right, I think I was able to figure this out. An updated version has been added to the PR on GitHub, and it works on both Ubuntu 12.04 and OSX 10.9. Loading the library in ctypes SHOULD be straightforward in Windows as well, but we still need to check that. We’d love it if anyone could take a look at our implementation and offer comments, since it’s different from what’s done in the python standard library. Nathan Woods On Apr 29, 2014, at 2:37 AM, Robert Kern <robert.kern@gmail.com> wrote:
On Tue, Apr 29, 2014 at 4:59 AM, Nathan Woods <charlesnwoods@gmail.com> wrote:
This sounds promising, but I will have to have a much better understanding of distutils than I do just at the moment to try it. I’ll see what I can figure out.
It's really easy. Let's say you put the test module as scipy/integrate/_ctypes_test.c. Just take a look at scipy/integrate/setup.py. Add a new line:
config.add_extension('_ctypes_test', sources=['_ctypes_test.c'])
And in the scipy/integrate/bento.info:
Extension: Sources: _ctypes_test.c
-- Robert Kern _______________________________________________ SciPy-Dev mailing list SciPy-Dev@scipy.org http://mail.scipy.org/mailman/listinfo/scipy-dev
participants (3)
-
Brian Lee Newsom
-
Nathan Woods
-
Robert Kern