Pyste: feature requests
Hi, I've been trying to wrap a fairly large library of mine using Boost.Python. I've been using Pyste to generate the wrapper code. Since the library is large, I've been using Pyste's --multiple option. Something like this: pyste.py --module=test --out build --multiple file1.pyste file2.pyste I find a few inconveniences and difficulties with this approach and it would be nice to have the following features: 1. Something like the C compiler's '-c' option. In this case Pyste should simply generate the wrapper code and none of the module code i.e. for an invocation like so: pyste.py --out build --multiple -c file1.pyste nothing should be added to test.cpp and just a _file.cpp (or whatever) should be generated which can be then compiled. Then invoking pyste.py --module=test --out build/ --multiple file1.pyste \ file2.pyste should generate test.cpp with the appropriate code. This makes it much easier to handle dependencies. Perhaps the option should be called '-w' for 'just-wrap'? 2. Instead of generating files for each header, it would be useful if one file were generated per pyste file when --multiple were used. The trouble with the current approach of --multiple is that several files can be generated per pyste file. This causes problems with dependencies since one does not (easily) know apriori what files will be generated. For example changing -D options could change the number of files generated. Perhaps instead of changing the behaviour of the --multiple option one one could add another option called --one-file (with a better name). Alternatively if the argument to --out is a filename (file.cpp) that may or may not exist then that file is written, if the argument is a directory (which exists) then multiple files can be generated in that directory. I think these options or something along these lines would make life easier when wrapping larger libraries. What do you folks think? Thanks! cheers, prabhu
Hi Prabhu, Prabhu Ramachandran wrote:
Hi,
1. Something like the C compiler's '-c' option. In this case Pyste should simply generate the wrapper code and none of the module code i.e. for an invocation like so:
pyste.py --out build --multiple -c file1.pyste
nothing should be added to test.cpp and just a _file.cpp (or whatever) should be generated which can be then compiled. Then invoking
pyste.py --module=test --out build/ --multiple file1.pyste \ file2.pyste
should generate test.cpp with the appropriate code. This makes it much easier to handle dependencies. Perhaps the option should be called '-w' for 'just-wrap'?
I don't understand exactly what you are saying. Are you suggesting that instead of generating: #include <...> BOOST_PYTHON_MODULE(module) { class_<A>(...); } You want to be able to generate just this: class_<A>(...); ? How does that help you handle dependencies?
2. Instead of generating files for each header, it would be useful if one file were generated per pyste file when --multiple were used. The trouble with the current approach of --multiple is that several files can be generated per pyste file. This causes problems with dependencies since one does not (easily) know apriori what files will be generated. For example changing -D options could change the number of files generated. Perhaps instead of changing the behaviour of the --multiple option one one could add another option called --one-file (with a better name). Alternatively if the argument to --out is a filename (file.cpp) that may or may not exist then that file is written, if the argument is a directory (which exists) then multiple files can be generated in that directory.
Good idea! When I implemented the --multiple option, I didn't consider dependencies. I will put in my TODO list, shouldn't be too hard to implement this.
I think these options or something along these lines would make life easier when wrapping larger libraries.
What do you folks think?
Thanks a lot for your suggestions! Regards, Nicodemus.
"N" == nicodemus <nicodemus@globalite.com.br> writes:
>> 1. Something like the C compiler's '-c' option. In this case >> Pyste >> should simply generate the wrapper code and none of the module >> code i.e. for an invocation like so: >> >> pyste.py --out build --multiple -c file1.pyste N> I don't understand exactly what you are saying. Are you N> suggesting that instead of generating: N> #include <...> N> BOOST_PYTHON_MODULE(module) { N> class_<A>(...); N> } N> You want to be able to generate just this: N> class_<A>(...); N> ? N> How does that help you handle dependencies? No, what I mean is 'pyste.py --multiple -c file.pyste' should generate this (edited for brevity): // ------------ _file.cpp ---------- // Includes ===== #include <boost/python.hpp> // Using ======= using namespace boost::python; // Declarations ==== namespace { struct test_A_Wrapper: test::A { [...] }; [ other wrappers etc. ] // Module =========== void _Export_file_pyste() { class_<...> } // ------------ _file.cpp ---------- And without -c this command: pyste.py --module=test --multiple file1.pyste file2.pyste should *only* generate: // ---------- test.cpp ---------- // Include ============== #include <boost/python.hpp> // Exports =============== void _Export_file_pyste(); // Module ================ BOOST_PYTHON_MODULE(test) { _Export_file_pyste(); } // ---------- test.cpp ---------- I hope this is clear. I think you will see what I am getting at here. This way if file.pyste changes then only _file.cpp changes. None of the other wrapper code sources will change and pyste will run faster since all you need to process is one of the pyste files. Its much easier for users to use and incrementally wrap their libraries and very convenient for development. >> 2. Instead of generating files for each header, it would be >> useful if >> one file were generated per pyste file when --multiple were N> Good idea! When I implemented the --multiple option, I didn't N> consider dependencies. I will put in my TODO list, shouldn't be N> too hard to implement this. That would be great! Thanks! N> Thanks a lot for your suggestions! My pleasure! cheers, prabhu
Hi Prabhu, Prabhu Ramachandran wrote:
"N" == nicodemus <nicodemus@globalite.com.br> writes:
1. Something like the C compiler's '-c' option. In this case Pyste should simply generate the wrapper code and none of the module code i.e. for an invocation like so:
pyste.py --out build --multiple -c file1.pyste
N> I don't understand exactly what you are saying. Are you N> suggesting that instead of generating:
N> #include <...> N> BOOST_PYTHON_MODULE(module) { N> class_<A>(...); N> }
N> You want to be able to generate just this: N> class_<A>(...); N> ?
N> How does that help you handle dependencies?
No, what I mean is 'pyste.py --multiple -c file.pyste' should generate this (edited for brevity):
// ------------ _file.cpp ---------- // Includes ===== #include <boost/python.hpp> // Using ======= using namespace boost::python; // Declarations ==== namespace { struct test_A_Wrapper: test::A { [...] }; [ other wrappers etc. ] // Module =========== void _Export_file_pyste() { class_<...> } // ------------ _file.cpp ----------
And without -c this command: pyste.py --module=test --multiple file1.pyste file2.pyste should *only* generate:
// ---------- test.cpp ---------- // Include ============== #include <boost/python.hpp> // Exports =============== void _Export_file_pyste();
// Module ================ BOOST_PYTHON_MODULE(test) { _Export_file_pyste(); } // ---------- test.cpp ----------
I hope this is clear. I think you will see what I am getting at here. This way if file.pyste changes then only _file.cpp changes. None of the other wrapper code sources will change and pyste will run faster since all you need to process is one of the pyste files. Its much easier for users to use and incrementally wrap their libraries and very convenient for development.
I see. Unfortunately this breaks backward compability. 8/ But notice that Pyste only overwrites files if they changed, otherwise it leaves them intact, so as to not disturb build systems out there that depend on time-stamps. I think a good option would be something like --xml-cache=<dir>, where Pyste would get the generated gccxml files, or write them there in case they are not found. The bottleneck with Pyste is always GCCXML, since it has to basically compile the source files. What do you think? Regards, Nicodemus.
Hi Nicodemus,
"N" == nicodemus <nicodemus@globalite.com.br> writes:
>> >> 1. Something like the C compiler's '-c' option. In this case >> >> Pyste >> >> should simply generate the wrapper code and none of the >> >> module code i.e. for an invocation like so: N> I see. Unfortunately this breaks backward compability. 8/ But N> notice that Pyste only overwrites files if they changed, N> otherwise it leaves them intact, so as to not disturb build N> systems out there that depend on time-stamps. Yes, I noticed the SmartFile while poking around. Neat. If backwards compatibility is the problem you could always define a new option that does not interfere with the current working. Something like this: --only-wrapper for individual files with only the export functions and without generating the module code. --only-main for the main module.cpp file that calls the exports and initializes the module. N> I think a good option would be something like N> --xml-cache=<dir>, where Pyste would get the generated gccxml N> files, or write them there in case they are not found. The N> bottleneck with Pyste is always GCCXML, since it has to N> basically compile the source files. What do you think? Yes, that also sounds OK to me. If the wrapper-generation time will drop and not every single file will be re-parsed/compiled and wrapped then I'm OK with a --xml-cache option. Basically, the trouble is that with many .pyste files its a pain changing just one Pyste file and waiting for a long while for pyste (gccxml actually) to finish. The other desireable feature would be to dump only one file per .pyste file. This makes dependencies with something like SCons a non-issue. If not its hard to figure out which files depend on what. Again if backwards compatibility is a problem perhaps another option --one-file needs to be added. Thanks! cheers, prabhu
Hi Prabhu, Prabhu Ramachandran wrote:
Hi Nicodemus,
N> I see. Unfortunately this breaks backward compability. 8/ But N> notice that Pyste only overwrites files if they changed, N> otherwise it leaves them intact, so as to not disturb build N> systems out there that depend on time-stamps.
Yes, I noticed the SmartFile while poking around. Neat.
If backwards compatibility is the problem you could always define a new option that does not interfere with the current working. Something like this:
--only-wrapper for individual files with only the export functions and without generating the module code.
--only-main for the main module.cpp file that calls the exports and initializes the module.
Yeah, it is an option too.
N> I think a good option would be something like N> --xml-cache=<dir>, where Pyste would get the generated gccxml N> files, or write them there in case they are not found. The N> bottleneck with Pyste is always GCCXML, since it has to N> basically compile the source files. What do you think?
Yes, that also sounds OK to me. If the wrapper-generation time will drop and not every single file will be re-parsed/compiled and wrapped then I'm OK with a --xml-cache option.
Great, I will implement that then.
Basically, the trouble is that with many .pyste files its a pain changing just one Pyste file and waiting for a long while for pyste (gccxml actually) to finish.
I understand.
The other desireable feature would be to dump only one file per .pyste file. This makes dependencies with something like SCons a non-issue. If not its hard to figure out which files depend on what. Again if backwards compatibility is a problem perhaps another option --one-file needs to be added.
I am working on this right now. 8) Regards, Nicodemus.
Nicodemus wrote:
Hi Prabhu,
Prabhu Ramachandran wrote:
The other desireable feature would be to dump only one file per .pyste file. This makes dependencies with something like SCons a non-issue. If not its hard to figure out which files depend on what. Again if backwards compatibility is a problem perhaps another option --one-file needs to be added.
I am working on this right now. 8)
Ok, it is in CVS now. If you find any bug, please report that I will try to fix it ASAP. Alas, I was planning to work more on Pyste tomorrow (specially fixing the staticmethod bug and the virtual operators), bug I just remembered that I have a final monday. Sorry everyone that was waiting for this fixes, I will only be able to work on them next week. Regards, Nicodemus.
"N" == nicodemus <nicodemus@globalite.com.br> writes:
>>> The other desireable feature would be to dump only one file >>> per .pyste file. This makes dependencies with something like >>> SCons a non-issue. If not its hard to figure out which files >>> depend on what. Again if backwards compatibility is a problem >>> perhaps another option --one-file needs to be added. >>> >> I am working on this right now. 8) N> Ok, it is in CVS now. If you find any bug, please report that I N> will try to fix it ASAP. Alas, I was planning to work more on Wow! That was fast. Your changes haven't propagated to the backup servers yet but I'll be able to check it out tommorow. N> Pyste tomorrow (specially fixing the staticmethod bug and the N> virtual operators), bug I just remembered that I have a final N> monday. Sorry everyone that was waiting for this fixes, I will N> only be able to work on them next week. No problem, good luck on the exam! Thanks again for taking the time to do all this in record time. Much appreciated. cheers, prabhu
participants (2)
-
Nicodemus -
Prabhu Ramachandran