[C++-sig] Re: Presenting a Python embedding tutorial for Boost.Python

[Dirk Gerrits]

David Abrahams wrote:
> Dirk Gerrits <dirk at gerrits.homeip.net> writes:
> 
> 
>>David Abrahams wrote:
> 
> 
>>>I also confess to not having any idea what piece of code you're
>>>talking about ;-)
>>
>>The two code examples in the manual reference counting section. ;)
> 
> 
> Ah. It's really good to have that section somewhere in the tutorial or
> maybe in the rationale documentation, to show why you want a C++
> wrapper over the 'C' API.

Then I shall compress it so that it can become a sidebar or a part of 
the rationale. With the upcoming embedding features of Boost.Python I 
would really prefer not to document both the 'C'-way and the BPL-way of 
doing everything. ;)

>>>See also http://www.mcmillan-inc.com/embed.html, FWIW.
>>
>>Thanks for the link. Taught me two things:
>>1) Any sane person will prefer Boost.Python over the Python C API ;)
>>2) It is possible to create multiple interpreters, even in different
>>threads. This could lead to a very cool python::interpreter class
>>indeed. :)
> 
> 
> That's a neat idea. I await your specific interface suggestions.

Well nothing concrete so far but I think you'll agree that the 
constructor and destructor are the most vital parts of the class. They 
do practically all the work of setting up/release (sub-)interpreters.

Furthermore, there should be a way to set/query the currently active 
interpreter. Member functions like this:

     bool is_current() const;
     void make_current();

were the first things that came to mind.

I also thought about a free function along the lines of:

     interpreter* current_interpreter();

But that would only make sense if the public class interface is 
expanded. After all, is_current() on the current_interpreter() is always 
true and make_current() is always a no-op, right? (Also, it would 
probably not work too well with threading maybe?)

A pseudo-implementation of the constructor and destructor could look like:

     interpreter()
     {
         // Create the main interpreter
         if (interpreter_count == 0)
         {
             Py_Initialize();
             ...
         }
         // Create a sub-interpreter
         else
         {
             ...
         }
         interpreter_count++;
     }
     ~interpreter()
     {
         // Release the main interpreter
         if (interpreter_count == 1)
         {
             Py_Finalize();
             ...
         }
         // Release a sub-interpreter
         else
         {
             ...
         }
         interpreter_count--;
     }

... treating the main interpreter and sub-interpreters equally as 
python::interpreter instances. I'm not sure if this is the best solution 
though, because there are some (very) subtle differences between the 
main interpreter and sub-interpreters. 
(http://www.python.org/doc/current/api/initialization.html#l2h-657)

Not making the distinction seems more natural though. ("Beautiful is 
better than ugly." "Simple is better than complex." ;))

>>>>However, adding Boost.Python facilities for the PyRun_ functions
>>>>probably means adding a load of other Python C API facilites as
>>>>well.
>>
>>>Yup. It was always the intention to add these. Dave Hawkes made a
>>>valiant effort:
>>>http://mail.python.org/pipermail/c++-sig/2002-June/001503.html, but it
>>>was in some ways just too much to take into the library at once.
>>
>>I see. Pity. Let's see how the incremental approach will fare.
> 
> 
> 
> OK! 
> 
> For reference you might want to look at Dave's work. I've attached
> a CVS log | cvslogfilter -b of his checkins at the bottom of the
> message.

Thanks. Downloaded the files to my harddrive for future reference. :)

>>>>I would like to have something like this in Boost.Python:
>>>>
>>>>object run_python_code(InputIterator begin, InputIterator end);
>>>
>>>These must have char as their value_type?
>>
>>I suppose so. At least I don't see any unicode/wchar_t in the Python C
>>API. ;) I was under the impression that Python supported unicode
>>source files though. How does that work then?
> 
> 
> Search me!

Searched for unicode (7 hits), wchar_t (no hits) and utf-8 (1 hit) on 
the C++ Sig archive on ASPN but couldn't find it, sorry.

>>Also, while this interface is nicely STL-ish, it is probably not very
>>efficient. I don't see how this can be implemented without some
>>std::copy to std::vector<char> or std::string. Thoughts?
> 
> 
> I don't know. Based on your remarks I'm guessing that the interface is
> a case of premature generalization. Suppose we just do this:
> 
>     object run_python_code(object x);
> 
>     template <class T>
>     object run_python_code(T const& x)
>     { 
>        return run_python_code(object(x));
>     }
> 
> ??
> 
> Now anything that can be converted to a Python string will work. Just
> a thought.

FWIW, I like it. :)

I think though that running code from a file would be at least as 
important as running it from a string. So the implementation of 
run_python_code (name might need work?) should IMHO check to see if x is 
(convertable to) a Python file. If so, then it should use PyRun_File 
with a FILE*, otherwise it should use PyRun_String with a char const*.

>>>>Also, it might be a good idea to wrap Py_Initialize and Py_Finalize in
>>>>a RAII class?
>>
>>>Absolutely. Just as soon as Py_Finalize becomes safe for us ;-)
>>>That might be a good project for you...
>>
>>I've been meaning to look into BPL's internals in more detail
>>anyway. :) I'm not sure if university will leave me with enough spare
>>time though. I think I should concentrate on the tutorial first.
> 
> 
> OK. BTW, this is not a hard problem to solve and it doesn't require a
> deep understanding of the internals. The first step is to identify all
> the places where static/global object or handle<> instances are
> created in the library or user code. There is also some global data in
> the registry. Then we'll need to set up some code for releasing the
> resources with Python's atexit() module.

Doesn't sound too hard. I'll get on it when I see the time (probably in 
the weekend). Perhaps somebody reading this thread will beat me to it 
though. ;)

Dirk Gerrits