[Python-bugs-list] [ python-Bugs-680429 ] module broken for extension classes

Fri, 07 Feb 2003 03:29:57 -0800

Bugs item #680429, was opened at 2003-02-04 19:55
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=680429&group_id=5470

Category: Python Interpreter Core
Group: Python 2.3
Status: Open
Resolution: None
Priority: 5
Submitted By: Ralf W. Grosse-Kunstleve (rwgk)
Assigned to: Michael Hudson (mwh)
Summary: __module__ broken for extension classes

Initial Comment:
We are having problems using Boost.Python with Python 
2.3a1 because
under some circumstances the result of __module__ is 
different compared
to earlier Python versions:

Python 2.2.1 (#2, Jun 17 2002, 12:06:51) 
[GCC 2.96 20000731 (Red Hat Linux 7.3 2.96-110)] on 
linux2
Type "help", "copyright", "credits" or "license" for more 
information.
>>> import simple
>>> simple.empty.__module__
'simple'
>>> 

Python 2.3a1 (#1, Jan  6 2003, 14:17:56) 
[GCC 2.96 20000731 (Red Hat Linux 7.3 2.96-110)] on 
linux2
Type "help", "copyright", "credits" or "license" for more 
information.
>>> import simple
>>> simple.empty.__module__
'__main__'
>>> 

Because of this we can no longer pickle our extension 
classes.
For your reference the code for the simple module is 
attached.
This is using Boost release 1.29.0 (www.boost.org).

We have done some debugging. Boost.Python's internal 
idea of the module
associated with an extension class is still correct even when 
using
Python 2.3a1. David Abrahams (main Boost.Python author) 
is telling me
that he "changed Boost.Python to work the way Guido 
suggested before
2.2.2." Therefore we suspect that the __module__ problem 
is due to
a change/bug in Python 2.3a1.

Ralf

#include <boost/python/module.hpp>
#include <boost/python/class.hpp>

namespace sandbx { namespace {

  struct empty {};

  void init_module()
  {
    using namespace boost::python;
    class_<empty>("empty");
  }

}} // namespace sandbx::<anonymous>

BOOST_PYTHON_MODULE(simple)
{
  sandbx::init_module();
}

----------------------------------------------------------------------

>Comment By: Michael Hudson (mwh)
Date: 2003-02-07 11:29

Message:
Logged In: YES 
user_id=6656

Ah, I start to remember the details now.

In 2.2, if tp_name contained a period, everything to the
left of the period was __module__, that to the right __name__.

However, at some point (maybe even by 2.3) we want

class X:
    class Y:
        pass
print X.Y.__name__

to print 'X.Y' (it prints 'Y' today).  Also we made __name__
assignable for 2.3 and this behaviour caused confusion.

So we made things so that the '__module__' entry in the
type's dict always wins.

In summary, if you poke a '__module__' key into the
boost::python::dict object you pass to the call of the
metatype and change the first argument to just 'name', I
think you'll be set.

I pretty sure this will also work with 2.2.x, but if I were
you I'd check.

HTH!

----------------------------------------------------------------------

Comment By: Ralf W. Grosse-Kunstleve (rwgk)
Date: 2003-02-06 18:36

Message:
Logged In: YES 
user_id=71407

> In 2.2.x, how are you telling Python that __module__ should
> be "simple"?  By setting tp_name to "simple.empty"?  What's
> simple.__name__?

Python 2.3a1 (#2, Feb  5 2003, 09:39:30) 
[GCC 2.96 20000731 (Red Hat Linux 7.3 2.96-110)] on linux2
Type "help", "copyright", "credits" or "license" for more 
information.
>>> import sandbx.simple
>>> sandbx.simple.__name__
'sandbx.simple'
>>> sandbx.simple.empty.__name__
'sandbx.simple.empty'
>>> sandbx.simple.empty.__module__
IS HEAPTYPE
type->tp_dict, __module__ = __main__
'__main__'
>>> 

> I think I know how __module__ is getting there, it's the
> chunk of code currently at typeobject.c:1750.

Yes, that's right. I've verified this by adding more print statements.

> Can you point me to the bit of the boost source that creates
> the typeobject?

Here is where it happens (file 
boost/libs/python/src/object/class.cpp):

object result = object(class_metatype())(module_prefix() + name, 
bases, d);

With print statements I've verified that module_prefix() + name is
correctly passed in as "sandbx.simple.empty" .

I am attaching the file class.cpp. Could it be that we have to
update our class_metatype_object?

Thanks!
        Ralf

----------------------------------------------------------------------

Comment By: Michael Hudson (mwh)
Date: 2003-02-06 11:22

Message:
Logged In: YES 
user_id=6656

OK, thanks for the info.  More questions, I'm afraid:

In 2.2.x, how are you telling Python that __module__ should
be "simple"?  By setting tp_name to "simple.empty"?  What's
simple.__name__?

I think I know how __module__ is getting there, it's the
chunk of code currently at typeobject.c:1750.

Can you point me to the bit of the boost source that creates
the typeobject?  'cept sf's just fallen off the net.

----------------------------------------------------------------------

Comment By: Ralf W. Grosse-Kunstleve (rwgk)
Date: 2003-02-05 18:25

Message:
Logged In: YES 
user_id=71407

> Look at typeobject.c:type_module() (about 100 lines in).
>
> Is simple.empty a HEAPTYPE?

Yes, it is a HEAPTYPE. I've established this by adding print 
statements
in typeobject.c:type_module():

static PyObject *
type_module(PyTypeObject *type, void *context)
{
        PyObject *mod;
        char *s;

        if (type->tp_flags & Py_TPFLAGS_HEAPTYPE) {
printf("IS HEAPTYPE\n");
                mod = PyDict_GetItemString(type-
>tp_dict, "__module__");
                Py_XINCREF(mod);
                s = PyString_AsString(mod);
printf("type->tp_dict, __module__ = %s\n", s);
                return mod;
        }

Result:

Python 2.3a1 (#2, Feb  5 2003, 09:39:30) 
[GCC 2.96 20000731 (Red Hat Linux 7.3 2.96-110)] on linux2
Type "help", "copyright", "credits" or "license" for more 
information.
>>> import simple
>>> simple.empty.__module__
IS HEAPTYPE
type->tp_dict, __module__ = __main__
'__main__'
>>> 

> If you can tell me what you want that function to do, I can
> have a go at it.

We expect "simple" as the result of simple.empty.__module__.

Current result if simple.so is moved to a package:

Python 2.3a1 (#2, Feb  5 2003, 09:39:30) 
[GCC 2.96 20000731 (Red Hat Linux 7.3 2.96-110)] on linux2
Type "help", "copyright", "credits" or "license" for more 
information.
>>> from sandbx import simple
>>> simple.empty.__module__
IS HEAPTYPE
type->tp_dict, __module__ = __main__
'__main__'
>>> 

Here we expect "sandbx.simple" .
Python 2.2.x produces the expected results using the exact same
Boost.Python source code.

David Abrahams provides this additional information:

  Remember that simple.empty is created by calling the metatype, 
so
  it's created on the heap, but it's Python's internal type creation
  mechanisms which do it -- we're not setting the flags manually.

Thanks!
        Ralf

----------------------------------------------------------------------

Comment By: Michael Hudson (mwh)
Date: 2003-02-05 14:10

Message:
Logged In: YES 
user_id=6656

There has certainly been a change here.

Look at typeobject.c:type_module() (about 100 lines in).

If you can tell me what you want that function to do, I can
have a go at it.

Is simple.empty a HEAPTYPE?

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=680429&group_id=5470

[Python-bugs-list] [ python-Bugs-680429 ] __module__ broken for extension classes

[Python-bugs-list] [ python-Bugs-680429 ] module broken for extension classes