[Python-bugs-list] [ python-Bugs-478339 ] Linking/compiling with DEBUG

[noreply@sourceforge.net]

Bugs item #478339, was opened at 2001-11-05 07:45
You can respond by visiting: 
http://sourceforge.net/tracker/?func=detail&atid=105470&aid=478339&group_id=5470

Category: Windows
Group: Python 2.1.1
Status: Open
Resolution: None
Priority: 5
Submitted By: Nobody/Anonymous (nobody)
Assigned to: Mark Hammond (mhammond)
Summary: Linking/compiling with DEBUG

Initial Comment:
I have an embedded use of the Python interpreter. I
want to compile and link the encapsulating application with
and without DEBUG. I have disabled the use of the
pragma in config.h to always use the library specified
in the link statement, which is always the standard
python library. I have ensured that all invocations of
Python.h undefine the DEBUG symbol before including them.

Compiling the application without debug: works fine if
you link non-debug. However, linking with debug causes
some change in the way python modules are looked up and
I'm getting undefined local modules. I have checked the
PYTHONPATH environment variable and it is correct, and
the .py module exists in the correct directory. (This
is a common scenario where some libraries are compiled
without DEBUG, some with DEBUG, and then the whole
application linked DEBUG.)

Compiling the application with debug: Link errors
result. Undefined -- _Py_RefTotal, _Py_Dealloc,
_PyInitModule4TraceRefs.

Python is making too many hidden assumptions about the
build environment when it is being used embedded. This
is a critical problem as it makes it impossible to
build the encapsulating application for debug purposes.

----------------------------------------------------------------------

>Comment By: Martin v. Löwis (loewis)
Date: 2001-11-08 15:28

Message:
Logged In: YES 
user_id=21627

AFAICT, PYTHONPATH is *not* ignored when running a debug
build. If you think it is, please run the debug and the
nodebug interpreter (i.e. python.exe and python_d.exe), and
report the values of sys.path for either case.

More likely, the cause of your problems is that the import
code looks for different DLL name *in sys.path*. I.e. when
you do 
import foo
in a nodebug build, it will try to find foo.pyd or foo.dll.
In a debug build, it will try to find foo_d.pyd or
foo_d.dll. So if you only have foo.pyd, then importing foo
will fail if Python was compiled with _DEBUG. 

Again, this is by design: foo.pyd might crash the debug
interpreter, since the object layout is different.

If you merely want to single-step through the Python
runtime, without making use of the Py_DEBUG features, you
should build the release version of Python, but activate the
generation of debug symbols.

----------------------------------------------------------------------

Comment By: Nobody/Anonymous (nobody)
Date: 2001-11-08 12:03

Message:
Logged In: NO 

You'll have to excuse the editorializing: I've found working
with the C API immensely frustrating in a variety of ways,
and then to have my empirical observations dismissed...

Let me step back and explain where I am right now.

I've built the 2.1 debug version of the python library using
the source distribution. When doing a final link without
debug (regardless of how all my client code was compiled)
I use the non-debug version of the Python library. When
doing a final link with debug (again, regardless of how all
of my client code is compiled) I use the debug version of
the python library. To get the latter scenario to work, I
edit the registry -- as well as the PYTHONPATH -- before
initializing python through the C API. For these two
cases, my unit tests run correctly to completion.

However, this means that our developers must compile the
library I wrote that wraps up the Python C API to match
how they are going to link it. I would like to find a way
around this constraint so that any combination of compiling
and linking my wrapper library works correctly.

(Let me also mention that I am aware that you are all
volunteers and I do appreciate your efforts. I am hopeful
that by resolving issues such as this Python can be made
more easy to use in a wider variety of circumstances.)

Thanks.

----------------------------------------------------------------------

Comment By: Tim Peters (tim_one)
Date: 2001-11-08 11:49

Message:
Logged In: YES 
user_id=31435

Anonymous, editorializing doesn't really help.  I 
understand that you want Python to work in a way other than 
it does, and repeating that isn't necessary.  I'm hoping 
Mark Hammond will comment on that.

In the meantime, I'm trying to determine why you're seeing 
the symptoms you are seeing.  I believe you're seeing them, 
but your analysis doesn't make sense, therefore something 
is going on you haven't told us about.  What *should* 
happen in your ideal world isn't relevant to determining 
what's actually happening.  Take this one step at a time, 
please.

----------------------------------------------------------------------

Comment By: Nobody/Anonymous (nobody)
Date: 2001-11-08 11:20

Message:
Logged In: NO 

It's not a question of *believing* that PYTHONPATH is
ignored: that is the observed behavior! (I am using
2.1.1.)

Whether I compile my client code with debug or not should
have nothing to do with what Python library I include.
(Note that I'm talking about *my* client code, not
compiling Python itself.) The Python #includes should not
be selecting a library for me, but should let me specify it
in my build environment. This is the way virtually every
3rd-party library is set up for both Windows and *nix
systems and the way I expected Python to behave. Whether
or not Windows defines _DEBUG shouldn't matter either --
I should have independent control. (I even when to the
lengths of undeffing DEBUG, _DEBUG, and Py_Debug prior to
including any Python headers -- still didn't help.)

Keep in mind that Python is not the core of my application.
It doesn't control the main loop. In fact I entirely
encapsulate the C API to Python so that my clients can
define commands without ever having to include Python.h
themselves. Python is just one piece of a very large system
(and a replacable piece at that) and can't dictate build
or compile requirements.

I'll check the system libraries, but I'm pretty sure they
are the multithreaded versions. (I think I'd be seeing
different symptoms if I had the wrong system libraries.)
Unfortunately, I can't send you samples due to
confidentiality concerns. Rest assured, however, that
I am describing the problems accurately.

Try it yourself! Uninstall python, just retaining the
normal link libraries and dlls and the headers. Now build
an application using just the C API. Not as simple as it
sounds (or is with competing tools such as Tcl).


----------------------------------------------------------------------

Comment By: Nobody/Anonymous (nobody)
Date: 2001-11-08 11:19

Message:
Logged In: NO 

It's not a question of *believing* that PYTHONPATH is
ignored: that is the observed behavior! (I am using
2.1.1.)

Whether I compile my client code with debug or not should
have nothing to do with what Python library I include.
(Note that I'm talking about *my* client code, not
compiling Python itself.) The Python #includes should not
be selecting a library for me, but should let me specify it
in my build environment. This is the way virtually every
3rd-party library is set up for both Windows and *nix
systems and the way I expected Python to behave. Whether
or not Windows defines _DEBUG shouldn't matter either --
I should have independent control. (I even when to the
lengths of undeffing DEBUG, _DEBUG, and Py_Debug prior to
including any Python headers -- still didn't help.)

Keep in mind that Python is not the core of my application.
It doesn't control the main loop. In fact I entirely
encapsulate the C API to Python so that my clients can
define commands without ever having to include Python.h
themselves. Python is just one piece of a very large system
(and a replacable piece at that) and can't dictate build
or compile requirements.

I'll check the system libraries, but I'm pretty sure they
are the multithreaded versions. (I think I'd be seeing
different symptoms if I had the wrong system libraries.)
Unfortunately, I can't send you samples due to
confidentiality concerns. Rest assured, however, that
I am describing the problems accurately.

Try it yourself! Uninstall python, just retaining the
normal link libraries and dlls and the headers. Now build
an application using just the C API. Not as simple as it
sounds (or is with competing tools such as Tcl).


----------------------------------------------------------------------

Comment By: Tim Peters (tim_one)
Date: 2001-11-08 09:48

Message:
Logged In: YES 
user_id=31435

Well, Martin is right, you know:  if you're not defining 
_DEBUG, and not letting MSVC define it either, then neither 
will pyconfig.h define Py_DEBUG, and it gets harder and 
harder to believe that what you're reporting is the full 
truth.

Perhaps you could attach a tiny example, along with the 
MSVC project file, or Makefile, you use.  It may well be 
academic, though, as it's *normal* for _DEBUG to get 
defined under MSVC in a debug build (and whether or not you 
add it explicitly -- as Martin implied, you would have had 
to explicitly remove _DEBUG from the MSVC auto-generated 
list of preprocesser debug-build #defines).

In answer to one of your earlier questions, PYTHONPATH is 
not ignored after a debug link.  I understand that you 
believe it is, but I expect that's another illusion due to 
something you haven't told us-- or not discovered --yet.  
Perhaps you're linking with the wrong system libraries?  
Python requires the multithreaded libraries.

----------------------------------------------------------------------

Comment By: Martin v. Löwis (loewis)
Date: 2001-11-08 09:33

Message:
Logged In: YES 
user_id=21627

What Python version are you using? You mentioned that you've
changed the pragma in config.h; atleast since Python 1.5.2,
this macro depends on _DEBUG, not on DEBUG. So if you didn't
define _DEBUG, you wouldn't need to change the pragma.Python
never changes its behaviour based on whether DEBUG is
defined, AFAICT.

Please check the project settings for the debug target in
your application again.

----------------------------------------------------------------------

Comment By: Nobody/Anonymous (nobody)
Date: 2001-11-08 09:22

Message:
Logged In: NO 

The only debug-type symbol I am defining when compiling my
code is DEBUG. I'm not deliberately defining _DEBUG or
Py_Debug.

----------------------------------------------------------------------

Comment By: Martin v. Löwis (loewis)
Date: 2001-11-08 09:09

Message:
Logged In: YES 
user_id=21627

AFAICT, even on Windows, defining DEBUG does not change
anything whatsoever. It is only when you define _DEBUG that
things start to change.

I'm not sure whether this is a bug or a feature: If you just
want to enable debugging in your application, just don't
define _DEBUG. Notice that MSVC does that for you unless you
tell it not to. AFAICT, _DEBUG is what Microsoft uses in the
libraries to indicate "enable all debugging code that you
have". So Python enables Py_DEBUG.

----------------------------------------------------------------------

Comment By: Tim Peters (tim_one)
Date: 2001-11-08 08:43

Message:
Logged In: YES 
user_id=31435

Changed Category to Windows, since I'm almost certain 
that's the missing info here.

Assigned to MarkH hoping he has a slick <wink> reply.  This 
kind of complaint comes up rarely but regularly, and I'm 
never sure what to say:  since I build Python from source, 
and always did, our Windows debug scheme is fine by me.  
But what are pure end-users expected to do?

----------------------------------------------------------------------

Comment By: Nobody/Anonymous (nobody)
Date: 2001-11-08 06:10

Message:
Logged In: NO 

I'm not recompiling the Python libraries. I am only
recompiling my personal libraries which use the Python
C API. I want to be able to compile my libraries with
debug and use the standard Python library.

This is not a "flawed" approach. It is extremely common
when using 3rd-party libraries to do this. Python should
be using a separate #define for its ref-counting debug
stuff so that clients can use the standard library and
still debug their own stuff.

Refer to my second comment regarding the use of PYTHONPATH
for another example of non-transparent substitution of
the debug and non-debug libraries.


----------------------------------------------------------------------

Comment By: Martin v. Löwis (loewis)
Date: 2001-11-07 14:11

Message:
Logged In: YES 
user_id=21627

I cannot understand what bug you are reporting. Of the
things you report, which of them did you not expect?

For example, that Py_RefTotal is undefined when you compile
with DEBUG, but link with the no-debug library is not a bug,
but by design: those functions are used in every debug
module, but available only in the debug library.

Your complete compilation approach is flawed: If you compile
with pydebug, you absolutely *must* link with the debug
library: the object layout of every object will change, so
crashes are likely if you link with the wrong library.

----------------------------------------------------------------------

Comment By: Nobody/Anonymous (nobody)
Date: 2001-11-05 09:29

Message:
Logged In: NO 

More info: I've now narrowed it down to the fact that when
the final link is done with debug, the PYTHONPATH
environment variable is ignored altogether. In my case, I
cannot rely on the registry being set up for Python, and the
system is resorting to the default relative path names
(which, in my case, are of no use).

How does the behavior of the (non-debug) python library know
to change based on the final link? And why is PYTHONPATH
ignored if it is linked in debug?

----------------------------------------------------------------------

You can respond by visiting: 
http://sourceforge.net/tracker/?func=detail&atid=105470&aid=478339&group_id=5470