[Python-Dev] No longer enable Py_TRACE_REFS by default in debug build

[Nathaniel Smith]

On Fri, Apr 12, 2019 at 5:05 PM Steve Dower <steve.dower at python.org> wrote:
>
> On 12Apr.2019 1643, Nathaniel Smith wrote:
> > On Thu, Apr 11, 2019 at 8:26 AM Steve Dower <steve.dower at python.org> wrote:
> >>
> >> On 10Apr2019 1917, Nathaniel Smith wrote:
> > I don't know how many people use Py_TRACE_REFS, but if we can't find
> > anyone on python-dev who uses it then it must be pretty rare. If
> > dropping Py_TRACE_REFS would let us converge the ABIs and get rid of
> > all the stuff above, then that seems like a pretty good trade! But
> > maybe the Windows C runtime issue will foil this...
>
> The very first question I asked was whether this would let us converge
> the ABIs, and the answer was "no".
>
> Otherwise I'd have said go for it, despite the C runtime issues.

I don't see that in the thread... just Victor saying he isn't sure
whether there might be other ABI incompatibilities lurking that he
hasn't found yet. Did I miss something?

I'm mostly interested in this because of the possibility of converging
the ABIs. If you think that the C runtime thing isn't a blocker for
that, then that's useful information. Though obviously we still need
to figure out whether there are any other blockers :-).

> >>>> The reason we ship debug Python binaries is because debug builds use a
> >>>> different C Runtime, so if you do a debug build of an extension module
> >>>> you're working on it won't actually work with a non-debug build of CPython.
> >>>
> >>> ...But this is an important point. I'd forgotten that MSVC has a habit
> >>> of changing the entire C runtime when you turn on the compiler's
> >>> debugging mode.
> >>
> >> Technically they are separate options, but most project files are
> >> configured such that *their* Debug/Release switch affects both the
> >> compiler options (optimization) and the linker options (C runtime linkage).
> >
> > So how do other projects handle this? I guess historically the main
> > target audience for Visual Studio was folks building monolithic apps,
> > where you can just rebuild everything with whatever options you want,
> > and compared to that Python extensions are messier. But Python isn't
> > the only project in this boat. Do ruby, nodejs, R, etc., all provide
> > separate debug builds with incompatible ABIs on Windows, and propagate
> > that information throughout their module/package ecosystem?
>
> Mostly I hear complaints about those languages *not* providing any help
> here. Python is renowned for having significantly better Windows support
> than any of them, so they're the wrong comparison to make in my opinion.
> Arguing that we should regress because other languages haven't caught up
> to us yet makes no sense.
>
> The tools that are better than Python typically don't ship debug builds
> either, unless you specifically request them. But they also don't leak
> their implementation details all over the place. If we had a better C
> API, we wouldn't have users who needed to match ABIs.

Do you happen to have a list of places where the C API leaks details
of the underlying CRT?

(I'm mostly curious because whenever I've looked my conclusion was
essentially: "Well....... I don't see any places that are *definitely*
broken, so maybe mixing CRTs is fine? but I have zero confidence that
I caught everything, so probably better to play it safe?". At least on
py3 – I know the py2 C API was definitely broken if you mixed CRTs,
because of the exposed FILE*.)

> For the most part, disabling optimizations in your own extension but
> using the non-debug ABI is sufficient, and if you're having to deal with
> other people's packages then maybe you don't have any choice (though I
> do know of people who have built debug versions of numpy before - turns
> out Windows developers are often just as capable as non-Windows
> developers when it comes to building things ;)

I'm not sure why you think I was implying otherwise? I'm sorry if you
thought I was attacking your users or something. I did say that I
thought most users downloading the debug builds were probably confused
about what they were actually getting, but I didn't mean because they
were stupid Windows users, I meant because the debug builds are so
confusing that even folks on the Python core team are confused about
what they're actually getting.

-n

-- 
Nathaniel J. Smith -- https://vorpus.org