On 10/4/2018 10:30 AM, INADA Naoki wrote:
> Hello,
>
> First of all, congratulations on passing all test on AIX.
>
>> My assumption is that it needs to (at least) pass all tests - and that
>> is why I keep asking for attention. All the PRs to fix individual tests
>> mean less if they are not merged, for whatever reason.
>>
>> However, maybe there is another way, or even something additional
>> needed. Maybe something I cannot provide and then I can adjust my
>> expectations and goals.
> As a one of core developer, I don't know anything about AIX.
> If my change breaks AIX build, I can't investigate what's happened.
>
> So I think we need following in devguide:
>
> * Brief description about AIX, from developer's point of view.
This I might be able to do. Bullet form:
* Committed to POSIX standard (valid when release came out, so AIX 5.3
confirms to a different standard than AIX 7.2)
* While Linux affinity is recognized - GNU (or GNP - GNU not POSIX)
integration is not guaranteed. - GNU rte is not provided under support.
There is a so-called Toolbox, GNU an other OSS utilities supplied by
many packaged as RPMs. Unfortunately, different RPM providers (Michael
Perlz, BULL Freeware, IBM, and others) have different numbering (the
part after the package version, e.g., python-2.7.10-XXX so they do not
mix well). Headache for both admins and developers trying to develop in
a GNU-like environment.
* As a consultant, fedup with what is called the "RPM hell" by many AIX
admins - I do not use any RPMs. I build everything myself, using xlc
(gcc introduces the need for a GNU RTE, e.g., glibc). I package using
installp (the "native") AIX package manager, and strive to make the
packages independent (these days). When there are dependencies I try to
build them as static libraries so that they do not become an additional
install dependency.
* finally, a bit deeper: while the AIX linker loader supports svr4
shared libraries (it is the data, not the file name) it also supports
having multiple shared libraries in a classic archive. So, rather that
.../lib/libxxx.so and .../lib64/libxxx.so AIX prefers .../lib/libxxx.a
with two so-called members, with same or different names. The one found
is not it's name, but the symbol name and size of the ABI (32-bit or 64-bit)
* Hope that is enough of the key issues for now.
** In general, GNU autotools (autoreconf -f -v) works well, as does
configure.ac et al. for createing OSS Makefiles.
> * How to run AIX on (VirtualBox, AWS EC2, Azure, GCP) easily.
Easily! ? :) - well, on a POWER server it was easy enough for me to
follow the step by step instructions for creating a buildbot. If I had a
POWER server with more resources I would examine using the AIX internal
WPAR - however, a POWER server configured to use PowerVC uses "EC2" aka
cloud-init for creating a virtual machine. With that environment it
should be "easy" to provide additional instructions to cloud-init-ec2.

Or, I provide you a login on my personal server that I run the buildbot
on. etc. etc. - Where there is a will, there is a way.
> * How to set up a development environment for Python.
Again, follow the instructions for setting up a buildbot.
> * How to build Python.
git clone ...
autoreconf -v -f (as needed)
./configure --with-pydebug  #gcc compiler
./configure --with-pydebug --without-computed-gotos # xlc compiler
make
make test
> * How to debug C code.
I learned, 40 years ago, using adb (a debugger) - I do a lot of
single-stepping. gdb is not the default debugger. If I were a developer
I would probably dig into the AIX debuggers (there are at least two, kdb
(kernel debugger, which I do use occaisionally for performance issues)
and another I try to avoid. I add fprintf statements and am looking at
learning how to use probevue.

In short, you probably have many much better ideas on how to debug C
than I do :)
>
> And even though there is a developer guide, it will take more long time
> than fixing issues on AIX, compared Linux, macOS, and Windows.
>
> But without this guide, it feels almost impossible to maintain AIX build to me.
IMHO: The AIX build is stable, but this is unrecognized because it does
have differences that cause tests to fail. I can think of one test that
PASSes, but should fail. And another test that passes, but should have
failed (in test_uuid) I have submitted a PR.

I tried to fix "all" in one PR, which confused people - so redid it as
two (got _uuid working in Python 3.7 ! yes!!) but the "original" to fix
uuid.py and test_uuid.py is still "awaiting change review".

My gut feeling to maintaining AIX is: a) all test pass so a potential
regression is flagged; b) someone such as myself who knows the platform
and can establish a "root cause" on why it is failing with AIX so that
c) a developer becomes aware and can decide to ignore or adjust; d)
alternatives - such as work around an implementation limitation (as I
try to do, e.g., for test_time and the related _datetime.c) is yet
another path.

In other words - it needs to be a shared responsibility - some
volunteers with a passion for platform stability (in this specific case
AIX and me as "passionate-person" and perhaps someone as youself who
wants to focus on the language itself - ideally without deep (or any!)
concern for platform differences.
>
> Regards,
My 6 bits!

On 05/10/2018 16:15, Michael Haubenwallner wrote:
> Hi Michael,
>
> being on a similar road with Gentoo Prefix, I really do appreciate
> your AIX related work!
>
> However, for two (not so minor) topics I've got a little different
> experience, which I think should be mentioned here for completion:
Always.
>
> On 10/04/2018 11:13 AM, Michael Felt wrote:
>> On 10/4/2018 10:30 AM, INADA Naoki wrote:
>>> Hello,
>>>
>>> First of all, congratulations on passing all test on AIX.
>>> As a one of core developer, I don't know anything about AIX.
>>> If my change breaks AIX build, I can't investigate what's happened.
>>>
>>> So I think we need following in devguide:
>>>
>>> * Brief description about AIX, from developer's point of view.
>> This I might be able to do. Bullet form:
>> ... I build everything myself, using xlc
>> (gcc introduces the need for a GNU RTE, e.g., glibc).
> Using gcc does *not* require to use glibc or even GNU binutils at all.
> Except for gcc's own runtime libraries, there's no need for a GNU RTE.
> In fact, in Gentoo Prefix I do use gcc as the compiler, configured to
> use AIX provided binutils (as, ld, nm, ...), with AIX libc as RTE.
Well, this is something I learned - second hand - from someone who
worked hard to make much more OSS available than I. Probably wrong then
- in how I came to my conclusion - but the few things I tried to compile
"asis" to shared libraries would not work without also a lot of the gcc
compiler libraries. While I could have bitten the bullet and just found
a way to add those I was warned that different versions of gcc need
different level of supporting files.
>> * finally, a bit deeper: while the AIX linker loader supports svr4
>> shared libraries (it is the data, not the file name) it also supports
>> having multiple shared libraries in a classic archive. So, rather that
>> .../lib/libxxx.so and .../lib64/libxxx.so AIX prefers .../lib/libxxx.a
>> with two so-called members, with same or different names. The one found
>> is not it's name, but the symbol name and size of the ABI (32-bit or 64-bit)
> While this all is true, having multiple *versions* of one shared library in
> one single file is a PITA for package managers - both human or software.
Yes, it is a necessary pain. My secret is a) do not touch /usr/lib -
leave what is as it is, and in the few situations where it must be in
/usr/lib I add/replace named archive members with my new ones - and all
the other ones get extracted, modify a flag in their respective header -
so that the linker knows they are only to be used for applications that
expect them - not for new applications.
> But fortunately, the AIX linker does support so called "Import Files",
> allowing for *filename based* shared library versioning like on Linux,
> while still allowing for both ABIs in a single library archive file.
>
> For example, libtool provides the --with-aix-soname={aix|svr4|both}
> configure flag since libtool-2.4.4.  Although the default will stay
> at 'aix' here, in Gentoo Prefix I do use 'svr4' only.  This actually
> is a package manager's decision, ideally for all depending packages.
> As gcc does use libtool, for more information please refer to
> https://gcc.gnu.org/install/configure.html#WithAixSoname
> But note that "Import Files" should work with xlc as well.
Actually, more detail than I really want to know. I recall the day when
a library was a collection of "static" .o files, and ranlib was nearly
always needed - Or you told the linker to link against that library
multiple times. And I recall going to my first conference where "RPC"
was the next greatest thing, and shared libraries were going to be such
a space savior - both on disk and in memory. And I was always more of a
bsd fan having schooled myself on UNIX v7, then bsd (2.9 iirc) and bsd
4.1, 4.2 (in comes tcpip) and 4.3. But I diverge :p

To return briefly to the question of what is AIX for the developer -
very flexible. You can choose your architecture and it generally, just
works. I wrote some scripts as a front-end for packaging and most
packages are a one-liner - that runs configure, make, makeinstall (using
DESTDIR) and then packaging the DESTDIR.

As far as python development and AIX goes I am open to helping others.
Been doing that for more years than I care to count. :)
>
> Thanks!
> /haubi/