[Wheel-builders] Packaging a library that uses PyGObject?

[Geoffrey Thomas]

On Sun, 23 Feb 2020, Daniel Alley wrote:

> I would like to package this library as a pre-built Python wheel: https://github.com/fedora-modularity/libmodulemd
> 
> This library uses PyGObject, so importing it looks like this:
> 
> import gi
> gi.require_version("Modulemd", "2.0")
> from gi.repository import Modulemd
> 
>  I can't find any examples of this, nor any documentation, nor discussion about it.  It seems like it would be, at minimum, a bit more complicated than libraries based on normal bindings.  Is this
> possible, and are there any special requirements that are needed to do so? 

It's been a bit since I've used gobject-introspection, but I _think_ the 
way this works for a normal (OS-installed) GObject package is

a) the package ordinarily provides no actual Python library
b) the package provides a girepository-1.0/Modulemd-2.0.typelib file in 
the system lib directory
c) when you import it, PyGObject loads the C libmodulemd library and 
generates Python bindings based on the typelib file
d) as a special case, a package _can_ provide a Python "override" library, 
but that amends the autogenerated bindings, it's not a complete set of 
bindings on its own (and it's Python code, not native code). Modulemd 
appears to do this.

So your users' code needs to be able to
- import PyGObject (and libgobject) itself, either from the system or from 
their virtualenv
- import the C libmodulemd library from your wheel, which you can compile 
for manylinux1
- find the typelib file, which you can put in your wheel
- point libgobject at the typelib file and the C library
- point PyGObject at the override file (which should hopefully be 
automatic if it's on sys.path)

The GObject docs https://developer.gnome.org/gi/stable/GIRepository.html 
say:

> GIRepository will typically look for a girepository-1.0 directory under 
> the library directory used when compiling gobject-introspection.
>
> It is possible to control the search paths programmatically, using 
> g_irepository_prepend_search_path(). It is also possible to modify the 
> search paths by using the GI_TYPELIB_PATH environment variable. The 
> environment variable takes precedence over the default search path and 
> the g_irepository_prepend_search_path() calls.

You will also need to make sure the C library is importable. For "normal" 
Python wheels, users import a compiled Python shared object (so that the 
usual Python path is used as the search path), and that shared object has 
a normal shared object dependency on the underlying C library which is 
also shipped in the wheel.auditwheel sets an $ORIGIN-relative rpath in 
that .so file (using patchelf) so that the Python module, having been 
found inside the virtualenv, can find its C library in a relative path to 
its own location. Since there is no compiled Python module in your case, 
because PyGObject is dynamically generating the bindings at runtime, I 
don't think there is a straightforward way of informing PyGObject of where 
to find the C library.

Personally, I'd approach this by first aiming for an 80% solution where I 
expect users to set GI_TYPELIB_PATH and LD_LIBRARY_PATH so that the 
typelib file and the C library can both be found, i.e., they use it by 
running something to the effect of

os.setenv("GI_TYPELIB_PATH", "myvenv/lib/girepository-1.0")
os.setenv("LD_LIBRARY_PATH", "myvenv/lib")
gi.require_version("Modulemd", "2.0")
from gi.repository import Modulemd

That would let me confirm that I've actually gotten all the libraries 
compiling properly inside the wheel and the code actually works. Then 
there's a question of how to do this automatically. A 90% solution would 
be to just decide that your wheel has a top-level Python module to do 
this, e.g., you tell your users that if they're using the wheel they just 
do "import Modulemd" and you create a Modulemd.py that does

os.setenv("GI_TYPELIB_PATH", some relative path from __file__)
etc.

(For bonus points, call g_irepository_prepend_search_path() / see if 
PyGObject has some binding to it, instead of setting $GI_TYPELIB_PATH, and 
use ctypes to load the actual C library using RTLD_GLOBAL so that it's 
already loaded when PyGObject goes looking for it, instead of setting 
$LD_LIBRARY_PATH.)

In my (naive) opinion, a 100% solution here would be teaching PyGObject 
how to find both typelib files and C libraries in paths based on sys.path, 
and then your users could use the standard upstream import instructions 
unmodified. (Actually, it's possible PyGObject does this already, but I 
don't immediately see anything about it in the docs, and my assumption is 
if you can't find examples of others doing this, the use case hasn't come 
up.)

One other question is whether your users are importing libgobject from the 
system or from a wheel. For the average desktop Linux user, it's probably 
fine to get libgobject from the system (and probably _preferable_ - you 
likely want the same version as the Gtk/GNOME/etc. libraries they might 
import, and if they're importing any of those, they almost certainly want 
the system version of Gtk etc.) It appears that PyGObject is on PyPI as 
sdists only, so if you don't want to assume your users have libgobject 
installed, you may have to first fight the battle of packaging up GObject, 
GLib, etc. into wheels.

(Relatedly, I'm guessing the reason nobody has done this yet is that most 
software that supports GObject introspection is GNOME-related in some 
fashion and therefore most people want it from their OS package manager 
and not from a wheel.)

Again, it's been a while since I've worked with GObject introspection, so 
if I got something wrong, anyone should feel free to correct me :)

-- 
Geoffrey Thomas
https://ldpreload.com
geofft at ldpreload.com