[C++-sig] Different behavior between local import and import from python-path

[Benedikt Tutzer]

Hello,

A few months ago I published Python wrappers for Yosys 
(https://github.com/YosysHQ/yosys). These can be used in two ways:

 1. Use Yosys in Python code by importing it and using it as a library
 2. Start the Yosys shell (regular command-line interaction) and load
    python code by specifying a source file. In that source file, one
    can implement passes by inheriting from the Pass class and these
    passes are then available in the Yosys shell.

This works well and was already merged into the main Yosys repo.

However, there is an issue I cannot wrap my head around. When Yosys is 
installed, I copy the shared library .so file together with a minimal 
__init__.py to /usr/lib/python3.5/site-packages/pyosys. I can then use 
that by writing

> from pyosys import libyosys as ys
This works great for case 1. I also need that import for case 2, so that 
users can inherit from the Pass class and use Yosys data structures. The 
import seems to work, but as soon as a command is issued, an exception 
is thrown. This exception seems not to concern the interoperability with 
Python, but it does not appear when I copy the .so file to the current 
working directory and import it by saying
> import libyosys as ys
In that case, everything is well and the whole thing works. The Pass 
defined in Python shows up in the Pass-list, can be executed and 
successfully interacts with the data structures created by C++ passes.

The obvious place to look would be the __init__.py file, since that is 
the only thing that is executed when importing from the path but not 
executed when importing from the local library, right? But all I do in 
the init file is:

> import os
> import sys
> sys.setdlopenflags(os.RTLD_NOW | os.RTLD_GLOBAL)
>
> __all__ = ["libyosys"]
And if I remove the first lines it doesn't work either, so they cannot 
(solely) be at fault.

Any other reason why the two ways to import would differ?

The exception thrown is a std::out_of_range exception where an empty 
vector is accessed at position 171. I introduced some debug-output and 
notice that the vector actually contains 179 elements. Is it somehow 
possible that Python and C++ see a different version of the vector? It 
is filled from C++ before calling any Python code and there is no place 
in the code where anything is removed from the vector, so I don't see 
why it should shrink in size.

There is a Github issue about this where you can find a minimal example, 
but you need to build yosys and it is kind of big -> 
https://github.com/YosysHQ/yosys/issues/1094

The Python wrappers are defined in kernel/python_wrappers.cc, but it is 
auto-generated from a Python-script during the build, so I attached it 
for your convenience.

Any thoughts and ideas are highly welcome! I am completely stuck and 
don't know where to start debugging.

BR

Benedikt

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cplusplus-sig/attachments/20190704/b8ec136f/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: python_wrappers.cc
Type: text/x-c++src
Size: 561529 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/cplusplus-sig/attachments/20190704/b8ec136f/attachment-0001.cc>