Packaging optional, arch-dependent, pre-built libraries
Hello, I'm the author of python-libusb1, a pure-python ctypes wrapper for libusb1. Until recently, I had been purely relying on OS-linker-provided libusb1 (distro-installed on GNU/Linux and *BSD, fink/macports/... on OSX, ...). Then, I've been requested to bundle the libusb1 dll on windows (x86 and x86_64 wheels) because otherwise distributions seems exceedingly painful for applications using my module. With some extra code to setup.py to fetch, unzip and copy[1] the dlls, plus a now even more multi-stage distribution process (sdist, both windows wheels, in addition to the existing sign and twine steps), and it ipso facto works. Now, I'm asked to add pyinstaller compatibility, as it on its own overlooks the dll. Which makes me feel that I am maybe not using the best possible way to bundle these. From my reading of distutils and setuptools, my understanding is that a package is that non-pure-python packages contain: - stuff they built themselves (build_ext & friends) - third-party libraries that the stuff they built themselves is linked against Having nothing to build, I cannot seem to reach the library inclusion step. What is the recommended way of bundling a third-party-built arch-dependent library in an otherwise pure-python package ? [1] https://github.com/vpelletier/python-libusb1/blob/49f7f846bdd3c3d0f2ec3a01c2... Regards, -- Vincent Pelletier GPG fingerprint 983A E8B7 3B91 1598 7A92 3845 CAC9 3691 4257 B0C1
If a file is not built or linked against, a dll in your wheel is essentially a plain data file from Python packaging’s perspective, no different from e.g. a text file. So you’re looking in the wrong direction for solutions. I believe the issue PyInstaller has with your package is that, since PyInstaller compiles a program into an executable, ctypes.util.find_library() won’t work (since there is no actual dll to find). If you know for sure the dll will be available, you can copy the binary to a temporary location (the “official” way to do this is through importlib.resources.path[1]), and use the path to load the dll directly instead. [1]: https://importlib-resources.readthedocs.io/en/latest/ -- Tzu-ping Chung (@uranusjr) uranusjr@gmail.com https://uranusjr.com
On 03/4/2021, at 06:57, Vincent Pelletier <plr.vincent@gmail.com> wrote:
Hello,
I'm the author of python-libusb1, a pure-python ctypes wrapper for libusb1.
Until recently, I had been purely relying on OS-linker-provided libusb1 (distro-installed on GNU/Linux and *BSD, fink/macports/... on OSX, ...).
Then, I've been requested to bundle the libusb1 dll on windows (x86 and x86_64 wheels) because otherwise distributions seems exceedingly painful for applications using my module. With some extra code to setup.py to fetch, unzip and copy[1] the dlls, plus a now even more multi-stage distribution process (sdist, both windows wheels, in addition to the existing sign and twine steps), and it ipso facto works.
Now, I'm asked to add pyinstaller compatibility, as it on its own overlooks the dll. Which makes me feel that I am maybe not using the best possible way to bundle these.
From my reading of distutils and setuptools, my understanding is that a package is that non-pure-python packages contain: - stuff they built themselves (build_ext & friends) - third-party libraries that the stuff they built themselves is linked against Having nothing to build, I cannot seem to reach the library inclusion step.
What is the recommended way of bundling a third-party-built arch-dependent library in an otherwise pure-python package ?
[1] https://github.com/vpelletier/python-libusb1/blob/49f7f846bdd3c3d0f2ec3a01c2...
Regards, -- Vincent Pelletier GPG fingerprint 983A E8B7 3B91 1598 7A92 3845 CAC9 3691 4257 B0C1 -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-leave@python.org https://mail.python.org/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/archives/list/distutils-sig@python.org/message/J6DDM...
- Builds manylinux, macOS 10.9+, and Windows wheels for CPython and PyPy - Works on GitHub Actions, Azure Pipelines, Travis CI, AppVeyor, CircleCI, and GitLab CI - Bundles shared library dependencies on Linux and macOS through auditwheel and delocate - Runs your library's tests against the wheel-installed version of your
Is there some easy way to solve this specifically with cibuildwheel? https://github.com/joerick/cibuildwheel : library On Mon, Apr 5, 2021, 12:18 Tzu-ping Chung <uranusjr@gmail.com> wrote:
If a file is not built or linked against, a dll in your wheel is essentially a plain data file from Python packaging’s perspective, no different from e.g. a text file. So you’re looking in the wrong direction for solutions.
I believe the issue PyInstaller has with your package is that, since PyInstaller compiles a program into an executable, ctypes.util.find_library() won’t work (since there is no actual dll to find). If you know for sure the dll will be available, you can copy the binary to a temporary location (the “official” way to do this is through importlib.resources.path[1]), and use the path to load the dll directly instead.
[1]: https://importlib-resources.readthedocs.io/en/latest/
-- Tzu-ping Chung (@uranusjr) uranusjr@gmail.com https://uranusjr.com
On 03/4/2021, at 06:57, Vincent Pelletier <plr.vincent@gmail.com> wrote:
Hello,
I'm the author of python-libusb1, a pure-python ctypes wrapper for libusb1.
Until recently, I had been purely relying on OS-linker-provided libusb1 (distro-installed on GNU/Linux and *BSD, fink/macports/... on OSX, ...).
Then, I've been requested to bundle the libusb1 dll on windows (x86 and x86_64 wheels) because otherwise distributions seems exceedingly painful for applications using my module. With some extra code to setup.py to fetch, unzip and copy[1] the dlls, plus a now even more multi-stage distribution process (sdist, both windows wheels, in addition to the existing sign and twine steps), and it ipso facto works.
Now, I'm asked to add pyinstaller compatibility, as it on its own overlooks the dll. Which makes me feel that I am maybe not using the best possible way to bundle these.
From my reading of distutils and setuptools, my understanding is that a package is that non-pure-python packages contain: - stuff they built themselves (build_ext & friends) - third-party libraries that the stuff they built themselves is linked against Having nothing to build, I cannot seem to reach the library inclusion step.
What is the recommended way of bundling a third-party-built arch-dependent library in an otherwise pure-python package ?
[1] https://github.com/vpelletier/python-libusb1/blob/49f7f846bdd3c3d0f2ec3a01c2...
Regards, -- Vincent Pelletier GPG fingerprint 983A E8B7 3B91 1598 7A92 3845 CAC9 3691 4257 B0C1 -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-leave@python.org https://mail.python.org/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/archives/list/distutils-sig@python.org/message/J6DDM...
-- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-leave@python.org https://mail.python.org/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/archives/list/distutils-sig@python.org/message/OBAR3...
Hello, On Tue, 6 Apr 2021 00:17:32 +0800, Tzu-ping Chung <uranusjr@gmail.com> wrote:
If a file is not built or linked against, a dll in your wheel is essentially a plain data file from Python packaging’s perspective, no different from e.g. a text file.
Thanks, I somehow did not get this until I saw it spelled out. There seems to be one catch, though: once I list the dll in package_data, it gets copied over to build/lib*, and the same dll gets used for all distributions, so if I build the win32 wheel first: - plat=win_amd64 gets a 32bits dll - plat=any gets a windows 32bits dll so I now have to clean --all between each. For a text file it could make no difference (although there are the EOL shenanigans which would arguably be platform-specific). So while I now agree the dll should be treated by my setup.py as a "plain data file", there is this annoying extra complication layer.
I believe the issue PyInstaller has with your package is that, since PyInstaller compiles a program into an executable, ctypes.util.find_library() won’t work (since there is no actual dll to find). If you know for sure the dll will be available, you can copy the binary to a temporary location (the “official” way to do this is through importlib.resources.path[1]), and use the path to load the dll directly instead.
Thanks for the pointer, I would love to use it. Unfortunately, this fails to install on 2.7: with install_requires=( "importlib_resources<=4.0.0;python_version<'3.0'", "importlib_resources;python_version>='3.0' and python_version<'3.7'", ), I get $ ./vpy2/bin/python setup.py install [...] Installed /home/vincent/git/python-libusb1/vpy2/lib/python2.7/site-packages/libusb1-1.9.2+4.g5aeb636.dirty-py2.7.egg Processing dependencies for libusb1==1.9.2+4.g5aeb636.dirty Searching for zipp>=0.4 Reading https://pypi.org/simple/zipp/ Downloading https://files.pythonhosted.org/packages/38/f9/4fa6df2753ded1bcc1ce2fdd8046f7... Best match: zipp 3.4.1 Processing zipp-3.4.1.tar.gz Writing /tmp/easy_install-ZDtgKM/zipp-3.4.1/setup.cfg Running zipp-3.4.1/setup.py -q bdist_egg --dist-dir /tmp/easy_install-ZDtgKM/zipp-3.4.1/egg-dist-tmp-GmefKD DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support pip 21.0 will remove support for this functionality. DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support pip 21.0 will remove support for this functionality. error: find_namespace: directive is unsupported on Python < 3.3 The contributor who requested pyinstaller support somehow got this to work with my archaic and unfortunately very zip-hostile os.path.join(dirname(__file__), 'libusb-1.0.dll') so I will be continuing with this until I finally drop 2.7 support. Regards, -- Vincent Pelletier GPG fingerprint 983A E8B7 3B91 1598 7A92 3845 CAC9 3691 4257 B0C1
“setup.py install” is pretty ancient at this point and lacks many of the remotely modern packaging syntax. I’d strongly advise to ignore it entirely. Use “pip install .” instead. -- Tzu-ping Chung (@uranusjr) uranusjr@gmail.com https://uranusjr.com
On 11/4/2021, at 08:59, Vincent Pelletier <plr.vincent@gmail.com> wrote:
Hello,
On Tue, 6 Apr 2021 00:17:32 +0800, Tzu-ping Chung <uranusjr@gmail.com> wrote:
If a file is not built or linked against, a dll in your wheel is essentially a plain data file from Python packaging’s perspective, no different from e.g. a text file.
Thanks, I somehow did not get this until I saw it spelled out.
There seems to be one catch, though: once I list the dll in package_data, it gets copied over to build/lib*, and the same dll gets used for all distributions, so if I build the win32 wheel first: - plat=win_amd64 gets a 32bits dll - plat=any gets a windows 32bits dll so I now have to clean --all between each.
For a text file it could make no difference (although there are the EOL shenanigans which would arguably be platform-specific). So while I now agree the dll should be treated by my setup.py as a "plain data file", there is this annoying extra complication layer.
I believe the issue PyInstaller has with your package is that, since PyInstaller compiles a program into an executable, ctypes.util.find_library() won’t work (since there is no actual dll to find). If you know for sure the dll will be available, you can copy the binary to a temporary location (the “official” way to do this is through importlib.resources.path[1]), and use the path to load the dll directly instead.
Thanks for the pointer, I would love to use it. Unfortunately, this fails to install on 2.7: with install_requires=( "importlib_resources<=4.0.0;python_version<'3.0'", "importlib_resources;python_version>='3.0' and python_version<'3.7'", ), I get $ ./vpy2/bin/python setup.py install [...] Installed /home/vincent/git/python-libusb1/vpy2/lib/python2.7/site-packages/libusb1-1.9.2+4.g5aeb636.dirty-py2.7.egg Processing dependencies for libusb1==1.9.2+4.g5aeb636.dirty Searching for zipp>=0.4 Reading https://pypi.org/simple/zipp/ Downloading https://files.pythonhosted.org/packages/38/f9/4fa6df2753ded1bcc1ce2fdd8046f7... Best match: zipp 3.4.1 Processing zipp-3.4.1.tar.gz Writing /tmp/easy_install-ZDtgKM/zipp-3.4.1/setup.cfg Running zipp-3.4.1/setup.py -q bdist_egg --dist-dir /tmp/easy_install-ZDtgKM/zipp-3.4.1/egg-dist-tmp-GmefKD DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support pip 21.0 will remove support for this functionality. DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support pip 21.0 will remove support for this functionality. error: find_namespace: directive is unsupported on Python < 3.3
The contributor who requested pyinstaller support somehow got this to work with my archaic and unfortunately very zip-hostile os.path.join(dirname(__file__), 'libusb-1.0.dll') so I will be continuing with this until I finally drop 2.7 support.
Regards, -- Vincent Pelletier GPG fingerprint 983A E8B7 3B91 1598 7A92 3845 CAC9 3691 4257 B0C1
On Sun, 2021-04-11 at 00:59 +0000, Vincent Pelletier wrote:
Thanks for the pointer, I would love to use it. Unfortunately, this fails to install on 2.7: with install_requires=( "importlib_resources<=4.0.0;python_version<'3.0'", "importlib_resources;python_version>='3.0' and python_version<'3.7'", ),
My bad: 4.0.0 was the first version *without* py2.7 support, so this was supposed to be: install_requires=( "importlib_resources<4.0.0;python_version<'3.0'", "importlib_resources;python_version>='3.0' and python_version<'3.7'", ), Somehow this does not seem to make a difference. Maybe the last wheel for 2.7 being the last supported version was making my bogus dependency accidentally work (for as far as importlib_resources version itself is concerned, so ignoring the zipp error). On Sun, 11 Apr 2021 09:26:47 +0800, Tzu-ping Chung <uranusjr@gmail.com> wrote:
“setup.py install” is pretty ancient at this point and lacks many of the remotely modern packaging syntax.
I’d strongly advise to ignore it entirely. Use “pip install .” instead.
Indeed: $ rm -r vpy2 $ virtualenv -p /usr/bin/python2 vpy2 created virtual environment CPython2.7.18.final.0-64 in 90ms creator CPython2Posix(dest=/home/vincent/git/python-libusb1/vpy2, clear=False, no_vcs_ignore=False, global=False) seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/vincent/.local/share/virtualenv) added seed packages: pip==20.3.4, pkg_resources==0.0.0, setuptools==44.1.1, wheel==0.34.2 activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator $ ./vpy2/bin/python setup.py install [...] Installed /home/vincent/git/python-libusb1/vpy2/lib/python2.7/site-packages/importlib_resources-3.3.1-py2.7.egg Searching for zipp>=0.4 Reading https://pypi.org/simple/zipp/ Downloading https://files.pythonhosted.org/packages/38/f9/4fa6df2753ded1bcc1ce2fdd8046f7... Best match: zipp 3.4.1 Processing zipp-3.4.1.tar.gz Writing /tmp/easy_install-kLJx7K/zipp-3.4.1/setup.cfg Running zipp-3.4.1/setup.py -q bdist_egg --dist-dir /tmp/easy_install-kLJx7K/zipp-3.4.1/egg-dist-tmp-10cETE DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support pip 21.0 will remove support for this functionality. DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support pip 21.0 will remove support for this functionality. error: find_namespace: directive is unsupported on Python < 3.3 But (with --no-cache-dir just in case cached versions would influence version choice): $ rm -r vpy2 $ virtualenv -p /usr/bin/python2 vpy2 created virtual environment CPython2.7.18.final.0-64 in 91ms creator CPython2Posix(dest=/home/vincent/git/python-libusb1/vpy2, clear=False, no_vcs_ignore=False, global=False) seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/vincent/.local/share/virtualenv) added seed packages: pip==20.3.4, pkg_resources==0.0.0, setuptools==44.1.1, wheel==0.34.2 activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator $ ./vpy2/bin/pip install --no-cache-dir . [...] Collecting zipp>=0.4; python_version < "3.8" Downloading zipp-1.2.0-py2.py3-none-any.whl (4.8 kB) [...] Successfully installed contextlib2-0.6.0.post1 importlib-resources-3.3.1 libusb1-1.9.2+4.g452c3e9.dirty pathlib2-2.3.5 scandir-1.10.0 singledispatch-3.6.1 six-1.15.0 typing-3.7.4.3 zipp-1.2.0 pip is choosing 1.2.0 while setup.py install is choosing 3.4.1 . -- Vincent Pelletier GPG fingerprint 983A E8B7 3B91 1598 7A92 3845 CAC9 3691 4257 B0C1
On Sun, 2021-04-11 at 00:59 +0000, Vincent Pelletier wrote:
Hello,
On Tue, 6 Apr 2021 00:17:32 +0800, Tzu-ping Chung <uranusjr@gmail.com> wrote:
If a file is not built or linked against, a dll in your wheel is essentially a plain data file from Python packaging’s perspective, no different from e.g. a text file.
Thanks, I somehow did not get this until I saw it spelled out.
There seems to be one catch, though: once I list the dll in package_data, it gets copied over to build/lib*, and the same dll gets used for all distributions, so if I build the win32 wheel first: - plat=win_amd64 gets a 32bits dll - plat=any gets a windows 32bits dll so I now have to clean --all between each.
For a text file it could make no difference (although there are the EOL shenanigans which would arguably be platform-specific). So while I now agree the dll should be treated by my setup.py as a "plain data file", there is this annoying extra complication layer.
I believe the issue PyInstaller has with your package is that, since PyInstaller compiles a program into an executable, ctypes.util.find_library() won’t work (since there is no actual dll to find). If you know for sure the dll will be available, you can copy the binary to a temporary location (the “official” way to do this is through importlib.resources.path[1]), and use the path to load the dll directly instead.
Thanks for the pointer, I would love to use it. Unfortunately, this fails to install on 2.7: with install_requires=( "importlib_resources<=4.0.0;python_version<'3.0'", "importlib_resources;python_version>='3.0' and python_version<'3.7'", ), I get $ ./vpy2/bin/python setup.py install [...] Installed /home/vincent/git/python-libusb1/vpy2/lib/python2.7/site- packages/libusb1-1.9.2+4.g5aeb636.dirty-py2.7.egg Processing dependencies for libusb1==1.9.2+4.g5aeb636.dirty Searching for zipp>=0.4 Reading https://pypi.org/simple/zipp/ Downloading https://files.pythonhosted.org/packages/38/f9/4fa6df2753ded1bcc1ce2fdd8046f7... Best match: zipp 3.4.1 Processing zipp-3.4.1.tar.gz Writing /tmp/easy_install-ZDtgKM/zipp-3.4.1/setup.cfg Running zipp-3.4.1/setup.py -q bdist_egg --dist-dir /tmp/easy_install- ZDtgKM/zipp-3.4.1/egg-dist-tmp-GmefKD DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support pi p 21.0 will remove support for this functionality. DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support pi p 21.0 will remove support for this functionality. error: find_namespace: directive is unsupported on Python < 3.3
The contributor who requested pyinstaller support somehow got this to work with my archaic and unfortunately very zip-hostile os.path.join(dirname(__file__), 'libusb-1.0.dll') so I will be continuing with this until I finally drop 2.7 support.
Regards, -- Vincent Pelletier GPG fingerprint 983A E8B7 3B91 1598 7A92 3845 CAC9 3691 4257 B0C1 -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-leave@python.org https://mail.python.org/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/archives/list/distutils-sig@python.org/message/QGADI...
Can you share in more detail how find_namespace is being used? Python 2.7 does not support namespace packages, I'd say you probably don't really to use them anyway, putting a __init__.py in the target package folder and using the find directive instead should solve the issue. Cheers, Filipe Laíns
On Wed, 14 Apr 2021 13:59:00 +0100, Filipe Laíns <lains@archlinux.org> wrote:
Thanks for the pointer, I would love to use it. Unfortunately, this fails to install on 2.7: with install_requires=( "importlib_resources<=4.0.0;python_version<'3.0'", "importlib_resources;python_version>='3.0' and python_version<'3.7'", ), Can you share in more detail how find_namespace is being used? Python 2.7 does not support namespace packages, I'd say you probably don't really to use them anyway, putting a __init__.py in the target package folder and using the find
On Sun, 2021-04-11 at 00:59 +0000, Vincent Pelletier wrote: directive instead should solve the issue.
This seems to actually be brought by the too-recent zipp that "setup.py install" is fetching. "pip install ." works. -- Vincent Pelletier GPG fingerprint 983A E8B7 3B91 1598 7A92 3845 CAC9 3691 4257 B0C1
participants (4)
-
Filipe Laíns -
Tzu-ping Chung -
Vincent Pelletier -
Wes Turner