Mailman 3 - Distutils-SIG

The Error NOT FOUND On My Project
by xelekid611＠mado34.com 25 Jul '20

25 Jul '20

I am facing the issue of not found "C:\Users\jaraco\projects\distutils_namespace\myns.projB\test\testBasic.py", can anyone guide me for this ? https://apkpalace.net/

1 0

Re: Fwd: Re: Use of "python" shebang an installation error?
by Filipe Laíns 22 Jul '20

22 Jul '20

On Tue, 2020-07-21 at 14:24 -0700, David Mathog wrote: > On Tue, Jul 21, 2020 at 2:03 PM Filipe Laíns <lains(a)archlinux.org> > wrote: > > On Tue, 2020-07-21 at 13:51 -0700, David Mathog wrote: > > > biopython-1.77, for instance, when installed into a virtualenv > > > with > > > pip3, has many of these shebangs: > > > > > > #!/usr/bin/env python > > > > > Is it being installed as an egg or wheel? Can you check what > > happens > > when you pass --use-pep517 to the pip install command (to force > > installing as a wheel)? > > > > It is wheel, it creates a directory > > biopython/lib/python3.6/site-packages/biopython-1.77.dist-info > > when installed either of these ways: > > pip3 install biopython > pip3 install biopython --use-pep517 > > Either way it has the same shebang in the files listed a few posts > up. > > Thanks, > > David Mathog You've fallen off-list again. The shebang handling for wheels is very well defined[1]. It looks like those shebangs come from biopython so it's their bug. No file a python package is supposed to be executed directly, so they shouldn't really have shebangs. The files that are superposed to be executed directly should be defined as scripts and have a #!python shebang so that the wheel installer can do its job properly. So unfortunately I think in most of the packages you are having issues with, the bug comes from the package upstream :/ [1] https://www.python.org/dev/peps/pep-0427/#recommended-installer-features Filipe Laíns

1 0

Fwd: Re: Use of "python" shebang an installation error?
by David Mathog 22 Jul '20

22 Jul '20

(oops, had to resend, forgot to change the destination to <distutils-sig(a)python.org>) On Mon, Jul 20, 2020 at 12:38 PM John Thorvald Wodder II <jwodder(a)gmail.com> wrote: > > On 2020 Jul 20, at 15:25, David Mathog <dmathog(a)gmail.com> wrote: > > Lately I have been working on a CentOS 8 machine, and it has "python2" > > and "python3", but no "python". Many packages install scripts with a > > shebang like: > > > > #!/usr/bin/env python > > > > and those do not work on this OS. Seems like rather a large missing > > dependency which goes by without triggering a fatal error. > > How exactly are these packages getting installed? Last time I checked, both pip and setuptools automatically set the shebang in scripts (both console_script entry points and scripts declared with the "scripts" argument to `setup()`) to use the path of the running Python interpreter. Are these packages installed using your system package manager? If so, you should take the problem up with its maintainers. Good point, I have been installing so many packages I get confused about which installer was used for which package. It turned out that many (but not all) of the files which contained #!/usr/bin/env python shebangs were installed using standard OS level tools (cmake, configure, make and the like). Example package, hisat2. I guess there isn't much choice for those but to scan the directories for python scripts and fix the shebangs. Installs that are initially into venvs and used pip3 are still an issue. Example: python3 -m venv johnnydep cd johnnydep grep -r '/usr/bin/env python$' . #finds: ./lib/python3.6/site-packages/pip/_vendor/appdirs.py:#!/usr/bin/env python ./lib/python3.6/site-packages/pip/_vendor/chardet/cli/chardetect.py:#!/usr/bin/env python ./lib/python3.6/site-packages/pip/_vendor/requests/certs.py:#!/usr/bin/env python ./lib/python3.6/site-packages/pkg_resources/_vendor/appdirs.py:#!/usr/bin/env python ./lib/python3.6/site-packages/johnnydep/pipper.py:#!/usr/bin/env python cd bin ls -1 | grep python lrwxrwxrwx. 1 modules modules 7 Jul 20 14:09 python -> python3 lrwxrwxrwx. 1 modules modules 16 Jul 20 14:09 python3 -> /usr/bin/python3 source activate pip3 install johnnydep head -1 johnnydep #!/home/common/lib/python3.6/Envs/johnnydep/bin/python #same for "tabulate" and all other shebangs in bin. cd .. grep -r '/usr/bin/env python$' . #same as before grep -r '/home/common/lib/python3.6/Envs/johnnydep/bin/python' . #just the files in the bin directory. It looks like none of the "#!/usr/bin/env python" shebangs within the venv are going to be used after the install, so perhaps those are harmless. The shebangs like #!/home/common/lib/python3.6/Envs/johnnydep/bin/python are OK within the venv, but once they are "devirtualized" they become a problem. That was a known problem though - my devirtualizer code already patches all of the ones in the bin directory. I have not seen any elsewhere (yet) within the venv, but there is probably no rule that keeps them from appearing in "share" or elsewhere. The "python" in use in the venv is just a symbolic link to "python3" which is itself a symbolic link to the actual program "/usr/bin/python3". It is constructed that way based on "python -m venv" which uses pieces which come from the CentOS 8 python3-libs-3.6.8-23.el8.x86_64 RPM. Is there some requirement that a venv have a "python"? Odd that RedHat (and so CentOS) provide a "python" there, but not in the OS itself. Thanks, David Mathog

6 8

Fwd: Re: Use of "python" shebang an installation error?
by David Mathog 21 Jul '20

21 Jul '20

(oops, had to resend, forgot to change the destination to <distutils-sig(a)python.org>) biopython-1.77, for instance, when installed into a virtualenv with pip3, has many of these shebangs: #!/usr/bin/env python And they are all over the place. They are: ./site-packages/Bio/bgzf.py: ./site-packages/Bio/PDB/parse_pdb_header.py: ./site-packages/Bio/PDB/PDBList.py: ./site-packages/Bio/Restriction/__init__.py: ./site-packages/Bio/Restriction/Restriction.py: ./site-packages/Bio/Restriction/PrintFormat.py: ./site-packages/Bio/Restriction/Restriction_Dictionary.py: ./site-packages/Bio/Wise/__init__.py: ./site-packages/Bio/Wise/psw.py: ./site-packages/Bio/Wise/dnal.py: ./site-packages/Bio/UniProt/GOA.py: ./site-packages/Bio/SeqUtils/__init__.py: Regards, David Mathog

2 1

Fwd: Re: Use of "python" shebang an installation error?
by David Mathog 21 Jul '20

21 Jul '20

On Mon, Jul 20, 2020 at 4:40 PM John Thorvald Wodder II <jwodder(a)gmail.com> wrote: > > First of all, your last two messages only went to me, not to the list. The mailing list doesn't set Reply-To on messages or the like, so you have to manually set "To: distutils-sig(a)python.org" when replying. Aargh, right. I use gmail for my home mail, and since I'm stuck working at home, that is what I used here. Gmail likes to hide, well, pretty much everything. I will repost those responses. > > As to your e-mail, though, are any of those files even meant to be executed? They're not in bin/; they just appear to be regular source files that some developer slapped a shebang on. That in a sense is the issue. I don't know, you don't know, maybe the developer knows (if he/she still remembers). I really don't want to do the work to dig through the code for every package I install to determine if a shebang is used or not. Yet if I don't figure this out some end user will run a script (one of a hundred in some package I installed for their use) which will blow up because of this issue. The best I can do now is run pdvctrl reshebang $TARGET_DIR or pdvctrl reshebang $ROOT_DIR... and fix them up after the fact. (pdvctrl from python_devirtualizer here: https://sourceforge.net/projects/python-devirtualizer/). Even then it usually has to guess that "python" means "python3" and not "python2", and sometimes it guesses wrong. Today's version of that recurring issue: https://github.com/lastz/lastz/issues/30 Regards, David Mathog

1 0

pip install very slow
by Stuart McGraw 21 Jul '20

21 Jul '20

Hello, First time trying to use Python's package tools and I'm attempting to use pip to install from a local project: python3 -m pip install --user --upgrade --no-deps ./ I have a simple directory structure very similar to that recommended in https://packaging.python.org/tutorials/packaging-projects/ project/ data/ mypackage/ __init__.py data/ *.csv module1.py module2.py tmpl/ *.jinja setup.py The project/data/ directory contains several GB of stuff I use during development. It is not supposed to be part of the package and it is not mentioned in setup.py. The problem is when I run the install command above it takes many minutes to finish. When I move the project/data/ sub-directory out of the project/ directory, it takes a few seconds. I thought perhaps that "packages=setuptools.find_packages()" in my setup.py file might be the culprit but replacing in with "packages=['mypackage']" made no difference. I'm not sure why pip seems to be roaming through project/data/ (if that's what's happening) but is there some way to get it not to? My apologies if this is covered in the docs somewhere; I'm still going though them but didn't see anything with an initial scan. If it helps, here is my setup.py file: setuptools.setup ( name="mypackage", version="...", author="Me", author_email="...", description="...", long_description="...", long_description_content_type="text/markdown", url="...", packages=['mypackage'], package_data={'mypackage': ['data/*.csv', 'tmpl/*.jinja',]}, zip_safe = False, classifiers=[...], python_requires='>=3.6', install_requires = ['jinja2',], ) Thanks for any enlightenment.

4 5

Use of "python" shebang an installation error?
by David Mathog 20 Jul '20

20 Jul '20

Lately I have been working on a CentOS 8 machine, and it has "python2" and "python3", but no "python". Many packages install scripts with a shebang like: #!/usr/bin/env python and those do not work on this OS. Seems like rather a large missing dependency which goes by without triggering a fatal error. In bioinformatics pipelines it is common for one package to invoke a script from another. So while the package which supplied a particular script might have avoided this issue by only invoking it with: python3 path/script that does not prevent another package from doing one of these: A path/script B python path/script In terms of analysis, it is trivial to find all python scripts installed by a package and examine the shebang line (if present) to see if this is an issue. I am adding a "reshebang" function to my python_devirtualizer specifically to handle the issue for scripts which are invoked directly. It is, however, not at all trivial to analyze all a package's code to see which scripts are called by other scripts, and how they are called. Moreover, they might be called from perl, or C, or some other language. So dealing with "B" above is not trivial. So, my question is, should the use of "python" (as opposed to "python2" or "python3") in a shebang be considered an installation error on a system for which "python" does not exist? I would argue yes, because we already know that python3 was not fully backwards compatible with python2, so we have reason to suspect that python4 (whenever that appears) might also not be fully backwards compatible with python3. By being picky about the python version now, that should prevent a lot of problems later. Regards, David Mathog

2 1

FASTEN and dependency analysis at call graph level
by Sumana Harihareswara 16 Jul '20

16 Jul '20

Last week I heard for the first time about the research project FASTEN https://www.fasten-project.eu/. "FASTEN stands for Fine-Grained Analysis of Software Ecosystems as Networks." > instead of analyzing dependencies at the package level, we will analyze them at the call graph level! This will allow us to be super precise when we are tracking dependencies, when we do change impact analysis, when we recommend clients to update packages etc. It will also open the door to new sophisticated applications, e.g. licensing compliance, dependency risk profiling and data-driven API evolution. That's from the blog post by Georgios Gousios, the PI, at http://www.gousios.gr/blog/Introducing-Fasten.html . More info: https://www.fasten-project.eu/view/Main/Introduction https://www.fasten-project.eu/view/Main/Overview https://www.fasten-project.eu/view/Main/Contacts And people who are interested in dynamically and statically analyzing call graphs in Python may be interested in "Graph Schema and its representation" in https://www.fasten-project.eu/view/Main/Deliverables . I've sent a note to FASTEN inviting the team to come talk about their project here on distutils-sig, because FASTEN's site says they aim to eventually integrate into PyPI -- I'm not 100% sure whether that means "create a service that people can use WITH PyPI" or "get FASTEN's work incorporated into pypi.org". -- Sumana Harihareswara pip project manager on contract with Python Software Foundation Changeset Consulting https://changeset.nyc

3 2

Setuptools adopts distutils
by Jason R. Coombs 13 Jul '20

13 Jul '20

I’m pleased to announce the release of Setuptools 48, which adopts the distutils codebase from CPython per pypa/setuptools#417<https://github.com/pypa/setuptools/issues/417> and pypa/packaging-problems#127<https://github.com/pypa/packaging-problems/issues/127>. Given the amount of change this effort involved, it’s likely unstable and thus the major version bump. Please report issues at the Setuptools issue tracker. I’ll be around today (IRC, Gitter, Slack) to either disable the functionality or add an escape hatch if needed.

6 6

PEP 541 / Transfer of https://pypi.org/project/Augur/ to my project https://github.com/chaoss/augur
by Sean Goggins 08 Jul '20

08 Jul '20

Hi, I am the code owner for the Augur Open Source Software project, which is part of the Linux Foundation’s CHAOSS Project. One of my developers secured our PiPy distribution a few years ago, but deleted their account before graduating. I’m hoping to secure it again: https://pypi.org/project/Augur/ <https://pypi.org/project/Augur/> Our project is at https://github.com/chaoss/augur and is part of the Linux Foundation’s CHAOSS Project at https://chaoss.community Thanks! Sean P. Goggins Associate Professor, Computer Science University of Missouri http://www.seangoggins.net <http://www.seangoggins.net/> visit: http://www.augurlabs.io <http://www.augurlabs.io/> (Research Lab) [Sloan Foundation, National Science Foundation] visit: http://www.globalconsentmanager.com <http://www.globalconsentmanager.com/> (Privacy Research) (Reynolds Journalism Institute and Mozilla) visit: <http://mhs.missouri.edu/>http://mhs.missouri.edu <http://mhs.missouri.edu/> (for mission hydro sci!) [i3 & IES] visit: http://www.sociotech.net <http://www.sociotech.net/> visit: <http://chaoss.community/> <http://chaoss.community/> <http://chaoss.community/>http://c <http://c/>haoss.community (for open source health metrics) [Sloan Foundation] visit: http://sociallycompute.io <http://sociallycompute.io/> (for code like things and Group Informatics) [National Science Foundation] “You build on failure. You use it as a stepping stone. Close the door on the past. You don't try to forget the mistakes, but you don't dwell on it. You don't let it have any of your energy, or any of your time, or any of your space.” -- Johnny Cash

3 2