TL;DR: OK to archive this mailing list? Reply by Aug 30th.
Below: Context, Proposal, Reasoning, and timeline.
*Context*:
For multiple years now, we've been advising users to use setuptools and
to move away from using distutils. There has been a long standing
plan [^1] to vendor distutils into setuptools to allow it to evolve
independently of the Python standard library and to allow for removal of
distutils from the Python standard library.
Recently, setuptools adopted the distutils [^2] which, as far as I can
tell, starts the long process of removing distutils from the Python
standard library. PEP 517 [^3] also removed the special status of
distutils/setuptools as the only pipeline that can be used for
generating distributions for Python projects.
For some time now, "distutils" has not been a "primary" tool in the
broader Python Packaging ecosystem, with setuptools being an overall
superior tool that also has a good interoperability story with the other
Python packaging tooling. This is acknowledged in the description of
this mailing list as:
> Now, it's better described as the "packaging interoperability SIG",
> where issues that cut across different parts of the Python packaging
> ecosystem get discussed and resolved.
However, this mailing list is no longer serving this stated role either,
with the Packaging category on discuss.python.org becoming the primary
location for packaging tool interoperability discussions.
Over the last year, the Packaging category on discuss.python.org had 841
active topics, with only 40 topics with 3 or fewer responses. [^5]
In the last 100 days, the Packaging category on discuss.python.org has
had 91 active topics. More than 10 PEPs have been discussed in the
Packaging category on discuss.python.org in the last 100 days.
Over the last year, distutils-sig had ~109 active threads, with
(based on a quick skim) most having 3 or fewer responses/posters. [^4]
In the last 100 days, distutils-sig has had 32 active threads (at least
7 of these have the same subject as another thread with Re:/Fwd: added).
There has been only 1 PEP-related feedback discussion on distutils-sig
in the last year. Most of the other threads are user support requests or
announcements.
*Proposal*:
I suggest that, one month from now, we stop posting to this list
(distutils-sig(a)python.org) and archive it.
*Reasoning*:
I think we do not use this mailing list for its dedicated purpose, and
it does not serve any secondary function that isn't better served by a
different communication channel already.
(1) this mailing list is no longer the primary location for
interoperability discussions
(2) we have better channels for user support requests (such as
issue trackers of various projects, packaging-problems etc.)
(3) we have other channels for making announcements (such as the
pypi-announce mailing list, the PyPA twitter account,
discuss.python.org etc.)
*Timeline*:
Here's what I suggest, and what I will carry out if there is no
objection.
In one month, on August 30th, I would verify that no one has argued here
for why this mailing list should not be closed/archived.
Or, even if a few people have objected to closing the list, I would
check for rough consensus, especially of people who are doing SOMETHING
productive having to do with Python Packaging (such as maintainers of
Python packages, maintainers of Python Packaging tooling, folks running
key infrastructure, etc.).
Then, I would request the list administrators to post a final message
to this list (marking its close and suggesting that people use
discuss.python.org instead) and, to archive this mailing list. This
would leave archives available at their current URLs, so links, browsing
and search would work.
And finally, I would look through relevant documentation within PyPA
repositories to see what needs updating (READMEs and so on pointing to
the old list), and submit pull requests.
I appreciate the work folks here have done to carry forward Python
packaging over the past 21 years of this mailing list. I don't mean to
diminish that or to insult anyone here. I want to help us out, and I
think closing this list will help focus our energy better. But I am
open to hearing that I am wrong.
Realizing-that-distutils-sig-is-older-than-me-ly,
Pradyun Gedam
[^1]:
https://www.pypa.io/en/latest/roadmap/#vendor-distutils-into-setuptools
[^2]: https://github.com/pypa/setuptools/pull/2143
[^3]: https://www.python.org/dev/peps/pep-0517/
[^4]:
https://mail.python.org/archives/list/distutils-sig@python.org/latest?count…
[^5]: https://discuss.python.org/c/packaging/14/l/top/yearly?order=posts
On behalf of the PyPA, I am pleased to announce that we have just released
pip 20.2, a new version of pip. You can install it by running python -m pip
install --upgrade pip.
The highlights for this release are:
- The beta of the next-generation dependency resolver is available
- Faster installations from wheel files
- Improved handling of wheels containing non-ASCII file contents
- Faster pip list using parallelized network operations
- Installed packages now contain metadata about whether they were
directly
requested by the user (PEP 376’s REQUESTED file)
The new dependency resolver is *off by default* because it is *not yet
ready for everyday use*. The new dependency resolver is significantly
stricter and more consistent when it receives incompatible instructions,
and reduces support for certain kinds of constraints files, so some
workarounds and workflows may break. Please test it with the
--use-feature=2020-resolver flag. Please see our guide on how to test and
migrate, and how to report issues
<https://pip.pypa.io/en/latest/user_guide/#changes-to-the-pip-dependency-res…>.
We are preparing to change the default dependency resolution behavior and
make the new resolver the default in pip 20.3 (in October 2020).
This release also partially optimizes pip’s network usage during
installation (as part of a Google Summer of Code project by McSinyx
<https://summerofcode.withgoogle.com/projects/#5428041779511296>). Please
test it with pip install --use-feature=fast-deps ... and report bugs to the
issue tracker
<https://github.com/pypa/pip/issues/new?template=bug-report.md>. This
functionality is *still experimental* and *not ready for everyday use*.
You can find more details (including deprecations and removals) in the
changelog <https://pip.pypa.io/en/stable/news/>.
As with all pip releases, a significant amount of the work was contributed
by pip’s user community. Huge thanks to all who have contributed, whether
through code, documentation, issue reports and/or discussion. Your help
keeps pip improving, and is hugely appreciated.
Specific thanks go to Mozilla (through its Mozilla Open Source Support
<https://www.mozilla.org/en-US/moss/> Awards) and to the Chan Zuckerberg
Initiative <https://chanzuckerberg.com/eoss/> DAF, an advised fund of
Silicon Valley Community Foundation, for their funding that enabled
substantial work on the new resolver.
facing same NOt found issue the issue code is "C:\Users\jaraco\projects\distutils_namespace\myns.projB\test\testBasic.py", can you guide me for this ?
https://thelifefoundry.com/hotmail/
Currently in the packaging space, we have a number of avenues for communication, which are:
- distutils-sig
- pypa-dev
- virtualenv-users
- Other project specific mailing lists
- IRC
- gitter
- Various issue trackers spread across multiple platforms.
- Probably more places I’m not remembering.
The result of this is that all discussion ends up being super fractured amongst the various places. Sometimes that is exactly what you want (for instance, someone who is working on the wheel specs probably doesn’t care about deprecation policy and internal module renaming in pip) and sometimes that ends up being the opposite of what you want (for instance, when you’re describing something that touches PyPI, setuptools, flit, pip, etc all at once).
Theoretically the idea is that distutils-sig is where cross project reaching stuff goes, IRC/gitter is where real time discussion goes, and the various project mailing lists and issue trackers are where the project specific bits go. The problem is that often times doesn’t actually happen in practice except for the largest and most obvious of changes.
I think our current “communications stack” kind of sucks, and I’d love to figure out a better way for us to handle this that solves the sort of weird “independent but related” set of projects we have here.
From my POV, a list of our major problems are:
* Discussion gets fractured across a variety of platforms and locations, which can make it difficult to actually keep up with what’s going on but also to know how to loop in someone relevant if their input would be valuable. You have to more or less simultaneously know someone’s email, Github username, IRC nick, bitbucket username, etc to be able to bring threads of discussion to people’s attention.
* It’s not always clear to users where a discussion should go, often times they’ll come to one location and need to get redirected to another location. If any discussion did happen in the incorrect location, it tends to need to get restarted in the new location (and depending on the specific platform, it may be impossible to actually redirect everyone over to the proper location, so you again, end up fractured with the discussion happening in two places).
* A lot of the technology in this stack is particularly old, and lacks a lot of the modern day affordances that newer things have. An example is being able to edit a discussion post to fix typos that can hinder the ability of others to actually understand whats being talked about. In your typical mailing list or IRC there’s no mechanism by which you can edit an already sent message, so your only option is to either let the problem ride and hope it doesn’t trip up too many people, or send an additional message to correct the error. However these show up as additional, later messages which someone might not even see until they’ve already been thoroughly confused by the first message (since people tend to read email/IRC in a linear fashion).
- There is a lot of things in this one, other things are things like being able to control in a more fine grained manner what email you’re going to get.
- Holy crap, formatting and typography to make things actually readable and not a big block of plaintext.
* We don’t have a clear way for users to get help, leaving users to treat random issues, discussion areas, etc as a support forum, rather than some place that’s actually optimized for that. Some of this ties back into some of the earlier things too, where it’s hard to actually redirect discussions
These aren’t *new* problems, and often times the existing members of a community are the least effected becasue they’ve already spent effort learning the ins and outs and also curating a (typically custom) workflow that they’ve grown accustomed too. The problem with that is that often times that means that new users are left out, and the community gets smaller and smaller as time goes on as people leave and aren’t replaced with new blood, because they’re driven off but the issues with the stack.
A big part of the place this is coming from, is me sitting back and realizing that I tend to be drawn towards pulling discussions into Github issues rather than onto the varying mailing lists, not because that’s always the most appropriate place for it, but because it’s the least painful place in terms of features and functionality. I figure if I’m doing that, when I already have a significant investment in setting up tooling and being involved here, that others (and particularly new users) are likely feeling the same way.
- Donald
I am facing the issue of not found
"C:\Users\jaraco\projects\distutils_namespace\myns.projB\test\testBasic.py", can anyone
guide me for this ? https://apkpalace.net/
On Tue, 2020-07-21 at 14:24 -0700, David Mathog wrote:
> On Tue, Jul 21, 2020 at 2:03 PM Filipe Laíns <lains(a)archlinux.org>
> wrote:
> > On Tue, 2020-07-21 at 13:51 -0700, David Mathog wrote:
> > > biopython-1.77, for instance, when installed into a virtualenv
> > > with
> > > pip3, has many of these shebangs:
> > >
> > > #!/usr/bin/env python
> > >
> > Is it being installed as an egg or wheel? Can you check what
> > happens
> > when you pass --use-pep517 to the pip install command (to force
> > installing as a wheel)?
> >
>
> It is wheel, it creates a directory
>
> biopython/lib/python3.6/site-packages/biopython-1.77.dist-info
>
> when installed either of these ways:
>
> pip3 install biopython
> pip3 install biopython --use-pep517
>
> Either way it has the same shebang in the files listed a few posts
> up.
>
> Thanks,
>
> David Mathog
You've fallen off-list again.
The shebang handling for wheels is very well defined[1].
It looks like those shebangs come from biopython so it's their bug.
No file a python package is supposed to be executed directly, so they
shouldn't really have shebangs. The files that are superposed to be
executed directly should be defined as scripts and have a #!python
shebang so that the wheel installer can do its job properly.
So unfortunately I think in most of the packages you are having issues
with, the bug comes from the package upstream :/
[1] https://www.python.org/dev/peps/pep-0427/#recommended-installer-features
Filipe Laíns
(oops, had to resend, forgot to change the destination to
<distutils-sig(a)python.org>)
On Mon, Jul 20, 2020 at 12:38 PM John Thorvald Wodder II
<jwodder(a)gmail.com> wrote:
>
> On 2020 Jul 20, at 15:25, David Mathog <dmathog(a)gmail.com> wrote:
> > Lately I have been working on a CentOS 8 machine, and it has "python2"
> > and "python3", but no "python". Many packages install scripts with a
> > shebang like:
> >
> > #!/usr/bin/env python
> >
> > and those do not work on this OS. Seems like rather a large missing
> > dependency which goes by without triggering a fatal error.
>
> How exactly are these packages getting installed? Last time I checked, both pip and setuptools automatically set the shebang in scripts (both console_script entry points and scripts declared with the "scripts" argument to `setup()`) to use the path of the running Python interpreter. Are these packages installed using your system package manager? If so, you should take the problem up with its maintainers.
Good point, I have been installing so many packages I get confused
about which installer was used for which package. It turned out that
many (but not all) of the files which contained
#!/usr/bin/env python
shebangs were installed using standard OS level tools (cmake,
configure, make and the like). Example package, hisat2. I guess there
isn't much choice for those but to scan the directories for python
scripts and fix the shebangs.
Installs that are initially into venvs and used pip3 are still an
issue. Example:
python3 -m venv johnnydep
cd johnnydep
grep -r '/usr/bin/env python$' .
#finds:
./lib/python3.6/site-packages/pip/_vendor/appdirs.py:#!/usr/bin/env python
./lib/python3.6/site-packages/pip/_vendor/chardet/cli/chardetect.py:#!/usr/bin/env
python
./lib/python3.6/site-packages/pip/_vendor/requests/certs.py:#!/usr/bin/env
python
./lib/python3.6/site-packages/pkg_resources/_vendor/appdirs.py:#!/usr/bin/env
python
./lib/python3.6/site-packages/johnnydep/pipper.py:#!/usr/bin/env python
cd bin
ls -1 | grep python
lrwxrwxrwx. 1 modules modules 7 Jul 20 14:09 python -> python3
lrwxrwxrwx. 1 modules modules 16 Jul 20 14:09 python3 -> /usr/bin/python3
source activate
pip3 install johnnydep
head -1 johnnydep
#!/home/common/lib/python3.6/Envs/johnnydep/bin/python
#same for "tabulate" and all other shebangs in bin.
cd ..
grep -r '/usr/bin/env python$' .
#same as before
grep -r '/home/common/lib/python3.6/Envs/johnnydep/bin/python' .
#just the files in the bin directory.
It looks like none of the "#!/usr/bin/env python" shebangs within the
venv are going to be used after the install, so perhaps those are
harmless.
The shebangs like
#!/home/common/lib/python3.6/Envs/johnnydep/bin/python
are OK within the venv, but once they are "devirtualized" they become
a problem. That was a known problem though - my devirtualizer code
already patches all of the ones in the bin directory. I have not seen
any elsewhere (yet) within the venv, but there is probably no rule
that keeps them from appearing in "share" or elsewhere.
The "python" in use in the venv is just a symbolic link to "python3"
which is itself a symbolic link to the actual program
"/usr/bin/python3". It is constructed that way based on "python -m
venv" which uses pieces which come from the CentOS 8
python3-libs-3.6.8-23.el8.x86_64 RPM. Is there some requirement that
a venv have a "python"? Odd that RedHat (and so CentOS) provide a
"python" there, but not in the OS itself.
Thanks,
David Mathog
(oops, had to resend, forgot to change the destination to
<distutils-sig(a)python.org>)
biopython-1.77, for instance, when installed into a virtualenv with
pip3, has many of these shebangs:
#!/usr/bin/env python
And they are all over the place. They are:
./site-packages/Bio/bgzf.py:
./site-packages/Bio/PDB/parse_pdb_header.py:
./site-packages/Bio/PDB/PDBList.py:
./site-packages/Bio/Restriction/__init__.py:
./site-packages/Bio/Restriction/Restriction.py:
./site-packages/Bio/Restriction/PrintFormat.py:
./site-packages/Bio/Restriction/Restriction_Dictionary.py:
./site-packages/Bio/Wise/__init__.py:
./site-packages/Bio/Wise/psw.py:
./site-packages/Bio/Wise/dnal.py:
./site-packages/Bio/UniProt/GOA.py:
./site-packages/Bio/SeqUtils/__init__.py:
Regards,
David Mathog
On Mon, Jul 20, 2020 at 4:40 PM John Thorvald Wodder II
<jwodder(a)gmail.com> wrote:
>
> First of all, your last two messages only went to me, not to the list. The mailing list doesn't set Reply-To on messages or the like, so you have to manually set "To: distutils-sig(a)python.org" when replying.
Aargh, right. I use gmail for my home mail, and since I'm stuck
working at home, that is what I used here. Gmail likes to hide, well,
pretty much everything. I will repost those responses.
>
> As to your e-mail, though, are any of those files even meant to be executed? They're not in bin/; they just appear to be regular source files that some developer slapped a shebang on.
That in a sense is the issue. I don't know, you don't know, maybe the
developer knows (if he/she still remembers). I really don't want to
do the work to dig through the code for every package I install to
determine if a shebang is used or not. Yet if I don't figure this out
some end user will run a script (one of a hundred in some package I
installed for their use) which will blow up because of this issue.
The best I can do now is run
pdvctrl reshebang $TARGET_DIR
or
pdvctrl reshebang $ROOT_DIR...
and fix them up after the fact. (pdvctrl from python_devirtualizer here:
https://sourceforge.net/projects/python-devirtualizer/). Even then it
usually has to guess that "python" means "python3" and not "python2",
and sometimes it guesses wrong. Today's version of that recurring
issue:
https://github.com/lastz/lastz/issues/30
Regards,
David Mathog
Hello,
First time trying to use Python's package tools and I'm attempting to use pip to install from a local project:
python3 -m pip install --user --upgrade --no-deps ./
I have a simple directory structure very similar to that recommended in https://packaging.python.org/tutorials/packaging-projects/
project/
data/
mypackage/
__init__.py
data/
*.csv
module1.py
module2.py
tmpl/
*.jinja
setup.py
The project/data/ directory contains several GB of stuff I use during development. It is not supposed to be part of the package and it is not mentioned in setup.py.
The problem is when I run the install command above it takes many minutes to finish. When I move the project/data/ sub-directory out of the project/ directory, it takes a few seconds.
I thought perhaps that "packages=setuptools.find_packages()" in my setup.py file might be the culprit but replacing in with "packages=['mypackage']" made no difference.
I'm not sure why pip seems to be roaming through project/data/ (if that's what's happening) but is there some way to get it not to? My apologies if this is covered in the docs somewhere; I'm still going though them but didn't see anything with an initial scan.
If it helps, here is my setup.py file:
setuptools.setup (
name="mypackage",
version="...",
author="Me",
author_email="...",
description="...",
long_description="...",
long_description_content_type="text/markdown",
url="...",
packages=['mypackage'],
package_data={'mypackage': ['data/*.csv', 'tmpl/*.jinja',]},
zip_safe = False,
classifiers=[...],
python_requires='>=3.6',
install_requires = ['jinja2',], )
Thanks for any enlightenment.
