Currently in the packaging space, we have a number of avenues for communication, which are:
- distutils-sig
- pypa-dev
- virtualenv-users
- Other project specific mailing lists
- IRC
- gitter
- Various issue trackers spread across multiple platforms.
- Probably more places I’m not remembering.
The result of this is that all discussion ends up being super fractured amongst the various places. Sometimes that is exactly what you want (for instance, someone who is working on the wheel specs probably doesn’t care about deprecation policy and internal module renaming in pip) and sometimes that ends up being the opposite of what you want (for instance, when you’re describing something that touches PyPI, setuptools, flit, pip, etc all at once).
Theoretically the idea is that distutils-sig is where cross project reaching stuff goes, IRC/gitter is where real time discussion goes, and the various project mailing lists and issue trackers are where the project specific bits go. The problem is that often times doesn’t actually happen in practice except for the largest and most obvious of changes.
I think our current “communications stack” kind of sucks, and I’d love to figure out a better way for us to handle this that solves the sort of weird “independent but related” set of projects we have here.
From my POV, a list of our major problems are:
* Discussion gets fractured across a variety of platforms and locations, which can make it difficult to actually keep up with what’s going on but also to know how to loop in someone relevant if their input would be valuable. You have to more or less simultaneously know someone’s email, Github username, IRC nick, bitbucket username, etc to be able to bring threads of discussion to people’s attention.
* It’s not always clear to users where a discussion should go, often times they’ll come to one location and need to get redirected to another location. If any discussion did happen in the incorrect location, it tends to need to get restarted in the new location (and depending on the specific platform, it may be impossible to actually redirect everyone over to the proper location, so you again, end up fractured with the discussion happening in two places).
* A lot of the technology in this stack is particularly old, and lacks a lot of the modern day affordances that newer things have. An example is being able to edit a discussion post to fix typos that can hinder the ability of others to actually understand whats being talked about. In your typical mailing list or IRC there’s no mechanism by which you can edit an already sent message, so your only option is to either let the problem ride and hope it doesn’t trip up too many people, or send an additional message to correct the error. However these show up as additional, later messages which someone might not even see until they’ve already been thoroughly confused by the first message (since people tend to read email/IRC in a linear fashion).
- There is a lot of things in this one, other things are things like being able to control in a more fine grained manner what email you’re going to get.
- Holy crap, formatting and typography to make things actually readable and not a big block of plaintext.
* We don’t have a clear way for users to get help, leaving users to treat random issues, discussion areas, etc as a support forum, rather than some place that’s actually optimized for that. Some of this ties back into some of the earlier things too, where it’s hard to actually redirect discussions
These aren’t *new* problems, and often times the existing members of a community are the least effected becasue they’ve already spent effort learning the ins and outs and also curating a (typically custom) workflow that they’ve grown accustomed too. The problem with that is that often times that means that new users are left out, and the community gets smaller and smaller as time goes on as people leave and aren’t replaced with new blood, because they’re driven off but the issues with the stack.
A big part of the place this is coming from, is me sitting back and realizing that I tend to be drawn towards pulling discussions into Github issues rather than onto the varying mailing lists, not because that’s always the most appropriate place for it, but because it’s the least painful place in terms of features and functionality. I figure if I’m doing that, when I already have a significant investment in setting up tooling and being involved here, that others (and particularly new users) are likely feeling the same way.
- Donald
Last week I heard for the first time about the research project FASTEN
https://www.fasten-project.eu/. "FASTEN stands for Fine-Grained Analysis
of Software Ecosystems as Networks."
> instead of analyzing dependencies at the package level, we will analyze them at the call graph level! This will allow us to be super precise when we are tracking dependencies, when we do change impact analysis, when we recommend clients to update packages etc. It will also open the door to new sophisticated applications, e.g. licensing compliance, dependency risk profiling and data-driven API evolution.
That's from the blog post by Georgios Gousios, the PI, at
http://www.gousios.gr/blog/Introducing-Fasten.html . More info:
https://www.fasten-project.eu/view/Main/Introductionhttps://www.fasten-project.eu/view/Main/Overviewhttps://www.fasten-project.eu/view/Main/Contacts
And people who are interested in dynamically and statically analyzing
call graphs in Python may be interested in "Graph Schema and its
representation" in https://www.fasten-project.eu/view/Main/Deliverables .
I've sent a note to FASTEN inviting the team to come talk about their
project here on distutils-sig, because FASTEN's site says they aim to
eventually integrate into PyPI -- I'm not 100% sure whether that means
"create a service that people can use WITH PyPI" or "get FASTEN's work
incorporated into pypi.org".
--
Sumana Harihareswara
pip project manager on contract with Python Software Foundation
Changeset Consulting
https://changeset.nyc
Hi all,
PyPi does not allow duplicate file names -- this makes lots of sense,
because you really don't want people to go to PyPi one day and grab a file,
and then go there another day, grab a file with exactly the same name, and
have it be a different file.
However....
We are all too human, and make mistakes when doing a release. All to often
someone pushed a broken file up to PyPi, often realizes it pretty quickly
-- before anyone has a chance to even download it (or only the dev team as,
for testing...).
In fact, I was in a sprint last summer, and we decided to push our package
up to PyPi -- granted, we were all careless amateurish noobs, but we ended
up making I think 4! minor version bumps because we had done something
stupid in the sdist.
Also -- the latest numpy release got caught in this, too:
"""
* We ran into a problem with pipy not allowing reuse of filenames and a
resulting proliferation of *.*.*.postN releases. Not only were the names
getting out of hand, some packages were unable to work with the postN
suffix.
"""
So -- I propose that PyPi allow projects to replace existing files if they
REALLY REALLY want to.
You should have to jump through all sorts of hoops, and make it really
clear that it is a BAD IDEA in the general case, but it'd be good to have
it be possible.
After all -- PyPi does not take on responsibility for anything else about
what's in those packages, and Python itself is all about "we're all
consenting adults here"
I suppose we could even put in some heuristics about how long the file as
been there, how many times it's been downloaded, etc.
Just a thought.....I really hate systems that don't let me roll back
mistakes, even when I discover them almost immediately...
-CHB
--
Christopher Barker, Ph.D.
Oceanographer
Emergency Response Division
NOAA/NOS/OR&R (206) 526-6959 voice
7600 Sand Point Way NE (206) 526-6329 fax
Seattle, WA 98115 (206) 526-6317 main reception
Chris.Barker(a)noaa.gov
I've got two projects: mynamespace.myprojectA and mynamespace.myprojectB
myprojectB depends on myprojectA. I'm using setuptools 0.6c8 to manage both
projects.
Both projects are registered using 'setup develop'. Both projects are
accessible from an interactive interpreter:
PS C:\Users\me\projects> python
Python 2.5.2 (r252:60911, Feb 21 2008, 13:11:45) [MSC v.1310 32 bit (Intel)]
on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import mynamespace.myprojectA
>>> import mynamespace.myprojectB
>>> from mynamespace.myprojectA import mymoduleZ
However, when I run 'setup test' in myprojectB, the tests fail with
File ".mymoduleZ.py", line NNN, in [some context]
from mynamespace.myprojectA.mymoduleZ import MyClassC
ImportError: No module named myprojectA.mymoduleZ
In setup.py, the test_suite is nose.collector.
I searched and couldn't find anyone else with this problem. Is this a
supported configuration? Is there something I can do to make tests work
with interdependent projects with the same root namespace?
If there's not something obvious I should be doing differently, I'm happy to
put together a minimal test case that reproduces the problem. Any
suggestions are appreciated.
Sincerely,
Jason R. Coombs
The project I am working on now needs to include license files for all of the 3rd party code that it includes. Since license files are not included in distribution packages, the process for doing this is exceedingly complex, error prone, and in some cases, impossible. The build process builds URLs to license files in package source code repositories based on version number, but this is dependent upon several things…
1. The source code repository is accessible via the Internet and still exists.
2. There is a tag for each version (not always true).
3. Tags follow a consistent naming pattern with respect to version numbers (not always true).
I believe that it makes sense to provide a standard means for distribution packages to include license files and then encourage package authors to use it.
Hi,
I'm modifying a python project which uses setuptools for installation.
The project installs an entry point as such:
setup_args["entry_points"] = { "console_scripts": ["w1thermsensor
= w1thermsensor.cli:cli [CLI]"] }
where [CLI] is a reference to an extra_requires entry. This entry
looks like this:
extras_require=dict(CLI="click>=7.0")
With these two lines, the current behavior is that the w1thermsensor
entry point is *always* installed when "setup.py install" is invoked,
but the entry point will cause an error if its dependencies are not
available.
I would like to change the behavior, so that the entry point is *not
installed* if the extra dependencies are not available (and perhaps
also show a warning, indicating that this entry point has not been
installed).
How can I do this?
Cheers,
Jonatan
It seems that the base images used for manylinux wheels are not publicly
available.
For example, https://quay.io/pypa/manylinux2010_centos-6-no-vsyscall gives
a 403 error ("you are not authorized to view this resource").
I think that it would make sense for reproducibility purposes to make these
images public. (I have opened an issue about this on the GitHub repository:
https://github.com/pypa/manylinux/issues/481).
Best,
/Cross-posting this announcement to both distutils-SIG and discourse.
The main discussion thread should be on the discourse
<https://discuss.python.org/t/pycon-us-2020-packaging-summit-registration-an…>./
Greetings one and all,
Though each of us cuts our own path through the harrowing jungle that is
the days of our lives, we are all inevitably drawn closer and closer to
a single fate; yes, of course I am talking about PyCon 2020’s Packaging
Summit! April is looming on the horizon, threatening to take us
unawares, but we are a crafty sort, and have prepared to meet its ambush
with a secret weapon: DILIGENT PREPARATION AND SPREADSHEETS!
In my last announcement, I asked that you save the date: *Thursday,
April 16, 2020*, but provided you no other action items. I am now coming
to you with TWO actions that you can take. /If you don’t know what the
packaging summit is, please see the website for more details
<https://us.pycon.org/2020/hatchery/packaging>./
1. *Register*! This year we will be collecting registration ahead of
time, and like the Language Summit we will be curating the list of
attendees to try and maximize the effectiveness of the limited time
we have for these discussions. In addition to people actually
working on packaging tools, we’re also looking to see people with
many different perspectives participating in the discussions, so if
you are on the fence as to whether or not you qualify but you’re
interested in participating, please register.
You can register at http://bit.ly/python-packaging-summit-2020-reg
Registration is open /now/ and we will close the registration form on
*March 7th, 2020*. All registration decisions should be finished by
*March 15th, 2020*.
2. *Propose topic for discussion*! In previous years, topics were
proposed in the room or ahead of time on discourse. This year, we’ll
be collecting discussion topics with something closer to a private
Call for Proposals, and then releasing all the discussion topics for
public voting at the same time. We’ll be doing it this way to
prevent biasing certain selections based on their position in the
voting list or length of time available to be voted on.
Topic suggestions will be evaluated independent of the submitter; please
propose any topic you would like to see discussed — you don’t have to be
the driver of an idea to propose it. We will schedule about 5 minutes at
the beginning of each discussion period for someone to give a quick
introduction to the topic. By default the person (or people) who
proposed the topic will be responsible for either giving that
introduction or delegating the task to someone else, but if one of your
proposals was selected and you cannot present or find someone to
present, the organizers will find someone to give the introduction.
You may find the topic proposal form at
http://bit.ly/python-packaging-summit-2020-topics
Topic proposal is open /now/ and will close on *March 7th, 2020*. Topic
voting is expected to run from March 9th - March 30th.
This and other information is available at our website:
https://us.pycon.org/2020/hatchery/packaging/
If you have any questions or concerns, feel free to respond on the
discourse
<https://discuss.python.org/t/pycon-us-2020-packaging-summit-registration-an…>
or to contact me or one of the other organizers.