Mailman 3 August 2018 - Distutils-SIG

Re: Packaging Advice for EFF's Certbot
by Brad Warren 06 Aug '18

06 Aug '18

>> On Jul 24, 2018, at 4:36 AM, Nick Coghlan <ncoghlan(a)gmail.com> wrote: >> >> However, there *are* folks that have been working on allowing >> applications to be defined primarily as Python projects, and then have >> the creation of wrapper native installers be a pushbutton exercise, >> rather than requiring careful human handholding. > > But it sounds like they also want to be able to install/remove/upgrade > *parts* of the Python project, for their plugin support. That’s correct. We currently have 18 official plugins for Certbot with plans to add more and a few dozen third-party plugins. > Do any of these tools allow that? This is a good question. If we went with something like dh-virtualenv or packaged virtualenvs with fpm, would we be able to have separate packages for our plugins that installed things in the same virtualenv? I haven’t looked into this yet, but I wouldn’t expect this to work. > That's the thing that really made me think about conda. My biggest concern with conda right now is I believe we (or our users) would be on their own for setting up a cron job or systemd timer for renewal. Is this correct? > On Jul 24, 2018, at 11:20 PM, Chris Jerdonek <chris.jerdonek(a)gmail.com> wrote: > > Just to be clear, I wasn't meaning to promote or recommend the Docker > option I described. Sure! After reading your 2nd email describing how this would work in more detail, I think this would require a pretty major rewrite to how Certbot currently works. Given the other downsides, I’m not sure this is the best approach for us, but I appreciate you throwing out the idea regardless just in case it was! > On Jul 26, 2018, at 3:20 AM, Ben Finney via Distutils-SIG <distutils-sig(a)python.org> wrote: > Just focus on Certbot, and cheer from the sidelines as the OS distributions > do the work of third party packages. > > Yes, that's a different set of problems (for example, keeping Certbot > compatible with those versions of the libraries the OS repositories > provide). We’ve done the work to maintain compatibility with the older versions of our dependencies available in the official OS repos where we are packaged. The source of our problems with official OS packaging is described below and in the Google Doc linked in my initial email. > On Jul 28, 2018, at 8:53 AM, matthew(a)woodcraft.me.uk wrote: > > I wouldn't be too put off by the idea of Debian politics. Certbot should be a good fit for stable-updates: We thought so too and getting updates like this was our main packaging plan when we launched in 2015. Unfortunately, it hasn’t gone well and is the main reason we’re seeking our own solution. Perhaps we did something incorrectly, but as Nathaniel pointed out, Certbot is broken in Debian Stretch and has been since January. The same and many other problems affect the packages in Ubuntu Xenial. We’ve also struggled to find people to help maintain our PPA for older, non-EOL’d versions of Ubuntu. If anyone reading this would like to help us solve these problems or know someone who would, please reach out off-list. While these packages exist in OS repos, some users will continue to use them regardless of the alternative packaging approach we take. Unless the current issues are resolved and we’re confident new ones in the future will be fixed quickly as well, I think we need to offer alternative packaging so we can provide our users some means of getting a working version of Certbot. > On Jul 28, 2018, at 12:00 PM, Wes Turner <wes.turner(a)gmail.com> wrote: > > Took a minute more to read the gdoc link. Wes Turner, while I don’t have any specific questions or comments right now, thank you very much for all the ideas and links.

3 3

Re: Proposal for timestamps of installed files
by Jeroen Demeyer 06 Aug '18

06 Aug '18

On 2018-08-06 19:49, Chris Jerdonek wrote: > Does this proposal mean that .pyc files would need to be regenerated > after installing from a wheel (because the timestamp inside the .pyc > file would no longer match the .py file from which it came)? No, I specifically mentioned that the timestamps of .py files would be preserved.

1 0

Proposal for timestamps of installed files
by Jeroen Demeyer 06 Aug '18

06 Aug '18

Proposal: change timestamps of files installed from a wheel to the time of installation, except for .py files. **Existing situation** Let's consider two ways of installing a Python package: installation from a source tarball and installation from a wheel, in both cases using "pip install". The installation roughly works as follows: (A) Create a temporary directory for pip to work in, with a filename like /tmp/pip-6N_2Kx-build (B) Extract the archive (sources or wheel) somewhere in the temporary directory (C) Build the package in the temporary directory (in both cases, this involves creating .pyc files for the .py files) (D) Copy the installation from the temporary directory to the final installation directory (typically somewhere in site-packages) (E) Delete the temporary directory Both (B) and (D) involve a copy operation of some sort, so the question is: what to do with timestamps? In both cases, (D) currently preserves the timestamp of the files. For (B), timestamps are preserved when extracting a tarball but not when extracting a wheel. **Proposal** Change step (D) for wheel installs to NOT preserve timestamps. It might be safer to make sure that all installed files have the same timestamp, so this would mean: give all installed files the timestamp of a fixed time, namely the time when step (D) was started in the installation process. Important exception: for .py files, do preserve the timestamp. I think that (B) should preserve timestamps. However, that is a different issue which should be fixed independently. **The problem** For a source installation, the timestamps are preserved in both (B) and (D), so the timestamp of the installed files can be much older than the time of installation. This is a problem for dependency checking: suppose that I am the developer of a package MYPKG which has a dependency on the C API of a package OTHERPKG. The C API of OTHERPKG is defined by .h files which are installed by OTHERPKG. The package MYPKG correctly declares this dependency using the "depends" keyword of the Extension objects. Now, when I install a different version of OTHERPKG, I want that distutils notices that the dependency has been changed and automatically rebuilds the Extensions with that dependency. This requires that the timestamp of the installed .h files is more recent than the timestamp of the compiled extension. In other words, the timestamp of installed files should be the time of installation. A similar situation happens for a Cython API exposed by installed .pxd (and possibly also .h) files. **Comparison with automake** Automake (which is probably the most widely used build system for open source/free software) installs files with timestamp the time of installation. So, my proposal is doing what automake already does. **The strange case of .pyc files** As noted in https://github.com/pypa/pip/issues/5648 it is important that the timestamp of .py files are preserved after the .pyc file is created. This is because Python checks the exact timestamp of the .py file to determine whether it is up to date. This explains the exception for .py files in the proposal. **Backwards compatibility** In other words: won't this break packages which rely on timestamps to be preserved? When installing a wheel, timestamps from the wheel are already not preserved in step (B). So any package which would rely on preserving timestamps would already be broken when installed as wheel. So this proposal can only break stuff which depends on timestamps from step (C), which happens to be the case for .py files. In my proposal, I'm only suggesting to make changes for wheel installs. I do think that the change makes sense for installs from source, but that will be irrelevant after PEP 517 is implemented anyway. Jeroen.

2 1

Re: PEP 541 Regarding emd
by Robert Collins 05 Aug '18

05 Aug '18

On 4 August 2018 at 03:34, Chathika Gunaratne <Chathika.Gunaratne(a)ucf.edu> wrote: > Dear Admin, > > > > I am writing to you regarding the Package Index name emd. I am a PhD student > at the University of Central Florida and for my dissertation, I have > recently developed a Python software, Evolutionary Model Discovery, which I > am assembling into a package, EMD. https://github.com/chathika/EMD > > > > However, I noticed that there is already a package named emd on pypi.org. > This package seems abandoned with just one release in February 2012 and no > contact information about the author, > https://pypi.org/project/emd/1.0/#history . > > > > I read about PEP 541 https://www.python.org/dev/peps/pep-0541/ and was > curious as to if it is still possible that I can acquire the package name > EMD. I wanted to make sure of this before I went ahead with the name EMD for > my package, please. Generally speaking its not whether a package has been updated that matters, its whether the package is in use: if its published in debian/suse/fedora, if its getting downloads, then taking over the name is going to have negative consequences. That said, this has had 0 downloads per the bigtable statistics (0 for all recorded time) so in this case that seems unlikely. -Rob

2 1

packaging guide and private packages index
by David Cournapeau 04 Aug '18

04 Aug '18

Hi, I see that there is almost no mention of private packages index in the packaging guide, and no recommendation on what to use. Currently googling for private packages mostly return obsolete (and not very practical) recommendations based on dependency links. In 2018, what would be the recommended practices for teams that develop private packages that depend on each other ? thank you, David

9 9

Re: Timestamp of installed files
by Jeroen Demeyer 04 Aug '18

04 Aug '18

On 2018-08-04 14:02, Thomas Kluyver wrote: > On Sat, Aug 4, 2018, at 9:34 AM, Paul Moore wrote: >> Whether timestamps are >> preserved by the wheel building process depends on the build system - >> so the question boils down to "does setup.py bdist_wheel preserve >> timestamps?" in the case of the setuptools backend - which is really a >> question for the wheel project. In the more general case, you'd have >> to ask the same question of flit, and any other backends you cared >> about. > > IIRC, Flit will preserve the timestamps of the files when you build a wheel For the record: my post wasn't about *building* a wheel, but about *installing* a wheel.

3 2

Re: Timestamp of installed files
by Jeroen Demeyer 04 Aug '18

04 Aug '18

On 2018-08-04 00:02, Chris Jerdonek wrote: > I'm not sure how relevant it is, but this issue was recently filed on > pip's issue tracker ("Reproducible installs"): > https://github.com/pypa/pip/issues/5648 This seems to be about preserving timestamps when extracting wheel files. > There is also an older (closed) pip issue that might be relevant > ("Preserving timestamps on copy"): > https://github.com/pypa/pip/issues/3201 This is a different issue about preserving timestamps from the source directory to the temporary build directory. So both are different issues, and I agree with both: during the source extraction and build process, you want to preserve timestamps as much as possible. But for the installation, you do NOT want to preserve timestamps.

4 4

Re: Timestamp of installed files
by Jeroen Demeyer 04 Aug '18

04 Aug '18

On 2018-08-04 13:16, Paul Moore wrote: > Can you give a > specific example of an end to end process where the packaging > toolchain's current behaviour gives demonstrably the wrong result? Yes I can. I will work out a detailed proposal about timestamps in the build/install process in general. But please give me some time to write it up.

2 1

Re: Timestamp of installed files
by Jeroen Demeyer 04 Aug '18

04 Aug '18

The fact that installs from wheels don't preserve timestamps is a very good argument that it's OK to NOT preserve timestamps in general. Any package which would rely on preserving timestamps would already be broken when installing using a wheel. By the way, initially I thought that this was a distutils issue, but it's clear now that it's really a pip issue. Jeroen.

2 1

Re: Timestamp of installed files
by Jeroen Demeyer 04 Aug '18

04 Aug '18

On 2018-08-04 10:34, Paul Moore wrote: > Jeroen seemed to say he agreed with this, but > I'm not sure I see how that matches his stated requirement for > installs to not preserve timestamps... The way how pip currently works (I assume that this stays the case) is that it uses a temporary build directory for everything, for example /tmp/pip-build-uvfWuR So concretely for a wheel install: (A) Extract the wheel in build directory (B) Generate .pyc files in build directory (C) Copy files from build directory to final installation directory in site-packages. The way I understand it, https://github.com/pypa/pip/issues/5648 is about (A) while my proposal is about (C). So I think that timestamps should be preserved in (A) but not for (C). Jeroen.

2 1