Mailman 3 - Distutils-SIG

RE:Re: approved screensaver
by MAILsweeper＠iena.com 29 Apr '18

29 Apr '18

Bonjour, Ceci est une réponse automatique. Le message de l'expéditeur distutils-sig(a)python.org et à destination de dirgen(a)journal-lunion.fr contient un virus. LE VIRUS A ETE SUPPRIME.

1 0

Your mail to tcm-users-request@listserv.cs.utwente.nl
by tcm-users-request＠cs.utwente.nl 29 Apr '18

29 Apr '18

This pre-recorded message is being sent in response to your recent email to tcm-users-request(a)listserv.cs.utwente.nl. All routine administrative requests (including subscriptions and unsubscriptions) concerning this mailing list are handled by an automated server. Please read this message carefully to find the information relevant to you. SUBSCRIBING =========== To subscribe to tcm-users, send the following in the body (not the subject line) of an email message to "Majordomo(a)listserv.cs.utwente.nl": subscribe tcm-users This will subscribe the account from which you send the message to the tcm-users list. If you wish to subscribe another address instead (such as a local redistribution list), you can use a command of the form: subscribe tcm-users other-address(a)your_site.your_net UNSUBSCRIBING ============= To unsubscribe from tcm-users, send the following in the body (not the subject line) of an email message to "Majordomo(a)listserv.cs.utwente.nl": unsubscribe tcm-users This will unsubscribe the account from which you send the message. If you are subscribed with some other address, you'll have to send a command of the following form instead: unsubscribe tcm-users other-address(a)your_site.your_net If you don't know what address you are subscribed with, you can send the following command to see who else is on the list (assuming that information isn't designated "private" by the owner of the list): who tcm-users If you want to search non-private lists at this server, you can do that by sending a command like: which string This will return a list of all entries on all lists that contain "string". HELP ==== To find out more about the automated server and the commands it understands, send the following command to "Majordomo(a)listserv.cs.utwente.nl": help If you feel you need to reach a human, send email to: tcm-users-approval(a)listserv.cs.utwente.nl

1 0

Re: [Distutils] distutils install probs on suse 9.0
by Mats Wichmann 29 Apr '18

29 Apr '18

At 05:43 AM 5/18/2004, David Handy wrote: >On Tue, 18 May 2004, Floris van der Tak wrote: > >> my attempt to install the distutils on suse 9.0 (which provides python >> 2.3) fails: > >Why are you installing distutils on a system that already has Python 2.3 >installed? Python 2.3 normally comes with distutils as part of the Python >standard library. > >Did suse distribute a version of Python that had distutils removed? It should come from python-devel

2 1

Warning: E-mail viruses detected
by MailScanner 29 Apr '18

29 Apr '18

Our virus detector has just been triggered by a message you sent:- To: <hinsen(a)cnrs-orleans.fr> Subject: Re: Your application Date: Sun Aug 24 22:37:08 2003 One or more of the attachments are on the list of unacceptable attachments for this site and will not have been delivered. Consider renaming the files or putting them into a "zip" file to avoid this constraint. The virus detector said this about the message: Report: Shortcuts to MS-Dos programs are very dangerous in email in details.pif -- MailScanner Email Virus Scanner

1 0

Sample setup.py for Numeric Python
by Greg Ward 29 Apr '18

29 Apr '18

Hi again -- [cc'd to Paul Dubois: you said you weren't following the distutils sig anymore, but this directly concerns NumPy and I'd like to get your input!] here's that sample setup.py for NumPy. See below for discussion (and questions!). ------------------------------------------------------------------------ #!/usr/bin/env python # Setup script example for building the Numeric extension to Python. # This does sucessfully compile all the .dlls. Nothing happens # with the .py files currently. # Move this file to the Numerical directory of the LLNL numpy # distribution and run as: # python numpysetup.py --verbose build_ext # # created 1999/08 Perry Stoll __rcsid__ = "$Id: numpysetup.py,v 1.1 1999/09/12 20:42:48 gward Exp $" from distutils.core import setup setup (name = "numerical", version = "0.01", description = "Numerical Extension to Python", url = "http://www.python.org/sigs/matrix-sig/", ext_modules = [ ( '_numpy', { 'sources' : [ 'Src/_numpymodule.c', 'Src/arrayobject.c', 'Src/ufuncobject.c' ], 'include_dirs' : ['./Include'], 'def_file' : 'Src/numpy.def' } ), ( 'multiarray', { 'sources' : ['Src/multiarraymodule.c'], 'include_dirs' : ['./Include'], 'def_file': 'Src/multiarray.def' } ), ( 'umath', { 'sources': ['Src/umathmodule.c'], 'include_dirs' : ['./Include'], 'def_file' : 'Src/umath.def' } ), ( 'fftpack', { 'sources': ['Src/fftpackmodule.c', 'Src/fftpack.c'], 'include_dirs' : ['./Include'], 'def_file' : 'Src/fftpack.def' } ), ( 'lapack_lite', { 'sources' : [ 'Src/lapack_litemodule.c', 'Src/dlapack_lite.c', 'Src/zlapack_lite.c', 'Src/blas_lite.c', 'Src/f2c_lite.c' ], 'include_dirs' : ['./Include'], 'def_file' : 'Src/lapack_lite.def' } ), ( 'ranlib', { 'sources': ['Src/ranlibmodule.c', 'Src/ranlib.c', 'Src/com.c', 'Src/linpack.c', ], 'include_dirs' : ['./Include'], 'def_file' : 'Src/ranlib.def' } ), ] ) ------------------------------------------------------------------------ First, what d'you think? Too clunky and verbose? Too much information for each extension? I kind of think so, but I'm not sure how to reduce it elegantly. Right now, the internal data structures needed to compile a module are pretty obviously exposed: is this a good thing? Or should there be some more compact form for setup.py that will be expanded later into the full glory we see above? I've already made one small step towards reducing the amount of cruft by factoring 'include_dirs' out and supplying it directly as a parameter to 'setup()'. (But that needs code not in the CVS archive yet, so I've left the sample setup.py the same for now.) The next thing I'd like to do is get that damn "def_file" out of there. To support it in MSVCCompiler, there's already an ugly hack that unnecessarily affects both the UnixCCompiler and CCompiler classes, and I want to get rid of that. (I refer to passing the 'build_info' dictionary into the compiler classes, if you're familiar with the code -- that dictionary is part of the Distutils extension-building system, and should not propagate into the more general compiler classes.) But I don't want to give these weird "def file" things standing on the order of source files, object files, libraries, etc., because they seem to me to be a bizarre artifact of one particular compiler, rather than something present in a wide range of C/C++ compilers. Based on the NumPy model, it seems like there's a not-too-kludgy way to handle this problem. Namely: if building extension "foo": if file "foo.def" found in same directory as "foo.c" add "/def:foo.def" to MSVC command line this will of course require some platform-specific code in the build_ext command class, but I figured that was coming eventually, so why put it off? ;-) To make this hack work with NumPy, one change would be necessary: rename Src/numpy.def to Src/_numpy.def to match Src/_numpy.c, which implements the _numpy module. Would this be too much to ask of NumPy? (Paul?) What about other module distributions that support MSVC++ and thus ship with "def" files? Could they be made to accomodate this scheme? Thanks for your feedback -- Greg -- Greg Ward - software developer gward(a)cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913

4 5

Builtin modules as external pyds
by Anthony Pfrunder 29 Apr '18

29 Apr '18

Hi, I'm in the middle of testing zmake and have run into an interesting problem. I assume that the problem is not with my code... but who knows ;). Compiler plugin is MSVC on windows 95. I'm compiling a super-small python lib with most modules external (such as array etc) so that I can pick'n'choose the python size. I have two zmake rules: Python-lib-modules-static and Python-lib-modules-dynamic. If you add the setup rule (ie array) to the first it is bound into the pythonlib without problems (adds it to config.c etc etc). If I add array to Python-lib-modules-dynamic it compiles but the link fails because: Unresolved external symbol _PyExc_ValueError... However, if I add the Zope Extensions to the python-lib-modules-dynamic rule then they compile and link fine. They of cause have plenty of references to PyExc_ValueError. The generated pythonlib (ex array etc) works fine with a pre-compiled python binary. I note in the Setup.in in Modules array is underneath a commented out *shared*. My question is: can array and friends be compiled as pyd's or do I have to add some Windows magic to them to get it to work? Cheers, Anthony Pfrunder

1 0

MSVC, ming and cygwin plugins
by Anthony Pfrunder 29 Apr '18

29 Apr '18

Hi, If you want some code for MSVC, ming and cygwin try: http://www.zope.org:1800/Members/NoneToBe/Zwin This contains a copy of Zmake which has rules for building python, the python library, a python console, extensions in Setup files, extensions in unix makefiles (simple ones only) and extensions in dsw (yes, it imports MSVC 6.0+ project files to zmake format!). It is also integreted with Gorden's freezing system. Simply "make" all the scripts you want to freeze and the system does the rest. Frozen scripts are placed in MPZ at the toplevel directory. It is supposed to work with the listed compilers but since I use VC only they may be broken. Next week I'll put up a new release with some documentation and a revamped python library builder. The new one generates a correct config.c based on which modules you want in the library. It has a SuperFreeze option where you point it to a directory containing a bunch of Setup files and a start script. It will then freeze everything - scripts, extensions, python library and a console into the one binary ;) Cheers, Anthony Pfrunder

1 0

Re: [Distutils] Can now compile simple extensions
by David Ascher 29 Apr '18

29 Apr '18

On Thu, 19 Aug 1999, Perry A. Stoll wrote: > Nice work, Greg! I thought I'd let David take a break on this one, so > attached is an MSVCCompiler, which works for me using MSVC 5.0. Not much has > changed on the command line usage for MSVC 6.0, so this should probably work > there also. Great! It's a very good thing for Greg given his talk on Monday! A very good thing for me (for obvious reasons), and, of course, a very good thing for Python users worldwide. Thanks, Perry! --david

1 0

Can now compile simple extensions
by Greg Ward 29 Apr '18

29 Apr '18

Hi all -- at long last, I found the time to hack in the ability to compile extension modules to the Distutils. Mainly, this meant adding a 'build_ext' command which uses a CCompiler instance for all its dirty work. I also had to add a few methods to CCompiler (and, of course, UnixCCompiler) to make this work. And I added a new module, 'spawn', which takes care of running sub-programs more efficiently and robustly (no shell involved) than os.system. That's needed, obviously, so we can run the compiler! If you're in the mood for grubbing over raw source code, then get the latest from CVS or download a current snapshot. See http://www.python.org/sigs/distutils-sig/implementation.html for a link to the code snapshot. I'm still waiting for more subclasses of CCompiler to appear. At the very least, we're going to need MSVCCompiler to build extensions on Windows. Any takers? Also, someone who knows the Mac, and how to run compilers programmatically there, will have to figure out how to write a Mac-specific concrete CCompiler subclass. The spawn module also needs a bit of work to be portable. I suspect that _win32_spawn() (the intended analog to my _posix_spawn()) will be easy to implement, if it even needs to go in a separate function at all. It looks from the Python Library documentation for 1.5.2 that the os.spawnv() function is all we need, but it's a bit hard to figure out just what's needed. Windows wizards, please take a look at the 'spawn()' function and see if you can make it work on Windows. As for actually compiling extensions: well, if you can figure out the build_ext command, go ahead and give it a whirl. It's a bit cryptic right now, since there's no documentation and no example setup.py. (I have a working example at home, but it's not available online.) If you feel up to it, though, see if you can read the code and figure out what's going on. I'm just hoping *I'll* be able to figure out what's going on when I get back from the O'Reilly conference next week... ;-) Enjoy -- Greg -- Greg Ward - software developer gward(a)cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913

2 1

Zmake build system
by Anthony Pfrunder 29 Apr '18

29 Apr '18

Hi, As part of the Zope project I've constructed a python build system which uses python itself and works cross-platform. Currently, only Visual C is implemented but in the next few weeks plug-ins for cygwin and mingw32 should come online. I would love to merge this in with dist-util so I can go back to playing with ZClasses ;) DESCRIPTION The system consists of: Make.py Python script which reads makefiles and 'runs' can import Setup files (for building extensions) and generic makefiles. Will soon be able to read ./configure output to get the variables Makefile.zmake The platform generic makefile, recursively loads Setup files and then builds a python console, lib and pyd's (choose your targets) msvc\msvc.zmake Platform stuff for visual c. Makefile hooks into generic one by adding stuff to dep lines and defing new rules. Also includes custom rc, ico files (ie all the stuff that is in the PC build dir) A simple rule (building the parser module): parser: os.path.join(PythonHome, 'Modules', 'parsermodule.obj') parser.pyd Note: you can recursively import Setup files and normal unix makefiles so you don't need to change anything (eventually...) This will compile parsermodule.c (dumping temp stuff including def files into directory pointed to by TMP) and link it to generate a pyd. The pyd will be placed in OUTDIRE for you. See the python-lib, python-lib-sources rules for a more complex example. (note, the PC specific stuff is added when msvc.zmake is loaded) Beware that it is pre-alpha and is release only to provide a tool to build Zope Extensions in VC. You can grab a snapshot at: http://student.uq.edu.au/~s341625/ZwinG2.zip Currently, it can build python-lin, console and extensions under Visual C Cheers, Anthony Pfrunder

1 0

inertia
by David Ascher 29 Apr '18

29 Apr '18

I'm reorganizing the PyOpenGL source tree. I'd love to move to a distutils distribution process instead of my current complicated homegrown system. However, I'm not willing to do that without the ability to generate .so's simply on Unices (I can do the win32 portion, see below). Is anyone working on this? I'd hoped to have the distribution story moved along further by the time the ORA conference rolls around, but it's not looking likely. Here's a dare: if someone does the Unix .c->.so->'simple installation' process, I'll do the win32 equivalent. My guess is that the first step is a pseudo-parse of the Makefile.pre.in file, no? Once the compilation process is in place (even buggy), it makes it more likely for others to contribute snippets for e.g. RPM generation, etc. --david

1 0

Two fixes (at last)
by Greg Ward 29 Apr '18

29 Apr '18

Hi all -- at long last, I have fixed two problems that a couple people noticed a while ago: * I folded in Amos Latteier's NT patches almost verbatim -- just changed an `os.path.sep == "/"' to `os.name == "posix"' and added some comments bitching about the inadequacy of the current library installation model (I think this is Python's fault, but for now Distutils is slavishly aping the situation in Python 1.5.x) * I fixed the problem whereby running "setup.py install" without doing anything else caused a crash (because 'build' hadn't yet been run). Now, the 'install' command automatically runs 'build' before doing anything; to make this bearable, I added a 'have_run' dictionary to the Distribution class to keep track of which commands have been run. So now not only are command classes singletons, but their 'run' method can only be invoked once -- both restrictions enforced by Distribution. The code is checked into CVS, or you can download a snapshot at http://www.python.org/sigs/distutils-sig/distutils-19990607.tar.gz Hope someone (Amos?) can try the new version under NT. Any takers for Mac OS? BTW, all parties involved in the Great "Where Do We Install Stuff?" Debate should take a good, hard look at the 'set_final_options()' method of the Install class in distutils/install.py; this is where all the policy decisions about where to install files are made. Currently it apes the Python 1.5 situation as closely as I could figure it out. Obviously, this is subject to change -- I just don't know to *what* it will change! Greg -- Greg Ward - software developer gward(a)cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913

2 1

Package versioning
by Timothy Docker 29 Apr '18

29 Apr '18

Hi all, I've been aware that the distutils sig has been simmerring away, but until recently it has not been directly relevant to what I do. I like the look of the proposed api, but have one question. Will this support an installed system that has multiple versions of the same package installed simultaneously? If not, then this would seem to be a significant limitation, especially when dependencies between packages are considered. Assuming it does, then how will this be achieved? I am presently managing this with a messy arrangement of symlinks. A package is installed with its version number in it's name, and a separate directory is created for an application with links from the unversioned package name to the versioned one. Then I just set the pythonpath to this directory. A sample of what the directory looks like is shown below. I'm sure there is a better solution that this, and I'm not sure that this would work under windows anyway (does windows have symlinks?). So, has this SIG considered such versioning issues yet? Cheers, Tim -------------------------------------------------------------- Tim Docker timd(a)macquarie.com.au Quantative Applications Division Macquarie Bank -------------------------------------------------------------- qad16:qad $ ls -l lib/python/ total 110 drwxr-xr-x 2 mts mts 512 Nov 11 11:23 1.1 -r--r----- 1 root mts 45172 Sep 1 1998 cdrmodule_0_7_1.so drwxr-xr-x 2 mts mts 512 Sep 1 1998 chart_1_1 drwxr-xr-x 3 mts mts 512 Sep 1 1998 Fnorb_0_7_1 dr-xr-x--- 3 mts mts 512 Nov 11 11:21 Fnorb_0_8 drwxr-xr-x 3 mts mts 1536 Mar 3 12:45 mts_1_1 dr-xr-x--- 7 mts mts 512 Nov 11 11:22 OpenGL_1_5_1 dr-xr-x--- 2 mts mts 1024 Nov 11 11:23 PIL_0_3 drwxr-xr-x 3 mts mts 512 Sep 1 1998 Pmw_0_7 dr-xr-x--- 2 mts mts 512 Nov 11 11:21 v3d_1_1 qad16:qad $ ls -l lib/python/1.1 total 30 lrwxrwxrwx 1 root other 29 Apr 10 10:43 _glumodule.so -> ../OpenGL_1_5_1/_glumodule.so lrwxrwxrwx 1 root other 30 Apr 10 10:43 _glutmodule.so -> ../OpenGL_1_5_1/_glutmodule.so lrwxrwxrwx 1 root other 22 Apr 10 10:43 _imaging.so -> ../PIL_0_3/_imaging.so lrwxrwxrwx 1 root other 36 Apr 10 10:43 _opengl_nummodule.so -> ../OpenGL_1_5_1/_opengl_nummodule.so lrwxrwxrwx 1 root other 27 Apr 10 10:43 _tkinter.so -> ../OpenGL_1_5_1/_tkinter.so lrwxrwxrwx 1 mts mts 21 Apr 10 10:43 cdrmodule.so -> ../cdrmodule_0_7_1.so lrwxrwxrwx 1 mts mts 12 Apr 10 10:43 chart -> ../chart_1_1 lrwxrwxrwx 1 root other 12 Apr 10 10:43 Fnorb -> ../Fnorb_0_8 lrwxrwxrwx 1 mts mts 12 Apr 10 10:43 mts -> ../mts_1_1 lrwxrwxrwx 1 root other 15 Apr 10 10:43 OpenGL -> ../OpenGL_1_5_1 lrwxrwxrwx 1 root other 33 Apr 10 10:43 opengltrmodule.so -> ../OpenGL_1_5_1/opengltrmodule.so lrwxrwxrwx 1 root other 33 Apr 10 10:43 openglutil_num.so -> ../OpenGL_1_5_1/openglutil_num.so lrwxrwxrwx 1 root other 10 Apr 10 10:43 PIL -> ../PIL_0_3 lrwxrwxrwx 1 mts mts 10 Apr 10 10:43 Pmw -> ../Pmw_0_7 lrwxrwxrwx 1 root other 10 Apr 10 10:43 v3d -> ../v3d_1_1

3 2

Re: Access to Python config info
by David Ascher 29 Apr '18

29 Apr '18

Robin: > Another vote for impeac^H^H^H^H^H^Ha separate .py file to store this. > Mostly it's better since it allows tweaks to be made easily, (say the > installpath changes, or a new compiler), a hundred different little > things can happen at a local site. It'll also help during the > development phases of this tool to have visibility into what's actually > happening. Same here. Furthermore, one can imagine writing a program which generates sysconfig.py for distributed heteregeneous systems, or even writing a sysconfig.py module which *does stuff* at import time. Why limit ourselves... --david

1 0

[Distutils-sig] distutils charter and interscript
by John Skaller 29 Apr '18

29 Apr '18

3 2

Re: [Distutils] Improving Communication
by Steve Piercy - Website Builder 27 Apr '18

27 Apr '18

My first impression of Zulip was "this feels like Slack". It is visually busier than Discourse, and I had a harder time understanding context. I had to step up the font twice to read it. I couldn't find how to hide the user list. I felt lost. Based on my first impressions, I probably wouldn't engage via Zulip. I had a negative first impression with Discourse, too, as "yet another forum software", but it grew on me. Dozens of UX details were done just right, like "I wonder how I can do this thing... oh, there it is!". Or it could have been the Plone community is engaging, or a wealth of information already existed by the time I joined. --steve On 4/22/18 at 1:19 PM, web(a)stevepiercy.com (Steve Piercy - Website Builder) pronounced: >I am an email and IRC recidivist. I have to admit that ever >since I joined the Plone community forum which uses Discourse, >I have been more engaged with that community than I would have >otherwise. I can choose which topics interest me, forget the >rest, and explore things that might interest me at some point >in the future. It also has a wealth of information easily >searchable and accessible. > >I noticed that my wireless carrier uses Discourse as well, and >I've found it exceptionally helpful to solve my phone problems >quickly, compared to the phone manufacturer's craptastic help experience. > >I haven't tried Zulip yet, but I just signed up to compare it to Discourse. > >I don't know what challenges might be for self-hosting. >Perhaps those who have experience with self-hosting either >Discourse or Zulip could share? > >--steve > > >On 4/21/18 at 11:51 PM, brett(a)python.org (Brett Cannon) pronounced: > >>I know this is no shock to Donald, but I agree with what he brings up >>below. One of motivators for trying out python.zulipchat.com is to see if >>it's real-time nature on top of topic-based discussion could act as a >>cross-section for email and IRC. >> >>For me, either something like Zulip or Discourse takes care of a lot of the >>issues raised. They provide modern tooling, allow for a central place to >>communicate, and allow for sectioning things into streams/sections to >>prevent overwhelming e.g. a single mailing list. > >-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- >Steve Piercy Website Builder Eugene, OR ><web(a)stevepiercy.com> <http://www.stevepiercy.com/> -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- Steve Piercy Website Builder Eugene, OR <web(a)stevepiercy.com> <http://www.stevepiercy.com/>

2 1

How to eliminate on part of a package?
by Skip Montanaro 27 Apr '18

27 Apr '18

I recently ported a package to Python 3. The overall structure is pretty straightforward: top/ client/ module.py server/ server.py There's more to it, but that's enough for demonstration. I am retaining Python 2.7 compatibility on the client side, but have dispensed with that business on the server. (I run the server, not my users.) Accordingly, I would like the py27 version of the package to not have a subpackage named "server". I thought it would be easy peasy to just trim the server bits from setuptools.find_packages() in my setup.py file (I have 36.5.0.post20170921 of setuptools), something like this: packages = setuptools.find_packages() if sys.version_info.major < 3: packages.remove("top.server") In either case, I pass the resulting list as the packages argument of setuptools.setup(...). That doesn't work though. I get some sort of mish-mash where part of the top/server tree is copied, part not. I eventually get an error from a cp command, something about ".../server is not a directory". It would seem there is more I need to do, but nothing jumped out at me in the online documentation (http://setuptools.readthedocs.io/en/latest/setuptools.html). FWIW, while I see a section on that page titled, "New and Changed setup() Keywords", I didn't see a corresponding "Old and Unchanged setup() Keywords" there or up a level. Any pointers, hints, or suggestions appreciated... Skip Montanaro

4 6

RFH with a problem with pip 10 on Windows (AppVeyor)
by Lele Gaifax 25 Apr '18

25 Apr '18

Hi, since some time, the CI job on AppVeyor for a package I maintain fails in a way I was not able to reproduce in my local demo Win7 VM. This is a link to the error: https://ci.appveyor.com/project/lelit/python-rapidjson/build/1.0.184#L66 and as you can see, it happens when cibuildwheel tries to install the "wheel" package, and the following happens: + pip install wheel Using default MSVC build environment for 32 bit architecture Executing: pip install wheel Traceback (most recent call last): File "c:\python34\lib\runpy.py", line 170, in _run_module_as_main "__main__", mod_spec) File "c:\python34\lib\runpy.py", line 85, in _run_code exec(code, run_globals) File "C:\Python34\Scripts\pip.exe\__main__.py", line 5, in <module> ImportError: cannot import name 'main' The problem may be related to something different in version 10 of pip, but that's just a guess: as always, on Windows there are no certainties :-) It used to work as expected, this is a link to latest successful run: https://ci.appveyor.com/project/lelit/python-rapidjson/build/1.0.181?fullLo… and since then the only (apparent) difference is the pip version... Can anybody understand what's going on? Thanks in advance, ciao, lele. -- nickname: Lele Gaifax | Quando vivrò di quello che ho pensato ieri real: Emanuele Gaifas | comincerò ad aver paura di chi mi copia. lele(a)metapensiero.it | -- Fortunato Depero, 1929.

3 4

[improvement proposal] Manage projects with namespaces
by herveberaud.pro＠gmail.com 25 Apr '18

25 Apr '18

Hi everybody! I want to propose an improvement to distutils and for python packaging management. *Overview* This is a feature proposal. Now when a project is already register on pypi it's not possible to users to test a fork of any projects with the same name when it's already exist, manage projects by namespace increase possiblities for the python community. With this feature we can introduce trusted packages by allow install/search without namespace and add namespaces on untrusted packages like docker behavior (docker pull nginx or docker pull 4383/nginx). On docker when the package is trusted (docker trusted image mean maintained by docker itself), namespace does not exist, and when a package is maintain by a third user namespace appear into the name. I don't want delegate official projects maintainance to the pypa team but we can introduce a vote system for the community and remove namespace when project obtain a certain number of votes from the community (users). I've already propose this feature to the pypa warehouse ( https://github.com/pypa/warehouse/issues/2589). Pypa team like this idea and we want to move forward so now we want your point of view about this. We can formalize this feature with a PEP for describe more formerly the behaviors etc... *Features* - Allow users to vote for trust project and allow download (install, search, etc...) without prefix with user namespace - Allow users to upload on pypi project with a name who already exist on pypi but prefixed by user namespace. *Benefits* - Improve project trust - Improve package trusting and discrease risk that users deal with a miscellaneous package come from a typo squatting example 1 <http://www.nbu.gov.sk/skcsirt-sa-20170909-pypi/index.html>, pypa github discussion <https://github.com/pypa/pypi-legacy/issues/644> - Allow users to provide forked version of an official project - Allow users to test that packaging work fine on pypi *Examples* With pip: $ pip install Django # trusted package $ pip install 4383/Django # untrusted package Url transposition: - https://pypi.org/project/Django/ - https://pypi.org/project/4383/Django/ *That all folks!* I hope this feature interesting you and you can consider this! Do not hesitate to ask me more questions and to ask me more descriptions! It's with pleasure that I want to help you (pypa, distutils) to implement this. All the best!

1 0

PyPI update: legacy shutdown 30 April, new classifiers page, seeking funding
by Sumana Harihareswara 25 Apr '18

25 Apr '18

Almost the end. On Monday April 30th we're going to shut down https://legacy.pypi.org/ . The URL pypi.python.org will continue to redirect to Warehouse (pypi.org). As you can see from https://status.python.org/ , Warehouse has been holding up well, and we don't see any reason to delay the shutdown of Legacy. If you need to compare new Warehouse behavior with old Legacy behavior, tell us about a redirect that isn't working right, etc., please do that this week. Older versions of JFrog's Artifactory have trouble with the pypi.python.org redirect. Users whose instances proxy/mirror PyPI should upgrade before April 30th. https://www.jfrog.com/jira/browse/RTFACT-16223?focusedCommentId=54641&page=… (more context[1]) We've been fixing up search[2], dealing with memory consumption[3] and reliability, adding metrics and monitoring, replying to user issues, reviewing volunteers' contributions, and improving PyPI admins' ability to do things like deprecate classifiers[4]. Check out the new page listing classifiers and linking to a search for each one! https://pypi.org/classifiers/ And we've been working on user research to help guide future design decisions and work. We're grateful for the 59 volunteers who have stepped up to participate in Nicole's user tests. And if you have a spare 5 minutes, we'd like for you to play our "buy a feature" game via this Google form! https://docs.google.com/forms/d/e/1FAIpQLSfABpsRcVYt7RDJEsbL_2CnyH-IKXRCRwa… (short URL: bit.ly/2HpsAWd & tweet to RT[5]) More in our weekly meeting notes[6]. Some open issues that could use comments from you: * Why does warehouse allow linux_armv6l and linux_armv7l wheels?[7] * Derive list of classifiers from a public, version-controlled source[8] * Offer a discouraged/deprecated releases option?[9] Thanks to jonparrott for adding sticky caching for release descriptions[10], to contrepoint for adding a browser warning for IE 10[11], and browniebroke for customizing an email address verification message[12]. As I said last week[13], we're running out of MOSS money. We will probably be able to deal with any issues that come up immediately following the legacy shutdown, but then this project (and the weekly emails from me) will be done. Of course Warehouse could use further sustained effort, so the Packaging Working Group has submitted some grant proposals and requests to some funders for amounts ranging from about USD$35,000 to about USD$150,000. Depending on the funders and their objectives, we've mentioned chunks of work that could happen faster (or at all) with funding, such as: * Adding support for two-factor authentication via TOTP and U2F/Fido. * Adding application-specific tokens scoped to individual users/projects (also covering adding token-based login support to twine and setuptools). * Adding a more advanced audit trail of user actions beyond the current journal (allowing publishers to track all actions taken by third- party services * on their behalf). * Performing accessibility repair work to follow an accessibility audit. * Researching and implementing localization and internationalization features. * Recruiting translators and integrating translations into PyPI. We also would like to accelerate work on group/organization support[14], better notifications, better staging/testing workflow for project maintainers, GitHub signon, and more. If you want details on predicted costs and are interested in hooking the Packaging Working Group[15] up with potential funders, email cochair Ewa Jodlowska at ewa at python dot org -- and she may advise that PSF sponsorship[16] is the route to take! (Also if I'm wrong here about how the PSF wants to do money things, trust actual PSF staffers and not me.) So, things you can do: * check legacy.pypi.org for any behavior, links, etc. you need * upgrade Artifactory * play our "buy a feature" game * comment on issues that need discussion * help us get more funding for future work Thanks and best wishes. -- Sumana Harihareswara Warehouse/PyPI project manager Changeset Consulting sh(a)changeset.nyc Links: 1. https://github.com/pypa/warehouse/issues/3275 2. https://github.com/pypa/warehouse/pull/3772 3. https://github.com/pypa/warehouse/pull/3774 4. https://github.com/pypa/warehouse/pull/3771 5. https://twitter.com/nlhkabu/status/988856279526465537 6. https://wiki.python.org/psf/PackagingWG/2018-04-23-Warehouse 7. https://github.com/pypa/warehouse/issues/3668 8. https://github.com/pypa/warehouse/issues/3786 9. https://github.com/pypa/warehouse/issues/3709 10. https://github.com/pypa/warehouse/pull/3745 11. https://github.com/pypa/warehouse/pull/3764 12. https://github.com/pypa/warehouse/pull/3789 13. https://groups.google.com/forum/#!topic/pypa-dev/MBa5300VlI8 14. https://github.com/pypa/warehouse/issues/201 15. https://wiki.python.org/psf/PackagingWG 16. https://www.python.org/psf/sponsorship/

1 1

saveopts without installing?
by Dirk Avery 23 Apr '18

23 Apr '18

For automation project, trying to get all options set by setup.py and setup.cfg without actually installing. Possible?

2 2

Pip 10.0.1 has been released
by Paul Moore 19 Apr '18

19 Apr '18

On behalf of the PyPA, I am pleased to announce that pip 10.0.1 has just been released. This release fixes a number of issues with the initial release of pip 10.0, notably: * A problem with running the "pip.exe" wrapper script on Windows from a directory with a space in the name. * A problem with get-pip.py needing to be renamed on Windows to avoid triggering a check in pip that aborts the run. * A problem with build isolation when pip is installed as --user * An issue with the vendored msgpack library on older versions of Python 2.7 * A problem with pip installing from non-editable VCS URLs Thanks to all the people who reported issues and helped with the fixes. Paul

1 0

File handling- tab separated files
by Niharika Jakhar 19 Apr '18

19 Apr '18

Hello I am having trouble opening and reading a Tab separated file(big data). I stored the data the data in a list and then used the split function (\t) to tried to print a list of 0 to 100 position. But it is resulting in random values from the files. Please let me know, if you deal with such queries, if not, can you please suggest me some good source in order to solve this problem. Thanking you in advance Best regards NIHARIKA

2 2

Warehouse/PyPI update: launch, project wrapup approaching
by Sumana Harihareswara 19 Apr '18

19 Apr '18

On Monday, we launched Warehouse and redirected[1] browser and API traffic so Warehouse is now the codebase, and http://pypi.org/ is the site, serving nearly everyone who requests files from PyPI. The old codebase is now up at http://legacy.pypi.org/ temporarily. We had a few hiccups (incident report)[2] and are now fixing up some search, indexing, caching, encoding, API compatibility, and UI issues. We're monitoring incoming bug reports and, so far, don't see anything new that absolutely needs fixing before we shut down the legacy site on Monday, April 30th. https://github.com/pypa/warehouse/projects/1 is the rollout board you can watch to see our progress, and our weekly meeting notes are up[3]. Please continue to report bugs -- if you know they're in PyPI, file them against Warehouse[4], and if you're not sure, file them in the "packaging problems" repository[5]. (The "nearly everyone" in that first sentence above is because of this User-Agent exclusion[6], and because I'm sure a few users are specifying legacy.pypi.org in their requests right now while working on forwards compatibility.) We've made a number of user-visible improvements in the past couple weeks. For instance, we added a "switch to desktop version" link in the mobile view[7], fixed dropdowns for accessibility[8], added user help[9] for folks affected by the TLS 1.0/1.1 deprecation (mea culpa[10], we should have done that sooner), and created a page thanking our sponsors[11]. And we've made many backend improvements to performance, API compatibility with legacy, sorting and indexing, instrumentation for metrics, and security -- and Donald implemented[12] email sending via SES. Thanks[13] to Noah Kantrowitz for reporting a privacy concern regarding Gravatar URLs and leaking users' email addresses (fixed[14]). It is not feasible for me to summarize all the work that volunteers put in, as testers, coders, code reviewers, writers, and user support helpers within the last two weeks. We have an embarrassment of riches here. Since April 3rd (two weeks ago) we've merged 98 PRs to Warehouse[15]; thanks to ymyzk, reaperhulk, glasnt, alex, RazerM, bskinn, saxenanurag, hugovk, waseem18, cheungnj, contrepoint, yeraydiazdiaz, jonparrott, jMuzsik, and aalmazan for those. And I'd also like to thank the many people who, on their own, provided and continue to provide help to affected users on IRC, Twitter, StackOverflow, and elsewhere. We don't have any virtual office hours coming up, but we have some other events planned. The Talk Python to Me podcast[16] just interviewed Dustin, Nicole, and Ernest for an upcoming episode. Dustin will be speaking on PyPI and packaging in general at PyCon NA in May[17] and at SciPy in July[18], and we're sprinting at PyCon and EuroPython[19] -- join us? And if you're in New York City, you can join Laura and me for a Warehouse sprint night[20] on Thursday, April 26th. And the project's starting to wrap up. Our MOSS funding (thanks to Mozilla[21] for their support[22] for the PyPI & Warehousework!) is nearly finished; after we shut down the legacy site on April 30th, the general pace of Warehouse development may slow down. (Warehouse has far more volunteer contributors than it did when we started MOSS-funded work in early December, but maintainer time available will diminish.) So we're seeking further funding, to speed up security and accessibility work, (potentially) localization, group/organization support[23], better notifications, better staging/testing workflow for project maintainers, GitHub signon, and more (see the "Post-Legacy shutdown" milestone[24] and "cool but not urgent" milestone[25]). We have submitted a few more grant proposals and are waiting to hear back. And donations to the Python Software Foundation’s Packaging Working Group, which works to sustain PyPI, pip, setuptools, and all other Python Packaging Ecosystem efforts, can now be made on a recurring basis! Please check out https://donate.pypi.org/ and consider pitching in or spreading the word. I'm working on writing up and sharing a more structured list explaining what we could do at various levels of funding. Thanks as always. Please keep the kind words, bug reports, and -- I hope -- funding coming! :)-- Sumana Harihareswara Warehouse/PyPI project manager Changeset Consulting sh(a)changeset.nyc Links: 1. https://github.com/python/pypi-infra/commits/master 2. https://status.python.org/incidents/mgjw1g5yjy5j 3. https://wiki.python.org/psf/PackagingWG/2018-04-17-Warehouse 4. https://github.com/pypa/warehouse/issues 5. https://github.com/pypa/packaging-problems/issues/ 6. https://github.com/pypa/warehouse/issues/3275 7. https://github.com/pypa/warehouse/pull/3602 8. https://github.com/pypa/warehouse/pull/3287 9. https://pypi.org/help/#tls-deprecation 10. https://groups.google.com/d/msg/pypa-dev/Oz6SGA7gefo/UXCu7jM6AQAJ 11. https://pypi.org/sponsors/ 12. https://github.com/pypa/warehouse/pull/3580 13. https://twitter.com/EWDurbin/status/984562893864816640 14. https://github.com/pypa/warehouse/pull/3648 15. https://github.com/pypa/warehouse/pulls?utf8=%E2%9C%93&q=is%3Apr+is%3Amerge… 16. https://talkpython.fm/ 17. https://us.pycon.org/2018/schedule/presentation/148/ 18. https://scipy2018.scipy.org/ehome/index.php?eventid=299527&tabid=712461&cid… 19. https://wiki.python.org/psf/PackagingSprints 20. https://www.meetup.com/nycpython/events/249722192/ 21. https://blog.mozilla.org/blog/2018/01/23/moss-q4-supporting-python-ecosyste… 22. https://pyfound.blogspot.com/2017/11/the-psf-awarded-moss-grant-pypi.html 23. https://github.com/pypa/warehouse/issues/201 24. https://github.com/pypa/warehouse/milestone/12 25. https://github.com/pypa/warehouse/milestone/11

1 0

ANNOUNCE: Sunsetting of uploading to the legacy PyPI/TestPyPI
by Donald Stufft 17 Apr '18

17 Apr '18

Hello All! As most people are aware, there has been an effort under way to rewrite PyPI in order to solve a lot of long standing problems. For those who aren't aware, that is currently available at https://pypi.org/ and it uses the same database that "Legacy" PyPI does, so the two are essentially just different views over the same data. For a awhile now, Python, setuptools, and twine have all defaulted to using this new code base for uploading artifacts to PyPI. Now that we've gotten some testing of that code base, the infrastructure team and myself feel comfortable directing everyone to using the new endpoint and we're planning on shutting down uploads to Legacy PyPI. If you're using the latest versions of Python, setuptools, or twine and you have not put an explicit URL in your ~/.pypirc file, then there's nothing you should need to do. If you are not, then you should ideally upgrade to the latest version of whatever tool you're using to upload (the preferred tool is twine) and edit your ~/.pypirc so that it removes any explicit mention of an URL. Thus it should look something like: [distutils] index-servers = pypi [pypi] username:yourusername password:yourpassword If for some reason you're not able to update to the newest version of your upload tool, then you can configure it to upload to the new code base by switching the URL to use https://upload.pypi.org/legacy/ instead of https://pypi.python.org/pypi. Thus your ~/.pypirc would then become: [distutils] index-servers = pypi [pypi] repository:https://upload.pypi.org/legacy/ username:yourusername password:yourpassword For those of you who are using TestPyPI, that will also be affected, and the required URL for the new upload endpoint for TestPyPI is https://test.pypi.org/legacy/. We plan to disable uploads to legacy PyPI on July 3rd, 2017 so any configuration change will need to be made before that date. In addition, we plan to have a "brownout" on June 29th where we will shut the legacy endpoint down for that day. For TestPyPI the change to disable uploads to legacy will be made in the next couple of days, likely this weekend. As part of the error message that users will get when attempting to upload to legacy PyPI, we will include a link to a page that details how to ensure that they are using the new code base and not the legacy code base. Thanks everyone! — Donald Stufft

9 29

ANN: distlib 0.2.7 released on PyPI
by Vinay Sajip 16 Apr '18

16 Apr '18

I've just released version 0.2.7 of distlib on PyPI [1]. For newcomers, distlib is a library of packaging functionality which is intended to be usable as the basis for third-party packaging tools. The main changes in this release are as follows: * Addressed #102: InstalledDistributions now have a modules attribute which is a list of top-level modules as read from top_level.txt, if that is in the distribution info. * Fixed #103: Now https downloads are preferred to those over http. Thanks to Saulius Žemaitaitis for the patch. * Fixed #104: Updated launcher binaries to properly handle interpreter paths with spaces. Thanks to Atsushi Odagiri for the diagnosis and fix. * Fixed #105: cache_from_source is now imported from importlib.util where available. * Added support for PEP 566 / Metadata 2.1. A more detailed change log is available at [2]. Please try it out, and if you find any problems or have any suggestions for improvements, please give some feedback using the issue tracker! [3] Regards, Vinay Sajip [1] https://pypi.org/project/distlib/0.2.7/ [2] https://goo.gl/M3kQzR [3] https://bitbucket.org/pypa/distlib/issues/new

1 0

New PyPI launched, legacy PyPI shutting down April 30
by Laura Hampton 16 Apr '18

16 Apr '18

New PyPI launched, legacy PyPI shutting down April 30[1] Starting today, the canonical Python Package Index is at https://pypi.org and uses the new Warehouse codebase. We announced the https://pypi.org beta on March 26 and your feedback and test usage have helped us get it production-ready. Monday April 16 (2018-04-16): We launched the new PyPI, redirecting browser traffic and API calls (including "pip install") from pypi.python.org to the new site. The old codebase is still available at https://legacy.pypi.org for now. Monday April 30 (2018-04-30): We plan to shut down legacy PyPI https://legacy.pypi.org . The address pypi.python.org will continue to redirect to Warehouse. For more details, see our roadmap: https://wiki.python.org/psf/WarehouseRoadmap If your site/service links to or uses pypi.python.org, you should start using pypi.org instead: https://warehouse.readthedocs.io/api-reference/integration-guide/#migrating… Thank you. [1] https://blog.python.org/2018/04/new-pypi-launched-legacy-pypi-shutting.html Laura Hampton laura at laura-hampton dot com

3 2

Package metadata: which fields are optional
by Thomas Kluyver 16 Apr '18

16 Apr '18

The core metadata specification (https://packaging.python.org/specifications/core-metadata/ ) notes whether each field is optional. However there are some discrepancies with my understanding: - Download-URL is not marked as optional, but in practice it's obsolete (since PEP 470) and not very helpful (there may be different places to download the same package). Flit has never set this, and I have had no bug reports about it, so in practice it definitely is optional. - Requires-Python is not marked as optional, though I'm pretty sure it also is in practice. - Only one of the multiple use fields is explicitly marked as optional, but my understanding is that 'multiple use' includes using them zero times, so they are all optional. I propose that we remove 'optional' from all the headings, and note at the top of the specification that the required fields are: - Metadata-Version - Name - Version - Summary (And we should check if Summary is really required) Thomas

3 4

Announcement: Pip 10 is coming, and will move all internal APIs
by Paul Moore 15 Apr '18

15 Apr '18

We're in the process of starting to plan for a release of pip (the long-awaited pip 10). We're likely still a month or two away from a release, but now is the time for people to start ensuring that everything works for them. One key change in the new version will be that all of the internal APIs of pip will no longer be available, so any code that currently calls functions in the "pip" namespace will break. Calling pip's internal APIs has never been supported, and always carried a risk of such breakage, so projects doing so should, in theory, be prepared for such things. However, reality is not always that simple, and we are aware that people will need time to deal with the implications. Just in case it's not clear, simply finding where the internal APIs have moved to and calling them under the new names is *not* what people should do. We can't stop people calling the internal APIs, obviously, but the idea of this change is to give people the incentive to find a supported approach, not just to annoy people who are doing things we don't want them to ;-) So please - if you're calling pip's internals in your code, take the opportunity *now* to check out the in-development version of pip, and ensure your project will still work when pip 10 is released. And many thanks to anyone else who helps by testing out the new version, as well :-) Thanks, Paul

12 33

Pip 10.0 has been released
by Paul Moore 15 Apr '18

15 Apr '18

On behalf of the PyPA, I am pleased to announce that pip 10.0 has just been released. This release has been the culmination of many months of work by the community. To install pip 10.0, you can run python -m pip install --upgrade pip or use get-pip, as described in https://pip.pypa.io/en/latest/installing. If you are using a version of pip supplied by your distribution vendor, vendor-supplied upgrades will be available in due course (or you can use pip 10 in a virtual environment). (One minor issue with using get-pip on Windows - when you download get-pip.py, rename it to something that doesn't include "pip" in the name, such as "gp.py", as the standard name triggers a check in pip that aborts the run - this is being tracked in https://github.com/pypa/pip/issues/5219). Highlights of the new release: * Python 2.6 is no longer supported - if you need pip on Python 2.6, you should stay on pip 9, which is the last version to support Python 2.6. * Support for PEP 518, which allows projects to specify what packages they require in order to build from source. (PEP 518 support is currently limited, with full support coming in future versions - see the documentation for details). * Significant improvements in Unicode handling for non-ASCII locales on Windows. * A new "pip config" command. * The default upgrade strategy has become "only-if-needed" * Many bug fixes and minor improvements. In addition, the previously announced reorganisation of pip's internals has now taken place. Unless you are the author of code that imports the pip module (or a user of such code), this change will not affect you. If you are affected, please report the issue to the author of the offending code (refer them to https://mail.python.org/pipermail/distutils-sig/2017-October/031642.html for the details of the announcement). Thanks to everyone who put so much effort into the new release. Many of the contributions came from community members, whether in the form of code, participation in design discussions, or bug reports. The pip development team is extremely grateful to everyone in the community for their contributions. Thanks, Paul

4 3

PEP 571: Any further manylinux2010 concerns or questions?
by Nick Coghlan 14 Apr '18

14 Apr '18

PEP link: https://www.python.org/dev/peps/pep-0571/ Thomas Kluyver has amended Mark Williams's PEP 571 to address the concerns & questions raised in the previous thread: * manylinux2 -> manylinux2010: https://github.com/python/peps/commit/70cbfda06534aedd6372f489090fdc8e1062d… * using glibc 2.12 as a compatibility marker: https://github.com/python/peps/commit/d43b984e021eddc11bdbc36863c5c285b473f… (We also dropped a potentially misleading aside that could be taken as implying inaccurate limitations on Anaconda platform compatibility) As I expect quite a few folks will be busy for the Easter weekend, I'm planning to accept the PEP a week from next Monday (i.e on April 9th) if no new concerns or objections are raised between now and then :) Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

5 10

please mark good first issues in your projects
by Sumana Harihareswara 13 Apr '18

13 Apr '18

Warehouse is attracting several newer contributors including people new to open source, which is great. As Warehouse matures, we have fewer and fewer easy small bugs *in the Python side* left. (So, we have more work for new frontend contributors, and less for Pythonists.) I'd love to refer these folks to other parts of the Python packaging and distribution ecosystem so we can improve the whole toolchain. Right now there are 29 open issues in PyPA projects on GitHub marked "good first issue", 11 in Warehouse and most of the rest in pip: https://github.com/issues?utf8=%E2%9C%93&q=user%3Apypa+is%3Aopen+label%3A%2… I'm totally fine with giving new volunteers teensy tiny doc fix tasks, "manually test this functionality" tasks, and "check whether this bug is still reproducible" tasks, in case you want to write up some of those. Here's a template we use to make good first issues in Warehouse, in case you want to emulate it: https://github.com/pypa/warehouse/issues/new?template=good-first-issue.md **Good First Issue**: This issue is good for first time contributors. If you've already contributed to Warehouse, please work on [another issue without this label](https://github.com/pypa/warehouse/issues?utf8=%E2%9C%93&q=is%3Aissue… instead. If there is not a corresponding pull request for this issue, it is up for grabs. For directions for getting set up, see our [Getting Started Guide](https://warehouse.pypa.io/development/getting-started/). If you are working on this issue and have questions, please feel free to ask them here, [`#pypa-dev` on Freenode](https://webchat.freenode.net/?channels=%23pypa-dev), or the [pypa-dev mailing list](https://groups.google.com/forum/#!forum/pypa-dev). If your project isn't under https://github.com/pypa , but you want to publicize your good first issues, reply to this thread? Thanks. -- Sumana Harihareswara Warehouse project manager Changeset Consulting https://changeset.nyc

2 2

Pip upgrade
by Niharika Jakhar 13 Apr '18

13 Apr '18

Hi I am having troubles upgrading my pip. terminal says: You are using pip version 9.0.1, however version 9.0.3 is available. You should consider upgrading via the 'pip install --upgrade pip' command. I used the same command as mentioned but it keeps on repeating the same lines. python version: Python 2.7.12 pip version: pip 9.0.1 Kindly help me in this regard. Thanking you in advance Best Regards NIHARIKA

3 4

providing a way for pip to communicate extra info to users
by Chris Jerdonek 12 Apr '18

12 Apr '18

On the pypa-dev Google group, a suggestion was raised about giving pip a way to communicate extra info to users. This was during a thread started by Matthew Brett about pip breaking for certain macOS users due to certain TLS changes ("Impending silent breakage of pip / macOS likely to cause severe confusion"). Donald said this behavior is governed by PEP 503 and that the topic was best discussed on distutils-sig: https://groups.google.com/d/msg/pypa-dev/Oz6SGA7gefo/RRXQBQSBBAAJ so I'm raising the suggestion here to continue the discussion. One of Donald's comments in response to the idea (and that occurred to me too and that I agree with) is that providing a way to communicate messages to users introduces another possible avenue for attack. A possible middle-ground could be to hard-code a message in pip. Pip could display the message in certain circumstances, e.g. in response to certain types of failures. For example, the message could tell users to check a certain URL maintained by PyPA for further information / possible announcements. What do people think? --Chris

8 9

wierd namespaces: nspkh.pth makes a package builtin
by Alessandro Dentella 12 Apr '18

12 Apr '18

Hi, I'm trying to debug a complex situazion in which a *nspkg.pth file creates a builtin package that brakes the import in a uwsgi process. In this case I'm still working with Python2.7. To be sure the package is correct I started with an "almost empty" one (whose content is shown below) and I have this strage behaviour: if I install using 'python setup.py install' everithing is ok, if I install with pip the namespace seems a builtin: root@argo-stretch:/tmp/jmb.vega# python setup.py install ... root@argo-stretch:/tmp/jmb.vega# python -c 'import jmb; print(jmb)' <module 'jmb' from '/usr/local/lib/python2.7/dist-packages/jmb.vega-0.1-py2.7.egg/jmb/__init__.pyc'> In this case the file 'jmb.vega-0.1-nspkg.pth' is NOT created and the egg is added to 'easy-install.pth' When doing the installation with pip root@argo-stretch:/tmp/jmb.vega# pip install . Processing /tmp/jmb.vega Requirement already satisfied: setuptools in /usr/lib/python2.7/dist-packages (from jmb.vega==0.1) Installing collected packages: jmb.vega Running setup.py install for jmb.vega ... done Successfully installed jmb.vega-0.1 the file 'jmb.tools-0.7-py2.7-nspkg.pth' is created and the modules seems a built-in root@argo-stretch:/tmp/jmb.vega# (cd ; python -c 'import jmb; print(jmb)') <module 'jmb' (built-in)> In the real case this is enought to break the import system of any call to namespace 'jmb'. the test package is jmb.vega/ ├── jmb │ ├── __init__.py │ └── vega │ └── __init__.py └── setup.py sandro@bluff:/tmp/jmb.vega$ cat setup.py from setuptools import setup, find_packages setup( name='jmb.vega', namespace_packages=['jmb'], version="0.1", description='Test package', author='Alessandro Dentella', packages=find_packages(exclude=['tests', 'tests.*']), platforms='any', zip_safe=False, install_requires=[ 'setuptools', ], ) While __init__ in jmb is: sandro@bluff:/tmp/jmb.vega$ cat jmb/__init__.py __import__('pkg_resources').declare_namespace(__name__) What's wrong with the configuration? Why pip makes it a builtin package? thanks in advance sandro *:-) -- Sandro Dentella *:-) http://trepalchi.it Il portale degli artisti http://www.reteisi.org Soluzioni libere per le scuole http://sqlkit.argolinux.org SQLkit home page - PyGTK/python/sqlalchemy

1 0

(no subject)
by VIVEK MATALIA 12 Apr '18

12 Apr '18

How to download graphics zelle? Sent from Mail for Windows 10

1 0

Execute command before pip install
by Jorge Maldonado Ventura 11 Apr '18

11 Apr '18

I need to execute a command automatically when running `pip install` to solve https://notabug.org/jorgesumle/boot-em-all/issues/1 I need to do that to compile the translation files. I found that overriding the `setuptools.command.install` makes it work with `python3 setup.py install`, but I want it to work with pip as well. Any advice or think I overlooked? Is there a clean or recommended way to do this? The whole code is free software, so you can check my `setup.py` file. The repository can be cloned executing `git clone https://notabug.org/jorgesumle/boot-em-all`.

3 2

setuptools package rpm issue
by xiaojun.gou＠nnct-nsn.com 11 Apr '18

11 Apr '18

hi， When i use the latest python version 3.6.5 to package the setuptools version 39.0.1 to rpm,the os is readhat 6.5.i have some problem. I use blow command to package it: python3 setup.py bdist_rpm --requires "Python3-nsn" --no-autoreq --packager "xiaojun.gou(a)nnct-nsn.com" --binary-only Then the error occured,the error messages is like this: Processing files: setuptools-39.0.1.post20180411-1.noarch error: Two files on one line: /usr/local/python3/lib/python3.6/site-packages/setuptools/script error: File must begin with "/": (dev).tmpl error: Two files on one line: /usr/local/python3/lib/python3.6/site-packages/setuptools/command/launcher error: File must begin with "/": manifest.xml RPM build errors: Two files on one line: /usr/local/python3/lib/python3.6/site-packages/setuptools/script File must begin with "/": (dev).tmpl Two files on one line: /usr/local/python3/lib/python3.6/site-packages/setuptools/command/launcher File must begin with "/": manifest.xml error: command 'rpmbuild' failed with exit status 1 I search the internet,and found this url:https://stackoverflow.com/questions/26718001/cannot-create-setuptools-r… . I followed it ,and found the above two files in the ./build/bdist.linux-x86_64/rpm/BUILD/setuptools-39.0.1.post20180411/INSTALLED_FILES file,there are spaces. Can you fix it? Email:xiaojun.gou@nnct-nsn.com

1 0

How to install examples files?
by Michael Schwager 11 Apr '18

11 Apr '18

Hello, I am trying to install a module with a package and also an examples directory. How do I get my examples installed on users' machines into a reasonable location, in a cross-platform kind of way? I notice that a number of Python examples are installed in /usr/share (in Linux) or in <home>\AppData\Local\Programs\Python\Python36\Share on Windows 8. So in my setup.py, I have this: from setuptools import setup, find_packages from codecs import open from os import path with open(path.join('.', 'README.md'), encoding='utf-8') as f: long_description = f.read() setup( name='kivydnd', version='0.5.0', description='Kivy Drag-n-Drop for Widgets', long_description=long_description, long_description_content_type='text/markdown', url='https://github.com/GreyGnome/KivyDnD', author='GreyGnome', author_email='myemail(a)example.com', license='Apache License 2.0', keywords='kivy drag-n-drop', packages=find_packages(exclude=[]), data_files=[('share/kivydnd-examples', ['examples/dndexample1.py',])], ) But when I try to install them on Linux, I don't see the dndexample1.py file anywhere: python setup.py sdist pip install --target=/home/schwager/lib/python dist/kivydnd-0.5.0.tar.gz --log /tmp/piplog (Note that I am using a different target for testing). The piplog shows that it at least tried to do something with the examples, but I can find a directory by that name anywhere in my home directory: <snippet from log> creating /tmp/tmpirrDx8/share/kivydnd-examples copying examples/dndexample1.py -> /tmp/tmpirrDx8/share/kivydnd-examples Thanks! -- -Mike Schwager

3 3

PyPI/Warehouse (short) weekly report: Progress towards launch milestone
by Laura Hampton 11 Apr '18

11 Apr '18

(This week's report will be a brief update, written by Laura Hampton. Sumana will return later this week.) We are continuing to approach the Launch: Redirect pypi.python.org to pypi.org milestone[1]. The redirect is scheduled to happen on Monday, April 16th. The Warehouse team has been busy closing and responding to issues from legacy PyPI [2], and Dustin is already preparing to remove the beta banner from Warehouse when we reach the Launch milestone[3]. Ernest is working on load-testing infrastructure in anticipation of the launch. We have continued to work on user testing[4], improving accessibility[5], and making changes based on our testers' feedback[6]. Thank you to Nicole for running the tests, and to everyone who helped test and contributed feedback. You can help this week by opining in our "needs discussion" issues [7]. If you have expertise in JavaScript, please consider looking at our frontend testing approach[8]. If you are new to the project, please check out our issues labeled "good first issue", or consider contacting Nicole about being a subject or interviewer for user tests[9]. If you will be at PyCon US in Cleveland next month, we will be sprinting, and we'd love to have you join us[10]. Thanks to Mozilla for their support for the PyPI & Warehouse work![11][12] Laura Hampton Laura at laura-hampton . com [1]https://github.com/pypa/warehouse/milestone/1 [2] https://github.com/pypa/pypi-legacy/issues?q=is%3Aissue+is%3Aclosed+sort%3A… [3]https://github.com/pypa/warehouse/pull/3566 [4] https://github.com/pypa/warehouse/issues/1317 [5] https://github.com/pypa/warehouse/issues/3334 [6] https://github.com/pypa/warehouse/issues/3162 [7] https://github.com/pypa/warehouse/issues?q=is%3Aissue+is%3Aopen+sort%3Aupda… [8] https://github.com/pypa/warehouse/issues/1297 [9]http://whoisnicoleharris.com/2018/03/13/user-testing-warehouse.html [10] https://wiki.python.org/psf/PackagingSprints [11] https://pyfound.blogspot.com/2017/11/the-psf-awarded-moss-grant-pypi.html [12] https://blog.mozilla.org/blog/2018/01/23/moss-q4-supporting-python-ecosyste…

1 0

Renaming this list to python-packaging
by Pradyun Gedam 10 Apr '18

10 Apr '18

Hey all! Given the direction we're moving with PEP 517/518, I propose that we rename this list to something like python-packaging or something similar, to make it a more relavantly named place for having discussions related to Python Packaging in general. I'm curious what others think about this. I'm definitely open to thoughts and other ideas. :) Cheers, Pradyun

6 7

Facing a strange issue while uploading my package using devpi
by Vinay Sharma 08 Apr '18

08 Apr '18

I am not sure if this is the right forum to ask this question. If it is not then please ignore this email and point me to the right forum (if anyone knows). I have a Jenkins Job to build my Python Package. At the end of this Jenkins Job there are bunch of commands to upload the recently build Python Package to devpi server. When I am trying to upload my application distribution using following command "devpi upload" in Jenkins I am getting an the following error :- my_application.tar.gz: does not contain PKGINFO, skipping I have checked everything and everything looks to be good like I have the proper README.rst etc from which the proper PKG-INFO file should be generated. Any pointers would be a great help... Regards -Vinay

4 6

Distributing packages to offline machines
by Eric Gorr 08 Apr '18

08 Apr '18

I had a question about distributing python packages to offline machines when the offline machine is running a different OS then a machine with an internet connection. The packages I am concerned with are third party upon which mine depend. Based on what I have learned so far, there are three solutions. (a) Use a CI to run a fleet of machines for each OS one needs to target to obtain the OS specific wheels. (b) 'pip download <package_list> --no-binary :all:' -- the intention here is to grab the source distribution without any OS specific code included. (c) use https://warehouse.pypa.io/api-reference/json to look for distributed wheels for the target OS and python version and download them directly. (This may make for a nice flag to add to pip somewhere...the ability to specify what wheel one wants when it isn’t for the machine pip is running on) The issue I see with (a) is the shear amount work it would take to setup and maintain such a system. The issue I see with (b) is that it is not 100% reliable as some packages are tricky to install and may not work well with 'pip download’. I have not played around with (c) yet, so I do not know how well it will work, but it seems like a viable solution. I was just wondering if anyone had any comments on this as I think though the ways to solve this problem. Thank you.

5 6

IRC/Twitter livechat hours March 27-April 5
by Sumana Harihareswara 05 Apr '18

05 Apr '18

Warehouse developers will be in IRC, in #pypa-dev on Freenode, and on Twitter (hashtag: #newpypi), available to talk about problems you run into, or about how to hack on Warehouse, for four livechats over the next few weeks: 1. Tuesday, March 27th, 9am-10am PDT, noon-1pm EDT, 18:00-19:00 CEST, 9:30pm-10:30pm India, 16:00-17:00 UTC https://www.timeanddate.com/worldclock/fixedtime.html?msg=Warehouse/PyPI+be… 2. Friday, March 30th, 10-11am EDT, 16:00-17:00 CEST, 7:30pm-8:30pm India, 14:00-15:00 UTC https://www.timeanddate.com/worldclock/fixedtime.html?msg=Warehouse/PyPI+be… 3. Tuesday, April 3rd, 8am-9am PDT, 11am-noon EDT, 17:00-18:00 CEST, 8:30pm-9:30pm India, 15:00-16:00 UTC https://www.timeanddate.com/worldclock/fixedtime.html?msg=Warehouse/PyPI+be… 4. Thursday, April 5th, 5pm-6pm PDT, 8pm-9pm EDT, (April 5th) 8am-9am Manila, (April 5th) 10am-11am Melbourne, (April 5th) 0:00-1:00 UTC https://www.timeanddate.com/worldclock/fixedtime.html?p1=24&iso=20180405T19… Feel free to drop in! (By participating, you agree to abide by the PyPA Code of Conduct: https://www.pypa.io/en/latest/code-of-conduct/ .) -- Sumana Harihareswara Changeset Consulting https://changeset.nyc

1 2

Github-Flavored Markdown is now supported on PyPI
by Jon Wayne Parrott 04 Apr '18

04 Apr '18

Hi distutils-sig, If you haven't yet heard, as of April 1st PyPI (warehouse) supports GitHub-Flavored Markdown for project descriptions. You can read more about this here: http://blog.jonparrott.com/github-flavored-markdown-on-pypi/ Thanks all!

2 1

how can I get the pdf version of library
by X.b. Gao 04 Apr '18

04 Apr '18

Dear Could you please help to understand if I can get the pdf version of library ? then I can look up modules of library locally and easily. Thanks Kind Regards, X.B

1 0

(no subject)
by Keisha Alexander 04 Apr '18

04 Apr '18

How can I get my Ip Address changed

1 0

PyPI/Warehouse update: new advice & launch, shutdown dates
by Sumana Harihareswara 04 Apr '18

04 Apr '18

Big news: we have a schedule for April, and we're advising less- intensive API users to switch to hitting pypi.org. April plans[1] (decided this in yesterday's Warehouse core developers' meeting[2]): 1. Sunday April 8th: TLS 1.0/1.1 removal[3] 2. Monday April 9th: publicize pypi.org launch date (especially to package maintainers and PyPI sponsors)3. Monday April 16: launch/redirect (API and browser traffic redirected to Warehouse)4. Monday April 30: shut down legacy PyPI And today in IRC we decided: If your site/service used to link or upload to pypi.python.org, you should possibly start using pypi.org instead[4]. Now. Specifically: If you have an automated tool that hits pypi.python.org less often than once a second, and/or you are fine with small service interruptions during the beta, then you should go ahead and start using pypi.org now[5], and subscribe to the PyPI announcement list (low-traffic) for further announcements[6]. If you run a mission-critical automated production setup, or an automated tool that hits pypi.python.org with 1+ requests/second, then you should not explicitly point to pypi.org yet, but you should test and prepare to switch[7]. You should also watch our status page[8], since we’re occasionally redirecting portions of that traffic in load tests that we announce on the status page. Subscribe to the PyPI announcement list (low-traffic)[9] to find out when you should switch permanently. Now the more general work update. It's huge because we've been doing so much, our volunteers are prolific, and I truncated last week's report so this is a double issue and I will understand if many of you say "TL;DR". We have 13 open issues till we can launch and redirect pypi.python.org traffic to Warehouse, and then we have 9 open issues till we can shut down legacy PyPI (overview[10]). We've been whittling away at that pile, and fixing other issues as they come up, while responding to new bug reports (and praise) over the last few weeks (a bit slower because of some (planned) team absences). Some improvements from the last two weeks from the MOSS-funded team: * our API docs are more accurate[11] and include policies[12] * our "browse" page has default text and suggested searches[13], and adjusts to page width[14] * prep in case we ever need to briefly turn the site read-only during maintenance[15] * project pages point you to sources for project stats[16] and release notifications[17] * notifications (banners) are dismissable[18] * performance and indexing/caching-related improvements, e.g., purging projects & user pages when someone's role changes[19], updating the trending projects[20], and updating the search index more often[21] * activity journal item order is more consistent and deterministic[22] * a project's last serial ID is available via Simple & JSON APIs[23] Also, Nicole did a bunch of cross-browser testing[24], Ernest worked on a docs proxy[25] and updated parts of legacy PyPI to point to Warehouse[26], Dustin fixed issues affecting twine --skip-existing[27] and a classifier error[28], and I improved developer docs[29] and classifier order[30]. And Warehouse founder and continuing contributor Donald Stufft moved email handling into a service[31], improved performance[32], and removed TLS 1.0/1.1 support[33]. It's deeply important to me to spread the word about the beta, so as many of our friends, neighbors, and colleagues as possible can try Warehouse before the big switch, and so we can know about potential showstoppers as early as possible. So we've held multiple virtual office hours (IRC logs from today's chat[34]), and we've posted to Python- specific spaces like the Python Insider blog[35] and python-announce[36] and debian-python[37], but we've also done some work to get the word out to programmers who don't particularly pay attention to English-language Python-specific news. We found volunteers to translate and post the announcement a little (Brazilian Portugese[38], Polish[39]), and reached out to (for instance) Software & Data Carpentry[40], NumFOCUS[41], The Changelog[42], FLOSS Weekly[43] (who'll interview Ernest and Dustin on May 2nd), and Public Lab[44]. I have a giant checklist and I have been crossing many things off. It's very satisfying. I tried out combining our IRC livechats with a Twitter hashtag, #newpypi, and have gotten approximately zero questions through that so far. Regardless: enjoy some screenshots! [45] And overlapping with the beta rollout, we've been deprecating TLS 1.0/1.1, which leads to issues on some Macs[46], which leads to a question from Guido that isn't answered yet[47]. And other volunteers wrote, by my count, 21 pull requests that we merged over the last 2 weeks[48]. jonparrott, nitinprakash96, venthur, pgadige, saxenanurag, cheungnj, yeraydiazdiaz, AyumuKasuga, jMuzsik, and aalmazan, thank you so much for all your work! I am particularly grateful that our volunteers are helping review each other's work, which helps everyone learn and improve PRs faster. How you can help: * forward the beta announcement[49] to downstreams * tell people on Macs to upgrade pip[50], and answer Guido's question[51] about which users are potentially affected * test[52] Warehouse pull requests, and consider making one[53] * talk with Nicole about being a subject or interviewer for user tests[54] * improve the official Python packaging guide[55] * remind well-off companies/foundations you know that further Warehouse work is more likely if they give the PSF donations[56], sponsorship[57], or grants Thanks again to the Mozilla Open Source Support grant[58] that makes this work possible. -- Sumana Harihareswara Warehouse project manager Changeset Consulting sh(a)changeset.nyc Links: 1. https://wiki.python.org/psf/WarehouseRoadmap 2. https://wiki.python.org/psf/PackagingWG/2018-04-02-Warehouse 3. https://github.com/pypa/warehouse/issues/3411 4. https://warehouse.readthedocs.io/api-reference/integration-guide/#migrating… 5. https://warehouse.readthedocs.io/api-reference/integration-guide/#migrating… 6. https://mail.python.org/mm3/mailman3/lists/pypi-announce.python.org/ 7. https://pyfound.blogspot.com/2018/03/warehouse-all-new-pypi-is-now-in-beta.… 8. http://status.python.org/ 9. https://mail.python.org/mm3/mailman3/lists/pypi-announce.python.org/ 10. https://github.com/pypa/warehouse/milestones 11. https://github.com/pypa/warehouse/pull/3503 12. https://github.com/pypa/warehouse/pull/3333 13. https://github.com/pypa/warehouse/pull/3327 14. https://github.com/pypa/warehouse/pull/3477 15. https://github.com/pypa/warehouse/pull/3393 16. https://github.com/pypa/warehouse/pull/3434 17. https://github.com/pypa/warehouse/pull/3418 18. https://github.com/pypa/warehouse/pull/3372 19. https://github.com/pypa/warehouse/pull/3396 20. https://github.com/pypa/warehouse/pull/3457 21. https://github.com/pypa/warehouse/pull/3459 22. https://github.com/pypa/warehouse/pull/3475 23. https://github.com/pypa/warehouse/pull/3429 24. https://github.com/pypa/warehouse/labels/cross%20browser%20bug%20%3Abug%3A 25. https://github.com/pypa/conveyor/pull/3 26. https://github.com/pypa/pypi-legacy/commits?author=ewdurbin&since=2018-03-0… 27. https://github.com/pypa/warehouse/pull/3522 28. https://github.com/pypa/warehouse/pull/3498 29. https://github.com/pypa/warehouse/pull/3320 30. https://github.com/pypa/warehouse/pull/3466 31. https://github.com/pypa/warehouse/pull/3493 32. https://github.com/pypa/warehouse/pull/3403 33. https://github.com/pypa/warehouse/pull/3354 34. http://kafka.dcpython.org/day/pypa-dev/2018-04-03 35. https://blog.python.org/2018/03/the-all-new-python-package-index-is-now.html 36. https://mail.python.org/pipermail/python-announce-list/2018-March/011883.ht… 37. https://lists.debian.org/debian-python/2018/04/msg00000.html 38. https://groups.google.com/forum/#!topic/python-brasil/Synj27Fczww 39. https://www.facebook.com/groups/pythonpl/permalink/1680880335336289/ 40. http://lists.software-carpentry.org/pipermail/discuss/2018-March/005891.html 41. https://groups.google.com/forum/#!topic/numfocus/uu8aGRmQ-oc 42. https://changelog.com/news/the-new-pypi-is-finally-in-beta-l66G 43. https://twit.tv/shows/floss-weekly 44. https://www.google.com/calendar/event?eid=cTNzdDByZWxmOGRsaXRiMWo3ZXJvY2lwa… 45. https://twitter.com/hashtag/newpypi?src=hash 46. https://mail.python.org/pipermail/python-announce-list/2018-April/011885.ht… 47. https://github.com/pypa/warehouse/issues/3293#issuecomment-378416605 48. https://github.com/pypa/warehouse/pulls?utf8=%E2%9C%93&q=3410+3448+3467+332… 49. https://mail.python.org/pipermail/python-announce-list/2018-March/011883.ht… 50. https://mail.python.org/pipermail/python-announce-list/2018-April/011885.ht… 51. https://github.com/pypa/warehouse/issues/3293#issuecomment-378416605 52. https://warehouse.readthedocs.io/development/reviewing-patches/#testing-bra… 53. https://warehouse.readthedocs.io/development/getting-started/ 54. http://whoisnicoleharris.com/2018/03/13/user-testing-warehouse.html 55. https://github.com/pypa/python-packaging-user-guide/issues?q=is%3Aissue+is%… 56. https://donate.pypi.org/ 57. https://www.python.org/psf/sponsorship/ 58. https://pyfound.blogspot.com/2017/11/the-psf-awarded-moss-grant-pypi.html

1 0

Mac users, upgrade to pip 9.0.3 (due to TLS deprecation)
by Sumana Harihareswara 03 Apr '18

03 Apr '18

Mac users who use pip and PyPI: If you are running macOS/OS X version 10.12 or older, then you ought to upgrade to the latest pip (9.0.3) to connect to the Python Package Index securely: curl https://bootstrap.pypa.io/get-pip.py | python and we recommend you do that by April 8th. Pip 9.0.3 supports TLSv1.2 when running under system Python on macOS < 10.13. Official release notes: https://pip.pypa.io/en/stable/news/ Context: As PSF blogged last year https://pyfound.blogspot.com/2017/01/time-to-upgrade-your-python-tls-v12.ht… , on June 30, 2018, Python.org sites are going to entirely stop supporting TLS versions 1.0 and 1.1, because our CDN provider is deprecating support for those versions. We are launching the new PyPI (in beta at https://pypi.org) this month and replacing the legacy PyPI (https://pypi.python.org). Here's the beta announcement for the new PyPI: https://pyfound.blogspot.com/2018/03/warehouse-all-new-pypi-is-now-in-beta.… Warehouse, the codebase for the new PyPI, does not support TLS 1.0 or 1.1. As of late March, the Python Package Index has started doing brownouts of the deprecated TLS versions. For some portion of each hour, anyone attempting to access PyPI with TLSv1.0 or TLSv1.1 will get a 403 response with an informative error. We are ramping up the amount of time the endpoint is down for the deprecated TLS versions, and plan to make the endpoint 100% unavailable (for the deprecated TLS versions) on and after April 8th, prior to the final deadline. That gives us a few months where, someone tries to "pip install", we can give a good error message -- once June 30th hits, it will just be an uninformative OpenSSL error. More info: * https://github.com/pypa/warehouse/issues/3293 * https://github.com/pypa/warehouse/issues/3411 * https://status.python.org/incidents/btjtz01lzp88 If you have problems accessing PyPI, upgrading pip, etc., please file an issue at https://github.com/pypa/packaging-problems/issues/ and we'll help figure it out. Thank you. Please publicize this. (I'm about to cross-post this to python-list/comp.lang.python.) -- Sumana Harihareswara Warehouse project manager Changeset Consulting https://changeset.nyc

1 0

Re: [Distutils] Beta release of pip version 10
by Paul Moore 02 Apr '18

02 Apr '18

Unfortunately there was an issue in one of the release files (the CA Certificate bundle contained Windows line endings instead of Unix line endings) which caused crashes on older versions of MacOS. As a result we have just released 10.0.0b2, fixing this issue. Anyone on older MacOS versions who have had problems with the 10.0.0b1 release should upgrade to 10.0.0b2. Thanks, Paul On 31 March 2018 at 12:11, Paul Moore <p.f.moore(a)gmail.com> wrote: > On behalf of the PyPA, I am pleased to announce that a beta release > 10.0.0b1 of pip has just been released for testing by the community. > We're planning on a final release in 2 weeks' time, over the weekend > of 14/15 April. > > To install pip 10.0.0.b1, you can run > > python -m pip install --upgrade --pre pip > > (obviously, you should not do this in a production environment!) > > We would be grateful for all testing that users could do, to ensure > that when pip 10 is released it's as solid as we can make it. > > Highlights of the new release: > > * Python 2.6 is no longer supported - if you need pip on Python 2.6, > you should stay on pip 9, which is the last version to support Python > 2.6. > * Support for PEP 518, which allows projects to specify what packages > they require in order to build from source. (PEP 518 support is > currently limited, with full support coming in future versions - see > the documentation for details). > * Significant improvements in Unicode handling for non-ASCII locales on Windows. > * A new "pip config" command. > * The default upgrade strategy has become "only-if-needed" > * Many bug fixes and minor improvements. > > In addition, the previously announced reorganisation of pip's > internals has now taken place. Unless you are the author of code that > imports the pip module (or a user of such code), this change will not > affect you. If you are, please report the issue to the author of the > affected code (refer them to > https://mail.python.org/pipermail/distutils-sig/2017-October/031642.html > for the details of the announcement). > > Please note that there is a minor issue with the NEWS file for this > release - the new features in 10.0.0b1 are reported as being for > "9.0.3 (2018-03-31)". > > If you discover any bugs while testing the new release, please report > them at https://github.com/pypa/pip/issues. > > Thanks to everyone who put so much effort into the new release. Many > of the contributions came from community members, whether in the form > of code, participation in design discussions, or bug reports. The pip > development team is extremely grateful to everyone in the community > for their contributions. > > Thanks, > Paul

1 0

Beta release of pip version 10
by Paul Moore 31 Mar '18

31 Mar '18

On behalf of the PyPA, I am pleased to announce that a beta release 10.0.0b1 of pip has just been released for testing by the community. We're planning on a final release in 2 weeks' time, over the weekend of 14/15 April. To install pip 10.0.0.b1, you can run python -m pip install --upgrade --pre pip (obviously, you should not do this in a production environment!) We would be grateful for all testing that users could do, to ensure that when pip 10 is released it's as solid as we can make it. Highlights of the new release: * Python 2.6 is no longer supported - if you need pip on Python 2.6, you should stay on pip 9, which is the last version to support Python 2.6. * Support for PEP 518, which allows projects to specify what packages they require in order to build from source. (PEP 518 support is currently limited, with full support coming in future versions - see the documentation for details). * Significant improvements in Unicode handling for non-ASCII locales on Windows. * A new "pip config" command. * The default upgrade strategy has become "only-if-needed" * Many bug fixes and minor improvements. In addition, the previously announced reorganisation of pip's internals has now taken place. Unless you are the author of code that imports the pip module (or a user of such code), this change will not affect you. If you are, please report the issue to the author of the affected code (refer them to https://mail.python.org/pipermail/distutils-sig/2017-October/031642.html for the details of the announcement). Please note that there is a minor issue with the NEWS file for this release - the new features in 10.0.0b1 are reported as being for "9.0.3 (2018-03-31)". If you discover any bugs while testing the new release, please report them at https://github.com/pypa/pip/issues. Thanks to everyone who put so much effort into the new release. Many of the contributions came from community members, whether in the form of code, participation in design discussions, or bug reports. The pip development team is extremely grateful to everyone in the community for their contributions. Thanks, Paul

1 0

Need how to install 3rd party python modules
by ?? ?? 29 Mar '18

29 Mar '18

I would like to install openpyxl, and I installed Python shown as follows: Python 3.6.4 (v3.6.4:d48eceb, Dec 19 2017, 06:04:45) [MSC v.1900 32 bit (Intel)] on win32 But as the following situation shows, I cannot succeed installing openpyxl. >>> pip install openpyxl File "<stdin>", line 1 pip install openpyxl ^ SyntaxError: invalid syntax How can I succeed installing openpyxl ? Best Regards, --------------------- Naoki Morihira TEL: 01181-90-6460-6265 ---------------------

5 7

PyPI/Warehouse (short) weekly report: beta, pythonhosted docs, PEP 541
by Sumana Harihareswara 29 Mar '18

29 Mar '18

(A short report this week that doesn't include as much detail about merged PRs, infrastructure work, and so on, because I've been doing stuff like writing to python-list/comp.lang.python to ask for testing[1], but next week's will have more.) The new PyPI went to beta this week![2] Now that PEP 541 is accepted, we are deciding on logistical stuff to implement it and help deal with name transfer requests[3]. Cool things from the notes from the weekly Warehouse core developers' meeting[4] include: if your PyPI project has docs already uploaded to pythonhosted.org, you can now delete those docs[5]. (Further work on docs transition is forthcoming.) And this week and next week some of our team are busy with other projects, so code review and issue resolution will be a little slower. We have livechats coming up in Twitter & IRC[6] (next one is Friday the 30th), and tomorrow I hope to speak on the OpenNews community call[7] at noon Eastern Time. You can help this week by spreading the word[8] about the new PyPI beta and testing it, by opining in our "needs discussion" issues[9], and by just being your wonderful selves. Thank you for your help and kind words through this whole process. It means a lot. More soon! -- Sumana Harihareswara Warehouse project manager Changeset Consulting sh(a)changeset.nyc Links: 1. https://mail.python.org/pipermail/python-list/2018-March/thread.html#732228 2. https://pyfound.blogspot.com/2018/03/warehouse-all-new-pypi-is-now-in-beta.… 3. https://github.com/pypa/warehouse/issues/1506#issuecomment-374626455 4. https://wiki.python.org/psf/PackagingWG/2018-03-26-Warehouse 5. https://github.com/pypa/warehouse/pull/3413 6. https://pyfound.blogspot.com/2018/03/warehouse-all-new-pypi-is-now-in-beta.… 7. https://opennews.org/what/community/calls/ 8. https://pyfound.blogspot.com/2018/03/warehouse-all-new-pypi-is-now-in-beta.… 9. https://github.com/pypa/warehouse/labels/needs%20discussion

1 0

Warehouse is in beta
by Sumana Harihareswara 26 Mar '18

26 Mar '18

As of today, the new Python Package Index at pypi.org, powered by Warehouse, is in beta. What does that mean? >From https://pypi.org/help/#beta : The site is robust, but not *fully* tested and 'production ready'. This means: * The UI may return unusual or erroneous results * Performance may not yet be optimised * Some features that you used in the old interface (pypi.python.org) may be missing * We need your feedback to get the site production ready While we are in beta, the pypi.org infrastructure cannot yet support all of the API traffic generated by our users running pip install, so don't explicitly point to it in automated production setups. But we want you to try it and test it, and we're occasionally redirecting portions of that traffic in load tests we announce on our status page. Uploads, search, and release management are working well, and project maintainers and owners should use this version of PyPI over the legacy site, which no longer supports uploading releases. More at https://pyfound.blogspot.com/2018/03/warehouse-all-new-pypi-is-now-in-beta.… . Congrats and thanks to the team, and onwards to the next milestone (see https://wiki.python.org/psf/WarehouseRoadmap ), the launch and redirect! -- Sumana Harihareswara Changeset Consulting https://changeset.nyc

1 0

quickly installing packages in "editable" mode
by Chris Jerdonek 26 Mar '18

26 Mar '18

Hi, I'm interested in finding ways to speed up a development workflow that involves installing several local packages in "editable" mode in a Docker image. For example, if I use `pip install -e project1 -e project2 ...` in a certain case, it takes on the order of 3-4 seconds. But if I create the .egg-link files, .egg-info directories, etc. "by hand," it only takes on the order of 0.07 seconds. I'm okay with using a hacky solution because these are simple, pure Python projects and it's in a development environment, but if I can rely on existing third-party libraries (via Python, not subprocess), that would be ideal. Do people know of libraries that can help with this sort of thing? As another example, it seems like one of the steps is to update "site-packages/easy-install.pth", but setuptools doesn't seem to expose this as a stand-alone function. I also briefly glanced at things like distlib, the "site" module, and pkg_resources, but I didn't seem to see an API for updating a .pth file. Or maybe I can bypass having to create / update a ".pth" by using symlinks. Thanks, --Chris

1 0

PEP 541 - Accepted
by Mark Mangoba 23 Mar '18

23 Mar '18

Hi All, As the BDFL-Delegate, I’m happy to announce PEP 541 has been accepted. PEP 541 has been voted by the packaging-wg (https://wiki.python.org/psf/P ackagingWG/Charter): - Donald Stufft - Dustin Ingram - Ernest W. Durbin III - Ewa Jodlowska - Kenneth Reitz - Mark Mangoba - Nathaniel J. Smith - Nick Coghlan - Nicole Harris - Sumana Harihareswara Thank you to the packaging-wg and to everyone that has contributed to PEP 541. Best regards, Mark -- Mark Mangoba | PSF IT Manager | Python Software Foundation | mmangoba(a)python.org | python.org | Infrastructure Staff: infrastructure-staff(a)python.org | GPG: 2DE4 D92B 739C 649B EBB8 CCF6 DC05 E024 5F4C A0D1

1 0

Removing wheel signing features from the wheel library
by alex.gronholm＠nextday.fi 23 Mar '18

23 Mar '18

After spending quite some time thinking about this, I've decided to cut out the wheel signature related features from the wheel codebase, unless there is significant resistance among the readers of this mailing list. For those not involved in the previous discussion, the reasoning is that the codebase can be significantly simplified by removing this rarely used feature whose practical value is questionable at best, given the lack of infrastructure for public key distribution. If, as a result, there is a public outcry (which I sincerely doubt), I can always add it back to the next release.

8 16

TUF, Warehouse, Pip, PyPA, ld-signatures, ed25519
by Wes Turner 23 Mar '18

23 Mar '18

TUF, Warehouse, Pip, PyPA, ld-signatures, ed "PEP 480 -- Surviving a Compromise of PyPI" https://www.python.org/dev/peps/pep-0458/ "PEP 480 -- Surviving a Compromise of PyPI: The Maximum Security Model" https://www.python.org/dev/peps/pep-0480/ I need to spend time reviewing these PEPs. Backseat driving here; sorry: Are there pypa/warehouse github issues for implementing the TUF trust root support in warehouse; and client support in pip (or a module that pip and other tools can use)? Warehouse is already a SPOF. That's a hefty responsibility that contributions should support. Would [offline] package mirrors and the CDN still work for/with TUF keys? ld-signatures has some normative language that could be useful. ld-signatures uses URIs for signature suites (a canonicalization algorithm, a message digest algorithm, and a signature algorithm) and JSONLD. That should be pretty future proof in regards to the NIST post-quantum algorithms call that's under review at this time. Blockcerts builds upon ld-signatures. There's a compact form of JSONLD. JSON[LD] can also be serialized as BSON (and RDFHDT). "Linked Data Signatures 1.0" (draft) https://w3c-dvcg.github.io/ld-signatures/ "Ed25519 Signature 2018" (draft) https://w3c-dvcg.github.io/lds-ed25519-2018/ - canonicalizationAlgorithm: https://w3id.org/security#URDNA2015 - digestAlgorithm: http://w3id.org/digests#sha512 - signatureAlgorithm: http://w3id.org/security#ed25519 https://theupdateframework.github.io/ https://github.com/theupdateframework/specification/blob/master/tuf-spec.md… On Thursday, March 22, 2018, Trishank Kuppusamy < trishank.kuppusamy(a)datadoghq.com> wrote: > Hi Wes, > > On Thu, Mar 22, 2018 at 4:40 PM, Wes Turner <wes.turner(a)gmail.com> wrote: > >> >> The hashes serve as file integrity check but provide no assurance that >> they are what the author intended to distribute because there is no >> cryptographic signature. >> >> File hashes help detect bit flips -- due to solar flares -- in storage or >> transit, but do not mitigate against malicious package modification to >> packages in storage or transit. >> >> AFAIU, TUF (The Update Framework) has a mechanism for limiting which >> signing keys are valid for which package? Are pre-shared keys then still >> necessary, or do we then rely on a PKI where one compromised CA cert can >> then forge any other cert? >> > > Yes, you are right, the hashes need to be signed: otherwise you have > integrity, but no authenticity. > > We wrote PEPs 458 <https://www.python.org/dev/peps/pep-0458/> and 480 > <https://www.python.org/dev/peps/pep-0480/> to discuss how TUF might be > deployed on PyPI / Warehouse. The PEPs go into details about public key > distribution. The TLDR is that is that clients (i.e., pip) need to be > shipped with one self-signed root metadata file, and the rest of the PKI is > bootstrapped from there. PyPI would act as an authority that distributes, > revokes, and replaces public keys for packages. > > More details on security vs usability also available in our Diplomat > <https://www.usenix.org/conference/nsdi16/technical-sessions/presentation/ku…> > paper. > > If the community is interested, we'd love to discuss how we could help > make this happen. > > Thanks, > Trishank > >

3 2

PyPI support for linux_ppc64le
by Matt McCormick 22 Mar '18

22 Mar '18

Hi, I created a wheel for Linux / ppc64le (built on CentOS 7). When I try to upload to PyPI via twine, the error message results: unsupported platform tag 'linux_ppc64le' What is the status of the ppc64le wheel support? Is pip / PyPI support expected soon? Thanks, Matt

5 6

TLS support policy & PyPI communications
by Sumana Harihareswara 22 Mar '18

22 Mar '18

PSF blogged last year https://pyfound.blogspot.com/2017/01/time-to-upgrade-your-python-tls-v12.ht… that > The more crucial deadline comes June 30, 2018. On that date all remaining python.org sites, including PyPI, will no longer support TSL 1.0 and 1.1. Older Python versions that do not implement TLSv1.2 will be prohibited from accessing PyPI. I asked Ernest W. Durbin III whether I ought to re-announce this to users in my PyPI announcements. He looked at our TLS trends/stats and told me we have a very very low proportion of traffic that will be affected when we shift over. Therefore, since it'll affect so few, I won't shout about TLS versions in my PyPI communications. Marking that here for the record. -- Sumana Harihareswara Changeset Consulting https://changeset.nyc

2 1

PyPI/Warehouse: infrastructure hardening & the CAPTCHA conundrum
by Sumana Harihareswara 21 Mar '18

21 Mar '18

So we aren't quite at beta yet, but we'll be shouting about pypi.org *really soon*. We have nearly all the Warehouse improvements we need for beta, and nearly all the infrastructure improvements we believe we'll need for the switchover. I'll tell you how you can help, then talk about the current state of things. * The big blocker keeping us from beta: China & CAPTCHAs. Help advise us.[1] * Comment on a "needs discussion" issue[2]. * Help us with large-scope JavaScript issues[3], like our frontend testing approach. * Please talk with Nicole about being a subject or interviewer for user tests[4]. * Tell me if you're planning to join us at sprints at PyCon or EuroPython[5]. * Check out our open good first Warehouse issues[6] (we usually have 10+ open) and get started[7]. If you follow https://status.python.org/ you saw we did some load testing last week and learned from it! We redirected some traffic, for a few periods, for `pip install`, from the old server to Warehouse, and learned from it. For instance, people running Ubuntu 14.04 LTS (long term service release)[8] are usually using a pretty old version of pip, and people on some versions of the Mac OS[9] have older versions of Python and old versions of security-related libraries that don't support the version of TLS that we want them to use. Ernest, Donald and Dustin did a bunch of work addressing this, including Donald putting out pip 9.0.2[10]. (A thing to understand about Ernest's continuing work on PyPI and distribution infrastructure is that it's in a lot of places. It's cabotage[11] & a test cabotage app[12], configuration with salt[13], conveyor[14], pip[15] & get-pip[16], and he filed a bug in Kubernetes[17] which I personally find particularly impressive. And it's in user-facing communication in IRC and GitHub comments and on our statuspage and Twitter, plus a lot of internal discussion with infrastructure colleagues. I have a harder time gathering links for Ernest's work for these emails than for my other teammates; regrets.) As usual, a summary of the past week's work is in our meeting notes[18]. We have new features like letting PyPI administrators add new trove classifiers easily[19], infrastructure improvements like this complexity reduction[20], ton of polish and bug fixing around layout, description content types (Markdown!), a FAQ restructuring[21], a more useful collaboration page[22], etc. And we reviewed and merged a lot of volunteers' pull requests! Thanks to our prolific volunteers: * pgadige making sure an error message reflects whether you're on PyPI or Test PyPI[23] * waseem18 providing an error message for the password reset[24] * cryvate fixing form requirements for password reset[25] * waseem18 fixing disabled button CSS[26] * yeraydiazdiaz fixing modal window behavior[27], then refixing[28] * berkerpeksag adding a "public profile" link to the user dropdown[29] * Mariatta sending notification email when a project collaborator's added[30] * berkerpeksag hiding the "view project" button for no-release-yet projects in maintainers' project lists[31] * alexwlchan renaming a CSS class for consistency[32] * jMuzsik improving documentation of owners' and maintainers' privileges[33] * yeraydiazdiaz adding JavaScript validation to show the user if "new password" and "confirm new password" don't match[34] * alexwlchan documenting all the modifiers in our SASS directory[35] * alanbato and yeraydiazdiaz adding a check to stop someone from uploading a file whose blake2 hash matches an already- uploaded file[36] * cryvate improving sorting of package versions in our /simple/ API[37] * jMuzsik improving how PyPI links look on Twitter, adding an image to our Twitter cards[38] * 9999years updating the Python Packaging User Guide[39] and sample project[40] for Markdown/PEP 566 And thanks to our many bug reporters, especially those who helped us learn from our load tests. Also, check out discussion on API key support/macaroons[41], supporting GitHub-flavored Markdown as Description-Content-Type[42], and project rating/ranking/stars[43]. And finally, we are ever closer to accepting PEP 541 (and planning followup tasks[44]) and are testing our PEP 566 compliance[45]. And I may start a PEP for a Python package index upload API specification[46]. More next week, as usual. *Thanks to Mozilla for their support[47] for the PyPI & Warehouse work[48]!* -- Sumana Harihareswara Warehouse project manager Changeset Consulting sh(a)changeset.nyc Links: 1. https://github.com/pypa/warehouse/issues/3174 2. https://github.com/pypa/warehouse/issues?q=is%3Aissue+is%3Aopen+sort%3Aupda… 3. https://github.com/pypa/warehouse/issues/1297 4. http://whoisnicoleharris.com/2018/03/13/user-testing-warehouse.html 5. https://wiki.python.org/psf/PackagingSprints 6. https://github.com/pypa/warehouse/issues?q=is%3Aissue+is%3Aopen+label%3A%22… 7. https://warehouse.readthedocs.io/development/getting-started/ 8. https://github.com/pypa/warehouse/issues/3280 9. https://github.com/pypa/warehouse/issues/3293 10. https://pypi.org/project/pip/9.0.2/ 11. https://github.com/cabotage/cabotage-app 12. https://github.com/cabotage/test-app/commits?author=ewdurbin&since=2018-03-… 13. https://github.com/python/pypi-salt/commit/1a20cd53ffce0fd3d018d989199d30e1… 14. https://github.com/pypa/conveyor/commits?author=ewdurbin&since=2018-03-13T0… 15. https://github.com/pypa/pip/pull/5076 16. https://github.com/pypa/get-pip/commits?author=ewdurbin&since=2018-03-01T05… 17. https://github.com/kubernetes/kubectl/issues/335 18. https://wiki.python.org/psf/PackagingWG/2018-03-19-Warehouse 19. https://github.com/pypa/warehouse/issues/2649 20. https://github.com/pypa/warehouse/pull/3289 21. https://github.com/pypa/warehouse/pull/3190 22. https://github.com/pypa/warehouse/pull/3047 23. https://github.com/pypa/warehouse/pull/3314 24. https://github.com/pypa/warehouse/pull/3220 25. https://github.com/pypa/warehouse/pull/3230 26. https://github.com/pypa/warehouse/pull/3254 27. https://github.com/pypa/warehouse/pull/3251 28. https://github.com/pypa/warehouse/pull/3291 29. https://github.com/pypa/warehouse/pull/3255 30. https://github.com/pypa/warehouse/pull/3155 31. https://github.com/pypa/warehouse/pull/3257 32. https://github.com/pypa/warehouse/pull/3261 33. https://github.com/pypa/warehouse/pull/3313 34. https://github.com/pypa/warehouse/pull/3219 35. https://github.com/pypa/warehouse/pull/3262 36. https://github.com/pypa/warehouse/pull/3310 37. https://github.com/pypa/warehouse/pull/2574 38. https://github.com/pypa/warehouse/pull/3304 39. https://github.com/pypa/python-packaging-user-guide/pull/457 40. https://github.com/pypa/sampleproject/pull/66 41. https://github.com/pypa/warehouse/issues/994 42. https://github.com/pypa/packaging-problems/issues/126 43. https://github.com/pypa/warehouse/issues/991#issuecomment-374665356 44. https://github.com/pypa/warehouse/issues/1506#issuecomment-374626455 45. https://github.com/pypa/warehouse/issues/3299 46. https://github.com/pypa/packaging-problems/issues/128 47. https://blog.mozilla.org/blog/2018/01/23/moss-q4-supporting-python-ecosyste… 48. https://pyfound.blogspot.com/2017/11/the-psf-awarded-moss-grant-pypi.html

1 0

prepping PEP 566 support in Twine for tomorrow
by Sumana Harihareswara 20 Mar '18

20 Mar '18

Per https://dustingram.com/articles/2018/03/16/markdown-descriptions-on-pypi , currently, Markdown support for a package long_description depends on a pre-release of Twine. I released Twine 1.11.0rc1 a few days ago. Today I'm fixing more bugs and putting out another release candidate, and then tomorrow I plan to release 1.11.0. Code review and testing is welcome, as is camaraderie in #pypa-dev on Freenode. -- Sumana Harihareswara Changeset Consulting https://changeset.nyc

1 1

timeline for 3.7 distribution testing
by Robin Becker 19 Mar '18

19 Mar '18

Anyone here know approximately when the manylinux docker or appveyor ci will be upgraded to supprt Python 3.7? -- Robin Becker

2 1

Cannot install pyopencl
by Tim Graham 19 Mar '18

19 Mar '18

Hello I wonder if you can help.... I am trying to install btcrecover which is no simple task. When I try and install pyopencl it keeps saying that it does not recognise it as a file name... Do you have any ideas? Thanks for your help. Best regards Tim Graham T: +61 8 9467 7664 M: +61 457 414 935 W: www.conveyorcn.com E: tgraham(a)minsep.com.au<mailto:tgraham@minsep.com.au> [cid:image003.jpg@01D3BD7C.2F1A3F40]

3 2

Fwd: Re: Pip 10 is nearing release
by Tim Golden 16 Mar '18

16 Mar '18

On 16/03/2018 09:08, Paul Moore wrote: > After much work, we're pleased to announce that pip 10 is finally > nearing release. We're planning on releasing a beta version over the > weekend of 31 Mar/1 Apr, with the final release to follow about a > fortnight after (assuming no major issues are found). > > Hopefully, as many people as possible will be able to test the new > version of pip during the beta period (or before - it's possible to > install pip from github if you don't want to wait!) Paul, am I right in thinking that this: https://github.com/pypa/pip is the place to look if we want an advanced preview? TJG

2 1

Pip 10 is nearing release
by Paul Moore 16 Mar '18

16 Mar '18

After much work, we're pleased to announce that pip 10 is finally nearing release. We're planning on releasing a beta version over the weekend of 31 Mar/1 Apr, with the final release to follow about a fortnight after (assuming no major issues are found). Hopefully, as many people as possible will be able to test the new version of pip during the beta period (or before - it's possible to install pip from github if you don't want to wait!) so that we can ensure that pip 10 is as robust as we can manage when it gets released. There are a number of new features in pip 10, as well as many months of bug fixes and minor improvements. Highlights include: 1. Support for PEP 518 (declaring build dependencies in `pyproject.toml`) 2. Fixes for Unicode handling in build tools (notably on Windows) 3. The previously announced reorganisation of pip's internals 4. A new `pip config` command 5. Default upgrade strategy becomes "only-if-needed" Thanks to everyone who has contributed to this version of pip - it's been a long time coming, and we apologise for the delay. The plan is to ensure that future releases aren't delayed as long as this one was! Paul

1 0

new stuff overview, beta next week, user tests, & other Warehouse updates
by Sumana Harihareswara 14 Mar '18

14 Mar '18

The new PyPI is still working towards our big public beta[1]. We have 7 open issues till we'll declare beta and make an outreach push (probably late this week or early next week), and then 19 more open issues till we can redirect/launch PyPI[2] probably in April (overview[3]). I've started preparing a draft overview of what's new in PyPI/packaging/distribution[4] to publicize along with the beta; it says "not to be publicized" but I'll let you in on the secret early. Maybe something in it is new to you as well! As usual, we had a Warehouse core developers' meeting on Monday[5]. The last week has seen a lot of polish and bugfixing and documentation for Warehouse. For instance, project deletion is cleaner[6], we more consistently indicate dangerous actions on a page[7], and there's now a migration guide for third-party services[8] which we told several projects about[9]. We've done some infrastructural work, like Datadog instrumentation[10], "Conveyor" (a shim for URL redirects)[11], and Cabotage improvements[12]. Here's an animated GIF demo of release phase commands (scale up, scale down).[13] And we improved other codebases as well, to fix Travis docs[14], get our HTTPS proxy service to deal with big embedded images[15], and deal better with parsing invalid URLs in READMEs[16]. Thanks to volunteers who got pull requests merged this week: * waseem[17]: we now send an email to primary email whenever primary email is changed * mds325[18]: clear input when the user closes the modal * dirn[19]: create a shortlink and redirect all requests for /p/<project>/ to /project/<project>/ * cryvate[20]: clarify project counter for searches with tons of results * Mariatta[21]: fix an email-sending issue And thanks to our many bug reporters, such as Andrew Nesbitt who noticed an RSS feed discrepancy[22]. Check out the current discussion[23] of API keys, a bearer token authentication scheme, and Macaroons in future PyPI. Want to help? * Talk with Nicole about being a subject or interviewer for user tests![24] She's been focusing on user tests and it's paid off, with a lot of bugs found and designs validated. * Got a good workaround for our CAPTCHA being blocked in China[25]? * Consider joining us at sprints[26] in the next few months. * We have 24 good first issues open[27], and a "getting started"[28] guide, and quick turnaround on code review. *Thanks to Mozilla Open Source Support[29] for their funding[30] for the PyPI & Warehouse work.* -- Sumana Harihareswara Warehouse project manager Changeset Consulting sh(a)changeset.nyc P.S. Usually I compose these weekly report emails in plain text; here I'm doing it in HTML with a plaintext fallback. Let me know if it's better, awful, etc. Also nearly no one *replies* to these emails so I'd also welcome your "hey this is useful to me!" offlist reply. Links: 1. https://github.com/pypa/warehouse/milestone/10 2. https://github.com/pypa/warehouse/milestone/1 3. https://github.com/pypa/warehouse/projects/1 4. https://wiki.python.org/psf/PackagingWG/PyPIBetaAnnouncement 5. https://wiki.python.org/psf/PackagingWG/2018-03-12-Warehouse 6. https://github.com/pypa/warehouse/pull/3212 7. https://github.com/pypa/warehouse/pull/3166 8. https://warehouse.readthedocs.io/api-reference/integration-guide/#migrating… 9. https://github.com/pypa/warehouse/issues/2935 10. https://github.com/pypa/warehouse/pull/3076 11. https://github.com/pypa/conveyor/commits?author=ewdurbin&since=2018-03-06T0… 12. https://github.com/cabotage/cabotage-app/commits?author=ewdurbin&since=2018… 13. https://ernest.ly/imgs/cabotage-release-scale-up-scale-down.gif 14. https://github.com/travis-ci/docs-travis-ci-com/pull/1726 15. https://github.com/pypa/warehouse-camo/pull/1 16. https://github.com/pypa/readme_renderer/pull/65 17. https://github.com/pypa/warehouse/pull/3158 18. https://github.com/pypa/warehouse/pull/3160 19. https://github.com/pypa/warehouse/pull/3165 20. https://github.com/pypa/warehouse/pull/3193 21. https://github.com/pypa/warehouse/pull/3214 22. https://github.com/pypa/warehouse/issues/3238 23. https://github.com/pypa/warehouse/issues/994 24. http://whoisnicoleharris.com/2018/03/13/user-testing-warehouse.html 25. https://github.com/pypa/warehouse/issues/3174 26. https://wiki.python.org/psf/PackagingSprints 27. https://github.com/pypa/warehouse/issues?q=is%3Aissue+is%3Aopen+label%3A%22… 28. https://warehouse.readthedocs.io/development/getting-started/ 29. https://blog.mozilla.org/blog/2018/01/23/moss-q4-supporting-python-ecosyste… 30. https://pyfound.blogspot.com/2017/11/the-psf-awarded-moss-grant-pypi.html

2 2

It should be possible to set a custom pypirc location
by Joel Croteau 11 Mar '18

11 Mar '18

Hardcoding ~/.pypirc as the only possible location for repo configs makes certain use cases, like deploying to a custom repo from Jenkins, difficult. It should be possible to set a custom pypirc location, either through an environment variable, or an argument to setup.

2 1

Twine 1.10.0rc1 on Test PyPI
by Sumana Harihareswara 08 Mar '18

08 Mar '18

(So it turns out I've taken on a volunteer gig, which is that I'm now one of the Twine maintainers. I may be wrong about how to do this - please feel free to comment on https://github.com/pypa/twine/pull/314 which is where I'm pulling together a new release checklist for myself.) https://test.pypi.org/manage/project/twine/release/1.10.0rc1/ This is a release candidate for Twine 1.10.0 which I'm planning to release early next week. This release improves project registration usage text (in some cases removing it where inapplicable), and updates `--repository[-url]` usage text, prints progress to `stdout` instead of `stderr`, improves the progressbar, and reorganizes and improves user and developer documentation. Please see the changelog https://twine.readthedocs.io/en/latest/changelog.html for detailed notes under "Next feature release". I believe this is how you test it out: pip install --upgrade --pre --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple twine Please check existing open issues at https://github.com/pypa/twine/issues and open new ones if you have problems. Thanks! -- Sumana Harihareswara Changeset Consulting https://changeset.nyc

3 6

Fwd: PyPI & Warehouse update: redirecting & shutting down legacy by end of April
by Sumana Harihareswara 07 Mar '18

07 Mar '18

Forwarding from pypa-dev (archived https://groups.google.com/forum/#!topic/pypa-dev/L9sF30_Yr2A ). -- Sumana Harihareswara ----- Original message ----- From: Sumana Harihareswara <sh(a)changeset.nyc> To: "pypa-dev" <pypa-dev(a)googlegroups.com> Subject: PyPI & Warehouse update: redirecting & shutting down legacy by end of April Date: Wed, 07 Mar 2018 14:48:34 -0500 The big PyPI news is that we're probably getting to the beta, which we'll publicize heavily, in the next 2 weeks, and redirecting traffic to the new PyPI and shutting down legacy PyPI by the end of April.[0] (Which is good, because that's about when our funding from Mozilla's Open Source Support[1][2] grant will probably run out.) We're working on making a list of third-party services to alert; please help us out.[3] The PyCon North America talk schedule is out -- including Dustin Ingram's "Inside the Cheeseshop: How Python Packaging Works".[4] And we hope you'll join us to hack on packaging and distribution at the sprints, May 14-17.[5] And Nicole Harris is also tentatively planning to lead a Warehouse sprint at EuroPython in July.[6] We've kept on working on features, bugfixes, testing, and infrastructure; here's a selection of the last week's work. Ernest has been continuing cabotage work to manage Kubernetes credentials,[7] and our new infrastructure is stood up & heavily tested. Nicole's doing user tests and taking lessons from that and turning them into issues -- feel free to ping her if you're open to talking with her for 30-60 minutes so she can see how you use the new PyPI.[8] Dustin fixed the issue "Version lookup should take PEP 440 normalization into account" #445 with multiple fixes involving a canonical version for each release.[9] And he also updated the official Python packaging guide to cover how you indicate multiple emails in core metadata.[10] And, thanks to Ernest, PyPI legacy now has a banner for logged-in users, asking them to test pypi.org.[11] Thanks to volunteers: * yeraydiazdiaz for password strength gauge[12] * jw for changing "Edit" to "Manage" in project management screen[13] * aalmazan for updating a checkbox to use the Stimulus framework[14] We also brought up the possibility of changing the PyPI URL structure, in case you want to weigh in.[15] Last week's office hours/IRC livechat went okay! Not as many participants as I would like, but this particular publicity/feedback structure is fairly new to the Python packagers community and I didn't do enough advance publicity. We got praise for the new PyPI, and we got bug reports and related comments and concerns (for Warehouse and related tools), and we shared tutorials and tools and command-line tips that some experienced packagers didn't know about. And we got people to subscribe to the announce mailing list.[16] Notes from the weekly Warehouse core developers' meeting are, as usual, on the wiki.[17] And you can keep up with our current and upcoming milestone progress at the GitHub rollout board overview.[18] And, thanks to Mark Mangoba, PEP 541 is going to get further progress within the Packaging WG this week.[19] We would love your help. Please test PyPI and let us know what works and doesn't work for you. Please let us know of third-party services that should get a heads-up about the changeover.[20] And please consider joining us and hacking on Warehouse.[21] We have 16 open good first issues.[22] Thanks to Mozilla for their support for the PyPI & Warehouse work, and thanks to the PSF for facilitating it! [0] https://wiki.python.org/psf/WarehouseRoadmap [1] https://pyfound.blogspot.com/2017/11/the-psf-awarded-moss-grant-pypi.html [2] https://blog.mozilla.org/blog/2018/01/23/moss-q4-supporting-python-ecosyste… [3] https://github.com/pypa/warehouse/issues/2935 [4] https://us.pycon.org/2018/schedule/presentation/148/ [5] https://us.pycon.org/2018/community/sprints/ [6] https://ep2018.europython.eu/ [7] https://github.com/cabotage/cabotage-app [8] https://twitter.com/nlhkabu/status/969644629644730368 [9] Several commits necessary, including: https://github.com/pypa/warehouse/pull/3113, https://github.com/pypa/warehouse/pull/3099, https://github.com/pypa/warehouse/pull/3102, https://github.com/pypa/packaging/commits?author=di&since=2018-02-01T05:00:… [10] https://github.com/pypa/python-packaging-user-guide/pull/429 [11] https://github.com/pypa/pypi-legacy/commits?author=ewdurbin&since=2018-02-2… [12] https://github.com/pypa/warehouse/pull/3128 [13] https://github.com/pypa/warehouse/pull/3130 [14] https://github.com/pypa/warehouse/pull/3136 [15] https://github.com/pypa/warehouse/issues/3143 [16] https://mail.python.org/mm3/mailman3/lists/pypi-announce.python.org/ [17] https://wiki.python.org/psf/PackagingWG/2018-03-06-Warehouse [18] https://github.com/pypa/warehouse/projects/1 [19] https://www.python.org/dev/peps/pep-0541/ [20] https://github.com/pypa/warehouse/issues/2935 [21] https://warehouse.readthedocs.io/development/getting-started/ [22] https://github.com/pypa/warehouse/issues?q=is%3Aissue+is%3Aopen+label%3A%22… -- Sumana Harihareswara Warehouse project manager Changeset Consulting https://changeset.nyc

1 0

/etc files
by Victor Porton 02 Mar '18

02 Mar '18

How to deal with the files to be placed into /etc or a similar dir? In the previous email I forgot to say I use setuptools not distutils.

2 1

Re: [Distutils] /etc files
by Nathaniel Smith 01 Mar '18

01 Mar '18

Note that this will make it impossible to distribute wheels of your package or to install it into a virtualenv, because those don't have an /etc. So it's mostly only suitable for projects that you use internally under a known and restricted set of deployment options, not for anything distributed on pypi. On Mar 1, 2018 5:01 AM, "Victor Porton" <porton(a)narod.ru> wrote: How to deal with the files to be placed into /etc or a similar dir? In the previous email I forgot to say I use setuptools not distutils. _______________________________________________ Distutils-SIG maillist - Distutils-SIG(a)python.org https://mail.python.org/mailman/listinfo/distutils-sig

1 0

Warehouse testing livechat/office hours in IRC today
by Sumana Harihareswara 01 Mar '18

01 Mar '18

In about 5 minutes in #pypa-dev on Freenode, the maintainers of the new PyPI pypi.org want to hear about any problems Python package maintainers are having with it, and help you learn to hack on Warehouse. We're holding a livechat/office hour; please drop in: https://webchat.freenode.net/?channels=#pypa-dev There'll be another one in a few hours. Timings: Thursday March 1st: 1700 UTC / noon-1pm EST Thursday March 1st: 2300 UTC / 6pm-7pm EST The PSF blog post https://pyfound.blogspot.com/2018/02/python-package-maintainers-help-test.h… has more info. -- Sumana Harihareswara Changeset Consulting https://changeset.nyc

1 0

/etc files
by Victor Porton 01 Mar '18

01 Mar '18

How to deal with files to be placed into /etc or a similar dir?

2 1

merger versus separation of publishing and download tools
by Sumana Harihareswara 28 Feb '18

28 Feb '18

I figured folks here would want to know that there's an ongoing discussion of the distinction between `pip` and `twine`, and related topics (should publishing tools be separate from download/consumption tools?), in: https://github.com/pypa/packaging-problems/issues/60 I don't want to break up the conversation into different threads, one here and one on GitHub,so please consider this a pointer rather than a fork of the conversation. :) -- Sumana Harihareswara Changeset Consulting https://changeset.nyc

1 0

Warehouse update: a week of testing, polish, & infrastructure
by Sumana Harihareswara 28 Feb '18

28 Feb '18

This week we're publicizing pypi.org to package maintainers and asking them/you to test it, and to spread the word to other package maintainers you know.[1][2] And we're probably within two or three weeks of the big public beta, given the existing open Warehouse issues in the next milestone,[3] the new issues we'll open based on user testing this week, and the infrastructure work in play. You can learn more in our weekly meeting notes.[4] We've been doing lots of performance and memory consumption work as we harden our infrastructure and codebase to make them production-ready, lots of polish as we get more testers and get their feedback.Examples: * reduce columns pulled for main project view in admin[5] * Guard against all tuples in metadata upload[6] * Add table style to project description, clean up titles[7] * Add SQLAlchemy error fix to troubleshooting docs[8] * Update PyPI migration info in packaging guide[9] And thanks to volunteers: * alex for "Don't install g++ from a PPA in travis"[10] * wasim for "Added help entry for File already exists error"[11] * HndrkMkt for "Redirect authenticated user from reset pw pages to index"[12] Our infrastructure work included more improvements to cabotage[13] -- see Ernest's demo[14]: > It’s super rough… but here’s first light of an end to end deployment on this thing I been building. still plenty of work to do, but already chock full of automated end-to-end TLS, secure storage of secrets with Vault, a bucket of Kubernetes, enough docker to make your head spin... And our infrastructure work included a restart of Nicole's user testing, both with the broad publicity to package maintainers and with Nicole leading folks through one-on-one exercises and data-gathering sessions. More about Nicole's current design process is in her blog update.[15] Some issue discussion this week that you might find relevant: * Which 3rd party services should we contact about the new pypi.org domain?[16] * APIs/feeds issues got a bit more sorted[17] -- and if there's anything you need in our API docs[18] that isn't in there, please let us know. * What should we show as the default search result on https://pypi.org/search/ ?[19] * What do new developers need in architecture documentation?[20] * Should we rename the "/legacy" URL?[21] As usual, you can get an overview of Warehouse development at our GitHub rollout board[22] And if you want to help out, this week, please do test the site, come to our IRC livechat hours, and spread the word. Thanks to Mozilla for their support for the PyPI & Warehouse work, and thanks to the Python Software Foundation for coordinating it![23][24] [1] https://pyfound.blogspot.com/2018/02/python-package-maintainers-help-test.h… [2] https://wiki.python.org/psf/WarehousePackageMaintainerTesting#IRC_livechat_… [3] https://github.com/pypa/warehouse/milestone/10 [4] https://wiki.python.org/psf/PackagingWG/2018-02-26-Warehouse [5] https://github.com/pypa/warehouse/pull/3043 [6] https://github.com/pypa/warehouse/pull/3049 [7] https://github.com/pypa/warehouse/pull/3040 [8] https://github.com/pypa/warehouse/pull/3048 [9] https://github.com/pypa/python-packaging-user-guide/pull/439 [10] https://github.com/pypa/warehouse/pull/3037 [11] https://github.com/pypa/warehouse/pull/2997 [12] https://github.com/pypa/warehouse/pull/2988 [13] https://github.com/cabotage/cabotage-app [14] https://twitter.com/EWDurbin/status/968315460101042176 [15] http://whoisnicoleharris.com/2015/12/31/designing-warehouse-an-overview.htm… [16] https://github.com/pypa/warehouse/issues/2935 [17] https://github.com/pypa/warehouse/issues?q=is%3Aissue+sort%3Aupdated-desc+l… [18] https://warehouse.readthedocs.io/api-reference/ [19] https://github.com/pypa/warehouse/issues/3062 [20] https://github.com/pypa/warehouse/issues/2794 [21] https://github.com/pypa/warehouse/issues/2285 [22] https://github.com/pypa/warehouse/projects/1 [23] https://pyfound.blogspot.com/2017/11/the-psf-awarded-moss-grant-pypi.html [24] https://blog.mozilla.org/blog/2018/01/23/moss-q4-supporting-python-ecosyste… -- Sumana Harihareswara Changeset Consulting https://changeset.nyc

1 0

Package maintainers, please help test PyPI
by Sumana Harihareswara 27 Feb '18

27 Feb '18

The Warehouse team has improved the new PyPI, available for you to check out at https://pypi.org/ , to the point where we would love for package maintainers to try it out, test it, and give us bug reports. https://wiki.python.org/psf/WarehousePackageMaintainerTesting has guidelines, things to test (like user registration and project removal), and how to contact Warehouse developers. We're hosting four livechat hours this week where Warehouse maintainers will be in IRC, in #pypa-dev on Freenode https://webchat.freenode.net/?channels=#pypa-dev , and specifically available to talk about problems you run into, or about how to hack on Warehouse. Tuesday Feb 27th: 1700 UTC / noon-1pm EST Tuesday Feb 27th: 2300 UTC / 6pm-7pm EST Thursday March 1st: 1700 UTC / noon-1pm EST Thursday March 1st: 2300 UTC / 6pm-7pm EST This isn't the big public beta yet, where we really push the message widely to get non-package-maintainer users to test the site. Since Warehouse must be a reimplementation of the existing PyPI, please focus initially on any differences, missing features, or incorrect behavior that pypi.org exhibits that affect your workflows for account management and package maintainership. We'll be soliciting feedback on other concerns soon! Feedback on user experience, accessibility, and overall ease of use are welcome. Thanks, -- Sumana Harihareswara Warehouse project manager Changeset Consulting https://changeset.nyc

1 0

Python Install
by peter＠mybetterlife.co.uk 26 Feb '18

26 Feb '18

I am new to Python but would like to install to play with some Fast Artificial Neural Network Library routines. However I ha a V.7 installed but could not access pip. I uninstalled 2.7 an then installed latest download which I understand should include pip. However nothing appears to work. Just need some poineters to get to first base. Hope you can help. The following is dialogue from Python Terminal Python 3.7.0a4 (v3.7.0a4:07c9d85, Jan 9 2018, 07:07:02) [MSC v.1900 64 bit (AMD64)] on win32 Type "copyright", "credits" or "license()" for more information. >>> pip install wheel SyntaxError: invalid syntax >>> =============================== RESTART: Shell =============================== >>> pip Traceback (most recent call last): File "<pyshell#1>", line 1, in <module> pip NameError: name 'pip' is not defined >>> python -m pip install SomePackage SyntaxError: invalid syntax >>> >>> python -m ensurepip --default-pip SyntaxError: invalid syntax >>> Regards Peter H. Williams Try BULB <https://join.bulb.co.uk/refer/peter6806> for great GREEN <https://join.bulb.co.uk/refer/peter6806> energy saving deals ~ a real alternative to the BIG six 113 Skelmorlie Castle Road Skelmorlie Ayrshire PA17 5AL TEL:- 01475 529946 Mobile:- 07773 348117 Skype:- peterhwuk

3 2

PEP 566 - Package metadata version 2.1
by Thomas Kluyver 24 Feb '18

24 Feb '18

I'd like to once again prod this PEP towards completion: https://www.python.org/dev/peps/pep-0566/ The version numbering question has been decided in favour of calling it 2.1. The remaining question I'm aware of is whether to make the body text (in the email format of the metadata file) officially represent the package long description. I'm in favour of doing so: at least twine and flit already use this for metadata in wheels. Thomas

6 9

Fwd: Warehouse: essential maintainer features & next steps
by Sumana Harihareswara 22 Feb '18

22 Feb '18

Forwarding from pypa-dev. -------- Forwarded Message -------- Subject: Warehouse: essential maintainer features & next steps Date: Wed, 21 Feb 2018 16:29:31 -0500 From: Sumana Harihareswara <sh(a)changeset.nyc> To: pypa-dev <pypa-dev(a)googlegroups.com> The big Warehouse news: we're now at the Maintainer Minimum Viable Product milestone! To quote our roadmap[0]: > give package maintainers a solid chance to try out Warehouse and report critical bugs early So we've started asking some package maintainers to test pypi.org, and probably later this week we'll broadcast that announcement and request more widely. Depending on the bugs we find as we reach out to project maintainers, and on some infrastructure work, we may hit Milestone 2 next week, which means we'd reach out to a lot of non-package-maintainer users, and start redirecting a portion of `pip` traffic to Warehouse. More on that in our weekly meeting notes.[1] Our team improved or added email management, account management including deletion, better password management and email confirmation of changed passwords to Warehouse last week.[2] We also continued to improve developer documentation[3] and API docs[4]. And we continued our cabotage work[5] and worked on some further improvements to Twine documentation.[6] I also want to highlight some work that Ernest W. Durbin III and Dustin Ingram have done on their own time, as volunteers, that help PyPI. Dustin's continuing work[7] on PEP 566[8] moves us closer to Markdown support for README files[9]. And Ernest put a BUNCH of time into spam-fighting on PyPI this past weekend. Thank you both. Thanks to Volcyy, waseem18, alanbato, zooba, alex, and HndrkMkt for their pull requests which we merged in the last week![10] In the past month, Warehouse has merged 72 pull requests from 11 distinct authors (excluding pyup-bot), and has closed 63 issues (and opened only 26 new ones).[11][12] We have 3 remaining issues between us and the next milestone (the End User MVP), and then ten more issues till we widely publicize the beta.[13] So, we're chugging along. What you can do: You can help improve Warehouse; we have seven open "good first contribution" issues[14] and a guide to getting started[15]. Ernest wants to help you dive in, and to give you stickers, and has 30-minute 1:1 slots available.[16] Please watch your email for a "hey please help us test" email to this very mailing list. Please file general packaging and distribution confusions, peeves, and suggestions in the packaging-problems issue repo.[17] Thanks to Mozilla's Open Source Support grant for funding this PyPI & Warehouse work![18][19] [0] https://wiki.python.org/psf/WarehouseRoadmap [1] https://wiki.python.org/psf/PackagingWG/2018-02-20-Warehouse [2] https://github.com/pypa/warehouse/milestone/8?closed=1 [3] https://warehouse.readthedocs.io/application/ [4] https://warehouse.readthedocs.io/api-reference/ [5] https://github.com/cabotage/cabotage-app/commits/master [6] https://github.com/pypa/twine/pull/297 [7] https://mail.python.org/pipermail/distutils-sig/2018-February/031997.html [8] https://www.python.org/dev/peps/pep-0566/ [9] https://github.com/pypa/warehouse/issues/869 [10] https://github.com/pypa/warehouse/pulls?utf8=%E2%9C%93&q=2948+2971+2975+296… [11] https://github.com/pypa/warehouse/pulse/monthly [12] https://github.com/pypa/warehouse/pulls?utf8=%E2%9C%93&q=is%3Apr+is%3Amerge… [13] https://github.com/pypa/warehouse/milestones [14] https://github.com/pypa/warehouse/issues?q=is%3Aissue+is%3Aopen+label%3A%22… [15] https://warehouse.readthedocs.io/development/getting-started/ [16] https://twitter.com/EWDurbin/status/955415184339849217 [17] https://github.com/pypa/packaging-problems [18] https://pyfound.blogspot.com/2017/11/the-psf-awarded-moss-grant-pypi.html [19] https://blog.mozilla.org/blog/2018/01/23/moss-q4-supporting-python-ecosyste… -- Sumana Harihareswara Changeset Consulting https://changeset.nyc

1 0

PEPs 426 & 459 (metadata 2.0) have been Withdrawn
by Nick Coghlan 21 Feb '18

21 Feb '18

With PEP 566 (metadata 2.1) nearing final review and acceptance, I've updated the PEPs repo to officially withdraw PEPs 426 and 459 in favour of PEP 566: https://github.com/python/peps/commit/0977d33b02920d4619c024b64e35a693220cc… I also reverted any mentions of metadata 3.0 in PEP 426: that's now back to referring solely to the never-adopted metadata 2.0. So while we may still revisit some of the specific ideas in 426 and 459 in future PEPs, any such proposals will share PEP 566's focus on ease of implementation and adoption. Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

1 0

Invalid Packages
by Matt Gieger 19 Feb '18

19 Feb '18

I would like to see a clause added to the "Ivalid Package" section of PEP541 that allows some mechanism for other pypi users to mark a package as spam. Every day i see more spam packages added to pypi and currently the only way to get them removed is to create an issue in github. -Meichthys

5 5

pypi.python Error 403 Forbidden
by Heiko L. 16 Feb '18

16 Feb '18

hallo Matthew, > Can you use pip to install your software instead? What are you no. > trying to install, exactly? Is it just this old version of https://www2.fh-lausitz.de/launic/comp/tmp/180212.install_python_setuptools… > setuptools? Why do you need that? > installation prog without issue. regards Heiko

2 1

pypi.python Error 403 Forbidden
by Heiko L. 16 Feb '18

16 Feb '18

Hallo Marius, >> After a few hours I found the following article: >> https://mail.python.org/pipermail/distutils-sig/2017-October/031712.html >> ...you can no longer access /simple/ and /packages/ over HTTP and you will have to directly go to HTTPS.... >> >> - It is true? > > Yes. > >> - Is this the right place if I have any questions? > > Yes. > I have not found a official website. I would have expected an info on the page https://pypi.python.org/pypi. Where can I find an official statement? Apparently exist prog on https://pypi.python.org which use http, but ypi.python.org does not allow http. It's not easy. A user should be able to decide for himself whether to use HTTP or HTTPS. regards Heiko.

2 1

Documenting project_urls to suggesting thanking or funding mechanisms
by Philip James 15 Feb '18

15 Feb '18

Hello distutils-sig! This email comes to you by way of an issue I filed ( https://github.com/pypa/setuptools/issues/1276), the tl:dr; of which is: I would like it to be easier for packages and package maintainers to indicate how they want to be thanked or funded for their work. I created https://pypi.org/project/thanks/ to help with this, but right now it's reading from a JSON blob maintained by the library. I would prefer to read from the packages themselves, and the `project_urls` dict seems like a good place to do so. Here is an example from the thanks project: https://github.com/phildini/thanks/commit/a4e549338eb3e3c70b1dd5628b38dcbdb… What I am planning to do: - Update the documentation and sample project with a `Project-URL: Funding` item, to encourage maintainers to add this information. What I would love to know from this list: - What do we think of the name "Funding"? Would you prefer "Thanks"? Cheers, and thank you for your time. Philip

2 2

pypi.python Error 403 Forbidden
by Heiko L. 14 Feb '18

14 Feb '18

hallo, I try to install a software from pypi.python.org and receive following errmsg: urllib2.HTTPError: HTTP Error 403: Forbidden and $ wget http://pypi.python.org/packages/2.7/s/setuptools/setuptools-0.6c11-py2.7.egg HTTP request sent, awaiting response... 403 SSL is required After a few hours I found the following article: https://mail.python.org/pipermail/distutils-sig/2017-October/031712.html ...you can no longer access /simple/ and /packages/ over HTTP and you will have to directly go to HTTPS.... - It is true? - Is this the right place if I have any questions? regards Heiko

3 2

Fwd: Warehouse: package manager features & question about advertising
by Sumana Harihareswara 14 Feb '18

14 Feb '18

Forwarded from pypa-dev https://groups.google.com/forum/#!topic/pypa-dev/xQb5RvDb5rc - the weekly Warehouse update. -------- Forwarded Message -------- Subject: Warehouse: package manager features & question about advertising Date: Tue, 13 Feb 2018 23:15:10 -0500 From: Sumana Harihareswara <sh(a)changeset.nyc> Here's your weekly update on Warehouse, powering the new PyPI.[0] Perhaps the biggest news is that the pace of our progress is making us optimistic; we expect to finish all the issues in the first milestone next week, which means Warehouse will have all the essential features package maintainers need.[1] When we get there, we'll be asking some active maintainers to take some time and poke at the site (in the browser and using the APIs) to let us know of any bugs or confusion. In the past week, we've made a ton of progress on, for instance, viewing releases[2] and managing user emails.[3] You can try those out right now at the pre-production site.[4] And the PyPI footer has various policies properly linked in the footer now -- thanks for your advice, PSF![5] Plus, a fix to human-friendly time indicators.[6] Also: Ever wonder how Twine is structured?[7] How does core metadata with multiple email addresses look?[8] And we continued our work on making our credentials handling for Kubernetes more robust.[9] Part of our work is setting up Warehouse on a good foundation for future work, so we spent some time sorting out stuff like: what API documentation do we need?[10] There's a new GitHub label for issues that ask: what APIs do we need?[11] And we restarted the discussion: How much work should we put into Warehouse localisation?[12] Luke Sneeringer volunteered to work on two-factor auth and PyPI API keys, which is great![13] As usual, the notes from our weekly meeting are on the Packaging Working Group wiki.[14] We've also introduced an overview of Warehouse's near-term progress using the GitHub "Projects" feature[15], in case you want to see what we're working on and what's next in a bit more detail than the roadmap.[16] Folks who want to help: we have several good first contribution issues[17] and a guide to getting started[18]. Also, as we prepare for future publicity pushes, please let me know (replying offlist is probably best): where should we advertise to reach occasional and non-Anglophone programmers?[19] Thanks to Mozilla and the PSF for their support for the PyPI & Warehouse work![20][21] [0] https://github.com/pypa/warehouse/ [1] https://github.com/pypa/warehouse/milestone/8 [2] https://github.com/pypa/warehouse/pull/2879 [3] https://github.com/pypa/warehouse/pull/2904 [4] https://pypi.org/ [5] https://github.com/pypa/warehouse/issues/1989 [6] https://github.com/pypa/warehouse/pull/2924 [7] https://github.com/pypa/twine/pull/296 [8] https://github.com/pypa/python-packaging-user-guide/pull/429 [9] https://github.com/cabotage/cabotage-app/commits/master [10] https://github.com/pypa/warehouse/issues/2913 [11] https://github.com/pypa/warehouse/labels/APIs%2Ffeeds [12] https://github.com/pypa/warehouse/issues/1453 [13] https://github.com/pypa/warehouse/issues/994 [14] https://wiki.python.org/psf/PackagingWG/2018-02-12-Warehouse [15] https://github.com/pypa/warehouse/projects/1 [16] https://wiki.python.org/psf/WarehouseRoadmap [17] https://github.com/pypa/warehouse/issues?q=is%3Aissue+is%3Aopen+label%3A%22… [18] https://warehouse.readthedocs.io/development/getting-started/ [19] https://ask.metafilter.com/319055/How-do-I-reach-occasional-and-non-Angloph… [20] https://pyfound.blogspot.com/2017/11/the-psf-awarded-moss-grant-pypi.html [21] https://blog.mozilla.org/blog/2018/01/23/moss-q4-supporting-python-ecosyste… -- Sumana Harihareswara Warehouse project manager Changeset Consulting https://changeset.nyc

1 0

draft PEP: manylinux2
by Mark Williams 13 Feb '18

13 Feb '18

Hi everyone! The manylinux1 platform tag has been tremendously useful, but unfortunately it's showing its age: https://mail.python.org/pipermail/distutils-sig/2017-April/030360.html https://mail.python.org/pipermail/wheel-builders/2016-December/000239.html Nathaniel identified a list of things to do for its successor, manylinux2: https://mail.python.org/pipermail/distutils-sig/2017-April/030361.html Please find below a draft PEP for manylinux2 that attempts to address these issues. I've also opened a PR against python/peps: https://github.com/python/peps/pull/565 Docker images for x86_64 (and soon i686) are available to test drive: https://hub.docker.com/r/markrwilliams/manylinux2/tags/ Thanks! ---- PEP: 9999 Title: The manylinux2 Platform Tag Version: $Revision$ Last-Modified: $Date$ Author: Mark Williams <mrw(a)enotuniq.org> BDFL-Delegate: Nick Coghlan <ncoghlan(a)gmail.com> Discussions-To: Distutils SIG <distutils-sig(a)python.org> Status: Active Type: Informational Content-Type: text/x-rst Created: Post-History: Resolution: Abstract ======== This PEP proposes the creation of a ``manylinux2`` platform tag to succeed the ``manylinux1`` tag introduced by PEP 513 [1]_. It also proposes that PyPI and ``pip`` both be updated to support uploading, downloading, and installing ``manylinux2`` distributions on compatible platforms. Rationale ========= True to its name, the ``manylinux1`` platform tag has made the installation of binary extension modules a reality on many Linux systems. Libraries like ``cryptography`` [2]_ and ``numpy`` [3]_ are more accessible to Python developers now that their installation on common architectures does not depend on fragile development environments and build toolchains. ``manylinux1`` wheels achieve their portability by allowing the extension modules they contain to link against only a small set of system-level shared libraries that export versioned symbols old enough to benefit from backwards-compatibility policies. Extension modules in a ``manylinux1`` wheel that rely on ``glibc``, for example, must be built against version 2.5 or earlier; they may then be run systems that provide more recent ``glibc`` version that still export the required symbols at version 2.5. PEP 513 drew its whitelisted shared libraries and their symbol versions from CentOS 5.11, which was the oldest supported CentOS release at the time of its writing. Unfortunately, CentOS 5.11 reached its end-of-life on March 31st, 2017 with a clear warning against its continued use. [4]_ No further updates, such as security patches, will be made available. This means that its packages will remain at obsolete versions that hamper the efforts of Python software packagers who use the ``manylinux1`` Docker image. CentOS 6.9 is now the oldest supported CentOS release, and will receive maintenance updates through November 30th, 2020. [5]_ We propose that a new PEP 425-style [6]_ platform tag called ``manylinux2`` be derived from CentOS 6.9 and that the ``manylinux`` toolchain, PyPI, and ``pip`` be updated to support it. The ``manylinux2`` policy ========================= The following criteria determine a ``linux`` wheel's eligibility for the ``manylinux2`` tag: 1. The wheel may only contain binary executables and shared objects compiled for one of the two architectures supported by CentOS 6.9: x86_64 or i686. [5]_ 2. The wheel's binary executables or shared objects may not link against externally-provided libraries except those in the following whitelist: :: libgcc_s.so.1 libstdc++.so.6 libm.so.6 libdl.so.2 librt.so.1 libcrypt.so.1 libc.so.6 libnsl.so.1 libutil.so.1 libpthread.so.0 libresolv.so.2 libX11.so.6 libXext.so.6 libXrender.so.1 libICE.so.6 libSM.so.6 libGL.so.1 libgobject-2.0.so.0 libgthread-2.0.so.0 libglib-2.0.so.0 This list is identical to the externally-provided libraries whitelisted for ``manylinux1``, minus ``libncursesw.so.5`` and ``libpanelw.so.5``. [7]_ ``libpythonX.Y`` remains ineligible for inclusion for the same reasons outlined in PEP 513. On Debian-based systems, these libraries are provided by the packages: ============ ======================================================= Package Libraries ============ ======================================================= libc6 libdl.so.2, libresolv.so.2, librt.so.1, libc.so.6, libpthread.so.0, libm.so.6, libutil.so.1, libcrypt.so.1, libnsl.so.1 libgcc1 libgcc_s.so.1 libgl1 libGL.so.1 libglib2.0-0 libgobject-2.0.so.0, libgthread-2.0.so.0, libglib-2.0.so.0 libice6 libICE.so.6 libsm6 libSM.so.6 libstdc++6 libstdc++.so.6 libx11-6 libX11.so.6 libxext6 libXext.so.6 libxrender1 libXrender.so.1 ============ ======================================================= On RPM-based systems, they are provided by these packages: ============ ======================================================= Package Libraries ============ ======================================================= glib2 libglib-2.0.so.0, libgthread-2.0.so.0, libgobject-2.0.so.0 glibc libresolv.so.2, libutil.so.1, libnsl.so.1, librt.so.1, libcrypt.so.1, libpthread.so.0, libdl.so.2, libm.so.6, libc.so.6 libICE libICE.so.6 libX11 libX11.so.6 libXext: libXext.so.6 libXrender libXrender.so.1 libgcc: libgcc_s.so.1 libstdc++ libstdc++.so.6 mesa libGL.so.1 ============ ======================================================= 3. If the wheel contains binary executables or shared objects linked against any whitelisted libraries that also export versioned symbols, they may only depend on the following maximum versions:: GLIBC_2.12 CXXABI_1.3.3 GLIBCXX_3.4.13 GCC_4.3.0 As an example, ``manylinux2`` wheels may include binary artifacts that require ``glibc`` symbols at version ``GLIBC_2.4``, because this an earlier version than the maximum of ``GLIBC_2.12``. 4. If a wheel is built for any version of CPython 2 or CPython versions 3.0 up to and including 3.2, it *must* include a CPython ABI tag indicating its Unicode ABI. A ``manylinux2`` wheel built against Python 2, then, must include either the ``cpy27mu`` tag indicating it was built against an interpreter with the UCS-4 ABI or the ``cpy27m`` tag indicating an interpeter with the UCS-2 ABI. *[Citation for UCS ABI tags?]* 5. A wheel *must not* require the ``PyFPE_jbuf`` symbol. This is achieved by building it against a Python compiled *without* the ``--with-fpectl`` ``configure`` flag. Compilation of Compliant Wheels =============================== Like ``manylinux1``, the ``auditwheel`` tool adds ```manylinux2`` platform tags to ``linux`` wheels built by ``pip wheel`` or ``bdist_wheel`` a ``manylinux2`` Docker container. Docker Images ------------- ``manylinux2`` Docker images based on CentOS 6.9 x86_64 and i686 are provided for building binary ``linux`` wheels that can reliably be converted to ``manylinux2`` wheels. [8]_ These images come with a full compiler suite installed (``gcc``, ``g++``, and ``gfortran`` 4.8.2) as well as the latest releases of Python and ``pip``. Compatibility with kernels that lack ``vsyscall`` ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ A Docker container assumes that its userland is compatible with its host's kernel. Unfortunately, an increasingly common kernel configuration breaks breaks this assumption for x86_64 CentOS 6.9 Docker images. Versions 2.14 and earlier of ``glibc`` require the kernel provide an archaic system call optimization known as ``vsyscall`` on x86_64. [9]_ To effect the optimization, the kernel maps a read-only page of frequently-called system calls -- most notably ``time(2)`` -- into each process at a fixed memory location. ``glibc`` then invokes these system calls by dereferencing a function pointer to the appropriate offset into the ``vsyscall`` page and calling it. This avoids the overhead associated with invoking the kernel that affects normal system call invocation. ``vsyscall`` has long been deprecated in favor of an equivalent mechanism known as vDSO, or "virtual dynamic shared object", in which the kernel instead maps a relocatable virtual shared object containing the optimized system calls into each process. [10]_ The ``vsyscall`` page has serious security implications because it does not participate in address space layout randomization (ASLR). Its predictable location and contents make it a useful source of gadgets used in return-oriented programming attacks. [11]_ At the same time, its elimination breaks the x86_64 ABI, because ``glibc`` versions that depend on ``vsyscall`` suffer from segmentation faults when attempting to dereference a system call pointer into a non-existent page. As a compromise, Linux 3.1 implemented an "emulated" ``vsyscall`` that reduced the executable code, and thus the material for ROP gadgets, mapped into the process. [12]_ ``vsyscall=emulated`` has been the default configuration in most distribution's kernels for many years. Unfortunately, ``vsyscall`` emulation still exposes predicatable code at a reliable memory location, and continues to be useful for return-oriented programming. [13]_ Because most distributions have now upgraded to ``glibc`` versions that do not depend on ``vsyscall``, they are beginning to ship kernels that do not support ``vsyscall`` at all. [14]_ CentOS 5.11 and 6.9 both include versions of ``glibc`` that depend on the ``vsyscall`` page (2.5 and 2.12.2 respectively), so containers based on either cannot run under kernels provided with many distribution's upcoming releases. [15]_ Continuum Analytics faces a related problem with its conda software suite, and as they point out, this will pose a significant obstacle to using these tools in hosted services. [16]_ If Travis CI, for example, begins running jobs under a kernel that does not provide the ``vsyscall`` interface, Python packagers will not be able to use our Docker images there to build ``manylinux`` wheels. [17]_ We have derived a patch from the ``glibc`` git repository that backports the removal of all dependencies on ``vsyscall`` to the version of ``glibc`` included with our ``manylinux2`` image. [18]_ Rebuilding ``glibc``, and thus building ``manylinux2`` image itself, still requires a host kernel that provides the ``vsyscall`` mechanism, but the resulting image can be both run on hosts that provide it and those that do not. Because the ``vsyscall`` interface is an optimization that is only applied to running processes, the ``manylinux2`` wheels built with this modified image should be identical to those built on an unmodified CentOS 6.9 system. Also, the ``vsyscall`` problem applies only to x86_64; it is not part of the i686 ABI. Auditwheel ---------- The ``auditwheel`` tool has also been updated to produce ``manylinux2`` wheels. [19]_ Its behavior and purpose are otherwise unchanged from PEP 513. Platform Detection for Installers ================================= Platforms may define a ``manylinux2_compatible`` boolean attribute on the ``_manylinux`` module described in PEP 513. A platform is considered incompatible with ``manylinux2`` if the attribute is ``False``. Backwards compatibility with ``manylinux1`` wheels ================================================== As explained in PEP 513, the specified symbol versions for ``manylinux1`` whitelisted libraries constitute an *upper bound*. The same is true for the symbol versions defined for ``manylinux2`` in this PEP. As a result, ``manylinux1`` wheels are considered ``manylinux2`` wheels. A ``pip`` that recognizes the ``manylinux2`` platform tag will thus install ``manylinux1`` wheels for ``manylinux2`` platforms -- even when explicitly set -- when no ``manylinux2`` wheels are available. [20]_ PyPI Support ============ PyPI should permit wheels containing the ``manylinux2`` platform tag to be uploaded in the same way that it permits ``manylinux1``. It should not attempt to verify the compatibility of ``manylinux2`` wheels. References ========== .. [1] PEP 513 -- A Platform Tag for Portable Linux Built Distributions (https://www.python.org/dev/peps/pep-0513/) .. [2] pyca/cryptography (https://cryptography.io/) .. [3] numpy (https://numpy.org) .. [4] CentOS 5.11 EOL announcement (https://lists.centos.org/pipermail/centos-announce/2017-April/022350.html) .. [5] CentOS Product Specifications (https://web.archive.org/web/20180108090257/https://wiki.centos.org/About/Pr…) .. [6] PEP 425 -- Compatibility Tags for Built Distributions (https://www.python.org/dev/peps/pep-0425/) .. [7] ncurses 5 -> 6 transition means we probably need to drop some libraries from the manylinux whitelist (https://github.com/pypa/manylinux/issues/94) .. [8] manylinux2 Docker images (https://hub.docker.com/r/markrwilliams/manylinux2/) .. [9] On vsyscalls and the vDSO (https://lwn.net/Articles/446528/) .. [10] vdso(7) (http://man7.org/linux/man-pages/man7/vdso.7.html) .. [11] Framing Signals -- A Return to Portable Shellcode (http://www.cs.vu.nl/~herbertb/papers/srop_sp14.pdf) .. [12] ChangeLog-3.1 (https://www.kernel.org/pub/linux/kernel/v3.x/ChangeLog-3.1) .. [13] Project Zero: Three bypasses and a fix for one of Flash's Vector.<*> mitigations (https://googleprojectzero.blogspot.com/2015/08/three-bypasses-and-fix-for-o…) .. [14] linux: activate CONFIG_LEGACY_VSYSCALL_NONE ? (https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=852620) .. [15] [Wheel-builders] Heads-up re: new kernel configurations breaking the manylinux docker image (https://mail.python.org/pipermail/wheel-builders/2016-December/000239.html) .. [16] Due to glibc 2.12 limitation, static executables that use time(), cpuinfo() and maybe a few others cannot be run on systems that do not support or use `vsyscall=emulate` (https://github.com/ContinuumIO/anaconda-issues/issues/8203) .. [17] Travis CI (https://travis-ci.org/) .. [18] remove-vsyscall.patch https://github.com/markrwilliams/manylinux/commit/e9493d55471d153089df3aafc… .. [19] auditwheel manylinux2 branch (https://github.com/markrwilliams/auditwheel/tree/manylinux2) .. [20] pip manylinux2 branch https://github.com/markrwilliams/pip/commits/manylinux2 Copyright ========= This document has been placed into the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: -- Mark Williams mrw(a)enotuniq.org

13 28

Installed Extras Metadata
by Pradyun Gedam 11 Feb '18

11 Feb '18

Hello! I hope everyone's had a great start to 2018! :) A few months back, while working on pip, I had noticed an oddity about extras. Installing a package with extras would not store information about the fact that the extras were requested. This means, later, it is not possible to know which extra-based optional dependencies of a package have to be considered when verifying that the packages are compatible with each other. This information is relavant for resolution/validation since without it, it is not possible to know which the extra-requirements to care about. As an example, installing ``requests[security]`` and then uninstalling ``PyOpenSSL`` leaves you in a state where you don't really satisfy what was asked for but there's no way to detect that either. Thus, obviously, I'm interested in making pip to be able to store this information. As I understand, this is done needs to be specified in a PEP and/or on PyPUG's specification page. To that end, here's seeding proposal for the discussion: a new `extras-requested.txt` file in the .dist-info directory, storing the extra names in a one-per-line format. Cheers! Pradyun

5 12

Re: [Distutils] draft PEP: manylinux2
by Alex Walters 08 Feb '18

08 Feb '18

-----Original Message----- From: Alex Walters [mailto:tritium-list@sdamon.com] Sent: Wednesday, February 7, 2018 12:21 AM To: 'Janzert' <janzert(a)janzert.com> Subject: RE: [Distutils] draft PEP: manylinux2 > -----Original Message----- > From: Distutils-SIG [mailto:distutils-sig-bounces+tritium- > list=sdamon.com(a)python.org] On Behalf Of Janzert > Sent: Tuesday, February 6, 2018 3:33 PM > To: Distutils-Sig(a)Python.Org > Subject: Re: [Distutils] draft PEP: manylinux2 > > On 2/5/2018 16:01, Nick Coghlan wrote: > > Compare: > > > > - manylinux1 vs manylinux2 vs manylinux3 > > - manylinux2007 vs manylinux2010 vs manylinux2014 > > > > I'll leave this just as a data point (anecdote) from someone that isn't > heavily involved with linux sysadmin or python packaging. Feel free to > make of it what you like. I generally run debian stable and occasionally > ubuntu lts on servers and the latest ubuntu for my workstation. > > If I were looking to install a package and one of the binaries available > is manylinux2010 I probably completely pass over that option and don't > even attempt using it. My assumption would be anything that old probably > isn't going to work on my 2016 or newer OS. Whereas manylinux(1, 2, 3) I > would think has a good chance of working on any reasonably modern linux. > > If not for having read the discussion here I would have interpreted a > date, especially a date that's the better part of a decade in the past, > completely the wrong way. > > Having said that, I'm pretty sure that pip should in general be handling > this decision for me and doing the right thing anyway? So it probably > doesn't matter too much. > > Janzert > This is a really good point. Since pip is the main interface to packages for end users anyways, we can call it manylinux8675309 and it wouldn't really matter to users - the name only really matters to package maintainers, not users. And because of that, manylinux2010, manylinux2014, etc makes more sense. A package maintainer is expected to be more educated about these matters, and that naming scheme is more useful to them. "Whats the oldest linux system my code will run on?" is a very likely question a maintainer would have when building binary packages, and the year-based naming scheme is the logical answer. +1 to manylinux2010, -0 manylinux2 > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG(a)python.org > https://mail.python.org/mailman/listinfo/distutils-sig

4 4

Building a Python package build service for warehouse
by Jannis Gebauer 07 Feb '18

07 Feb '18

Hi! I’m currently working on a package build server. My goal is to produce useful additional meta data for all packages available on PyPi. This includes: - Transitive dependencies - Is the package installable under Python 3? - Various automated “code quality” tests like pylint, pyflakes, pep8, mccabe etc. - Automated security tests - (possibly changelogs, commit logs) - Licenses! The main idea is to run the build process in a restricted “sandbox” docker container that pulls the package from PyPi, installs it and runs a couple of tools on it. Code is still pretty rough, nothing to look at at the moment I’m afraid. Is there any interest in working on this together? Maybe even with the goal to make it an open API that can be consumed by warehouse et al.? Interested in any thoughts on this! Cheers, Jannis P.S: I’m currently crunching trough the data on a 96 CPU cluster. There’s an API available, but it’s sitting behind HTTP Basic Auth as it is basically an endpoint for remote code execution (and throws lots of 500s :D). Send me a mail to jay(a)pyup.io <mailto:jay@pyup.io> if you want to play around with it.

3 2

Moving PEP 541 (PyPI's name management policy) forward
by Nick Coghlan 07 Feb '18

07 Feb '18

Hi folks, As Sumana mentioned in her latest summary email, we've been looking at PEP 541 again, and figuring out how best to manage the process for getting that approved so that the folks handling name management requests can start relying on it. The key challenge with it has been the fact that this document is essentially defining a new operational policy for the Python Software Foundation (as the legal entity behind PyPI), and given that it relates to naming things on the internet, we can reasonably expect it to prove potentially contentious (and perhaps even turn litigious if someone *really* doesn't like a decision that was made in accordance with the policy). Approving that kind of policy would be a significant burden to place on the shoulders of a single volunteer, so we discussed it in the PSF's Packaging working group [1] (which decides how best to allocate any funding the PSF receives or allocates specifically to support the packaging infrastructure), and amended the approval process for this particular PEP as follows: - the BDFL-Delegate is now Mark Mangoba, in his role as the PSF's IT manager - rather than approving the PEP directly, Mark will instead put it forward for a vote in the Packaging Working Group (similar to the way we handle funding allocation decisions) - input from the PSF's General Counsel will be explicitly requested prior to the vote The commit making those changes can be found here: https://github.com/python/peps/commit/6c356f0472b38e69e8a59a2dee1f275b61aea… distutils-sig will remain the mailing list for discussing the actual contents of the policy (and any future amendments to it) - the change in approach just makes it clearer that the policy needs to be explicitly adopted by the PSF as an entity, rather than by an individual volunteer. Regards, Nick. [1] https://wiki.python.org/psf/PackagingWG -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

1 0

Fwd: Warehouse update: still on track, new features
by Sumana Harihareswara 07 Feb '18

07 Feb '18

Our weekly Warehouse update just went out to pypa-dev and is included below. -------- Forwarded Message -------- Subject: Warehouse update: still on track, new features Date: Tue, 6 Feb 2018 22:09:12 -0500 From: Sumana Harihareswara <sh(a)changeset.nyc> To: pypa-dev <pypa-dev(a)googlegroups.com> Here's your weekly update on Warehouse, powering the new PyPI.[0] You can see some noticeable improvements to Warehouse right now compared to last week. There's a mobile UI for managing projects[1], and a project owner can now delete a project.[2] We also have several CSS tweaks and other continuing design improvements -- we're lucky to be working with Nicole on this.[3] Less visibly, we have further Kubernetes security work by Ernest in cabotage[4] and Dustin's work on a generic token service[5]. We're still on track to hit the Maintainer MVP milestone at the end of this month.[6] On the documentation and outreach side, Laura and I have been preparing to contact very active maintainers when we hit that milestone, and we've been improving the packaging user guide,[7] and working a bit on Twine (e.g., documentation for using python-keyring with Twine to avoid having to use a .pypirc).[8] Thanks to Jon Wayne Parrott for fixing an issue Dustin spotted[9] so that pypa.io gets fresh updates again.[10] In PEP progress, PEP 541 is moving forward again, with a pull request for a change in BDFL-Delegate.[11] As usual, meeting notes from our weekly discussion are on the wiki.[12] And if you want to get started contributing to Warehouse, Ernest wants to help you and give you stickers, and has 30-minute 1:1 slots available.[13] Right now we have eleven open issues marked as good for newcomers.[14] Thanks to Mozilla for their support for the PyPI & Warehouse work, and thanks to the PSF for facilitating and supporting this work![15][16] [0] https://pypi.org/ [1] https://github.com/pypa/warehouse/pull/2865 [2] https://github.com/pypa/warehouse/pull/2821 [3] http://whoisnicoleharris.com/warehouse/ [4] https://github.com/cabotage/cabotage-app/commits/master [5] https://github.com/pypa/warehouse/pull/2864 [6] https://github.com/pypa/warehouse/milestone/8 [7] https://github.com/pypa/python-packaging-user-guide/pull/426 [8] https://github.com/pypa/python-packaging-user-guide/issues/297#issuecomment… [9] https://groups.google.com/forum/#!topic/pypa-dev/jzXR3A3E-dw [10] https://www.pypa.io/en/latest/roadmap/ [11] https://github.com/python/peps/pull/566 [12] https://wiki.python.org/psf/PackagingWG/2018-02-05-Warehouse [13] https://twitter.com/EWDurbin/status/955415184339849217 [14] https://github.com/pypa/warehouse/issues?q=is%3Aissue+is%3Aopen+label%3A%22… [15] https://pyfound.blogspot.com/2017/11/the-psf-awarded-moss-grant-pypi.html [16] https://blog.mozilla.org/blog/2018/01/23/moss-q4-supporting-python-ecosyste… -- Sumana Harihareswara Warehouse project manager Changeset Consulting https://changeset.nyc -- Sumana Harihareswara Changeset Consulting https://changeset.nyc

1 0

draft PEP: manylinux2
by Mark Williams 02 Feb '18

02 Feb '18

Hi everyone! The manylinux1 platform tag has been tremendously useful, but unfortunately it's showing its age: https://mail.python.org/pipermail/distutils-sig/2017-April/030360.html https://mail.python.org/pipermail/wheel-builders/2016-December/000239.html Nathaniel identified a list of things to do for its successor, manylinux2: https://mail.python.org/pipermail/distutils-sig/2017-April/030361.html Please find below a draft PEP for manylinux2 that attempts to address these issues. I've also opened a PR against python/peps: https://github.com/python/peps/pull/565 Docker images for x86_64 and i686 are available to test drive: https://hub.docker.com/r/markrwilliams/manylinux2/tags/ Thanks! ---- PEP: 9999 Title: The manylinux2 Platform Tag Version: $Revision$ Last-Modified: $Date$ Author: Mark Williams <mrw(a)enotuniq.org> BDFL-Delegate: Nick Coghlan <ncoghlan(a)gmail.com> Discussions-To: Distutils SIG <distutils-sig(a)python.org> Status: Active Type: Informational Content-Type: text/x-rst Created: Post-History: Resolution: Abstract ======== This PEP proposes the creation of a ``manylinux2`` platform tag to succeed the ``manylinux1`` tag introduced by PEP 513 [1]_. It also proposes that PyPI and ``pip`` both be updated to support uploading, downloading, and installing ``manylinux2`` distributions on compatible platforms. Rationale ========= True to its name, the ``manylinux1`` platform tag has made the installation of binary extension modules a reality on many Linux systems. Libraries like ``cryptography`` [2]_ and ``numpy`` [3]_ are more accessible to Python developers now that their installation on common architectures does not depend on fragile development environments and build toolchains. ``manylinux1`` wheels achieve their portability by allowing the extension modules they contain to link against only a small set of system-level shared libraries that export versioned symbols old enough to benefit from backwards-compatibility policies. Extension modules in a ``manylinux1`` wheel that rely on ``glibc``, for example, must be built against version 2.5 or earlier; they may then be run systems that provide more recent ``glibc`` version that still export the required symbols at version 2.5. PEP 513 drew its whitelisted shared libraries and their symbol versions from CentOS 5.11, which was the oldest supported CentOS release at the time of its writing. Unfortunately, CentOS 5.11 reached its end-of-life on March 31st, 2017 with a clear warning against its continued use. [4]_ No further updates, such as security patches, will be made available. This means that its packages will remain at obsolete versions that hamper the efforts of Python software packagers who use the ``manylinux1`` Docker image. CentOS 6.9 is now the oldest supported CentOS release, and will receive maintenance updates through November 30th, 2020. [5]_ We propose that a new PEP 425-style [6]_ platform tag called ``manylinux2`` be derived from CentOS 6.9 and that the ``manylinux`` toolchain, PyPI, and ``pip`` be updated to support it. The ``manylinux2`` policy ========================= The following criteria determine a ``linux`` wheel's eligibility for the ``manylinux2`` tag: 1. The wheel may only contain binary executables and shared objects compiled for one of the two architectures supported by CentOS 6.9: x86_64 or i686. [5]_ 2. The wheel's binary executables or shared objects may not link against externally-provided libraries except those in the following whitelist: :: libgcc_s.so.1 libstdc++.so.6 libm.so.6 libdl.so.2 librt.so.1 libcrypt.so.1 libc.so.6 libnsl.so.1 libutil.so.1 libpthread.so.0 libresolv.so.2 libX11.so.6 libXext.so.6 libXrender.so.1 libICE.so.6 libSM.so.6 libGL.so.1 libgobject-2.0.so.0 libgthread-2.0.so.0 libglib-2.0.so.0 This list is identical to the externally-provided libraries whitelisted for ``manylinux1``, minus ``libncursesw.so.5`` and ``libpanelw.so.5``. [7]_ ``libpythonX.Y`` remains ineligible for inclusion for the same reasons outlined in PEP 513. On Debian-based systems, these libraries are provided by the packages: ============ ======================================================= Package Libraries ============ ======================================================= libc6 libdl.so.2, libresolv.so.2, librt.so.1, libc.so.6, libpthread.so.0, libm.so.6, libutil.so.1, libcrypt.so.1, libnsl.so.1 libgcc1 libgcc_s.so.1 libgl1 libGL.so.1 libglib2.0-0 libgobject-2.0.so.0, libgthread-2.0.so.0, libglib-2.0.so.0 libice6 libICE.so.6 libsm6 libSM.so.6 libstdc++6 libstdc++.so.6 libx11-6 libX11.so.6 libxext6 libXext.so.6 libxrender1 libXrender.so.1 ============ ======================================================= On RPM-based systems, they are provided by these packages: ============ ======================================================= Package Libraries ============ ======================================================= glib2 libglib-2.0.so.0, libgthread-2.0.so.0, libgobject-2.0.so.0 glibc libresolv.so.2, libutil.so.1, libnsl.so.1, librt.so.1, libcrypt.so.1, libpthread.so.0, libdl.so.2, libm.so.6, libc.so.6 libICE libICE.so.6 libX11 libX11.so.6 libXext: libXext.so.6 libXrender libXrender.so.1 libgcc: libgcc_s.so.1 libstdc++ libstdc++.so.6 mesa libGL.so.1 ============ ======================================================= 3. If the wheel contains binary executables or shared objects linked against any whitelisted libraries that also export versioned symbols, they may only depend on the following maximum versions:: GLIBC_2.12 CXXABI_1.3.3 GLIBCXX_3.4.13 GCC_4.3.0 As an example, ``manylinux2`` wheels may include binary artifacts that require ``glibc`` symbols at version ``GLIBC_2.4``, because this an earlier version than the maximum of ``GLIBC_2.12``. 4. If a wheel is built for any version of CPython 2 or CPython versions 3.0 up to and including 3.2, it *must* include a CPython ABI tag indicating its Unicode ABI. A ``manylinux2`` wheel built against Python 2, then, must include either the ``cpy27mu`` tag indicating it was built against an interpreter with the UCS-4 ABI or the ``cpy27m`` tag indicating an interpeter with the UCS-2 ABI. *[Citation for UCS ABI tags?]* 5. A wheel *must not* require the ``PyFPE_jbuf`` symbol. This is achieved by building it against a Python compiled *without* the ``--with-fpectl`` ``configure`` flag. Compilation of Compliant Wheels =============================== Like ``manylinux1``, the ``auditwheel`` tool adds ```manylinux2`` platform tags to ``linux`` wheels built by ``pip wheel`` or ``bdist_wheel`` a ``manylinux2`` Docker container. Docker Images ------------- ``manylinux2`` Docker images based on CentOS 6.9 x86_64 and i686 are provided for building binary ``linux`` wheels that can reliably be converted to ``manylinux2`` wheels. [8]_ These images come with a full compiler suite installed (``gcc``, ``g++``, and ``gfortran`` 4.8.2) as well as the latest releases of Python and ``pip``. Compatibility with kernels that lack ``vsyscall`` ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ A Docker container assumes that its userland is compatible with its host's kernel. Unfortunately, an increasingly common kernel configuration breaks breaks this assumption for x86_64 CentOS 6.9 Docker images. Versions 2.14 and earlier of ``glibc`` require the kernel provide an archaic system call optimization known as ``vsyscall`` on x86_64. [9]_ To effect the optimization, the kernel maps a read-only page of frequently-called system calls -- most notably ``time(2)`` -- into each process at a fixed memory location. ``glibc`` then invokes these system calls by dereferencing a function pointer to the appropriate offset into the ``vsyscall`` page and calling it. This avoids the overhead associated with invoking the kernel that affects normal system call invocation. ``vsyscall`` has long been deprecated in favor of an equivalent mechanism known as vDSO, or "virtual dynamic shared object", in which the kernel instead maps a relocatable virtual shared object containing the optimized system calls into each process. [10]_ The ``vsyscall`` page has serious security implications because it does not participate in address space layout randomization (ASLR). Its predictable location and contents make it a useful source of gadgets used in return-oriented programming attacks. [11]_ At the same time, its elimination breaks the x86_64 ABI, because ``glibc`` versions that depend on ``vsyscall`` suffer from segmentation faults when attempting to dereference a system call pointer into a non-existent page. As a compromise, Linux 3.1 implemented an "emulated" ``vsyscall`` that reduced the executable code, and thus the material for ROP gadgets, mapped into the process. [12]_ ``vsyscall=emulated`` has been the default configuration in most distribution's kernels for many years. Unfortunately, ``vsyscall`` emulation still exposes predicatable code at a reliable memory location, and continues to be useful for return-oriented programming. [13]_ Because most distributions have now upgraded to ``glibc`` versions that do not depend on ``vsyscall``, they are beginning to ship kernels that do not support ``vsyscall`` at all. [14]_ CentOS 5.11 and 6.9 both include versions of ``glibc`` that depend on the ``vsyscall`` page (2.5 and 2.12.2 respectively), so containers based on either cannot run under kernels provided with many distribution's upcoming releases. [15]_ Continuum Analytics faces a related problem with its conda software suite, and as they point out, this will pose a significant obstacle to using these tools in hosted services. [16]_ If Travis CI, for example, begins running jobs under a kernel that does not provide the ``vsyscall`` interface, Python packagers will not be able to use our Docker images there to build ``manylinux`` wheels. [17]_ We have derived a patch from the ``glibc`` git repository that backports the removal of all dependencies on ``vsyscall`` to the version of ``glibc`` included with our ``manylinux2`` image. [18]_ Rebuilding ``glibc``, and thus building ``manylinux2`` image itself, still requires a host kernel that provides the ``vsyscall`` mechanism, but the resulting image can be both run on hosts that provide it and those that do not. Because the ``vsyscall`` interface is an optimization that is only applied to running processes, the ``manylinux2`` wheels built with this modified image should be identical to those built on an unmodified CentOS 6.9 system. Also, the ``vsyscall`` problem applies only to x86_64; it is not part of the i686 ABI. Auditwheel ---------- The ``auditwheel`` tool has also been updated to produce ``manylinux2`` wheels. [19]_ Its behavior and purpose are otherwise unchanged from PEP 513. Platform Detection for Installers ================================= Platforms may define a ``manylinux2_compatible`` boolean attribute on the ``_manylinux`` module described in PEP 513. A platform is considered incompatible with ``manylinux2`` if the attribute is ``False``. Backwards compatibility with ``manylinux1`` wheels ================================================== As explained in PEP 513, the specified symbol versions for ``manylinux1`` whitelisted libraries constitute an *upper bound*. The same is true for the symbol versions defined for ``manylinux2`` in this PEP. As a result, ``manylinux1`` wheels are considered ``manylinux2`` wheels. A ``pip`` that recognizes the ``manylinux2`` platform tag will thus install ``manylinux1`` wheels for ``manylinux2`` platforms -- even when explicitly set -- when no ``manylinux2`` wheels are available. [20]_ PyPI Support ============ PyPI should permit wheels containing the ``manylinux2`` platform tag to be uploaded in the same way that it permits ``manylinux1``. It should not attempt to verify the compatibility of ``manylinux2`` wheels. References ========== .. [1] PEP 513 -- A Platform Tag for Portable Linux Built Distributions (https://www.python.org/dev/peps/pep-0513/) .. [2] pyca/cryptography (https://cryptography.io/) .. [3] numpy (https://numpy.org) .. [4] CentOS 5.11 EOL announcement (https://lists.centos.org/pipermail/centos-announce/2017-April/022350.html) .. [5] CentOS Product Specifications (https://web.archive.org/web/20180108090257/https://wiki.centos.org/About/Pr…) .. [6] PEP 425 -- Compatibility Tags for Built Distributions (https://www.python.org/dev/peps/pep-0425/) .. [7] ncurses 5 -> 6 transition means we probably need to drop some libraries from the manylinux whitelist (https://github.com/pypa/manylinux/issues/94) .. [8] manylinux2 Docker images (https://hub.docker.com/r/markrwilliams/manylinux2/) .. [9] On vsyscalls and the vDSO (https://lwn.net/Articles/446528/) .. [10] vdso(7) (http://man7.org/linux/man-pages/man7/vdso.7.html) .. [11] Framing Signals -- A Return to Portable Shellcode (http://www.cs.vu.nl/~herbertb/papers/srop_sp14.pdf) .. [12] ChangeLog-3.1 (https://www.kernel.org/pub/linux/kernel/v3.x/ChangeLog-3.1) .. [13] Project Zero: Three bypasses and a fix for one of Flash's Vector.<*> mitigations (https://googleprojectzero.blogspot.com/2015/08/three-bypasses-and-fix-for-o…) .. [14] linux: activate CONFIG_LEGACY_VSYSCALL_NONE ? (https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=852620) .. [15] [Wheel-builders] Heads-up re: new kernel configurations breaking the manylinux docker image (https://mail.python.org/pipermail/wheel-builders/2016-December/000239.html) .. [16] Due to glibc 2.12 limitation, static executables that use time(), cpuinfo() and maybe a few others cannot be run on systems that do not support or use `vsyscall=emulate` (https://github.com/ContinuumIO/anaconda-issues/issues/8203) .. [17] Travis CI (https://travis-ci.org/) .. [18] remove-vsyscall.patch https://github.com/markrwilliams/manylinux/commit/e9493d55471d153089df3aafc… .. [19] auditwheel manylinux2 branch (https://github.com/markrwilliams/auditwheel/tree/manylinux2) .. [20] pip manylinux2 branch https://github.com/markrwilliams/pip/commits/manylinux2 Copyright ========= This document has been placed into the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: -- Mark Williams mrw(a)twistedmatrix.com

1 0

draft PEP: manylinux2
by Mark Williams 02 Feb '18

02 Feb '18

Hi everyone! The manylinux1 platform tag has been tremendously useful, but unfortunately it's showing its age: https://mail.python.org/pipermail/distutils-sig/2017-April/030360.html https://mail.python.org/pipermail/wheel-builders/2016-December/000239.html Nathaniel identified a list of things to do for its successor, manylinux2: https://mail.python.org/pipermail/distutils-sig/2017-April/030361.html Please find below a draft PEP for manylinux2 that attempts to address these issues. I've also opened a PR against python/peps: https://github.com/python/peps/pull/565 Docker images for x86_64 and i686 are available to test drive: https://hub.docker.com/r/markrwilliams/manylinux2/tags/ Thanks! ---- PEP: 9999 Title: The manylinux2 Platform Tag Version: $Revision$ Last-Modified: $Date$ Author: Mark Williams <mrw(a)enotuniq.org> BDFL-Delegate: Nick Coghlan <ncoghlan(a)gmail.com> Discussions-To: Distutils SIG <distutils-sig(a)python.org> Status: Active Type: Informational Content-Type: text/x-rst Created: Post-History: Resolution: Abstract ======== This PEP proposes the creation of a ``manylinux2`` platform tag to succeed the ``manylinux1`` tag introduced by PEP 513 [1]_. It also proposes that PyPI and ``pip`` both be updated to support uploading, downloading, and installing ``manylinux2`` distributions on compatible platforms. Rationale ========= True to its name, the ``manylinux1`` platform tag has made the installation of binary extension modules a reality on many Linux systems. Libraries like ``cryptography`` [2]_ and ``numpy`` [3]_ are more accessible to Python developers now that their installation on common architectures does not depend on fragile development environments and build toolchains. ``manylinux1`` wheels achieve their portability by allowing the extension modules they contain to link against only a small set of system-level shared libraries that export versioned symbols old enough to benefit from backwards-compatibility policies. Extension modules in a ``manylinux1`` wheel that rely on ``glibc``, for example, must be built against version 2.5 or earlier; they may then be run systems that provide more recent ``glibc`` version that still export the required symbols at version 2.5. PEP 513 drew its whitelisted shared libraries and their symbol versions from CentOS 5.11, which was the oldest supported CentOS release at the time of its writing. Unfortunately, CentOS 5.11 reached its end-of-life on March 31st, 2017 with a clear warning against its continued use. [4]_ No further updates, such as security patches, will be made available. This means that its packages will remain at obsolete versions that hamper the efforts of Python software packagers who use the ``manylinux1`` Docker image. CentOS 6.9 is now the oldest supported CentOS release, and will receive maintenance updates through November 30th, 2020. [5]_ We propose that a new PEP 425-style [6]_ platform tag called ``manylinux2`` be derived from CentOS 6.9 and that the ``manylinux`` toolchain, PyPI, and ``pip`` be updated to support it. The ``manylinux2`` policy ========================= The following criteria determine a ``linux`` wheel's eligibility for the ``manylinux2`` tag: 1. The wheel may only contain binary executables and shared objects compiled for one of the two architectures supported by CentOS 6.9: x86_64 or i686. [5]_ 2. The wheel's binary executables or shared objects may not link against externally-provided libraries except those in the following whitelist: :: libgcc_s.so.1 libstdc++.so.6 libm.so.6 libdl.so.2 librt.so.1 libcrypt.so.1 libc.so.6 libnsl.so.1 libutil.so.1 libpthread.so.0 libresolv.so.2 libX11.so.6 libXext.so.6 libXrender.so.1 libICE.so.6 libSM.so.6 libGL.so.1 libgobject-2.0.so.0 libgthread-2.0.so.0 libglib-2.0.so.0 This list is identical to the externally-provided libraries whitelisted for ``manylinux1``, minus ``libncursesw.so.5`` and ``libpanelw.so.5``. [7]_ ``libpythonX.Y`` remains ineligible for inclusion for the same reasons outlined in PEP 513. On Debian-based systems, these libraries are provided by the packages: ============ ======================================================= Package Libraries ============ ======================================================= libc6 libdl.so.2, libresolv.so.2, librt.so.1, libc.so.6, libpthread.so.0, libm.so.6, libutil.so.1, libcrypt.so.1, libnsl.so.1 libgcc1 libgcc_s.so.1 libgl1 libGL.so.1 libglib2.0-0 libgobject-2.0.so.0, libgthread-2.0.so.0, libglib-2.0.so.0 libice6 libICE.so.6 libsm6 libSM.so.6 libstdc++6 libstdc++.so.6 libx11-6 libX11.so.6 libxext6 libXext.so.6 libxrender1 libXrender.so.1 ============ ======================================================= On RPM-based systems, they are provided by these packages: ============ ======================================================= Package Libraries ============ ======================================================= glib2 libglib-2.0.so.0, libgthread-2.0.so.0, libgobject-2.0.so.0 glibc libresolv.so.2, libutil.so.1, libnsl.so.1, librt.so.1, libcrypt.so.1, libpthread.so.0, libdl.so.2, libm.so.6, libc.so.6 libICE libICE.so.6 libX11 libX11.so.6 libXext: libXext.so.6 libXrender libXrender.so.1 libgcc: libgcc_s.so.1 libstdc++ libstdc++.so.6 mesa libGL.so.1 ============ ======================================================= 3. If the wheel contains binary executables or shared objects linked against any whitelisted libraries that also export versioned symbols, they may only depend on the following maximum versions:: GLIBC_2.12 CXXABI_1.3.3 GLIBCXX_3.4.13 GCC_4.3.0 As an example, ``manylinux2`` wheels may include binary artifacts that require ``glibc`` symbols at version ``GLIBC_2.4``, because this an earlier version than the maximum of ``GLIBC_2.12``. 4. If a wheel is built for any version of CPython 2 or CPython versions 3.0 up to and including 3.2, it *must* include a CPython ABI tag indicating its Unicode ABI. A ``manylinux2`` wheel built against Python 2, then, must include either the ``cpy27mu`` tag indicating it was built against an interpreter with the UCS-4 ABI or the ``cpy27m`` tag indicating an interpeter with the UCS-2 ABI. *[Citation for UCS ABI tags?]* 5. A wheel *must not* require the ``PyFPE_jbuf`` symbol. This is achieved by building it against a Python compiled *without* the ``--with-fpectl`` ``configure`` flag. Compilation of Compliant Wheels =============================== Like ``manylinux1``, the ``auditwheel`` tool adds ```manylinux2`` platform tags to ``linux`` wheels built by ``pip wheel`` or ``bdist_wheel`` a ``manylinux2`` Docker container. Docker Images ------------- ``manylinux2`` Docker images based on CentOS 6.9 x86_64 and i686 are provided for building binary ``linux`` wheels that can reliably be converted to ``manylinux2`` wheels. [8]_ These images come with a full compiler suite installed (``gcc``, ``g++``, and ``gfortran`` 4.8.2) as well as the latest releases of Python and ``pip``. Compatibility with kernels that lack ``vsyscall`` ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ A Docker container assumes that its userland is compatible with its host's kernel. Unfortunately, an increasingly common kernel configuration breaks breaks this assumption for x86_64 CentOS 6.9 Docker images. Versions 2.14 and earlier of ``glibc`` require the kernel provide an archaic system call optimization known as ``vsyscall`` on x86_64. [9]_ To effect the optimization, the kernel maps a read-only page of frequently-called system calls -- most notably ``time(2)`` -- into each process at a fixed memory location. ``glibc`` then invokes these system calls by dereferencing a function pointer to the appropriate offset into the ``vsyscall`` page and calling it. This avoids the overhead associated with invoking the kernel that affects normal system call invocation. ``vsyscall`` has long been deprecated in favor of an equivalent mechanism known as vDSO, or "virtual dynamic shared object", in which the kernel instead maps a relocatable virtual shared object containing the optimized system calls into each process. [10]_ The ``vsyscall`` page has serious security implications because it does not participate in address space layout randomization (ASLR). Its predictable location and contents make it a useful source of gadgets used in return-oriented programming attacks. [11]_ At the same time, its elimination breaks the x86_64 ABI, because ``glibc`` versions that depend on ``vsyscall`` suffer from segmentation faults when attempting to dereference a system call pointer into a non-existent page. As a compromise, Linux 3.1 implemented an "emulated" ``vsyscall`` that reduced the executable code, and thus the material for ROP gadgets, mapped into the process. [12]_ ``vsyscall=emulated`` has been the default configuration in most distribution's kernels for many years. Unfortunately, ``vsyscall`` emulation still exposes predicatable code at a reliable memory location, and continues to be useful for return-oriented programming. [13]_ Because most distributions have now upgraded to ``glibc`` versions that do not depend on ``vsyscall``, they are beginning to ship kernels that do not support ``vsyscall`` at all. [14]_ CentOS 5.11 and 6.9 both include versions of ``glibc`` that depend on the ``vsyscall`` page (2.5 and 2.12.2 respectively), so containers based on either cannot run under kernels provided with many distribution's upcoming releases. [15]_ Continuum Analytics faces a related problem with its conda software suite, and as they point out, this will pose a significant obstacle to using these tools in hosted services. [16]_ If Travis CI, for example, begins running jobs under a kernel that does not provide the ``vsyscall`` interface, Python packagers will not be able to use our Docker images there to build ``manylinux`` wheels. [17]_ We have derived a patch from the ``glibc`` git repository that backports the removal of all dependencies on ``vsyscall`` to the version of ``glibc`` included with our ``manylinux2`` image. [18]_ Rebuilding ``glibc``, and thus building ``manylinux2`` image itself, still requires a host kernel that provides the ``vsyscall`` mechanism, but the resulting image can be both run on hosts that provide it and those that do not. Because the ``vsyscall`` interface is an optimization that is only applied to running processes, the ``manylinux2`` wheels built with this modified image should be identical to those built on an unmodified CentOS 6.9 system. Also, the ``vsyscall`` problem applies only to x86_64; it is not part of the i686 ABI. Auditwheel ---------- The ``auditwheel`` tool has also been updated to produce ``manylinux2`` wheels. [19]_ Its behavior and purpose are otherwise unchanged from PEP 513. Platform Detection for Installers ================================= Platforms may define a ``manylinux2_compatible`` boolean attribute on the ``_manylinux`` module described in PEP 513. A platform is considered incompatible with ``manylinux2`` if the attribute is ``False``. Backwards compatibility with ``manylinux1`` wheels ================================================== As explained in PEP 513, the specified symbol versions for ``manylinux1`` whitelisted libraries constitute an *upper bound*. The same is true for the symbol versions defined for ``manylinux2`` in this PEP. As a result, ``manylinux1`` wheels are considered ``manylinux2`` wheels. A ``pip`` that recognizes the ``manylinux2`` platform tag will thus install ``manylinux1`` wheels for ``manylinux2`` platforms -- even when explicitly set -- when no ``manylinux2`` wheels are available. [20]_ PyPI Support ============ PyPI should permit wheels containing the ``manylinux2`` platform tag to be uploaded in the same way that it permits ``manylinux1``. It should not attempt to verify the compatibility of ``manylinux2`` wheels. References ========== .. [1] PEP 513 -- A Platform Tag for Portable Linux Built Distributions (https://www.python.org/dev/peps/pep-0513/) .. [2] pyca/cryptography (https://cryptography.io/) .. [3] numpy (https://numpy.org) .. [4] CentOS 5.11 EOL announcement (https://lists.centos.org/pipermail/centos-announce/2017-April/022350.html) .. [5] CentOS Product Specifications (https://web.archive.org/web/20180108090257/https://wiki.centos.org/About/Pr…) .. [6] PEP 425 -- Compatibility Tags for Built Distributions (https://www.python.org/dev/peps/pep-0425/) .. [7] ncurses 5 -> 6 transition means we probably need to drop some libraries from the manylinux whitelist (https://github.com/pypa/manylinux/issues/94) .. [8] manylinux2 Docker images (https://hub.docker.com/r/markrwilliams/manylinux2/) .. [9] On vsyscalls and the vDSO (https://lwn.net/Articles/446528/) .. [10] vdso(7) (http://man7.org/linux/man-pages/man7/vdso.7.html) .. [11] Framing Signals -- A Return to Portable Shellcode (http://www.cs.vu.nl/~herbertb/papers/srop_sp14.pdf) .. [12] ChangeLog-3.1 (https://www.kernel.org/pub/linux/kernel/v3.x/ChangeLog-3.1) .. [13] Project Zero: Three bypasses and a fix for one of Flash's Vector.<*> mitigations (https://googleprojectzero.blogspot.com/2015/08/three-bypasses-and-fix-for-o…) .. [14] linux: activate CONFIG_LEGACY_VSYSCALL_NONE ? (https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=852620) .. [15] [Wheel-builders] Heads-up re: new kernel configurations breaking the manylinux docker image (https://mail.python.org/pipermail/wheel-builders/2016-December/000239.html) .. [16] Due to glibc 2.12 limitation, static executables that use time(), cpuinfo() and maybe a few others cannot be run on systems that do not support or use `vsyscall=emulate` (https://github.com/ContinuumIO/anaconda-issues/issues/8203) .. [17] Travis CI (https://travis-ci.org/) .. [18] remove-vsyscall.patch https://github.com/markrwilliams/manylinux/commit/e9493d55471d153089df3aafc… .. [19] auditwheel manylinux2 branch (https://github.com/markrwilliams/auditwheel/tree/manylinux2) .. [20] pip manylinux2 branch https://github.com/markrwilliams/pip/commits/manylinux2 Copyright ========= This document has been placed into the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: -- Mark Williams mrw(a)enotuniq.org

1 0

Fwd: Warehouse update: Estimate for 1st milestone, new announce list
by Sumana Harihareswara 30 Jan '18

30 Jan '18

Our weekly Warehouse update just went out to pypa-dev: https://groups.google.com/forum/#!topic/pypa-dev/es_-fC-sdpk and is included below. -------- Forwarded Message -------- Subject: Warehouse update: Estimate for 1st milestone, new announce list Date: Tue, 30 Jan 2018 14:33:36 -0500 From: Sumana Harihareswara <sh(a)changeset.nyc> To: pypa-dev(a)googlegroups.com The big news in Warehouse world is that we tentatively believe we can get our first milestone[0] out by the end of February. So around then you can expect emails and other announcements asking package maintainers to test pypi.org and give us bug reports. Speaking of announcements, we now have a new PyPI-announce mailing list.[1] I encourage you to subscribe. It'll be low-traffic and we'll only post there with major PyPI news. Ernest is nearing the end of his concentration on infrastructure work, especially on Cabotage[2] which helps manage our Kubernetes security.[3] Nicole and Dustin are steadily finishing views and design for maintainer features (e.g., project edit button[4]). And Laura and Sumana are preparing for the publicity push around the Maintainer MVP, and have improved some docs to improve developer experience (e.g., twine[5], Warehouse testing instructions[6], and the PyPA roadmap[7]). Details are in our meeting notes from yesterday.[8] Thanks alanbato for improving Warehouse's error messages![9] If you want to get started contributing to Warehouse, Ernest wants to help you and give you stickers, and has 30-minute 1:1 slots available.[10] Thanks to Mozilla for their support for the PyPI & Warehouse work![11] [0] https://github.com/pypa/warehouse/milestone/8 [1] https://mail.python.org/mm3/mailman3/lists/pypi-announce.python.org/ [2] https://github.com/cabotage/cabotage-app [3] https://github.com/python/pypi-infra/pull/3 [4] https://github.com/pypa/warehouse/pull/2823 [5] https://github.com/pypa/twine/pull/292 [6] https://github.com/pypa/warehouse/pull/2758 [7] https://github.com/pypa/pypa.io/pull/23 [8] https://wiki.python.org/psf/PackagingWG/2018-01-29-Warehouse [9] https://github.com/pypa/warehouse/pull/2767 [10] https://twitter.com/EWDurbin/status/955415184339849217 [11] https://pyfound.blogspot.com/2017/11/the-psf-awarded-moss-grant-pypi.html -- Sumana Harihareswara Warehouse project manager Changeset Consulting https://changeset.nyc

1 0

print module
by Kevin Urban 30 Jan '18

30 Jan '18

Just starting out unable to use print function on a mac. Which module do I import to use print? Kevin Urban

3 2

Current status of ``setup.py register --list-classifiers``
by Takayuki Shimizukawa 30 Jan '18

30 Jan '18

Hi, I tried getting a list of Trove Classifiers by running ``setup.py register --list-classifiers`` command, the HTML of https://upload.pypi.org/legacy/ was displayed on the console. Does this mean that the ``--list-classifiers`` option was broken when the PyPI endpoint changed? Or was it made intentional change? Either way, I could not find such a report or announcement. I confirmed the execution of ``setup.py register --list-classifiers`` with the following combinations. Python-3.5 + distutils setup: OK Python-3.5 + setuptools-26 setup: OK Python-3.5 + setuptools-27 setup: NG Python-3.5 + setuptools-38 setup: NG Python-3.6 + distutils setup: NG Python-3.6 + setuptools-26 setup: NG Python-3.6 + setuptools-27 setup: NG Python-3.6 + setuptools-38 setup: NG Also, in any of the above combinations, it was OK if I specified https://pypi.python.org/pypi as the repository. (BTW, although document says "The --repository or -r option lets you specify a PyPI server different from the default", if there is no URL specified in ``.pypirc`` it will display "ValueError: <URL> not found in .pypirc") So, in my understanding, ``register --list-classifiers`` does not work as expected. Regards, -- Takayuki Shimizukawa

3 4

Any interest in Perforce support in pip?
by Barney Gale 30 Jan '18

30 Jan '18

Hi all, Is there any interest in support for Perforce in pip? Perforce is a commercial version control system used in mostly corporate environments. It's also used for FreeBSD development. I have a pull request against the pypa/pip project (#4857) that allows the following: pip install p4+p4://host/depot/path ... but as the pip developers lack experience with Perforce, I've been directed to this mailing list to gauge whether this is a feature worth adding. I'm aware that Perforce is used in the game and film industries, but beyond a github issue requesting support (#2754) I'm not sure who else might use this! I'd appreciate any feedback. Thanks. Barney

4 4

Begin to Code With Python
by Brian Sujecki 29 Jan '18

29 Jan '18

To whom it may concern: My name is Brian Sujecki and I am currently reading Begin to Code With Python and am unable to import the library snaps even after downloading the library from the command shell on my mac. It's as if shell hasn't updated the library. Can you help me? it's driving me crazy I can't move on. Brian Sujecki

2 1

PEP 508, environment markers & the possibility of Python 3.10
by Nick Coghlan 28 Jan '18

28 Jan '18

Hi folks, In https://github.com/python/peps/issues/560, a user pointed out that the current definition of python_version in PEP 508 assumes single-digit major and minor version numbers: platform.python_version()[:3] There's a reasonable chance we'll see 3.10 rather than 4.0 in a few years time, at which point that definition would break. The suggested fix is to amend that definition to be: ".".join(platform.python_version_tuple()[:2]) This seems like a good suggestion to me, so my inclination is to handle this in a similar way to https://www.python.org/dev/peps/pep-0440/#summary-of-changes-to-pep-440: fix it in place, and add a section at the end of the PEP listing the post-publication changes. Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

4 4

Fwd: Warehouse update: role management & welcoming first-time contributors
by Sumana Harihareswara 24 Jan '18

24 Jan '18

Forwarding from pypa-dev <https://groups.google.com/forum/#!topic/pypa-dev/f7VZ_gh5Je0>. -------- Forwarded Message -------- Subject: Warehouse update: role management & welcoming first-time contributors Date: Tue, 23 Jan 2018 22:33:46 -0500 From: Sumana Harihareswara <sh(a)changeset.nyc> To: pypa-dev(a)googlegroups.com In the past week, the Warehouse team's continued making progress despite a few of us getting sick. The biggest news is that the master branch now includes the foundation for a bunch of useful UI for maintainers. Several people collaborated on a role management feature[0] so a project Owner can add and remove Maintainer and Owner roles for their projects. This enables us to work on further release management features. We made progress on more improvements, including to developer experience, that you'll see in future updates. And thanks to Srinivas Garlapati for starting a password reset feature PR that we were able to finish up and merge.[1] We've turned a number of umbrella issues into more specific issues for the maintainer MVP milestone[2] which we continue working on. And if you're looking for a good first issue as you start contributing to Warehouse, there's one in our current milestone we'd love help with: "Valid `Author-email` and `Maintainer-email` fields are rejected".[3] If you are or know someone who wants to be a first-time contributor, check out Ernest's offer of neat stickers and mentorship time![4] As we get closer to the maintainer MVP milestone, we're preparing to publicize it and future milestones, including to developers who don't usually watch distribution and packaging discussions. So we're making lists of places to post notices, and we're using PyPI data and libraries.io to find projects and maintainers to personally contact. And we're working on future announcement channels, e.g., banners and a special announcement mailing list.[5] Once again, thanks to Mozilla for their support for this project.[6] More next week! [0] https://github.com/pypa/warehouse/pull/2705 [1] https://github.com/pypa/warehouse/pull/2764 [2] https://github.com/pypa/warehouse/milestone/8 [3] https://github.com/pypa/warehouse/issues/2679 [4] https://twitter.com/EWDurbin/status/955413628408205313 [5] https://github.com/python/psf-infra-meta/issues/1 [6] https://pyfound.blogspot.com/2017/11/the-psf-awarded-moss-grant-pypi.html -- Sumana Harihareswara Changeset Consulting https://changeset.nyc

1 0

PEP 566 - metadata 1.3 changes
by Thomas Kluyver 23 Jan '18

23 Jan '18

I hope everyone has had a good break. :-) I'd like to see PEP 566 move forwards. From the last thread, I think people were mostly happy with it as stands, but there were some proposals to introduce further metadata changes. I'd suggest that we finalise the PEP as it currently stands, and save up other changes for a future metadata revision. One change I would like to the current text is to make more explicit that the format of several fields allowing version specifiers (Requires-Dist, Provides-Dist, Obsoletes-Dist, Requires-External) has changed, as PEP 508 removes the need for parentheses around the version specifier. The section on version specifiers currently refers to PEP 440, which doesn't mention the parentheses at all, and says they are otherwise unchanged from PEP 345, which says parentheses are required. I would like to add some text to that section, such as: "Following PEP 508, version specifiers no longer need to be surrounded by parentheses in the fields Requires-Dist, Provides-Dist, Obsoletes-Dist or Requires-External, so e.g. `requests >= 2.8.1` is now a valid value. The recommended format is without parentheses, but tools parsing metadata should also be able to handle version specifiers in parentheses." Thanks, Thomas

7 21

pipenv use cases
by Paul Moore 19 Jan '18

19 Jan '18

On 16 January 2018 at 10:03, Nick Coghlan <ncoghlan(a)gmail.com> wrote: > On 16 January 2018 at 19:47, Paul Moore <p.f.moore(a)gmail.com> wrote: >> I think that if the pipenv docs had some better guidance on what use >> cases it was intended to cover (and what it wasn't, in relation to the >> broader range of pip+virtualenv use cases) that would help people >> better understand its place in the ecosystem. > > That's fair, and making the PyPUG tutorial specifically about > *application* dependency management was an initial step in that > direction - for the library development use case, you're generally > going to step out of pipenv's world as soon as you try to run your > tests across multiple versions. > > The basic usage constraint is that pipenv expects you to control your > target Python version, and it expects you to have exactly one - to go > beyond that (as is needed for multi-version library support), you'll > still need to drop down to the lower level tooling, either skipping > the use of pipenv entirely, or else by using `pipfile lock > --requirements` to shift levels. New subject as I don't want to hijack the original thread to rehash my old issue, but I do want to make a couple of points on this (and I agree, it's hard to find a good forum to discuss things like this as it stands). 1. "pipenv expects you to control your target Python version" - I'm not 100% sure what that means, but if it's saying "pipenv is only really for code that will only be used with a single Python version" then that's basically excluding a huge chunk of Python development. Specifically, library development of all forms. Admittedly my experience of what's "typical" is biased strongly by the communities I work with, but conversely the "writing standalone apps in Python" community doesn't really seem to have a web presence that we can promote pipenv through, so it's becoming visible via the "general Python development" community, which quite biased to libraries, and so is not the right target audience, from what you say. 2. My specific issues weren't outside that constraint - I *was* writing code that would only ever need to run under Python 3.6. My problem was the need for "local" build dependencies, which seems entirely within that use case. In fairness, the pipenv devs weren't totally against the functionality I needed, it's just not something they had considered important. Maybe the problem here is that there isn't a good set of "developing apps in Python" best practices (as opposed to the library development situation - use install_requires, test with tox, ...), so I didn't know the "recommended" solution to my problem, that pipenv would have been expecting me to use. Maybe it's a chicken-and-egg situation - the "best practice" is to use pipenv, but until pipenv gets to encounter all the various use cases, that "best practice" doesn't properly cover every situation... Paul

8 17

Deprecating/Removing OpenID/Google login support for PyPI
by Donald Stufft 18 Jan '18

18 Jan '18

As folks are likely aware, legacy PyPI currently supports logging in using OpenID and Google Auth while Warehouse does not. After much deliberation, I’ve decided that Warehouse will not be implementing OpenID or Google logins, and once we shutdown legacy PyPI, OpenID/ and Google logins to PyPI will no longer be possible. This decision was made for a few reasons: * Very few people actually are using OpenID or Google logins as it is. In one month we had ~15k logins using the web form, ~5k using basic auth, and 62 using Google and 7 using OpenID. This is a several orders of magnitude difference. * Regardless of how you log into PyPI (Password or Google/OpenID) you’re required to have a password added to your account to actually upload anything to PyPI. This negates much of the benefit of a federated authentication for PyPI as it stands. * Keeping these requires ongoing maintenance to deal with any changes in the specification or to update as Google deprecates/changes things. * Adding support for them to Warehouse requires additional work that could better be used elsewhere, where it would have a higher impact. - Donald

8 7

barriers to Warehouse contribution
by Sumana Harihareswara 16 Jan '18

16 Jan '18

I've sent a question to pypa-dev https://groups.google.com/forum/#!topic/pypa-dev/U6NO55bExbE which can be summarized: what should we fix or set up to make it easier for you to hack on Warehouse, especially with a goal of a Warehouse/packaging sprint at PyCon in May? I'd appreciate discussion on pypa-dev to keep it in one place (that list tends to get ~15 posts/month), but if you aren't subscribed there and don't want to be, go ahead & reply here. :) -- Sumana Harihareswara Changeset Consulting https://changeset.nyc

1 0

pipenv best practices?
by Chris Withers 16 Jan '18

16 Jan '18

Hi All, Couldn't find an obviously better discussion forum than this for pipenv stuff, but please point me to where the rest of the discussions are happening... So, with pipenv, what files do we version control for a project? both Pipfile and Pipfile.lock? Hopefully one I missed from the docs: with the correct files source controlled, how do I reproduce the environment on another machine? Continuing on the deployment theme: I'm used to being able to put /path/to/some/ve/bin/some_script in crontabs and the like, where some_script comes form a setuptools entry point. What's the canonical way of doing this with pipenv? Last up, how should pipenv be used for library development? (ie: where dependencies and minimal constraints are expressed in setup.py and where you want to test against as many python versions and library combinations as you support). Other than sadness around capitalisation of the config file names, it looks really exciting :-) cheers, Chris

4 7

(no subject)
by bige12655 . 16 Jan '18

16 Jan '18

Thanks

1 0

Fwd: Warehouse update: recent progress
by Sumana Harihareswara 16 Jan '18

16 Jan '18

Forwarding from pypa-dev. -------- Forwarded Message -------- Subject: Warehouse update: recent progress Date: Mon, 15 Jan 2018 22:32:45 -0500 From: Sumana Harihareswara <sh(a)changeset.nyc> To: pypa-dev(a)googlegroups.com Happy new year! After the late-December holiday season, the Warehouse team had a meeting and followup chats last week to discuss the first milestone, the MVP for package maintainers to try Warehouse and give feedback.[0] Full notes from a January 10th meeting and its followups are on the PSF wiki.[1] Highlights from that meeting and related work in the last week and change: * A number of items were blocked on Nicole Harris's availability -- she's the designer working on our front-end views. We sorted out things she needs and are making better progress on those now. * Donald wrote to distutils-sig[2] about removing OpenID/Google login support for PyPI.[3] * Ernest and Nicole are working on a maintainer views meta-issue to "Classify, describe, and share the *current* functionality implemented by PyPI to assist contributors while developing warehouse's maintainer UI."[4] * The Warehouse roadmap, which previously only lived in GitHub milestone descriptions, is now all in one wiki page for easier reading and linking.[5] * Dustin has been working on adding the ability for a project owner to add/remove roles for their projects.[6] * Ernest has been continuing to work on deployment infrastructure, which is getting more robust and resilient. * We've created, reviewed, improved, and/or merged several other pull requests to Warehouse and related projects. In particular, development is now getting easier, as we have an SMTP service for development[7] and a better Makefile[8], and are getting better testing instructions.[9] We are incrementally getting a clearer picture of what work is left in this milestone so I can get you an estimate of when the Maintainer MVP will be. And Warehouse continues to get easier to hack on; feel free to join us on Freenode in case you want any help as you poke around the Warehouse code![10] [0] https://github.com/pypa/warehouse/milestone/8 [1] https://wiki.python.org/psf/PackagingWG/2018-01-10-Warehouse [2] https://mail.python.org/pipermail/distutils-sig/2018-January/031855.html [3] https://github.com/pypa/warehouse/issues/61 [4] https://github.com/pypa/warehouse/issues/2734 [5] https://wiki.python.org/psf/WarehouseRoadmap [6] https://github.com/pypa/warehouse/pull/2705 [7] https://github.com/pypa/warehouse/pull/2709 [8] https://github.com/pypa/warehouse/pull/2728 [9] https://github.com/pypa/warehouse/pull/2758 [10] https://webchat.freenode.net/?channels=%23pypa-dev -- Sumana Harihareswara Changeset Consulting https://changeset.nyc

1 0

Re: [Distutils] Should abi3 tag be usable on Windows?
by Cosimo Lupo 13 Jan '18

13 Jan '18

Thank you, Steve. Ok so, setuptools can successfully build extension modules with Py_LIMITED_API on Windows, with a generic *.pyd suffix and no extra tags. However, there are still two problems before I can distribute them: 1) it's not clear how wheels containing such extension modules should be tagged on Windows, given that `abi3` tag is not recognized for that platform. The function imp.get_suffixes() used by both pip and wheel to get the list of supported tags does not return any abi3 suffixes on Windows, and for this reason the bdist_wheel command assumes from this that Windows does not support extensions built with Py_LIMITED_API, and gives an error. Some have resorted to tagging their Windows wheels with abi "none" to work around this, however "none" seems to imply that extension does not use the Python ABI which is not true. Alex Grönholm (current maintainer of pypa/wheel package) pointed me to this other related discussion on distutils-sig (from 2013) where Paul Moore explains what the problem is: https://mail.python.org/pipermail/distutils-sig/2013-February/020022.html Has anything happened since the to address this either in the PEPs and/or the tooling? 2) the current virtualenv module (not the built-in venv) does not copy over the required PYTHON3.DLL and thus importing such extension modules from within the virtual environment fails. The latter issue was already fixed in virtualenv's master branch, but virtualenv package hasn't seen a release since then (the last release 15.10.0 was in November 2016). https://github.com/pypa/virtualenv/pull/1083 I would love to take advantage of CPython limited API feature, and be able to distribute a one/two wheels per platform instead of four (multiply that by two for 32 vs 64 bit arch), and having to rebuild one for each new Python 3.x release. It works fine for Linux and macOS; I hope that Windows python users will also be able to enjoy this soon. I am willing to contribute my time to help solving this issue. Thanks, Cosimo On Mon, Jan 8, 2018 at 1:31 AM Steve Dower <steve.dower(a)python.org> wrote: > There is a solution to that problem on the linked issue. Basically, you > need to declare Py_LIMITED_API in your code or as an extra preprocessor > variable. > > > > Windows doesn’t use a filename suffix for python3.dll linked extensions as > it will be handled at load time. The tag for the wheel is outside of my > area, so hopefully someone can chime in on that for you. > > > > Cheers, > > Steve > > > > Top-posted from my Windows phone > > > > *From: *Cosimo Lupo <cosimo(a)anthrotype.com> > *Sent: *Monday, January 8, 2018 10:32 > *To: *distutils-sig(a)python.org > *Subject: *Re: [Distutils] Should abi3 tag be usable on Windows? > > > > It turns out setuptools is still linking to PYTHON36.DLL instead of > PYTHNO3.DLL even when py_limited_api=True > > https://github.com/pypa/setuptools/issues/1248 > > > > This is a different, though somewhat related, problem from the one > concering wheel/pip pep425tags disallowing `abi3` tag on Windows. > > > > > > On Sun, Jan 7, 2018 at 12:39 PM Cosimo Lupo <cosimo(a)anthrotype.com> wrote: > > I jut found this related pip issue: > https://github.com/pypa/pip/issues/4445 > > > > (I myself as @anthrotype commented on that thread back in April last year > but then completely forgot...) > > > > After re-reading it now, it's still not clear to me what the resolution on > that issue was. > > > > On Sun, Jan 7, 2018 at 10:12 AM Cosimo Lupo <cosimo(a)anthrotype.com> wrote: > > Hello, > > CFFI has recently added support for Py_LIMITED_API extension modules built > for CPython 3. > The wheel module since version 0.30.0 also supports passing > —py-limited-api cp3X to bdist_wheel command to allow the generated .whl to > be installed on all CPython versions equal or greater than the one > specified. > > Yesterday I was trying to apply this on a cffi-built extension module, and > it worked for Linux and macOS but failed for Windows: > > https://github.com/hynek/argon2_cffi/pull/32 > <https://github.com/hynek/argon2_cffi/pull/32#issuecomment-355771832> > The AssertionError from wheel.pep425tags complains that a tag with abi3 > would be unsupported for the target platform. > > Alex Gronholm commented > > > imp.get_suffixes() does not seem to contain any ABI3 suffixes, but I'm > not sure if this is even applicable on Windows. > > Incidentally, I noticed one specific package, PyQt5, that distributes both > abi3-tagged wheels for Mac and manylinux and Windows wheels for a range of > cp35.cp36.cp37 but with abi tag set as “none”, and they do seem to work. > > So, can one make such py_limited_api wheels work on Windows with the > current state of the tooling, and if so how? > > Thank you in advance > > > > Cosimo Lupo > > > > -- > > Cosimo Lupo > > > > -- > > Cosimo Lupo > > > -- Cosimo Lupo

1 0