Python-Dev February 2013

python-dev@python.org

103 participants
84 discussions

by Daniel Holth

This is PEP 427: Wheel. A binary package format for Python. Because newegg was taken. Since the last submission, the signature specification has been made clearly optional / informative and does not attempt to specify any specific signing algorithm or how the signatures would fit into a security scheme that would necessarily exist outside of a single archive. JWS signatures are used in their current form by OpenID Connect and Mozilla Personas and are a useful way to implement basically raw public or secret key signatures. The embedded signature scheme in wheel should also not affect the current effort to define end-to-end security for PyPI in any way; it might be a useful complement for packages that are not hosted on PyPI at all. This version also does not depend on the in-process Metadata PEP 426. It has been cleaned up in several places regarding which PEPs it references to describe its contents. The WHEEL metadata file should contain all the information in the wheel filename itself, and the non-alphanumeric/unicode filename escaping rules are made official. Thanks. For your consideration, PEP: 427 Title: The Wheel Binary Package Format 0.1 Version: $Revision$ Last-Modified: $Date$ Author: Daniel Holth <dholth(a)fastmail.fm> BDFL-Delegate: Nick Coghlan <ncoghlan(a)gmail.com> Discussions-To: <distutils-sig(a)python.org> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 20-Sep-2012 Post-History: 18-Oct-2012, 15-Feb-2013 Abstract ======== This PEP describes a built-package format for Python called "wheel". A wheel is a ZIP-format archive with a specially formatted file name and the ``.whl`` extension. It contains a single distribution nearly as it would be installed according to PEP 376 with a particular installation scheme. Although a specialized installer is recommended, a wheel file may be installed by simply unpacking into site-packages with the standard 'unzip' tool while preserving enough information to spread its contents out onto their final paths at any later time. Note ==== This draft PEP describes version 0.1 of the "wheel" format. When the PEP is accepted, the version will be changed to 1.0. (The major version is used to indicate potentially backwards-incompatible changes to the format.) Rationale ========= Python needs a package format that is easier to install than sdist. Python's sdist packages are defined by and require the distutils and setuptools build systems, running arbitrary code to build-and-install, and re-compile, code just so it can be installed into a new virtualenv. This system of conflating build-install is slow, hard to maintain, and hinders innovation in both build systems and installers. Wheel attempts to remedy these problems by providing a simpler interface between the build system and the installer. The wheel binary package format frees installers from having to know about the build system, saves time by amortizing compile time over many installations, and removes the need to install a build system in the target environment. Details ======= Installing a wheel 'distribution-1.0.py32.none.any.whl' ------------------------------------------------------- Wheel installation notionally consists of two phases: - Unpack. a. Parse ``distribution-1.0.dist-info/WHEEL``. b. Check that installer is compatible with Wheel-Version. Warn if minor version is greater, abort if major version is greater. c. If Root-Is-Purelib == 'true', unpack archive into purelib (site-packages). d. Else unpack archive into platlib (site-packages). - Spread. a. Unpacked archive includes ``distribution-1.0.dist-info/`` and (if there is data) ``distribution-1.0.data/``. b. Move each subtree of ``distribution-1.0.data/`` onto its destination path. Each subdirectory of ``distribution-1.0.data/`` is a key into a dict of destination directories, such as ``distribution-1.0.data/(purelib|platlib|headers|scripts|data)``. The initially supported paths are taken from ``distutils.command.install``. c. If applicable, update scripts starting with ``#!python`` to point to the correct interpreter. d. Update ``distribution-1.0.dist.info/RECORD`` with the installed paths. e. Remove empty ``distribution-1.0.data`` directory. f. Compile any installed .py to .pyc. (Uninstallers should be smart enough to remove .pyc even if it is not mentioned in RECORD.) Recommended installer features '''''''''''''''''''''''''''''' Rewrite ``#!python``. In wheel, scripts are packaged in ``{distribution}-{version}.data/scripts/``. If the first line of a file in ``scripts/`` starts with exactly ``b'#!python'``, rewrite to point to the correct interpreter. Unix installers may need to add the +x bit to these files if the archive was created on Windows. Generate script wrappers. In wheel, scripts packaged on Unix systems will certainly not have accompanying .exe wrappers. Windows installers may want to add them during install. File Format ----------- File name convention '''''''''''''''''''' The wheel filename is ``{distribution}-{version}(-{build tag})?-{python tag}-{abi tag}-{platform tag}.whl``. distribution Distribution name, e.g. 'django', 'pyramid'. version Distribution version, e.g. 1.0. build tag Optional build number. Must start with a digit. A tie breaker if two wheels have the same version. Sort as the empty string if unspecified, else sort the initial digits as a number, and the remainder lexicographically. language implementation and version tag E.g. 'py27', 'py2', 'py3'. abi tag E.g. 'cp33m', 'abi3', 'none'. platform tag E.g. 'linux_x86_64', 'any'. For example, ``distribution-1.0-1-py27-none-any.whl`` is the first build of a package called 'distribution', and is compatible with Python 2.7 (any Python 2.7 implementation), with no ABI (pure Python), on any CPU architecture. The last three components of the filename before the extension are called "compatibility tags." The compatibility tags express the package's basic interpreter requirements and are detailed in PEP 425. Escaping and Unicode '''''''''''''''''''' Each component of the filename is escaped by replacing runs of non-alphanumeric characters with an underscore ``_``:: re.sub("[^\w\d.]+", "_", distribution, re.UNICODE) The filename is Unicode. It will be some time before the tools are updated to support non-ASCII filenames, but they are supported in this specification. File contents ''''''''''''' The contents of a wheel file, where {distribution} is replaced with the name of the package, e.g. ``beaglevote`` and {version} is replaced with its version, e.g. ``1.0.0``, consist of: #. ``/``, the root of the archive, contains all files to be installed in ``purelib`` or ``platlib`` as specified in ``WHEEL``. ``purelib`` and ``platlib`` are usually both ``site-packages``. #. ``{distribution}-{version}.dist-info/`` contains metadata. #. ``{distribution}-{version}.data/`` contains one subdirectory for each non-empty install scheme key not already covered, where the subdirectory name is an index into a dictionary of install paths (e.g. ``data``, ``scripts``, ``include``, ``purelib``, ``platlib``). #. Python scripts must appear in ``scripts`` and begin with exactly ``b'#!python'`` in order to enjoy script wrapper generation and ``#!python`` rewriting at install time. They may have any or no extension. #. ``{distribution}-{version}.dist-info/METADATA`` is Metadata version 1.1 or greater (PEP 314, PEP 345, PEP 426) format metadata. #. ``{distribution}-{version}.dist-info/WHEEL`` is metadata about the archive itself in the same basic key: value format:: Wheel-Version: 0.1 Generator: bdist_wheel 0.7 Root-Is-Purelib: true Tag: py2-none-any Tag: py3-none-any Build: 1 #. ``Wheel-Version`` is the version number of the Wheel specification. ``Generator`` is the name and optionally the version of the software that produced the archive. ``Root-Is-Purelib`` is true if the top level directory of the archive should be installed into purelib; otherwise the root should be installed into platlib. ``Tag`` is the wheel's expanded compatibility tags; in the example the filename would contain ``py2.py3-none-any``. ``Build`` is the build number and is omitted if there is no build number. #. A wheel installer should warn if Wheel-Version is greater than the version it supports, and must fail if Wheel-Version has a greater major version than the version it supports. #. Wheel, being an installation format that is intended to work across multiple versions of Python, does not generally include .pyc files. #. Wheel does not contain setup.py or setup.cfg. This version of the wheel specification is based on the distutils install schemes and does not define how to install files to other locations. The layout offers a superset of the functionality provided by the existing wininst and egg binary formats. The .dist-info directory ^^^^^^^^^^^^^^^^^^^^^^^^ #. Wheel .dist-info directories include at a minimum METADATA, WHEEL, and RECORD. #. METADATA is the package metadata, the same format as PKG-INFO as found at the root of sdists. #. WHEEL is the wheel metadata specific to a build of the package. #. RECORD is a list of (almost) all the files in the wheel and their secure hashes. Unlike PEP 376, every file except RECORD, which cannot contain a hash of itself, must include its hash. The hash algorithm must be sha256 or better; specifically, md5 and sha1 are not permitted, as signed wheel files rely on the strong hashes in RECORD to validate the integrity of the archive. #. PEP 376's INSTALLER and REQUESTED are not included in the archive. #. RECORD.jws is used for digital signatures. It is not mentioned in RECORD. #. RECORD.p7s is allowed as a courtesy to anyone who would prefer to use S/MIME signatures to secure their wheel files. It is not mentioned in RECORD. #. During extraction, wheel installers verify all the hashes in RECORD against the file contents. Apart from RECORD and its signatures, installation will fail if any file in the archive is not both mentioned and correctly hashed in RECORD. The .data directory ^^^^^^^^^^^^^^^^^^^ Any file that is not normally installed inside site-packages goes into the .data directory, named as the .dist-info directory but with the .data/ extension:: distribution-1.0.dist-info/ distribution-1.0.data/ The .data directory contains subdirectories with the scripts, headers, documentation and so forth from the distribution. During installation the contents of these subdirectories are moved onto their destination paths. Signed wheel files ------------------ Wheel files include an extended RECORD that enables digital signatures. PEP 376's RECORD is altered to include a secure hash ``digestname=urlsafe_b64encode_nopad(digest)`` (urlsafe base64 encoding with no trailing = characters) as the second column instead of an md5sum. All possible entries are hashed, including any generated files such as .pyc files, but not RECORD which cannot contain its own hash. For example:: file.py,sha256=AVTFPZpEKzuHr7OvQZmhaU3LvwKz06AJw8mT\_pNh2yI,3144 distribution-1.0.dist-info/RECORD,, The signature file(s) RECORD.jws and RECORD.p7s are not mentioned in RECORD at all since they can only be added after RECORD is generated. Every other file in the archive must have a correct hash in RECORD or the installation will fail. If JSON web signatures are used, one or more JSON Web Signature JSON Serialization (JWS-JS) signatures is stored in a file RECORD.jws adjacent to RECORD. JWS is used to sign RECORD by including the SHA-256 hash of RECORD as the signature's JSON payload:: { "hash": "sha256=ADD-r2urObZHcxBW3Cr-vDCu5RJwT4CaRTHiFmbcIYY" } If RECORD.p7s is used, it must contain a detached S/MIME format signature of RECORD. A wheel installer is not required to understand digital signatures but MUST verify the hashes in RECORD against the extracted file contents. When the installer checks file hashes against RECORD, a separate signature checker only needs to establish that RECORD matches the signature. See - http://self-issued.info/docs/draft-ietf-jose-json-web-signature.html - http://self-issued.info/docs/draft-jones-jose-jws-json-serialization.html - http://self-issued.info/docs/draft-ietf-jose-json-web-key.html - http://self-issued.info/docs/draft-jones-jose-json-private-key.html Comparison to .egg ------------------ #. Wheel is an installation format; egg is importable. Wheel archives do not need to include .pyc and are less tied to a specific Python version or implementation. Wheel can install (pure Python) packages built with previous versions of Python so you don't always have to wait for the packager to catch up. #. Wheel uses .dist-info directories; egg uses .egg-info. Wheel is compatible with the new world of Python packaging and the new concepts it brings. #. Wheel has a richer file naming convention for today's multi-implementation world. A single wheel archive can indicate its compatibility with a number of Python language versions and implementations, ABIs, and system architectures. Historically the ABI has been specific to a CPython release, wheel is ready for the stable ABI. #. Wheel is lossless. The first wheel implementation bdist_wheel always generates egg-info, and then converts it to a .whl. It is also possible to convert existing eggs and bdist_wininst distributions. #. Wheel is versioned. Every wheel file contains the version of the wheel specification and the implementation that packaged it. Hopefully the next migration can simply be to Wheel 2.0. #. Wheel is a reference to the other Python. FAQ === Wheel defines a .data directory. Should I put all my data there? This specification does not have an opinion on how you should organize your code. The .data directory is just a place for any files that are not normally installed inside ``site-packages`` or on the PYTHONPATH. In other words, you may continue to use ``pkgutil.get_data(package, resource)`` even though *those* files will usually not be distributed in *wheel's* ``.data`` directory. Why does wheel include attached signatures? Attached signatures are more convenient than detached signatures because they travel with the archive. Since only the individual files are signed, the archive can be recompressed without invalidating the signature or individual files can be verified without having to download the whole archive. Why does wheel allow JWS signatures? The JOSE specifications of which JWS is a part are designed to be easy to implement, a feature that is also one of wheel's primary design goals. JWS yields a useful, concise pure-Python implementation. Why does wheel also allow S/MIME signatures? S/MIME signatures are allowed for users who need or want to use existing public key infrastructure with wheel. Signed packages are only a basic building block in a secure package update system. Wheel only provides the building block. Appendix ======== Example urlsafe-base64-nopad implementation:: # urlsafe-base64-nopad for Python 3 import base64 def urlsafe_b64encode_nopad(data): return base64.urlsafe_b64encode(data).rstrip(b'=') def urlsafe_b64decode_nopad(data): pad = b'=' * (4 - (len(data) & 3)) return base64.urlsafe_b64decode(data + pad) Copyright ========= This document has been placed into the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

8 years, 1 month

Point of building without threads?

by Antoine Pitrou

Hello, I guess a long time ago, threading support in operating systems wasn't very widespread, but these days all our supported platforms have it. Is it still useful for production purposes to configure --without-threads? Do people use this option for something else than curiosity of mind? Regards Antoine.

8 years, 1 month

Proposing "Argument Clinic", a new way of specifying arguments to builtins for CPython

by Larry Hastings

Say there, the Python core development community! Have I got a question for you! *ahem* Which of the following four options do you dislike least? ;-) 1) CPython continues to provide no "function signature" objects (PEP 362) or inspect.getfullargspec() information for any function implemented in C. 2) We add new hand-coded data structures representing the metadata necessary for function signatures for builtins. Which means that, when defining arguments to functions in C, we'd need to repeat ourselves *even more* than we already do. 3) Builtin function arguments are defined using some seriously uncomfortable and impenetrable C preprocessor macros, which produce all the various types of output we need (argument processing code, function signature metadata, possibly the docstrings too). 4) Builtin function arguments are defined in a small DSL; these are expanded to code and data using a custom compile-time preprocessor step. All the core devs I've asked said "given all that, I'd prefer the hairy preprocessor macros". But by the end of the conversation they'd changed their minds to prefer the custom DSL. Maybe I'll make a believer out of you too--read on! I've named this DSL preprocessor "Argument Clinic", or Clinic for short**. Clinic works similarly to Ned Batchelder's brilliant "Cog" tool: http://nedbatchelder.com/code/cog/ You embed the input to Clinic in a comment in your C file, and the output is written out immediately after that comment. The output's overwritten every time the preprocessor is run. In short it looks something like this: /*[clinic] input to the DSL [clinic]*/ ... output from the DSL, overwritten every time ... /*[clinic end:<checksum>]*/ The input to the DSL includes all the metadata about the function that we need for the function signature: * the name of the function, * the return annotation (if any), * each parameter to the function, including * its name, * its type (in C), * its default value, * and a per-parameter docstring; * and the docstring for the function as a whole. The resulting output contains: * the docstring for the function, * declarations for all your parameters, * C code handling all argument processing for you, * and a #define'd methoddef structure for adding the function to the module. I discussed this with Mark "HotPy" Shannon, and he suggested we break our existing C functions into two. We put the argument processing into its own function, generated entirely by Clinic, and have the implementation in a second function called from the first. I like this approach simply because it makes the code cleaner. (Note that this approach should not cause any overhead with a modern compiler, as both functions will be "static".) But it also provides an optimization opportunity for HotPy: it could read the metadata, and when generating the JIT'd code it could skip building the PyObjects and argument tuple (and possibly keyword argument dict), and the subsequent unpacking/decoding, and just call the implementation function directly, giving it a likely-measurable speed boost. And we can go further! If we add a new extension type API allowing you to register both functions, and external modules start using it, sophisticated Python implementations like PyPy might be able to skip building the tuple for extension type function calls--speeding those up too! Another plausible benefit: alternate implementations of Python could read the metadata--or parse the input to Clinic themselves--to ensure their reimplementations of the Python standard library conform to the same API! Clinic can also run general-purpose Python code ("/*[python]"). All output from "print" is redirected into the output section after the Python code. As you've no doubt already guessed, I've made a prototype of Argument Clinic. You can see it--and some sample conversions of builtins using it for argument processing--at this BitBucket repo: https://bitbucket.org/larry/python-clinic I don't claim that it's fabulous, production-ready code. But it's a definite start! To save you a little time, here's a preview of using Clinic for dbm.open(). The stuff at the same indent as a declaration are options; see the "clinic.txt" in the repo above for full documentation. /*[clinic] dbm.open -> mapping basename=dbmopen const char *filename; The filename to open. const char *flags="r"; How to open the file. "r" for reading, "w" for writing, etc. int mode=0666; default=0o666 If creating a new file, the mode bits for the new file (e.g. os.O_RDWR). Returns a database object. [clinic]*/ PyDoc_STRVAR(dbmopen__doc__, "dbm.open(filename[, flags=\'r\'[, mode=0o666]]) -> mapping\n" "\n" " filename\n" " The filename to open.\n" "\n" " flags\n" " How to open the file. \"r\" for reading, \"w\" for writing, etc.\n" "\n" " mode\n" " If creating a new file, the mode bits for the new file\n" " (e.g. os.O_RDWR).\n" "\n" "Returns a database object.\n" "\n"); #define DBMOPEN_METHODDEF \ {"open", (PyCFunction)dbmopen, METH_VARARGS | METH_KEYWORDS, dbmopen__doc__} static PyObject * dbmopen_impl(PyObject *self, const char *filename, const char *flags, int mode); static PyObject * dbmopen(PyObject *self, PyObject *args, PyObject *kwargs) { const char *filename; const char *flags = "r"; int mode = 0666; static char *_keywords[] = {"filename", "flags", "mode", NULL}; if (!PyArg_ParseTupleAndKeywords(args, kwargs, "s|si", _keywords, &filename, &flags, &mode)) return NULL; return dbmopen_impl(self, filename, flags, mode); } static PyObject * dbmopen_impl(PyObject *self, const char *filename, const char *flags, int mode) /*[clinic end:eddc886e542945d959b44b483258bf038acf8872]*/ As of this writing, I also have sample conversions in the following files available for your perusal: Modules/_cursesmodule.c Modules/_dbmmodule.c Modules/posixmodule.c Modules/zlibmodule.c Just search in C files for '[clinic]' and you'll find everything soon enough. As you can see, Clinic has already survived some contact with the enemy. I've already converted some tricky functions--for example, os.stat() and curses.window.addch(). The latter required adding a new positional-only processing mode for functions using a legacy argument processing approach. (See "clinic.txt" for more.) If you can suggest additional tricky functions to support, please do! Big unresolved questions: * How would we convert all the builtins to use Clinic? I fear any solution will involve some work by hand. Even if we can automate big chunks of it, fully automating it would require parsing arbitrary C. This seems like overkill for a one-shot conversion. (Mark Shannon says he has some ideas.) * How do we create the Signature objects? My current favorite idea: Clinic also generates a new, TBD C structure defining all the information necessary for the signature, which is also passed in to the new registration API (you remember, the one that takes both the argument-processing function and the implementation function). This is secreted away in some new part of the C function object. At runtime this is converted on-demand into a Signature object. Default values for arguments are represented in C as strings; the conversion process attempts eval() on the string, and if that works it uses the result, otherwise it simply passes through the string. * Right now Clinic paves over the PyArg_ParseTuple API for you. If we convert CPython to use Clinic everywhere, theoretically we could replace the parsing API with something cleaner and/or faster. Does anyone have good ideas (and time, and energy) here? * There's actually a fifth option, proposed by Brett Cannon. We constrain the format of docstrings for builtin functions to make them machine-readable, then generate the function signature objects from that. But consider: generating *everything* in the signature object may get a bit tricky (e.g. Parameter.POSITIONAL_ONLY), and this might gunk up the docstring. But the biggest unresolved question... is this all actually a terrible idea? //arry/ ** "Is this the right room for an argument?" "I've told you once...!"

8 years, 1 month

Re: [Python-Dev] A new webpage promoting Compiler technology for CPython

by Paul Boddie

Travis Oliphant wrote: > > With Numba and Blaze we have been doing a lot of work on what essentially > is compiler technology and realizing more and more that we are treading on > ground that has been plowed before with many other projects. So, we > wanted to create a web-site and perhaps even a mailing list or forum where > people could coordinate and communicate about compiler projects, compiler > tools, and ways to share efforts and ideas. > > The website is: http://compilers.pydata.org/ This is a rather nice resource. Thank you for letting us know about it! There has been an attempt to record different Python implementations on the Python Wiki, and now that this is available again, I'd like to remind people about it: http://wiki.python.org/moin/PythonImplementations I know that this isn't quite the same thing as a page about compiler technology, but there is a substantial overlap. > This page is specifically for Compiler projects that either integrate with > or work directly with the CPython run-time which is why PyPy is not > presently listed. The PyPy project is a great project but we just felt > that we wanted to explicitly create a collection of links to compilation > projects that are accessible from CPython which are likely less well known. I think that given the scope of the projects listed, it shouldn't preclude PyPy from being listed, really. After all, interoperability with CPython extensions is something of a focus area for PyPy: http://pypy.org/compat.html I don't have an agenda here - I don't use PyPy actively, my only involvement with Shedskin (which is referenced and which can produce CPython extension modules) is in packaging it for Debian, and although I do have a static analysis library I see no pressing need to promote it extensively - but I feel that when it comes to educational resources people should be a bit careful about drawing boundaries that exclude things that would benefit people substantially if they only knew about it. My reason for speaking up about this is that I've had to tell a room full of people who were told to use Cython, NumPy and even plain C to make their Python programs faster that PyPy existed. (Of course, one can justify ignoring the elephant in the room by claiming things like scientific users rely on native libraries or CPython extensions - since "science" is a very broad term, this obviously isn't universally true - but I think people should be entitled to make their own minds up, and I was not completely certain that everyone involved in the case in question was oblivious to PyPy's existence or status.) > But that is just where we started from. The website is intended to be a > community website constructed from a github repository. So, we welcome > pull requests from anyone who would like to see the website updated to > reflect their related project. Jon Riehl (Mython, PyFront, ROFL, and > many other interesting projects) and Stephen Diehl (Blaze) and I will be > moderating the pull requests to begin with. But, we welcome others with > similar interests to participate in that effort of moderation. > > The github repository is here: https://github.com/pydata/compilers-webpage > > This is intended to be a community website for information spreading, and > so we welcome any and all contributions. There is also the python-static-type-checking Google Group: https://groups.google.com/forum/?fromgroups#!forum/python-static-type-check… If no-one beats me to it, I may post details of the site to that group because it may well be of interest to the members. Thanks once again for bringing such information together in one place! Paul

8 years, 1 month

Re: [Python-Dev] A new webpage promoting Compiler technology for CPython

by Paul Boddie

Stefan Behnel wrote: > > This is off-topic for this list, but the main problem with PyPy is that > you'll quickly hit a lot of walls when you try to use it for anything > serious in the area. It's true that there is a certain level of > interoperability with CPython extensions, but calling it a "focus area" > makes it sound bigger than it actually is in my ears. It is a focus area. They even have a Web page for it, which I mentioned. > Even trying to get bugs fixed to at least make things work at all often > means running against walls on their side. I can tell, trying to port > Cython mostly consisted of bugging PyPy developers to fix stuff, which took > anything from days to still-not-done, each time. And, by design, PyPy makes > it very hard and time consuming to debug it and to try to fix bugs in their > code base. I'm sure it is tricky to debug PyPy. However, complete compatibility with CPython for Python programs is a goal of the project, and I have no reason to doubt that the project is committed to supporting broader compatibility with CPython. > So, while I agree that PyPy is worth a try in certain application areas, > and can be helpful for some special needs, also in the field of scientific > computing, it's lightyears away from a production-ready state in that area. You are generalising too broadly, which is precisely what I mentioned in my last message. There are also plenty of people doing science who aren't using "numeric" libraries or relying on native code libraries. What I most took exception to was either the lack of awareness of PyPy amongst people giving advice on how to speed up Python programs - not specifically "numeric" programs - to an audience of people who happened to be scientists, or the pretense that the first port of call was to break out Cython or NumPy when the easiest thing for them to do was to try their code on PyPy, provided they could get a package for it. One of my colleagues thought that "you could always rewrite your code in C", which is what he took away from such recommendations, was hilarious advice for speeding up Python programs. You write your Python code in another language! Why not just use C in the first place? Not such a stupid question, really. It's a question that has been hanging around for far too long without a really good answer. > It just doesn't integrate with the huge bulk of software that people use in > their daily work. And once you rely on that software, which is hand tuned, > well integrated and real-world proven in so many ways, over the time span > of way more than a decade, the most visible advantage of PyPy to make > Python code run faster becomes almost pointless. In that light, telling > people to try PyPy and to drop (most of) their current ecosystem for it > doesn't really sound helpful and clearly appears outside of the focus of > the web site in question. For a very long time, and even now to some extent, you could say the same thing about Python 3. Personally, I suspect that some people have such a problem with PyPy that they would rather not mention it or see it mentioned. That's obviously a shame because not only does PyPy offer people an alternative, but it also encourages the development of software in Python, some of which appears on the resource being discussed, that would otherwise be written in C because "that's what we would usually write this kind of software in". And although I don't actively use PyPy, mostly because of what the default Python implementation is on the platforms I use, I would like to see more actual Python code written. But I'm certainly not going to dwell on this matter any more than I've already done. I'm sure people will get something from the referenced resource regardless of whether any particular project is mentioned by it or not. Paul

8 years, 1 month

Summary of Python tracker Issues

by Python tracker

ACTIVITY SUMMARY (2013-02-08 - 2013-02-15) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 3845 ( -2) closed 25141 (+53) total 28986 (+51) Open issues with patches: 1664 Issues opened (38) ================== #6083: Reference counting bug in PyArg_ParseTuple and PyArg_ParseTupl http://bugs.python.org/issue6083 reopened by serhiy.storchaka #17162: Py_LIMITED_API needs a PyType_GenericDealloc http://bugs.python.org/issue17162 opened by bfroehle #17163: Fix test discovery for test_file.py http://bugs.python.org/issue17163 opened by zach.ware #17164: MozillaCookieJar does not handle session cookies http://bugs.python.org/issue17164 opened by piotr.dobrogost #17167: python man page contains $Date$ in page footer http://bugs.python.org/issue17167 opened by ned.deily #17170: string method lookup is too slow http://bugs.python.org/issue17170 opened by gvanrossum #17172: Add turtledemo to IDLE menu http://bugs.python.org/issue17172 opened by rhettinger #17174: Posix os.path.join should raise TypeError when passed unusable http://bugs.python.org/issue17174 opened by thomas.scrace #17175: update PEP 430 http://bugs.python.org/issue17175 opened by tshepang #17176: Document imp.NullImporter is NOT used anymore by import http://bugs.python.org/issue17176 opened by brett.cannon #17177: Document/deprecate imp http://bugs.python.org/issue17177 opened by brett.cannon #17178: In all.__doc__ and any.__doc__ need to clarify the behaviour w http://bugs.python.org/issue17178 opened by py.user #17179: Incorrect use of type function in types.new_class http://bugs.python.org/issue17179 opened by cjw296 #17180: shutil copy* unsafe on POSIX - they preserve setuid/setgit bit http://bugs.python.org/issue17180 opened by milko.krachounov #17181: SSLContext.set_servername_callback should be able to set argum http://bugs.python.org/issue17181 opened by grooverdan #17182: signal.default_int_handler should set signal number on the rai http://bugs.python.org/issue17182 opened by pitrou #17183: Small enhancements to Lib/_markupbase.py http://bugs.python.org/issue17183 opened by guido.reina #17184: re.VERBOSE doesn't respect whitespace in '( ?P<foo>...)' http://bugs.python.org/issue17184 opened by roysmith #17185: create_autospec http://bugs.python.org/issue17185 opened by cjw296 #17186: no way to introspect registered atexit handlers http://bugs.python.org/issue17186 opened by cjw296 #17187: Python segfaults from improperly formed and called function http://bugs.python.org/issue17187 opened by larry #17188: Document 'from None' in raise statement doc. http://bugs.python.org/issue17188 opened by terry.reedy #17189: Add zip64 support to shutil http://bugs.python.org/issue17189 opened by william.mallard #17191: pdb list shows unexpected code when stack frame includes a try http://bugs.python.org/issue17191 opened by AbramClark #17192: libffi-3.0.12 import http://bugs.python.org/issue17192 opened by doko #17193: Use binary prefixes http://bugs.python.org/issue17193 opened by serhiy.storchaka #17197: c/profile refactoring http://bugs.python.org/issue17197 opened by giampaolo.rodola #17198: dbm.whichdb references unitialized 'ndbm' variable http://bugs.python.org/issue17198 opened by pjenvey #17200: telnetlib.read_until() timeout uses the wrong units http://bugs.python.org/issue17200 opened by Reuben.D'Netto #17201: Use allowZip64=True by default http://bugs.python.org/issue17201 opened by serhiy.storchaka #17202: Add .bat line to .hgeol http://bugs.python.org/issue17202 opened by zach.ware #17203: add long option names to unittest discovery docs http://bugs.python.org/issue17203 opened by chris.jerdonek #17204: argparser's subparsers.add_parser() should accept an ArgumentP http://bugs.python.org/issue17204 opened by chris.jerdonek #17206: Py_XDECREF() expands its argument multiple times http://bugs.python.org/issue17206 opened by dmalcolm #17208: add note/warning about daemon threads http://bugs.python.org/issue17208 opened by pitrou #17209: get_wch() doesn't handle KeyboardInterrupt http://bugs.python.org/issue17209 opened by raymontag #17210: documentation of PyUnicode_Format() states wrong argument type http://bugs.python.org/issue17210 opened by scoder #17211: pkgutil.iter_modules and walk_packages should return a namedtu http://bugs.python.org/issue17211 opened by Ramchandra.Apte Most recent 15 issues with no replies (15) ========================================== #17211: pkgutil.iter_modules and walk_packages should return a namedtu http://bugs.python.org/issue17211 #17210: documentation of PyUnicode_Format() states wrong argument type http://bugs.python.org/issue17210 #17209: get_wch() doesn't handle KeyboardInterrupt http://bugs.python.org/issue17209 #17204: argparser's subparsers.add_parser() should accept an ArgumentP http://bugs.python.org/issue17204 #17203: add long option names to unittest discovery docs http://bugs.python.org/issue17203 #17202: Add .bat line to .hgeol http://bugs.python.org/issue17202 #17191: pdb list shows unexpected code when stack frame includes a try http://bugs.python.org/issue17191 #17188: Document 'from None' in raise statement doc. http://bugs.python.org/issue17188 #17186: no way to introspect registered atexit handlers http://bugs.python.org/issue17186 #17185: create_autospec http://bugs.python.org/issue17185 #17182: signal.default_int_handler should set signal number on the rai http://bugs.python.org/issue17182 #17178: In all.__doc__ and any.__doc__ need to clarify the behaviour w http://bugs.python.org/issue17178 #17177: Document/deprecate imp http://bugs.python.org/issue17177 #17176: Document imp.NullImporter is NOT used anymore by import http://bugs.python.org/issue17176 #17175: update PEP 430 http://bugs.python.org/issue17175 Most recent 15 issues waiting for review (15) ============================================= #17202: Add .bat line to .hgeol http://bugs.python.org/issue17202 #17201: Use allowZip64=True by default http://bugs.python.org/issue17201 #17200: telnetlib.read_until() timeout uses the wrong units http://bugs.python.org/issue17200 #17193: Use binary prefixes http://bugs.python.org/issue17193 #17189: Add zip64 support to shutil http://bugs.python.org/issue17189 #17187: Python segfaults from improperly formed and called function http://bugs.python.org/issue17187 #17180: shutil copy* unsafe on POSIX - they preserve setuid/setgit bit http://bugs.python.org/issue17180 #17175: update PEP 430 http://bugs.python.org/issue17175 #17172: Add turtledemo to IDLE menu http://bugs.python.org/issue17172 #17170: string method lookup is too slow http://bugs.python.org/issue17170 #17163: Fix test discovery for test_file.py http://bugs.python.org/issue17163 #17159: Remove explicit type check from inspect.Signature.from_functio http://bugs.python.org/issue17159 #17148: mingw: nt thread model detection http://bugs.python.org/issue17148 #17145: memoryview(array.array) http://bugs.python.org/issue17145 #17143: trace.py uses _warn without importing it http://bugs.python.org/issue17143 Top 10 most discussed issues (10) ================================= #16997: subtests http://bugs.python.org/issue16997 20 msgs #17170: string method lookup is too slow http://bugs.python.org/issue17170 18 msgs #16278: os.rename documentation slightly inaccurate http://bugs.python.org/issue16278 11 msgs #17130: Add runcall() function to profile.py and cProfile.py http://bugs.python.org/issue17130 9 msgs #17180: shutil copy* unsafe on POSIX - they preserve setuid/setgit bit http://bugs.python.org/issue17180 9 msgs #16612: Integrate "Argument Clinic" specialized preprocessor into CPyt http://bugs.python.org/issue16612 8 msgs #17159: Remove explicit type check from inspect.Signature.from_functio http://bugs.python.org/issue17159 8 msgs #17193: Use binary prefixes http://bugs.python.org/issue17193 8 msgs #17189: Add zip64 support to shutil http://bugs.python.org/issue17189 7 msgs #17200: telnetlib.read_until() timeout uses the wrong units http://bugs.python.org/issue17200 7 msgs Issues closed (52) ================== #4591: 32-bits unsigned user/group identifier http://bugs.python.org/issue4591 closed by serhiy.storchaka #5308: cannot marshal objects with more than 2**31 elements http://bugs.python.org/issue5308 closed by serhiy.storchaka #6426: imaplib.IMAP4 "command illegal in this state" is unhelpful err http://bugs.python.org/issue6426 closed by r.david.murray #6975: symlinks incorrectly resolved on POSIX platforms http://bugs.python.org/issue6975 closed by serhiy.storchaka #7145: UTF-16 BOM is not written by socket.makefile (but is expected http://bugs.python.org/issue7145 closed by serhiy.storchaka #7358: cStringIO not 64-bit safe http://bugs.python.org/issue7358 closed by serhiy.storchaka #7365: grp and pwd should treat uid and gid as unsigned http://bugs.python.org/issue7365 closed by serhiy.storchaka #9067: Use macros from pyctype.h http://bugs.python.org/issue9067 closed by skrah #9874: Message.attach() loses empty attachments http://bugs.python.org/issue9874 closed by r.david.murray #10355: SpooledTemporaryFile's name property is broken http://bugs.python.org/issue10355 closed by serhiy.storchaka #11311: StringIO.readline(0) returns incorrect results http://bugs.python.org/issue11311 closed by serhiy.storchaka #11397: os.path.realpath() may produce incorrect results http://bugs.python.org/issue11397 closed by serhiy.storchaka #11649: startElementNS in xml.sax.saxutils.XMLGenerator ignored encodi http://bugs.python.org/issue11649 closed by serhiy.storchaka #11837: smtplib._quote_periods triggers spurious type error in re.sub http://bugs.python.org/issue11837 closed by serhiy.storchaka #12983: byte string literals with invalid hex escape codes raise Value http://bugs.python.org/issue12983 closed by serhiy.storchaka #13555: cPickle MemoryError when loading large file (while pickle work http://bugs.python.org/issue13555 closed by serhiy.storchaka #13773: Support sqlite3 uri filenames http://bugs.python.org/issue13773 closed by pitrou #15220: Reduce parsing overhead in email.feedparser.BufferedSubFile http://bugs.python.org/issue15220 closed by r.david.murray #16564: email.generator.BytesGenerator fails with bytes payload http://bugs.python.org/issue16564 closed by r.david.murray #16686: audioop overflow issues http://bugs.python.org/issue16686 closed by serhiy.storchaka #16743: mmap on Windows can mishandle files larger than sys.maxsize http://bugs.python.org/issue16743 closed by sbt #16800: tempfile._get_default_tempdir() leaves files behind when HD is http://bugs.python.org/issue16800 closed by serhiy.storchaka #16967: Keyword only argument default values are evaluated before othe http://bugs.python.org/issue16967 closed by python-dev #16996: Reuse shutil.which() in webbrowser module http://bugs.python.org/issue16996 closed by serhiy.storchaka #17043: Invalid read in test_codecs http://bugs.python.org/issue17043 closed by serhiy.storchaka #17052: unittest discovery should use self.testLoader http://bugs.python.org/issue17052 closed by python-dev #17064: Fix sporadic buildbot failures for test_mailbox http://bugs.python.org/issue17064 closed by r.david.murray #17089: Expat parser parses strings only when XML encoding is UTF-8 http://bugs.python.org/issue17089 closed by serhiy.storchaka #17111: test_surrogates of test_fileio fails sometimes on OS X 10.4 http://bugs.python.org/issue17111 closed by ned.deily #17141: random.vonmisesvariate() hangs for large kappa http://bugs.python.org/issue17141 closed by serhiy.storchaka #17147: BytesIO should be mentioned in SpooledTemporaryFile documentat http://bugs.python.org/issue17147 closed by serhiy.storchaka #17149: random.vonmisesvariate() results range is inconsistent for sma http://bugs.python.org/issue17149 closed by mark.dickinson #17152: Array module should support "boolean" natively http://bugs.python.org/issue17152 closed by jcea #17155: logging can raise UnicodeEncodeError http://bugs.python.org/issue17155 closed by vinay.sajip #17156: Tools/i18n/pygettext.py doesn't parse unicode string. http://bugs.python.org/issue17156 closed by serhiy.storchaka #17158: help() module searcher text improvement http://bugs.python.org/issue17158 closed by python-dev #17160: test_urllib2net fails http://bugs.python.org/issue17160 closed by r.david.murray #17161: make install misses the man and the static library http://bugs.python.org/issue17161 closed by ned.deily #17165: Use "except ImportError" instead of bare except in _strptime.p http://bugs.python.org/issue17165 closed by pitrou #17166: Fix the suggested usage of _dummy_thread module http://bugs.python.org/issue17166 closed by r.david.murray #17168: test.support 3.x docs mentions stringio.stringio http://bugs.python.org/issue17168 closed by python-dev #17169: Restore errno in tempfile exceptions http://bugs.python.org/issue17169 closed by serhiy.storchaka #17171: email.encoders.encode7or8bit does not work with binary data http://bugs.python.org/issue17171 closed by r.david.murray #17173: uses of locale-dependent ctype functions http://bugs.python.org/issue17173 closed by pitrou #17190: _FAST opcodes do no range checking http://bugs.python.org/issue17190 closed by rhettinger #17194: operator.attrgetter is slower than a lambda http://bugs.python.org/issue17194 closed by flox #17195: Reading source code from file on exception http://bugs.python.org/issue17195 closed by flox #17196: crash http://bugs.python.org/issue17196 closed by michael.foord #17199: Slightly wrong behavior of logging.StrFormatStyle.usesTime http://bugs.python.org/issue17199 closed by vinay.sajip #17205: In the help() function the order of methods changes http://bugs.python.org/issue17205 closed by r.david.murray #17207: string format precision misbehaving http://bugs.python.org/issue17207 closed by r.david.murray #1470548: Bugfix for #1470540 (XMLGenerator cannot output UTF-16) http://bugs.python.org/issue1470548 closed by serhiy.storchaka

8 years, 1 month

PYTHONSTARTUP and -i

by Draic Kin

Hello. Python doc says that the PYTHONSTARTUP file is not read when a script is run with the -i option. Is there a reason for this? Is there a way to get the functionality i.e. to be able to run specified Python file just before the first prompt of interactive interpreter? Use case: I want to run my own repl on Windows, because there's a problem with unicode IO in the console. See http://bugs.python.org/issue1602 . Regards, Drekin

8 years, 1 month

A new webpage promoting Compiler technology for CPython

by Travis Oliphant

Hey all, With Numba and Blaze we have been doing a lot of work on what essentially is compiler technology and realizing more and more that we are treading on ground that has been plowed before with many other projects. So, we wanted to create a web-site and perhaps even a mailing list or forum where people could coordinate and communicate about compiler projects, compiler tools, and ways to share efforts and ideas. The website is: http://compilers.pydata.org/ This page is specifically for Compiler projects that either integrate with or work directly with the CPython run-time which is why PyPy is not presently listed. The PyPy project is a great project but we just felt that we wanted to explicitly create a collection of links to compilation projects that are accessible from CPython which are likely less well known. But that is just where we started from. The website is intended to be a community website constructed from a github repository. So, we welcome pull requests from anyone who would like to see the website updated to reflect their related project. Jon Riehl (Mython, PyFront, ROFL, and many other interesting projects) and Stephen Diehl (Blaze) and I will be moderating the pull requests to begin with. But, we welcome others with similar interests to participate in that effort of moderation. The github repository is here: https://github.com/pydata/compilers-webpage This is intended to be a community website for information spreading, and so we welcome any and all contributions. Thank you, Travis Oliphant

8 years, 1 month

build from source on mac uses clang, python.org binary uses gcc

by Chris Withers

Hi All, I've run into "some issues" installing lxml for python 3.3 on my mac: One of the stumbling blocks I've hit is that I built python 3.3 from source (./configure && make && make altinstall), and it used clang: buzzkill:virtualenvs chris$ /src/Python-3.3.0/python.exe Python 3.3.0 (default, Jan 23 2013, 09:56:03) [GCC 4.2.1 Compatible Apple Clang 3.0 (tags/Apple/clang-211.12)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> ...which makes lxml pretty unhappy. However, when I install the binary Python 3.3 from python.org, it's built with GCC: buzzkill:virtualenvs chris$ python3.3 Python 3.3.0 (v3.3.0:bd8afb90ebf2, Sep 29 2012, 01:25:11) [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> Why the difference? cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk

8 years, 1 month

Status of the PEP 433?

by Victor Stinner

Hi, I read again all emails related to PEP 433. There are different opposition to this change and I tried to make a summary. My question is: how can I move this PEP forward (accept, defer or reject it)? Can I accept it myself, or do we need a global agreement, or should someone else decide for this PEP (who? Antoine maybe)? -- Most complains concern issues if cloexec is set by default. I don't understand why so many people feel concerned because my PEP proposes *to not change anything to the default behaviour*: cloexec will not be set by default, as it was always the case since Python 1. You have to explicitly use -e command line option, PYTHONCLOEXEC=1 environment variable, or call sys.setdefaultcloexec(True). If you *explicitly* change the default value of the cloexec parameter, you must expect to change the behaviour of your application and maybe discover "bugs" in your application (application incompatible with cloexec set by default). (I don't think that any library would like to play with sys.setdefaultcloexec(), since most developers agree that it might break applications.) -- Ok, let's see all oppositions, presented as a FAQ. (I hope that I didn't misuse email quotes.) Q: the PEP does not solve file descriptor inherance after fork() A: Correct, it only solves issues with exec(), as explained in the PEP. Q: the PEP changes the POSIX behaviour. A: Antoine replied that "Python is not POSIX", Nick wrote "While many of Python's APIs are heavily inspired by POSIX, it still isn't POSIX" (he was talking about integer division). Perl sets cloexec flag by default since its first version, something like 25 years ago, except for explicit POSIX::* calls. Ruby 2 will also set cloexec flag by default. Q: The PEP may break applications. A: Most developers agree that it is (very) unlikely. If file descriptor inherance matters, subprocess must be used (because it rocks!) with pass_fds. If subprocess is used without pass_fds, applications stops working since python 3.2 (since python 3.0 on Windows). pass_fds will clear the close-on-exec flag on the file descriptors with the PEP. If an application breaks because it relies on file descriptor inherance, it's easy to detect it (EBADF error) and simple to fix it (ex: just call sys.setdefaultcloexec(False) at startup, or set explicitly cloexec parameter on the few functions causing trouble). Q: Performance overhead if cloexec is set by default (extra syscalls). A: The PEP now contains a benchmark: the overhead *on a microbenchmark* is between 1 and 3%. I bet that nobody will be able to notice a difference on a real application. Q: The default value of cloexec must not be configurable: "libraries cannot rely on it and have to make file descriptors cloexec on an individual basis, since the default flag can be disabled". A: IMO inherance of file descriptor doesn't matter in most cases, so only a few function must be modified to explicitly set or clear cloexec flag. Said differently: libraries should not care of the default value of the cloexec parameter, which should only be configured by applications (not libraries). Q: The default value of cloexec must not be configurable: "just by looking at this code, you have no way to know whether the file descriptor will be inherited by the child process." A: Guido wrote "Honestly, what happened to the idea that we're all adults here? The likelihood that someone is clueless enough to misuse the flag and yet clueful enough to find out how to change it seems remote -- and they pretty much deserve what they get." -- Another alternative (not listed in the PEP yet) is to leave the posix module (its name depends on the platform: posix, nt or ce) unchanged, and only modify the os module to add cloexec parameter. I implemented this alternative in the "posixmod" branch of the PEP 433 repository. It gives a better traceback on error because the cloexec flag is set in Python, so it's possible to know if the error comes from the creation of the file descriptor or setting the flag in the traceback. (Does it really matter? I never seen an error on setting or clearing the flag.) The problem is that it only concernes posix and os modules: the PEP does also modify socket, io and select modules, which don't have "low" and "high" level modules. So it's possible possible to create a socket using the POSIX behaviour. (Does it matter, it was said that Python is not POSIX.) I'm not convinced that we should do something special for posix/os, I prefer the actual version of the PEP (modify also posix). Victor

8 years, 1 month

← Newer
1
2
3
4
5
6
7
8
9
Older →

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

1999

Python-Dev February 2013