This is PEP 427: Wheel. A binary package format for Python. Because newegg
was taken.
Since the last submission, the signature specification has been made
clearly optional / informative and does not attempt to specify any specific
signing algorithm or how the signatures would fit into a security scheme
that would necessarily exist outside of a single archive. JWS signatures
are used in their current form by OpenID Connect and Mozilla Personas and
are a useful way to implement basically raw public or secret key
signatures. The embedded signature scheme in wheel should also not affect
the current effort to define end-to-end security for PyPI in any way; it
might be a useful complement for packages that are not hosted on PyPI at
all.
This version also does not depend on the in-process Metadata PEP 426. It
has been cleaned up in several places regarding which PEPs it references to
describe its contents. The WHEEL metadata file should contain all the
information in the wheel filename itself, and the non-alphanumeric/unicode
filename escaping rules are made official.
Thanks.
For your consideration,
PEP: 427
Title: The Wheel Binary Package Format 0.1
Version: $Revision$
Last-Modified: $Date$
Author: Daniel Holth <dholth(a)fastmail.fm>
BDFL-Delegate: Nick Coghlan <ncoghlan(a)gmail.com>
Discussions-To: <distutils-sig(a)python.org>
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 20-Sep-2012
Post-History: 18-Oct-2012, 15-Feb-2013
Abstract
========
This PEP describes a built-package format for Python called "wheel".
A wheel is a ZIP-format archive with a specially formatted file name and
the ``.whl`` extension. It contains a single distribution nearly as it
would be installed according to PEP 376 with a particular installation
scheme. Although a specialized installer is recommended, a wheel file
may be installed by simply unpacking into site-packages with the standard
'unzip' tool while preserving enough information to spread its contents
out onto their final paths at any later time.
Note
====
This draft PEP describes version 0.1 of the "wheel" format. When the PEP
is accepted, the version will be changed to 1.0. (The major version
is used to indicate potentially backwards-incompatible changes to the
format.)
Rationale
=========
Python needs a package format that is easier to install than sdist.
Python's sdist packages are defined by and require the distutils and
setuptools build systems, running arbitrary code to build-and-install,
and re-compile, code just so it can be installed into a new
virtualenv. This system of conflating build-install is slow, hard to
maintain, and hinders innovation in both build systems and installers.
Wheel attempts to remedy these problems by providing a simpler
interface between the build system and the installer. The wheel
binary package format frees installers from having to know about the
build system, saves time by amortizing compile time over many
installations, and removes the need to install a build system in the
target environment.
Details
=======
Installing a wheel 'distribution-1.0.py32.none.any.whl'
-------------------------------------------------------
Wheel installation notionally consists of two phases:
- Unpack.
a. Parse ``distribution-1.0.dist-info/WHEEL``.
b. Check that installer is compatible with Wheel-Version. Warn if
minor version is greater, abort if major version is greater.
c. If Root-Is-Purelib == 'true', unpack archive into purelib
(site-packages).
d. Else unpack archive into platlib (site-packages).
- Spread.
a. Unpacked archive includes ``distribution-1.0.dist-info/`` and (if
there is data) ``distribution-1.0.data/``.
b. Move each subtree of ``distribution-1.0.data/`` onto its
destination path. Each subdirectory of ``distribution-1.0.data/``
is a key into a dict of destination directories, such as
``distribution-1.0.data/(purelib|platlib|headers|scripts|data)``.
The initially supported paths are taken from
``distutils.command.install``.
c. If applicable, update scripts starting with ``#!python`` to point
to the correct interpreter.
d. Update ``distribution-1.0.dist.info/RECORD`` with the installed
paths.
e. Remove empty ``distribution-1.0.data`` directory.
f. Compile any installed .py to .pyc. (Uninstallers should be smart
enough to remove .pyc even if it is not mentioned in RECORD.)
Recommended installer features
''''''''''''''''''''''''''''''
Rewrite ``#!python``.
In wheel, scripts are packaged in
``{distribution}-{version}.data/scripts/``. If the first line of
a file in ``scripts/`` starts with exactly ``b'#!python'``, rewrite to
point to the correct interpreter. Unix installers may need to add
the +x bit to these files if the archive was created on Windows.
Generate script wrappers.
In wheel, scripts packaged on Unix systems will certainly not have
accompanying .exe wrappers. Windows installers may want to add them
during install.
File Format
-----------
File name convention
''''''''''''''''''''
The wheel filename is ``{distribution}-{version}(-{build
tag})?-{python tag}-{abi tag}-{platform tag}.whl``.
distribution
Distribution name, e.g. 'django', 'pyramid'.
version
Distribution version, e.g. 1.0.
build tag
Optional build number. Must start with a digit. A tie breaker
if two wheels have the same version. Sort as the empty string
if unspecified, else sort the initial digits as a number, and the
remainder lexicographically.
language implementation and version tag
E.g. 'py27', 'py2', 'py3'.
abi tag
E.g. 'cp33m', 'abi3', 'none'.
platform tag
E.g. 'linux_x86_64', 'any'.
For example, ``distribution-1.0-1-py27-none-any.whl`` is the first
build of a package called 'distribution', and is compatible with
Python 2.7 (any Python 2.7 implementation), with no ABI (pure Python),
on any CPU architecture.
The last three components of the filename before the extension are
called "compatibility tags." The compatibility tags express the
package's basic interpreter requirements and are detailed in PEP 425.
Escaping and Unicode
''''''''''''''''''''
Each component of the filename is escaped by replacing runs of
non-alphanumeric characters with an underscore ``_``::
re.sub("[^\w\d.]+", "_", distribution, re.UNICODE)
The filename is Unicode. It will be some time before the tools are
updated to support non-ASCII filenames, but they are supported in this
specification.
File contents
'''''''''''''
The contents of a wheel file, where {distribution} is replaced with the
name of the package, e.g. ``beaglevote`` and {version} is replaced with
its version, e.g. ``1.0.0``, consist of:
#. ``/``, the root of the archive, contains all files to be installed in
``purelib`` or ``platlib`` as specified in ``WHEEL``. ``purelib`` and
``platlib`` are usually both ``site-packages``.
#. ``{distribution}-{version}.dist-info/`` contains metadata.
#. ``{distribution}-{version}.data/`` contains one subdirectory
for each non-empty install scheme key not already covered, where
the subdirectory name is an index into a dictionary of install paths
(e.g. ``data``, ``scripts``, ``include``, ``purelib``, ``platlib``).
#. Python scripts must appear in ``scripts`` and begin with exactly
``b'#!python'`` in order to enjoy script wrapper generation and
``#!python`` rewriting at install time. They may have any or no
extension.
#. ``{distribution}-{version}.dist-info/METADATA`` is Metadata version 1.1
or greater (PEP 314, PEP 345, PEP 426) format metadata.
#. ``{distribution}-{version}.dist-info/WHEEL`` is metadata about the
archive
itself in the same basic key: value format::
Wheel-Version: 0.1
Generator: bdist_wheel 0.7
Root-Is-Purelib: true
Tag: py2-none-any
Tag: py3-none-any
Build: 1
#. ``Wheel-Version`` is the version number of the Wheel specification.
``Generator`` is the name and optionally the version of the software
that produced the archive. ``Root-Is-Purelib`` is true if the top
level directory of the archive should be installed into purelib;
otherwise the root should be installed into platlib. ``Tag`` is the
wheel's expanded compatibility tags; in the example the filename would
contain ``py2.py3-none-any``. ``Build`` is the build number and is
omitted if there is no build number.
#. A wheel installer should warn if Wheel-Version is greater than the
version it supports, and must fail if Wheel-Version has a greater
major version than the version it supports.
#. Wheel, being an installation format that is intended to work across
multiple versions of Python, does not generally include .pyc files.
#. Wheel does not contain setup.py or setup.cfg.
This version of the wheel specification is based on the distutils install
schemes and does not define how to install files to other locations.
The layout offers a superset of the functionality provided by the existing
wininst and egg binary formats.
The .dist-info directory
^^^^^^^^^^^^^^^^^^^^^^^^
#. Wheel .dist-info directories include at a minimum METADATA, WHEEL,
and RECORD.
#. METADATA is the package metadata, the same format as PKG-INFO as
found at the root of sdists.
#. WHEEL is the wheel metadata specific to a build of the package.
#. RECORD is a list of (almost) all the files in the wheel and their
secure hashes. Unlike PEP 376, every file except RECORD, which
cannot contain a hash of itself, must include its hash. The hash
algorithm must be sha256 or better; specifically, md5 and sha1 are
not permitted, as signed wheel files rely on the strong hashes in
RECORD to validate the integrity of the archive.
#. PEP 376's INSTALLER and REQUESTED are not included in the archive.
#. RECORD.jws is used for digital signatures. It is not mentioned in
RECORD.
#. RECORD.p7s is allowed as a courtesy to anyone who would prefer to
use S/MIME signatures to secure their wheel files. It is not
mentioned in RECORD.
#. During extraction, wheel installers verify all the hashes in RECORD
against the file contents. Apart from RECORD and its signatures,
installation will fail if any file in the archive is not both
mentioned and correctly hashed in RECORD.
The .data directory
^^^^^^^^^^^^^^^^^^^
Any file that is not normally installed inside site-packages goes into
the .data directory, named as the .dist-info directory but with the
.data/ extension::
distribution-1.0.dist-info/
distribution-1.0.data/
The .data directory contains subdirectories with the scripts, headers,
documentation and so forth from the distribution. During installation the
contents of these subdirectories are moved onto their destination paths.
Signed wheel files
------------------
Wheel files include an extended RECORD that enables digital
signatures. PEP 376's RECORD is altered to include a secure hash
``digestname=urlsafe_b64encode_nopad(digest)`` (urlsafe base64
encoding with no trailing = characters) as the second column instead
of an md5sum. All possible entries are hashed, including any
generated files such as .pyc files, but not RECORD which cannot contain its
own hash. For example::
file.py,sha256=AVTFPZpEKzuHr7OvQZmhaU3LvwKz06AJw8mT\_pNh2yI,3144
distribution-1.0.dist-info/RECORD,,
The signature file(s) RECORD.jws and RECORD.p7s are not mentioned in
RECORD at all since they can only be added after RECORD is generated.
Every other file in the archive must have a correct hash in RECORD
or the installation will fail.
If JSON web signatures are used, one or more JSON Web Signature JSON
Serialization (JWS-JS) signatures is stored in a file RECORD.jws adjacent
to RECORD. JWS is used to sign RECORD by including the SHA-256 hash of
RECORD as the signature's JSON payload::
{ "hash": "sha256=ADD-r2urObZHcxBW3Cr-vDCu5RJwT4CaRTHiFmbcIYY" }
If RECORD.p7s is used, it must contain a detached S/MIME format signature
of RECORD.
A wheel installer is not required to understand digital signatures but
MUST verify the hashes in RECORD against the extracted file contents.
When the installer checks file hashes against RECORD, a separate signature
checker only needs to establish that RECORD matches the signature.
See
- http://self-issued.info/docs/draft-ietf-jose-json-web-signature.html
- http://self-issued.info/docs/draft-jones-jose-jws-json-serialization.html
- http://self-issued.info/docs/draft-ietf-jose-json-web-key.html
- http://self-issued.info/docs/draft-jones-jose-json-private-key.html
Comparison to .egg
------------------
#. Wheel is an installation format; egg is importable. Wheel archives
do not need to include .pyc and are less tied to a specific Python
version or implementation. Wheel can install (pure Python) packages
built with previous versions of Python so you don't always have to
wait for the packager to catch up.
#. Wheel uses .dist-info directories; egg uses .egg-info. Wheel is
compatible with the new world of Python packaging and the new
concepts it brings.
#. Wheel has a richer file naming convention for today's
multi-implementation world. A single wheel archive can indicate
its compatibility with a number of Python language versions and
implementations, ABIs, and system architectures. Historically the
ABI has been specific to a CPython release, wheel is ready for the
stable ABI.
#. Wheel is lossless. The first wheel implementation bdist_wheel
always generates egg-info, and then converts it to a .whl. It is
also possible to convert existing eggs and bdist_wininst
distributions.
#. Wheel is versioned. Every wheel file contains the version of the
wheel specification and the implementation that packaged it.
Hopefully the next migration can simply be to Wheel 2.0.
#. Wheel is a reference to the other Python.
FAQ
===
Wheel defines a .data directory. Should I put all my data there?
This specification does not have an opinion on how you should organize
your code. The .data directory is just a place for any files that are
not normally installed inside ``site-packages`` or on the PYTHONPATH.
In other words, you may continue to use ``pkgutil.get_data(package,
resource)`` even though *those* files will usually not be distributed
in *wheel's* ``.data`` directory.
Why does wheel include attached signatures?
Attached signatures are more convenient than detached signatures
because they travel with the archive. Since only the individual files
are signed, the archive can be recompressed without invalidating
the signature or individual files can be verified without having
to download the whole archive.
Why does wheel allow JWS signatures?
The JOSE specifications of which JWS is a part are designed to be easy
to implement, a feature that is also one of wheel's primary design
goals. JWS yields a useful, concise pure-Python implementation.
Why does wheel also allow S/MIME signatures?
S/MIME signatures are allowed for users who need or want to use
existing public key infrastructure with wheel.
Signed packages are only a basic building block in a secure package
update system. Wheel only provides the building block.
Appendix
========
Example urlsafe-base64-nopad implementation::
# urlsafe-base64-nopad for Python 3
import base64
def urlsafe_b64encode_nopad(data):
return base64.urlsafe_b64encode(data).rstrip(b'=')
def urlsafe_b64decode_nopad(data):
pad = b'=' * (4 - (len(data) & 3))
return base64.urlsafe_b64decode(data + pad)
Copyright
=========
This document has been placed into the public domain.
..
Local Variables:
mode: indented-text
indent-tabs-mode: nil
sentence-end-double-space: t
fill-column: 70
coding: utf-8
End:
Hello,
I guess a long time ago, threading support in operating systems wasn't
very widespread, but these days all our supported platforms have it.
Is it still useful for production purposes to configure
--without-threads? Do people use this option for something else than
curiosity of mind?
Regards
Antoine.
Say there, the Python core development community! Have I got
a question for you!
*ahem*
Which of the following four options do you dislike least? ;-)
1) CPython continues to provide no "function signature"
objects (PEP 362) or inspect.getfullargspec() information
for any function implemented in C.
2) We add new hand-coded data structures representing the
metadata necessary for function signatures for builtins.
Which means that, when defining arguments to functions in C,
we'd need to repeat ourselves *even more* than we already do.
3) Builtin function arguments are defined using some seriously
uncomfortable and impenetrable C preprocessor macros, which
produce all the various types of output we need (argument
processing code, function signature metadata, possibly
the docstrings too).
4) Builtin function arguments are defined in a small DSL; these
are expanded to code and data using a custom compile-time
preprocessor step.
All the core devs I've asked said "given all that, I'd prefer the
hairy preprocessor macros". But by the end of the conversation
they'd changed their minds to prefer the custom DSL. Maybe I'll
make a believer out of you too--read on!
I've named this DSL preprocessor "Argument Clinic", or Clinic
for short**. Clinic works similarly to Ned Batchelder's brilliant
"Cog" tool:
http://nedbatchelder.com/code/cog/
You embed the input to Clinic in a comment in your C file,
and the output is written out immediately after that comment.
The output's overwritten every time the preprocessor is run.
In short it looks something like this:
/*[clinic]
input to the DSL
[clinic]*/
... output from the DSL, overwritten every time ...
/*[clinic end:<checksum>]*/
The input to the DSL includes all the metadata about the
function that we need for the function signature:
* the name of the function,
* the return annotation (if any),
* each parameter to the function, including
* its name,
* its type (in C),
* its default value,
* and a per-parameter docstring;
* and the docstring for the function as a whole.
The resulting output contains:
* the docstring for the function,
* declarations for all your parameters,
* C code handling all argument processing for you,
* and a #define'd methoddef structure for adding the
function to the module.
I discussed this with Mark "HotPy" Shannon, and he suggested we break
our existing C functions into two. We put the argument processing
into its own function, generated entirely by Clinic, and have the
implementation in a second function called from the first. I like
this approach simply because it makes the code cleaner. (Note that
this approach should not cause any overhead with a modern compiler,
as both functions will be "static".)
But it also provides an optimization opportunity for HotPy: it could
read the metadata, and when generating the JIT'd code it could skip
building the PyObjects and argument tuple (and possibly keyword
argument dict), and the subsequent unpacking/decoding, and just call
the implementation function directly, giving it a likely-measurable
speed boost.
And we can go further! If we add a new extension type API allowing
you to register both functions, and external modules start using it,
sophisticated Python implementations like PyPy might be able to skip
building the tuple for extension type function calls--speeding those
up too!
Another plausible benefit: alternate implementations of Python could
read the metadata--or parse the input to Clinic themselves--to ensure
their reimplementations of the Python standard library conform to the
same API!
Clinic can also run general-purpose Python code ("/*[python]").
All output from "print" is redirected into the output section
after the Python code.
As you've no doubt already guessed, I've made a prototype of
Argument Clinic. You can see it--and some sample conversions of
builtins using it for argument processing--at this BitBucket repo:
https://bitbucket.org/larry/python-clinic
I don't claim that it's fabulous, production-ready code. But it's
a definite start!
To save you a little time, here's a preview of using Clinic for
dbm.open(). The stuff at the same indent as a declaration are
options; see the "clinic.txt" in the repo above for full documentation.
/*[clinic]
dbm.open -> mapping
basename=dbmopen
const char *filename;
The filename to open.
const char *flags="r";
How to open the file. "r" for reading, "w" for writing, etc.
int mode=0666;
default=0o666
If creating a new file, the mode bits for the new file
(e.g. os.O_RDWR).
Returns a database object.
[clinic]*/
PyDoc_STRVAR(dbmopen__doc__,
"dbm.open(filename[, flags=\'r\'[, mode=0o666]]) -> mapping\n"
"\n"
" filename\n"
" The filename to open.\n"
"\n"
" flags\n"
" How to open the file. \"r\" for reading, \"w\" for writing,
etc.\n"
"\n"
" mode\n"
" If creating a new file, the mode bits for the new file\n"
" (e.g. os.O_RDWR).\n"
"\n"
"Returns a database object.\n"
"\n");
#define DBMOPEN_METHODDEF \
{"open", (PyCFunction)dbmopen, METH_VARARGS | METH_KEYWORDS,
dbmopen__doc__}
static PyObject *
dbmopen_impl(PyObject *self, const char *filename, const char *flags,
int mode);
static PyObject *
dbmopen(PyObject *self, PyObject *args, PyObject *kwargs)
{
const char *filename;
const char *flags = "r";
int mode = 0666;
static char *_keywords[] = {"filename", "flags", "mode", NULL};
if (!PyArg_ParseTupleAndKeywords(args, kwargs,
"s|si", _keywords,
&filename, &flags, &mode))
return NULL;
return dbmopen_impl(self, filename, flags, mode);
}
static PyObject *
dbmopen_impl(PyObject *self, const char *filename, const char *flags,
int mode)
/*[clinic end:eddc886e542945d959b44b483258bf038acf8872]*/
As of this writing, I also have sample conversions in the following files
available for your perusal:
Modules/_cursesmodule.c
Modules/_dbmmodule.c
Modules/posixmodule.c
Modules/zlibmodule.c
Just search in C files for '[clinic]' and you'll find everything soon
enough.
As you can see, Clinic has already survived some contact with the
enemy. I've already converted some tricky functions--for example,
os.stat() and curses.window.addch(). The latter required adding a
new positional-only processing mode for functions using a legacy
argument processing approach. (See "clinic.txt" for more.) If you
can suggest additional tricky functions to support, please do!
Big unresolved questions:
* How would we convert all the builtins to use Clinic? I fear any
solution will involve some work by hand. Even if we can automate
big chunks of it, fully automating it would require parsing arbitrary
C. This seems like overkill for a one-shot conversion.
(Mark Shannon says he has some ideas.)
* How do we create the Signature objects? My current favorite idea:
Clinic also generates a new, TBD C structure defining all the
information necessary for the signature, which is also passed in to
the new registration API (you remember, the one that takes both the
argument-processing function and the implementation function). This
is secreted away in some new part of the C function object. At
runtime this is converted on-demand into a Signature object. Default
values for arguments are represented in C as strings; the conversion
process attempts eval() on the string, and if that works it uses the
result, otherwise it simply passes through the string.
* Right now Clinic paves over the PyArg_ParseTuple API for you.
If we convert CPython to use Clinic everywhere, theoretically we
could replace the parsing API with something cleaner and/or faster.
Does anyone have good ideas (and time, and energy) here?
* There's actually a fifth option, proposed by Brett Cannon. We
constrain the format of docstrings for builtin functions to make
them machine-readable, then generate the function signature objects
from that. But consider: generating *everything* in the signature
object may get a bit tricky (e.g. Parameter.POSITIONAL_ONLY), and
this might gunk up the docstring.
But the biggest unresolved question... is this all actually a terrible
idea?
//arry/
** "Is this the right room for an argument?"
"I've told you once...!"
Travis Oliphant wrote:
>
> With Numba and Blaze we have been doing a lot of work on what essentially
> is compiler technology and realizing more and more that we are treading on
> ground that has been plowed before with many other projects. So, we
> wanted to create a web-site and perhaps even a mailing list or forum where
> people could coordinate and communicate about compiler projects, compiler
> tools, and ways to share efforts and ideas.
>
> The website is: http://compilers.pydata.org/
This is a rather nice resource. Thank you for letting us know about it!
There has been an attempt to record different Python implementations on the
Python Wiki, and now that this is available again, I'd like to remind people
about it:
http://wiki.python.org/moin/PythonImplementations
I know that this isn't quite the same thing as a page about compiler
technology, but there is a substantial overlap.
> This page is specifically for Compiler projects that either integrate with
> or work directly with the CPython run-time which is why PyPy is not
> presently listed. The PyPy project is a great project but we just felt
> that we wanted to explicitly create a collection of links to compilation
> projects that are accessible from CPython which are likely less well known.
I think that given the scope of the projects listed, it shouldn't preclude
PyPy from being listed, really. After all, interoperability with CPython
extensions is something of a focus area for PyPy:
http://pypy.org/compat.html
I don't have an agenda here - I don't use PyPy actively, my only involvement
with Shedskin (which is referenced and which can produce CPython extension
modules) is in packaging it for Debian, and although I do have a static
analysis library I see no pressing need to promote it extensively - but I
feel that when it comes to educational resources people should be a bit
careful about drawing boundaries that exclude things that would benefit
people substantially if they only knew about it.
My reason for speaking up about this is that I've had to tell a room full of
people who were told to use Cython, NumPy and even plain C to make their
Python programs faster that PyPy existed. (Of course, one can justify
ignoring the elephant in the room by claiming things like scientific users
rely on native libraries or CPython extensions - since "science" is a very
broad term, this obviously isn't universally true - but I think people should
be entitled to make their own minds up, and I was not completely certain that
everyone involved in the case in question was oblivious to PyPy's existence
or status.)
> But that is just where we started from. The website is intended to be a
> community website constructed from a github repository. So, we welcome
> pull requests from anyone who would like to see the website updated to
> reflect their related project. Jon Riehl (Mython, PyFront, ROFL, and
> many other interesting projects) and Stephen Diehl (Blaze) and I will be
> moderating the pull requests to begin with. But, we welcome others with
> similar interests to participate in that effort of moderation.
>
> The github repository is here: https://github.com/pydata/compilers-webpage
>
> This is intended to be a community website for information spreading, and
> so we welcome any and all contributions.
There is also the python-static-type-checking Google Group:
https://groups.google.com/forum/?fromgroups#!forum/python-static-type-check…
If no-one beats me to it, I may post details of the site to that group because
it may well be of interest to the members. Thanks once again for bringing
such information together in one place!
Paul
Stefan Behnel wrote:
>
> This is off-topic for this list, but the main problem with PyPy is that
> you'll quickly hit a lot of walls when you try to use it for anything
> serious in the area. It's true that there is a certain level of
> interoperability with CPython extensions, but calling it a "focus area"
> makes it sound bigger than it actually is in my ears.
It is a focus area. They even have a Web page for it, which I mentioned.
> Even trying to get bugs fixed to at least make things work at all often
> means running against walls on their side. I can tell, trying to port
> Cython mostly consisted of bugging PyPy developers to fix stuff, which took
> anything from days to still-not-done, each time. And, by design, PyPy makes
> it very hard and time consuming to debug it and to try to fix bugs in their
> code base.
I'm sure it is tricky to debug PyPy. However, complete compatibility with
CPython for Python programs is a goal of the project, and I have no reason to
doubt that the project is committed to supporting broader compatibility with
CPython.
> So, while I agree that PyPy is worth a try in certain application areas,
> and can be helpful for some special needs, also in the field of scientific
> computing, it's lightyears away from a production-ready state in that area.
You are generalising too broadly, which is precisely what I mentioned in my
last message. There are also plenty of people doing science who aren't
using "numeric" libraries or relying on native code libraries. What I most
took exception to was either the lack of awareness of PyPy amongst people
giving advice on how to speed up Python programs - not specifically "numeric"
programs - to an audience of people who happened to be scientists, or the
pretense that the first port of call was to break out Cython or NumPy when
the easiest thing for them to do was to try their code on PyPy, provided they
could get a package for it.
One of my colleagues thought that "you could always rewrite your code in C",
which is what he took away from such recommendations, was hilarious advice
for speeding up Python programs. You write your Python code in another
language! Why not just use C in the first place? Not such a stupid question,
really. It's a question that has been hanging around for far too long without
a really good answer.
> It just doesn't integrate with the huge bulk of software that people use in
> their daily work. And once you rely on that software, which is hand tuned,
> well integrated and real-world proven in so many ways, over the time span
> of way more than a decade, the most visible advantage of PyPy to make
> Python code run faster becomes almost pointless. In that light, telling
> people to try PyPy and to drop (most of) their current ecosystem for it
> doesn't really sound helpful and clearly appears outside of the focus of
> the web site in question.
For a very long time, and even now to some extent, you could say the same
thing about Python 3. Personally, I suspect that some people have such a
problem with PyPy that they would rather not mention it or see it mentioned.
That's obviously a shame because not only does PyPy offer people an
alternative, but it also encourages the development of software in Python,
some of which appears on the resource being discussed, that would otherwise
be written in C because "that's what we would usually write this kind of
software in". And although I don't actively use PyPy, mostly because of what
the default Python implementation is on the platforms I use, I would like to
see more actual Python code written.
But I'm certainly not going to dwell on this matter any more than I've already
done. I'm sure people will get something from the referenced resource
regardless of whether any particular project is mentioned by it or not.
Paul
Hello.
Python doc says that the PYTHONSTARTUP file is not read when a script is
run with the -i option. Is there a reason for this? Is there a way to get
the functionality i.e. to be able to run specified Python file just before
the first prompt of interactive interpreter?
Use case: I want to run my own repl on Windows, because there's a problem
with unicode IO in the console. See http://bugs.python.org/issue1602 .
Regards, Drekin
Hey all,
With Numba and Blaze we have been doing a lot of work on what essentially is compiler technology and realizing more and more that we are treading on ground that has been plowed before with many other projects. So, we wanted to create a web-site and perhaps even a mailing list or forum where people could coordinate and communicate about compiler projects, compiler tools, and ways to share efforts and ideas.
The website is: http://compilers.pydata.org/
This page is specifically for Compiler projects that either integrate with or work directly with the CPython run-time which is why PyPy is not presently listed. The PyPy project is a great project but we just felt that we wanted to explicitly create a collection of links to compilation projects that are accessible from CPython which are likely less well known.
But that is just where we started from. The website is intended to be a community website constructed from a github repository. So, we welcome pull requests from anyone who would like to see the website updated to reflect their related project. Jon Riehl (Mython, PyFront, ROFL, and many other interesting projects) and Stephen Diehl (Blaze) and I will be moderating the pull requests to begin with. But, we welcome others with similar interests to participate in that effort of moderation.
The github repository is here: https://github.com/pydata/compilers-webpage
This is intended to be a community website for information spreading, and so we welcome any and all contributions.
Thank you,
Travis Oliphant
Hi All,
I've run into "some issues" installing lxml for python 3.3 on my mac:
One of the stumbling blocks I've hit is that I built python 3.3 from
source (./configure && make && make altinstall), and it used clang:
buzzkill:virtualenvs chris$ /src/Python-3.3.0/python.exe
Python 3.3.0 (default, Jan 23 2013, 09:56:03)
[GCC 4.2.1 Compatible Apple Clang 3.0 (tags/Apple/clang-211.12)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>>
...which makes lxml pretty unhappy.
However, when I install the binary Python 3.3 from python.org, it's
built with GCC:
buzzkill:virtualenvs chris$ python3.3
Python 3.3.0 (v3.3.0:bd8afb90ebf2, Sep 29 2012, 01:25:11)
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>>
Why the difference?
cheers,
Chris
--
Simplistix - Content Management, Batch Processing & Python Consulting
- http://www.simplistix.co.uk
Hi,
I read again all emails related to PEP 433. There are different
opposition to this change and I tried to make a summary.
My question is: how can I move this PEP forward (accept, defer or
reject it)? Can I accept it myself, or do we need a global agreement,
or should someone else decide for this PEP (who? Antoine maybe)?
--
Most complains concern issues if cloexec is set by default. I don't
understand why so many people feel concerned because my PEP proposes
*to not change anything to the default behaviour*: cloexec will not be
set by default, as it was always the case since Python 1.
You have to explicitly use -e command line option, PYTHONCLOEXEC=1
environment variable, or call sys.setdefaultcloexec(True). If you
*explicitly* change the default value of the cloexec parameter, you
must expect to change the behaviour of your application and maybe
discover "bugs" in your application (application incompatible with
cloexec set by default).
(I don't think that any library would like to play with
sys.setdefaultcloexec(), since most developers agree that it might
break applications.)
--
Ok, let's see all oppositions, presented as a FAQ.
(I hope that I didn't misuse email quotes.)
Q: the PEP does not solve file descriptor inherance after fork()
A: Correct, it only solves issues with exec(), as explained in the PEP.
Q: the PEP changes the POSIX behaviour.
A: Antoine replied that "Python is not POSIX", Nick wrote "While many
of Python's APIs are heavily inspired by POSIX, it still isn't POSIX"
(he was talking about integer division). Perl sets cloexec flag by
default since its first version, something like 25 years ago, except
for explicit POSIX::* calls. Ruby 2 will also set cloexec flag by
default.
Q: The PEP may break applications.
A: Most developers agree that it is (very) unlikely. If file
descriptor inherance matters, subprocess must be used (because it
rocks!) with pass_fds. If subprocess is used without pass_fds,
applications stops working since python 3.2 (since python 3.0 on
Windows). pass_fds will clear the close-on-exec flag on the file
descriptors with the PEP. If an application breaks because it relies
on file descriptor inherance, it's easy to detect it (EBADF error) and
simple to fix it (ex: just call sys.setdefaultcloexec(False) at
startup, or set explicitly cloexec parameter on the few functions
causing trouble).
Q: Performance overhead if cloexec is set by default (extra syscalls).
A: The PEP now contains a benchmark: the overhead *on a
microbenchmark* is between 1 and 3%. I bet that nobody will be able to
notice a difference on a real application.
Q: The default value of cloexec must not be configurable: "libraries
cannot rely on it and have to make file descriptors cloexec on an
individual basis, since the default flag can be disabled".
A: IMO inherance of file descriptor doesn't matter in most cases, so
only a few function must be modified to explicitly set or clear
cloexec flag. Said differently: libraries should not care of the
default value of the cloexec parameter, which should only be
configured by applications (not libraries).
Q: The default value of cloexec must not be configurable: "just by
looking at this code, you have no way to know whether the file
descriptor will be inherited by the child process."
A: Guido wrote "Honestly, what happened to the idea that we're all
adults here? The likelihood that someone is clueless enough to misuse
the flag and yet clueful enough to find out how to change it seems
remote -- and they pretty much deserve what they get."
--
Another alternative (not listed in the PEP yet) is to leave the posix
module (its name depends on the platform: posix, nt or ce) unchanged,
and only modify the os module to add cloexec parameter.
I implemented this alternative in the "posixmod" branch of the PEP 433
repository. It gives a better traceback on error because the cloexec
flag is set in Python, so it's possible to know if the error comes
from the creation of the file descriptor or setting the flag in the
traceback. (Does it really matter? I never seen an error on setting or
clearing the flag.)
The problem is that it only concernes posix and os modules: the PEP
does also modify socket, io and select modules, which don't have "low"
and "high" level modules. So it's possible possible to create a socket
using the POSIX behaviour. (Does it matter, it was said that Python is
not POSIX.)
I'm not convinced that we should do something special for posix/os, I
prefer the actual version of the PEP (modify also posix).
Victor