Martin v. Löwis wrote:
> - PyGetSetDef (name, get, set, doc, closure)
Is it fully decided that the generally-unused closure parameter will
stay until python 4?
> The accessor macros to these fields (Py_REFCNT, Py_TYPE, Py_SIZE)
> are also available to applications.
There have been several experiments in memory management, ranging from
not bothering to change the refcount on permanent objects like None,
to proxying objects across multiple threads or processes. I also
believe (but don't remember for sure) that some of the proposed
Unicode (or String?) optimizations changed the memory layout a bit.
So far, these have all been complicated (or slow) enough that they
didn't get integrated, but if it ever happens ... I don't think it
would justify python 4.0
> New Python
> versions may introduce new slot ids, but slot ids will never be
> recycled. Slots may get deprecated, but continue to be supported
> throughout Python 3.x.
Weren't there already a few ready for deprecation? Do you really want
to commit to them forever? Even if you aren't willing to settle for
less than "3.x from now on", it might make sense to at least start
with 3.2, rather than 3.0.
At 04:04 PM 5/19/2009 +0200, Tarek Ziadé wrote:
>On Sat, May 16, 2009 at 6:55 PM, P.J. Eby <pje(a)telecommunity.com> wrote:
> > 1. Why ';' separation, instead of tabs as in PEP 262? Aren't semicolons a
> > valid character in filenames?
>I am changing this into a <tab>. for now.
>What about Antoine's idea about doing a quote() on the names ?
I like the CSV idea better, since the csv module is available in 2.3
and up. We should just pick a dialect with unambiguous quoting rules.
> From my point of view <tabs> seems more simple to deal with, if 3rd-party
>tools want to work on these files without using pkgutil or Python.
True, but then CSV files are still pretty common.
One other possibility that might work is using a vertical bar as a separator.
My preference rank at the moment is probably tabs, CSV, or vertical
bar. But I don't really care all that much, so let the people who care decide.
Personally, though, I don't see much point to cross-language
manipulation of the file. System packaging tools have their own way
of keeping track of this stuff. So unless somebody's using it to
*build* system packages (e.g. making an RPM builder), they don't need this.
Now, about the APIs...
> > 4. There should probably be a way to iterate over the projects in a
> > directory, since it's otherwise impossible for an installation tool to find
> > out what project(s) "own" a file that conflicts with something being
> > installed. Alternatively, reshaping the file API to allow querying by path
> > as well as by project might work.
>I am adding a "get_projects" api:
> get_projects() -> iterator
> Provides an iterator that will return (name, path) tuples, where `name`
> is the name of a registered project and `path` the path to its `egg-info`
>But for the use case you are mentioning, what about an explicit API:
> get_owners(paths) -> sequence of project names
> returns a sequence of tuple. For each path in the "paths" list, a
>tuple of project names
> is returned
> > 5. If any cache mechanisms are to be used by the API, the API
> *must* make it
> > possible to bypass or explicitly manage that cache, as otherwise
> > installation tools and tools that manipulate sys.path at runtime may end up
> > using incorrect data.
>work in progress - (I am afraid I have to write an advanced prototype
>to be able to know
>exaclty how the cache might work, and so, what API we should have)
I think it would be simpler to have explicit object types
representing things like a directory, a collection of directories,
and individual projects, and these object types should be part of the API.
Any function-oriented API should just be exposed as the methods of a
default singleton. Other Python modules follow this pattern -- and
it's what I copied for the pkg_resources design. It gives a nice
tradeoff between keeping the simple things simple, and complex things
possible, as well as keeping mechanism and policy separate.
Right now, the API design you're trying to do is being burdened by
using strings and tuples to represent things that could just as
easily be objects with their own methods, instead of things you have
to pass back into other APIs. This also makes caching more complex,
because you can't just have one main object with stuff hanging off;
you've got to have a bunch of dictionaries, tuples, lists, sets, etc.
My perspective is as follows:
1) If PEP-384 had always been in place, my life would now be a lot easier.
2) Since it hasn't always been in place, its introduction won't help me
in the short term: there are an awful lot of extension modules that use
excluded functions (for example, all(?) PyCxx modules use PyCode_New and
PyFrame_New to get nicer tracebacks), and I'll still have to handle all
these cases until everyone is up-to-date with whatever version of Python
this gets accepted into.
3) Regardless, this PEP makes me very happy, because I can now look
forward to the glorious day when all extension modules are
384-compatible (and even *some* modules becoming compatible will make me
However, I'm not sure exactly how we can get there from here; I suspect
that certain features of certain extensions already depend critically
upon implementation details which will become hidden. The most extreme
illustrative example I know is from NumPy (in scalarmathmodule.c), and
looks like this:
PyInt_Type.tp_as_number = PyLongArrType_Type.tp_as_number;
PyInt_Type.tp_compare = PyLongArrType_Type.tp_compare;
PyInt_Type.tp_richcompare = PyLongArrType_Type.tp_richcompare;
...and I fear that many many similar (if perhaps less frightening)
dependencies exist elsewhere.
Regardless, in answer to the two specific questions you ask:
a) We don't really have that option. However, I would have a much higher
degree of confidence in running PEP-384-compatible modules under
Ironclad than I do with current modules, simply because I would no
longer need to worry about (say) edge cases in which extension writers
suddenly try to directly access op->ob_type->tp_as_number->nb_power.
b) I can't think of any more useful restrictions. The PEP would solve my
biggest current worry, which is that my current implementation allows
managed/unmanaged lists to fall out of sync in certain circumstances
(but if every list mutation happened via an API call, it wouldn't be an
Michael Foord wrote:
> The questions from Martin v. Lowis are in the email below.
> The PEP under discussion is:
> I can proxy any replies you want to send, or you can join Python-dev.
> All the best,
> -------- Original Message --------
> Subject: Re: [Python-Dev] PEP 384: Defining a Stable ABI
> Date: Mon, 18 May 2009 08:00:57 +0200
> From: "Martin v. Löwis" <martin(a)v.loewis.de>
> To: Michael Foord <fuzzyman(a)voidspace.org.uk>
> CC: Dino Viehland <dinov(a)microsoft.com>, Python-Dev
> <python-dev(a)python.org>, Unladen Swallow
> <unladen-swallow(a)googlegroups.com>, Python List <python-list(a)python.org>
> References: <4A107988.3020202(a)v.loewis.de>
> <4A1097F5.2050009(a)v.loewis.de> <4A10A368.1060406(a)voidspace.org.uk>
> >>> It also might make it easier for alternate implementations to support
> >>> the same API so some modules could work cross implementation - but I
> >>> suspect that's a non-goal of this PEP :).
> >> Indeed :-) I'm also skeptical that this would actually allow
> >> cross-implementation modules to happen. The list of functions that
> >> an alternate implementation would have to provide is fairly long.
> > Just in case you're unaware of it; the company I work for has an open
> > source project called Ironclad.
> I was unaware indeed; thanks for pointing this out.
> IIUC, it's not just an API emulation, but also an ABI emulation.
> > In particular we have had to address the issue of the GIL and extensions
> > (IronPython has no GIL) and reference counting (which IronPython also
> > doesn't) use.
> I think this somewhat strengthens the point I was trying to make: An
> alternate implementation that tries to be API compatible has to consider
> so many things that it is questionable whether making Py_INCREF/DECREF
> functions would be any simplification.
> So I just ask:
> a) Would it help IronClad if it could restrict itself to PEP 384
> compatible modules?
> b) Would further restrictions in the PEP help that cause?
since our python installation is located on a symlink'ed directory,
our variables "sys.exec_prefix" and "sys.executable" can have different
paths. Therefore, the respective test in build_ext.py fails (line 202)
and a wrong
library directory is obtained.
To fix this issue, I have attached a patch that uses "os.path.samefile"
to see whether two files are identical irrespective of its path.
ps: please CC answers to me, I'm not on the list :-)
pps: I hope the attachment isn't inline...
I've got a patch to add some documentation for lnotab and its use in
tracing at http://bugs.python.org/issue6042. I think it's correct, but
it's complicated so I'm looking for someone who was around when it was
designed to check. I'm also proposing a change to the semantics of
PyCode_CheckLineNumber and want to know whether I should consider it
Thanks to anyone who takes a look!
The inspect module (inspect.get_argspec etc) work fine for Python
functions and classes in IronPython, but they don't work on .NET types
which don't have the Python function attributes like im_func etc.
I have IronPython specific versions of several of these functions which
use .NET reflection and inspect could fallback to if sys.platform ==
'cli'. Would it be ok for me to add these to the inspect module?
Obviously the tests would only run on IronPython... The behaviour for
CPython would be unaffected.
All the best,
At 06:06 PM 5/16/2009 +0200, Tarek Ziadé wrote:
>Ok I've changed the PEP with all the points you mentioned, if you want
>to take a look.
1. Why ';' separation, instead of tabs as in PEP 262? Aren't
semicolons a valid character in filenames?
2. "if the installed file is located in a directory in site-packages"
should refer not to site-packages but to the directory containing the
3. get_egg_info_file needs to be specified as using '/'-separated
paths and converting to OS paths if appropriate. There's also the
problem that the mode it opens the file in (binary or text) is unspecified.
4. There should probably be a way to iterate over the projects in a
directory, since it's otherwise impossible for an installation tool
to find out what project(s) "own" a file that conflicts with
something being installed. Alternatively, reshaping the file API to
allow querying by path as well as by project might work.
5. If any cache mechanisms are to be used by the API, the API *must*
make it possible to bypass or explicitly manage that cache, as
otherwise installation tools and tools that manipulate sys.path at
runtime may end up using incorrect data.
6. get_files() doesn't document whether the yielded paths are
absolute or relative, local or cross-platform, etc.
>I need to find back your comments for this part, I must have missed
>the last part I didn't work out yet on the current PEP revision.
Well, if you can't find them, the EggFormats doc explains how these
file/dir structures are currently laid out by setuptools,
easy_install, pip, etc., and the PEP should probably reference that.
Technically, this PEP doesn't so much propose a change to the
EggFormats standard, as simply add a RECORD file to it, and propose
stdlib support for reading and writing it. So, the PEP really should
reference (i.e. link to) the existing standard. The EggFormats doc
in turn cites pkg_resources doc for lower-level format issues, such
as name and version normalization, filename escaping, file parsing, etc.
This PEP should also probably be framed as a replacement for PEP 262,
proposing to extend the de-facto standard for an installation
database with uninstall support, and blessing selected portions of
the de facto standard as an official standard. (Since that's pretty
much exactly what it is.)
I want to remove the usage of the "tar" command in Distutils in favor
or the "tarfile" module.
But, there's an option in Distutils.make_archive to create a tarball
using the "compress"  program rather than gzip or bzip2.
Using tar -Z, it will pipe it to the compress program if present. This
program implements the LZW algorithm .
The LZW used to be patented but this patent seem to be expired in
every country now .
On Distutils side I can work things out so the tar archive created can
be piped to an arbitraty compression program when it is
not compressed using bzip2 or gzip;
But I was wondering if we should we add a LZW support in tarinfo,
besides gzip and bzip2 ?
Although this compression standard doesn't seem very used these days,
Tarek Ziadé | http://ziade.org