Mailman 3 April 2009 - Python-Dev

Re: [Python-Dev] PEP 382: Namespace Packages
by P.J. Eby April 17, 2009

April 17, 2009

At 03:46 AM 4/16/2009 +0000, glyph(a)divmod.com wrote: >On 15 Apr, 09:11 pm, pje(a)telecommunity.com wrote: >>I think that there is some confusion here. A "main" package or >>buildout that assembles a larger project from components is not the >>same thing as having a "base" package for a namespace package. > >I'm certainly confused. > >Twisted has its own system for "namespace" packages, and I'm not >really sure where we fall in this discussion. I haven'… [View More]t been able >to follow the whole thread, but my original understanding was that >the PEP supports "defining packages", which we now seem to be >calling "base packages", just fine. Yes, it does. The discussion since the original proposal, however, has been dominated by MAL's counterproposal, which *requires* a defining package. There is a slight distinction between "base package" and "defining package", although I suppose I've been using them a bit interchangeably. Base package describes a use case: you have a base package which is extended in the same namespace. In that use case, you may want to place your base package in the defining package. In contrast, setuptools does not support a defining package, so if you have a base package, you must place it in a submodule or subpackage of the namespace. Does that all make sense now? MAL's proposal requires a defining package, which is counterproductive if you have a pure package with no base, since it now requires you to create an additional project on PyPI just to hold your defining package. >I'd appreciate it if the PEP could also be extended cover Twisted's >very similar mechanism for namespace packages, >"twisted.plugin.pluginPackagePaths". I know this is not quite as >widely used as setuptools' namespace package support, but its >existence belies a need for standardization. > >The PEP also seems a bit vague with regard to the treatment of other >directories containing __init__.py and *.pkg files. Do you have a clarification to suggest? My understanding (probably a projection) is that to be a nested namespace package, you have to have a parent namespace package. > The concept of a "defining package" seems important to avoid > conflicts like this one: > > http://twistedmatrix.com/trac/ticket/2339 > >More specifically I don't quite understand the PEP's intentions >towards hierarchical packages. It says that all of sys.path will be >searched, but what about this case? > >In Twisted, the suggested idiom to structure a project which wants >to provide Twisted plugins is to have a directory structure like this: > > MyProject/ > myproject/ > __init__.py > twisted/ > plugins/ > myproject_plugin.py > >If you then put MyProject on PYTHONPATH, MyProject/twisted/plugins >will be picked up automatically by the plugin machinery. Namespaces are not plugins and vice versa. The purpose of a namespace package is to allow projects managed by the same entity to share a namespace (ala Java "package" names) and avoid naming conflicts with other authors. A plugin system, by contrast, is explicitly intended for use by multiple authors, so the use case is rather different... and using namespace packages for plugins actually *increases* the possibility of naming conflicts, unless you add back in another level of hierarchy. (As apparently you are recommending via "myproject_plugin".) > However, as "twisted" is *not* a "namespace" package in the same > way, .py files in MyProject/twisted/ would not be picked up - this > is very much intentional, since the "twisted" namespace is intended > to be reserved for packages that we actually produce. If either > MyProject/twisted or MyProject/twisted/plugins/ had an __init__.py, > then no modules in MyProject/twisted/plugins/ would be picked up, > because it would be considered a conflicting package. Precisely. Note, however, that neither is twisted.plugins a namespace package, and it should not contain any .pkg files. I don't think it's reasonable to abuse PEP 382 namespace packages as a plugin system. In setuptools' case, a different mechanism is provided for locating plugin code, and of course Twisted already has its own system for the same thing. It would be nice to have a standardized way of locating plugins in the stdlib, but that will need to be a different PEP. >I hope this all makes sense. As I understand it, both setuptools >and the proposed standard would either still have the bug described >by ticket 2339 above, or would ignore twisted/plugins/ as a >namespace package because its parent isn't a namespace package. If twisted/ lacked an __init__.py, then setuptools would ignore it. Under PEP 382, the same, unless it had .pkg files. (Again, setuptools explicitly does not support using namespace packages as a plugin mechanism.) >P.S.: vendor packaging systems *ARE* a major use case for just about >any aspect of Python's package structure. I really liked MvL's >coverage of "vendor packages", in the PEP, since this could equally >well apply to MSIs, python libraries distributed in bundles on OS X, >debs, or RPMs. If this use-case were to be ignored, as one >particular fellow seems to be advocating, then the broken packages >and user confusion that has been going on for the last 5 years or so >is just going to get worse. Indeed. [View Less]

2 1

Re: [Python-Dev] PEP 382: Namespace Packages
by P.J. Eby April 16, 2009

April 16, 2009

At 10:00 PM 4/15/2009 +0200, Tarek Ziadé wrote: >Now for the "base" or "core" package, what peoplethat uses setuptools >do most of the time: > >1- they use zc.buildout so they don't need a base package : they list >in a configuration files all packages needed > to build the application, and one of these package happen to have >the scripts to launch the application. > >2 - they have a "main" package that doesn't use the same namespace, >but uses setuptools … [View More]instal_requires metadata > to include namespaced packages. It acts like zc.buildout in some ways. > >For example, you mentioned atomisator.* in your example, this app has >a main package called "Atomisator" (notice the upper A) >that uses strategy #2 I think that there is some confusion here. A "main" package or buildout that assembles a larger project from components is not the same thing as having a "base" package for a namespace package. A base or core package is one that is depended upon by most or all of the related projects. In other words, the dependencies are in the *opposite direction* from what you described above. To have a base package in setuptools, you would move the target code from the namespace package __init__.py to another module or subpackage within your namespace, then make all your other projects depend on the project containing that module or subpackage. And I explicitly excluded from my survey any packages that were following this strategy, on the assumption that they might consider switching to an __init__.py or __pkg__.py strategy if some version of PEP 382 were supported by setuptools, since they already have a "base" or "core" project -- in that case, they are only changing ONE of their packages' distribution metadata to adopt the new strategy, because the dependencies already exist. >So : >- having namespaces natively in Python is a big win (Namespaces are >one honking great idea -- let's do more of those!) >- being able to still write some code under the primary namespace is >something I (and lots of people) wish we could do > with setuptools, so it's a big win too. Yes, that's why I support Martin's proposal: it would allow setuptools to support this case in the future, and it would also allow improved startup times for installations with many setuptools-based namespace packages installed in flat form. (Contra MAL's claims of decreased performance: adopting Martin's proposal allows there to be *fewer* .pth files read at startup, because only .pkg files for an actually-imported package need to be read.) [View Less]

2 1

Why does read() return bytes instead of bytearray?
by Dan Eloff April 15, 2009

April 15, 2009

>No, the read() method did not change from the 2.x series. It returns a new object on each call. I think you misunderstand me, but the readinto() method looks like a perfectly reasonable solution, I didn't realize it existed, as it's not in the library reference on file objects. Thanks for enlightening me, I feel a little stupid now :) Python 3, lookout, here I come! -Dan

5 4

Correction: Python Bug Day on April 25
by Georg Brandl April 15, 2009

April 15, 2009

Hi, I managed to screw up the date, so here it goes again: I'd like to announce that there will be a Python Bug Day on April 25. As always, this is a perfect opportunity to get involved in Python development, or bring your own issues to attention, discuss them and (hopefully) resolve them together with the core developers. We will coordinate over IRC, in #python-dev on irc.freenode.net, and the Wiki page http://wiki.python.org/moin/PythonBugDay has all important information and a short list … [View More]

3 2

Python Bug Day on April 23
by Georg Brandl April 15, 2009

April 15, 2009

Hi, I'd like to announce that there will be a Python Bug Day on April 23. As always, this is a perfect opportunity to get involved in Python development, or bring your own issues to attention, discuss them and (hopefully) resolve them together with the core developers. We will coordinate over IRC, in #python-dev on irc.freenode.net, and the Wiki page http://wiki.python.org/moin/PythonBugDay has all important information and a short list of steps how to get set up. Please spread the word! Georg

1 0

IDLE timeout.
by Alessio Giovanni Baroni April 15, 2009

April 15, 2009

Hi to all, I write on this list, because the error concerns the internals (I think). The IDLE has a strange behaviour. Sometimes, randomly, the IDLE restart the interpreter, with the follow exception on console: ---------------------------------------- Unhandled server exception! Thread: SockThread Client Address: ('127.0.0.1', 8833) Request: <socket.socket object, fd=3, family=2, type=1, proto=0> Traceback (most recent call last): File "/opt/python301/lib/python3.0/socketserver.py", … [View More]

2 2

Why does read() return bytes instead of bytearray?
by Dan Eloff April 15, 2009

April 15, 2009

Hi, Can someone please explain why read() should return an immutable bytes type instead of a mutable bytearray? It's not like read() from a file and use buffer as a key in a dict is common. Certainly read() from file or stream, modify, write is very common. I don't understand why the common case pays the price in performance and simplicity. It seemed to me that the immutable bytes was described as being useful in niche situations, but it actually seems to have been favored over bytearray in … [View More]

2 1

Shorter float repr in Python 3.1?
by Mark Dickinson April 14, 2009

April 14, 2009

Executive summary (details and discussion points below) ================= Some time ago, Noam Raphael pointed out that for a float x, repr(x) can often be much shorter than it currently is, without sacrificing the property that eval(repr(x)) == x, and proposed changing Python accordingly. See http://bugs.python.org/issue1580 For example, instead of the current behaviour: Python 3.1a2+ (py3k:71353:71354, Apr 7 2009, 12:55:16) [GCC 4.0.1 (Apple Inc. build 5490)] on darwin Type "help", "… [View More]copyright", "credits" or "license" for more information. >>> 0.01 0.01 >>> 0.02 0.02 >>> 0.03 0.029999999999999999 >>> 0.04 0.040000000000000001 >>> 0.04 == eval(repr(0.04)) True we'd have this: Python 3.1a2+ (py3k-short-float-repr:71350:71352M, Apr 7 2009, ) [GCC 4.0.1 (Apple Inc. build 5490)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> 0.01 0.01 >>> 0.02 0.02 >>> 0.03 0.03 >>> 0.04 0.04 >>> 0.04 == eval(repr(0.04)) True Initial attempts to implement this encountered various difficulties, and at some point Tim Peters pointed out (I'm paraphrasing horribly here) that one can't have all three of {fast, easy, correct}. One PyCon 2009 sprint later, Eric Smith and I have produced the py3k-short-float-repr branch, which implements short repr of floats and also does some major cleaning up of the current float formatting functions. We've gone for the {fast, correct} pairing. We'd like to get this into Python 3.1. Any thoughts/objections/counter-proposals/...? More details ============ Our solution is based on an adaptation of David Gay's 'perfect rounding' code for inclusion in Python. To make eval(repr(x)) roundtripping work, one needs to have correctly rounded float -> decimal *and* decimal -> float conversions: Gay's code provides correctly rounded dtoa and strtod functions for these two conversions. His code is well-known and well-tested: it's used as the basis of the glibc strtod, and is also in OS X. It's available from http://www.netlib.org/fp/dtoa.c So our branch contains a new file Python/dtoa.c, which is a cut down version of Gay's original file. (We've removed stuff for VAX and IBM floating-point formats, hex NaNs, hex floating-point formats, locale-aware interpretation of the decimal separator, K&R headers, code for correct setting of the inexact flag, and various other bits and pieces that Python doesn't care about.) Most of the rest of the work is in the existing file Python/pystrtod.c. Every float -> string or string -> float conversion goes through a function in this file at some point. Gay's code also provides the opportunity to clean up the current float formatting code, and Eric has reworked a lot of the float formatting in the py3k-short-float-repr branch. This reworking should make finishing off the implementation of things like thousands separators much more straightforward. One example of this: the previous string -> float conversion used the system strtod, which is locale-aware, so the code had to first replace the '.' by the current locale's decimal separator, *then* call strtod. There was a similar dance in the reverse direction when doing float -> string conversion. Both these are now unnecessary. The current code is pretty close to ready for merging to py3k. I've uploaded a patchset to Rietveld: http://codereview.appspot.com/33084/show Apart from the short float repr, and a couple of bugfixes, all behaviour should be unchanged from before. There are a few exceptions: - format(1e200, '<') doesn't behave quite as it did before. See item (3) below for details - repr switches to using exponential notation at 1e16 instead of the previous 1e17. This avoids a subtle issue where the 'short float repr' result is padded with bogus zeros. - a similar change applies to str, which switches to exponential notation at 1e11, not 1e12. This fixes the following minor annoyance, which goes back at least as far as Python 2.5 (and probably much further): >>> x = 1e11 + 0.5 >>> x 100000000000.5 >>> print(x) 100000000000.0 That .0 seems wrong to me: if we're going to go to the trouble of printing extra digits (str usually only gives 12 significant digits; here there are 13), they should be the *right* extra digits. Discussion points ================= (1) Any objections to including this into py3k? If there's controversy, then I guess we'll need a PEP. (2) Should other Python implementations (Jython, IronPython, etc.) be expected to use short float repr, or should it just be considered an implementation detail of CPython? I propose the latter, except that all implementations should be required to satisfy eval(repr(x)) == x for finite floats x. (3) There's a PEP 3101 line we don't know what to do with. In py3k, we currently have: >>> format(1e200, '<') '1.0e+200' but in our py3k-short-float-repr branch: >>> format(1e200, '<') '1e+200' Which is correct? The py3k behaviour comes from the 'Standard Format Specifiers' section of PEP 3101, where it says: """ The available floating point presentation types are: [... list of other format codes omitted here ...] '' (None) - similar to 'g', except that it prints at least one digit after the decimal point. """ It's that 'at least one digit after the decimal point' bit that's at issue. I understood this to apply only to floats converted to a string *without* an exponent; this is the way that repr and str work, adding a .0 to floats formatted without an exponent, but leaving the .0 out when the exponent is present. Should the .0 always be added? Or is it required only when it would be necessary to distinguish a float string from an integer string? My preference is for the latter (i.e., format(x, '<') should behave in the same way as repr and str in this respect). But I'm biased, not least because the other behaviour would be a pain to implement. Does anyone care? This email is already too long. I'll stop now. Mark [View Less]

10 29

Google Summer of Code/core Python projects - RFC
by C. Titus Brown April 13, 2009

April 13, 2009

Hi all, this year we have 10-12 GSoC applications that I've put in the "relevant to core Python development" category. These projects, if mentors etc are found, are *guaranteed* a slot under the PSF GSoC umbrella. As backup GSoC admin and general busybody, I've taken on the work of coordinating these as a special subgroup within the PSF GSoC, and I thought it would be good to mention them to python-dev. Note that all of them have been run by a few different committers, including Martin, … [View More]Tarek, Benjamin, and Brett, and they've been obliging enough to triage a few of them. Thanks, guys! Here's what's left after that triage. Note that except for the four at the top, these have all received positive support from *someone* who is a committer and I don't think we need to discuss them here -- patches etc. can go through normal "python-dev" channels during the course of the summer. I am looking for feedback on the first four, though. Can these reasonably be considered "core" priorites for Python? Remember, this "costs" us something in the sense of preferring these over Python subprojects like (random example) Cython, NumPy, PySoy, Tahoe, Gajim, etc. --- Questionable "core": 2x "port NumPy to py3k" -- NumPy is a major Python module and porting it to py3k fits with Guido's request that "more stuff get ported". To be clear, I don't think anyone expects all of NumPy to get ported this summer, but these students will work through issues associated with porting big chunks o' code to py3k. One medium/strong proposal, one medium/weak proposal. Comments/thoughts? 2x "improve testing tools for py3k" -- variously focus on improving test coverage and testing wrappers. One proposes to provide a nice wrapper to make nose and py.test capable of running the regrtests, which (with no change to regrtest) would let people run tests in parallel, distribute or run tests across multiple machines (including Snakebite), tag and run subsets of tests with personal and/or public tags, and otherwise take advantage of many of the nice features of nose and py.test. The other proposes to measure & increase the code coverage of the py3k tests in both Python and C, integrate across multiple machines, and otherwise provide a nice set of integrated reports that anyone can generate on their own machines. This proposal, in particular, could move smoothly towards the effort to produce a "Python-wide" test suite for CPython/IronPython/PyPy/Jython. (This wasn't integrated into the proposal because I only found out about it after the proposals were due.) I personally think that both testing proposals are good, and they grew out of conversations I had with Brett, who thinks that the general ideas are good. So, err, I'm looking for pushback, I guess ;). I can expand on these ideas a bit if people are interested. Both proposals are medium at least, and I've personally been positively impressed with the student interaction. Comments/thoughts? --- Unquestionably "core" by my criteria above: 3to2 tool -- 'nuff said. subprocess improvement -- integrating, testing, and proposing some of the various subprocess improvements that have passed across this list & the bug tracker IDLE/Tkinter patch integration & improvement -- deal with ~120 tracker issues relating to IDLE and Tkinter. roundup VCS integration / build tools to support core development -- a single student proposed both of these and has received some support. See http://slexy.org/view/s2pFgWxufI for details. sphinx framework improvement -- support for per-paragraph comments and user/developer interface for submitting/committing fixes 2x "keyring package" -- see http://tarekziade.wordpress.com/2009/03/27/pycon-hallway-session-1-a-keyrin…. The poorer one of these will probably be axed unless Tarek gives it strong support. -- --titus -- C. Titus Brown, ctb(a)msu.edu [View Less]

10 18

Contributor Agreements for Patches - was [Jython-dev] Jython on Google AppEngine!
by Jim Baker April 13, 2009

April 13, 2009

A question that arose on this thread, which I'm forwarding for context (and we're quite happy about it too!): - What is the scope of a patch that requires a contributor agreement? This particular patch on #1188 simply adds obvious (in retrospect of course) handling on SecurityException so that it's treated in a similar fashion to IOException (possibly a bit more buried), so it seems like a minor patch. - Do Google employees, working on company time, automatically get treated … [View More]

3 5