From victor.stinner at gmail.com Sun Sep 1 00:24:30 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Sun, 1 Sep 2013 00:24:30 +0200 Subject: [Python-Dev] Add a new tracemalloc module to trace memory allocations In-Reply-To: <CAGE7PNJ9DrKQ8Zb-E-F7QPRTP=t+S8StBvm3OQgvct8539gFPg@mail.gmail.com> References: <CAMpsgwaa3pABNrMSw2S7-Og111u8DdXK6dYnWVJzS4TAKbFNEQ@mail.gmail.com> <CAMpsgwatDdFQg4q=b3s7W6BZ+WQmpo=KTeJZd7ESbuGG+RzaKA@mail.gmail.com> <CAGE7PNJ9DrKQ8Zb-E-F7QPRTP=t+S8StBvm3OQgvct8539gFPg@mail.gmail.com> Message-ID: <CAMpsgwZBeN7W+GK6hLSHp4VYXAYgWFQBTx_-K3J7KzOMDB3H=g@mail.gmail.com> Le 31 ao?t 2013 19:09, "Gregory P. Smith" <greg at krypto.org> a ?crit : > First, I really like this. +1 Current votes: +3 (i also want tracemalloc!). No opposition against such addition. > We should be consistent with faulthandler's options. Why do you not want to support both the env var and enable()/disable() functions? The reason was technical but I have issues with enabling tracemalloc before Python is fully initialized. Enabling tracemalloc when Python is almost initialized and disabling it before Python exit is more reliable. In the last implementation, enable() and disable() are back. PYTHONTRACEMALLOC is still available. I will also add -X tracemalloc command line. > Users are likely to want snapshots captured by enable()/disable() around particular pieces of code just as much as whole program information. enable()/disable() are useful for tests of the tracemalloc module! > Taking that further: file and line information is great, but what if you extend the concept: could you allow for C API or even Python hooks to gather additional information at the time of each allocation or free? for example: Gathering the actual C and Python stack traces for correlation to figure out what call patterns lead allocations is powerful. There is no portable function to retrieve the C traceback. For the Python traceback: it may be possible to get it but you have to remember that the hook is on PyMem_Malloc(), a low level memory allocator. The GIL is hold. It is possible to call Python code in some cases, but they are corner cases where it would lead to a loop, deadlock or worse. I prefer to only read available Python data without calling any Python code and try to write a reliable debug tool. I still have some issues like issues with reentrant calls to the hook, but I'm working on them. A nice improvement compared to the implementation on PyPI would be to hook PyMem_RawMalloc(), to trace also the memory used by gzip, bz2, lzma and other C modules. Locking the GIL in PyMem_RawMalloc() to get the filename and locking internals tracemalloc structures causes also new issues (like a funny deadlock related to subinterpreter). > (Yes, this gets messy fast as hooks should not trigger calls back into themselves when they allocate or free, similar to the "fun" involved in writing coverage tools) tracemalloc uses a "reentrant" variable to do nothing on a reentrant call to the hook. > let me know if you think i'm crazy. :) You are not crazy. It is just hard to implement it. I'm not sure that it is possible. Victor -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130901/d9caffd6/attachment.html> From ncoghlan at gmail.com Sun Sep 1 03:28:36 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 1 Sep 2013 11:28:36 +1000 Subject: [Python-Dev] Pre-PEP: Redesigning extension modules In-Reply-To: <kvtfek$23p$1@ger.gmane.org> References: <kv77o2$f96$1@ger.gmane.org> <20130823111822.49cba700@pitrou.net> <CADiSq7dhzpTWMw7gcBRrzZot7jaS=Aj=aukq0+jh_cK2+F_kGQ@mail.gmail.com> <CADiSq7dBruZiWhObnjKbcneji9+ziXmyRpg1+72a8YeMWbyO3A@mail.gmail.com> <kvabt8$jfc$1@ger.gmane.org> <CADiSq7c6OjFa8zo7sMB=k2zLSmaf1w-tAwhYKSQ26T3caLyj6A@mail.gmail.com> <kvaka3$sbu$1@ger.gmane.org> <CADiSq7dzQqyH9i6g9QYWp6_ZHHUD4+4npYo6of6mCNahQ+w8kw@mail.gmail.com> <kvcr9f$4gs$1@ger.gmane.org> <CADiSq7cfNH1M29W19BW88-BBQaWEpaMkE=s09wNyT5VGOrPB+g@mail.gmail.com> <kvtfek$23p$1@ger.gmane.org> Message-ID: <CADiSq7d3L1+Jc5E4dWYDRUQPNDCmxqyc3GzbgXJvGMe4+OnWrQ@mail.gmail.com> On 1 Sep 2013 05:18, "Stefan Behnel" <stefan_ml at behnel.de> wrote: > > Nick Coghlan, 31.08.2013 18:49: > > On 25 Aug 2013 21:56, "Stefan Behnel" wrote: > >>>>> One key point to note is that it *doesn't* call > >>>>> _PyImport_FixupExtensionObject, which is the API that handles all the > >>>>> PEP 3121 per-module state stuff. Instead, the idea will be for modules > >>>>> that don't need additional C level state to just implement > >>>>> PyImportExec_NAME, while those that *do* need C level state implement > >>>>> PyImportCreate_NAME and return a custom object (which may or may not > >>>>> be a module subtype). > >>>> > >>>> Is it really a common case for an extension module not to need any C > >>>> level > >>>> state at all? I mean, this might work for very simple accelerator > >>>> modules > >>>> with only a few stand-alone functions. But anything non-trivial will > >>>> almost > >>>> certainly have some kind of global state, cache, external library, > >>>> etc., > >>>> and that state is best stored at the C level for safety reasons. > > > > In my experience, most extension authors aren't writing high performance C > > accelerators, they're exposing an existing C API to Python. It's the cffi > > use case rather than the Cython use case. > > Interesting. I can't really remember a case where I could afford the > runtime overhead of implementing a wrapper in Python and going through > something like ctypes or cffi. I mean, testing C libraries with Python > tools would be one, but then, you wouldn't want to write an extension > module for that and instead want to call it directly from the test code as > directly as possible. > > I'm certainly aware that that use case exists, though, and also the case of > just wanting to get things done as quickly and easily as possible. Keep in mind I first came to Python as a tool for test automation of custom C++ hardware APIs that could be written to be SWIG friendly. I now work for an OS vendor where the 3 common languages for system utilities are C, C++ and Python. For those use cases, dropping a bunch of standard Python objects in a module dict is often going to be a quick and easy solution that avoids a lot of nasty pointer lifecycle issues at the C level. This style of extension code would suffer similar runtime checking overhead as Python, including for function calls, but, like CPython itself, would still often be "fast enough". However, as soon as you want to manually optimise for *speed* at all, you're going to want to remove those module internal indirections through the Python API. There are at least three ways to do this (internally, CPython uses all of them in various places): * type checks followed by direct function calls on the optimised path and falling back to the abstract object APIs on the compatibility path * type checks followed by an exception for unknown types * hidden state that isn't exposed directly at the Python level and hence can be trusted to only be changed through the module APIs. The third approach can be implemented in three ways, with various consequences: * C static variables. For mutable state, including pointers to Python types, this breaks subinterpreters, reloading in place and loading a fresh copy of the module * PEP 3121 per-interpreter shared state. Handles subinterpreters, *may* handle reloading (but may segfault if references are held to old types and functions from before the reload), doesn't handle loading a fresh copy at all. * PEP 3121 with a size of "0". As above, but avoids the module state APIs in order to support reloading. All module state (including type cross-references) is stored in hidden state (e.g. an instance of a custom type not exposed to Python, with a reference stored on each custom type object defined in the module, and any module level "functions" actually being methods of a hidden object). Still doesn't support loading a *fresh* copy due to the hidden PEP 3121 module cache. The proposed new approach is to bypass the PEP 3121 cache entirely, and instead recommend providing an instance of a custom type to be placed in sys.modules. Extension modules will be given the ability to explicitly disallow in-place reloading *or* to make it work reliably, rather than the status quo where the import system assumes it will work, and instead may fail in unusual ways. > > Mutable module global state is always a recipe for obscure bugs, and not > > something I will ever let through code review without a really good > > rationale. Hidden process global state is never good, just sometimes a > > necessary evil. > > I'm not necessarily talking about mutable state. Rather about things like > pre-initialised data or imported functionality. For example, I often have a > bound method of a compiled regex lying around somewhere in my Python > modules as a utility function. And the same kind of stuff exists in C code, > some may be local to a class, but other things can well be module global. > And given that we are talking about module internals here I'd always keep > them at the C level rather than exposing them through the module dict. The > module dict involves a much higher access overhead, in addition to the > reduced safety due to user accessibility. > > Exported C-APIs are also a use case. You'd import the C-API of another > module at init time and from that point on only go through function > pointers etc. Those are (sub-)interpreter specific, i.e. they are module > global state that is specific to the currently loaded module instances. Due to refcounting, all instances of Python objects qualify as mutable state. Hopefully my elaboration above helps make it clear why I think it's worthwhile to clearly separate out the "no custom C level state needed" case. > > However, keep in mind my patch is currently just the part I can implement > > without PEP 451 module spec objects. > > Understood. > > > >> Note that even global functions usually hold state, be it in the form of > >> globally imported modules, global caches, constants, ... > > > > If they can be shared safely across multiple instances of the module (e.g. > > immutable constants), then these can be shared at the C level. Otherwise, a > > custom Python type will be needed to make them instance specific. > > I assume you meant a custom module (extension) type here. Not sure yet. For PEP 451, we still need to support arbitrary objects in sys.modules, so it's still possible that freedom will be made available to extension modules. > > Just to be clear, the "module state at the C-level" is meant to be stored > in the object struct fields of the extension type that implements the > module, at least for modules that want to support reloading and > sub-interpreters. Obviously, nothing should be stored in static (global) > variables etc. Right. > >>> We also need the create/exec split to properly support reloading. Reload > >>> *must* reinitialize the object already in sys.modules instead of > >>> inserting > >>> a different object or it completely misses the point of reloading > >>> modules > >>> over deleting and reimporting them (i.e. implicitly affecting the > >>> references from other modules that imported the original object). > >> > >> Interesting. I never thought of it that way. > >> > >> I'm not sure this can be done in general. What if the module has threads > >> running that access the global state? In that case, reinitialising the > >> module object itself would almost certainly lead to a crash. That's why I want a way for loaders in general (and extension modules in particular) to clearly say "in-place reloading not supported", rather than Python blundering ahead with it and risking a crash. > > My current proposal on import-sig is to make the first hook > > "prepare_module", and pass in the existing object in the reload case. For > > the extension loader, this would be reflected in the signature of the C > > level hook as well, so the module could decide for itself if it supported > > reloading. > > I really don't like the idea of reloading by replacing module state. It > would be much simpler if the module itself would be replaced, then the > original module could stay alive and could still be used by those who hold > a reference to it or parts of its contents. Especially the from-import case > would benefit from this. Obviously, you could still run into obscure bugs > where a function you call rejects the input because it expects an older > version of a type, for example. But I can't see that being worse (or even > just different) from the reload-by-refilling-dict case. Sure, this is what we do in the test suite in "test.support.import_fresh_module". It was actually Eli trying to use that in the etree tests that triggered our recent investigation of the limits of PEP 3121 (it breaks for stateful extension modules due to the per-interpreter caching). It's a different operation from imp.reload, though. Assuming we can get this stable and reliable in the new API, I expect we'll be able to add "imp.reload_fresh" as a supported API in 3.5. > You seemed to be ok with my idea of making the loader return a wrapped > extension module instead of the module itself. We should actually try that. Sure, that's just a variant of the "hidden state object" idea I described above. It should actually work today with the PEP 3121 custom storage size set to zero. > > This is actually my primary motivation for trying to improve the > > "can this be reloaded or not?" aspects of the loader API in PEP 451. > > I assume you mean that the extension module would be able to clearly signal > that it can't be reloaded, right? I agree that that's helpful. If you're > wrapping a C library, then the way that library is implemented might simply > force you to prevent any attempts at reloading the wrapper module. But if > reloading is possible at all, it would be even more helpful if we could > make it really easy to properly support it. Yep, that's my goal (and why it's really good to be having this discussion while PEP 451 is still in development). > > > > (keep in mind existing extension modules using the existing API will still > > never be reloaded) > > Sure, that's the cool thing. We can really design this totally from scratch > without looking back. Well, not *quite*. We need to ensure a module can implement both APIs can coexist in the same module for source compatibility without nasty ifdef hacks, and that there is a reasonable migration path for existing handwritten extension modules. > >>> Take a look at the current example - everything gets stored in the > >>> module dict for the simple case with no C level global state. > >> > >> Well, you're storing types there. And those types are your module API. I > >> understand that it's just an example, but I don't think it matches a > >> common > >> case. As far as I can see, the types are not even interacting with each > >> other, let alone doing any C-level access of each other. We should try to > >> focus on the normal case that needs C-level state and C-level field access > >> of extension types. Once that's solved, we can still think about how to > >> make the really simple cases simpler, if it turns out that they are not > >> simple enough. > > > > Our experience is very different - my perspective is that the normal case > > either eschews C level global state in the extension module, because it > > causes so many problems, or else just completely ignores subinterpreter > > support and proper module cleanup. > > As soon as you have more than one extension type in your module, and they > interact with each other, they will almost certainly have to do type checks > against each other to make sure users haven't passed them rubbish before > they access any C struct fields of the object. Doing a type check means > that at least one type has a pointer to the other, meaning that it holds > global module state. Sure, but you can use the CPython API rather than writing normal C code. We do this fairly often in CPython when we're dealing with things stored in modules that can be manipulated from Python. It incurs CPython's dynamic dispatch overhead, but sometimes that's worth it to avoid needing to deal with C level lifecycle issues. > I really think that having some kind of global module state is the > exceedingly common case for an extension module. I wouldn't be willing to make the call about which of stateless vs stateful is more common without a lot more research :) They're both common enough that I think they should both be well supported, and making the "no custom C level state" case as simple as possible. > >> I didn't know about PyType_FromSpec(), BTW. It looks like a nice addition > >> for manually written code (although useless for Cython). > > > > This is the only way to create custom types when using the stable ABI. Can > > I take your observation to mean that Cython doesn't currently offer the > > option of limiting itself to the stable ABI? > > Correct. I've taken a bird's view at it back then, and keep stumbling over > "wow - I couldn't even use that?" kind of declarations in the header files. > I don't think it makes sense for Cython. Existing CPython versions are easy > to support because they don't change anymore, and new major releases most > likely need adaptations anyway, if only to adapt to new features and > performance changes. Cython actually knows quite a lot about the inner > workings of CPython and its various releases. Going only through the stable > ABI parts of the C-API would make the code horribly slow in comparison, so > there are huge drawbacks for the benefit it might give. > > The Cython way of doing it is more like: you want your code to run on a new > CPython version, then use a recent Cython release to compile it. It may > still work with older ones, but what you actually want is the newest > anyway, and you also want to compile the C code for the specific CPython > version at hand to get the most out of it. It's the C code that adapts, not > the runtime code (or Cython itself). > > We run continuous integration tests with all of CPython's development > branches since 2.4, so we usually support new CPython releases long before > they are out. And new releases of CPython rarely affect Cython user code. The main advantage of the stable ABI is being able to publish cross-version binary extension modules. I guess if Cython already supports generating binaries for each new version of CPython before we release it, that capability is indeed less useful than it is for those that are maintaining extension modules by hand. Cheers, Nick. > > Stefan > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130901/aa8d7cd8/attachment.html> From ncoghlan at gmail.com Sun Sep 1 03:36:51 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 1 Sep 2013 11:36:51 +1000 Subject: [Python-Dev] Add a new tracemalloc module to trace memory allocations In-Reply-To: <CAMpsgwZBeN7W+GK6hLSHp4VYXAYgWFQBTx_-K3J7KzOMDB3H=g@mail.gmail.com> References: <CAMpsgwaa3pABNrMSw2S7-Og111u8DdXK6dYnWVJzS4TAKbFNEQ@mail.gmail.com> <CAMpsgwatDdFQg4q=b3s7W6BZ+WQmpo=KTeJZd7ESbuGG+RzaKA@mail.gmail.com> <CAGE7PNJ9DrKQ8Zb-E-F7QPRTP=t+S8StBvm3OQgvct8539gFPg@mail.gmail.com> <CAMpsgwZBeN7W+GK6hLSHp4VYXAYgWFQBTx_-K3J7KzOMDB3H=g@mail.gmail.com> Message-ID: <CADiSq7dKojZ5f1bprhZZxu6DjovjWtwy-8o=-BYjYRvHZWqoFw@mail.gmail.com> +1 from me for both tracemalloc and failmalloc in the same vein as faulthandler (and using similar API concepts to faulthandler). While I like the flat top level naming structure, we should clearly document these as implementation dependent runtime services. Other implementations may not provide them at all, and even if they do, many details will likely be different. The gc module may even fall into the same category. Cheers, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130901/4bed69d1/attachment.html> From stefan_ml at behnel.de Sun Sep 1 10:11:36 2013 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 01 Sep 2013 10:11:36 +0200 Subject: [Python-Dev] Pre-PEP: Redesigning extension modules In-Reply-To: <CADiSq7d3L1+Jc5E4dWYDRUQPNDCmxqyc3GzbgXJvGMe4+OnWrQ@mail.gmail.com> References: <kv77o2$f96$1@ger.gmane.org> <20130823111822.49cba700@pitrou.net> <CADiSq7dhzpTWMw7gcBRrzZot7jaS=Aj=aukq0+jh_cK2+F_kGQ@mail.gmail.com> <CADiSq7dBruZiWhObnjKbcneji9+ziXmyRpg1+72a8YeMWbyO3A@mail.gmail.com> <kvabt8$jfc$1@ger.gmane.org> <CADiSq7c6OjFa8zo7sMB=k2zLSmaf1w-tAwhYKSQ26T3caLyj6A@mail.gmail.com> <kvaka3$sbu$1@ger.gmane.org> <CADiSq7dzQqyH9i6g9QYWp6_ZHHUD4+4npYo6of6mCNahQ+w8kw@mail.gmail.com> <kvcr9f$4gs$1@ger.gmane.org> <CADiSq7cfNH1M29W19BW88-BBQaWEpaMkE=s09wNyT5VGOrPB+g@mail.gmail.com> <kvtfek$23p$1@ger.gmane.org> <CADiSq7d3L1+Jc5E4dWYDRUQPNDCmxqyc3GzbgXJvGMe4+OnWrQ@mail.gmail.com> Message-ID: <kvusrg$2vm$1@ger.gmane.org> Nick Coghlan, 01.09.2013 03:28: > On 1 Sep 2013 05:18, "Stefan Behnel" wrote: >> I can't really remember a case where I could afford the >> runtime overhead of implementing a wrapper in Python and going through >> something like ctypes or cffi. I mean, testing C libraries with Python >> tools would be one, but then, you wouldn't want to write an extension >> module for that and instead want to call it directly from the test code as >> directly as possible. >> >> I'm certainly aware that that use case exists, though, and also the case >> of just wanting to get things done as quickly and easily as possible. > > Keep in mind I first came to Python as a tool for test automation of custom > C++ hardware APIs that could be written to be SWIG friendly. Interesting again. Would you still do it that way? I recently had a discussion with Holger Krekel of py.test fame about testing C code with Cython, and we quickly agreed that wrapping the code in an extension module was both too cumbersome and too inflexible for testing purposes. Specifically, neither of Cython's top selling points fits here, not speed, not clarity, not API design. It's most likely different for SWIG, which involves less (not no, just less) manual work and gives you API-wise more of less exactly what you put in. However, cffi is almost certainly the better way to do it, because it gives you all sorts of flexibility for your test code without having to think about the wrapper design all the time. The situation is also different for C++ where you have less options for wrapping it. I can imagine SWIG still being the tool of choice on that front when it comes to bare and direct testing of large code bases. > I now work for an OS vendor where the 3 common languages for system > utilities are C, C++ and Python. > > For those use cases, dropping a bunch of standard Python objects in a > module dict is often going to be a quick and easy solution that avoids a > lot of nasty pointer lifecycle issues at the C level. That's yet another use case, BTW. When you control the whole application, then safety doesn't really matter at these points and keeping a bunch of stuff in a dict will usually work just fine. I'm mainly used to writing libraries for (sometimes tons of) other people, in which case the requirements are so diverse on user side that safety is a top thing to care about. Anything you can keep inside of C code should stay there. (Especially when dealing with libxml2&friends in lxml which continuously present their 'interesting' usability characteristics.) > * PEP 3121 with a size of "0". As above, but avoids the module state APIs > in order to support reloading. All module state (including type > cross-references) is stored in hidden state (e.g. an instance of a custom > type not exposed to Python, with a reference stored on each custom type > object defined in the module, and any module level "functions" actually > being methods of a hidden object). Thanks for elaborating. I had completely failed to make the mental link that you could simply stick bound methods as functions into the module dict, i.e. that they don't even have to be methods of the module itself. That's something that Cython could already use in older CPythons, even as a preparation for any future import protocol changes. The object that they are methods of would then eventually become the module instance. You'd still suffer a slight performance hit from going from a static global C variable to a pointer indirection - for everything: string constants, cached Python objects, all user defined global C variables would have to go there as Cython cannot know if they are module instance specific state or not (they usually will be, I guess). But that has to be done anyway if the goal is to get rid of static state to enable sub-interpreters. I can't wait seeing lxml run threaded in mod_wsgi... ;-) >> You seemed to be ok with my idea of making the loader return a wrapped >> extension module instead of the module itself. We should actually try >> that. > > Sure, that's just a variant of the "hidden state object" idea I described > above. It should actually work today with the PEP 3121 custom storage size > set to zero. True. The only difference is whether you leave it to the extension type itself or make it a part of the loader architecture. Anyway, I promise I'll give it a try in Cython. Will be some work, though, to rewrite Cython's use of global variables, create a module state type, migrate everything to heap types, ... I had wanted to do that for a couple of years, but it's clearly not something for a happy afternoon or two. Plus, it would even have to be optional in the compiler to avoid performance regressions for modules that want to continue using fast static globals simply because they cannot support multiple instances anyway (e.g. due to external C library dependencies). Let's see if we can solve that at C compilation time by throwing in a couple of macros. That would at least help keeping the Cython compiler itself simple in that regard... (I guess it would also help with testing as we could just duplicate the test suite runs for both design modes) >> As soon as you have more than one extension type in your module, and they >> interact with each other, they will almost certainly have to do type >> checks >> against each other to make sure users haven't passed them rubbish before >> they access any C struct fields of the object. Doing a type check means >> that at least one type has a pointer to the other, meaning that it holds >> global module state. > > Sure, but you can use the CPython API rather than writing normal C code. We > do this fairly often in CPython when we're dealing with things stored in > modules that can be manipulated from Python. > > It incurs CPython's dynamic dispatch overhead, but sometimes that's worth > it to avoid needing to deal with C level lifecycle issues. Not so much of a problem in Cython, because all you usually have to do to get fast C level access to something is to change a "def" into a "cdef" somewhere, or add a decorator, or an assignment to a known extension type variable. Once the module global state is 'virtualised', this will also be a safe thing to do in the face of multiple module instances, and still be much faster than going through Python calls. >> I really think that having some kind of global module state is the >> exceedingly common case for an extension module. > > I wouldn't be willing to make the call about which of stateless vs stateful > is more common without a lot more research :) > > They're both common enough that I think they should both be well supported, > and making the "no custom C level state" case as simple as possible. Agreed. >>>> I didn't know about PyType_FromSpec(), BTW. It looks like a nice >>>> addition for manually written code (although useless for Cython). >>> >>> This is the only way to create custom types when using the stable ABI. I actually think I recall reading about it in the PEP back when it was designed, decided that it made sense in the given context, and then forgot about it as I didn't consider it relevant. > The main advantage of the stable ABI is being able to publish cross-version > binary extension modules. I guess if Cython already supports generating > binaries for each new version of CPython before we release it, that > capability is indeed less useful than it is for those that are maintaining > extension modules by hand. I consider it mostly interesting for Linux distributions and closed source module vendors as it reduces the build/support overhead. But compiling the generated C code for the specific CPython version at hand really has some major advantages. Stefan From ncoghlan at gmail.com Sun Sep 1 14:23:02 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 1 Sep 2013 22:23:02 +1000 Subject: [Python-Dev] Pre-PEP: Redesigning extension modules In-Reply-To: <kvusrg$2vm$1@ger.gmane.org> References: <kv77o2$f96$1@ger.gmane.org> <20130823111822.49cba700@pitrou.net> <CADiSq7dhzpTWMw7gcBRrzZot7jaS=Aj=aukq0+jh_cK2+F_kGQ@mail.gmail.com> <CADiSq7dBruZiWhObnjKbcneji9+ziXmyRpg1+72a8YeMWbyO3A@mail.gmail.com> <kvabt8$jfc$1@ger.gmane.org> <CADiSq7c6OjFa8zo7sMB=k2zLSmaf1w-tAwhYKSQ26T3caLyj6A@mail.gmail.com> <kvaka3$sbu$1@ger.gmane.org> <CADiSq7dzQqyH9i6g9QYWp6_ZHHUD4+4npYo6of6mCNahQ+w8kw@mail.gmail.com> <kvcr9f$4gs$1@ger.gmane.org> <CADiSq7cfNH1M29W19BW88-BBQaWEpaMkE=s09wNyT5VGOrPB+g@mail.gmail.com> <kvtfek$23p$1@ger.gmane.org> <CADiSq7d3L1+Jc5E4dWYDRUQPNDCmxqyc3GzbgXJvGMe4+OnWrQ@mail.gmail.com> <kvusrg$2vm$1@ger.gmane.org> Message-ID: <CADiSq7cN17Maf5wdrmyVR=xG4EUqxeuzDfm7ZWa6=6oMFS_iLA@mail.gmail.com> On 1 September 2013 18:11, Stefan Behnel <stefan_ml at behnel.de> wrote: > Nick Coghlan, 01.09.2013 03:28: >> On 1 Sep 2013 05:18, "Stefan Behnel" wrote: >>> I can't really remember a case where I could afford the >>> runtime overhead of implementing a wrapper in Python and going through >>> something like ctypes or cffi. I mean, testing C libraries with Python >>> tools would be one, but then, you wouldn't want to write an extension >>> module for that and instead want to call it directly from the test code as >>> directly as possible. >>> >>> I'm certainly aware that that use case exists, though, and also the case >>> of just wanting to get things done as quickly and easily as possible. >> >> Keep in mind I first came to Python as a tool for test automation of custom >> C++ hardware APIs that could be written to be SWIG friendly. > > Interesting again. Would you still do it that way? I recently had a > discussion with Holger Krekel of py.test fame about testing C code with > Cython, and we quickly agreed that wrapping the code in an extension module > was both too cumbersome and too inflexible for testing purposes. > Specifically, neither of Cython's top selling points fits here, not speed, > not clarity, not API design. It's most likely different for SWIG, which > involves less (not no, just less) manual work and gives you API-wise more > of less exactly what you put in. However, cffi is almost certainly the > better way to do it, because it gives you all sorts of flexibility for your > test code without having to think about the wrapper design all the time. > > The situation is also different for C++ where you have less options for > wrapping it. I can imagine SWIG still being the tool of choice on that > front when it comes to bare and direct testing of large code bases. To directly wrap C++, I'd still use SWIG. It makes a huge difference when you can tweak the C++ side of the API to be SWIG friendly rather than having to live with whatever a third party C++ library provides. Having classes in C++ map directly to classes in Python is the main benefit of doing it this way over using a C wrapper and cffi. However, for an existing C API, or a custom API where I didn't need the direct object mapping that C++ can provide, using cffi would be a more attractive option than SWIG these days (the stuff I was doing with SWIG was back around 2003 or so). I think this is getting a little off topic for the list, though :) >> I now work for an OS vendor where the 3 common languages for system >> utilities are C, C++ and Python. >> >> For those use cases, dropping a bunch of standard Python objects in a >> module dict is often going to be a quick and easy solution that avoids a >> lot of nasty pointer lifecycle issues at the C level. > > That's yet another use case, BTW. When you control the whole application, > then safety doesn't really matter at these points and keeping a bunch of > stuff in a dict will usually work just fine. I'm mainly used to writing > libraries for (sometimes tons of) other people, in which case the > requirements are so diverse on user side that safety is a top thing to care > about. Anything you can keep inside of C code should stay there. > (Especially when dealing with libxml2&friends in lxml which continuously > present their 'interesting' usability characteristics.) I don't think it's a coincidence that it was the etree interface with expat that highlighted the deficiencies of the current extension module hooks when it comes to working properly with test.support.import_fresh_module :) >> * PEP 3121 with a size of "0". As above, but avoids the module state APIs >> in order to support reloading. All module state (including type >> cross-references) is stored in hidden state (e.g. an instance of a custom >> type not exposed to Python, with a reference stored on each custom type >> object defined in the module, and any module level "functions" actually >> being methods of a hidden object). > > Thanks for elaborating. I had completely failed to make the mental link > that you could simply stick bound methods as functions into the module > dict, i.e. that they don't even have to be methods of the module itself. > That's something that Cython could already use in older CPythons, even as a > preparation for any future import protocol changes. The object that they > are methods of would then eventually become the module instance. > > You'd still suffer a slight performance hit from going from a static global > C variable to a pointer indirection - for everything: string constants, > cached Python objects, all user defined global C variables would have to go > there as Cython cannot know if they are module instance specific state or > not (they usually will be, I guess). But that has to be done anyway if the > goal is to get rid of static state to enable sub-interpreters. I can't wait > seeing lxml run threaded in mod_wsgi... ;-) To be honest, I didn't realise that such a trick might already be possible until I was writing down this list of alternatives. If you manage to turn it into a real solution for lxml (or Cython in general), it would be great to hear more about how you turned the general idea into something real :) That means the powers any new extension initialisation API will offer will be limited to: * letting the module know its own name (and other details) * letting the module explicitly block reloading * letting the module support loading multiple copies at once by taking the initial import out of sys.modules (but keeping a separate reference to it alive) <snip> >>> As soon as you have more than one extension type in your module, and they >>> interact with each other, they will almost certainly have to do type >>> checks >>> against each other to make sure users haven't passed them rubbish before >>> they access any C struct fields of the object. Doing a type check means >>> that at least one type has a pointer to the other, meaning that it holds >>> global module state. >> >> Sure, but you can use the CPython API rather than writing normal C code. We >> do this fairly often in CPython when we're dealing with things stored in >> modules that can be manipulated from Python. >> >> It incurs CPython's dynamic dispatch overhead, but sometimes that's worth >> it to avoid needing to deal with C level lifecycle issues. > > Not so much of a problem in Cython, because all you usually have to do to > get fast C level access to something is to change a "def" into a "cdef" > somewhere, or add a decorator, or an assignment to a known extension type > variable. Once the module global state is 'virtualised', this will also be > a safe thing to do in the face of multiple module instances, and still be > much faster than going through Python calls. It's kinda cool how designing a next generation API can sometimes reveal hidden possibilities of an *existing* API :) We had another one of those recently on distutils-sig, when I realised the much-maligned .pth modules are actually a decent solution to sharing distributions between virtual environments. I'm so used to disliking their global side effects when used with the system Python that it took me a long time to recognise the validity of using them to make implicit path additions in a more controlled virtual environment :) In terms of where we go from here - do you mind if I use your pre-PEP as the initial basis for a PEP of my own some time in the next week or two (listing you as co-author)? Improving extension module initialisation has been the driver for most of the PEP 451 feedback I've been giving to Eric over on import-sig, so I have some definite ideas on how I think that API should look :) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From stefan_ml at behnel.de Sun Sep 1 14:41:50 2013 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 01 Sep 2013 14:41:50 +0200 Subject: [Python-Dev] Pre-PEP: Redesigning extension modules In-Reply-To: <CADiSq7cN17Maf5wdrmyVR=xG4EUqxeuzDfm7ZWa6=6oMFS_iLA@mail.gmail.com> References: <kv77o2$f96$1@ger.gmane.org> <20130823111822.49cba700@pitrou.net> <CADiSq7dhzpTWMw7gcBRrzZot7jaS=Aj=aukq0+jh_cK2+F_kGQ@mail.gmail.com> <CADiSq7dBruZiWhObnjKbcneji9+ziXmyRpg1+72a8YeMWbyO3A@mail.gmail.com> <kvabt8$jfc$1@ger.gmane.org> <CADiSq7c6OjFa8zo7sMB=k2zLSmaf1w-tAwhYKSQ26T3caLyj6A@mail.gmail.com> <kvaka3$sbu$1@ger.gmane.org> <CADiSq7dzQqyH9i6g9QYWp6_ZHHUD4+4npYo6of6mCNahQ+w8kw@mail.gmail.com> <kvcr9f$4gs$1@ger.gmane.org> <CADiSq7cfNH1M29W19BW88-BBQaWEpaMkE=s09wNyT5VGOrPB+g@mail.gmail.com> <kvtfek$23p$1@ger.gmane.org> <CADiSq7d3L1+Jc5E4dWYDRUQPNDCmxqyc3GzbgXJvGMe4+OnWrQ@mail.gmail.com> <kvusrg$2vm$1@ger.gmane.org> <CADiSq7cN17Maf5wdrmyVR=xG4EUqxeuzDfm7ZWa6=6oMFS_iLA@mail.gmail.com> Message-ID: <kvvcm6$okb$1@ger.gmane.org> Nick Coghlan, 01.09.2013 14:23: > That means the powers any new extension initialisation API will offer > will be limited to: > > * letting the module know its own name (and other details) > * letting the module explicitly block reloading > * letting the module support loading multiple copies at once by taking > the initial import out of sys.modules (but keeping a separate > reference to it alive) Which, all by themselves, can be considered a huge benefit, IMHO. Plus, if we design the protocol broad enough now, specifically as a two-way interface (info in, module out), we won't have to make any major changes to it again anywhere in the near future, because incremental changes can just be integrated into what's there then, in case we need any. It's sad that we didn't see these requirements for Py3.0. > In terms of where we go from here - do you mind if I use your pre-PEP > as the initial basis for a PEP of my own some time in the next week or > two (listing you as co-author)? Improving extension module > initialisation has been the driver for most of the PEP 451 feedback > I've been giving to Eric over on import-sig, so I have some definite > ideas on how I think that API should look :) Go for it. I'm not sure how much time I can actively spend on this during the next weeks anyway, so I'm happy if this continues to get pushed onwards in the meantime. Stefan From solipsis at pitrou.net Sun Sep 1 15:03:51 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 1 Sep 2013 15:03:51 +0200 Subject: [Python-Dev] Pre-PEP: Redesigning extension modules References: <kv77o2$f96$1@ger.gmane.org> <20130823111822.49cba700@pitrou.net> <CADiSq7dhzpTWMw7gcBRrzZot7jaS=Aj=aukq0+jh_cK2+F_kGQ@mail.gmail.com> <CADiSq7dBruZiWhObnjKbcneji9+ziXmyRpg1+72a8YeMWbyO3A@mail.gmail.com> <kvabt8$jfc$1@ger.gmane.org> <CADiSq7c6OjFa8zo7sMB=k2zLSmaf1w-tAwhYKSQ26T3caLyj6A@mail.gmail.com> <kvaka3$sbu$1@ger.gmane.org> <CADiSq7dzQqyH9i6g9QYWp6_ZHHUD4+4npYo6of6mCNahQ+w8kw@mail.gmail.com> <kvcr9f$4gs$1@ger.gmane.org> <CADiSq7cfNH1M29W19BW88-BBQaWEpaMkE=s09wNyT5VGOrPB+g@mail.gmail.com> <kvtfek$23p$1@ger.gmane.org> <CADiSq7d3L1+Jc5E4dWYDRUQPNDCmxqyc3GzbgXJvGMe4+OnWrQ@mail.gmail.com> Message-ID: <20130901150351.2f2aa44c@fsol> On Sun, 1 Sep 2013 11:28:36 +1000 Nick Coghlan <ncoghlan at gmail.com> wrote: > * PEP 3121 with a size of "0". As above, but avoids the module state APIs > in order to support reloading. All module state (including type > cross-references) is stored in hidden state (e.g. an instance of a custom > type not exposed to Python, with a reference stored on each custom type > object defined in the module, and any module level "functions" actually > being methods of a hidden object). Still doesn't support loading a *fresh* > copy due to the hidden PEP 3121 module cache. Not sure what you mean by that: >>> import atexit >>> id(atexit) 140031896222680 >>> import sys >>> del sys.modules['atexit'] >>> import atexit >>> id(atexit) 140031896221400 > Due to refcounting, all instances of Python objects qualify as mutable > state. That's an overly broad definition. Many objects are shared between subinterpreters without any problems (None, the empty tuple, built-in types and most C extension types, etc.). As long as the state is an internal implementation detail, there shouldn't be any problem. > I wouldn't be willing to make the call about which of stateless vs stateful > is more common without a lot more research :) > > They're both common enough that I think they should both be well supported, > and making the "no custom C level state" case as simple as possible. Agreed. Regards Antoine. From ncoghlan at gmail.com Sun Sep 1 16:10:08 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 2 Sep 2013 00:10:08 +1000 Subject: [Python-Dev] Pre-PEP: Redesigning extension modules In-Reply-To: <20130901150351.2f2aa44c@fsol> References: <kv77o2$f96$1@ger.gmane.org> <20130823111822.49cba700@pitrou.net> <CADiSq7dhzpTWMw7gcBRrzZot7jaS=Aj=aukq0+jh_cK2+F_kGQ@mail.gmail.com> <CADiSq7dBruZiWhObnjKbcneji9+ziXmyRpg1+72a8YeMWbyO3A@mail.gmail.com> <kvabt8$jfc$1@ger.gmane.org> <CADiSq7c6OjFa8zo7sMB=k2zLSmaf1w-tAwhYKSQ26T3caLyj6A@mail.gmail.com> <kvaka3$sbu$1@ger.gmane.org> <CADiSq7dzQqyH9i6g9QYWp6_ZHHUD4+4npYo6of6mCNahQ+w8kw@mail.gmail.com> <kvcr9f$4gs$1@ger.gmane.org> <CADiSq7cfNH1M29W19BW88-BBQaWEpaMkE=s09wNyT5VGOrPB+g@mail.gmail.com> <kvtfek$23p$1@ger.gmane.org> <CADiSq7d3L1+Jc5E4dWYDRUQPNDCmxqyc3GzbgXJvGMe4+OnWrQ@mail.gmail.com> <20130901150351.2f2aa44c@fsol> Message-ID: <CADiSq7e=E+WCstffBsREc6KKrBU5mbKFWiriDRt6orAKsQWnVQ@mail.gmail.com> On 1 September 2013 23:03, Antoine Pitrou <solipsis at pitrou.net> wrote: > On Sun, 1 Sep 2013 11:28:36 +1000 > Nick Coghlan <ncoghlan at gmail.com> wrote: >> * PEP 3121 with a size of "0". As above, but avoids the module state APIs >> in order to support reloading. All module state (including type >> cross-references) is stored in hidden state (e.g. an instance of a custom >> type not exposed to Python, with a reference stored on each custom type >> object defined in the module, and any module level "functions" actually >> being methods of a hidden object). Still doesn't support loading a *fresh* >> copy due to the hidden PEP 3121 module cache. > > Not sure what you mean by that: > >>>> import atexit >>>> id(atexit) > 140031896222680 >>>> import sys >>>> del sys.modules['atexit'] >>>> import atexit >>>> id(atexit) > 140031896221400 Ah, you're right - I misremembered the exact problem that broke xml.etree.ElementTree testing. PyModule_GetState is actually fine (since that pointer is hidden state on the module object), it's only PyState_GetModule that is broken when you import a second copy. So, here, when the second import happens, it breaks the original atexit module's callbacks, even though the two callback registries are properly isolated: $ ./python Python 3.4.0a1+ (default:575071257c92+, Aug 25 2013, 00:42:17) [GCC 4.7.2 20121109 (Red Hat 4.7.2-8)] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import atexit >>> atexit.register(print, "Hello World!") <built-in function print> >>> import sys >>> del sys.modules["atexit"] >>> import atexit as atexit2 >>> atexit2.register(print, "Goodbye World!") <built-in function print> >>> Goodbye World! So I think PEP 3121 is actually as good as we can get on the hidden state front, but the important point is that it is the *PyState_GetModule* API that can't handle fresh imports - the second import will always replace the first one. So anyone affected needs to find some other way of passing the state, like using bound methods of a hidden type rather than ordinary callables. If you have to interoperate with a C API that only accepts a C callback without allowing additional state arguments, you're going to have trouble. I think atexit serves as a good example, though - that _Py_PyAtExit call will *always* be destructive (even if you still have a reference to the original module), so there should be a way for the module to explicitly indicate to the import system "you can only create this module once, and then you're committed - unloading it and importing it again won't work properly due to side effects on the process state". Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From rdmurray at bitdance.com Sun Sep 1 18:28:06 2013 From: rdmurray at bitdance.com (R. David Murray) Date: Sun, 01 Sep 2013 12:28:06 -0400 Subject: [Python-Dev] Completing the email6 API changes. In-Reply-To: <878uzhdh24.fsf@uwakimon.sk.tsukuba.ac.jp> References: <20130831052136.42D4B2507F5@webabinitio.net> <87bo4echcr.fsf@uwakimon.sk.tsukuba.ac.jp> <20130831142352.600462507F5@webabinitio.net> <878uzhdh24.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <20130901162807.0C345250813@webabinitio.net> On Sun, 01 Sep 2013 00:18:59 +0900, "Stephen J. Turnbull" <stephen at xemacs.org> wrote: > R. David Murray writes: > > > Full validation is something that is currently a "future > > objective". > > I didn't mean it to be anything else. :-) > > > There's infrastructure to do it, but not all of the necessary knowledge > > has been coded in yet. > > Well, I assume you already know that there's no way that can ever > happen (at least until we abandon messaging entirely): new RFCs will > continue to be published. So it needs to be an extensible mechanism, > a "pipeline" of checks (Barry would say a "chain of rules", I think). My idea was to encode as much of the current known rules as as we have the stomach for, and to have a validation flag that you turn on if you want to check your message against those standards. But without that flag the code allows you to set arbitrary parameters and headers. As you say, an extensible mechanism for the validators is a good idea. So I take it back that the infrastructure is in place, since extensibility doesn't exist for that feature yet. --David From solipsis at pitrou.net Sun Sep 1 19:40:48 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 1 Sep 2013 19:40:48 +0200 Subject: [Python-Dev] Pre-PEP: Redesigning extension modules In-Reply-To: <CADiSq7e=E+WCstffBsREc6KKrBU5mbKFWiriDRt6orAKsQWnVQ@mail.gmail.com> References: <kv77o2$f96$1@ger.gmane.org> <20130823111822.49cba700@pitrou.net> <CADiSq7dhzpTWMw7gcBRrzZot7jaS=Aj=aukq0+jh_cK2+F_kGQ@mail.gmail.com> <CADiSq7dBruZiWhObnjKbcneji9+ziXmyRpg1+72a8YeMWbyO3A@mail.gmail.com> <kvabt8$jfc$1@ger.gmane.org> <CADiSq7c6OjFa8zo7sMB=k2zLSmaf1w-tAwhYKSQ26T3caLyj6A@mail.gmail.com> <kvaka3$sbu$1@ger.gmane.org> <CADiSq7dzQqyH9i6g9QYWp6_ZHHUD4+4npYo6of6mCNahQ+w8kw@mail.gmail.com> <kvcr9f$4gs$1@ger.gmane.org> <CADiSq7cfNH1M29W19BW88-BBQaWEpaMkE=s09wNyT5VGOrPB+g@mail.gmail.com> <kvtfek$23p$1@ger.gmane.org> <CADiSq7d3L1+Jc5E4dWYDRUQPNDCmxqyc3GzbgXJvGMe4+OnWrQ@mail.gmail.com> <20130901150351.2f2aa44c@fsol> <CADiSq7e=E+WCstffBsREc6KKrBU5mbKFWiriDRt6orAKsQWnVQ@mail.gmail.com> Message-ID: <20130901194048.0d7d1b75@fsol> On Mon, 2 Sep 2013 00:10:08 +1000 Nick Coghlan <ncoghlan at gmail.com> wrote: > > $ ./python > Python 3.4.0a1+ (default:575071257c92+, Aug 25 2013, 00:42:17) > [GCC 4.7.2 20121109 (Red Hat 4.7.2-8)] on linux > Type "help", "copyright", "credits" or "license" for more information. > >>> import atexit > >>> atexit.register(print, "Hello World!") > <built-in function print> > >>> import sys > >>> del sys.modules["atexit"] > >>> import atexit as atexit2 > >>> atexit2.register(print, "Goodbye World!") > <built-in function print> > >>> > Goodbye World! Yeah, atexit is a very particular example, because it interacts with global state by design (the main interpreter instance), and no amount of module initialization magic can prevent that :-) Speaking of which, it also doesn't work (well) with subinterpreters: http://bugs.python.org/issue18618 Regards Antoine. From tjreedy at udel.edu Sun Sep 1 22:02:33 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 01 Sep 2013 16:02:33 -0400 Subject: [Python-Dev] 'Subinterpreter' (was Re: Pre-PEP: Redesigning extension modules) In-Reply-To: <20130901194048.0d7d1b75@fsol> References: <kv77o2$f96$1@ger.gmane.org> <20130823111822.49cba700@pitrou.net> <CADiSq7dhzpTWMw7gcBRrzZot7jaS=Aj=aukq0+jh_cK2+F_kGQ@mail.gmail.com> <CADiSq7dBruZiWhObnjKbcneji9+ziXmyRpg1+72a8YeMWbyO3A@mail.gmail.com> <kvabt8$jfc$1@ger.gmane.org> <CADiSq7c6OjFa8zo7sMB=k2zLSmaf1w-tAwhYKSQ26T3caLyj6A@mail.gmail.com> <kvaka3$sbu$1@ger.gmane.org> <CADiSq7dzQqyH9i6g9QYWp6_ZHHUD4+4npYo6of6mCNahQ+w8kw@mail.gmail.com> <kvcr9f$4gs$1@ger.gmane.org> <CADiSq7cfNH1M29W19BW88-BBQaWEpaMkE=s09wNyT5VGOrPB+g@mail.gmail.com> <kvtfek$23p$1@ger.gmane.org> <CADiSq7d3L1+Jc5E4dWYDRUQPNDCmxqyc3GzbgXJvGMe4+OnWrQ@mail.gmail.com> <20130901150351.2f2aa44c@fsol> <CADiSq7e=E+WCstffBsREc6KKrBU5mbKFWiriDRt6orAKsQWnVQ@mail.gmail.com> <20130901194048.0d7d1b75@fsol> Message-ID: <l006gp$jb7$1@ger.gmane.org> > Speaking of which, it also doesn't work (well) with subinterpreters: Could someone briefly explain 'subinterpreter' or point me somewhere in the docs? It appears throughout this thread but there is no index or glossary entry. -- Terry Jan Reedy From solipsis at pitrou.net Sun Sep 1 22:06:49 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 1 Sep 2013 22:06:49 +0200 Subject: [Python-Dev] 'Subinterpreter' (was Re: Pre-PEP: Redesigning extension modules) References: <kv77o2$f96$1@ger.gmane.org> <20130823111822.49cba700@pitrou.net> <CADiSq7dhzpTWMw7gcBRrzZot7jaS=Aj=aukq0+jh_cK2+F_kGQ@mail.gmail.com> <CADiSq7dBruZiWhObnjKbcneji9+ziXmyRpg1+72a8YeMWbyO3A@mail.gmail.com> <kvabt8$jfc$1@ger.gmane.org> <CADiSq7c6OjFa8zo7sMB=k2zLSmaf1w-tAwhYKSQ26T3caLyj6A@mail.gmail.com> <kvaka3$sbu$1@ger.gmane.org> <CADiSq7dzQqyH9i6g9QYWp6_ZHHUD4+4npYo6of6mCNahQ+w8kw@mail.gmail.com> <kvcr9f$4gs$1@ger.gmane.org> <CADiSq7cfNH1M29W19BW88-BBQaWEpaMkE=s09wNyT5VGOrPB+g@mail.gmail.com> <kvtfek$23p$1@ger.gmane.org> <CADiSq7d3L1+Jc5E4dWYDRUQPNDCmxqyc3GzbgXJvGMe4+OnWrQ@mail.gmail.com> <20130901150351.2f2aa44c@fsol> <CADiSq7e=E+WCstffBsREc6KKrBU5mbKFWiriDRt6orAKsQWnVQ@mail.gmail.com> <20130901194048.0d7d1b75@fsol> <l006gp$jb7$1@ger.gmane.org> Message-ID: <20130901220649.3e9c3ab4@fsol> On Sun, 01 Sep 2013 16:02:33 -0400 Terry Reedy <tjreedy at udel.edu> wrote: > > > Speaking of which, it also doesn't work (well) with subinterpreters: > > Could someone briefly explain 'subinterpreter' or point me somewhere in > the docs? It appears throughout this thread but there is no index or > glossary entry. http://docs.python.org/dev/c-api/init.html#sub-interpreter-support Subinterpreters are a somewhat borderline feature that allows embedding applications to host multiple Python programs in a single process. A well-known example is mod_wsgi. Regards Antoine. From solipsis at pitrou.net Sun Sep 1 23:04:14 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 1 Sep 2013 23:04:14 +0200 Subject: [Python-Dev] cpython (merge 3.3 -> default): Merge fix from 3.3 into default. References: <3cSn1P710Hz7LjX@mail.python.org> Message-ID: <20130901230414.44af5426@fsol> On Sun, 1 Sep 2013 23:02:17 +0200 (CEST) tim.peters <python-checkins at python.org> wrote: > http://hg.python.org/cpython/rev/25211a22228b > changeset: 85495:25211a22228b > parent: 85493:267e09700978 > parent: 85494:8efcf3c823f9 > user: Tim Peters <tim at python.org> > date: Sun Sep 01 16:01:46 2013 -0500 > summary: > Merge fix from 3.3 into default. > > Fix issue 18889: test_sax: multiple failures on Windows desktop. > > "The fix" is to tell Mercurial that the test files are binary. > > Windows developers: to get the correct line endings in your checkout, > delete Lib\test\xmltestdata, and then "hg revert" that directory. > > Why the Windows buildbots didn't fail test_sax remains a mystery :-( Probably because they don't have the hgeol extension enabled. Regards Antoine. From stefan_ml at behnel.de Sun Sep 1 23:13:36 2013 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 01 Sep 2013 23:13:36 +0200 Subject: [Python-Dev] 'Subinterpreter' (was Re: Pre-PEP: Redesigning extension modules) In-Reply-To: <20130901220649.3e9c3ab4@fsol> References: <kv77o2$f96$1@ger.gmane.org> <20130823111822.49cba700@pitrou.net> <CADiSq7dhzpTWMw7gcBRrzZot7jaS=Aj=aukq0+jh_cK2+F_kGQ@mail.gmail.com> <CADiSq7dBruZiWhObnjKbcneji9+ziXmyRpg1+72a8YeMWbyO3A@mail.gmail.com> <kvabt8$jfc$1@ger.gmane.org> <CADiSq7c6OjFa8zo7sMB=k2zLSmaf1w-tAwhYKSQ26T3caLyj6A@mail.gmail.com> <kvaka3$sbu$1@ger.gmane.org> <CADiSq7dzQqyH9i6g9QYWp6_ZHHUD4+4npYo6of6mCNahQ+w8kw@mail.gmail.com> <kvcr9f$4gs$1@ger.gmane.org> <CADiSq7cfNH1M29W19BW88-BBQaWEpaMkE=s09wNyT5VGOrPB+g@mail.gmail.com> <kvtfek$23p$1@ger.gmane.org> <CADiSq7d3L1+Jc5E4dWYDRUQPNDCmxqyc3GzbgXJvGMe4+OnWrQ@mail.gmail.com> <20130901150351.2f2aa44c@fsol> <CADiSq7e=E+WCstffBsREc6KKrBU5mbKFWiriDRt6orAKsQWnVQ@mail.gmail.com> <20130901194048.0d7d1b75@fsol> <l006gp$jb7$1@ger.gmane.org> <20130901220649.3e9c3ab4@fsol> Message-ID: <l00alo$qlc$1@ger.gmane.org> Antoine Pitrou, 01.09.2013 22:06: > On Sun, 01 Sep 2013 16:02:33 -0400 > Terry Reedy wrote: >>> Speaking of which, it also doesn't work (well) with subinterpreters: >> >> Could someone briefly explain 'subinterpreter' or point me somewhere in >> the docs? It appears throughout this thread but there is no index or >> glossary entry. > > http://docs.python.org/dev/c-api/init.html#sub-interpreter-support > > Subinterpreters are a somewhat borderline feature that allows embedding > applications to host multiple Python programs in a single process. A > well-known example is mod_wsgi. And extension modules usually don't play well with subinterpreters because each subinterpreter requires its own separate version of the module and extension modules are rarely designed to keep their state completely local to an interpreter, let alone being prepared for having their module init function be called more than once. Stefan From rdmurray at bitdance.com Mon Sep 2 00:10:39 2013 From: rdmurray at bitdance.com (R. David Murray) Date: Sun, 01 Sep 2013 18:10:39 -0400 Subject: [Python-Dev] Completing the email6 API changes. In-Reply-To: <87bo4echcr.fsf@uwakimon.sk.tsukuba.ac.jp> References: <20130831052136.42D4B2507F5@webabinitio.net> <87bo4echcr.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <20130901221040.DC76E250818@webabinitio.net> On Sat, 31 Aug 2013 18:57:56 +0900, "Stephen J. Turnbull" <stephen at xemacs.org> wrote: > R. David Murray writes: > > > But I would certainly appreciate review from anyone so moved, since I > > haven't gotten any yet. > > I'll try to make time for a serious (but obviously partial) review by > Monday. > > I don't know if this is "serious" bikeshedding, but I have a comment > or two on the example: > > > from email.message import MIMEMessage > > from email.headerregistry import Address > > fullmsg = MIMEMessage() > > fullmsg['To'] = Address('Fo?? Bar', 'fbar at example.com') > > fullmsg['From'] = "m???? <me at example.com>" > > fullmsg['Subject'] = "j'ai un probl??me de python." > > This is very nice! *I* *love* it. > > But (sorry!) I worry that it's not obvious to "naive" users. Maybe it > would be useful to have a Message() factory which has one semantic > difference from MIMEMessage: it "requires" RFC 822-required headers > (or optionally RFC 1036 for news). Eg: > > # This message will be posted and mailed > # These would conform to the latest Draft Standards > # and be DKIM-signed > fullmsg = Message('rfc822', 'rfc1036', 'dmarc') > > I'm not sure how "required" would be implemented (perhaps through a > .validate() method). So the signature of the API suggested above is > Message(*validators, **kw). Adding new constructor arguments to the existing Message class is possible. However, given the new architecture, the more logical way to do this is to put it in the policy. So currently the idea would be for this to be spelled like this: fullmsg = Message(policy=policy.SMTP+policy.strict) Then what would happen is that when the message is serialized (be it via str(), bytes(), by passing it to smtplib.sendmail or smtplib.sendmessage, or by an explicit call to a Generator), an error would be raised if the minimum required headers are not present. As I said in an earlier message, currently there's no extensibility mechanism for the validation. If the parser recognizes a defect, whether or not an error is raised is controlled by the policy. But there's no mechanism for adding new defect checking that the parser doesn't already know about, or for issues that are not parse-time defects. (There is currently one non-parsing defect for which there is a custom control: the maximum number of headers of a given type that are allowed to be added to a Message object.) So we need some way to add additional constraints as well. Probably a list of validation functions that take a Message/MIMEPart as the argument and do a raise if they want to reject the message. The tricky bit is that currently raise_on_defect means you get an error as soon as a (parsing) defect is discovered. Likewise, if max_count is being enforced for headers, the error is raised as soon as the duplicate header is added. Generating errors early when building messages was one of or original design goals, and *only* detecting problems via validators runs counter to that unless all the validators are called every time an operation is performed that modifies a message. Maybe that would be OK, but it feels ugly. For the missing header problem, the custom solution could be to add a 'headers' argument to Message that would allow you to write: fullmsg = Message(header=( Header('Date', email.utils.localtime()), Header('To', Address('Fred', 'abc at xyz.com')), Header('From', Address('Sally, 'fgd at xyz.com')), Header('Subject', 'Foo'), ), policy=policy.SMTP+policy.Strict) This call could then immediately raise an error if not all of the required headers are present. (Header is unfortunately not a good choice of name here because we already have a 'Header' class that has a different API). Aside: I could also imagine adding a 'content' argument that would let you generate a simple text message via a single call...which means you could also extend this model to specifying the entire message in a single call, if you wrote a suitable content manager function for tuples: fullmsg = Message( policy=policy.SMTP+policy.Strict, header=( Header('Date', datetime.datetime.now()), Header('To', Address('Fred', 'abc at xyz.com')), Header('From', Address('Sally, 'fgd at xyz.com')), Header('Subject', 'Foo'), ), content=( ( 'This is the text part', ( '<p>Here is the html</p><img src="image1" \>', {'image1': b'image data'}, ), ), b'attachment data', ) But that is probably a little bit crazy...easier to just write a custom function for your application that calls the message building APIs. Well, anyway, coming back from that little tangent, it seems to me in summary that if we want to raise errors as soon as possible, we need to add custom mechanisms for the error detection as we figure out each class of error we want to handle, so that the validation gets done right away without adding too much overhead. For parsing defects, that means if you want to control which ones are raised you hook handle_defect and make your decision there (there's no way to *add* parsing defects via that hook). For duplicate headers, you hook header_max_count (or set max_count on your custom header classes). For missing headers we *could* introduce something like the above headers argument to Message, which would be required in 'strict' mode. But clearly we should also have a general "validation function" list on the policy, where all the functions would be called before serialization and have an opportunity to reject the message by raising errors. That would provide a generalized, if heavier handed, hook to use when there is no specific hook that provides extendability. For missing headers I'm inclined to use a serialization-time check rather than the Message constructor check I speculated about above. My logic is that the message isn't complete until you are ready to send it, so generating an error earlier probably isn't wanted in many cases, nor will generating it at serialization time loose you much debugging information. So I'm inclined to implement it as a default validator in the proposed list of serialization time validators. It is still probably worthwhile providing a utility function for creating a message in a single call. I think there's even an open issue in the tracker for that. But I'm inclined to postpone that until 3.5. > For MIMEMessage, I think I prefer the name "MIMEPart". To naive > users, the idea of MIMEMessages containing MIMEMessages is a bit > disconcerting, except in the case of multipart/digest, I think. Or message/rfc822: [...] try: smtplib.sendmessage(msg) except Exception as exc: errormsg = MIMEMEssage() errormsg['To'] = msg['sender'] or msg['from'] errormsg['From'] = 'robot at mydomain.org' errormsg.set_content("I'm sorry, sending failed: {}".format(exc)) errormsg.add_attachment(orig_msg, disposition='inline') smtplib.sendmessage(errormsg) (Terrible code, but you get the idea :) > > fullmsg.set_content("et la il est mont?? sur moi et il commence" > > " a m'????touffer.") > > htmlmsg = MIMEMessage() > > htmlmsg.set_content("<p>et la il est mont?? sur moi et il commence" > > " a m'??touffer.</p><img src='image1' />", > > subtype='html') > > I think I'd like to express the suite above as > > fullmsg.payload.add_alternative(...) > fullmsg.payload.add_alternative(..., subtype='html') > > This would automatically convert the MIME type of fullmsg to > 'multipart/alternative', and .payload to a list where necessary. > .set_content() would be available but it's "dangerous" (it could > replace an arbitrary multipart -- this would be useful operation to > replace it with a textual URL or external-body part). Having an attribute with methods that affect the parent object is not a natural act in Python. (Is it in OO in general?) But aside from that, I'm not sure I see the point of a two level name here. You are still talking about mutating the message model object, which is exactly what is going on in my example: if you call add_alternative on a text part, it gets turned into a multipart/alternative with the first part being the text part and the second part being the thing you just added. That's in the docs but I didn't refer to it explicitly in my example. Does that make things clearer? It is a good point about unintentionally overwriting the existing contents. That would be simple to fix: make it an error to call set_content if payload is not None. Possibly a 'clear' method would then be useful, especially since so many other Python datatypes have one. > Aside: it occurs to me that the .payload attribute (and other such > attributes) could be avoided by the device of using names prefixed by > ":" such as ":payload" as keys: "fullmsg[':payload']" since colon is > illegal in field names (cf RFC 5322). Probably I've just been writing > too much Common Lisp, though.<wink/> I think maybe so :). I'd rather write 'msg.payload' than 'msg[:payload']. So I don't have a motivation to "avoid" those attributes. (Nor would I find msg[':payload'].sommethod mutating the parent object any less surprising.) > I'm not sure whether "payload" is a better name than "content" for > that attribute. > > Now the suite > > > with open('python.jpg', 'rb') as python: > > htmlmsg.add_related(python.read(), 'image', 'jpg', cid='image1' > > disposition='inline') > > fullmsg.make_alternative() > > fullmsg.attach(htmlmsg) > > becomes just > > with open('python.jpg', 'rb') as python: > fullmsg.payload['text/html'].add_related(...) This doesn't work, though, because you could (although you usually won't) have more than one 'text/html' part in a single multipart. The reason behind the structure of my example is to avoid having to think about indexing into already existing parts. It *could* have been written: fullmsg.add_alternative(...) fullmsg.add_alternative(...., subtype='html') with open('python.jpg', 'rb') as python: fullmsg.get_payload()[1].add_related(...) But that means you have to think about payload lists and order of parts and figure out the right index to use. But perhaps some people will find that more congenial than the make_alternative() dance, and I should probably make sure that I include that alternative in the examples (that I haven't yet written for the docs). (I dislike the 'get_payload' API, by the way.) The awkwardness of the code in my example arises from the fact that a Message object is a valid thing to want to use as content (see message/rfc822 above), so I can't have the code just assume that: fullmsg.add_alternative(htmlmsg) means "make fullmsg into a multipart/alternative and attach htmlmsg", since in the logic of my API what it naturally means is "make_fullmsg into a multipart/alternative and attach a new message/rfc822 part containing htmlmsg". Thus the make/attach dance instead. However, what I really want, as I mentioned in my original proposal but have dropped from my 3.4 proposed code addition, is: with open('python.jpg', 'rb') as python: fullmsg.add_alternative( Webpage('<p>example<\p><img src=image1 \>', {'image1': python.read()} )) I dropped it from the code proposal because it seems to me that we should give the community an opportunity to experiment with the content manager interface before we decide what the best stuff is to include in the stdlib. > At this point, "fullmsg.add_related()" without the .payload attribute > would be an error, unless a "insertPart=True" keyword argument were > present. With "insertPart=True", a new top-level multipart/related > would be interposed with the existing multipart/alternative as its > first child, and the argument of add_related as the second. Maybe > that's too complicated, but I suspect it's harder for people who think > of MIME messages as trees, than for people who think of messages as > documents and don't want to hear about mimes other than Marcel > Marceau.<wink/> I *think* I see what you are suggesting, but I don't see that it is easier for the non-tree thinker than my proposal. My proposal doesn't even let you make the conversion your insertPart=True would produce. (That could indeed be a bug, so we may want a "I know what I'm doing" keyword for 'make_related' and friends). > The indexing of the .payload attribute by part type is perhaps too > smart for my own good, haven't thought carefully about it. It's > plausible, though, since a message with multiple parts of the same > type can only have one displayed -- normally that shouldn't happen. > OTOH, this wouldn't work without modification for multipart/mixed or > multipart/related. Could use Yet Another Keyword Argument, maybe. Ah, good point. But the mechanism doesn't generalize to other types (eg: you can have any number of 'image/jpg'), which would make the API quite inconsistent (you can index by 'text/html' but not by 'image/jpeg'? Wat?) > (BTW, it's really annoying when the text/plain part refers to images > etc that are attached only to the text/html part. AFAICT from RFC > 2387 it ought to be possible to put the multipart/related part at the > top so both text/html and text/plain can refer to it.) Is there a text part format that can actually refer to related parts? text/plain isn't one, as far as I know. As I said in another message, it is worth thinking about supporting more unusual structures. The tradeoff is not getting an error when you do something that is in most cases a mistake. Absent information about a reasonably common text format that actually embeds images, I'm inclined to keep the restriction. We can always remove it later, but we can't add it back later. > > with open('police-report.txt') as report: > > fullmsg.add_attachment(report.read(), filename='p????lice-report.txt', > > params=dict(wrap='flow'), headers=( > > 'X-Secret-Level: top', > > 'X-Authorization: Monty')) > > I can't find an RFC that specifies a "wrap" parameter to text/plain. > Do you mean RFC 2646 'format="flowed"' here? Yes, I did. I just didn't bother to look it up (my apologies), since the point of the example was that you could add arbitrary extra parameters. > (A "validate" method could raise a warning on unregistered parameters.) Yes, which if we want it to happen at the time the parameter is set, means we probably want a custom (extendible) mechanism for this in the policy. > > (Hmm. Looking at that I see I didn't fully fix a bug I had meant to fix: > > some of the parts have a MIME-Version header that don't need it.) > > Another reason why the top-level part should be treated differently in > the API. True. And if we add MIMEPart but keep MIMEMessage as the top level part rather than re-using the existing Message, we can make the MIMEMessage policy strict by default. --David From eliben at gmail.com Mon Sep 2 00:57:02 2013 From: eliben at gmail.com (Eli Bendersky) Date: Sun, 1 Sep 2013 15:57:02 -0700 Subject: [Python-Dev] [Python-checkins] cpython: Further reduce the cost of hash collisions by inspecting an additional nearby In-Reply-To: <3cSLzl490rzSc9@mail.python.org> References: <3cSLzl490rzSc9@mail.python.org> Message-ID: <CAF-Rda8zmjwLTVsAZ+hoT1hrUEqEFU0fXcQ=Hqgq1UFrUcM-OA@mail.gmail.com> On Sat, Aug 31, 2013 at 9:29 PM, raymond.hettinger < python-checkins at python.org> wrote: > http://hg.python.org/cpython/rev/d40a65658ff0 > changeset: 85486:d40a65658ff0 > parent: 85482:4d604f1f0219 > user: Raymond Hettinger <python at rcn.com> > date: Sat Aug 31 21:27:08 2013 -0700 > summary: > Further reduce the cost of hash collisions by inspecting an additional > nearby entry. > Hi Raymond, I'm curious about the benchmarks used to test such micro-optimizations. Do you use one of the existing Python benchmark suites or do you have a particular set of micro-benchmarks you're running this on? Eli > > files: > Objects/setobject.c | 43 +++++++++++++++++++++++++++++--- > 1 files changed, 39 insertions(+), 4 deletions(-) > > > diff --git a/Objects/setobject.c b/Objects/setobject.c > --- a/Objects/setobject.c > +++ b/Objects/setobject.c > @@ -65,10 +65,11 @@ > The initial probe index is computed as hash mod the table size. Subsequent > probe indices are computed as explained in Objects/dictobject.c. > > -To improve cache locality, each probe is done in pairs. > -After the probe is examined, an adjacent entry is then examined as well. > -The likelihood is that an adjacent entry is in the same cache line and > -can be examined more cheaply than another probe elsewhere in memory. > +To improve cache locality, each probe inspects nearby entries before > +moving on to probes elsewhere in memory. Depending on alignment and the > +size of a cache line, the nearby entries are cheaper to inspect than > +other probes elsewhere in memory. This probe strategy reduces the cost > +of hash collisions. > > All arithmetic on hash should ignore overflow. > > @@ -130,6 +131,26 @@ > if (entry->key == dummy && freeslot == NULL) > freeslot = entry; > > + entry = &table[j ^ 2]; > + if (entry->key == NULL) > + break; > + if (entry->key == key) > + return entry; > + if (entry->hash == hash && entry->key != dummy) { > + PyObject *startkey = entry->key; > + Py_INCREF(startkey); > + cmp = PyObject_RichCompareBool(startkey, key, Py_EQ); > + Py_DECREF(startkey); > + if (cmp < 0) > + return NULL; > + if (table != so->table || entry->key != startkey) > + return set_lookkey(so, key, hash); > + if (cmp > 0) > + return entry; > + } > + if (entry->key == dummy && freeslot == NULL) > + freeslot = entry; > + > i = i * 5 + perturb + 1; > j = i & mask; > perturb >>= PERTURB_SHIFT; > @@ -190,6 +211,17 @@ > if (entry->key == dummy && freeslot == NULL) > freeslot = entry; > > + entry = &table[j ^ 2]; > + if (entry->key == NULL) > + break; > + if (entry->key == key > + || (entry->hash == hash > + && entry->key != dummy > + && unicode_eq(entry->key, key))) > + return entry; > + if (entry->key == dummy && freeslot == NULL) > + freeslot = entry; > + > i = i * 5 + perturb + 1; > j = i & mask; > perturb >>= PERTURB_SHIFT; > @@ -258,6 +290,9 @@ > entry = &table[j ^ 1]; > if (entry->key == NULL) > break; > + entry = &table[j ^ 2]; > + if (entry->key == NULL) > + break; > i = i * 5 + perturb + 1; > j = i & mask; > perturb >>= PERTURB_SHIFT; > > -- > Repository URL: http://hg.python.org/cpython > > _______________________________________________ > Python-checkins mailing list > Python-checkins at python.org > http://mail.python.org/mailman/listinfo/python-checkins > > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130901/1873bb9b/attachment.html> From eliben at gmail.com Mon Sep 2 00:58:54 2013 From: eliben at gmail.com (Eli Bendersky) Date: Sun, 1 Sep 2013 15:58:54 -0700 Subject: [Python-Dev] [Python-checkins] cpython: Issue #11798: fix tests for regrtest -R : In-Reply-To: <3cSMds1ZCzzPrk@mail.python.org> References: <3cSMds1ZCzzPrk@mail.python.org> Message-ID: <CAF-Rda_5sPNkM+pOh5VB3S43t7+dh5jSQSpQ0q_bemMesgRjrA@mail.gmail.com> On Sat, Aug 31, 2013 at 9:58 PM, andrew.svetlov <python-checkins at python.org>wrote: > http://hg.python.org/cpython/rev/39781c3737f8 > changeset: 85490:39781c3737f8 > user: Andrew Svetlov <andrew.svetlov at gmail.com> > date: Sun Sep 01 07:58:41 2013 +0300 > summary: > Issue #11798: fix tests for regrtest -R : > > files: > Lib/test/regrtest.py | 5 +++++ > Lib/unittest/suite.py | 8 ++++++-- > Lib/unittest/test/test_suite.py | 8 ++++++++ > 3 files changed, 19 insertions(+), 2 deletions(-) > > > Hi Andrew, It would help if you could add more details into the commit message. This would make both post-commit reviews and future code archeology simpler. Eli > diff --git a/Lib/test/regrtest.py b/Lib/test/regrtest.py > --- a/Lib/test/regrtest.py > +++ b/Lib/test/regrtest.py > @@ -496,6 +496,8 @@ > > if ns.slaveargs is not None: > args, kwargs = json.loads(ns.slaveargs) > + if kwargs.get('huntrleaks'): > + unittest.BaseTestSuite._cleanup = False > try: > result = runtest(*args, **kwargs) > except KeyboardInterrupt: > @@ -528,6 +530,9 @@ > #gc.set_debug(gc.DEBUG_SAVEALL) > found_garbage = [] > > + if ns.huntrleaks: > + unittest.BaseTestSuite._cleanup = False > + > if ns.single: > filename = os.path.join(TEMPDIR, 'pynexttest') > try: > diff --git a/Lib/unittest/suite.py b/Lib/unittest/suite.py > --- a/Lib/unittest/suite.py > +++ b/Lib/unittest/suite.py > @@ -16,6 +16,8 @@ > class BaseTestSuite(object): > """A simple test suite that doesn't provide class or module shared > fixtures. > """ > + _cleanup = True > + > def __init__(self, tests=()): > self._tests = [] > self.addTests(tests) > @@ -61,7 +63,8 @@ > if result.shouldStop: > break > test(result) > - self._removeTestAtIndex(index) > + if self._cleanup: > + self._removeTestAtIndex(index) > return result > > def _removeTestAtIndex(self, index): > @@ -115,7 +118,8 @@ > else: > test.debug() > > - self._removeTestAtIndex(index) > + if self._cleanup: > + self._removeTestAtIndex(index) > > if topLevel: > self._tearDownPreviousClass(None, result) > diff --git a/Lib/unittest/test/test_suite.py > b/Lib/unittest/test/test_suite.py > --- a/Lib/unittest/test/test_suite.py > +++ b/Lib/unittest/test/test_suite.py > @@ -303,6 +303,9 @@ > suite.run(unittest.TestResult()) > > def test_remove_test_at_index(self): > + if not unittest.BaseTestSuite._cleanup: > + raise unittest.SkipTest("Suite cleanup is disabled") > + > suite = unittest.TestSuite() > > suite._tests = [1, 2, 3] > @@ -311,6 +314,9 @@ > self.assertEqual([1, None, 3], suite._tests) > > def test_remove_test_at_index_not_indexable(self): > + if not unittest.BaseTestSuite._cleanup: > + raise unittest.SkipTest("Suite cleanup is disabled") > + > suite = unittest.TestSuite() > suite._tests = None > > @@ -318,6 +324,8 @@ > suite._removeTestAtIndex(2) > > def assert_garbage_collect_test_after_run(self, TestSuiteClass): > + if not unittest.BaseTestSuite._cleanup: > + raise unittest.SkipTest("Suite cleanup is disabled") > > class Foo(unittest.TestCase): > def test_nothing(self): > > -- > Repository URL: http://hg.python.org/cpython > > _______________________________________________ > Python-checkins mailing list > Python-checkins at python.org > http://mail.python.org/mailman/listinfo/python-checkins > > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130901/71a4d40b/attachment.html> From eliben at gmail.com Mon Sep 2 00:59:12 2013 From: eliben at gmail.com (Eli Bendersky) Date: Sun, 1 Sep 2013 15:59:12 -0700 Subject: [Python-Dev] [Python-checkins] cpython: Update whatsnew/3.4.rst wrt. the socket constants switch to IntEnum In-Reply-To: <522281B1.50004@udel.edu> References: <3cSBmf31XfzPkT@mail.python.org> <522281B1.50004@udel.edu> Message-ID: <CAF-Rda8ho7KwWrQSSRxe6THq9zTzwQWCbc+jDg=TTE=NOUq=Ug@mail.gmail.com> On Sat, Aug 31, 2013 at 4:52 PM, Terry Reedy <tjreedy at udel.edu> wrote: > On 8/31/2013 6:19 PM, eli.bendersky wrote: > >> http://hg.python.org/cpython/**rev/4d604f1f0219<http://hg.python.org/cpython/rev/4d604f1f0219> >> changeset: 85482:4d604f1f0219 >> user: Eli Bendersky <eliben at gmail.com> >> date: Sat Aug 31 15:18:48 2013 -0700 >> summary: >> Update whatsnew/3.4.rst wrt. the socket constants switch to IntEnum >> >> [issue #18730] >> > > > Wrong issue number I think. > _______________________________________________ > Oops, yes. Sorry about that. Eli -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130901/2c77d9d2/attachment.html> From v+python at g.nevcal.com Mon Sep 2 01:08:18 2013 From: v+python at g.nevcal.com (Glenn Linderman) Date: Sun, 01 Sep 2013 16:08:18 -0700 Subject: [Python-Dev] Completing the email6 API changes. In-Reply-To: <20130901221040.DC76E250818@webabinitio.net> References: <20130831052136.42D4B2507F5@webabinitio.net> <87bo4echcr.fsf@uwakimon.sk.tsukuba.ac.jp> <20130901221040.DC76E250818@webabinitio.net> Message-ID: <5223C8E2.2000006@g.nevcal.com> On 9/1/2013 3:10 PM, R. David Murray wrote: > This doesn't work, though, because you could (although you usually > won't) have more than one 'text/html' part in a single multipart. I was traveling and your original message is still unread in my queue of "things to look at later" :( I haven't caught up with old stuff yet, but am trying to stay current on current stuff... The quoted issue was mentioned in another message in this thread, though in different terms. I recall being surprised when first seeing messages generated by Apple Mail software, that are multipart/related, having a sequence of intermixed text/plain and image/jpeg parts. This is apparently how Apple Mail generates messages that have inline pictures, without resorting to use of HTML mail. Other email clients handle this relatively better or worse, depending on the expectations of their authors! Several of them treat all the parts after the initial text/html part as attachments; some of them display inline attachments if they are text/html or image/jpeg and others do not. I can't say for sure if there are other ways they are treated; I rather imagine that Apple Mail displays the whole message with interspersed pictures quite effectively, without annoying the user with attachment "markup", but I'm not an Apple Mail user so I couldn't say for sure. You should, of course, ensure that it is possible to create such a message. Whether Apple Mail does that with other embedded image/* formats, or with other text/* formats, or other non-image, non-text formats, I couldn't say. I did attempt to determine if it was non-standard usage: it is certainly non-common usage, but I found nothing in the email/MIME RFCs that precludes such usage. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130901/d4f7c39c/attachment-0001.html> From tjreedy at udel.edu Mon Sep 2 01:25:31 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 01 Sep 2013 19:25:31 -0400 Subject: [Python-Dev] cpython (merge 3.3 -> default): Merge fix from 3.3 into default. In-Reply-To: <20130901230414.44af5426@fsol> References: <3cSn1P710Hz7LjX@mail.python.org> <20130901230414.44af5426@fsol> Message-ID: <l00ida$1v5$1@ger.gmane.org> On 9/1/2013 5:04 PM, Antoine Pitrou wrote: > On Sun, 1 Sep 2013 23:02:17 +0200 (CEST) > tim.peters <python-checkins at python.org> wrote: >> Windows developers: to get the correct line endings in your checkout, >> delete Lib\test\xmltestdata, and then "hg revert" that directory. Or, in Tortoisehg Workbenck, select the four Working Directory patch entries, right click, and revert. >> Why the Windows buildbots didn't fail test_sax remains a mystery :-( > > Probably because they don't have the hgeol extension enabled. Since the tests also failed on installed Python, it seems that the .msi installer is created in a repository with the extension enabled. If so, I think that the buildbots should have the extension enabled also, so that they are testing Python as it will be distributed. Or we could stop supporting Notepad and change what we distribute ;-). -- Terry Jan Reedy From tjreedy at udel.edu Mon Sep 2 01:38:58 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 01 Sep 2013 19:38:58 -0400 Subject: [Python-Dev] 'Subinterpreter' (was Re: Pre-PEP: Redesigning extension modules) In-Reply-To: <l00alo$qlc$1@ger.gmane.org> References: <kv77o2$f96$1@ger.gmane.org> <20130823111822.49cba700@pitrou.net> <CADiSq7dhzpTWMw7gcBRrzZot7jaS=Aj=aukq0+jh_cK2+F_kGQ@mail.gmail.com> <CADiSq7dBruZiWhObnjKbcneji9+ziXmyRpg1+72a8YeMWbyO3A@mail.gmail.com> <kvabt8$jfc$1@ger.gmane.org> <CADiSq7c6OjFa8zo7sMB=k2zLSmaf1w-tAwhYKSQ26T3caLyj6A@mail.gmail.com> <kvaka3$sbu$1@ger.gmane.org> <CADiSq7dzQqyH9i6g9QYWp6_ZHHUD4+4npYo6of6mCNahQ+w8kw@mail.gmail.com> <kvcr9f$4gs$1@ger.gmane.org> <CADiSq7cfNH1M29W19BW88-BBQaWEpaMkE=s09wNyT5VGOrPB+g@mail.gmail.com> <kvtfek$23p$1@ger.gmane.org> <CADiSq7d3L1+Jc5E4dWYDRUQPNDCmxqyc3GzbgXJvGMe4+OnWrQ@mail.gmail.com> <20130901150351.2f2aa44c@fsol> <CADiSq7e=E+WCstffBsREc6KKrBU5mbKFWiriDRt6orAKsQWnVQ@mail.gmail.com> <20130901194048.0d7d1b75@fsol> <l006gp$jb7$1@ger.gmane.org> <20130901220649.3e9c3ab4@fsol> <l00alo$qlc$1@ger.gmane.org> Message-ID: <l00j6i$97n$1@ger.gmane.org> On 9/1/2013 5:13 PM, Stefan Behnel wrote: > Antoine Pitrou, 01.09.2013 22:06: >> On Sun, 01 Sep 2013 16:02:33 -0400 >> Terry Reedy wrote: >>>> Speaking of which, it also doesn't work (well) with subinterpreters: >>> >>> Could someone briefly explain 'subinterpreter' or point me somewhere in >>> the docs? It appears throughout this thread but there is no index or >>> glossary entry. >> >> http://docs.python.org/dev/c-api/init.html#sub-interpreter-support So cpython specific. >> Subinterpreters are a somewhat borderline feature that allows embedding >> applications to host multiple Python programs in a single process. A >> well-known example is mod_wsgi. Thank you for both the link *and* the explanatory example, which just what I needed to make the past discussion more intelligible. I imagine that wsgi uses a sub-interpreter for each user connection. > And extension modules usually don't play well with subinterpreters because > each subinterpreter requires its own separate version of the module and > extension modules are rarely designed to keep their state completely local > to an interpreter, let alone being prepared for having their module init > function be called more than once. I can see now why this is a bit of a 'hair-puller';-). -- Terry Jan Reedy From tjreedy at udel.edu Mon Sep 2 01:42:10 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 01 Sep 2013 19:42:10 -0400 Subject: [Python-Dev] [Python-checkins] cpython: Update whatsnew/3.4.rst wrt. the socket constants switch to IntEnum In-Reply-To: <CAF-Rda8ho7KwWrQSSRxe6THq9zTzwQWCbc+jDg=TTE=NOUq=Ug@mail.gmail.com> References: <3cSBmf31XfzPkT@mail.python.org> <522281B1.50004@udel.edu> <CAF-Rda8ho7KwWrQSSRxe6THq9zTzwQWCbc+jDg=TTE=NOUq=Ug@mail.gmail.com> Message-ID: <l00jch$97n$2@ger.gmane.org> On 9/1/2013 6:59 PM, Eli Bendersky wrote: > > > > On Sat, Aug 31, 2013 at 4:52 PM, Terry Reedy <tjreedy at udel.edu > <mailto:tjreedy at udel.edu>> wrote: > > On 8/31/2013 6:19 PM, eli.bendersky wrote: > [issue #18730] > Wrong issue number I think. > _______________________________________________ > > > Oops, yes. Sorry about that. I unlinked it from 18730. -- Terry Jan Reedy From eliben at gmail.com Mon Sep 2 03:02:30 2013 From: eliben at gmail.com (Eli Bendersky) Date: Sun, 1 Sep 2013 18:02:30 -0700 Subject: [Python-Dev] SEEK_* constants in io and os Message-ID: <CAF-Rda_2j=HTDJ5nEdKWxX1mCmbHLgUNe_zyDYXzGHE6-LUBmg@mail.gmail.com> Hello, I was looking at the possibility of replacing the SEEK_* constants by IntEnums, and the first thing that catches attention is that these constants are defined in both Lib/os.py and Lib/io.py; both places also recently started supporting SEEK_HOLE and SEEK_DATA (though here io refers to os.SEEK_HOLE and os.SEEK_DATA). Additional data points: other modules take these constants as arguments - * mmap: directs to use os.SEEK_* * chunk and fcntk: spell out the numeric values. os seems to import io in some functions; can this be done always? If yes, we can just define the constants once and os.SEEK_* will alias io.SEEK_*? The other way (io taking from os) is also a possibility (maybe the preferred one because io already refers to os.SEEK_HOLE/DATA, at least in the documentation). Any ideas and suggestions are welcome, Eli -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130901/e9a9c8cb/attachment.html> From db3l.net at gmail.com Mon Sep 2 03:08:57 2013 From: db3l.net at gmail.com (David Bolen) Date: Sun, 01 Sep 2013 21:08:57 -0400 Subject: [Python-Dev] cpython (merge 3.3 -> default): Merge fix from 3.3 into default. References: <3cSn1P710Hz7LjX@mail.python.org> <20130901230414.44af5426@fsol> <l00ida$1v5$1@ger.gmane.org> Message-ID: <m2sixouj12.fsf@valheru.db3l.homeip.net> Terry Reedy <tjreedy at udel.edu> writes: > On 9/1/2013 5:04 PM, Antoine Pitrou wrote: >> Probably because they don't have the hgeol extension enabled. Yes, I believe that's correct, at least for my Windows buildbots. > Since the tests also failed on installed Python, it seems that the > .msi installer is created in a repository with the extension > enabled. If so, I think that the buildbots should have the extension > enabled also, so that they are testing Python as it will be > distributed. If it should be enabled, I believe it will have to be done globally, so affecting all buildbot branches (not that it sounds like that would be an issue). The individual branch repositories are physically removed at times (such as during a failure to update, which escalates to a removal followed by full clone, as well as I believe during a custom build), so I don't believe persistent local changes to the repository's own hgrc are possible in a reliable way. In the event we did want per-repository control, then I think we'd need to have some sort of additional script that ran as part of the buildbot build process to ensure the right contents for the local repository hgrc following a clone, or just prior to the update step. -- David From andrew.svetlov at gmail.com Mon Sep 2 04:37:20 2013 From: andrew.svetlov at gmail.com (Andrew Svetlov) Date: Mon, 2 Sep 2013 05:37:20 +0300 Subject: [Python-Dev] [Python-checkins] cpython: Issue #11798: fix tests for regrtest -R : In-Reply-To: <CAF-Rda_5sPNkM+pOh5VB3S43t7+dh5jSQSpQ0q_bemMesgRjrA@mail.gmail.com> References: <3cSMds1ZCzzPrk@mail.python.org> <CAF-Rda_5sPNkM+pOh5VB3S43t7+dh5jSQSpQ0q_bemMesgRjrA@mail.gmail.com> Message-ID: <CAL3CFcUoq2ihwQ4GEOopCsVNC29Dyvu7wx8ofhuBcJa-fUK=oQ@mail.gmail.com> regrtest -R runs test suites several times. That's why test cleanup should be disabled for this case. Details discussed in issue. I'll do more expressive commit messages next time. Thanks. On Mon, Sep 2, 2013 at 1:58 AM, Eli Bendersky <eliben at gmail.com> wrote: > > > > On Sat, Aug 31, 2013 at 9:58 PM, andrew.svetlov <python-checkins at python.org> > wrote: >> >> http://hg.python.org/cpython/rev/39781c3737f8 >> changeset: 85490:39781c3737f8 >> user: Andrew Svetlov <andrew.svetlov at gmail.com> >> date: Sun Sep 01 07:58:41 2013 +0300 >> summary: >> Issue #11798: fix tests for regrtest -R : >> >> files: >> Lib/test/regrtest.py | 5 +++++ >> Lib/unittest/suite.py | 8 ++++++-- >> Lib/unittest/test/test_suite.py | 8 ++++++++ >> 3 files changed, 19 insertions(+), 2 deletions(-) >> >> > > Hi Andrew, > > It would help if you could add more details into the commit message. This > would make both post-commit reviews and future code archeology simpler. > > Eli > > >> >> diff --git a/Lib/test/regrtest.py b/Lib/test/regrtest.py >> --- a/Lib/test/regrtest.py >> +++ b/Lib/test/regrtest.py >> @@ -496,6 +496,8 @@ >> >> if ns.slaveargs is not None: >> args, kwargs = json.loads(ns.slaveargs) >> + if kwargs.get('huntrleaks'): >> + unittest.BaseTestSuite._cleanup = False >> try: >> result = runtest(*args, **kwargs) >> except KeyboardInterrupt: >> @@ -528,6 +530,9 @@ >> #gc.set_debug(gc.DEBUG_SAVEALL) >> found_garbage = [] >> >> + if ns.huntrleaks: >> + unittest.BaseTestSuite._cleanup = False >> + >> if ns.single: >> filename = os.path.join(TEMPDIR, 'pynexttest') >> try: >> diff --git a/Lib/unittest/suite.py b/Lib/unittest/suite.py >> --- a/Lib/unittest/suite.py >> +++ b/Lib/unittest/suite.py >> @@ -16,6 +16,8 @@ >> class BaseTestSuite(object): >> """A simple test suite that doesn't provide class or module shared >> fixtures. >> """ >> + _cleanup = True >> + >> def __init__(self, tests=()): >> self._tests = [] >> self.addTests(tests) >> @@ -61,7 +63,8 @@ >> if result.shouldStop: >> break >> test(result) >> - self._removeTestAtIndex(index) >> + if self._cleanup: >> + self._removeTestAtIndex(index) >> return result >> >> def _removeTestAtIndex(self, index): >> @@ -115,7 +118,8 @@ >> else: >> test.debug() >> >> - self._removeTestAtIndex(index) >> + if self._cleanup: >> + self._removeTestAtIndex(index) >> >> if topLevel: >> self._tearDownPreviousClass(None, result) >> diff --git a/Lib/unittest/test/test_suite.py >> b/Lib/unittest/test/test_suite.py >> --- a/Lib/unittest/test/test_suite.py >> +++ b/Lib/unittest/test/test_suite.py >> @@ -303,6 +303,9 @@ >> suite.run(unittest.TestResult()) >> >> def test_remove_test_at_index(self): >> + if not unittest.BaseTestSuite._cleanup: >> + raise unittest.SkipTest("Suite cleanup is disabled") >> + >> suite = unittest.TestSuite() >> >> suite._tests = [1, 2, 3] >> @@ -311,6 +314,9 @@ >> self.assertEqual([1, None, 3], suite._tests) >> >> def test_remove_test_at_index_not_indexable(self): >> + if not unittest.BaseTestSuite._cleanup: >> + raise unittest.SkipTest("Suite cleanup is disabled") >> + >> suite = unittest.TestSuite() >> suite._tests = None >> >> @@ -318,6 +324,8 @@ >> suite._removeTestAtIndex(2) >> >> def assert_garbage_collect_test_after_run(self, TestSuiteClass): >> + if not unittest.BaseTestSuite._cleanup: >> + raise unittest.SkipTest("Suite cleanup is disabled") >> >> class Foo(unittest.TestCase): >> def test_nothing(self): >> >> -- >> Repository URL: http://hg.python.org/cpython >> >> _______________________________________________ >> Python-checkins mailing list >> Python-checkins at python.org >> http://mail.python.org/mailman/listinfo/python-checkins >> > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/andrew.svetlov%40gmail.com > -- Thanks, Andrew Svetlov From stephen at xemacs.org Mon Sep 2 05:03:40 2013 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Mon, 02 Sep 2013 12:03:40 +0900 Subject: [Python-Dev] Completing the email6 API changes. In-Reply-To: <5223C8E2.2000006@g.nevcal.com> References: <20130831052136.42D4B2507F5@webabinitio.net> <87bo4echcr.fsf@uwakimon.sk.tsukuba.ac.jp> <20130901221040.DC76E250818@webabinitio.net> <5223C8E2.2000006@g.nevcal.com> Message-ID: <874na3diwj.fsf@uwakimon.sk.tsukuba.ac.jp> This is getting off-topic IMO; we should probably take this thread to email-sig. Glenn Linderman writes: > I recall being surprised when first seeing messages generated by > Apple Mail software, that are multipart/related, having a sequence > of intermixed text/plain and image/jpeg parts. This is apparently > how Apple Mail generates messages that have inline pictures, > without resorting to use of HTML mail. (Are you sure you mean "text/plain" above? I've not seen this form of message. And you mention only "text/html" below.) This practice (like my suggestion) is based on the conjecture that MUAs that implement multipart/related will treat it as multipart/mixed if the "main" subpart isn't known to implement links to external entities. > Other email clients handle this relatively better or worse, > depending on the expectations of their authors! Sure. After all, this is a world in which some MUAs have a long history of happily executing virus executables. > I did attempt to determine if it was non-standard usage: it is > certainly non-common usage, but I found nothing in the email/MIME > RFCs that precludes such usage. Clearly RFCs 2046 and 2387 envision a fallback to multipart/mixed, but are silent on how to do it for MUAs that implement multipart/related. RFC 2387 says: MIME User Agents that do recognize Multipart/Related entities but are unable to process the given type should give the user the option of suppressing the entire Multipart/Related body part shall be. [...] Handling Multipart/Related differs [from handling of existing composite subtypes] in that processing cannot be reduced to handling the individual entities. I think that the sane policy is that when processing multipart/related internally, the MUA should treat the whole as multipart/mixed, unless it knows how links are implemented in the "start" part. But the RFC doesn't say that. > Several of them treat all the parts after the initial text/html > part as attachments; They don't implement RFC 2387 (promoted to draft standard in 1998, following two others, the earlier being RFC 1872 from 1995). Too bad for their users. But what I'm worried about is a different issue, which is how to ensure that multipart/alternative messages present all relevant content entities in both presentations. For example, the following hypothetical structure is efficient: multipart/alternative text/plain multipart/related text/html application/x-opentype-font because the text/plain can't use the font. But this multipart/alternative text/plain multipart/related text/html image/png image/png often cost the text/plain receiver a view of the images, and I don't see any way to distinguish the two cases. (The images might be character glyphs, for example, effectively a "poor man's font".) OTOH, if the message is structured multipart/related multipart/alternative text/plain text/html image/png image/png the receiver can infer that the images are related to both text/* parts and DTRT for each. Steve From v+python at g.nevcal.com Mon Sep 2 07:55:58 2013 From: v+python at g.nevcal.com (Glenn Linderman) Date: Sun, 01 Sep 2013 22:55:58 -0700 Subject: [Python-Dev] Completing the email6 API changes. In-Reply-To: <874na3diwj.fsf@uwakimon.sk.tsukuba.ac.jp> References: <20130831052136.42D4B2507F5@webabinitio.net> <87bo4echcr.fsf@uwakimon.sk.tsukuba.ac.jp> <20130901221040.DC76E250818@webabinitio.net> <5223C8E2.2000006@g.nevcal.com> <874na3diwj.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <5224286E.4040409@g.nevcal.com> On 9/1/2013 8:03 PM, Stephen J. Turnbull wrote: > This is getting off-topic IMO; we should probably take this thread to > email-sig. Probably, but you didn't :) > Glenn Linderman writes: > > > I recall being surprised when first seeing messages generated by > > Apple Mail software, that are multipart/related, having a sequence > > of intermixed text/plain and image/jpeg parts. This is apparently > > how Apple Mail generates messages that have inline pictures, > > without resorting to use of HTML mail. > > (Are you sure you mean "text/plain" above? I've not seen this form of > message. And you mention only "text/html" below.) Yes, I'm sure it was text/plain. I may be able to access the archived discussion from a non-Python mailing list about it, to verify, if that becomes important. But now that you mention mulitpart/mixed, I'm not sure if it was multipart/related or mulitpart/mixed for the grouping MIME part. Perhaps someone with Apple Mail could produce one... probably by composing a message as text/plain, and dragging in a picture or two. The other references to text/html was in error. > This practice (like my suggestion) is based on the conjecture that > MUAs that implement multipart/related will treat it as multipart/mixed > if the "main" subpart isn't known to implement links to external > entities. > > > Other email clients handle this relatively better or worse, > > depending on the expectations of their authors! > > Sure. After all, this is a world in which some MUAs have a long > history of happily executing virus executables. > > > I did attempt to determine if it was non-standard usage: it is > > certainly non-common usage, but I found nothing in the email/MIME > > RFCs that precludes such usage. > > Clearly RFCs 2046 and 2387 envision a fallback to multipart/mixed, but > are silent on how to do it for MUAs that implement multipart/related. > RFC 2387 says: > > MIME User Agents that do recognize Multipart/Related entities but > are unable to process the given type should give the user the > option of suppressing the entire Multipart/Related body part shall > be. [...] Handling Multipart/Related differs [from handling of > existing composite subtypes] in that processing cannot be reduced > to handling the individual entities. > > I think that the sane policy is that when processing multipart/related > internally, the MUA should treat the whole as multipart/mixed, unless > it knows how links are implemented in the "start" part. But the RFC > doesn't say that. > > > Several of them treat all the parts after the initial text/html > > part as attachments; > > They don't implement RFC 2387 (promoted to draft standard in 1998, > following two others, the earlier being RFC 1872 from 1995). Too bad > for their users. Correct... but the MUA receiving the Apple Mail message I was talking about being a text-mostly MUA, it is probably a reasonable method of handling them. > But what I'm worried about is a different issue, > which is how to ensure that multipart/alternative messages present all > relevant content entities in both presentations. For example, the > following hypothetical structure is efficient: > > multipart/alternative > text/plain > multipart/related > text/html > application/x-opentype-font > > because the text/plain can't use the font. But this > > multipart/alternative > text/plain > multipart/related > text/html > image/png > image/png > > often cost the text/plain receiver a view of the images, and I don't > see any way to distinguish the two cases. (The images might be > character glyphs, for example, effectively a "poor man's font".) Yes, that issue is handled by some text MUA by showing the image/png (or anything in such a position) as attachments. Again, being text-mostly, that might be a reasonable way of handling them. Perhaps the standard says they should be ignored, when displaying text/plain alternative. > OTOH, if the message is structured > > multipart/related > multipart/alternative > text/plain > text/html > image/png > image/png > > the receiver can infer that the images are related to both text/* > parts and DTRT for each. > With the images being treated as attachments. Or is there a syntax to allow the text/html to embed the images and the text/plain to see them as attachments? I think the text/html wants to refer to things within its containing multipart/related, but am not sure if that allows the intervening multipart/alternative. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130901/874f57b8/attachment.html> From stephen at xemacs.org Mon Sep 2 09:06:53 2013 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Mon, 02 Sep 2013 16:06:53 +0900 Subject: [Python-Dev] Completing the email6 API changes. In-Reply-To: <5224286E.4040409@g.nevcal.com> References: <20130831052136.42D4B2507F5@webabinitio.net> <87bo4echcr.fsf@uwakimon.sk.tsukuba.ac.jp> <20130901221040.DC76E250818@webabinitio.net> <5223C8E2.2000006@g.nevcal.com> <874na3diwj.fsf@uwakimon.sk.tsukuba.ac.jp> <5224286E.4040409@g.nevcal.com> Message-ID: <87y57fbt2q.fsf@uwakimon.sk.tsukuba.ac.jp> >>>>> Glenn writes: > >>>>> Steve writes: >> OTOH, if the message is structured >> >> multipart/related >> multipart/alternative >> text/plain >> text/html >> image/png >> image/png >> >> the receiver can infer that the images are related to both text/* >> parts and DTRT for each. >With the images being treated as attachments. Or is there a syntax to >allow the text/html to embed the images and the text/plain to see them >as attachments? I believe the above is that syntax. But the standard doesn't say anything about this. The standard for multipart/alternative is RFC 2046, which doesn't know about multipart/related. RFC 2387 doesn't update RFC 2046, so it doesn't say anything about multipart/alternative within multipart/related, either. >I think the text/html wants to refer to things within its containing >multipart/related, but am not sure if that allows the intervening >multipart/alternative. I don't see why not. But it would depend on the implementations, which we'll have to test before recommending the structure I (theoretically :-) prefer.e From solipsis at pitrou.net Mon Sep 2 10:16:42 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 2 Sep 2013 10:16:42 +0200 Subject: [Python-Dev] Pre-PEP: Redesigning extension modules References: <kv77o2$f96$1@ger.gmane.org> <CALeMXf5oxi0oX13zSfHwgVi1eXF_z_coY23MfBP-YOK8WVkWsg@mail.gmail.com> <kvctoj$qsv$1@ger.gmane.org> <kvstdj$jo0$1@ger.gmane.org> <CADiSq7c5xtEcPBXGPWqQosfgGcb2n+2AnHiAWFoATeYfQTUcEQ@mail.gmail.com> Message-ID: <20130902101642.04b69588@pitrou.net> Le Sun, 1 Sep 2013 02:19:48 +1000, Nick Coghlan <ncoghlan at gmail.com> a ?crit : > On 1 Sep 2013 00:10, "Stefan Behnel" <stefan_ml at behnel.de> wrote: > > > > *bump* > > > > Does this sound like a viable solution? > > This isn't likely to progress until we have Eric's PEP 451 to a point > where it's ready for python-dev discussion and pronouncement. > > However, the revised loader API is being designed to allow for the > loader returning arbitrary objects, so something along these lines > should work. There will likely be some adjustments to the API > signature to allow extension modules to optionally support reloading > if they so desire. I think the biggest challenge here is to propose an API that's simple and easy to use (i.e. that doesn't make extension module writing more complicated than it currently is). The basic concept of putting custom module objects in sys.modules is sound, IMHO. As for "extension module as a wrapper", though, it shounds like the kind of complication I would personally prefer to stay away from. Also, it would make extension modules less like Python modules, rather than more. Regards Antoine. > > Cheers, > Nick. > > > > > Stefan > > > > > > Stefan Behnel, 25.08.2013 14:36: > > > Hi, > > > > > > thanks for bringing this up. It clearly shows that there is more > > > to this problem than I initially thought. > > > > > > Let me just add one idea that your post gave me. > > > > > > PJ Eby, 25.08.2013 06:12: > > >> My "Importing" package offers lazy imports by creating module > > >> objects in sys.modules that are a subtype of ModuleType, and use > > >> a __getattribute__ hook so that trying to use them fires off a > > >> reload() of the module. > > > > > > I wonder if this wouldn't be an approach to fix the reloading > > > problem in general. What if extension module loading, at least > > > with the new scheme, didn't return the module object itself and > > > put it into sys.modules but created a wrapper that redirects its > > > __getattr__ and __setattr__ to the actual module object? That > > > would have a tiny performance impact on attribute access, but I'd > > > expect that to be negligible given that the > usual > > > reason for the extension module to exist is that it does > > > non-trivial > stuff > > > in whatever its API provides. Reloading could then really create a > > > completely new module object and replace the reference inside of > > > the > wrapper. > > > > > > That way, code that currently uses "from extmodule import xyz" > > > would continue to see the original version of the module as of > > > the time of its import, and code that just did "import extmodule" > > > and then used > attribute > > > access at need would always see the current content of the module > > > as it > was > > > last loaded. I think that, together with keeping module global > > > state in > the > > > module object itself, would nicely fix both cases. > > > > > > Stefan > > > > > > _______________________________________________ > > Python-Dev mailing list > > Python-Dev at python.org > > http://mail.python.org/mailman/listinfo/python-dev > > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com > From solipsis at pitrou.net Mon Sep 2 10:18:17 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 2 Sep 2013 10:18:17 +0200 Subject: [Python-Dev] Add a new tracemalloc module to trace memory allocations References: <CAMpsgwaa3pABNrMSw2S7-Og111u8DdXK6dYnWVJzS4TAKbFNEQ@mail.gmail.com> <CAMpsgwatDdFQg4q=b3s7W6BZ+WQmpo=KTeJZd7ESbuGG+RzaKA@mail.gmail.com> <CAGE7PNJ9DrKQ8Zb-E-F7QPRTP=t+S8StBvm3OQgvct8539gFPg@mail.gmail.com> <CAMpsgwZBeN7W+GK6hLSHp4VYXAYgWFQBTx_-K3J7KzOMDB3H=g@mail.gmail.com> <CADiSq7dKojZ5f1bprhZZxu6DjovjWtwy-8o=-BYjYRvHZWqoFw@mail.gmail.com> Message-ID: <20130902101817.1a6d1c5d@pitrou.net> Le Sun, 1 Sep 2013 11:36:51 +1000, Nick Coghlan <ncoghlan at gmail.com> a ?crit : > +1 from me for both tracemalloc and failmalloc in the same vein as > faulthandler (and using similar API concepts to faulthandler). > > While I like the flat top level naming structure, we should clearly > document these as implementation dependent runtime services. Other > implementations may not provide them at all, and even if they do, many > details will likely be different. So, it sounds like Victor may write a new PEP soon :-) +1 from me on a flat namespace. Non-designed naming hierarchies are just a pain to work with. Regards Antoine. From solipsis at pitrou.net Mon Sep 2 10:24:37 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 2 Sep 2013 10:24:37 +0200 Subject: [Python-Dev] SEEK_* constants in io and os References: <CAF-Rda_2j=HTDJ5nEdKWxX1mCmbHLgUNe_zyDYXzGHE6-LUBmg@mail.gmail.com> Message-ID: <20130902102437.197c1708@pitrou.net> Le Sun, 1 Sep 2013 18:02:30 -0700, Eli Bendersky <eliben at gmail.com> a ?crit : > Hello, > > I was looking at the possibility of replacing the SEEK_* constants by > IntEnums, and the first thing that catches attention is that these > constants are defined in both Lib/os.py and Lib/io.py; both places > also recently started supporting SEEK_HOLE and SEEK_DATA (though here > io refers to os.SEEK_HOLE and os.SEEK_DATA). What is the runtime cost of doing so? os is a fundamental module that is imported by almost every Python program. Regards Antoine. From ncoghlan at gmail.com Mon Sep 2 15:02:24 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 2 Sep 2013 23:02:24 +1000 Subject: [Python-Dev] Pre-PEP: Redesigning extension modules In-Reply-To: <20130902101642.04b69588@pitrou.net> References: <kv77o2$f96$1@ger.gmane.org> <CALeMXf5oxi0oX13zSfHwgVi1eXF_z_coY23MfBP-YOK8WVkWsg@mail.gmail.com> <kvctoj$qsv$1@ger.gmane.org> <kvstdj$jo0$1@ger.gmane.org> <CADiSq7c5xtEcPBXGPWqQosfgGcb2n+2AnHiAWFoATeYfQTUcEQ@mail.gmail.com> <20130902101642.04b69588@pitrou.net> Message-ID: <CADiSq7c+R92r17c8g3ygW0T4p=VG8LiSf5XSGuOJsFHrOtOJDg@mail.gmail.com> On 2 September 2013 18:16, Antoine Pitrou <solipsis at pitrou.net> wrote: > Le Sun, 1 Sep 2013 02:19:48 +1000, > Nick Coghlan <ncoghlan at gmail.com> a ?crit : >> On 1 Sep 2013 00:10, "Stefan Behnel" <stefan_ml at behnel.de> wrote: >> > >> > *bump* >> > >> > Does this sound like a viable solution? >> >> This isn't likely to progress until we have Eric's PEP 451 to a point >> where it's ready for python-dev discussion and pronouncement. >> >> However, the revised loader API is being designed to allow for the >> loader returning arbitrary objects, so something along these lines >> should work. There will likely be some adjustments to the API >> signature to allow extension modules to optionally support reloading >> if they so desire. > > I think the biggest challenge here is to propose an API that's simple > and easy to use (i.e. that doesn't make extension module writing more > complicated than it currently is). > > The basic concept of putting custom module objects in sys.modules is > sound, IMHO. The hook API I currently have in mind is a two step initialisation: PyImport_PrepareNAME (optional) PyImport_ExecNAME If you don't define prepare, the import system takes care of creating a module object for you, and passing it in to the exec hook. The return result from that is just an integer indicating success or failure (on failure, an exception should be set). If you *do* define the prepare hook, then it's similar to the existing init hook, but receives a PEP 451 module spec object with info about the module being imported (see PEP 451 for the draft details) and is permitted to return an arbitrary PyObject reference. The main open questions I have are how to deal with clearly indicating whether modules support in-place reloading, unloading, loading in subinterpreters and/or loading a second copy in the same interpreter. That's actually more a question for PEP 451 though, so I'll post some more detailed thoughts on that over on import-sig, which may eventually make their way into a PEP 451 draft. > As for "extension module as a wrapper", though, it shounds like the > kind of complication I would personally prefer to stay away from. Also, > it would make extension modules less like Python modules, rather than > more. Yeah, that will be allowed (since we'll probably permit returning arbitrary objects), but definitely not required. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From eliben at gmail.com Mon Sep 2 15:18:31 2013 From: eliben at gmail.com (Eli Bendersky) Date: Mon, 2 Sep 2013 06:18:31 -0700 Subject: [Python-Dev] SEEK_* constants in io and os In-Reply-To: <20130902102437.197c1708@pitrou.net> References: <CAF-Rda_2j=HTDJ5nEdKWxX1mCmbHLgUNe_zyDYXzGHE6-LUBmg@mail.gmail.com> <20130902102437.197c1708@pitrou.net> Message-ID: <CAF-Rda-_Z7QTQNf+jonLC1FqWj4pdLVk2h5r5ORTPaiUCwGauA@mail.gmail.com> On Mon, Sep 2, 2013 at 1:24 AM, Antoine Pitrou <solipsis at pitrou.net> wrote: > Le Sun, 1 Sep 2013 18:02:30 -0700, > Eli Bendersky <eliben at gmail.com> a ?crit : > > Hello, > > > > I was looking at the possibility of replacing the SEEK_* constants by > > IntEnums, and the first thing that catches attention is that these > > constants are defined in both Lib/os.py and Lib/io.py; both places > > also recently started supporting SEEK_HOLE and SEEK_DATA (though here > > io refers to os.SEEK_HOLE and os.SEEK_DATA). > > What is the runtime cost of doing so? os is a fundamental module that is > imported by almost every Python program. > Theoretically, it should be very low given that we just need to add an import and define one class. os already does a number of things in its toplevel (mostly a few imports which transitively do other things). Compounded with import caching, since this is done just once per run, doesn't seem like a problem. Empirically, I tried measuring it but I can't discern a difference with/without translating SEEK_* to enums. There's a fluctuation of ~1usec which I can't distinguish from noise. Let me know if you have a good methodology of benchmarking these things Eli -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130902/144c8345/attachment.html> From solipsis at pitrou.net Mon Sep 2 15:45:22 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 2 Sep 2013 15:45:22 +0200 Subject: [Python-Dev] SEEK_* constants in io and os References: <CAF-Rda_2j=HTDJ5nEdKWxX1mCmbHLgUNe_zyDYXzGHE6-LUBmg@mail.gmail.com> <20130902102437.197c1708@pitrou.net> <CAF-Rda-_Z7QTQNf+jonLC1FqWj4pdLVk2h5r5ORTPaiUCwGauA@mail.gmail.com> Message-ID: <20130902154522.495b1697@pitrou.net> Le Mon, 2 Sep 2013 06:18:31 -0700, Eli Bendersky <eliben at gmail.com> a ?crit : > On Mon, Sep 2, 2013 at 1:24 AM, Antoine Pitrou <solipsis at pitrou.net> > wrote: > > > Le Sun, 1 Sep 2013 18:02:30 -0700, > > Eli Bendersky <eliben at gmail.com> a ?crit : > > > Hello, > > > > > > I was looking at the possibility of replacing the SEEK_* > > > constants by IntEnums, and the first thing that catches attention > > > is that these constants are defined in both Lib/os.py and > > > Lib/io.py; both places also recently started supporting SEEK_HOLE > > > and SEEK_DATA (though here io refers to os.SEEK_HOLE and > > > os.SEEK_DATA). > > > > What is the runtime cost of doing so? os is a fundamental module > > that is imported by almost every Python program. > > > > Theoretically, it should be very low given that we just need to add an > import and define one class. os already does a number of things in its > toplevel (mostly a few imports which transitively do other things). > Compounded with import caching, since this is done just once per run, > doesn't seem like a problem. > > Empirically, I tried measuring it but I can't discern a difference > with/without translating SEEK_* to enums. There's a fluctuation of > ~1usec which I can't distinguish from noise. Let me know if you have > a good methodology of benchmarking these things How did you get that result? You have to remove to "os" from sys.modules before importing it again, otherwise "import os" will simply return the already imported module. Regards Antoine. From solipsis at pitrou.net Mon Sep 2 15:51:09 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 2 Sep 2013 15:51:09 +0200 Subject: [Python-Dev] SEEK_* constants in io and os References: <CAF-Rda_2j=HTDJ5nEdKWxX1mCmbHLgUNe_zyDYXzGHE6-LUBmg@mail.gmail.com> <20130902102437.197c1708@pitrou.net> <CAF-Rda-_Z7QTQNf+jonLC1FqWj4pdLVk2h5r5ORTPaiUCwGauA@mail.gmail.com> <20130902154522.495b1697@pitrou.net> Message-ID: <20130902155109.7da869bd@pitrou.net> Le Mon, 2 Sep 2013 15:45:22 +0200, Antoine Pitrou <solipsis at pitrou.net> a ?crit : > Le Mon, 2 Sep 2013 06:18:31 -0700, > Eli Bendersky <eliben at gmail.com> a ?crit : > > On Mon, Sep 2, 2013 at 1:24 AM, Antoine Pitrou <solipsis at pitrou.net> > > wrote: > > > > > Le Sun, 1 Sep 2013 18:02:30 -0700, > > > Eli Bendersky <eliben at gmail.com> a ?crit : > > > > Hello, > > > > > > > > I was looking at the possibility of replacing the SEEK_* > > > > constants by IntEnums, and the first thing that catches > > > > attention is that these constants are defined in both Lib/os.py > > > > and Lib/io.py; both places also recently started supporting > > > > SEEK_HOLE and SEEK_DATA (though here io refers to os.SEEK_HOLE > > > > and os.SEEK_DATA). > > > > > > What is the runtime cost of doing so? os is a fundamental module > > > that is imported by almost every Python program. > > > > > > > Theoretically, it should be very low given that we just need to add > > an import and define one class. os already does a number of things > > in its toplevel (mostly a few imports which transitively do other > > things). Compounded with import caching, since this is done just > > once per run, doesn't seem like a problem. > > > > Empirically, I tried measuring it but I can't discern a difference > > with/without translating SEEK_* to enums. There's a fluctuation of > > ~1usec which I can't distinguish from noise. Let me know if you have > > a good methodology of benchmarking these things > > How did you get that result? You have to remove to "os" from > sys.modules before importing it again, otherwise "import os" will > simply return the already imported module. Oh and remove "enum" too... Regards Antoine. From eliben at gmail.com Mon Sep 2 16:49:31 2013 From: eliben at gmail.com (Eli Bendersky) Date: Mon, 2 Sep 2013 07:49:31 -0700 Subject: [Python-Dev] SEEK_* constants in io and os In-Reply-To: <20130902155109.7da869bd@pitrou.net> References: <CAF-Rda_2j=HTDJ5nEdKWxX1mCmbHLgUNe_zyDYXzGHE6-LUBmg@mail.gmail.com> <20130902102437.197c1708@pitrou.net> <CAF-Rda-_Z7QTQNf+jonLC1FqWj4pdLVk2h5r5ORTPaiUCwGauA@mail.gmail.com> <20130902154522.495b1697@pitrou.net> <20130902155109.7da869bd@pitrou.net> Message-ID: <CAF-Rda86kyZg+ZuJcw4jff8w6LgCYo__N=VDyR_GQqdEcQ6ePg@mail.gmail.com> On Mon, Sep 2, 2013 at 6:51 AM, Antoine Pitrou <solipsis at pitrou.net> wrote: > Le Mon, 2 Sep 2013 15:45:22 +0200, > Antoine Pitrou <solipsis at pitrou.net> a ?crit : > > Le Mon, 2 Sep 2013 06:18:31 -0700, > > Eli Bendersky <eliben at gmail.com> a ?crit : > > > On Mon, Sep 2, 2013 at 1:24 AM, Antoine Pitrou <solipsis at pitrou.net> > > > wrote: > > > > > > > Le Sun, 1 Sep 2013 18:02:30 -0700, > > > > Eli Bendersky <eliben at gmail.com> a ?crit : > > > > > Hello, > > > > > > > > > > I was looking at the possibility of replacing the SEEK_* > > > > > constants by IntEnums, and the first thing that catches > > > > > attention is that these constants are defined in both Lib/os.py > > > > > and Lib/io.py; both places also recently started supporting > > > > > SEEK_HOLE and SEEK_DATA (though here io refers to os.SEEK_HOLE > > > > > and os.SEEK_DATA). > > > > > > > > What is the runtime cost of doing so? os is a fundamental module > > > > that is imported by almost every Python program. > > > > > > > > > > Theoretically, it should be very low given that we just need to add > > > an import and define one class. os already does a number of things > > > in its toplevel (mostly a few imports which transitively do other > > > things). Compounded with import caching, since this is done just > > > once per run, doesn't seem like a problem. > > > > > > Empirically, I tried measuring it but I can't discern a difference > > > with/without translating SEEK_* to enums. There's a fluctuation of > > > ~1usec which I can't distinguish from noise. Let me know if you have > > > a good methodology of benchmarking these things > > > > How did you get that result? You have to remove to "os" from > > sys.modules before importing it again, otherwise "import os" will > > simply return the already imported module. > > Oh and remove "enum" too... > Yes, now I see a 500 usec difference timed within the Python script. When timing the whole execution of Python: $ sudo nice -n -20 perf stat -r 100 python -c 'import os' It still gets lost in the noise, though. I usually get around 34 ms per run with 1 - 1.5% jitter. Since 1.5% of 34 ms is ~0.5 ms, it's difficult to distinguish the runs clearly. That 0.5 ms seems to be a one-time penalty for the importing of enum (most time goes to importing, not defining the new class based on IntEnum), no matter where/how it's used in the stdlib and the user code. Eli -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130902/77095dfd/attachment.html> From victor.stinner at gmail.com Mon Sep 2 17:48:01 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Mon, 2 Sep 2013 17:48:01 +0200 Subject: [Python-Dev] SEEK_* constants in io and os In-Reply-To: <CAF-Rda86kyZg+ZuJcw4jff8w6LgCYo__N=VDyR_GQqdEcQ6ePg@mail.gmail.com> References: <CAF-Rda_2j=HTDJ5nEdKWxX1mCmbHLgUNe_zyDYXzGHE6-LUBmg@mail.gmail.com> <20130902102437.197c1708@pitrou.net> <CAF-Rda-_Z7QTQNf+jonLC1FqWj4pdLVk2h5r5ORTPaiUCwGauA@mail.gmail.com> <20130902154522.495b1697@pitrou.net> <20130902155109.7da869bd@pitrou.net> <CAF-Rda86kyZg+ZuJcw4jff8w6LgCYo__N=VDyR_GQqdEcQ6ePg@mail.gmail.com> Message-ID: <CAMpsgwYoR9p+F5Q9rXw-UbLKV2fFyEsZARR=VKXvjSLwGv8i6A@mail.gmail.com> 2013/9/2 Eli Bendersky <eliben at gmail.com>: > Yes, now I see a 500 usec difference timed within the Python script. When > timing the whole execution of Python: (...) Can you please provide the list of imported modules by: python -c 'import sys; print(sys.modules)' For python with default options and for python with -S (no site module) options? And also with your patch? Python should be fast to write "hello world" (or "python -c pass), it's a dummy but common benchmark (to compare Python to other VM / other programming languages). Victor From eliben at gmail.com Mon Sep 2 18:02:00 2013 From: eliben at gmail.com (Eli Bendersky) Date: Mon, 2 Sep 2013 09:02:00 -0700 Subject: [Python-Dev] SEEK_* constants in io and os In-Reply-To: <CAMpsgwYoR9p+F5Q9rXw-UbLKV2fFyEsZARR=VKXvjSLwGv8i6A@mail.gmail.com> References: <CAF-Rda_2j=HTDJ5nEdKWxX1mCmbHLgUNe_zyDYXzGHE6-LUBmg@mail.gmail.com> <20130902102437.197c1708@pitrou.net> <CAF-Rda-_Z7QTQNf+jonLC1FqWj4pdLVk2h5r5ORTPaiUCwGauA@mail.gmail.com> <20130902154522.495b1697@pitrou.net> <20130902155109.7da869bd@pitrou.net> <CAF-Rda86kyZg+ZuJcw4jff8w6LgCYo__N=VDyR_GQqdEcQ6ePg@mail.gmail.com> <CAMpsgwYoR9p+F5Q9rXw-UbLKV2fFyEsZARR=VKXvjSLwGv8i6A@mail.gmail.com> Message-ID: <CAF-Rda_msMu--d-8Q8kPy8ReaJdNavKQ8keLrVQPOjMxpA41Yg@mail.gmail.com> On Mon, Sep 2, 2013 at 8:48 AM, Victor Stinner <victor.stinner at gmail.com>wrote: > 2013/9/2 Eli Bendersky <eliben at gmail.com>: > > Yes, now I see a 500 usec difference timed within the Python script. When > > timing the whole execution of Python: (...) > > Can you please provide the list of imported modules by: > python -c 'import sys; print(sys.modules)' > > For python with default options and for python with -S (no site > module) options? And also with your patch? > > Python should be fast to write "hello world" (or "python -c pass), > it's a dummy but common benchmark (to compare Python to other VM / > other programming languages). > The sorted list for both default and -S (they're identical) is http://pastebin.com/4vzSMCu7 - there are 55 entries there, including things like itertools, heapq, functools and collections. With my patch, enum is also added (which makes sense since os is in the list). So the 0.5ms increase for the 34ms runtime kind-of makes sense. Eli -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130902/d7f87272/attachment.html> From rdmurray at bitdance.com Mon Sep 2 23:40:55 2013 From: rdmurray at bitdance.com (R. David Murray) Date: Mon, 02 Sep 2013 17:40:55 -0400 Subject: [Python-Dev] Completing the email6 API changes. In-Reply-To: <87y57fbt2q.fsf@uwakimon.sk.tsukuba.ac.jp> References: <20130831052136.42D4B2507F5@webabinitio.net> <87bo4echcr.fsf@uwakimon.sk.tsukuba.ac.jp> <20130901221040.DC76E250818@webabinitio.net> <5223C8E2.2000006@g.nevcal.com> <874na3diwj.fsf@uwakimon.sk.tsukuba.ac.jp> <5224286E.4040409@g.nevcal.com> <87y57fbt2q.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <20130902214056.ADC342502EA@webabinitio.net> On Mon, 02 Sep 2013 16:06:53 +0900, "Stephen J. Turnbull" <stephen at xemacs.org> wrote: > >>>>> Glenn writes: > > >>>>> Steve writes: > > >> OTOH, if the message is structured > >> > >> multipart/related > >> multipart/alternative > >> text/plain > >> text/html > >> image/png > >> image/png > >> > >> the receiver can infer that the images are related to both text/* > >> parts and DTRT for each. > > >With the images being treated as attachments. Or is there a syntax to > >allow the text/html to embed the images and the text/plain to see them > >as attachments? > > I believe the above is that syntax. But the standard doesn't say > anything about this. The standard for multipart/alternative is RFC > 2046, which doesn't know about multipart/related. RFC 2387 doesn't > update RFC 2046, so it doesn't say anything about > multipart/alternative within multipart/related, either. > > >I think the text/html wants to refer to things within its containing > >multipart/related, but am not sure if that allows the intervening > >multipart/alternative. > > I don't see why not. But it would depend on the implementations, > which we'll have to test before recommending the structure I > (theoretically :-) prefer.e I'm still not understanding how the text/plain part *refers* to the related parts. I can understand the structure Glen found in Applemail: a series of text/plain parts interspersed with image/jpg, with all parts after the first being marked 'Contentent-Disposition: inline'. Any MUA that can display text and images *ought* to handle that correctly and produce the expected result. But that isn't what your structure above would produce. If you did: multipart/related multipart/alternative text/html text/plain image/png text/plain image/png text/plain and only referred to the png parts in the text/html part and marked all the parts as 'inline' (even though that is irrelevant in the text/html related case), an MUA that *knew* about this technique *could* display it "correctly", but an MUA that is just following the standards most likely won't. I don't see any way short of duplicating the image parts to make it likely that a typical MUA would display images for both a text/plain sequence and a text/html related part. On the other hand, my experience with MUAs is actually quite limited :) Unless there is some standard for referring to related parts in a text/plain part? I'm not aware of any, but you have much more experience with this stuff than I do. (Even text/enriched (RFC 1896) doesn't seem to have one, though of course there could be "extensions" that define both that and the font support you used as an example.) --David From v+python at g.nevcal.com Tue Sep 3 00:52:59 2013 From: v+python at g.nevcal.com (Glenn Linderman) Date: Mon, 02 Sep 2013 15:52:59 -0700 Subject: [Python-Dev] Completing the email6 API changes. In-Reply-To: <20130902214056.ADC342502EA@webabinitio.net> References: <20130831052136.42D4B2507F5@webabinitio.net> <87bo4echcr.fsf@uwakimon.sk.tsukuba.ac.jp> <20130901221040.DC76E250818@webabinitio.net> <5223C8E2.2000006@g.nevcal.com> <874na3diwj.fsf@uwakimon.sk.tsukuba.ac.jp> <5224286E.4040409@g.nevcal.com> <87y57fbt2q.fsf@uwakimon.sk.tsukuba.ac.jp> <20130902214056.ADC342502EA@webabinitio.net> Message-ID: <522516CB.1030705@g.nevcal.com> On 9/2/2013 2:40 PM, R. David Murray wrote: > I'm still not understanding how the text/plain part*refers* to the > related parts. I don't think the text/plain part can refer to the related parts, but, like you, am willing to be educated if there is a way; but while the text/html may be able to if things like cid: URIs can reach up a level in a given MUA, the text/plain would be left with the additional parts being attachments, methinks. This is less interesting than the technique Apple Mail uses, but more interesting than not even seeing the attached pictures. MUAs tend to be able to display what they produce themselves, but I have situations where they don't handle what other MUAs produce. One nice thing about this email6 toolkit might be the ability to produce, more easily than before, a variety of MIME combinations to exercise and test a variety of MUAs. While perhaps most of them have been tested with some obviously standard MIME combinations, I suspect most of them will produce strange results with combinations that are out of the ordinary. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130902/7d746b6b/attachment.html> From rdmurray at bitdance.com Tue Sep 3 03:27:15 2013 From: rdmurray at bitdance.com (R. David Murray) Date: Mon, 02 Sep 2013 21:27:15 -0400 Subject: [Python-Dev] Completing the email6 API changes. In-Reply-To: <522516CB.1030705@g.nevcal.com> References: <20130831052136.42D4B2507F5@webabinitio.net> <87bo4echcr.fsf@uwakimon.sk.tsukuba.ac.jp> <20130901221040.DC76E250818@webabinitio.net> <5223C8E2.2000006@g.nevcal.com> <874na3diwj.fsf@uwakimon.sk.tsukuba.ac.jp> <5224286E.4040409@g.nevcal.com> <87y57fbt2q.fsf@uwakimon.sk.tsukuba.ac.jp> <20130902214056.ADC342502EA@webabinitio.net> <522516CB.1030705@g.nevcal.com> Message-ID: <20130903012716.1E3FF250166@webabinitio.net> On Mon, 02 Sep 2013 15:52:59 -0700, Glenn Linderman <v+python at g.nevcal.com> wrote: > MUAs tend to be able to display what they produce themselves, but I have > situations where they don't handle what other MUAs produce. > > One nice thing about this email6 toolkit might be the ability to > produce, more easily than before, a variety of MIME combinations to > exercise and test a variety of MUAs. While perhaps most of them have > been tested with some obviously standard MIME combinations, I suspect > most of them will produce strange results with combinations that are out > of the ordinary. Yeah, RFC compliance and other types of testing is something I want this package to be good for. The API under discussion here, though, is oriented toward people using the library for easily generating emails from their application and/or easily accessing the information from emails sent to their application. --David From stephen at xemacs.org Tue Sep 3 03:56:36 2013 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Tue, 03 Sep 2013 10:56:36 +0900 Subject: [Python-Dev] Completing the email6 API changes. In-Reply-To: <20130902214056.ADC342502EA@webabinitio.net> References: <20130831052136.42D4B2507F5@webabinitio.net> <87bo4echcr.fsf@uwakimon.sk.tsukuba.ac.jp> <20130901221040.DC76E250818@webabinitio.net> <5223C8E2.2000006@g.nevcal.com> <874na3diwj.fsf@uwakimon.sk.tsukuba.ac.jp> <5224286E.4040409@g.nevcal.com> <87y57fbt2q.fsf@uwakimon.sk.tsukuba.ac.jp> <20130902214056.ADC342502EA@webabinitio.net> Message-ID: <87r4d6brcb.fsf@uwakimon.sk.tsukuba.ac.jp> R. David Murray writes: > I'm still not understanding how the text/plain part *refers* to the > related parts. Like this: "Check out this picture of my dog!" Or this: "The terms of the contract are found in the attached PDF. Please print it and sign it, then return it by carrier pigeon (attached)." With this structure multipart/alternative text/plain multipart/related text/html application/pdf application/rfc6214-transport the rendering of the text/plain part will not show evidence of the PDF at all (eg, a view/download button), at least in some of the MUAs I've tested. And it *should* not, in an RFC-conforming MUA. > I can understand the structure Glen found in Applemail: > a series of text/plain parts interspersed with image/jpg, with all parts > after the first being marked 'Contentent-Disposition: inline'. Any MUA > that can display text and images *ought* to handle that correctly and > produce the expected result. But that isn't what your structure above > would produce. If you did: > > multipart/related > multipart/alternative > text/html > text/plain > image/png > text/plain > image/png > text/plain > > and only referred to the png parts in the text/html part and marked all > the parts as 'inline' (even though that is irrelevant in the text/html > related case), an MUA that *knew* about this technique *could* display it > "correctly", but an MUA that is just following the standards most > likely won't. OK, I see that now. It requires non-MIME information about the treatment of the root entity by the implementation. On the other hand, it shouldn't *hurt*. RFC 2387 explicitly specifies that at least some parts of a contained multipart/related part should be able to refer to entities related via the containing multipart/related. Since it does not mention *any* restrictions on contained root entities, I take it that it implicitly specifies that any contained multipart may make such references. But I suspect it's not implemented by most MUAs. I'll have to test. > I don't see any way short of duplicating the image parts to make it > likely that a typical MUA would display images for both a text/plain > sequence and a text/html related part. On the other hand, my experience > with MUAs is actually quite limited :) > > Unless there is some standard for referring to related parts in a > text/plain part? No, the whole point is that we MUA implementers *know* that there is no machine-parsable way to refer to the related parts in text/plain, and therefore the only way to communicate even the *presence* of the attachment in multipart/related text/plain image/jpeg; name="dog-photo.jpg" to the receiving user is to make an exception in the implementation and treat it as multipart/mixed.[1] It *does* make sense, i.e., doesn't require any information not already available to the implementation. I wonder if use of external bodies could avoid the duplication in current implementations. Probably too fragile, though. Footnotes: [1] This is conformant to the RFC, as the mechanism of "relation" is explicitly application-dependent. From rdmurray at bitdance.com Tue Sep 3 16:01:42 2013 From: rdmurray at bitdance.com (R. David Murray) Date: Tue, 03 Sep 2013 10:01:42 -0400 Subject: [Python-Dev] Completing the email6 API changes. In-Reply-To: <87r4d6brcb.fsf@uwakimon.sk.tsukuba.ac.jp> References: <20130831052136.42D4B2507F5@webabinitio.net> <87bo4echcr.fsf@uwakimon.sk.tsukuba.ac.jp> <20130901221040.DC76E250818@webabinitio.net> <5223C8E2.2000006@g.nevcal.com> <874na3diwj.fsf@uwakimon.sk.tsukuba.ac.jp> <5224286E.4040409@g.nevcal.com> <87y57fbt2q.fsf@uwakimon.sk.tsukuba.ac.jp> <20130902214056.ADC342502EA@webabinitio.net> <87r4d6brcb.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <20130903140143.4EC862502EA@webabinitio.net> On Tue, 03 Sep 2013 10:56:36 +0900, "Stephen J. Turnbull" <stephen at xemacs.org> wrote: > R. David Murray writes: > > I can understand the structure Glen found in Applemail: > > a series of text/plain parts interspersed with image/jpg, with all parts > > after the first being marked 'Contentent-Disposition: inline'. Any MUA > > that can display text and images *ought* to handle that correctly and > > produce the expected result. But that isn't what your structure above > > would produce. If you did: > > > > multipart/related > > multipart/alternative > > text/html > > text/plain > > image/png > > text/plain > > image/png > > text/plain > > > > and only referred to the png parts in the text/html part and marked all > > the parts as 'inline' (even though that is irrelevant in the text/html > > related case), an MUA that *knew* about this technique *could* display it > > "correctly", but an MUA that is just following the standards most > > likely won't. > > OK, I see that now. It requires non-MIME information about the > treatment of the root entity by the implementation. On the other > hand, it shouldn't *hurt*. RFC 2387 explicitly specifies that at > least some parts of a contained multipart/related part should be able > to refer to entities related via the containing multipart/related. > Since it does not mention *any* restrictions on contained root > entities, I take it that it implicitly specifies that any contained > multipart may make such references. But I suspect it's not > implemented by most MUAs. I'll have to test. OK, I see what you are driving at now. Whether or not it works is dependent on whether or not typical MUAs handle a multipart/related with a text/plain root part by treating it as if it were a multipart/mixed with inline or attachment sub-parts. So yes, whether or not we should support and/or document this technique very much depends on whether or not typical MUAs do so. I will, needless to say, be very interested in the results of your research :) --David From rdmurray at bitdance.com Tue Sep 3 17:28:32 2013 From: rdmurray at bitdance.com (R. David Murray) Date: Tue, 03 Sep 2013 11:28:32 -0400 Subject: [Python-Dev] Completing the email6 API changes. In-Reply-To: <20130903140143.4EC862502EA@webabinitio.net> References: <20130831052136.42D4B2507F5@webabinitio.net> <87bo4echcr.fsf@uwakimon.sk.tsukuba.ac.jp> <20130901221040.DC76E250818@webabinitio.net> <5223C8E2.2000006@g.nevcal.com> <874na3diwj.fsf@uwakimon.sk.tsukuba.ac.jp> <5224286E.4040409@g.nevcal.com> <87y57fbt2q.fsf@uwakimon.sk.tsukuba.ac.jp> <20130902214056.ADC342502EA@webabinitio.net> <87r4d6brcb.fsf@uwakimon.sk.tsukuba.ac.jp> <20130903140143.4EC862502EA@webabinitio.net> Message-ID: <20130903152832.EB219250166@webabinitio.net> On Tue, 03 Sep 2013 10:01:42 -0400, "R. David Murray" <rdmurray at bitdance.com> wrote: > On Tue, 03 Sep 2013 10:56:36 +0900, "Stephen J. Turnbull" <stephen at xemacs.org> wrote: > > R. David Murray writes: > > > I can understand the structure Glen found in Applemail: > > > a series of text/plain parts interspersed with image/jpg, with all parts > > > after the first being marked 'Contentent-Disposition: inline'. Any MUA > > > that can display text and images *ought* to handle that correctly and > > > produce the expected result. But that isn't what your structure above > > > would produce. If you did: > > > > > > multipart/related > > > multipart/alternative > > > text/html > > > text/plain > > > image/png > > > text/plain > > > image/png > > > text/plain > > > > > > and only referred to the png parts in the text/html part and marked all > > > the parts as 'inline' (even though that is irrelevant in the text/html > > > related case), an MUA that *knew* about this technique *could* display it > > > "correctly", but an MUA that is just following the standards most > > > likely won't. > > > > OK, I see that now. It requires non-MIME information about the > > treatment of the root entity by the implementation. On the other > > hand, it shouldn't *hurt*. RFC 2387 explicitly specifies that at > > least some parts of a contained multipart/related part should be able > > to refer to entities related via the containing multipart/related. > > Since it does not mention *any* restrictions on contained root > > entities, I take it that it implicitly specifies that any contained > > multipart may make such references. But I suspect it's not > > implemented by most MUAs. I'll have to test. > > OK, I see what you are driving at now. Whether or not it works is > dependent on whether or not typical MUAs handle a multipart/related with > a text/plain root part by treating it as if it were a multipart/mixed I meant "a text/plain root part *inside* a multipart/alternative", which is what you said, I just didn't understand it at first :) Although I wonder how many GUI MUAs do the fallback to multipart/mixed with just a normal text/plain root part, too. I would expect a text-only MUA would, since it has no other way to display a multipart/related...but a graphical MUA might just assume that there will always be an html part in a multipart/related. > with inline or attachment sub-parts. So yes, whether or not we should > support and/or document this technique very much depends on whether or > not typical MUAs do so. I will, needless to say, be very interested in > the results of your research :) > > --David > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/rdmurray%40bitdance.com From barry at python.org Tue Sep 3 20:51:05 2013 From: barry at python.org (Barry Warsaw) Date: Tue, 3 Sep 2013 14:51:05 -0400 Subject: [Python-Dev] Python 2.6 to end support with 2.6.9 in October 2013 Message-ID: <20130903145105.76944e79@anarchist> Hello Pythonistas, Python 2.6.9 is the last planned release of the 2.6.x series. This will be a security-only source-only release. It is currently scheduled for October 2013, and after this Python 2.6 will have reached its end-of-life and the branch will be retired. http://www.python.org/dev/peps/pep-0361/ I would like to release 2.6.9rc1 on Monday, September 30th, and 2.6.9 final on Monday, October 28th. I've added both dates to the Python calendar. Here are the list of candidates still to be fixed for 2.6.9: - 18747 - Re-seed OpenSSL's PRNG after fork - 16037 - httplib: header parsing is not delimited - 16038 - ftplib: unlimited readline() from connection - 16039 - imaplib: unlimited readline() from connection - 16040 - nntplib: unlimited readline() from connection - 16041 - poplib: unlimited readline() from connection - 16042 - smtplib: unlimited readline() from connection - 16043 - xmlrpc: gzip_decode has unlimited read() These were the ones I previously had on my list, and I've now marked these all as release blockers for 2.6.9... for now. If you know of any others that I should be aware of, please let me know. If you can contribute to 2.6.9 by reviewing, testing, developing, or commenting, that would be greatly appreciated. I will be spending some time triaging these and any other issues that get identified as possible 2.6.9 candidates. If you have any questions regarding 2.6.9, please contact me via mailing list or IRC. Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: <http://mail.python.org/pipermail/python-dev/attachments/20130903/d62efb81/attachment.sig> From rymg19 at gmail.com Tue Sep 3 21:53:53 2013 From: rymg19 at gmail.com (Ryan) Date: Tue, 03 Sep 2013 14:53:53 -0500 Subject: [Python-Dev] Python 2.6 to end support with 2.6.9 in October 2013 In-Reply-To: <20130903145105.76944e79@anarchist> References: <20130903145105.76944e79@anarchist> Message-ID: <ec8efc66-d6e7-4a4c-a77a-f6590f7fecea@email.android.com> I'm still waiting on Python 2.7 for Android! Stuck on 2.6 for now...ugh! Wonder if I can build it myself... Barry Warsaw <barry at python.org> wrote: >Hello Pythonistas, > >Python 2.6.9 is the last planned release of the 2.6.x series. This >will be a >security-only source-only release. It is currently scheduled for >October >2013, and after this Python 2.6 will have reached its end-of-life and >the >branch will be retired. > >http://www.python.org/dev/peps/pep-0361/ > >I would like to release 2.6.9rc1 on Monday, September 30th, and 2.6.9 >final on >Monday, October 28th. I've added both dates to the Python calendar. > >Here are the list of candidates still to be fixed for 2.6.9: > >- 18747 - Re-seed OpenSSL's PRNG after fork >- 16037 - httplib: header parsing is not delimited >- 16038 - ftplib: unlimited readline() from connection >- 16039 - imaplib: unlimited readline() from connection >- 16040 - nntplib: unlimited readline() from connection >- 16041 - poplib: unlimited readline() from connection >- 16042 - smtplib: unlimited readline() from connection >- 16043 - xmlrpc: gzip_decode has unlimited read() > >These were the ones I previously had on my list, and I've now marked >these all >as release blockers for 2.6.9... for now. > >If you know of any others that I should be aware of, please let me >know. > >If you can contribute to 2.6.9 by reviewing, testing, developing, or >commenting, that would be greatly appreciated. I will be spending some >time >triaging these and any other issues that get identified as possible >2.6.9 >candidates. > >If you have any questions regarding 2.6.9, please contact me via >mailing list >or IRC. > >Cheers, >-Barry > > >------------------------------------------------------------------------ > >_______________________________________________ >Python-Dev mailing list >Python-Dev at python.org >https://mail.python.org/mailman/listinfo/python-dev >Unsubscribe: >https://mail.python.org/mailman/options/python-dev/rymg19%40gmail.com -- Sent from my Android phone with K-9 Mail. Please excuse my brevity. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130903/97f55af0/attachment.html> From victor.stinner at gmail.com Wed Sep 4 01:27:15 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 4 Sep 2013 01:27:15 +0200 Subject: [Python-Dev] RFC: PEP 454: Add a new tracemalloc module Message-ID: <CAMpsgwZTOBdXy2Rcxs0itnPZYZgHU4X4tPNf5syoZrOJ5gYCFw@mail.gmail.com> Hi, Antoine Pitrou suggested me to write a PEP to discuss the API of the new tracemalloc module that I proposed to add to Python 3.4. Here you have. If you prefer to read the HTML version: http://www.python.org/dev/peps/pep-0454/ See also the documentation of the current implementation of the module. http://hg.python.org/features/tracemalloc/file/tip/Doc/library/tracemalloc.rst The documentaion contains examples and a short "tutorial". PEP: 454 Title: Add a new tracemalloc module to trace Python memory allocations Version: $Revision$ Last-Modified: $Date$ Author: Victor Stinner <victor.stinner at gmail.com> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 3-September-2013 Python-Version: 3.4 Abstract ======== Add a new ``tracemalloc`` module to trace Python memory allocations. Rationale ========= Common debug tools tracing memory allocations read the C filename and number. Using such tool to analyze Python memory allocations does not help because most memory allocations are done in the same C function, ``PyMem_Malloc()`` for example. There are debug tools dedicated to the Python languages like ``Heapy`` and ``PySizer``. These projects analyze objects type and/or content. These tools are useful when the most memory leak are instances of the same type and this type in allocated only in a few functions. The problem is when the object type is very common like ``str`` or ``tuple``, and it is hard to identify where these objects are allocated. Finding reference cycles is also a difficult task. There are different tools to draw a diagram of all references. These tools cannot be used huge on large applications with thousands of objects because the diagram is too huge to be analyzed manually. Proposal ======== Using the PEP 445, it becomes easy to setup an hook on Python memory allocators. The hook can inspect the current Python frame to get the Python filename and line number. This PEP proposes to add a new ``tracemalloc`` module. It is a debug tool to trace memory allocations made by Python. The module provides the following information: * Statistics on Python memory allocations per Python filename and line number: size, number, and average size of allocations * Compute differences between two snapshots of Python memory allocations * Location of a Python memory allocation: size in bytes, Python filename and line number Command line options ==================== The ``python -m tracemalloc`` command can be used to analyze and compare snapshots. The command takes a list of snapshot filenames and has the following options. ``-g``, ``--group-per-file`` Group allocations per filename, instead of grouping per line number. ``-n NTRACES``, ``--number NTRACES`` Number of traces displayed per top (default: 10). ``--first`` Compare with the first snapshot, instead of comparing with the previous snapshot. ``--include PATTERN`` Only include filenames matching pattern *PATTERN*. The option can be specified multiple times. See ``fnmatch.fnmatch()`` for the syntax of patterns. ``--exclude PATTERN`` Exclude filenames matching pattern *PATTERN*. The option can be specified multiple times. See ``fnmatch.fnmatch()`` for the syntax of patterns. ``-S``, ``--hide-size`` Hide the size of allocations. ``-C``, ``--hide-count`` Hide the number of allocations. ``-A``, ``--hide-average`` Hide the average size of allocations. ``-P PARTS``, ``--filename-parts=PARTS`` Number of displayed filename parts (default: 3). ``--color`` Force usage of colors even if ``sys.stdout`` is not a TTY device. ``--no-color`` Disable colors if ``sys.stdout`` is a TTY device. API === To trace the most Python memory allocations, the module should be enabled as early as possible in your application by calling ``tracemalloc.enable()`` function, by setting the ``PYTHONTRACEMALLOC`` environment variable to ``1``, or by using ``-X tracemalloc`` command line option. Functions --------- ``enable()`` function: Start tracing Python memory allocations. ``disable()`` function: Stop tracing Python memory allocations and stop the timer started by ``start_timer()``. ``is_enabled()`` function: Get the status of the module: ``True`` if it is enabled, ``False`` otherwise. ``get_object_address(obj)`` function: Get the address of the memory block of the specified Python object. ``get_object_trace(obj)`` function: Get the trace of a Python object *obj* as a ``trace`` instance. Return ``None`` if the tracemalloc module did not save the location when the object was allocated, for example if the module was disabled. ``get_process_memory()`` function: Get the memory usage of the current process as a meminfo namedtuple with two attributes: * ``rss``: Resident Set Size in bytes * ``vms``: size of the virtual memory in bytes Return ``None`` if the platform is not supported. Use the ``psutil`` module if available. ``get_stats()`` function: Get statistics on Python memory allocations per Python filename and per Python line number. Return a dictionary ``{filename: str -> {line_number: int -> stats: line_stat}}`` where *stats* in a ``line_stat`` instance. *filename* and *line_number* can be ``None``. Return an empty dictionary if the tracemalloc module is disabled. ``get_traces(obj)`` function: Get all traces of a Python memory allocations. Return a dictionary ``{pointer: int -> trace}`` where *trace* is a ``trace`` instance. Return an empty dictionary if the ``tracemalloc`` module is disabled. ``start_timer(delay: int, func: callable, args: tuple=(), kwargs: dict={})`` function: Start a timer calling ``func(*args, **kwargs)`` every *delay* seconds. The timer is based on the Python memory allocator, it is not real time. *func* is called after at least *delay* seconds, it is not called exactly after *delay* seconds if no Python memory allocation occurred. If ``start_timer()`` is called twice, previous parameters are replaced. The timer has a resolution of 1 second. ``start_timer()`` is used by ``DisplayTop`` and ``TakeSnapshot`` to run regulary a task. ``stop_timer()`` function: Stop the timer started by ``start_timer()``. trace class ----------- ``trace`` class: This class represents debug information of an allocated memory block. ``size`` attribute: Size in bytes of the memory block. ``filename`` attribute: Name of the Python script where the memory block was allocated, ``None`` if unknown. ``lineno`` attribute: Line number where the memory block was allocated, ``None`` if unknown. line_stat class ---------------- ``line_stat`` class: Statistics on Python memory allocations of a specific line number. ``size`` attribute: Total size in bytes of all memory blocks allocated on the line. ``count`` attribute: Number of memory blocks allocated on the line. DisplayTop class ---------------- ``DisplayTop(count: int=10, file=sys.stdout)`` class: Display the list of the *count* biggest memory allocations into *file*. ``display()`` method: Display the top once. ``start(delay: int)`` method: Start a task using ``tracemalloc`` timer to display the top every *delay* seconds. ``stop()`` method: Stop the task started by the ``DisplayTop.start()`` method ``color`` attribute: If ``True``, ``display()`` uses color. The default value is ``file.isatty()``. ``compare_with_previous`` attribute: If ``True`` (default value), ``display()`` compares with the previous snapshot. If ``False``, compare with the first snapshot. ``filename_parts`` attribute: Number of displayed filename parts (int, default: ``3``). Extra parts are replaced with ``"..."``. ``group_per_file`` attribute: If ``True``, group memory allocations per Python filename. If ``False`` (default value), group allocation per Python line number. ``show_average`` attribute: If ``True`` (default value), ``display()`` shows the average size of allocations. ``show_count`` attribute: If ``True`` (default value), ``display()`` shows the number of allocations. ``show_size`` attribute: If ``True`` (default value), ``display()`` shows the size of allocations. ``user_data_callback`` attribute: Optional callback collecting user data (callable, default: ``None``). See ``Snapshot.create()``. Snapshot class -------------- ``Snapshot()`` class: Snapshot of Python memory allocations. Use ``TakeSnapshot`` to take regulary snapshots. ``create(user_data_callback=None)`` method: Take a snapshot. If *user_data_callback* is specified, it must be a callable object returning a list of ``(title: str, format: str, value: int)``. *format* must be ``'size'``. The list must always have the same length and the same order to be able to compute differences between values. Example: ``[('Video memory', 'size', 234902)]``. ``filter_filenames(patterns: list, include: bool)`` method: Remove filenames not matching any pattern of *patterns* if *include* is ``True``, or remove filenames matching a pattern of *patterns* if *include* is ``False`` (exclude). See ``fnmatch.fnmatch()`` for the syntax of a pattern. ``load(filename)`` classmethod: Load a snapshot from a file. ``write(filename)`` method: Write the snapshot into a file. ``pid`` attribute: Identifier of the process which created the snapshot (int). ``process_memory`` attribute: Result of the ``get_process_memory()`` function, can be ``None``. ``stats`` attribute: Result of the ``get_stats()`` function (dict). ``timestamp`` attribute: Creation date and time of the snapshot, ``datetime.datetime`` instance. ``user_data`` attribute: Optional list of user data, result of *user_data_callback* in ``Snapshot.create()`` (default: None). TakeSnapshot class ------------------ ``TakeSnapshot`` class: Task taking snapshots of Python memory allocations: write them into files. By default, snapshots are written in the current directory. ``start(delay: int)`` method: Start a task taking a snapshot every delay seconds. ``stop()`` method: Stop the task started by the ``TakeSnapshot.start()`` method. ``take_snapshot()`` method: Take a snapshot. ``filename_template`` attribute: Template (``str``) used to create a filename. The following variables can be used in the template: * ``$pid``: identifier of the current process * ``$timestamp``: current date and time * ``$counter``: counter starting at 1 and incremented at each snapshot The default pattern is ``'tracemalloc-$counter.pickle'``. ``user_data_callback`` attribute: Optional callback collecting user data (callable, default: ``None``). See ``Snapshot.create()``. Links ===== Python issues: * `#18874: Add a new tracemalloc module to trace Python memory allocations <http://bugs.python.org/issue18874>`_ Similar projects: * `Meliae: Python Memory Usage Analyzer <https://pypi.python.org/pypi/meliae>`_ * `Guppy-PE: umbrella package combining Heapy and GSL <http://guppy-pe.sourceforge.net/>`_ * `PySizer <http://pysizer.8325.org/>`_: developed for Python 2.4 * `memory_profiler <https://pypi.python.org/pypi/memory_profiler>`_ * `pympler <http://code.google.com/p/pympler/>`_ * `Dozer <https://pypi.python.org/pypi/Dozer>`_: WSGI Middleware version of the CherryPy memory leak debugger * `objgraph <http://mg.pov.lt/objgraph/>`_ * `caulk <https://github.com/smartfile/caulk/>`_ Copyright ========= This document has been placed into the public domain. From victor.stinner at gmail.com Wed Sep 4 01:56:21 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 4 Sep 2013 01:56:21 +0200 Subject: [Python-Dev] RFC: PEP 454: Add a new tracemalloc module In-Reply-To: <CAMpsgwZTOBdXy2Rcxs0itnPZYZgHU4X4tPNf5syoZrOJ5gYCFw@mail.gmail.com> References: <CAMpsgwZTOBdXy2Rcxs0itnPZYZgHU4X4tPNf5syoZrOJ5gYCFw@mail.gmail.com> Message-ID: <CAMpsgwagcq-vNr3W7mQ7kPx=1sq=tio6cDyGNbRiEc6=EQXHNA@mail.gmail.com> > ``get_object_trace(obj)`` function: > > Get the trace of a Python object *obj* as a ``trace`` instance. > > Return ``None`` if the tracemalloc module did not save the location > when the object was allocated, for example if the module was > disabled. This function and get_traces() can be reused by other debug tools like Heapy and objgraph to add where objects were allocated. > ``get_stats()`` function: > > Get statistics on Python memory allocations per Python filename and > per Python line number. > > Return a dictionary > ``{filename: str -> {line_number: int -> stats: line_stat}}`` > where *stats* in a ``line_stat`` instance. *filename* and > *line_number* can be ``None``. > > Return an empty dictionary if the tracemalloc module is disabled. > > ``get_traces(obj)`` function: > > Get all traces of a Python memory allocations. > Return a dictionary ``{pointer: int -> trace}`` where *trace* > is a ``trace`` instance. > > Return an empty dictionary if the ``tracemalloc`` module is disabled. get_stats() can computed from get_traces(), example: ----- import pprint, tracemalloc traces = tracemalloc.get_traces() stats = {} for trace in traces.values(): if trace.filename not in stats: stats[trace.filename] = line_stats = {} else: line_stats = stats[trace.filename] if trace.lineno not in line_stats: line_stats[trace.lineno] = line_stat = tracemalloc.line_stat((0, 0)) size = trace.size count = 1 else: line_stat = line_stats[trace.lineno] size = line_stat.size + trace.size count = line_stat.count + 1 line_stats[trace.lineno] = tracemalloc.line_stat((size, count)) pprint.pprint(stats) ----- The problem is the efficiency. At startup, Python already allocated more than 20,000 memory blocks: $ ./python -X tracemalloc -c 'import tracemalloc; print(len(tracemalloc.get_traces()))' 21704 At the end of the Python test suite, Python allocated more than 500,000 memory blocks. Storing all these traces in a snapshot eats a lot of memory, disk space and uses CPU to build the statistics. > ``start_timer(delay: int, func: callable, args: tuple=(), kwargs: > dict={})`` function: > > Start a timer calling ``func(*args, **kwargs)`` every *delay* > seconds. (...) > > If ``start_timer()`` is called twice, previous parameters are > replaced. The timer has a resolution of 1 second. > > ``start_timer()`` is used by ``DisplayTop`` and ``TakeSnapshot`` to > run regulary a task. So DisplayTop and TakeSnapshot cannot be used at the same time. It would be convinient to be able to register more than one function. What do you think? > ``trace`` class: > This class represents debug information of an allocated memory block. > > ``size`` attribute: > Size in bytes of the memory block. > ``filename`` attribute: > Name of the Python script where the memory block was allocated, > ``None`` if unknown. > ``lineno`` attribute: > Line number where the memory block was allocated, ``None`` if > unknown. I though twice and it would be posible to store more than 1 frame per trace instance, to be able to rebuild a (partial) Python traceback. The hook on the memory allocator has access to the chain of Python frames. The API should be changed to support such enhancement. > ``DisplayTop(count: int=10, file=sys.stdout)`` class: > Display the list of the *count* biggest memory allocations into > *file*. > (...) > ``group_per_file`` attribute: > > If ``True``, group memory allocations per Python filename. If > ``False`` (default value), group allocation per Python line number. This attribute is very important. We may add it to the constructor. By the way, the self.stream attribute is not documented. > Snapshot class > -------------- > > ``Snapshot()`` class: > > Snapshot of Python memory allocations. > > Use ``TakeSnapshot`` to take regulary snapshots. > > ``create(user_data_callback=None)`` method: > > Take a snapshot. If *user_data_callback* is specified, it must be a > callable object returning a list of > ``(title: str, format: str, value: int)``. > *format* must be ``'size'``. The list must always have the same > length and the same order to be able to compute differences between > values. > > Example: ``[('Video memory', 'size', 234902)]``. (Oops, create() is a class method, not a method.) Having to call a class method to build an instance of a class is surprising. But I didn't find a way to implement the load() class method otherwise. The user_data_callback API can be improved. The "format must be size" is not very convinient. Victor From stephen at xemacs.org Wed Sep 4 03:30:28 2013 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Wed, 04 Sep 2013 10:30:28 +0900 Subject: [Python-Dev] Completing the email6 API changes. In-Reply-To: <20130903152832.EB219250166@webabinitio.net> References: <20130831052136.42D4B2507F5@webabinitio.net> <87bo4echcr.fsf@uwakimon.sk.tsukuba.ac.jp> <20130901221040.DC76E250818@webabinitio.net> <5223C8E2.2000006@g.nevcal.com> <874na3diwj.fsf@uwakimon.sk.tsukuba.ac.jp> <5224286E.4040409@g.nevcal.com> <87y57fbt2q.fsf@uwakimon.sk.tsukuba.ac.jp> <20130902214056.ADC342502EA@webabinitio.net> <87r4d6brcb.fsf@uwakimon.sk.tsukuba.ac.jp> <20130903140143.4EC862502EA@webabinitio.net> <20130903152832.EB219250166@webabinitio.net> Message-ID: <87eh95bcgb.fsf@uwakimon.sk.tsukuba.ac.jp> R. David Murray writes: > I meant "a text/plain root part *inside* a multipart/alternative", which > is what you said, I just didn't understand it at first :) Although I > wonder how many GUI MUAs do the fallback to multipart/mixed with just a > normal text/plain root part, too. I would expect a text-only MUA would, > since it has no other way to display a multipart/related...but a > graphical MUA might just assume that there will always be an html part > in a multipart/related. It's not really a problem with text vs. GUI, or an assumption of HMTL. There are plenty of formats that have such links, and some which don't have links, but rather assigned roles such as "Mac files" (with data fork and resource fork) and digital signatures (though that turned out to be worth designing a new multipart subtype). The problem is that "multipart/related" says "pass all the part entities to the handler appropriate to the root part entity, which will process the links found in the root part entity". If you implement that in the natural way, you just pass the text/plain part to the text/plain handler, which won't find any links for the simple reason that it has no protocol for representing them. This means that the kind of multipart/related handler I envision needs to implement linking itself, rather than delegate them to the root part handler. This requires checking the type of the root part: # not intended to look like Email API def handle_multipart_related (part_list, root_part): if root_part.content_type in ['text/plain']: # just display the parts in order handle_multipart_mixed (part_list) else: # cid -> entities in internal representation entity_map = extract_entity_map(part_list) root_part.content_type.handle(root_part, entity_map) From victor.stinner at gmail.com Wed Sep 4 13:20:11 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 4 Sep 2013 13:20:11 +0200 Subject: [Python-Dev] RFC: PEP 454: Add a new tracemalloc module In-Reply-To: <CAMpsgwagcq-vNr3W7mQ7kPx=1sq=tio6cDyGNbRiEc6=EQXHNA@mail.gmail.com> References: <CAMpsgwZTOBdXy2Rcxs0itnPZYZgHU4X4tPNf5syoZrOJ5gYCFw@mail.gmail.com> <CAMpsgwagcq-vNr3W7mQ7kPx=1sq=tio6cDyGNbRiEc6=EQXHNA@mail.gmail.com> Message-ID: <CAMpsgwbtfpYVXSTHiygYAxY+ktf9Tj9vW-J-Nv8HV-75q7oaUw@mail.gmail.com> >> ``trace`` class: >> This class represents debug information of an allocated memory block. >> >> ``size`` attribute: >> Size in bytes of the memory block. >> ``filename`` attribute: >> Name of the Python script where the memory block was allocated, >> ``None`` if unknown. >> ``lineno`` attribute: >> Line number where the memory block was allocated, ``None`` if >> unknown. > > I though twice and it would be posible to store more than 1 frame per > trace instance, to be able to rebuild a (partial) Python traceback. > The hook on the memory allocator has access to the chain of Python > frames. The API should be changed to support such enhancement. Oh, it was much easier than expected to retrieve the traceback (maximum 10 frames) instead of only the current frame. I modified the trace class to replace filename and lineno with a new frames attribute which is list of frames. Script example: --- import tracemalloc, linecache def g(): return object() def f(): return g() tracemalloc.enable() obj = f() trace = tracemalloc.get_object_trace(obj) print("Traceback (most recent first):") for frame in trace.frames: print(' File "%s", line %s' % (frame.filename, frame.lineno)) line = linecache.getline(frame.filename, frame.lineno) if line: print(" " + line.strip()) --- Output of the script: --- Traceback (most recent first): File "x.py", line 4 return object() File "x.py", line 7 return g() File "x.py", line 10 obj = f() --- I updated the PEP 454 (add a new frame class, update trace class): frame class ----------- ``frame`` class: Trace of a Python frame. ``filename`` attribute (``str``): Python filename, ``None`` if unknown. ``lineno`` attribute (``int``): Python line number, ``None`` if unknown. trace class ----------- ``trace`` class: This class represents debug information of an allocated memory block. ``size`` attribute (``int``): Size in bytes of the memory block. ``frames`` attribute (``list``): Traceback where the memory block was allocated as a list of ``frame`` instances (most recent first). The list can be empty or incomplete if the tracemalloc module was unable to retrieve the full traceback. For efficiency, the traceback is truncated to 10 frames. Victor From alexander.belopolsky at gmail.com Thu Sep 5 02:46:21 2013 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Wed, 4 Sep 2013 20:46:21 -0400 Subject: [Python-Dev] RFC: PEP 454: Add a new tracemalloc module In-Reply-To: <CAMpsgwZTOBdXy2Rcxs0itnPZYZgHU4X4tPNf5syoZrOJ5gYCFw@mail.gmail.com> References: <CAMpsgwZTOBdXy2Rcxs0itnPZYZgHU4X4tPNf5syoZrOJ5gYCFw@mail.gmail.com> Message-ID: <CAP7h-xauzgw8UJqKJAb2WPMqPM9-_BvDhHgcj5RnKvj5yaMiYw@mail.gmail.com> On Tue, Sep 3, 2013 at 7:27 PM, Victor Stinner <victor.stinner at gmail.com>wrote: > API > === > > To trace the most Python memory allocations, the module should be > enabled as early as possible in your application by calling > ``tracemalloc.enable()`` function, by setting the ``PYTHONTRACEMALLOC`` > environment variable to ``1``, or by using ``-X tracemalloc`` command > line option. > > > Functions > --------- > > ``enable()`` function: > > Start tracing Python memory allocations. > > ``disable()`` function: > > Stop tracing Python memory allocations and stop the timer started by > ``start_timer()``. > > ``is_enabled()`` function: > > Get the status of the module: ``True`` if it is enabled, ``False`` > otherwise. > Please mention that this API is similar to that of faulthandler and add a link to faulthandler docs. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130904/311e3a96/attachment.html> From jcea at jcea.es Thu Sep 5 19:31:59 2013 From: jcea at jcea.es (Jesus Cea) Date: Thu, 05 Sep 2013 19:31:59 +0200 Subject: [Python-Dev] Offtopic: OpenID Providers Message-ID: <5228C00F.7000209@jcea.es> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 I just received an email from my OpenID provider, "myOpenID", saying that they drop OpenID service next February. I wonder what other OpenID providers are used by other python-dev fellows. What are you using?. bugs.python.org admins could share some data? I agree than OpenID is (quite) dead, but I rather prefer OpenID to use user/pass. I have big hopes for Mozilla Persona, looking forward Python infrastructure support :). PS: I use "http://www.jcea.es/" as my OpenID identity, and I delegate the actual service to "myOpenID". I can switch delegation trivially. - -- Jes?s Cea Avi?n _/_/ _/_/_/ _/_/_/ jcea at jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ Twitter: @jcea _/_/ _/_/ _/_/_/_/_/ jabber / xmpp:jcea at jabber.org _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQCVAwUBUijAD5lgi5GaxT1NAQLH0wQAkKDORrAtfJFzzIHl3lHRp7GfxOzdqdNP uiuW65l/pas+p9+B0G6qR6EE2AAL7YPozcNF5AkmuGmxkpyn/JyUYKJcUWmUotpj V9Buz9jz3qpPuv7AlTnMbjBBQK4YTYenbdk2HgI41SVQHZHkU/+y4CL3Y1hWyJNo C8CCWfR0VlA= =YXIq -----END PGP SIGNATURE----- From phd at phdru.name Thu Sep 5 20:12:31 2013 From: phd at phdru.name (Oleg Broytman) Date: Thu, 5 Sep 2013 22:12:31 +0400 Subject: [Python-Dev] Offtopic: OpenID Providers In-Reply-To: <5228C00F.7000209@jcea.es> References: <5228C00F.7000209@jcea.es> Message-ID: <20130905181231.GA25306@iskra.aviel.ru> On Thu, Sep 05, 2013 at 07:31:59PM +0200, Jesus Cea <jcea at jcea.es> wrote: > I just received an email from my OpenID provider, "myOpenID", saying > that they drop OpenID service next February. I wonder what other > OpenID providers are used by other python-dev fellows. > > What are you using?. bugs.python.org admins could share some data? > > I agree than OpenID is (quite) dead, but I rather prefer OpenID to use > user/pass. I have big hopes for Mozilla Persona, looking forward > Python infrastructure support :). > > PS: I use "http://www.jcea.es/" as my OpenID identity, and I delegate > the actual service to "myOpenID". I can switch delegation trivially. I used to use myOpenID and became my own provider using poit[1]. These days I seldom use OpenID -- there are too few sites that allow full-featured login with OpenID. The future lies in OAuth 2.0. 1. http://yangman.ca/poit/ Oleg. -- Oleg Broytman http://phdru.name/ phd at phdru.name Programmers don't die, they just GOSUB without RETURN. From donald at stufft.io Thu Sep 5 20:16:29 2013 From: donald at stufft.io (Donald Stufft) Date: Thu, 5 Sep 2013 14:16:29 -0400 Subject: [Python-Dev] Offtopic: OpenID Providers In-Reply-To: <20130905181231.GA25306@iskra.aviel.ru> References: <5228C00F.7000209@jcea.es> <20130905181231.GA25306@iskra.aviel.ru> Message-ID: <8593FDCE-E3E0-4AD4-897E-2FB1D751086C@stufft.io> On Sep 5, 2013, at 2:12 PM, Oleg Broytman <phd at phdru.name> wrote: > On Thu, Sep 05, 2013 at 07:31:59PM +0200, Jesus Cea <jcea at jcea.es> wrote: >> I just received an email from my OpenID provider, "myOpenID", saying >> that they drop OpenID service next February. I wonder what other >> OpenID providers are used by other python-dev fellows. >> >> What are you using?. bugs.python.org admins could share some data? >> >> I agree than OpenID is (quite) dead, but I rather prefer OpenID to use >> user/pass. I have big hopes for Mozilla Persona, looking forward >> Python infrastructure support :). >> >> PS: I use "http://www.jcea.es/" as my OpenID identity, and I delegate >> the actual service to "myOpenID". I can switch delegation trivially. > > I used to use myOpenID and became my own provider using poit[1]. > These days I seldom use OpenID -- there are too few sites that allow > full-featured login with OpenID. The future lies in OAuth 2.0. The Auth in OAuth stands for Authorization not Authentication. > > 1. http://yangman.ca/poit/ > > Oleg. > -- > Oleg Broytman http://phdru.name/ phd at phdru.name > Programmers don't die, they just GOSUB without RETURN. > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: <http://mail.python.org/pipermail/python-dev/attachments/20130905/0f57ca25/attachment.sig> From phd at phdru.name Thu Sep 5 20:25:22 2013 From: phd at phdru.name (Oleg Broytman) Date: Thu, 5 Sep 2013 22:25:22 +0400 Subject: [Python-Dev] Offtopic: OpenID Providers In-Reply-To: <8593FDCE-E3E0-4AD4-897E-2FB1D751086C@stufft.io> References: <5228C00F.7000209@jcea.es> <20130905181231.GA25306@iskra.aviel.ru> <8593FDCE-E3E0-4AD4-897E-2FB1D751086C@stufft.io> Message-ID: <20130905182522.GA26090@iskra.aviel.ru> On Thu, Sep 05, 2013 at 02:16:29PM -0400, Donald Stufft <donald at stufft.io> wrote: > > On Sep 5, 2013, at 2:12 PM, Oleg Broytman <phd at phdru.name> wrote: > > I used to use myOpenID and became my own provider using poit[1]. > > These days I seldom use OpenID -- there are too few sites that allow > > full-featured login with OpenID. The future lies in OAuth 2.0. > > The Auth in OAuth stands for Authorization not Authentication. There is no authorization without authentication, so OAuth certainly performs authentication: http://oauth.net/core/1.0a/#anchor9 , http://tools.ietf.org/html/rfc5849#section-3 Oleg. -- Oleg Broytman http://phdru.name/ phd at phdru.name Programmers don't die, they just GOSUB without RETURN. From barry at python.org Thu Sep 5 20:30:43 2013 From: barry at python.org (Barry Warsaw) Date: Thu, 5 Sep 2013 14:30:43 -0400 Subject: [Python-Dev] Offtopic: OpenID Providers In-Reply-To: <5228C00F.7000209@jcea.es> References: <5228C00F.7000209@jcea.es> Message-ID: <20130905143043.590a12b2@anarchist> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On Sep 05, 2013, at 07:31 PM, Jesus Cea wrote: >I just received an email from my OpenID provider, "myOpenID", saying >that they drop OpenID service next February. I wonder what other >OpenID providers are used by other python-dev fellows. > >What are you using?. bugs.python.org admins could share some data? Launchpad. It's not going anywhere. >I agree than OpenID is (quite) dead, but I rather prefer OpenID to use >user/pass. I have big hopes for Mozilla Persona, looking forward >Python infrastructure support :). We at the Mailman project like Persona a lot. It'll be the primary way people can log into Postorius (the new web ui). - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.14 (GNU/Linux) iQIcBAEBCAAGBQJSKM3TAAoJEBJutWOnSwa/jP8QAJ4BkV8285TexZq+9GuTwzFf qx2OlUwtiv6lhDZq9uY0sXijyKl76N0/nTsSf5Fr8nDsn5KaETd14PFGfwbg6WVS jMZcKz+3j7eq6zq/Qm47qFbcBb2moyxptkE96BMCqEw+DCnPo87q8DXOb2W67ROI XDsRW2T/D8v5SvAqkmgOPlWsXWc1QzOpJCGkYcufzDeCjkFPXvck3mFI0TpizXnv 7L4IjrK9Qn6D1zNvKmovTkQOu3sk3C2FvPT2QG0b+DWkPtERE6o/xK2OTkhIzbBd /YfLEkb60tJYtwKqfHei+n4pVreExT0DtCDFHh5bYbdcD23VuHfDZtaHCDzYEhK9 rjuY2odydxD/xH9F21enHqYuU76TBr8ceodYsP2FNlqO5XeOJwgAXhWdpBYwN+SU G0B5k3wDb6VCDkqM1u7VGcA/yQLeGQ43s/klxxAxo08sA3url9HcIeuO5rnmnB5l lGIehq/SJPVQXQevEJvL8Atm7uLnsaWEJg34+T04MyR+TLeUcjKsNrUZBS8bGOZu agygKg3M/zi1k9LW/uxKR7IKLyBK+axYoEQS9yydqpgO1e8BQa6XlLQ89Sv9r38Y y1qTFmGdRRQSItrlS1urMQ9Q8m7pJBSyfBywlsck07hpXlZLRwRm2S3DnMBrF6It I7f28XXirrnBf/la/+6g =xqI7 -----END PGP SIGNATURE----- From donald at stufft.io Thu Sep 5 20:35:16 2013 From: donald at stufft.io (Donald Stufft) Date: Thu, 5 Sep 2013 14:35:16 -0400 Subject: [Python-Dev] Offtopic: OpenID Providers In-Reply-To: <20130905182522.GA26090@iskra.aviel.ru> References: <5228C00F.7000209@jcea.es> <20130905181231.GA25306@iskra.aviel.ru> <8593FDCE-E3E0-4AD4-897E-2FB1D751086C@stufft.io> <20130905182522.GA26090@iskra.aviel.ru> Message-ID: <2C429087-4E06-4F0F-B524-25D0F6FB4FFF@stufft.io> On Sep 5, 2013, at 2:25 PM, Oleg Broytman <phd at phdru.name> wrote: > On Thu, Sep 05, 2013 at 02:16:29PM -0400, Donald Stufft <donald at stufft.io> wrote: >> >> On Sep 5, 2013, at 2:12 PM, Oleg Broytman <phd at phdru.name> wrote: >>> I used to use myOpenID and became my own provider using poit[1]. >>> These days I seldom use OpenID -- there are too few sites that allow >>> full-featured login with OpenID. The future lies in OAuth 2.0. >> >> The Auth in OAuth stands for Authorization not Authentication. > > There is no authorization without authentication, so OAuth certainly > performs authentication: http://oauth.net/core/1.0a/#anchor9 , > http://tools.ietf.org/html/rfc5849#section-3 They are separate topics and authorization does not need to imply authentication, it so happens that in many particular instances of OAuth you can estimate authentication. https://en.wikipedia.org/wiki/OAuth#OpenID_vs._pseudo-authentication_using_OAuth Persona is the logical successor to OpenID. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: <http://mail.python.org/pipermail/python-dev/attachments/20130905/065d9b65/attachment.sig> From a.badger at gmail.com Thu Sep 5 20:33:47 2013 From: a.badger at gmail.com (Toshio Kuratomi) Date: Thu, 5 Sep 2013 11:33:47 -0700 Subject: [Python-Dev] Offtopic: OpenID Providers In-Reply-To: <20130905182522.GA26090@iskra.aviel.ru> References: <5228C00F.7000209@jcea.es> <20130905181231.GA25306@iskra.aviel.ru> <8593FDCE-E3E0-4AD4-897E-2FB1D751086C@stufft.io> <20130905182522.GA26090@iskra.aviel.ru> Message-ID: <20130905183347.GO10747@unaka.lan> On Thu, Sep 05, 2013 at 10:25:22PM +0400, Oleg Broytman wrote: > On Thu, Sep 05, 2013 at 02:16:29PM -0400, Donald Stufft <donald at stufft.io> wrote: > > > > On Sep 5, 2013, at 2:12 PM, Oleg Broytman <phd at phdru.name> wrote: > > > I used to use myOpenID and became my own provider using poit[1]. > > > These days I seldom use OpenID -- there are too few sites that allow > > > full-featured login with OpenID. The future lies in OAuth 2.0. > > > > The Auth in OAuth stands for Authorization not Authentication. > > There is no authorization without authentication, so OAuth certainly > performs authentication: http://oauth.net/core/1.0a/#anchor9 , > http://tools.ietf.org/html/rfc5849#section-3 > Sortof.... The way OAuth looks to me, it's designed to prove that a given client is authorized to perform an action. It's not designed to prove that the given client is a specific person. In some cases, you really want to know the latter and not merely the former. So I think in these situations Donald's separation of Authz and Authn makes sense. -Toshio -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 198 bytes Desc: not available URL: <http://mail.python.org/pipermail/python-dev/attachments/20130905/d6c3e7f9/attachment.sig> From phd at phdru.name Thu Sep 5 20:43:37 2013 From: phd at phdru.name (Oleg Broytman) Date: Thu, 5 Sep 2013 22:43:37 +0400 Subject: [Python-Dev] Offtopic: OpenID Providers In-Reply-To: <2C429087-4E06-4F0F-B524-25D0F6FB4FFF@stufft.io> References: <5228C00F.7000209@jcea.es> <20130905181231.GA25306@iskra.aviel.ru> <8593FDCE-E3E0-4AD4-897E-2FB1D751086C@stufft.io> <20130905182522.GA26090@iskra.aviel.ru> <2C429087-4E06-4F0F-B524-25D0F6FB4FFF@stufft.io> Message-ID: <20130905184337.GA26926@iskra.aviel.ru> On Thu, Sep 05, 2013 at 02:35:16PM -0400, Donald Stufft <donald at stufft.io> wrote: > Persona is the logical successor to OpenID. OpenID lived a short life and died a quiet death. I'm afraid Persona wouldn't live even that much. Dead-born idea, in my so humble opinion. Oleg. -- Oleg Broytman http://phdru.name/ phd at phdru.name Programmers don't die, they just GOSUB without RETURN. From eliben at gmail.com Thu Sep 5 20:45:01 2013 From: eliben at gmail.com (Eli Bendersky) Date: Thu, 5 Sep 2013 11:45:01 -0700 Subject: [Python-Dev] Offtopic: OpenID Providers In-Reply-To: <5228C00F.7000209@jcea.es> References: <5228C00F.7000209@jcea.es> Message-ID: <CAF-Rda_aajDr71ZU=xJ8m1diBLuG_jr__TO6tR8kAMS625q7hg@mail.gmail.com> On Thu, Sep 5, 2013 at 10:31 AM, Jesus Cea <jcea at jcea.es> wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > I just received an email from my OpenID provider, "myOpenID", saying > that they drop OpenID service next February. I wonder what other > OpenID providers are used by other python-dev fellows. > > What are you using?. bugs.python.org admins could share some data? > > I agree than OpenID is (quite) dead, but I rather prefer OpenID to use > user/pass. I have big hopes for Mozilla Persona, looking forward > Python infrastructure support :). > > PS: I use "http://www.jcea.es/" as my OpenID identity, and I delegate > the actual service to "myOpenID". I can switch delegation trivially. > http://bugs.python.org/?@action=openid_login&provider=Google Eli -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130905/3d56e85c/attachment.html> From donald at stufft.io Thu Sep 5 20:50:44 2013 From: donald at stufft.io (Donald Stufft) Date: Thu, 5 Sep 2013 14:50:44 -0400 Subject: [Python-Dev] Offtopic: OpenID Providers In-Reply-To: <20130905184337.GA26926@iskra.aviel.ru> References: <5228C00F.7000209@jcea.es> <20130905181231.GA25306@iskra.aviel.ru> <8593FDCE-E3E0-4AD4-897E-2FB1D751086C@stufft.io> <20130905182522.GA26090@iskra.aviel.ru> <2C429087-4E06-4F0F-B524-25D0F6FB4FFF@stufft.io> <20130905184337.GA26926@iskra.aviel.ru> Message-ID: <667454FE-47D1-46BB-99B6-4C0B10986269@stufft.io> On Sep 5, 2013, at 2:43 PM, Oleg Broytman <phd at phdru.name> wrote: > On Thu, Sep 05, 2013 at 02:35:16PM -0400, Donald Stufft <donald at stufft.io> wrote: >> Persona is the logical successor to OpenID. > > OpenID lived a short life and died a quiet death. I'm afraid Persona > wouldn't live even that much. Dead-born idea, in my so humble opinion. I don't think there's much evidence to support this. I'm seeing more sites support Persona not less. It solves some of the major problems with OpenID. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: <http://mail.python.org/pipermail/python-dev/attachments/20130905/dedf861c/attachment.sig> From barry at python.org Thu Sep 5 20:53:43 2013 From: barry at python.org (Barry Warsaw) Date: Thu, 5 Sep 2013 14:53:43 -0400 Subject: [Python-Dev] Offtopic: OpenID Providers In-Reply-To: <20130905183347.GO10747@unaka.lan> References: <5228C00F.7000209@jcea.es> <20130905181231.GA25306@iskra.aviel.ru> <8593FDCE-E3E0-4AD4-897E-2FB1D751086C@stufft.io> <20130905182522.GA26090@iskra.aviel.ru> <20130905183347.GO10747@unaka.lan> Message-ID: <20130905145343.08d40db8@anarchist> On Sep 05, 2013, at 11:33 AM, Toshio Kuratomi wrote: >Sortof.... The way OAuth looks to me, it's designed to prove that a given >client is authorized to perform an action. It's not designed to prove that >the given client is a specific person. In some cases, you really want to >know the latter and not merely the former. So I think in these situations >Donald's separation of Authz and Authn makes sense. This probably isn't the only application of these technologies, but I've always thought about OAuth as delegating authority to scripts and programs to act on your behalf. For example, you can write a script to interact with Launchpad's REST API, but before you can use the script, you have to interact with the web ui once (since your browser is trusted, presumably) to receive a token which the script can then use to prove that it's acting on your behalf. If at some point you stop trusting that script, you can revoke the token to disable its access, without having to reset your password. To me, OpenID is about logging into web sites using single-sign on. For example, once I've logged into Launchpad, I can essentially go anywhere that accepts OpenID, type my OpenID and generally not have to log in again (things like two-factor auth and such may change that interaction pattern). Or to summarize to a rough approximation: OpenID is for logins, OAuth is for scripts. Persona seems to fit the OpenID use case. You'd still want OAuth for scripting. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: <http://mail.python.org/pipermail/python-dev/attachments/20130905/9a5db88e/attachment-0001.sig> From skip at pobox.com Thu Sep 5 21:07:11 2013 From: skip at pobox.com (Skip Montanaro) Date: Thu, 5 Sep 2013 14:07:11 -0500 Subject: [Python-Dev] Offtopic: OpenID Providers In-Reply-To: <667454FE-47D1-46BB-99B6-4C0B10986269@stufft.io> References: <5228C00F.7000209@jcea.es> <20130905181231.GA25306@iskra.aviel.ru> <8593FDCE-E3E0-4AD4-897E-2FB1D751086C@stufft.io> <20130905182522.GA26090@iskra.aviel.ru> <2C429087-4E06-4F0F-B524-25D0F6FB4FFF@stufft.io> <20130905184337.GA26926@iskra.aviel.ru> <667454FE-47D1-46BB-99B6-4C0B10986269@stufft.io> Message-ID: <CANc-5UxNJQp46dA2b54Ad-rifUGv8OCL1H2Yky1gr8EqYxcABA@mail.gmail.com> >> On Thu, Sep 05, 2013 at 02:35:16PM -0400, Donald Stufft <donald at stufft.io> wrote: >>> Persona is the logical successor to OpenID. >> >> OpenID lived a short life and died a quiet death. I'm afraid Persona >> wouldn't live even that much. Dead-born idea, in my so humble opinion. > > I don't think there's much evidence to support this. I'm seeing more sites support Persona > not less. It solves some of the major problems with OpenID. I was completely unaware of OpenID's demise. Can someone point me to/provide an explanation? I much prefer using OpenID to login to a site than having to either come up with yet another username/password which I will just forget, or using Facebook or similar (I don't really trust them with my info). Thx, Skip From solipsis at pitrou.net Thu Sep 5 21:08:18 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 5 Sep 2013 21:08:18 +0200 Subject: [Python-Dev] Offtopic: OpenID Providers References: <5228C00F.7000209@jcea.es> Message-ID: <20130905210818.5fd499f8@fsol> On Thu, 05 Sep 2013 19:31:59 +0200 Jesus Cea <jcea at jcea.es> wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > I just received an email from my OpenID provider, "myOpenID", saying > that they drop OpenID service next February. I wonder what other > OpenID providers are used by other python-dev fellows. I use a self-hosted SimpleID instance: > > What are you using?. bugs.python.org admins could share some data? > > I agree than OpenID is (quite) dead, but I rather prefer OpenID to use > user/pass. I have big hopes for Mozilla Persona, looking forward > Python infrastructure support :). > > PS: I use "http://www.jcea.es/" as my OpenID identity, and I delegate > the actual service to "myOpenID". I can switch delegation trivially. > > - -- > Jes?s Cea Avi?n _/_/ _/_/_/ _/_/_/ > jcea at jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ > Twitter: @jcea _/_/ _/_/ _/_/_/_/_/ > jabber / xmpp:jcea at jabber.org _/_/ _/_/ _/_/ _/_/ _/_/ > "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ > "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ > "El amor es poner tu felicidad en la felicidad de otro" - Leibniz > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.10 (GNU/Linux) > Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ > > iQCVAwUBUijAD5lgi5GaxT1NAQLH0wQAkKDORrAtfJFzzIHl3lHRp7GfxOzdqdNP > uiuW65l/pas+p9+B0G6qR6EE2AAL7YPozcNF5AkmuGmxkpyn/JyUYKJcUWmUotpj > V9Buz9jz3qpPuv7AlTnMbjBBQK4YTYenbdk2HgI41SVQHZHkU/+y4CL3Y1hWyJNo > C8CCWfR0VlA= > =YXIq > -----END PGP SIGNATURE----- From solipsis at pitrou.net Thu Sep 5 21:10:51 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 5 Sep 2013 21:10:51 +0200 Subject: [Python-Dev] Offtopic: OpenID Providers References: <5228C00F.7000209@jcea.es> Message-ID: <20130905211051.307f38ce@fsol> On Thu, 05 Sep 2013 19:31:59 +0200 Jesus Cea <jcea at jcea.es> wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > I just received an email from my OpenID provider, "myOpenID", saying > that they drop OpenID service next February. I wonder what other > OpenID providers are used by other python-dev fellows. > > What are you using?. bugs.python.org admins could share some data? I use a self-hosted SimpleID instance: http://simpleid.koinic.net/ It works fine with python.org, except recent problems with PyPI for which I haven't had an answer yet: https://mail.python.org/pipermail/distutils-sig/2013-September/022583.html (sorry for the previous message, keyboard mishap) Regards Antoine. From a.badger at gmail.com Thu Sep 5 21:15:30 2013 From: a.badger at gmail.com (Toshio Kuratomi) Date: Thu, 5 Sep 2013 12:15:30 -0700 Subject: [Python-Dev] Offtopic: OpenID Providers In-Reply-To: <20130905145343.08d40db8@anarchist> References: <5228C00F.7000209@jcea.es> <20130905181231.GA25306@iskra.aviel.ru> <8593FDCE-E3E0-4AD4-897E-2FB1D751086C@stufft.io> <20130905182522.GA26090@iskra.aviel.ru> <20130905183347.GO10747@unaka.lan> <20130905145343.08d40db8@anarchist> Message-ID: <20130905191529.GP10747@unaka.lan> On Thu, Sep 05, 2013 at 02:53:43PM -0400, Barry Warsaw wrote: > > This probably isn't the only application of these technologies, but I've > always thought about OAuth as delegating authority to scripts and programs to > act on your behalf. For example, you can write a script to interact with > Launchpad's REST API, but before you can use the script, you have to interact > with the web ui once (since your browser is trusted, presumably) to receive a > token which the script can then use to prove that it's acting on your behalf. > If at some point you stop trusting that script, you can revoke the token to > disable its access, without having to reset your password. > > To me, OpenID is about logging into web sites using single-sign on. For > example, once I've logged into Launchpad, I can essentially go anywhere that > accepts OpenID, type my OpenID and generally not have to log in again (things > like two-factor auth and such may change that interaction pattern). > > Or to summarize to a rough approximation: OpenID is for logins, OAuth is for > scripts. > > Persona seems to fit the OpenID use case. You'd still want OAuth for > scripting. > <nod> However, in some cases, Persona/OpenID can make more sense for scripts. For instance, if you have a script that is primarily interactive in nature, it may be better to have the user login via that script than to have an OAuth token laying around on the filesystem all the time (Contrariwise, if the script is primarily run from cron or similar, it's better to have a token with limited permissions laying around on the filesystem than your OpenID password ;-) It's probably also useful to point out that OAuth (because it was developed to let third party websites have limited permission to act on your behalf) is more paranoid than strictly required for many scripts where that "third-party" is a script that you've written running on a box that you control. If that's the main use case for your service, OAuth may not be a good fit for your authz needs. -Toshio -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 198 bytes Desc: not available URL: <http://mail.python.org/pipermail/python-dev/attachments/20130905/b8ea2fbb/attachment.sig> From solipsis at pitrou.net Thu Sep 5 21:07:33 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 5 Sep 2013 21:07:33 +0200 Subject: [Python-Dev] Offtopic: OpenID Providers References: <5228C00F.7000209@jcea.es> <20130905181231.GA25306@iskra.aviel.ru> <8593FDCE-E3E0-4AD4-897E-2FB1D751086C@stufft.io> <20130905182522.GA26090@iskra.aviel.ru> <2C429087-4E06-4F0F-B524-25D0F6FB4FFF@stufft.io> <20130905184337.GA26926@iskra.aviel.ru> <667454FE-47D1-46BB-99B6-4C0B10986269@stufft.io> Message-ID: <20130905210733.29975b0a@fsol> On Thu, 5 Sep 2013 14:50:44 -0400 Donald Stufft <donald at stufft.io> wrote: > > On Sep 5, 2013, at 2:43 PM, Oleg Broytman <phd at phdru.name> wrote: > > > On Thu, Sep 05, 2013 at 02:35:16PM -0400, Donald Stufft <donald at stufft.io> wrote: > >> Persona is the logical successor to OpenID. > > > > OpenID lived a short life and died a quiet death. I'm afraid Persona > > wouldn't live even that much. Dead-born idea, in my so humble opinion. > > I don't think there's much evidence to support this. I'm seeing more sites support Persona > not less. Which sites exactly? I can login to BitBucket and *.python.org using OpenID, not Persona. From barry at python.org Thu Sep 5 21:40:44 2013 From: barry at python.org (Barry Warsaw) Date: Thu, 5 Sep 2013 15:40:44 -0400 Subject: [Python-Dev] Offtopic: OpenID Providers In-Reply-To: <20130905210733.29975b0a@fsol> References: <5228C00F.7000209@jcea.es> <20130905181231.GA25306@iskra.aviel.ru> <8593FDCE-E3E0-4AD4-897E-2FB1D751086C@stufft.io> <20130905182522.GA26090@iskra.aviel.ru> <2C429087-4E06-4F0F-B524-25D0F6FB4FFF@stufft.io> <20130905184337.GA26926@iskra.aviel.ru> <667454FE-47D1-46BB-99B6-4C0B10986269@stufft.io> <20130905210733.29975b0a@fsol> Message-ID: <20130905154044.44d93e67@anarchist> On Sep 05, 2013, at 09:07 PM, Antoine Pitrou wrote: >Which sites exactly? I can login to BitBucket and *.python.org using >OpenID, not Persona. I think Persona is just too new to see it around much yet. Or maybe Mozilla needs better PR. -Barry From skip at pobox.com Thu Sep 5 21:52:29 2013 From: skip at pobox.com (Skip Montanaro) Date: Thu, 5 Sep 2013 14:52:29 -0500 Subject: [Python-Dev] Offtopic: OpenID Providers In-Reply-To: <20130905154044.44d93e67@anarchist> References: <5228C00F.7000209@jcea.es> <20130905181231.GA25306@iskra.aviel.ru> <8593FDCE-E3E0-4AD4-897E-2FB1D751086C@stufft.io> <20130905182522.GA26090@iskra.aviel.ru> <2C429087-4E06-4F0F-B524-25D0F6FB4FFF@stufft.io> <20130905184337.GA26926@iskra.aviel.ru> <667454FE-47D1-46BB-99B6-4C0B10986269@stufft.io> <20130905210733.29975b0a@fsol> <20130905154044.44d93e67@anarchist> Message-ID: <CANc-5Uw6k5Susim4wAcPK3nUkV4ewOu+Y9c0iP7fSuSrfUPfNw@mail.gmail.com> > I think Persona is just too new to see it around much yet. Or maybe Mozilla > needs better PR. The Persona site touts: "Signing in using Persona requires only a valid email address; allowing you to provide personal information on as-needed basis, when and where you think it?s appropriate." They clearly need a better example site. They chose something called Voost. Sure enough, all I needed to enter was my Gmail address. That got me signed in, but then Voost asked me for a bunch of other personal information (name, gender, birthdate, etc), and wouldn't let me go any farther without that. :-/ Skip From ben+python at benfinney.id.au Thu Sep 5 21:53:25 2013 From: ben+python at benfinney.id.au (Ben Finney) Date: Fri, 06 Sep 2013 05:53:25 +1000 Subject: [Python-Dev] Offtopic: OpenID Providers References: <5228C00F.7000209@jcea.es> <20130905181231.GA25306@iskra.aviel.ru> <8593FDCE-E3E0-4AD4-897E-2FB1D751086C@stufft.io> <20130905182522.GA26090@iskra.aviel.ru> <2C429087-4E06-4F0F-B524-25D0F6FB4FFF@stufft.io> <20130905184337.GA26926@iskra.aviel.ru> <667454FE-47D1-46BB-99B6-4C0B10986269@stufft.io> <CANc-5UxNJQp46dA2b54Ad-rifUGv8OCL1H2Yky1gr8EqYxcABA@mail.gmail.com> Message-ID: <7wk3ivyrii.fsf@benfinney.id.au> Skip Montanaro <skip at pobox.com> writes: > >> On Thu, Sep 05, 2013 at 02:35:16PM -0400, Donald Stufft <donald at stufft.io> wrote: > >>> Persona is the logical successor to OpenID. > >> > >> OpenID lived a short life and died a quiet death. I'm afraid Persona > >> wouldn't live even that much. Dead-born idea, in my so humble opinion. > > > > I don't think there's much evidence to support this. I'm seeing more sites support Persona > > not less. It solves some of the major problems with OpenID. > > I was completely unaware of OpenID's demise. It has failed at its stated purpose, which was to obviate the need for services to provide their own ad hoc systems and allow users to consolidate their digital identities. This is evident by lookig at how few sites have added OpenID login in the past several years, and how many that once had it have dropped it. If you're unaware of that, I can only surmise you haven't been trying to log in with an OpenID to anything newer than about 2009. > Can someone point me to/provide an explanation? An explanation in terms of what? I can point you to punditry <URL:http://www.25hoursaday.com/weblog/2011/01/30/LearningFromOurMistakesTheFailureOfOpenIDAtomPubAndXMLOnTheWeb.aspx> and hand-wringing <URL:http://geekyschmidt.com/2011/01/31/openid-death-greatly-exaggerated>. My own take is that most people choose convenience and expedience over security and freedom, hence Facebook and Twitter and Google have taken over the online identity game instead of a federated identity system. > I much prefer using OpenID to login to a site than having to either > come up with yet another username/password which I will just forget, > or using Facebook or similar (I don't really trust them with my info). Agreed. Our preferences are not enough though. -- \ ?Anyone who believes exponential growth can go on forever in a | `\ finite world is either a madman or an economist.? ?Kenneth | _o__) Boulding | Ben Finney From phd at phdru.name Thu Sep 5 22:29:21 2013 From: phd at phdru.name (Oleg Broytman) Date: Fri, 6 Sep 2013 00:29:21 +0400 Subject: [Python-Dev] Offtopic: OpenID Providers In-Reply-To: <667454FE-47D1-46BB-99B6-4C0B10986269@stufft.io> References: <5228C00F.7000209@jcea.es> <20130905181231.GA25306@iskra.aviel.ru> <8593FDCE-E3E0-4AD4-897E-2FB1D751086C@stufft.io> <20130905182522.GA26090@iskra.aviel.ru> <2C429087-4E06-4F0F-B524-25D0F6FB4FFF@stufft.io> <20130905184337.GA26926@iskra.aviel.ru> <667454FE-47D1-46BB-99B6-4C0B10986269@stufft.io> Message-ID: <20130905202921.GA30331@iskra.aviel.ru> On Thu, Sep 05, 2013 at 02:50:44PM -0400, Donald Stufft <donald at stufft.io> wrote: > On Sep 5, 2013, at 2:43 PM, Oleg Broytman <phd at phdru.name> wrote: > > On Thu, Sep 05, 2013 at 02:35:16PM -0400, Donald Stufft <donald at stufft.io> wrote: > >> Persona is the logical successor to OpenID. > > > > OpenID lived a short life and died a quiet death. I'm afraid Persona > > wouldn't live even that much. Dead-born idea, in my so humble opinion. > > I don't think there's much evidence to support this. I'm seeing more sites support Persona > not less. It solves some of the major problems with OpenID. I have seen exactly 0 (zero) sites that support Persona. Can you point me? Oleg. -- Oleg Broytman http://phdru.name/ phd at phdru.name Programmers don't die, they just GOSUB without RETURN. From tseaver at palladion.com Thu Sep 5 22:30:31 2013 From: tseaver at palladion.com (Tres Seaver) Date: Thu, 05 Sep 2013 16:30:31 -0400 Subject: [Python-Dev] Offtopic: OpenID Providers In-Reply-To: <CANc-5Uw6k5Susim4wAcPK3nUkV4ewOu+Y9c0iP7fSuSrfUPfNw@mail.gmail.com> References: <5228C00F.7000209@jcea.es> <20130905181231.GA25306@iskra.aviel.ru> <8593FDCE-E3E0-4AD4-897E-2FB1D751086C@stufft.io> <20130905182522.GA26090@iskra.aviel.ru> <2C429087-4E06-4F0F-B524-25D0F6FB4FFF@stufft.io> <20130905184337.GA26926@iskra.aviel.ru> <667454FE-47D1-46BB-99B6-4C0B10986269@stufft.io> <20130905210733.29975b0a@fsol> <20130905154044.44d93e67@anarchist> <CANc-5Uw6k5Susim4wAcPK3nUkV4ewOu+Y9c0iP7fSuSrfUPfNw@mail.gmail.com> Message-ID: <l0apkv$m40$1@ger.gmane.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 09/05/2013 03:52 PM, Skip Montanaro wrote: >> I think Persona is just too new to see it around much yet. Or maybe >> Mozilla needs better PR. > > The Persona site touts: "Signing in using Persona requires only a > valid email address; allowing you to provide personal information on > as-needed basis, when and where you think it?s appropriate." > > They clearly need a better example site. They chose something called > Voost. Sure enough, all I needed to enter was my Gmail address. That > got me signed in, but then Voost asked me for a bunch of other > personal information (name, gender, birthdate, etc), and wouldn't let > me go any farther without that. :-/ As sith OpenID, the key element to Persona is SSO: you can authenticate without needing to create / remember passwords for every site you visit. Whether a given site chooses to authroize an authenticated-but-otherwise-unknown user to do anything meaningful is logically distinct. +1 for supporting Persona as an alternative to OpenID on all *.python.org servers. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlIo6ecACgkQ+gerLs4ltQ6gOwCgrIokRYnddNaNVIVPoY/M4d0k kKcAni6hxXQE4T4QMij3bQHAJwBFX1uW =9ZJP -----END PGP SIGNATURE----- From phd at phdru.name Thu Sep 5 22:36:51 2013 From: phd at phdru.name (Oleg Broytman) Date: Fri, 6 Sep 2013 00:36:51 +0400 Subject: [Python-Dev] Offtopic: OpenID Providers In-Reply-To: <CANc-5UxNJQp46dA2b54Ad-rifUGv8OCL1H2Yky1gr8EqYxcABA@mail.gmail.com> References: <5228C00F.7000209@jcea.es> <20130905181231.GA25306@iskra.aviel.ru> <8593FDCE-E3E0-4AD4-897E-2FB1D751086C@stufft.io> <20130905182522.GA26090@iskra.aviel.ru> <2C429087-4E06-4F0F-B524-25D0F6FB4FFF@stufft.io> <20130905184337.GA26926@iskra.aviel.ru> <667454FE-47D1-46BB-99B6-4C0B10986269@stufft.io> <CANc-5UxNJQp46dA2b54Ad-rifUGv8OCL1H2Yky1gr8EqYxcABA@mail.gmail.com> Message-ID: <20130905203651.GB30331@iskra.aviel.ru> On Thu, Sep 05, 2013 at 02:07:11PM -0500, Skip Montanaro <skip at pobox.com> wrote: > >> OpenID lived a short life and died a quiet death. I'm afraid Persona > >> wouldn't live even that much. Dead-born idea, in my so humble opinion. > > > I was completely unaware of OpenID's demise. There was no demise. Because there was no take-off. OpenID was never popular. I can remember a very limited set of major sites that allow login using OpenID: SourceForge, LiveJournal, BitBucket. The first two in the list allow limited login. BitBucket doesn't allow even that. They only allow full-featured login if you have already created an account and linked your OpenID URL with that account. You cannot login using OpenID to most interesting popular sites. GMail? No. Twitter? No. Facebook? FriendFeed? identi.ca? No, no, no. Small uninteresting blogs? Yes, but who cares? Oleg. -- Oleg Broytman http://phdru.name/ phd at phdru.name Programmers don't die, they just GOSUB without RETURN. From v+python at g.nevcal.com Thu Sep 5 22:33:48 2013 From: v+python at g.nevcal.com (Glenn Linderman) Date: Thu, 05 Sep 2013 13:33:48 -0700 Subject: [Python-Dev] Offtopic: OpenID Providers In-Reply-To: <l0apkv$m40$1@ger.gmane.org> References: <5228C00F.7000209@jcea.es> <20130905181231.GA25306@iskra.aviel.ru> <8593FDCE-E3E0-4AD4-897E-2FB1D751086C@stufft.io> <20130905182522.GA26090@iskra.aviel.ru> <2C429087-4E06-4F0F-B524-25D0F6FB4FFF@stufft.io> <20130905184337.GA26926@iskra.aviel.ru> <667454FE-47D1-46BB-99B6-4C0B10986269@stufft.io> <20130905210733.29975b0a@fsol> <20130905154044.44d93e67@anarchist> <CANc-5Uw6k5Susim4wAcPK3nUkV4ewOu+Y9c0iP7fSuSrfUPfNw@mail.gmail.com> <l0apkv$m40$1@ger.gmane.org> Message-ID: <5228EAAC.2020800@g.nevcal.com> On 9/5/2013 1:30 PM, Tres Seaver wrote: > +1 for supporting Persona as an alternative to OpenID on all *.python.org > servers. Is there a Python implementation of Persona I can install on my web server? -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130905/1e708f42/attachment-0001.html> From solipsis at pitrou.net Thu Sep 5 22:40:47 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 5 Sep 2013 22:40:47 +0200 Subject: [Python-Dev] Offtopic: OpenID Providers References: <5228C00F.7000209@jcea.es> <20130905181231.GA25306@iskra.aviel.ru> <8593FDCE-E3E0-4AD4-897E-2FB1D751086C@stufft.io> <20130905182522.GA26090@iskra.aviel.ru> <2C429087-4E06-4F0F-B524-25D0F6FB4FFF@stufft.io> <20130905184337.GA26926@iskra.aviel.ru> <667454FE-47D1-46BB-99B6-4C0B10986269@stufft.io> <20130905210733.29975b0a@fsol> <20130905154044.44d93e67@anarchist> Message-ID: <20130905224047.1867fdee@fsol> On Thu, 5 Sep 2013 15:40:44 -0400 Barry Warsaw <barry at python.org> wrote: > On Sep 05, 2013, at 09:07 PM, Antoine Pitrou wrote: > > >Which sites exactly? I can login to BitBucket and *.python.org using > >OpenID, not Persona. > > I think Persona is just too new to see it around much yet. Or maybe Mozilla > needs better PR. Well, OpenID at least got some publicity since it appeared. Persona is almost unknown at this point (though it was publicly launched two years ago, according to Wikipedia). Comparing the size of the respective Wikipedia pages actually tells quite a bit: http://en.wikipedia.org/wiki/OpenID http://en.wikipedia.org/wiki/Mozilla_Persona Regards Antoine. From fred at fdrake.net Thu Sep 5 22:37:25 2013 From: fred at fdrake.net (Fred Drake) Date: Thu, 5 Sep 2013 16:37:25 -0400 Subject: [Python-Dev] Offtopic: OpenID Providers In-Reply-To: <20130905202921.GA30331@iskra.aviel.ru> References: <5228C00F.7000209@jcea.es> <20130905181231.GA25306@iskra.aviel.ru> <8593FDCE-E3E0-4AD4-897E-2FB1D751086C@stufft.io> <20130905182522.GA26090@iskra.aviel.ru> <2C429087-4E06-4F0F-B524-25D0F6FB4FFF@stufft.io> <20130905184337.GA26926@iskra.aviel.ru> <667454FE-47D1-46BB-99B6-4C0B10986269@stufft.io> <20130905202921.GA30331@iskra.aviel.ru> Message-ID: <CAFT4OTFUsLBQFBW6nyFufq_3+eo68V80JwErYJ=KGVsevU0boQ@mail.gmail.com> On Thu, Sep 5, 2013 at 4:29 PM, Oleg Broytman <phd at phdru.name> wrote: > I have seen exactly 0 (zero) sites that support Persona. Can you > point me? We have an internal app that uses Persona, but we did that mostly to play with it. I've not run across any sites that use Persona in the wild, either. -Fred -- Fred L. Drake, Jr. <fred at fdrake.net> "A storm broke loose in my mind." --Albert Einstein From benjamin at python.org Thu Sep 5 22:45:57 2013 From: benjamin at python.org (Benjamin Peterson) Date: Thu, 5 Sep 2013 16:45:57 -0400 Subject: [Python-Dev] Offtopic: OpenID Providers In-Reply-To: <5228EAAC.2020800@g.nevcal.com> References: <5228C00F.7000209@jcea.es> <20130905181231.GA25306@iskra.aviel.ru> <8593FDCE-E3E0-4AD4-897E-2FB1D751086C@stufft.io> <20130905182522.GA26090@iskra.aviel.ru> <2C429087-4E06-4F0F-B524-25D0F6FB4FFF@stufft.io> <20130905184337.GA26926@iskra.aviel.ru> <667454FE-47D1-46BB-99B6-4C0B10986269@stufft.io> <20130905210733.29975b0a@fsol> <20130905154044.44d93e67@anarchist> <CANc-5Uw6k5Susim4wAcPK3nUkV4ewOu+Y9c0iP7fSuSrfUPfNw@mail.gmail.com> <l0apkv$m40$1@ger.gmane.org> <5228EAAC.2020800@g.nevcal.com> Message-ID: <CAPZV6o9DHy6LOWgP6zj+GsZjZyyf4K7SsN=qXmV-RArpaSMhuA@mail.gmail.com> There's some sample Python code here: https://developer.mozilla.org/en-US/docs/Mozilla/Persona/Quick_Setup The API is so simple something generic like requests suffices. 2013/9/5 Glenn Linderman <v+python at g.nevcal.com>: > On 9/5/2013 1:30 PM, Tres Seaver wrote: > > +1 for supporting Persona as an alternative to OpenID on all *.python.org > servers. > > > Is there a Python implementation of Persona I can install on my web server? > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/benjamin%40python.org > -- Regards, Benjamin From skip at pobox.com Thu Sep 5 22:42:17 2013 From: skip at pobox.com (Skip Montanaro) Date: Thu, 5 Sep 2013 15:42:17 -0500 Subject: [Python-Dev] Offtopic: OpenID Providers In-Reply-To: <l0apkv$m40$1@ger.gmane.org> References: <5228C00F.7000209@jcea.es> <20130905181231.GA25306@iskra.aviel.ru> <8593FDCE-E3E0-4AD4-897E-2FB1D751086C@stufft.io> <20130905182522.GA26090@iskra.aviel.ru> <2C429087-4E06-4F0F-B524-25D0F6FB4FFF@stufft.io> <20130905184337.GA26926@iskra.aviel.ru> <667454FE-47D1-46BB-99B6-4C0B10986269@stufft.io> <20130905210733.29975b0a@fsol> <20130905154044.44d93e67@anarchist> <CANc-5Uw6k5Susim4wAcPK3nUkV4ewOu+Y9c0iP7fSuSrfUPfNw@mail.gmail.com> <l0apkv$m40$1@ger.gmane.org> Message-ID: <CANc-5UzpJcwsCa_0ScBQ=T43vM8ac1UVrn60vq6m=We7SmiQGw@mail.gmail.com> > Whether a given site chooses to authroize an > authenticated-but-otherwise-unknown user to do anything meaningful is > logically distinct. But the least they could have done was pick a demo site that didn't do exactly what they contend you shouldn't need to do: cough up all sorts of personal information to use their site. Skip From barry at python.org Thu Sep 5 22:53:18 2013 From: barry at python.org (Barry Warsaw) Date: Thu, 5 Sep 2013 16:53:18 -0400 Subject: [Python-Dev] Offtopic: OpenID Providers In-Reply-To: <20130905203651.GB30331@iskra.aviel.ru> References: <5228C00F.7000209@jcea.es> <20130905181231.GA25306@iskra.aviel.ru> <8593FDCE-E3E0-4AD4-897E-2FB1D751086C@stufft.io> <20130905182522.GA26090@iskra.aviel.ru> <2C429087-4E06-4F0F-B524-25D0F6FB4FFF@stufft.io> <20130905184337.GA26926@iskra.aviel.ru> <667454FE-47D1-46BB-99B6-4C0B10986269@stufft.io> <CANc-5UxNJQp46dA2b54Ad-rifUGv8OCL1H2Yky1gr8EqYxcABA@mail.gmail.com> <20130905203651.GB30331@iskra.aviel.ru> Message-ID: <20130905165318.2fa40bab@anarchist> On Sep 06, 2013, at 12:36 AM, Oleg Broytman wrote: > You cannot login using OpenID to most interesting popular sites. >GMail? No. Twitter? No. Facebook? FriendFeed? identi.ca? No, no, no. I'd be surprised if you ever saw the big social networking sites support OpenID or Persona. They want to own that space themselves, so probably have no business incentive to support 3rd party systems. We're open source, and I think it benefits our mission to support open, decentralized, and free systems like OpenID and Persona. -Barry From donald at stufft.io Thu Sep 5 22:57:08 2013 From: donald at stufft.io (Donald Stufft) Date: Thu, 5 Sep 2013 16:57:08 -0400 Subject: [Python-Dev] Offtopic: OpenID Providers In-Reply-To: <20130905165318.2fa40bab@anarchist> References: <5228C00F.7000209@jcea.es> <20130905181231.GA25306@iskra.aviel.ru> <8593FDCE-E3E0-4AD4-897E-2FB1D751086C@stufft.io> <20130905182522.GA26090@iskra.aviel.ru> <2C429087-4E06-4F0F-B524-25D0F6FB4FFF@stufft.io> <20130905184337.GA26926@iskra.aviel.ru> <667454FE-47D1-46BB-99B6-4C0B10986269@stufft.io> <CANc-5UxNJQp46dA2b54Ad-rifUGv8OCL1H2Yky1gr8EqYxcABA@mail.gmail.com> <20130905203651.GB30331@iskra.aviel.ru> <20130905165318.2fa40bab@anarchist> Message-ID: <00E21A32-DEC0-4CB0-A9AA-CDE3E2307D86@stufft.io> On Sep 5, 2013, at 4:53 PM, Barry Warsaw <barry at python.org> wrote: > On Sep 06, 2013, at 12:36 AM, Oleg Broytman wrote: > >> You cannot login using OpenID to most interesting popular sites. >> GMail? No. Twitter? No. Facebook? FriendFeed? identi.ca? No, no, no. > > I'd be surprised if you ever saw the big social networking sites support > OpenID or Persona. They want to own that space themselves, so probably have > no business incentive to support 3rd party systems. Not that it changes this statement at all but you wouldn't expect to see a Persona login for gmail as persona solves the problem that people don't think of urls as personal identifiers by replacing it with emails. So Gmail would be the Persona IdP > > We're open source, and I think it benefits our mission to support open, > decentralized, and free systems like OpenID and Persona. > > -Barry > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: <http://mail.python.org/pipermail/python-dev/attachments/20130905/48e3cb8a/attachment.sig> From tseaver at palladion.com Thu Sep 5 22:58:19 2013 From: tseaver at palladion.com (Tres Seaver) Date: Thu, 05 Sep 2013 16:58:19 -0400 Subject: [Python-Dev] Offtopic: OpenID Providers In-Reply-To: <20130905202921.GA30331@iskra.aviel.ru> References: <5228C00F.7000209@jcea.es> <20130905181231.GA25306@iskra.aviel.ru> <8593FDCE-E3E0-4AD4-897E-2FB1D751086C@stufft.io> <20130905182522.GA26090@iskra.aviel.ru> <2C429087-4E06-4F0F-B524-25D0F6FB4FFF@stufft.io> <20130905184337.GA26926@iskra.aviel.ru> <667454FE-47D1-46BB-99B6-4C0B10986269@stufft.io> <20130905202921.GA30331@iskra.aviel.ru> Message-ID: <l0ar93$758$1@ger.gmane.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 09/05/2013 04:29 PM, Oleg Broytman wrote: > On Thu, Sep 05, 2013 at 02:50:44PM -0400, Donald Stufft > <donald at stufft.io> wrote: >> On Sep 5, 2013, at 2:43 PM, Oleg Broytman <phd at phdru.name> wrote: >>> On Thu, Sep 05, 2013 at 02:35:16PM -0400, Donald Stufft >>> <donald at stufft.io> wrote: >>>> Persona is the logical successor to OpenID. >>> >>> OpenID lived a short life and died a quiet death. I'm afraid >>> Persona wouldn't live even that much. Dead-born idea, in my so >>> humble opinion. >> >> I don't think there's much evidence to support this. I'm seeing more >> sites support Persona not less. It solves some of the major problems >> with OpenID. > > I have seen exactly 0 (zero) sites that support Persona. Can you point > me? - From the "Mozilla Identity" blog: - - https://webmaker.org/ - - http://bornthiswayfoundation.org/ - - http://firebase.com/ - - https://orionhub.org/ - - http://ting.com/ - - http://www.gnu.org/software/mailman/ - - http://discourse.org/ - - https://dailycred.com/ Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlIo8GoACgkQ+gerLs4ltQ5OtACdEv/XvYKwGuFQESuLn+uBkNzm UUAAn2YY22+YL+tS0WhE+95tIb0USmV7 =cB96 -----END PGP SIGNATURE----- From dirkjan at ochtman.nl Thu Sep 5 23:03:17 2013 From: dirkjan at ochtman.nl (Dirkjan Ochtman) Date: Thu, 5 Sep 2013 23:03:17 +0200 Subject: [Python-Dev] Offtopic: OpenID Providers In-Reply-To: <5228EAAC.2020800@g.nevcal.com> References: <5228C00F.7000209@jcea.es> <20130905181231.GA25306@iskra.aviel.ru> <8593FDCE-E3E0-4AD4-897E-2FB1D751086C@stufft.io> <20130905182522.GA26090@iskra.aviel.ru> <2C429087-4E06-4F0F-B524-25D0F6FB4FFF@stufft.io> <20130905184337.GA26926@iskra.aviel.ru> <667454FE-47D1-46BB-99B6-4C0B10986269@stufft.io> <20130905210733.29975b0a@fsol> <20130905154044.44d93e67@anarchist> <CANc-5Uw6k5Susim4wAcPK3nUkV4ewOu+Y9c0iP7fSuSrfUPfNw@mail.gmail.com> <l0apkv$m40$1@ger.gmane.org> <5228EAAC.2020800@g.nevcal.com> Message-ID: <CAKmKYaCeohRSmfjag3DBG7SAksh1pa2E6k8qQEesrHLx+C7=Ag@mail.gmail.com> On Thu, Sep 5, 2013 at 10:33 PM, Glenn Linderman <v+python at g.nevcal.com> wrote: > Is there a Python implementation of Persona I can install on my web server? If you mean to use your web server as an Identity Provider, try this: https://bitbucket.org/djc/persona-totp It currently only implements TOTP-based authentication (i.e. no passwords), but it should be easy to add a password or 2FA-mode if you'd prefer that. Cheers, Dirkjan From tseaver at palladion.com Thu Sep 5 23:01:23 2013 From: tseaver at palladion.com (Tres Seaver) Date: Thu, 05 Sep 2013 17:01:23 -0400 Subject: [Python-Dev] Offtopic: OpenID Providers In-Reply-To: <5228EAAC.2020800@g.nevcal.com> References: <5228C00F.7000209@jcea.es> <20130905181231.GA25306@iskra.aviel.ru> <8593FDCE-E3E0-4AD4-897E-2FB1D751086C@stufft.io> <20130905182522.GA26090@iskra.aviel.ru> <2C429087-4E06-4F0F-B524-25D0F6FB4FFF@stufft.io> <20130905184337.GA26926@iskra.aviel.ru> <667454FE-47D1-46BB-99B6-4C0B10986269@stufft.io> <20130905210733.29975b0a@fsol> <20130905154044.44d93e67@anarchist> <CANc-5Uw6k5Susim4wAcPK3nUkV4ewOu+Y9c0iP7fSuSrfUPfNw@mail.gmail.com> <l0apkv$m40$1@ger.gmane.org> <5228EAAC.2020800@g.nevcal.com> Message-ID: <l0ares$a70$1@ger.gmane.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 09/05/2013 04:33 PM, Glenn Linderman wrote: > On 9/5/2013 1:30 PM, Tres Seaver wrote: >> +1 for supporting Persona as an alternative to OpenID on all >> *.python.org servers. > > Is there a Python implementation of Persona I can install on my web > server? - - https://readthedocs.org/projects/django-browserid/ - - https://pyramid_persona.readthedocs.org/en/latest/ Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlIo8SMACgkQ+gerLs4ltQ57xgCfT2xVkJfMtuDjmed6jlhfsD8I fxMAoNHT69tbi3iKtpkyEcSY0lwJosqc =TTJ8 -----END PGP SIGNATURE----- From dirkjan at ochtman.nl Thu Sep 5 23:05:55 2013 From: dirkjan at ochtman.nl (Dirkjan Ochtman) Date: Thu, 5 Sep 2013 23:05:55 +0200 Subject: [Python-Dev] Offtopic: OpenID Providers In-Reply-To: <00E21A32-DEC0-4CB0-A9AA-CDE3E2307D86@stufft.io> References: <5228C00F.7000209@jcea.es> <20130905181231.GA25306@iskra.aviel.ru> <8593FDCE-E3E0-4AD4-897E-2FB1D751086C@stufft.io> <20130905182522.GA26090@iskra.aviel.ru> <2C429087-4E06-4F0F-B524-25D0F6FB4FFF@stufft.io> <20130905184337.GA26926@iskra.aviel.ru> <667454FE-47D1-46BB-99B6-4C0B10986269@stufft.io> <CANc-5UxNJQp46dA2b54Ad-rifUGv8OCL1H2Yky1gr8EqYxcABA@mail.gmail.com> <20130905203651.GB30331@iskra.aviel.ru> <20130905165318.2fa40bab@anarchist> <00E21A32-DEC0-4CB0-A9AA-CDE3E2307D86@stufft.io> Message-ID: <CAKmKYaDjssPK4F_=9vrC4687aP=BpjzYgbq1cygJ9fTVXXpAMg@mail.gmail.com> On Thu, Sep 5, 2013 at 10:57 PM, Donald Stufft <donald at stufft.io> wrote: > Not that it changes this statement at all but you wouldn't expect to see a Persona login > for gmail as persona solves the problem that people don't think of urls as personal > identifiers by replacing it with emails. So Gmail would be the Persona IdP And actually you can already trivially login with your GMail account on any Persona-based Relying Party (that is, a site that uses Persona to authenticate you). This is because one of the nice parts of the current implementation of Persona is that Mozilla has implemented bridges that allow GMail and Yahoo addresses to be authenticated via their respective OAuth implementations, such that you don't need to setup an account at Mozilla's fallback IdP (which acts as an Identity Provider for email addresses that don't currently have an IdP available to them). Cheers, Dirkjan From dirkjan at ochtman.nl Thu Sep 5 23:08:13 2013 From: dirkjan at ochtman.nl (Dirkjan Ochtman) Date: Thu, 5 Sep 2013 23:08:13 +0200 Subject: [Python-Dev] Offtopic: OpenID Providers In-Reply-To: <l0apkv$m40$1@ger.gmane.org> References: <5228C00F.7000209@jcea.es> <20130905181231.GA25306@iskra.aviel.ru> <8593FDCE-E3E0-4AD4-897E-2FB1D751086C@stufft.io> <20130905182522.GA26090@iskra.aviel.ru> <2C429087-4E06-4F0F-B524-25D0F6FB4FFF@stufft.io> <20130905184337.GA26926@iskra.aviel.ru> <667454FE-47D1-46BB-99B6-4C0B10986269@stufft.io> <20130905210733.29975b0a@fsol> <20130905154044.44d93e67@anarchist> <CANc-5Uw6k5Susim4wAcPK3nUkV4ewOu+Y9c0iP7fSuSrfUPfNw@mail.gmail.com> <l0apkv$m40$1@ger.gmane.org> Message-ID: <CAKmKYaBVYE6HN2XDgHNT4b5gHuQ3W5GzL8mr5x_b8vqLtZs-gg@mail.gmail.com> On Thu, Sep 5, 2013 at 10:30 PM, Tres Seaver <tseaver at palladion.com> wrote: > +1 for supporting Persona as an alternative to OpenID on all *.python.org > servers. BTW, I'd be happy to assist with any Persona RP implementations for python.org services. The MDN docs are pretty good, I got my first RP going in just a few hours of looking at code (and you can probably do better if you're more into frontend webdev stuff). There's also ongoing work that will replace ReadTheDocs accounts with Persona support. Cheers, Dirkjan From phd at phdru.name Thu Sep 5 23:09:24 2013 From: phd at phdru.name (Oleg Broytman) Date: Fri, 6 Sep 2013 01:09:24 +0400 Subject: [Python-Dev] Offtopic: OpenID Providers In-Reply-To: <20130905165318.2fa40bab@anarchist> References: <5228C00F.7000209@jcea.es> <20130905181231.GA25306@iskra.aviel.ru> <8593FDCE-E3E0-4AD4-897E-2FB1D751086C@stufft.io> <20130905182522.GA26090@iskra.aviel.ru> <2C429087-4E06-4F0F-B524-25D0F6FB4FFF@stufft.io> <20130905184337.GA26926@iskra.aviel.ru> <667454FE-47D1-46BB-99B6-4C0B10986269@stufft.io> <CANc-5UxNJQp46dA2b54Ad-rifUGv8OCL1H2Yky1gr8EqYxcABA@mail.gmail.com> <20130905203651.GB30331@iskra.aviel.ru> <20130905165318.2fa40bab@anarchist> Message-ID: <20130905210924.GA30649@iskra.aviel.ru> On Thu, Sep 05, 2013 at 04:53:18PM -0400, Barry Warsaw <barry at python.org> wrote: > On Sep 06, 2013, at 12:36 AM, Oleg Broytman wrote: > > You cannot login using OpenID to most interesting popular sites. > >GMail? No. Twitter? No. Facebook? FriendFeed? identi.ca? No, no, no. > > I'd be surprised if you ever saw the big social networking sites support > OpenID or Persona. They want to own that space themselves, so probably have > no business incentive to support 3rd party systems. But of course! And that IMO spells the end of the feature. Things that aren't available for millions seldom are available for a few, and if they are -- they are available for big price. > We're open source, and I think it benefits our mission to support open, > decentralized, and free systems like OpenID and Persona. But they also have disadvantages. Implementing such a major feature is a significant burden to sysadmins and is an additional vein for security breaches. That said, I don't mind if pydotorg would get such features. If FSF pays salaries and admins are willing to work -- no objections from me. But I am not going to use it. What gain if I can login to one site? I will change my mind when Google and GitHub start using them. Oleg. -- Oleg Broytman http://phdru.name/ phd at phdru.name Programmers don't die, they just GOSUB without RETURN. From phd at phdru.name Thu Sep 5 23:25:10 2013 From: phd at phdru.name (Oleg Broytman) Date: Fri, 6 Sep 2013 01:25:10 +0400 Subject: [Python-Dev] Offtopic: OpenID Providers In-Reply-To: <l0ar93$758$1@ger.gmane.org> References: <5228C00F.7000209@jcea.es> <20130905181231.GA25306@iskra.aviel.ru> <8593FDCE-E3E0-4AD4-897E-2FB1D751086C@stufft.io> <20130905182522.GA26090@iskra.aviel.ru> <2C429087-4E06-4F0F-B524-25D0F6FB4FFF@stufft.io> <20130905184337.GA26926@iskra.aviel.ru> <667454FE-47D1-46BB-99B6-4C0B10986269@stufft.io> <20130905202921.GA30331@iskra.aviel.ru> <l0ar93$758$1@ger.gmane.org> Message-ID: <20130905212510.GB30649@iskra.aviel.ru> On Thu, Sep 05, 2013 at 04:58:19PM -0400, Tres Seaver <tseaver at palladion.com> wrote: > On 09/05/2013 04:29 PM, Oleg Broytman wrote: > > I have seen exactly 0 (zero) sites that support Persona. Can you point > > me? > > - From the "Mozilla Identity" blog: > > - - https://webmaker.org/ > - - http://bornthiswayfoundation.org/ > - - http://firebase.com/ > - - https://orionhub.org/ > - - http://ting.com/ > - - http://www.gnu.org/software/mailman/ > - - http://discourse.org/ > - - https://dailycred.com/ Thank you! Never heard of these sites. Well, I saw WebMaker once, but it's a Mozilla site, no wonder it supports Persona. Oleg. -- Oleg Broytman http://phdru.name/ phd at phdru.name Programmers don't die, they just GOSUB without RETURN. From barry at python.org Thu Sep 5 23:29:07 2013 From: barry at python.org (Barry Warsaw) Date: Thu, 5 Sep 2013 17:29:07 -0400 Subject: [Python-Dev] Offtopic: OpenID Providers In-Reply-To: <20130905210924.GA30649@iskra.aviel.ru> References: <5228C00F.7000209@jcea.es> <20130905181231.GA25306@iskra.aviel.ru> <8593FDCE-E3E0-4AD4-897E-2FB1D751086C@stufft.io> <20130905182522.GA26090@iskra.aviel.ru> <2C429087-4E06-4F0F-B524-25D0F6FB4FFF@stufft.io> <20130905184337.GA26926@iskra.aviel.ru> <667454FE-47D1-46BB-99B6-4C0B10986269@stufft.io> <CANc-5UxNJQp46dA2b54Ad-rifUGv8OCL1H2Yky1gr8EqYxcABA@mail.gmail.com> <20130905203651.GB30331@iskra.aviel.ru> <20130905165318.2fa40bab@anarchist> <20130905210924.GA30649@iskra.aviel.ru> Message-ID: <20130905172907.67a08d29@anarchist> On Sep 06, 2013, at 01:09 AM, Oleg Broytman wrote: >I will change my mind when Google and GitHub start using them. Neither Google nor GitHub are free or open. Bitbucket and Facebook aren't either. I'm not saying they're bad services because of that of course, but I don't want to have to rely on any of them to access python.org resources, and I don't want my choice to be log into Facebook or manage a slew of passwords. But I'm not volunteering to do the work, so I don't get to decide. I'm just stating that I think our principle should be that you *can* (not *must*) use free and open services to access our resources. -Barry From victor.stinner at gmail.com Thu Sep 5 23:30:03 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Thu, 5 Sep 2013 23:30:03 +0200 Subject: [Python-Dev] RFC: PEP 454: Add a new tracemalloc module In-Reply-To: <CAP7h-xauzgw8UJqKJAb2WPMqPM9-_BvDhHgcj5RnKvj5yaMiYw@mail.gmail.com> References: <CAMpsgwZTOBdXy2Rcxs0itnPZYZgHU4X4tPNf5syoZrOJ5gYCFw@mail.gmail.com> <CAP7h-xauzgw8UJqKJAb2WPMqPM9-_BvDhHgcj5RnKvj5yaMiYw@mail.gmail.com> Message-ID: <CAMpsgwaXPgNdvtrajVX2AJ5+61JcSPA_vYPZbQYi=e+TGA-Jrw@mail.gmail.com> 2013/9/5 Alexander Belopolsky <alexander.belopolsky at gmail.com>: > Please mention that this API is similar to that of faulthandler and add a > link to faulthandler docs. Done. Victor From victor.stinner at gmail.com Thu Sep 5 23:30:53 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Thu, 5 Sep 2013 23:30:53 +0200 Subject: [Python-Dev] Add a new tracemalloc module to trace memory allocations In-Reply-To: <CADiSq7dKojZ5f1bprhZZxu6DjovjWtwy-8o=-BYjYRvHZWqoFw@mail.gmail.com> References: <CAMpsgwaa3pABNrMSw2S7-Og111u8DdXK6dYnWVJzS4TAKbFNEQ@mail.gmail.com> <CAMpsgwatDdFQg4q=b3s7W6BZ+WQmpo=KTeJZd7ESbuGG+RzaKA@mail.gmail.com> <CAGE7PNJ9DrKQ8Zb-E-F7QPRTP=t+S8StBvm3OQgvct8539gFPg@mail.gmail.com> <CAMpsgwZBeN7W+GK6hLSHp4VYXAYgWFQBTx_-K3J7KzOMDB3H=g@mail.gmail.com> <CADiSq7dKojZ5f1bprhZZxu6DjovjWtwy-8o=-BYjYRvHZWqoFw@mail.gmail.com> Message-ID: <CAMpsgwa5YaeCXCRZrZcM=+V+0xSJhj246zJtP-MUqtaggfALYA@mail.gmail.com> 2013/9/1 Nick Coghlan <ncoghlan at gmail.com>: > +1 from me for both tracemalloc and failmalloc in the same vein as > faulthandler (and using similar API concepts to faulthandler). > > While I like the flat top level naming structure, we should clearly document > these as implementation dependent runtime services. Other implementations > may not provide them at all, and even if they do, many details will likely > be different. > > The gc module may even fall into the same category. I updated the PEP 454 to mention that tracemalloc has been written for CPython. Victor From victor.stinner at gmail.com Thu Sep 5 23:37:52 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Thu, 5 Sep 2013 23:37:52 +0200 Subject: [Python-Dev] Add a new tracemalloc module to trace memory allocations In-Reply-To: <CAGE7PNJ9DrKQ8Zb-E-F7QPRTP=t+S8StBvm3OQgvct8539gFPg@mail.gmail.com> References: <CAMpsgwaa3pABNrMSw2S7-Og111u8DdXK6dYnWVJzS4TAKbFNEQ@mail.gmail.com> <CAMpsgwatDdFQg4q=b3s7W6BZ+WQmpo=KTeJZd7ESbuGG+RzaKA@mail.gmail.com> <CAGE7PNJ9DrKQ8Zb-E-F7QPRTP=t+S8StBvm3OQgvct8539gFPg@mail.gmail.com> Message-ID: <CAMpsgwbGNUCtNjkR-+1cSnoBF32s9tRw-b4k=q-+0nY+HWE_4g@mail.gmail.com> 2013/8/31 Gregory P. Smith <greg at krypto.org>: > Think of the possibilities, you could even setup a test runner to > enable/disable before and after each test, test suite or test module to > gather narrow statistics as to what code actually _caused_ the allocations > rather than the ultimate individual file/line doing it. It may be used with "-m test -R" to detect memory leaks, you need to repeat a test. > Taking that further: file and line information is great, but what if you > extend the concept: could you allow for C API or even Python hooks to gather > additional information at the time of each allocation or free? for example: > Gathering the actual C and Python stack traces for correlation to figure out > what call patterns lead allocations is powerful. I modified the PEP 454 and the implementation. By default, tracemalloc only stores 1 "frame" instance (filename and line number). But you can now use tracemalloc.set_number_frame(10) to store 10 frames per memory allocation. Don't store too much frames, remember that tracemalloc traces all Python memory allocations! So when Python allocates 1 byte, tracemalloc may need to allocate 1000 bytes to store the trace... tracemalloc cannot get the C traceback. It was discussed in the design of the PEP 445 (Add new APIs to customize Python memory allocators) how to pass the C filename and line number. It was decided to not add them to have a simpler API. In general, a C function is binded to a Python function. Knowing the Python function should be enough. Victor From phd at phdru.name Thu Sep 5 23:56:22 2013 From: phd at phdru.name (Oleg Broytman) Date: Fri, 6 Sep 2013 01:56:22 +0400 Subject: [Python-Dev] Offtopic: OpenID Providers In-Reply-To: <20130905172907.67a08d29@anarchist> References: <8593FDCE-E3E0-4AD4-897E-2FB1D751086C@stufft.io> <20130905182522.GA26090@iskra.aviel.ru> <2C429087-4E06-4F0F-B524-25D0F6FB4FFF@stufft.io> <20130905184337.GA26926@iskra.aviel.ru> <667454FE-47D1-46BB-99B6-4C0B10986269@stufft.io> <CANc-5UxNJQp46dA2b54Ad-rifUGv8OCL1H2Yky1gr8EqYxcABA@mail.gmail.com> <20130905203651.GB30331@iskra.aviel.ru> <20130905165318.2fa40bab@anarchist> <20130905210924.GA30649@iskra.aviel.ru> <20130905172907.67a08d29@anarchist> Message-ID: <20130905215622.GA31833@iskra.aviel.ru> On Thu, Sep 05, 2013 at 05:29:07PM -0400, Barry Warsaw <barry at python.org> wrote: > I don't want my choice to be log into Facebook or manage a slew of passwords. The last part is unavoidable. I regularly login to LiveJournal, Twitter, SourceForge, BitBucket, Gitorious, GitHub and to hundreds of other sites -- blogs, torrents, web shops. I already manage hundreds of passwords. OpenID promised to save me from that and failed. Do you think Persona would succeed in this regard (saved me from managing all those passwords)? And if not -- what are the benefits? I already manage hundreds of passwords -- two or three additional passwords for bugs.python.org, wiki.python.org and so on don't make the situation worse. IMO the very idea of single sign-on in the open web is meaningless. > But I'm not volunteering to do the work, so I don't get to decide. I'm just > stating that I think our principle should be that you *can* (not *must*) use > free and open services to access our resources. Well, I can only use services that are available, not those that are promised. If python.org grows support for Persona -- who will be my provider and for what price? I am not going to install and manage additional software on my servers -- I don't want to be my own provider, I have enough job already. Oleg. -- Oleg Broytman http://phdru.name/ phd at phdru.name Programmers don't die, they just GOSUB without RETURN. From donald at stufft.io Fri Sep 6 00:07:36 2013 From: donald at stufft.io (Donald Stufft) Date: Thu, 5 Sep 2013 18:07:36 -0400 Subject: [Python-Dev] Offtopic: OpenID Providers In-Reply-To: <20130905215622.GA31833@iskra.aviel.ru> References: <8593FDCE-E3E0-4AD4-897E-2FB1D751086C@stufft.io> <20130905182522.GA26090@iskra.aviel.ru> <2C429087-4E06-4F0F-B524-25D0F6FB4FFF@stufft.io> <20130905184337.GA26926@iskra.aviel.ru> <667454FE-47D1-46BB-99B6-4C0B10986269@stufft.io> <CANc-5UxNJQp46dA2b54Ad-rifUGv8OCL1H2Yky1gr8EqYxcABA@mail.gmail.com> <20130905203651.GB30331@iskra.aviel.ru> <20130905165318.2fa40bab@anarchist> <20130905210924.GA30649@iskra.aviel.ru> <20130905172907.67a08d29@anarchist> <20130905215622.GA31833@iskra.aviel.ru> Message-ID: <637EAFBD-9870-48BD-9206-A8B67B4DBCD3@stufft.io> On Sep 5, 2013, at 5:56 PM, Oleg Broytman <phd at phdru.name> wrote: > On Thu, Sep 05, 2013 at 05:29:07PM -0400, Barry Warsaw <barry at python.org> wrote: >> I don't want my choice to be log into Facebook or manage a slew of passwords. > > The last part is unavoidable. I regularly login to LiveJournal, > Twitter, SourceForge, BitBucket, Gitorious, GitHub and to hundreds of > other sites -- blogs, torrents, web shops. I already manage hundreds of > passwords. OpenID promised to save me from that and failed. Do you think > Persona would succeed in this regard (saved me from managing all those > passwords)? And if not -- what are the benefits? I already manage > hundreds of passwords -- two or three additional passwords for > bugs.python.org, wiki.python.org and so on don't make the situation > worse. > IMO the very idea of single sign-on in the open web is meaningless. > >> But I'm not volunteering to do the work, so I don't get to decide. I'm just >> stating that I think our principle should be that you *can* (not *must*) use >> free and open services to access our resources. > > Well, I can only use services that are available, not those that are > promised. If python.org grows support for Persona -- who will be my > provider and for what price? I am not going to install and manage > additional software on my servers -- I don't want to be my own provider, > I have enough job already. Theoretically whoever runs the domain for your email address (since Persona uses email as your identifier). In order to make it work as a stop gap they also have more openid like idP's which they run a major one (that also offers a "bridge" to make things like Gmail work on Persona without needing to register for anything else). > > Oleg. > -- > Oleg Broytman http://phdru.name/ phd at phdru.name > Programmers don't die, they just GOSUB without RETURN. > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: <http://mail.python.org/pipermail/python-dev/attachments/20130905/aadf01d0/attachment.sig> From stephen at xemacs.org Fri Sep 6 03:09:26 2013 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Fri, 06 Sep 2013 10:09:26 +0900 Subject: [Python-Dev] Offtopic: OpenID Providers In-Reply-To: <20130905165318.2fa40bab@anarchist> References: <5228C00F.7000209@jcea.es> <20130905181231.GA25306@iskra.aviel.ru> <8593FDCE-E3E0-4AD4-897E-2FB1D751086C@stufft.io> <20130905182522.GA26090@iskra.aviel.ru> <2C429087-4E06-4F0F-B524-25D0F6FB4FFF@stufft.io> <20130905184337.GA26926@iskra.aviel.ru> <667454FE-47D1-46BB-99B6-4C0B10986269@stufft.io> <CANc-5UxNJQp46dA2b54Ad-rifUGv8OCL1H2Yky1gr8EqYxcABA@mail.gmail.com> <20130905203651.GB30331@iskra.aviel.ru> <20130905165318.2fa40bab@anarchist> Message-ID: <87hadyah89.fsf@uwakimon.sk.tsukuba.ac.jp> Barry Warsaw writes: > On Sep 06, 2013, at 12:36 AM, Oleg Broytman wrote: > > You cannot login using OpenID to most interesting popular sites. > >GMail? No. Twitter? No. Facebook? FriendFeed? identi.ca? No, no, no. > > I'd be surprised if you ever saw the big social networking sites support > OpenID or Persona. They want to own that space themselves, so probably have > no business incentive to support 3rd party systems. Quite the reverse, unfortunately. That's why *those* sites *all* appear on most sites that support OpenID. They're not going to delegate to each other until they are forced to. > We're open source, and I think it benefits our mission to support open, > decentralized, and free systems like OpenID and Persona. Thus speaks an employee of yet another Provider-That-Won't-Accept-My- Third-Party-Credentials. Sorry, Barry, but you see the problem: Unfortunately, we can't do it alone. What needs to happen is there needs to be a large network of sites that support login via O-D-F systems like OpenID and Persona. Too many of the sites I use (news sources, GMail, etc) don't support them and my browser manages my logins to most of them, so why bother learning OpenID, and then setting it up site by site? I'm not against it, but it's quixotic (and therefore valuable). One reason that OpenID and Persona fail to achieve penetration is that they overstate their mission. A protocol that any email provider can support is a protocol that provides authentication without identification (imagine what havoc Dogbert could wreak with his own Persona provider), and therefore cannot be used in authorization (except trivially). Think ident (port tcp/113). And most general- audience sites that want to provide high-quality "Web 2.0" service are going to start by asking for your demographics. It's probably at least as effective as CAPTCHA for classifying mammals and 'bots, too! The reason that the "big" providers can take advantage of these protocols as providers without reciprocating as clients is that identities on these sites are very valuable to at least 95% of people who use them (that may or may not correspond to as much as 50% of the accounts). Losing your Facebook site for abuse of TOS is very costly: you can't even contact your "circle" easily. Nor do you want multiple logins on one of these sites, because that will double the amount of spam they send you. Bottom line: A login via Facebook-provided OpenID means that the login is unlikely to perform random mischief. Of course, those issues are easy to deal with if you have even a bit of Internet savvy. So sites still have to worry about a deliberate attack from a Facebook user, but a serious intruder has many ways to get in the front door, so you need to lock up your Waterford crystal and Noritake china anyway whether you support global ID logins or not. From jcea at jcea.es Fri Sep 6 03:43:53 2013 From: jcea at jcea.es (Jesus Cea) Date: Fri, 06 Sep 2013 03:43:53 +0200 Subject: [Python-Dev] Offtopic: OpenID Providers In-Reply-To: <7wk3ivyrii.fsf@benfinney.id.au> References: <5228C00F.7000209@jcea.es> <20130905181231.GA25306@iskra.aviel.ru> <8593FDCE-E3E0-4AD4-897E-2FB1D751086C@stufft.io> <20130905182522.GA26090@iskra.aviel.ru> <2C429087-4E06-4F0F-B524-25D0F6FB4FFF@stufft.io> <20130905184337.GA26926@iskra.aviel.ru> <667454FE-47D1-46BB-99B6-4C0B10986269@stufft.io> <CANc-5UxNJQp46dA2b54Ad-rifUGv8OCL1H2Yky1gr8EqYxcABA@mail.gmail.com> <7wk3ivyrii.fsf@benfinney.id.au> Message-ID: <52293359.4080304@jcea.es> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 05/09/13 21:53, Ben Finney wrote: > My own take is that most people choose convenience and expedience > over security and freedom, hence Facebook and Twitter and Google > have taken over the online identity game instead of a federated > identity system. That is one of the Persona improvements: If your email provider is not supporting it, you can still use your email with "persona" (thru Mozilla servers). If you provider support OAUTH authentication (let say, Facebook, twitter, Google), You can use that identity to prove your identity to Mozilla, and Mozilla to prove your email ID to any Persona consumer. In the process, you get privacy (facebook doesn't know where are you using authentication, beside Mozilla). Being a Persona provider is easy, being a verifier is trivial. An interesting property of Persona is that if its popularity grows, it becomes decentralized "autom?gically". Anyway, I was asking for alternative OpenID providers, not to open a debate about single sign on methods :). - -- Jes?s Cea Avi?n _/_/ _/_/_/ _/_/_/ jcea at jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ Twitter: @jcea _/_/ _/_/ _/_/_/_/_/ jabber / xmpp:jcea at jabber.org _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQCVAwUBUikzWZlgi5GaxT1NAQJMbgP/enbARAwUiDXzJ9PkBmvlAi/OrhXrrXwE Vf1f6XHab+ERyJ6kj1V5yz6F0D0zxl6s7GL+abz7P0JEdGEGMfEJc+aK15HM2r3b LAm9k5ofe+ysdJx0HEd/V6/viqAeK7medb5Hh3xNwxbY6qRe4VkXbAvc0KQuo7eR 1Uf005ibUT8= =i0/D -----END PGP SIGNATURE----- From jcea at jcea.es Fri Sep 6 03:51:30 2013 From: jcea at jcea.es (Jesus Cea) Date: Fri, 06 Sep 2013 03:51:30 +0200 Subject: [Python-Dev] Offtopic: OpenID Providers In-Reply-To: <20130905203651.GB30331@iskra.aviel.ru> References: <5228C00F.7000209@jcea.es> <20130905181231.GA25306@iskra.aviel.ru> <8593FDCE-E3E0-4AD4-897E-2FB1D751086C@stufft.io> <20130905182522.GA26090@iskra.aviel.ru> <2C429087-4E06-4F0F-B524-25D0F6FB4FFF@stufft.io> <20130905184337.GA26926@iskra.aviel.ru> <667454FE-47D1-46BB-99B6-4C0B10986269@stufft.io> <CANc-5UxNJQp46dA2b54Ad-rifUGv8OCL1H2Yky1gr8EqYxcABA@mail.gmail.com> <20130905203651.GB30331@iskra.aviel.ru> Message-ID: <52293522.2060304@jcea.es> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 05/09/13 22:36, Oleg Broytman wrote: > There was no demise. Because there was no take-off. OpenID was > never popular. I can remember a very limited set of major sites > that allow login using OpenID: SourceForge, LiveJournal, BitBucket. > The first two I remember a day where OpenID was the ONLY authentication method in StackOverflow. - -- Jes?s Cea Avi?n _/_/ _/_/_/ _/_/_/ jcea at jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ Twitter: @jcea _/_/ _/_/ _/_/_/_/_/ jabber / xmpp:jcea at jabber.org _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQCVAwUBUik1Iplgi5GaxT1NAQKHGQP/ayFi3hs5CWwlNCciFguNkUesQBfRaPFF TxHrloNM2Uo9wB2dt2oAVhAQnGSTw3lIYoRMzhk5+tfx4Bn16QVPiDXnbotO5xqn QkAH3ZE9q9/YTDDqPJPFXHP/7eoQwk9Ou4LrTJk97ofp0DM7JcfVm3W0Sys53wEQ ddwoNkrIk/s= =vFd+ -----END PGP SIGNATURE----- From jcea at jcea.es Fri Sep 6 03:52:35 2013 From: jcea at jcea.es (Jesus Cea) Date: Fri, 06 Sep 2013 03:52:35 +0200 Subject: [Python-Dev] Offtopic: OpenID Providers In-Reply-To: <20130905215622.GA31833@iskra.aviel.ru> References: <8593FDCE-E3E0-4AD4-897E-2FB1D751086C@stufft.io> <20130905182522.GA26090@iskra.aviel.ru> <2C429087-4E06-4F0F-B524-25D0F6FB4FFF@stufft.io> <20130905184337.GA26926@iskra.aviel.ru> <667454FE-47D1-46BB-99B6-4C0B10986269@stufft.io> <CANc-5UxNJQp46dA2b54Ad-rifUGv8OCL1H2Yky1gr8EqYxcABA@mail.gmail.com> <20130905203651.GB30331@iskra.aviel.ru> <20130905165318.2fa40bab@anarchist> <20130905210924.GA30649@iskra.aviel.ru> <20130905172907.67a08d29@anarchist> <20130905215622.GA31833@iskra.aviel.ru> Message-ID: <52293563.5070206@jcea.es> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 05/09/13 23:56, Oleg Broytman wrote: > Well, I can only use services that are available, not those that > are promised. If python.org grows support for Persona -- who will > be my provider and for what price? I am not going to install and > manage additional software on my servers -- I don't want to be my > own provider, I have enough job already. If your email provider is not supporting Persona, the automatic fallback is Mozilla, a non-profit, free web, I care about your privacy, organization. - -- Jes?s Cea Avi?n _/_/ _/_/_/ _/_/_/ jcea at jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ Twitter: @jcea _/_/ _/_/ _/_/_/_/_/ jabber / xmpp:jcea at jabber.org _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQCVAwUBUik1Y5lgi5GaxT1NAQLKjwP9EXXmMTVORDPOCK5ByBfrwveL71nG/1IJ oTc0ZUjZPMp/JD2UBnibEUIhxJHtmAkTsuMTmNsSbzqx6F74n9zYokfMK4Y0iscC 2EpqDe7lcAzEjJXpIa93A/K8/fh3gaufWHAXND3Ynr7gMdlLkn9jRiXoVMHHX3Sm lEIX/47kzlI= =4+OP -----END PGP SIGNATURE----- From jcea at jcea.es Fri Sep 6 03:54:01 2013 From: jcea at jcea.es (Jesus Cea) Date: Fri, 06 Sep 2013 03:54:01 +0200 Subject: [Python-Dev] Offtopic: OpenID Providers In-Reply-To: <20130905202921.GA30331@iskra.aviel.ru> References: <5228C00F.7000209@jcea.es> <20130905181231.GA25306@iskra.aviel.ru> <8593FDCE-E3E0-4AD4-897E-2FB1D751086C@stufft.io> <20130905182522.GA26090@iskra.aviel.ru> <2C429087-4E06-4F0F-B524-25D0F6FB4FFF@stufft.io> <20130905184337.GA26926@iskra.aviel.ru> <667454FE-47D1-46BB-99B6-4C0B10986269@stufft.io> <20130905202921.GA30331@iskra.aviel.ru> Message-ID: <522935B9.9030208@jcea.es> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 05/09/13 22:29, Oleg Broytman wrote: > I have seen exactly 0 (zero) sites that support Persona. Can you > point me? "Python Espa?a" (Python Spain) association is going to provide Persona Only login. Deployment in four weeks. - -- Jes?s Cea Avi?n _/_/ _/_/_/ _/_/_/ jcea at jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ Twitter: @jcea _/_/ _/_/ _/_/_/_/_/ jabber / xmpp:jcea at jabber.org _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQCVAwUBUik1uZlgi5GaxT1NAQIA3wP/eH0HA+gxa/9c1S679lmC7GVoBqahWyWS mmOnIg170MaozIxyjgLruReTKdnfhcFa/QxxWLpS7brGenOSA1UCXdOhRHRTpf5y JMtQ/f1pmS3sdjEIal2F6Fm1Dt1i6YFZr813j9hdzVHgcx96PuxTx5UVO3LpAAkf +edD8PBn3eM= =y1Ri -----END PGP SIGNATURE----- From jcea at jcea.es Fri Sep 6 03:56:55 2013 From: jcea at jcea.es (Jesus Cea) Date: Fri, 06 Sep 2013 03:56:55 +0200 Subject: [Python-Dev] Offtopic: OpenID Providers In-Reply-To: <CAF-Rda_aajDr71ZU=xJ8m1diBLuG_jr__TO6tR8kAMS625q7hg@mail.gmail.com> References: <5228C00F.7000209@jcea.es> <CAF-Rda_aajDr71ZU=xJ8m1diBLuG_jr__TO6tR8kAMS625q7hg@mail.gmail.com> Message-ID: <52293667.8030609@jcea.es> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 05/09/13 20:45, Eli Bendersky wrote: > PS: I use "http://www.jcea.es/" as my OpenID identity, and I > delegate the actual service to "myOpenID". I can switch delegation > trivially. > > http://bugs.python.org/?@action=openid_login&provider=Google Sorry, Google, Facebook, Twitter, etc., are not acceptable OpenID providers for me. I should have made that point in my original email. My excuses. Any other suggestion? - -- Jes?s Cea Avi?n _/_/ _/_/_/ _/_/_/ jcea at jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ Twitter: @jcea _/_/ _/_/ _/_/_/_/_/ jabber / xmpp:jcea at jabber.org _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQCVAwUBUik2Z5lgi5GaxT1NAQJEtQP+NdOaMbAB4LzgsUfoL1yelPcYA8wg2rnd WjM7IEsfcs/3NFrmpg9XxjJqSwoxgqW8NqpcQLapGSUdqUxMrhYz3xUnXzSUmM75 IwxUWlTuVVQkscRWwrmIjqMWD20EI3KtmGBtCH5J1Tnmb1EeTH4+wrtpBcl3hBUi Ph8+Afy+E6w= =VkCq -----END PGP SIGNATURE----- From ericsnowcurrently at gmail.com Fri Sep 6 07:26:31 2013 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Thu, 5 Sep 2013 23:26:31 -0600 Subject: [Python-Dev] Pre-PEP: Redesigning extension modules In-Reply-To: <kvctoj$qsv$1@ger.gmane.org> References: <kv77o2$f96$1@ger.gmane.org> <CALeMXf5oxi0oX13zSfHwgVi1eXF_z_coY23MfBP-YOK8WVkWsg@mail.gmail.com> <kvctoj$qsv$1@ger.gmane.org> Message-ID: <CALFfu7Bfvrp1yT0=X08P-f+CF8aLGifZzk8-Mv3YyzL69hUHnA@mail.gmail.com> On Sat, Aug 24, 2013 at 7:07 AM, Stefan Behnel <stefan_ml at behnel.de> wrote: > PEP 3121 would no longer be necessary. Extension types can do all we need. > No more special casing of modules, that was the idea. > One nice thing about PEP 3121 is the addition of md_state to module objects to store internal module state. Wouldn't we be better served by improving the related API rather than abandoning it? If md_state were the home for all mutable internal state then load/reload could focus directly on just md_state and md_dict and not worry about other internal state, since all remaining state would be immutable (refcounts notwithstanding). If the API made this easier then we could leverage the strengths of PEP 3121 to make loading safer and more independent. Of course, we could certainly go the other way and actively discourage mutable internal state... This, coupled with the PEP 451-compatible API and with a proxying wrapper, would go a long way to various "reloading" issues that extension modules have. On Sun, Aug 25, 2013 at 5:54 AM, Stefan Behnel <stefan_ml at behnel.de> wrote: > (regarding reloading into the existing module's namespace) > > I'm not sure this can be done in general. What if the module has threads > running that access the global state? In that case, reinitialising the > module object itself would almost certainly lead to a crash. > > And what if you do "from extmodule import some_function" in a Python > module? Then reloading couldn't replace that reference, just as for normal > Python modules. Meaning that you'd still have to keep both modules properly > alive in order to prevent crashes due to lost global state of the imported > function. > > The difference to Python modules here is that in Python code, you'll get > some kind of exception if state is lost during a reload. In C code, you'll > most likely get a crash. > > How would you even make sure global state is properly cleaned up? Would you > call tp_clear() on the module object before re-running the init code? Or > how else would you enable the init code to do the right thing during both > the first run (where global state is uninitialised) and subsequent runs > (where global state may hold valid state and owned Python references)? > > Even tp_clear() may not be enough, because it's only meant to clean up > Python references, not C-level state. Basically, for reloading to be > correct without changing the object reference, it would have to go all the > way through tp_dealloc(), catch the object at the very end, right before it > gets freed, and then re-initialise it. > Right. It would probably require a separate `PyImportInitializeState_<module>(PyObject *mod)` and/or some API that helps make it easier to manage mutable internal module state (on md_state). On Sun, Aug 25, 2013 at 6:36 AM, Stefan Behnel <stefan_ml at behnel.de> wrote: > PJ Eby, 25.08.2013 06:12: > > My "Importing" package offers lazy imports by creating module objects > > in sys.modules that are a subtype of ModuleType, and use a > > __getattribute__ hook so that trying to use them fires off a reload() > > of the module. > > I wonder if this wouldn't be an approach to fix the reloading problem in > general. What if extension module loading, at least with the new scheme, > didn't return the module object itself and put it into sys.modules but > created a wrapper that redirects its __getattr__ and __setattr__ to the > actual module object? That would have a tiny performance impact on > attribute access, but I'd expect that to be negligible given that the usual > reason for the extension module to exist is that it does non-trivial stuff > in whatever its API provides. Reloading could then really create a > completely new module object and replace the reference inside of the > wrapper. > > That way, code that currently uses "from extmodule import xyz" would > continue to see the original version of the module as of the time of its > import, and code that just did "import extmodule" and then used attribute > access at need would always see the current content of the module as it was > last loaded. I think that, together with keeping module global state in the > module object itself, would nicely fix both cases. > At first blush I like this. -eric p.s. Bear with me if I've missed something in the thread. I'm slogging through a backlog of email -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130905/20ddd112/attachment.html> From solipsis at pitrou.net Fri Sep 6 07:43:48 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 6 Sep 2013 07:43:48 +0200 Subject: [Python-Dev] Pre-PEP: Redesigning extension modules References: <kv77o2$f96$1@ger.gmane.org> <CALeMXf5oxi0oX13zSfHwgVi1eXF_z_coY23MfBP-YOK8WVkWsg@mail.gmail.com> <kvctoj$qsv$1@ger.gmane.org> <CALFfu7Bfvrp1yT0=X08P-f+CF8aLGifZzk8-Mv3YyzL69hUHnA@mail.gmail.com> Message-ID: <20130906074348.2779de48@fsol> On Thu, 5 Sep 2013 23:26:31 -0600 Eric Snow <ericsnowcurrently at gmail.com> wrote: > On Sat, Aug 24, 2013 at 7:07 AM, Stefan Behnel <stefan_ml at behnel.de> wrote: > > > PEP 3121 would no longer be necessary. Extension types can do all we need. > > No more special casing of modules, that was the idea. > > > > One nice thing about PEP 3121 is the addition of md_state to module objects > to store internal module state. Wouldn't we be better served by improving > the related API rather than abandoning it? md_state isn't a PyObject and therefore its lifetime management is quirky (as Py_buffer, same bad idea). So I'd be happy for it to disappear from the next API. > This, coupled with the PEP 451-compatible API and with a proxying wrapper, > would go a long way to various "reloading" issues that extension modules > have. Proxying wrapper? We shouldn't need that kind of tricks. Regards Antoine. From ericsnowcurrently at gmail.com Fri Sep 6 07:47:25 2013 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Thu, 5 Sep 2013 23:47:25 -0600 Subject: [Python-Dev] Pre-PEP: Redesigning extension modules In-Reply-To: <CALeMXf5oxi0oX13zSfHwgVi1eXF_z_coY23MfBP-YOK8WVkWsg@mail.gmail.com> References: <kv77o2$f96$1@ger.gmane.org> <CALeMXf5oxi0oX13zSfHwgVi1eXF_z_coY23MfBP-YOK8WVkWsg@mail.gmail.com> Message-ID: <CALFfu7CTDWvTEmDRa5zvZFTOywjONaBjBNgkqCLBv3kr83qBVw@mail.gmail.com> On Sat, Aug 24, 2013 at 10:12 PM, PJ Eby <pje at telecommunity.com> wrote: > I haven't had a chance to address this on the import-sig discussion > yet about ModuleSpec, but I would like to just mention that one > property of the existing module system that I'm not sure either this > proposal or the ModuleSpec proposal preserves is that it's possible to > implement lazy importing of modules using standard reload() semantics. > > My "Importing" package offers lazy imports by creating module objects > in sys.modules that are a subtype of ModuleType, and use a > __getattribute__ hook so that trying to use them fires off a reload() > of the module. Because the dummy module doesn't have __file__ or > anything else initialized, the import system searches for the module > and then loads it, reusing the existing module object, even though > it's actually only executing the module code for the first time. > > That the existing object be reused is important, because once the > dummy is in sys.modules, it can also be imported by other modules, so > references to it can abound everywhere, and we wish only for it to be > loaded lazily, without needing to trace down and replace all instances > of it. This also preserves other invariants of the module system. > > Anyway, the reason I was asking why reloading is being handled as a > special case in the ModuleSpec proposal -- and the reason I'm curious > about certain provisions of this proposal -- is that making the > assumption you can only reload something with the same > spec/location/etc. it was originally loaded with, and/or that if you > are reloading a module then you previously had a chance to do things > to it, doesn't jibe with the way things work currently. > That is to say, in the pure PEP 302 world, there is no special status > for "reload" that is different from "load" -- the *only* thing that's > different is that there is already a module object to use, and there > is *no guarantee that it's a module object that was initialized by the > loader now being invoked*. > In Python 3.3 (#13959) import.reload() was updated to reuse a module's __loader__, which must now be set. If __loader__ is not set, you get an AttributeError. If that's a problem we can create a tracker issue and discuss there. In Python 3.4 imp.reload() is just an alias to importlib.reload(), but it works basically the same. With ModuleSpec things won't work that differently. If you reload such a module as you described, it will look for __spec__ and call it's reload() method. If __spec__ is not set, you get an AttributeError. It wouldn't be that hard to build a spec from the module if need be and then use that. -eric > AFAICT both this proposal and the ModuleSpec one are making an invalid > assumption per PEP 302, and aren't explicitly proposing to change the > status quo: they just assume things that aren't actually assured by > the prior specs or implementations. > > So, for example, this extension module proposal needs to cover what > happens if an extension module is reloaded and the module object is > not of the type or instance it's expecting. Must it do its own > checking? Error handling? Will some other portion of the import > system be expected to handle it? > > For that matter, what happens (in either proposal) if you reload() a > module which only has a __name__, and no other attributes? I haven't > tested with importlib, but with earlier Pythons this results in a > standard module search being done by reload(). But the ModuleSpec > proposal and this one seem to assume that a reload()-ed module must > already be associated with a loader, location, and/or spec. > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/ericsnowcurrently%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130905/3cad8555/attachment.html> From chris at simplistix.co.uk Fri Sep 6 07:50:07 2013 From: chris at simplistix.co.uk (Chris Withers) Date: Fri, 06 Sep 2013 06:50:07 +0100 Subject: [Python-Dev] windows file closing race condition? Message-ID: <52296D0F.2030400@simplistix.co.uk> Hi All, Continuous testing is a wonderful thing when it comes to finding weird edge case problems, like this one: http://jenkins.simplistix.co.uk/job/testfixtures-tox/COMPONENTS=zc,PYTHON=3.3,label=windows/149/testReport/junit/testfixtures.tests.test_tempdirectory/TempDirectoryTests/test_check_all_tuple/ File "C:\Jenkins\workspace\testfixtures-tox\e8666d4e\.tox\3.3-zc\lib\site-packages\testfixtures\tempdirectory.py", line 323, in __exit__ self.cleanup() File "C:\Jenkins\workspace\testfixtures-tox\e8666d4e\.tox\3.3-zc\lib\site-packages\testfixtures\tempdirectory.py", line 78, in cleanup rmtree(self.path) File "C:\Jenkins\workspace\testfixtures-tox\e8666d4e\.tox\3.3-zc\lib\shutil.py", line 460, in rmtree return _rmtree_unsafe(path, onerror) File "C:\Jenkins\workspace\testfixtures-tox\e8666d4e\.tox\3.3-zc\lib\shutil.py", line 362, in _rmtree_unsafe _rmtree_unsafe(fullname, onerror) File "C:\Jenkins\workspace\testfixtures-tox\e8666d4e\.tox\3.3-zc\lib\shutil.py", line 371, in _rmtree_unsafe onerror(os.rmdir, path, sys.exc_info()) File "C:\Jenkins\workspace\testfixtures-tox\e8666d4e\.tox\3.3-zc\lib\shutil.py", line 369, in _rmtree_unsafe os.rmdir(path) OSError: [WinError 145] The directory is not empty: 'c:\\users\\jenkins\\appdata\\local\\temp\\tmpkeg4d7\\a' I'm 99% certain my code is correct here, the only place I open files for writing in that directory is here: https://github.com/Simplistix/testfixtures/blob/master/testfixtures/tempdirectory.py#L275 So, from my perspective, I'm either looking at a bug in shutil.rmtree (which would be trying to delete a directory before deleting its content or failing to delete a file but ignoring an error) or the file object, when being used as a context manager, going through __exit__ without closing the file and releasing the handle. This happens very infrequently, the OS is Windows 7 and the filesystem is NTFS, if that helps... Any ideas? Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From ericsnowcurrently at gmail.com Fri Sep 6 07:52:40 2013 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Thu, 5 Sep 2013 23:52:40 -0600 Subject: [Python-Dev] Pre-PEP: Redesigning extension modules In-Reply-To: <kvtfek$23p$1@ger.gmane.org> References: <kv77o2$f96$1@ger.gmane.org> <20130823111822.49cba700@pitrou.net> <CADiSq7dhzpTWMw7gcBRrzZot7jaS=Aj=aukq0+jh_cK2+F_kGQ@mail.gmail.com> <CADiSq7dBruZiWhObnjKbcneji9+ziXmyRpg1+72a8YeMWbyO3A@mail.gmail.com> <kvabt8$jfc$1@ger.gmane.org> <CADiSq7c6OjFa8zo7sMB=k2zLSmaf1w-tAwhYKSQ26T3caLyj6A@mail.gmail.com> <kvaka3$sbu$1@ger.gmane.org> <CADiSq7dzQqyH9i6g9QYWp6_ZHHUD4+4npYo6of6mCNahQ+w8kw@mail.gmail.com> <kvcr9f$4gs$1@ger.gmane.org> <CADiSq7cfNH1M29W19BW88-BBQaWEpaMkE=s09wNyT5VGOrPB+g@mail.gmail.com> <kvtfek$23p$1@ger.gmane.org> Message-ID: <CALFfu7CKZU1g8mrhZcpT-0a3coqghT23BpynmEM=+eE4m1ynfQ@mail.gmail.com> On Sat, Aug 31, 2013 at 1:16 PM, Stefan Behnel <stefan_ml at behnel.de> wrote: > Nick Coghlan, 31.08.2013 18:49: > > This is actually my primary motivation for trying to improve the > > "can this be reloaded or not?" aspects of the loader API in PEP 451. > > I assume you mean that the extension module would be able to clearly signal > that it can't be reloaded, right? I agree that that's helpful. If you're > wrapping a C library, then the way that library is implemented might simply > force you to prevent any attempts at reloading the wrapper module. But if > reloading is possible at all, it would be even more helpful if we could > make it really easy to properly support it. > When loader.exec_module() gets called, it should raise ImportError if the module does not support reloading. -eric -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130905/2522477b/attachment.html> From ericsnowcurrently at gmail.com Fri Sep 6 07:57:56 2013 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Thu, 5 Sep 2013 23:57:56 -0600 Subject: [Python-Dev] Pre-PEP: Redesigning extension modules In-Reply-To: <20130902101642.04b69588@pitrou.net> References: <kv77o2$f96$1@ger.gmane.org> <CALeMXf5oxi0oX13zSfHwgVi1eXF_z_coY23MfBP-YOK8WVkWsg@mail.gmail.com> <kvctoj$qsv$1@ger.gmane.org> <kvstdj$jo0$1@ger.gmane.org> <CADiSq7c5xtEcPBXGPWqQosfgGcb2n+2AnHiAWFoATeYfQTUcEQ@mail.gmail.com> <20130902101642.04b69588@pitrou.net> Message-ID: <CALFfu7BZzRFWb0=_xfZHM6O=VnV5MhiNuej6f_sKNebBZ7KzHA@mail.gmail.com> On Mon, Sep 2, 2013 at 2:16 AM, Antoine Pitrou <solipsis at pitrou.net> wrote: > I think the biggest challenge here is to propose an API that's simple > and easy to use (i.e. that doesn't make extension module writing more > complicated than it currently is). > +1 > > The basic concept of putting custom module objects in sys.modules is > sound, IMHO. > > As for "extension module as a wrapper", though, it shounds like the > kind of complication I would personally prefer to stay away from. Also, > it would make extension modules less like Python modules, rather than > more. > It all depends on how useful it would be to be able to safely reload extension modules from their files. -eric -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130905/49715ee2/attachment.html> From solipsis at pitrou.net Fri Sep 6 08:10:20 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 6 Sep 2013 08:10:20 +0200 Subject: [Python-Dev] windows file closing race condition? References: <52296D0F.2030400@simplistix.co.uk> Message-ID: <20130906081020.76e37fba@fsol> On Fri, 06 Sep 2013 06:50:07 +0100 Chris Withers <chris at simplistix.co.uk> wrote: > > So, from my perspective, I'm either looking at a bug in shutil.rmtree > (which would be trying to delete a directory before deleting its content > or failing to delete a file but ignoring an error) or the file object, > when being used as a context manager, going through __exit__ without > closing the file and releasing the handle. It seems someone forgot to remove the following snippet from FileIO.__exit__: if random.random() > 0.0001: # Sometimes we leave the fd open, to annoy Chris os.close(self.fd) > This happens very infrequently, the OS is Windows 7 and the filesystem > is NTFS, if that helps... It should help indeed: http://blogs.msdn.com/b/oldnewthing/archive/2012/09/07/10347136.aspx :-) Regards Antoine. From stefan_ml at behnel.de Fri Sep 6 08:11:06 2013 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 06 Sep 2013 08:11:06 +0200 Subject: [Python-Dev] Pre-PEP: Redesigning extension modules In-Reply-To: <20130906074348.2779de48@fsol> References: <kv77o2$f96$1@ger.gmane.org> <CALeMXf5oxi0oX13zSfHwgVi1eXF_z_coY23MfBP-YOK8WVkWsg@mail.gmail.com> <kvctoj$qsv$1@ger.gmane.org> <CALFfu7Bfvrp1yT0=X08P-f+CF8aLGifZzk8-Mv3YyzL69hUHnA@mail.gmail.com> <20130906074348.2779de48@fsol> Message-ID: <l0brli$73j$1@ger.gmane.org> Antoine Pitrou, 06.09.2013 07:43: > Proxying wrapper? We shouldn't need that kind of tricks. The advantage is that it's controlled by the loader, and transparent to the module itself. Stefan From ncoghlan at gmail.com Fri Sep 6 09:14:18 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 6 Sep 2013 17:14:18 +1000 Subject: [Python-Dev] windows file closing race condition? In-Reply-To: <52296D0F.2030400@simplistix.co.uk> References: <52296D0F.2030400@simplistix.co.uk> Message-ID: <CADiSq7dYFHRSGgLMROXc+XNfTfbsstrkbwdfvzJGOXqdR7MwTA@mail.gmail.com> On 6 September 2013 15:50, Chris Withers <chris at simplistix.co.uk> wrote: > Hi All, > > Continuous testing is a wonderful thing when it comes to finding weird edge > case problems, like this one: > > http://jenkins.simplistix.co.uk/job/testfixtures-tox/COMPONENTS=zc,PYTHON=3.3,label=windows/149/testReport/junit/testfixtures.tests.test_tempdirectory/TempDirectoryTests/test_check_all_tuple/ > > File > "C:\Jenkins\workspace\testfixtures-tox\e8666d4e\.tox\3.3-zc\lib\site-packages\testfixtures\tempdirectory.py", > line 323, in __exit__ > self.cleanup() > File > "C:\Jenkins\workspace\testfixtures-tox\e8666d4e\.tox\3.3-zc\lib\site-packages\testfixtures\tempdirectory.py", > line 78, in cleanup > rmtree(self.path) > File > "C:\Jenkins\workspace\testfixtures-tox\e8666d4e\.tox\3.3-zc\lib\shutil.py", > line 460, in rmtree > return _rmtree_unsafe(path, onerror) > File > "C:\Jenkins\workspace\testfixtures-tox\e8666d4e\.tox\3.3-zc\lib\shutil.py", > line 362, in _rmtree_unsafe > _rmtree_unsafe(fullname, onerror) > File > "C:\Jenkins\workspace\testfixtures-tox\e8666d4e\.tox\3.3-zc\lib\shutil.py", > line 371, in _rmtree_unsafe > onerror(os.rmdir, path, sys.exc_info()) > File > "C:\Jenkins\workspace\testfixtures-tox\e8666d4e\.tox\3.3-zc\lib\shutil.py", > line 369, in _rmtree_unsafe > os.rmdir(path) > OSError: [WinError 145] The directory is not empty: > 'c:\\users\\jenkins\\appdata\\local\\temp\\tmpkeg4d7\\a' > > I'm 99% certain my code is correct here, the only place I open files for > writing in that directory is here: > > https://github.com/Simplistix/testfixtures/blob/master/testfixtures/tempdirectory.py#L275 > > So, from my perspective, I'm either looking at a bug in shutil.rmtree (which > would be trying to delete a directory before deleting its content or failing > to delete a file but ignoring an error) or the file object, when being used > as a context manager, going through __exit__ without closing the file and > releasing the handle. > > This happens very infrequently, the OS is Windows 7 and the filesystem is > NTFS, if that helps... > > Any ideas? This feels a lot like an issue we were seeing on the Windows buildbots, which we ended up working around in the test support library: http://bugs.python.org/issue15496 That would be some awfully ugly code to upgrade from "hack in the test support library" to "this is just how Python unlinks files on Windows", though :P Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From chris at simplistix.co.uk Fri Sep 6 09:48:50 2013 From: chris at simplistix.co.uk (Chris Withers) Date: Fri, 06 Sep 2013 08:48:50 +0100 Subject: [Python-Dev] windows file closing race condition? In-Reply-To: <20130906081020.76e37fba@fsol> References: <52296D0F.2030400@simplistix.co.uk> <20130906081020.76e37fba@fsol> Message-ID: <522988E2.20005@simplistix.co.uk> On 06/09/2013 07:10, Antoine Pitrou wrote: >> This happens very infrequently, the OS is Windows 7 and the filesystem >> is NTFS, if that helps... > > It should help indeed: > http://blogs.msdn.com/b/oldnewthing/archive/2012/09/07/10347136.aspx The box in questions runs no AV software or indexing services... Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From stephen at xemacs.org Fri Sep 6 09:28:21 2013 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Fri, 06 Sep 2013 16:28:21 +0900 Subject: [Python-Dev] windows file closing race condition? In-Reply-To: <20130906081020.76e37fba@fsol> References: <52296D0F.2030400@simplistix.co.uk> <20130906081020.76e37fba@fsol> Message-ID: <87a9jq9zoq.fsf@uwakimon.sk.tsukuba.ac.jp> Antoine Pitrou writes: > http://blogs.msdn.com/b/oldnewthing/archive/2012/09/07/10347136.aspx That was worth it just for the comment from Australia! From mail at timgolden.me.uk Fri Sep 6 09:58:06 2013 From: mail at timgolden.me.uk (Tim Golden) Date: Fri, 06 Sep 2013 08:58:06 +0100 Subject: [Python-Dev] windows file closing race condition? In-Reply-To: <CADiSq7dYFHRSGgLMROXc+XNfTfbsstrkbwdfvzJGOXqdR7MwTA@mail.gmail.com> References: <52296D0F.2030400@simplistix.co.uk> <CADiSq7dYFHRSGgLMROXc+XNfTfbsstrkbwdfvzJGOXqdR7MwTA@mail.gmail.com> Message-ID: <52298B0E.7090508@timgolden.me.uk> On 06/09/2013 08:14, Nick Coghlan wrote: > On 6 September 2013 15:50, Chris Withers <chris at simplistix.co.uk> wrote: >> Continuous testing is a wonderful thing when it comes to finding weird edge >> case problems, like this one: [... snip ...] >> os.rmdir(path) >> OSError: [WinError 145] The directory is not empty: >> 'c:\\users\\jenkins\\appdata\\local\\temp\\tmpkeg4d7\\a' > This feels a lot like an issue we were seeing on the Windows > buildbots, which we ended up working around in the test support > library: http://bugs.python.org/issue15496 > > That would be some awfully ugly code to upgrade from "hack in the test > support library" to "this is just how Python unlinks files on > Windows", though :P I think that any kind of delay-retry loop falls squarely within the remit of the calling application, not core Python. This isn't a problem of Python's making: IIUC, you would see the same effect if you used any other language or simply deleted the folder from within Explorer. (I don't know whether Explorer itself does anything canny under the covers to retry). Obviously our test suite *is* a calling application, and so it makes perfect sense to put some workaround in place. The trouble with this class of problem, where a share-delete handle allows operations to succeed and to fail later which would normally fail early, is that the bad effect is at one remove from its cause. Here, by the time the rmtree has reached the point of removing a parent directory, it's long-since left behind the file which has a still-open handle: the DeleteFile succeeded and the code moved on. You can't even tell which file it was. A related problem arises where the DeleteFile succeeds and an error occurs when a subsequent CreateFile fails for the same filepath, again because a share-delete handle is still open for a file at that path. This is another one which hits our test suite because of an overuse of one temp filename. What should Python do? With some effort it could look for open file handles against the file it's trying to delete, but what then? Wait until they're all closed? That could leave it hanging. And even with a timeout it would introduce a delay which might be unnecessary. A lot of the time, no harm will come of the file existing a few seconds after the DeleteFile has succeeded. In short, I don't believe there's any mileage in introducing extra code into Python's os or io modules. It falls on the shoulders of the calling code to implement retry loops or whatever logic as needed. TJG From chris at simplistix.co.uk Fri Sep 6 09:59:19 2013 From: chris at simplistix.co.uk (Chris Withers) Date: Fri, 06 Sep 2013 08:59:19 +0100 Subject: [Python-Dev] windows file closing race condition? In-Reply-To: <CADiSq7dYFHRSGgLMROXc+XNfTfbsstrkbwdfvzJGOXqdR7MwTA@mail.gmail.com> References: <52296D0F.2030400@simplistix.co.uk> <CADiSq7dYFHRSGgLMROXc+XNfTfbsstrkbwdfvzJGOXqdR7MwTA@mail.gmail.com> Message-ID: <52298B57.2020108@simplistix.co.uk> On 06/09/2013 08:14, Nick Coghlan wrote: > This feels a lot like an issue we were seeing on the Windows > buildbots, which we ended up working around in the test support > library: http://bugs.python.org/issue15496 Wow :'( > That would be some awfully ugly code to upgrade from "hack in the test > support library" to "this is just how Python unlinks files on > Windows", though :P Indeed... Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From solipsis at pitrou.net Fri Sep 6 12:14:36 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 6 Sep 2013 12:14:36 +0200 Subject: [Python-Dev] windows file closing race condition? References: <52296D0F.2030400@simplistix.co.uk> <CADiSq7dYFHRSGgLMROXc+XNfTfbsstrkbwdfvzJGOXqdR7MwTA@mail.gmail.com> <52298B0E.7090508@timgolden.me.uk> Message-ID: <20130906121436.7db391f2@pitrou.net> Le Fri, 06 Sep 2013 08:58:06 +0100, Tim Golden <mail at timgolden.me.uk> a ?crit : > > What should Python do? Maybe using FILE_SHARE_DELETE could help? http://bugs.python.org/issue15244 Regards Antoine. From mail at timgolden.me.uk Fri Sep 6 12:23:39 2013 From: mail at timgolden.me.uk (Tim Golden) Date: Fri, 06 Sep 2013 11:23:39 +0100 Subject: [Python-Dev] windows file closing race condition? In-Reply-To: <20130906121436.7db391f2@pitrou.net> References: <52296D0F.2030400@simplistix.co.uk> <CADiSq7dYFHRSGgLMROXc+XNfTfbsstrkbwdfvzJGOXqdR7MwTA@mail.gmail.com> <52298B0E.7090508@timgolden.me.uk> <20130906121436.7db391f2@pitrou.net> Message-ID: <5229AD2B.5030206@timgolden.me.uk> On 06/09/2013 11:14, Antoine Pitrou wrote: > Le Fri, 06 Sep 2013 08:58:06 +0100, > Tim Golden <mail at timgolden.me.uk> a ?crit : >> >> What should Python do? > > Maybe using FILE_SHARE_DELETE could help? > http://bugs.python.org/issue15244 I don't think so. It's the use of FILE_SHARE_DELETE (by other programs, eg Virus Checkers) that typically causes the problem. IOW, the sequence is: * [Some Other Prog] takes FILE_SHARE_DELETE handle, allowing other programs to delete the file even while this handle is still open * Python calls DeleteFile -- succeeds if the only open handles are FILE_SHARE_DELETE; carries on * File has apparently disappeared but still has open handles * Any attempt to create a file of the same name or to delete a containing directory fail because the file is still open, even though it's successfully been deleted. * (Some time later) [Some Other Prog] closes its handle and the file is now completely gone Unless I'm missing something, there's nothing Python can do to help here apart from the sort of delay-retry dance which test.support uses and which I'm advocating against as a core feature. TJG From shibturn at gmail.com Fri Sep 6 13:50:34 2013 From: shibturn at gmail.com (Richard Oudkerk) Date: Fri, 06 Sep 2013 12:50:34 +0100 Subject: [Python-Dev] windows file closing race condition? In-Reply-To: <5229AD2B.5030206@timgolden.me.uk> References: <52296D0F.2030400@simplistix.co.uk> <CADiSq7dYFHRSGgLMROXc+XNfTfbsstrkbwdfvzJGOXqdR7MwTA@mail.gmail.com> <52298B0E.7090508@timgolden.me.uk> <20130906121436.7db391f2@pitrou.net> <5229AD2B.5030206@timgolden.me.uk> Message-ID: <l0cfia$dl$1@ger.gmane.org> On 06/09/2013 11:23am, Tim Golden wrote: > On 06/09/2013 11:14, Antoine Pitrou wrote: >> Le Fri, 06 Sep 2013 08:58:06 +0100, >> Tim Golden <mail at timgolden.me.uk> a ?crit : >>> >>> What should Python do? >> >> Maybe using FILE_SHARE_DELETE could help? >> http://bugs.python.org/issue15244 > > I don't think so. It's the use of FILE_SHARE_DELETE (by other programs, > eg Virus Checkers) that typically causes the problem. IOW, the sequence is: > > * [Some Other Prog] takes FILE_SHARE_DELETE handle, allowing other > programs to delete the file even while this handle is still open > > * Python calls DeleteFile -- succeeds if the only open handles are > FILE_SHARE_DELETE; carries on > > * File has apparently disappeared but still has open handles > > * Any attempt to create a file of the same name or to delete a > containing directory fail because the file is still open, even though > it's successfully been deleted. > > * (Some time later) [Some Other Prog] closes its handle and the file is > now completely gone > > > Unless I'm missing something, there's nothing Python can do to help here > apart from the sort of delay-retry dance which test.support uses and > which I'm advocating against as a core feature. Instead of deleting, the file could be moved to a temporary name in the root directory (or some other permanent directory on the same drive) and then deleted. That would allow the directory to be closed even if a FILE_SHARE_DELETE handle is still open for the file. -- Richard From mail at timgolden.me.uk Fri Sep 6 14:21:35 2013 From: mail at timgolden.me.uk (Tim Golden) Date: Fri, 06 Sep 2013 13:21:35 +0100 Subject: [Python-Dev] windows file closing race condition? In-Reply-To: <l0cfia$dl$1@ger.gmane.org> References: <52296D0F.2030400@simplistix.co.uk> <CADiSq7dYFHRSGgLMROXc+XNfTfbsstrkbwdfvzJGOXqdR7MwTA@mail.gmail.com> <52298B0E.7090508@timgolden.me.uk> <20130906121436.7db391f2@pitrou.net> <5229AD2B.5030206@timgolden.me.uk> <l0cfia$dl$1@ger.gmane.org> Message-ID: <5229C8CF.2050308@timgolden.me.uk> On 06/09/2013 12:50, Richard Oudkerk wrote: > On 06/09/2013 11:23am, Tim Golden wrote: >> On 06/09/2013 11:14, Antoine Pitrou wrote: >>> Le Fri, 06 Sep 2013 08:58:06 +0100, >>> Tim Golden <mail at timgolden.me.uk> a ?crit : >>>> >>>> What should Python do? >>> >>> Maybe using FILE_SHARE_DELETE could help? >>> http://bugs.python.org/issue15244 >> >> I don't think so. It's the use of FILE_SHARE_DELETE (by other programs, >> eg Virus Checkers) that typically causes the problem. IOW, the >> sequence is: >> >> * [Some Other Prog] takes FILE_SHARE_DELETE handle, allowing other >> programs to delete the file even while this handle is still open >> >> * Python calls DeleteFile -- succeeds if the only open handles are >> FILE_SHARE_DELETE; carries on >> >> * File has apparently disappeared but still has open handles >> >> * Any attempt to create a file of the same name or to delete a >> containing directory fail because the file is still open, even though >> it's successfully been deleted. >> >> * (Some time later) [Some Other Prog] closes its handle and the file is >> now completely gone >> >> >> Unless I'm missing something, there's nothing Python can do to help here >> apart from the sort of delay-retry dance which test.support uses and >> which I'm advocating against as a core feature. > > Instead of deleting, the file could be moved to a temporary name in the > root directory (or some other permanent directory on the same drive) and > then deleted. That would allow the directory to be closed even if a > FILE_SHARE_DELETE handle is still open for the file. > True, but then you're into determining a temporary name somewhere on the same volume if possible and avoiding collisions etc. Again, I don't think this is something we need to be doing by default in core Python. TJG From p.f.moore at gmail.com Fri Sep 6 14:55:17 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 6 Sep 2013 13:55:17 +0100 Subject: [Python-Dev] windows file closing race condition? In-Reply-To: <5229C8CF.2050308@timgolden.me.uk> References: <52296D0F.2030400@simplistix.co.uk> <CADiSq7dYFHRSGgLMROXc+XNfTfbsstrkbwdfvzJGOXqdR7MwTA@mail.gmail.com> <52298B0E.7090508@timgolden.me.uk> <20130906121436.7db391f2@pitrou.net> <5229AD2B.5030206@timgolden.me.uk> <l0cfia$dl$1@ger.gmane.org> <5229C8CF.2050308@timgolden.me.uk> Message-ID: <CACac1F8QJeLf=KXo-Eq+LrKBJo_sNWKdpZpjEgsPtdxKU+buig@mail.gmail.com> On 6 September 2013 13:21, Tim Golden <mail at timgolden.me.uk> wrote: > True, but then you're into determining a temporary name somewhere on the > same volume if possible and avoiding collisions etc. Again, I don't > think this is something we need to be doing by default in core Python. Agreed. There simply is no solution that works in all cases. If you rename the file to anywhere but the same directory, you potentially have permission issues. And if you rename to the same directory, you still can't delete the directory. It's a shame, because a helper to do this would be useful for Unix users wanting to ensure that their code works properly on Windows, but there really isn't an answer other than knowing how things work (and crafting an appropriate solution for your application), here... Paul From shibturn at gmail.com Fri Sep 6 15:16:25 2013 From: shibturn at gmail.com (Richard Oudkerk) Date: Fri, 06 Sep 2013 14:16:25 +0100 Subject: [Python-Dev] windows file closing race condition? In-Reply-To: <CACac1F8QJeLf=KXo-Eq+LrKBJo_sNWKdpZpjEgsPtdxKU+buig@mail.gmail.com> References: <52296D0F.2030400@simplistix.co.uk> <CADiSq7dYFHRSGgLMROXc+XNfTfbsstrkbwdfvzJGOXqdR7MwTA@mail.gmail.com> <52298B0E.7090508@timgolden.me.uk> <20130906121436.7db391f2@pitrou.net> <5229AD2B.5030206@timgolden.me.uk> <l0cfia$dl$1@ger.gmane.org> <5229C8CF.2050308@timgolden.me.uk> <CACac1F8QJeLf=KXo-Eq+LrKBJo_sNWKdpZpjEgsPtdxKU+buig@mail.gmail.com> Message-ID: <l0ckja$qv2$1@ger.gmane.org> On 06/09/2013 1:55pm, Paul Moore wrote: > ... If you rename the > file to anywhere but the same directory, you potentially have > permission issues. Using the root directory avoids permission issues -- users always have write access there. -- Richard From rdmurray at bitdance.com Fri Sep 6 15:26:52 2013 From: rdmurray at bitdance.com (R. David Murray) Date: Fri, 06 Sep 2013 09:26:52 -0400 Subject: [Python-Dev] windows file closing race condition? In-Reply-To: <5229C8CF.2050308@timgolden.me.uk> References: <52296D0F.2030400@simplistix.co.uk> <CADiSq7dYFHRSGgLMROXc+XNfTfbsstrkbwdfvzJGOXqdR7MwTA@mail.gmail.com> <52298B0E.7090508@timgolden.me.uk> <20130906121436.7db391f2@pitrou.net> <5229AD2B.5030206@timgolden.me.uk> <l0cfia$dl$1@ger.gmane.org> <5229C8CF.2050308@timgolden.me.uk> Message-ID: <20130906132652.D566A2507F5@webabinitio.net> On Fri, 06 Sep 2013 13:21:35 +0100, Tim Golden <mail at timgolden.me.uk> wrote: > On 06/09/2013 12:50, Richard Oudkerk wrote: > > On 06/09/2013 11:23am, Tim Golden wrote: > >> On 06/09/2013 11:14, Antoine Pitrou wrote: > >>> Le Fri, 06 Sep 2013 08:58:06 +0100, > >>> Tim Golden <mail at timgolden.me.uk> a ??crit : > >>>> > >>>> What should Python do? > >>> > >>> Maybe using FILE_SHARE_DELETE could help? > >>> http://bugs.python.org/issue15244 > >> > >> I don't think so. It's the use of FILE_SHARE_DELETE (by other programs, > >> eg Virus Checkers) that typically causes the problem. IOW, the > >> sequence is: > >> > >> * [Some Other Prog] takes FILE_SHARE_DELETE handle, allowing other > >> programs to delete the file even while this handle is still open > >> > >> * Python calls DeleteFile -- succeeds if the only open handles are > >> FILE_SHARE_DELETE; carries on > >> > >> * File has apparently disappeared but still has open handles > >> > >> * Any attempt to create a file of the same name or to delete a > >> containing directory fail because the file is still open, even though > >> it's successfully been deleted. > >> > >> * (Some time later) [Some Other Prog] closes its handle and the file is > >> now completely gone > >> > >> > >> Unless I'm missing something, there's nothing Python can do to help here > >> apart from the sort of delay-retry dance which test.support uses and > >> which I'm advocating against as a core feature. > > > > Instead of deleting, the file could be moved to a temporary name in the > > root directory (or some other permanent directory on the same drive) and > > then deleted. That would allow the directory to be closed even if a > > FILE_SHARE_DELETE handle is still open for the file. > > > > True, but then you're into determining a temporary name somewhere on the > same volume if possible and avoiding collisions etc. Again, I don't > think this is something we need to be doing by default in core Python. What about moving the test.support delete helper to somewhere in unittest, since this will come up in pretty much every test suite that runs on Windows? --David From ethan at stoneleaf.us Fri Sep 6 15:32:46 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 06 Sep 2013 06:32:46 -0700 Subject: [Python-Dev] inspect and metaclasses Message-ID: <5229D97E.9080103@stoneleaf.us> Part of the fix for issue #18693 is to fix inspect to look in the metaclass for class attributes (http://bugs.python.org/issue18929). In inspect.py in function get_mro() we can either add the metaclass unconditionally, or only if it is not 'type'. If we add unconditionally, then help() adds the following: class A(builtins.object) | Hello and goodbye + | + | Method resolution order: + | A + | builtins.object + | builtins.type | | Methods defined here: Do we want that, or should we just add the metaclass if it is not 'type'? -- ~Ethan~ From p.f.moore at gmail.com Fri Sep 6 16:01:44 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 6 Sep 2013 15:01:44 +0100 Subject: [Python-Dev] windows file closing race condition? In-Reply-To: <l0ckja$qv2$1@ger.gmane.org> References: <52296D0F.2030400@simplistix.co.uk> <CADiSq7dYFHRSGgLMROXc+XNfTfbsstrkbwdfvzJGOXqdR7MwTA@mail.gmail.com> <52298B0E.7090508@timgolden.me.uk> <20130906121436.7db391f2@pitrou.net> <5229AD2B.5030206@timgolden.me.uk> <l0cfia$dl$1@ger.gmane.org> <5229C8CF.2050308@timgolden.me.uk> <CACac1F8QJeLf=KXo-Eq+LrKBJo_sNWKdpZpjEgsPtdxKU+buig@mail.gmail.com> <l0ckja$qv2$1@ger.gmane.org> Message-ID: <CACac1F_VtkEYoqWy=vp2vMwfSoyQ3W5KsLcnwYbnx82jzwW7tw@mail.gmail.com> On 6 September 2013 14:16, Richard Oudkerk <shibturn at gmail.com> wrote: > On 06/09/2013 1:55pm, Paul Moore wrote: >> >> ... If you rename the >> >> file to anywhere but the same directory, you potentially have >> permission issues. > > > Using the root directory avoids permission issues -- users always have write > access there. I don't believe that's true. IIRC, some Windows versions protect the root of the system drive from non-admin users. And I *know* I have had problems with NTFS-formatted removable drives plugged into systems they were not originally formatted on, where the owner of the drive no longer exists. They are rare corner cases, for sure, but they do happen and frankly are probably harder to explain when they come up than saying "your virus checker is interfering". Everyone expects their virus checker to screw up their system (cure worse than disease syndrome :-)) Paul From eliben at gmail.com Fri Sep 6 16:14:11 2013 From: eliben at gmail.com (Eli Bendersky) Date: Fri, 6 Sep 2013 07:14:11 -0700 Subject: [Python-Dev] SEEK_* constants in io and os In-Reply-To: <CAF-Rda_msMu--d-8Q8kPy8ReaJdNavKQ8keLrVQPOjMxpA41Yg@mail.gmail.com> References: <CAF-Rda_2j=HTDJ5nEdKWxX1mCmbHLgUNe_zyDYXzGHE6-LUBmg@mail.gmail.com> <20130902102437.197c1708@pitrou.net> <CAF-Rda-_Z7QTQNf+jonLC1FqWj4pdLVk2h5r5ORTPaiUCwGauA@mail.gmail.com> <20130902154522.495b1697@pitrou.net> <20130902155109.7da869bd@pitrou.net> <CAF-Rda86kyZg+ZuJcw4jff8w6LgCYo__N=VDyR_GQqdEcQ6ePg@mail.gmail.com> <CAMpsgwYoR9p+F5Q9rXw-UbLKV2fFyEsZARR=VKXvjSLwGv8i6A@mail.gmail.com> <CAF-Rda_msMu--d-8Q8kPy8ReaJdNavKQ8keLrVQPOjMxpA41Yg@mail.gmail.com> Message-ID: <CAF-Rda_NWd3FxvZRYnQJ7uoF1N3WK8G-_ZfKuQKDg+X=z=+veg@mail.gmail.com> > On Mon, Sep 2, 2013 at 8:48 AM, Victor Stinner <victor.stinner at gmail.com>wrote: > >> 2013/9/2 Eli Bendersky <eliben at gmail.com>: >> > Yes, now I see a 500 usec difference timed within the Python script. >> When >> > timing the whole execution of Python: (...) >> >> Can you please provide the list of imported modules by: >> python -c 'import sys; print(sys.modules)' >> >> For python with default options and for python with -S (no site >> module) options? And also with your patch? >> >> Python should be fast to write "hello world" (or "python -c pass), >> it's a dummy but common benchmark (to compare Python to other VM / >> other programming languages). >> > > The sorted list for both default and -S (they're identical) is > http://pastebin.com/4vzSMCu7 - there are 55 entries there, including > things like itertools, heapq, functools and collections. With my patch, > enum is also added (which makes sense since os is in the list). So the > 0.5ms increase for the 34ms runtime kind-of makes sense. > This question is still kind-of open. I haven't received additional feedback - only Antoine's and Victor's concerns wrt. runtime cost, which I believe I addressed. Note that the runtime cost is globally one-time in the sense that if additional modules (whether used at start-up or not) use enum, this is a price only paid once. So, is it worth it the extra 0.5ms of start-up time, or not? Eli -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130906/692d8d35/attachment.html> From ethan at stoneleaf.us Fri Sep 6 16:23:46 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 06 Sep 2013 07:23:46 -0700 Subject: [Python-Dev] SEEK_* constants in io and os In-Reply-To: <CAF-Rda_2j=HTDJ5nEdKWxX1mCmbHLgUNe_zyDYXzGHE6-LUBmg@mail.gmail.com> References: <CAF-Rda_2j=HTDJ5nEdKWxX1mCmbHLgUNe_zyDYXzGHE6-LUBmg@mail.gmail.com> Message-ID: <5229E572.5090509@stoneleaf.us> On 09/01/2013 06:02 PM, Eli Bendersky wrote: > > os seems to import io in some functions; can this be done always? If yes, we can just define the constants once and > os.SEEK_* will alias io.SEEK_*? The other way (io taking from os) is also a possibility (maybe the preferred one because > io already refers to os.SEEK_HOLE/DATA, at least in the documentation). > > Any ideas and suggestions are welcome, Ideally we should only define them once. If these are values that could change per O/S then they should be in the os module. Since they /can/ change per O/S (even if they don't currently), we should put them in os. I'd say it's worth the extra 0.5ms startup time. -- ~Ethan~ From arigo at tunes.org Fri Sep 6 16:47:15 2013 From: arigo at tunes.org (Armin Rigo) Date: Fri, 6 Sep 2013 16:47:15 +0200 Subject: [Python-Dev] inspect and metaclasses In-Reply-To: <5229D97E.9080103@stoneleaf.us> References: <5229D97E.9080103@stoneleaf.us> Message-ID: <CAMSv6X1k30-PAWct_9tDTmwir4mDO0jAs-mbrvvBrJc5XE6VTQ@mail.gmail.com> Hi Ethan, Are you suggesting that inspect.get_mro(A) would return (A, object, type)? That seems very wrong to me. If the goal is to fix `inspect.classify_class_attrs()`, then this function only needs a specific fix, along the lines of looking in `get_mro(A) + get_mro(type(A))`. (A more minor issue is that the bug report suggests `... + (type(A),)` only, but that's wrong: Python will also look in all the base classes of type(A).) "Fixing" inspect.get_mro() as suggested would break a lot of other usages of it. A bient?t, Armin. From jcea at jcea.es Fri Sep 6 17:08:23 2013 From: jcea at jcea.es (Jesus Cea) Date: Fri, 06 Sep 2013 17:08:23 +0200 Subject: [Python-Dev] DTRACE support Message-ID: <5229EFE7.8040702@jcea.es> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 As far as I know, Erlang, Ruby, PHP, Perl, etc., support Dtrace. Python is embarrasingly missing from this list. Some examples: <http://crypt.codemancers.com/posts/2013-04-16-profile-ruby-apps-dtrace-part1/> <http://www.phpdeveloper.org/news/18859> <http://www.erlang.org/doc/apps/runtime_tools/DTRACE.html> I have spend a very long time on a patch for Dtrace support in most platforms with dtrace available. Currently working under Solaris and derivatives, and MacOS X. Last time I checked, it would crash FreeBSD because bugs in the dtrace port, but that was a long time ago. I would like to push this to Python 3.4, and the window is going to be closed soon, so I think this is the time to ask for opinions and support here. Does Python-Dev have any opinion or interest in this project?. Should I push for it? Some (not current) details: http://bugs.python.org/issue13405 DTrace is amazing: http://dtrace.org/blogs/ PS: I can't release this code as a PyPI project, because it mess with the inner core of Python. - -- Jes?s Cea Avi?n _/_/ _/_/_/ _/_/_/ jcea at jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ Twitter: @jcea _/_/ _/_/ _/_/_/_/_/ jabber / xmpp:jcea at jabber.org _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQCVAwUBUinv55lgi5GaxT1NAQIK7gP8DeaWxNVwNFa5PnfZe00Aay5l1rWzgj8d CY0D3W+PAdPkBci9SYPmfv7ajXrQWo/ccANYIRaUdI/U9Zjq/od7eNemOFqyL7U6 BrQpAUMySI6tMlL+gYEfQ8O47SManvTqoyNvOFAz9mVJute8IxKsbCIK/jiRHDXz vWyG7YrYN1A= =4E7+ -----END PGP SIGNATURE----- From jcea at jcea.es Fri Sep 6 17:08:28 2013 From: jcea at jcea.es (Jesus Cea) Date: Fri, 06 Sep 2013 17:08:28 +0200 Subject: [Python-Dev] Details of "import lock" in 3.3 Message-ID: <5229EFEC.1010503@jcea.es> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 With importlib and other recent changes, what are the current details of "import lock"?. That is, the lock/locks held when Python code does "import", specially in the case of multithreading. Is that documented anywhere? Thanks! - -- Jes?s Cea Avi?n _/_/ _/_/_/ _/_/_/ jcea at jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ Twitter: @jcea _/_/ _/_/ _/_/_/_/_/ jabber / xmpp:jcea at jabber.org _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQCVAwUBUinv7Jlgi5GaxT1NAQLX7QP/SBNuy869xSKRVNlWmAiDLpZOR9IHm77K fy+AG/aJDpk4BMcm+KNoHZyUtJJs2kndfz+OVTeU/mmlr+aXu/wA1yNB1HAjAwWO 4ZWyaN9WLrhv8xXomUlT8iqikc/9pEspgQbJAljUvMLVouixIdxga3IqtaT3FQZh wkV9v04G4OA= =P3PI -----END PGP SIGNATURE----- From ethan at stoneleaf.us Fri Sep 6 16:51:06 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 06 Sep 2013 07:51:06 -0700 Subject: [Python-Dev] when to fix cross-version bugs? Message-ID: <5229EBDA.1030902@stoneleaf.us> I recently committed a fix for unicodeobject.c so that the %d, %i, and %u format specifiers always output values (otherwise, in subclasses, the str() was used instead). Should this be fixed in 3.3 as well? What guidelines determine when a bug is fixed in previous versions? -- ~Ethan~ From guido at python.org Fri Sep 6 17:14:04 2013 From: guido at python.org (Guido van Rossum) Date: Fri, 6 Sep 2013 08:14:04 -0700 Subject: [Python-Dev] DTRACE support In-Reply-To: <5229EFE7.8040702@jcea.es> References: <5229EFE7.8040702@jcea.es> Message-ID: <CAP7+vJJsPVn_EHjfYeYZhK=nd3VfBZ3h_b6tTsShTt6gUSVNaw@mail.gmail.com> I've heard good things about DTRACE but never used it myself. Do I understand correctly that you have to build a separate Python executable with it turned on? I noticed one odd thing in the patch: apparently the dtrace module only exists to have a flag that tells whether it is enabled. Can't that flag be added to the sys module? On Fri, Sep 6, 2013 at 8:08 AM, Jesus Cea <jcea at jcea.es> wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > As far as I know, Erlang, Ruby, PHP, Perl, etc., support Dtrace. > Python is embarrasingly missing from this list. > > Some examples: > > <http://crypt.codemancers.com/posts/2013-04-16-profile-ruby-apps-dtrace-part1/> > <http://www.phpdeveloper.org/news/18859> > <http://www.erlang.org/doc/apps/runtime_tools/DTRACE.html> > > I have spend a very long time on a patch for Dtrace support in most > platforms with dtrace available. Currently working under Solaris and > derivatives, and MacOS X. Last time I checked, it would crash FreeBSD > because bugs in the dtrace port, but that was a long time ago. > > I would like to push this to Python 3.4, and the window is going to be > closed soon, so I think this is the time to ask for opinions and > support here. > > Does Python-Dev have any opinion or interest in this project?. Should > I push for it? > > Some (not current) details: http://bugs.python.org/issue13405 > > DTrace is amazing: http://dtrace.org/blogs/ > > PS: I can't release this code as a PyPI project, because it mess with > the inner core of Python. > > - -- > Jes?s Cea Avi?n _/_/ _/_/_/ _/_/_/ > jcea at jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ > Twitter: @jcea _/_/ _/_/ _/_/_/_/_/ > jabber / xmpp:jcea at jabber.org _/_/ _/_/ _/_/ _/_/ _/_/ > "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ > "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ > "El amor es poner tu felicidad en la felicidad de otro" - Leibniz > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.10 (GNU/Linux) > Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ > > iQCVAwUBUinv55lgi5GaxT1NAQIK7gP8DeaWxNVwNFa5PnfZe00Aay5l1rWzgj8d > CY0D3W+PAdPkBci9SYPmfv7ajXrQWo/ccANYIRaUdI/U9Zjq/od7eNemOFqyL7U6 > BrQpAUMySI6tMlL+gYEfQ8O47SManvTqoyNvOFAz9mVJute8IxKsbCIK/jiRHDXz > vWyG7YrYN1A= > =4E7+ > -----END PGP SIGNATURE----- > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%40python.org -- --Guido van Rossum (python.org/~guido) From A.J.Miller at bcs.org.uk Fri Sep 6 11:54:45 2013 From: A.J.Miller at bcs.org.uk (Andrew Miller) Date: Fri, 6 Sep 2013 10:54:45 +0100 Subject: [Python-Dev] unicodedata module is out of date Message-ID: <CAPMPPoRq1Uuag3BP0OnfsJPhKVsZapN3JS03i+dN2enAj=saYg@mail.gmail.com> The unicodedata module only contains data up to Unicode 5.2 (October 2009), so attempting to reference any character from a later version e.g: unicodedata.lookup("TURKISH LIRA SIGN") results in a KeyError. Also, it seems to be limited to properties in the UnicodeData.txt file and does not contain any data from the other files from the Unicode Character Database (the perl library Unicode::UCD is far more complete). Are there any plans to update this module to the latest Unicode version (6.2, with 6.3 being released shortly), or is there another module that provides more up to date information? Thanks, Andrew -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130906/48935204/attachment.html> From skip at pobox.com Fri Sep 6 17:18:49 2013 From: skip at pobox.com (Skip Montanaro) Date: Fri, 6 Sep 2013 10:18:49 -0500 Subject: [Python-Dev] DTRACE support In-Reply-To: <5229EFE7.8040702@jcea.es> References: <5229EFE7.8040702@jcea.es> Message-ID: <CANc-5UzS_jUBOmixhRjKz+ooEMFYjOEPvNkCTjzWfROCKH=nbw@mail.gmail.com> > I have spend a very long time on a patch for Dtrace support in most > platforms with dtrace available. Currently working under Solaris and > derivatives, and MacOS X. Last time I checked, it would crash FreeBSD > because bugs in the dtrace port, but that was a long time ago. I looked at this several years ago. As I recall, the problem at the time was that the Apple and Sun DTrace implementations were incompatible, or that the probes they had inserted into their own /usr/bin/python instances were incompatible. (Don't remember which off the top of my head.) Of course, the DTrace folks at Apple and Sun weren't really interested in holding hands... Skip From jcea at jcea.es Fri Sep 6 17:21:06 2013 From: jcea at jcea.es (Jesus Cea) Date: Fri, 06 Sep 2013 17:21:06 +0200 Subject: [Python-Dev] DTRACE support In-Reply-To: <CAP7+vJJsPVn_EHjfYeYZhK=nd3VfBZ3h_b6tTsShTt6gUSVNaw@mail.gmail.com> References: <5229EFE7.8040702@jcea.es> <CAP7+vJJsPVn_EHjfYeYZhK=nd3VfBZ3h_b6tTsShTt6gUSVNaw@mail.gmail.com> Message-ID: <5229F2E2.6090301@jcea.es> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 06/09/13 17:14, Guido van Rossum wrote: > I've heard good things about DTRACE but never used it myself. > > Do I understand correctly that you have to build a separate Python > executable with it turned on? It is a patch you apply on stock Python and you get a new configure option: "--with-dtrace". If you compile Python with that flag, you get a python interpreter with dtrace probes on it. > I noticed one odd thing in the patch: apparently the dtrace module > only exists to have a flag that tells whether it is enabled. Can't > that flag be added to the sys module? My plan is to (in the future) use that module to create new probes on the fly, like <http://chrisa.github.io/blog/2011/12/04/libusdt-runtime-dtrace-providers/>, and to export functions to communicate data to dtrace scripts, if running. That is, this module is currently a shell I plan to fill. - -- Jes?s Cea Avi?n _/_/ _/_/_/ _/_/_/ jcea at jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ Twitter: @jcea _/_/ _/_/ _/_/_/_/_/ jabber / xmpp:jcea at jabber.org _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQCVAwUBUiny4plgi5GaxT1NAQLngQP/eFoQ3rdPy12nmw8pPPO/+/RYjlhIi67+ T0n1xBJbxIvpFpIRufB+tkCfLWdb7YZKcf3Nb4V6pVsOPdbd9s6RcVWNA9Ds5mNx ISrO644CAIqRF5XgeeS5uhsODL1DrZTEfX0QBhGsU6lcqyOzSfRX2t9hRBMh3f5y 2dPMHNznJhs= =dV5g -----END PGP SIGNATURE----- From jcea at jcea.es Fri Sep 6 17:24:08 2013 From: jcea at jcea.es (Jesus Cea) Date: Fri, 06 Sep 2013 17:24:08 +0200 Subject: [Python-Dev] DTRACE support In-Reply-To: <CANc-5UzS_jUBOmixhRjKz+ooEMFYjOEPvNkCTjzWfROCKH=nbw@mail.gmail.com> References: <5229EFE7.8040702@jcea.es> <CANc-5UzS_jUBOmixhRjKz+ooEMFYjOEPvNkCTjzWfROCKH=nbw@mail.gmail.com> Message-ID: <5229F398.3040802@jcea.es> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 06/09/13 17:18, Skip Montanaro wrote: > I looked at this several years ago. As I recall, the problem at the > time was that the Apple and Sun DTrace implementations were > incompatible, or that the probes they had inserted into their own > /usr/bin/python instances were incompatible. (Don't remember which > off the top of my head.) Of course, the DTrace folks at Apple and > Sun weren't really interested in holding hands... Right. They use two different set of probes because Python doesn't provide "official" probes :-). If I recall correctly, they were basically identical, with an extra parameter somewhere. I think my patch provides a superset. Anyway, I don't think neither of them support Python 3. Time to step in and liderate. - -- Jes?s Cea Avi?n _/_/ _/_/_/ _/_/_/ jcea at jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ Twitter: @jcea _/_/ _/_/ _/_/_/_/_/ jabber / xmpp:jcea at jabber.org _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQCVAwUBUinzmJlgi5GaxT1NAQLxwQP/cTq90CCfZhiUVLsutP0dQk4Br62kvCyn mmMV9E/ZPJEO3LYYQT+CSiIRnG5panqdZulMDAGBBvgIbJP/E8spuyoFZkspWtPM fk5n2etHq8JMZs36mbxnhw++W3/jlZ4osdVWZZKoKdwVBRFz2YvpLa24q+Y06gU/ 5YEXHuNDP/4= =YFm2 -----END PGP SIGNATURE----- From jcea at jcea.es Fri Sep 6 17:28:56 2013 From: jcea at jcea.es (Jesus Cea) Date: Fri, 06 Sep 2013 17:28:56 +0200 Subject: [Python-Dev] DTRACE support In-Reply-To: <5229EFE7.8040702@jcea.es> References: <5229EFE7.8040702@jcea.es> Message-ID: <5229F4B8.7070200@jcea.es> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 06/09/13 17:08, Jesus Cea wrote: > Does Python-Dev have any opinion or interest in this project?. > Should I push for it? I have using this code for ages on my Solaris machines: Python 2.7.5 (dtrace-issue13405_2.7:f96ea83cd766, Aug 19 2013, 02:55:15) Python 3.3.2 (dtrace-issue13405_3.3:23dafaf73d29, Aug 19 2013, 05:52:34) Python 3.4.0a1+ (dtrace-issue13405:d7be45bb06a3, Aug 19 2013, 07:03:24) - -- Jes?s Cea Avi?n _/_/ _/_/_/ _/_/_/ jcea at jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ Twitter: @jcea _/_/ _/_/ _/_/_/_/_/ jabber / xmpp:jcea at jabber.org _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQCVAwUBUin0uJlgi5GaxT1NAQJP/wQAmkxZ5UJk9L+wTZb94icd3vqUGdo+mgR6 CNs0/oyemd7fL2DSaHGycqDwVPqyqNVFE+aD/ziEmWOwmiFQebudX0Z7md4ZSF89 LU4ddBPMj1bI5E/EYtlqgZg4TqwbUP3XPy9IrCEeiiSKcIk2TJxYiZ12IlcFyXU+ ff3E+DyDkBY= =svJD -----END PGP SIGNATURE----- From brian at python.org Fri Sep 6 17:29:00 2013 From: brian at python.org (Brian Curtin) Date: Fri, 6 Sep 2013 10:29:00 -0500 Subject: [Python-Dev] when to fix cross-version bugs? In-Reply-To: <5229EBDA.1030902@stoneleaf.us> References: <5229EBDA.1030902@stoneleaf.us> Message-ID: <CAD+XWwpHdGLGFen+thhpmkaV9ArWB-V8pggBjsfjzFXmFGtPoA@mail.gmail.com> On Fri, Sep 6, 2013 at 9:51 AM, Ethan Furman <ethan at stoneleaf.us> wrote: > I recently committed a fix for unicodeobject.c so that the %d, %i, and %u > format specifiers always output values (otherwise, in subclasses, the str() > was used instead). > > Should this be fixed in 3.3 as well? > > What guidelines determine when a bug is fixed in previous versions? If it's a bug in that version and the version is accepting bug fixes, i.e., not in security mode, go for it. This includes crossing the 2/3 boundary if applicable. From cf.natali at gmail.com Fri Sep 6 17:29:12 2013 From: cf.natali at gmail.com (=?ISO-8859-1?Q?Charles=2DFran=E7ois_Natali?=) Date: Fri, 6 Sep 2013 17:29:12 +0200 Subject: [Python-Dev] DTRACE support In-Reply-To: <5229EFE7.8040702@jcea.es> References: <5229EFE7.8040702@jcea.es> Message-ID: <CAH_1eM2gG2fVsXU88j1T3y8TeGJ+fnDu4BBR2aMVg2hw=37_pg@mail.gmail.com> > As far as I know, Erlang, Ruby, PHP, Perl, etc., support Dtrace. > Python is embarrasingly missing from this list. > > Some examples: > > <http://crypt.codemancers.com/posts/2013-04-16-profile-ruby-apps-dtrace-part1/> > <http://www.phpdeveloper.org/news/18859> > <http://www.erlang.org/doc/apps/runtime_tools/DTRACE.html> > > I have spend a very long time on a patch for Dtrace support in most > platforms with dtrace available. Currently working under Solaris and > derivatives, and MacOS X. Last time I checked, it would crash FreeBSD > because bugs in the dtrace port, but that was a long time ago. > > I would like to push this to Python 3.4, and the window is going to be > closed soon, so I think this is the time to ask for opinions and > support here. > > Does Python-Dev have any opinion or interest in this project?. Should > I push for it? IMO, that's a large, intrusive patch, which distracts the reader from the main code and logic. Here's an extract from Modules/gcmodule.c: static void dtrace_gc_done(Py_ssize_t value) { PYTHON_GC_DONE((long) value); /* * Currently a USDT tail-call will not receive the correct arguments. * Disable the tail call here. */ #if defined(__sparc) asm("nop"); #endif } Also have a look at cevalc.c: http://bugs.python.org/review/13405/diff/6152/Python/ceval.c IMO it's not worth it (personally strace/gdb/valgrind are more than enough for me, and we''re about to gain memory tracing with Victor's tracemalloc). cf From rdmurray at bitdance.com Fri Sep 6 17:31:53 2013 From: rdmurray at bitdance.com (R. David Murray) Date: Fri, 06 Sep 2013 11:31:53 -0400 Subject: [Python-Dev] when to fix cross-version bugs? In-Reply-To: <5229EBDA.1030902@stoneleaf.us> References: <5229EBDA.1030902@stoneleaf.us> Message-ID: <20130906153153.D5DDF250845@webabinitio.net> On Fri, 06 Sep 2013 07:51:06 -0700, Ethan Furman <ethan at stoneleaf.us> wrote: > I recently committed a fix for unicodeobject.c so that the %d, %i, and %u format specifiers always output values > (otherwise, in subclasses, the str() was used instead). > > Should this be fixed in 3.3 as well? > > What guidelines determine when a bug is fixed in previous versions? The basic guideline is: we try very hard not to break currently working code in a maintenance release. Making that decision very much depends on the details of each individual case. I'd say this one is borderline...it would probably be OK to backport it, since programs depending on the str of number subclasses (that is what we are talking about here, right? Even though you say the fix is in unicodeobject.c...) are likely to be rare, or already have a workaround that won't get broken by the change, but by the same token it probably doesn't have much positive impact if it does get backported, so is it worth the (small) chance of breaking someone's code? --David From stefan_ml at behnel.de Fri Sep 6 17:33:47 2013 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 06 Sep 2013 17:33:47 +0200 Subject: [Python-Dev] unicodedata module is out of date In-Reply-To: <CAPMPPoRq1Uuag3BP0OnfsJPhKVsZapN3JS03i+dN2enAj=saYg@mail.gmail.com> References: <CAPMPPoRq1Uuag3BP0OnfsJPhKVsZapN3JS03i+dN2enAj=saYg@mail.gmail.com> Message-ID: <l0cskj$tc1$1@ger.gmane.org> Andrew Miller, 06.09.2013 11:54: > The unicodedata module only contains data up to Unicode 5.2 (October 2009), > so attempting to reference any character from a later version e.g: > > unicodedata.lookup("TURKISH LIRA SIGN") > > results in a KeyError. > > Also, it seems to be limited to properties in the UnicodeData.txt file and > does not contain any data from the other files from the Unicode Character > Database (the perl library Unicode::UCD is far more complete). > > Are there any plans to update this module to the latest Unicode version > (6.2, with 6.3 being released shortly) It's been updated to 6.2 almost a year ago, so Python 3.3 should have that. I don't think 6.3 support will be added before Python 3.4, assuming it's final by then. You should open a ticket so that it won't be forgotten before the release. http://bugs.python.org/ That being said, the module is (mostly) generated, so you might be able to fix it up yourself if you need it sooner in a local installation. Stefan From rdmurray at bitdance.com Fri Sep 6 17:35:06 2013 From: rdmurray at bitdance.com (R. David Murray) Date: Fri, 06 Sep 2013 11:35:06 -0400 Subject: [Python-Dev] unicodedata module is out of date In-Reply-To: <CAPMPPoRq1Uuag3BP0OnfsJPhKVsZapN3JS03i+dN2enAj=saYg@mail.gmail.com> References: <CAPMPPoRq1Uuag3BP0OnfsJPhKVsZapN3JS03i+dN2enAj=saYg@mail.gmail.com> Message-ID: <20130906153510.6CCB0250833@webabinitio.net> On Fri, 06 Sep 2013 10:54:45 +0100, Andrew Miller <A.J.Miller at bcs.org.uk> wrote: > Are there any plans to update this module to the latest Unicode version > (6.2, with 6.3 being released shortly), or is there another module that > provides more up to date information? Python 3.4 currently has 6.2. If 6.3 gets released before the first RC, I'm guessing we will probably upgrade to it. --David From python at mrabarnett.plus.com Fri Sep 6 17:38:03 2013 From: python at mrabarnett.plus.com (MRAB) Date: Fri, 06 Sep 2013 16:38:03 +0100 Subject: [Python-Dev] unicodedata module is out of date In-Reply-To: <CAPMPPoRq1Uuag3BP0OnfsJPhKVsZapN3JS03i+dN2enAj=saYg@mail.gmail.com> References: <CAPMPPoRq1Uuag3BP0OnfsJPhKVsZapN3JS03i+dN2enAj=saYg@mail.gmail.com> Message-ID: <5229F6DB.40000@mrabarnett.plus.com> On 06/09/2013 10:54, Andrew Miller wrote: > The unicodedata module only contains data up to Unicode 5.2 (October > 2009), so attempting to reference any character from a later version e.g: > > unicodedata.lookup("TURKISH LIRA SIGN") > > results in a KeyError. > > Also, it seems to be limited to properties in the UnicodeData.txt file > and does not contain any data from the other files from the Unicode > Character Database (the perl library Unicode::UCD is far more complete). > > Are there any plans to update this module to the latest Unicode version > (6.2, with 6.3 being released shortly), or is there another module that > provides more up to date information? > Which version of Python are you talking about? Python 3.3 uses Unicode version 6.1. From ethan at stoneleaf.us Fri Sep 6 17:14:09 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 06 Sep 2013 08:14:09 -0700 Subject: [Python-Dev] inspect and metaclasses In-Reply-To: <CAMSv6X1k30-PAWct_9tDTmwir4mDO0jAs-mbrvvBrJc5XE6VTQ@mail.gmail.com> References: <5229D97E.9080103@stoneleaf.us> <CAMSv6X1k30-PAWct_9tDTmwir4mDO0jAs-mbrvvBrJc5XE6VTQ@mail.gmail.com> Message-ID: <5229F141.2050007@stoneleaf.us> On 09/06/2013 07:47 AM, Armin Rigo wrote: > > Are you suggesting that inspect.getmro(A) would return (A, object, > type)? That seems very wrong to me. Currently, `inspect.getmro(A)` returns `(A, object)`. Considering that Python actually will look in A's metaclass to find a class attribute, I think returning `(A, object, type(A)` is appropriate: ================================================================================= Python 3.4.0a1+ (default:61ca4732399b+, Sep 4 2013, 22:28:04) [GCC 4.7.3] on linux Type "help", "copyright", "credits" or "license" for more information. --> class Meta(type): ... meta_attr = 42 ... --> class Class(metaclass=Meta): ... cls_attr = 'Vitamin-soaked towel' ... def __init__(self): ... self.inst_attr = 'dolphins' ... --> test = Class() --> test.inst_attr 'dolphins' --> test.cls_attr 'Vitamin-soaked towel' --> test.meta_attr Traceback (most recent call last): File "<stdin>", line 1, in <module> AttributeError: 'Class' object has no attribute 'meta_attr' --> Class.cls_attr 'Vitamin-soaked towel' --> Class.meta_attr 42 --> import inspect --> inspect.getmro(Class) # with patch in place (<class '__main__.Class'>, <class 'object'>, <class '__main__.Meta'>) ================================================================================= > If the goal is to fix `inspect.classify_class_attrs()`, then this > function only needs a specific fix, along the lines of looking in > `getmro(A) + getmro(type(A))`. (A more minor issue is that the bug > report suggests `... + (type(A),)` only, but that's wrong: Python will > also look in all the base classes of type(A).) Good point. Will incorporate that into the final fix, whichever way it ends up. > "Fixing" inspect.getmro() as suggested would break a lot of other usages of it. Any examples? -- ~Ethan~ From rdmurray at bitdance.com Fri Sep 6 17:39:01 2013 From: rdmurray at bitdance.com (R. David Murray) Date: Fri, 06 Sep 2013 11:39:01 -0400 Subject: [Python-Dev] unicodedata module is out of date In-Reply-To: <l0cskj$tc1$1@ger.gmane.org> References: <CAPMPPoRq1Uuag3BP0OnfsJPhKVsZapN3JS03i+dN2enAj=saYg@mail.gmail.com> <l0cskj$tc1$1@ger.gmane.org> Message-ID: <20130906153901.9ED57250845@webabinitio.net> On Fri, 06 Sep 2013 17:33:47 +0200, Stefan Behnel <stefan_ml at behnel.de> wrote: > Andrew Miller, 06.09.2013 11:54: > > The unicodedata module only contains data up to Unicode 5.2 (October 2009), > > so attempting to reference any character from a later version e.g: > > > > unicodedata.lookup("TURKISH LIRA SIGN") > > > > results in a KeyError. > > > > Also, it seems to be limited to properties in the UnicodeData.txt file and > > does not contain any data from the other files from the Unicode Character > > Database (the perl library Unicode::UCD is far more complete). > > > > Are there any plans to update this module to the latest Unicode version > > (6.2, with 6.3 being released shortly) > > It's been updated to 6.2 almost a year ago, so Python 3.3 should have that. 3.3 shipped with 6.1. --David From solipsis at pitrou.net Fri Sep 6 17:41:11 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 6 Sep 2013 17:41:11 +0200 Subject: [Python-Dev] DTRACE support References: <5229EFE7.8040702@jcea.es> Message-ID: <20130906174111.446ea2c4@pitrou.net> Le Fri, 06 Sep 2013 17:08:23 +0200, Jesus Cea <jcea at jcea.es> a ?crit : > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > As far as I know, Erlang, Ruby, PHP, Perl, etc., support Dtrace. > Python is embarrasingly missing from this list. > > Some examples: > > <http://crypt.codemancers.com/posts/2013-04-16-profile-ruby-apps-dtrace-part1/> > <http://www.phpdeveloper.org/news/18859> > <http://www.erlang.org/doc/apps/runtime_tools/DTRACE.html> > > I have spend a very long time on a patch for Dtrace support in most > platforms with dtrace available. Currently working under Solaris and > derivatives, and MacOS X. Last time I checked, it would crash FreeBSD > because bugs in the dtrace port, but that was a long time ago. > > I would like to push this to Python 3.4, and the window is going to be > closed soon, so I think this is the time to ask for opinions and > support here. You should start by addressing review comments: http://bugs.python.org/issue13405#msg151751 Right now, I agree with Charles-Fran?ois: your patch is too intrusive. Regards Antoine. From rdmurray at bitdance.com Fri Sep 6 17:44:08 2013 From: rdmurray at bitdance.com (R. David Murray) Date: Fri, 06 Sep 2013 11:44:08 -0400 Subject: [Python-Dev] inspect and metaclasses In-Reply-To: <5229F141.2050007@stoneleaf.us> References: <5229D97E.9080103@stoneleaf.us> <CAMSv6X1k30-PAWct_9tDTmwir4mDO0jAs-mbrvvBrJc5XE6VTQ@mail.gmail.com> <5229F141.2050007@stoneleaf.us> Message-ID: <20130906154408.BFAAD250845@webabinitio.net> On Fri, 06 Sep 2013 08:14:09 -0700, Ethan Furman <ethan at stoneleaf.us> wrote: > On 09/06/2013 07:47 AM, Armin Rigo wrote: > > > > Are you suggesting that inspect.getmro(A) would return (A, object, > > type)? That seems very wrong to me. > > Currently, `inspect.getmro(A)` returns `(A, object)`. Which matches A.__mro__. EOD, I think. --David From solipsis at pitrou.net Fri Sep 6 17:43:13 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 6 Sep 2013 17:43:13 +0200 Subject: [Python-Dev] Details of "import lock" in 3.3 References: <5229EFEC.1010503@jcea.es> Message-ID: <20130906174313.63e37ac3@pitrou.net> Le Fri, 06 Sep 2013 17:08:28 +0200, Jesus Cea <jcea at jcea.es> a ?crit : > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > With importlib and other recent changes, what are the current details > of "import lock"?. That is, the lock/locks held when Python code does > "import", specially in the case of multithreading. Is that documented > anywhere? Quick summary here: http://docs.python.org/3/whatsnew/3.3.html#a-finer-grained-import-lock Otherwise, I'm afraid the source code has the most information, e.g.: http://hg.python.org/cpython/file/1d88d04aade2/Lib/importlib/_bootstrap.py#l124 Regards Antoine. From benjamin at python.org Fri Sep 6 17:45:14 2013 From: benjamin at python.org (Benjamin Peterson) Date: Fri, 6 Sep 2013 11:45:14 -0400 Subject: [Python-Dev] unicodedata module is out of date In-Reply-To: <CAPMPPoRq1Uuag3BP0OnfsJPhKVsZapN3JS03i+dN2enAj=saYg@mail.gmail.com> References: <CAPMPPoRq1Uuag3BP0OnfsJPhKVsZapN3JS03i+dN2enAj=saYg@mail.gmail.com> Message-ID: <CAPZV6o98M=fb1Re=R3b3RSBgqZdAF_Lo=NaxDTT1aq3nC5Bjhw@mail.gmail.com> 2013/9/6 Andrew Miller <A.J.Miller at bcs.org.uk>: > The unicodedata module only contains data up to Unicode 5.2 (October 2009), > so attempting to reference any character from a later version e.g: > > unicodedata.lookup("TURKISH LIRA SIGN") > > results in a KeyError. > > Also, it seems to be limited to properties in the UnicodeData.txt file and > does not contain any data from the other files from the Unicode Character > Database (the perl library Unicode::UCD is far more complete). > > Are there any plans to update this module to the latest Unicode version > (6.2, with 6.3 being released shortly), or is there another module that > provides more up to date information? I usually keep the latest Python version up to date with the latest Unicode version, so 3.4 will have Unicode 6.2. -- Regards, Benjamin From jcea at jcea.es Fri Sep 6 17:55:52 2013 From: jcea at jcea.es (Jesus Cea) Date: Fri, 06 Sep 2013 17:55:52 +0200 Subject: [Python-Dev] DTRACE support In-Reply-To: <CAH_1eM2gG2fVsXU88j1T3y8TeGJ+fnDu4BBR2aMVg2hw=37_pg@mail.gmail.com> References: <5229EFE7.8040702@jcea.es> <CAH_1eM2gG2fVsXU88j1T3y8TeGJ+fnDu4BBR2aMVg2hw=37_pg@mail.gmail.com> Message-ID: <5229FB08.20603@jcea.es> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 06/09/13 17:29, Charles-Fran?ois Natali wrote: > IMO, that's a large, intrusive patch, which distracts the reader > from the main code and logic. Yes, the patch is intrusive. It must be, to get its goals. Could be improved, nevertheless. Help and suggestions welcome. I want to write a probe for the GIL, but I didn't because I know adding the cost of an extra single machine code branch would be anathema here :-) (lets do baby steps), but I would love to be able to watch with detail all interaction between the GIL, threads, and OS scheduling on my Solaris :). That is very valuable information to have, for instance, to guide future improvements of the GIL. How can you get that kind of information with any other tool? > IMO it's not worth it (personally strace/gdb/valgrind are more > than enough for me, and we''re about to gain memory tracing with > Victor's tracemalloc). The main value of DTrace is systemwide observability. You can see something "strange" at kernel level and trace it to a particular line of code in a random Python script. There is no other tool that can do that. You have complete transversal observability of ALL the code running in your computer, kernel or usermode, clean reports with threads, etc. Valgrind doesn't work on Solaris and *BSD support is unclean. You can run a dtrace script in a long running python process if you need it, after launching it, at runtime, and when the dtrace script is done, the python program keeps running without any penalty. Just poking around, for instance. I do it constantly for, for instance, profiling covering both Python as any C code/libraries called from it. Maybe the biggest objection would be that most python-devs are running Linux, and you don't have dtrace support on linux unless you are running Oracle distribution. But world is larger than linux, and there are some efforts to port DTrace to Linux itself. DTrace is available on Solaris and derivatives, MacOS X and FreeBSD. - -- Jes?s Cea Avi?n _/_/ _/_/_/ _/_/_/ jcea at jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ Twitter: @jcea _/_/ _/_/ _/_/_/_/_/ jabber / xmpp:jcea at jabber.org _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQCVAwUBUin7CJlgi5GaxT1NAQKbfwP+PjOqf3pjlHHq78ggA8qyNkjPFXEoRVj9 9PKslZ7FiU+JWxzXY52k1GNspHhIox+PAcvdBL/gWU3rOiqjEm5fU8FX61Lh6FMj bB+QRyZLpLfmANrLX43sBvwDbG9gTuq8FVUvqmtSme615vX2ygITwNZysQ7xPoD/ jXbeOAF0LZU= =asEC -----END PGP SIGNATURE----- From A.J.Miller at bcs.org.uk Fri Sep 6 17:29:32 2013 From: A.J.Miller at bcs.org.uk (Andrew Miller) Date: Fri, 6 Sep 2013 16:29:32 +0100 Subject: [Python-Dev] Fwd: unicodedata module is out of date In-Reply-To: <CAPMPPoRq1Uuag3BP0OnfsJPhKVsZapN3JS03i+dN2enAj=saYg@mail.gmail.com> References: <CAPMPPoRq1Uuag3BP0OnfsJPhKVsZapN3JS03i+dN2enAj=saYg@mail.gmail.com> Message-ID: <CAPMPPoTuTB6T-FBq4__DSNoZUtH5XY8moMJR9CgLS76mZ5xhoA@mail.gmail.com> The unicodedata module only contains data up to Unicode 5.2 (October 2009), so attempting to reference any character from a later version e.g: unicodedata.lookup("TURKISH LIRA SIGN") results in a KeyError. Also, it seems to be limited to properties in the UnicodeData.txt file and does not contain any data from the other files from the Unicode Character Database (the perl library Unicode::UCD is far more complete). Are there any plans to update this module to the latest Unicode version (6.2, with 6.3 being released shortly), or is there another module that provides more up to date information? Thanks, Andrew -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130906/ce3e7ce2/attachment.html> From A.J.Miller at bcs.org.uk Fri Sep 6 17:55:22 2013 From: A.J.Miller at bcs.org.uk (Andrew Miller) Date: Fri, 6 Sep 2013 16:55:22 +0100 Subject: [Python-Dev] unicodedata module is out of date In-Reply-To: <5229F6DB.40000@mrabarnett.plus.com> References: <CAPMPPoRq1Uuag3BP0OnfsJPhKVsZapN3JS03i+dN2enAj=saYg@mail.gmail.com> <5229F6DB.40000@mrabarnett.plus.com> Message-ID: <CAPMPPoRMBsBbAasu8ewEwqJOJARLAsctJricjKdVYVkXdLy8Tg@mail.gmail.com> I've just checked on Python 2.7.5 and Python 3.3.2 (Win32 versions). In Python 3.3.2 unicodedata.unidata_version is set to '6.1.0'. In Python 2.7.5 it is set to '5.2.0' so it looks as though this version is no longer being updated. Since my initial post I've downloaded the Python 2.7.5 source and have found the makeunicodedata.py script which creates this module. Are there plans to add the extra data from the other UCD files to this module? At the moment I am using a module from https://gist.github.com/anonymous/2204527 to obtain the script of a character but it would be nice if this was available from the standard library. On 6 September 2013 16:38, MRAB <python at mrabarnett.plus.com> wrote: > On 06/09/2013 10:54, Andrew Miller wrote: > >> The unicodedata module only contains data up to Unicode 5.2 (October >> 2009), so attempting to reference any character from a later version e.g: >> >> unicodedata.lookup("TURKISH LIRA SIGN") >> >> results in a KeyError. >> >> Also, it seems to be limited to properties in the UnicodeData.txt file >> and does not contain any data from the other files from the Unicode >> Character Database (the perl library Unicode::UCD is far more complete). >> >> Are there any plans to update this module to the latest Unicode version >> (6.2, with 6.3 being released shortly), or is there another module that >> provides more up to date information? >> >> Which version of Python are you talking about? Python 3.3 uses Unicode > version 6.1. > > ______________________________**_________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/**mailman/listinfo/python-dev<https://mail.python.org/mailman/listinfo/python-dev> > Unsubscribe: https://mail.python.org/**mailman/options/python-dev/a.** > j.miller%40bcs.org.uk<https://mail.python.org/mailman/options/python-dev/a.j.miller%40bcs.org.uk> > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130906/f58de89a/attachment.html> From jcea at jcea.es Fri Sep 6 18:02:32 2013 From: jcea at jcea.es (Jesus Cea) Date: Fri, 06 Sep 2013 18:02:32 +0200 Subject: [Python-Dev] Details of "import lock" in 3.3 In-Reply-To: <20130906174313.63e37ac3@pitrou.net> References: <5229EFEC.1010503@jcea.es> <20130906174313.63e37ac3@pitrou.net> Message-ID: <5229FC98.9080106@jcea.es> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 06/09/13 17:43, Antoine Pitrou wrote: > Quick summary here: > http://docs.python.org/3/whatsnew/3.3.html#a-finer-grained-import-lock > > Otherwise, I'm afraid the source code has the most information, > e.g.: > http://hg.python.org/cpython/file/1d88d04aade2/Lib/importlib/_bootstrap.py#l124 Yes, > I was depending of the global import lock in my code. I am evaluating impact under Python 3.3. Under Python 3.3, metafinder/loader MUST BE reentrant?. Can they count on the fact that they could be called concurrently by multiple threads but NOT for importing the SAME module?. - -- Jes?s Cea Avi?n _/_/ _/_/_/ _/_/_/ jcea at jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ Twitter: @jcea _/_/ _/_/ _/_/_/_/_/ jabber / xmpp:jcea at jabber.org _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQCVAwUBUin8mJlgi5GaxT1NAQKQugP+N0VI/BhqKvcaUicu7YMR09VhFweoAE6h c48ltOrhlWhrauYc9XYeoEPZ2ksl9/SDnMt83cE+qLunSBltkSbYv3j/0LXWyHwk ZGjzh5vQGTgCu3/2iCkFWq/tSAsgUfk8YfbwPQK82L1gs7LbC5vt2jWlRwNQs0nl 2Sr1Nlbf7og= =GL0o -----END PGP SIGNATURE----- From status at bugs.python.org Fri Sep 6 18:07:41 2013 From: status at bugs.python.org (Python tracker) Date: Fri, 6 Sep 2013 18:07:41 +0200 (CEST) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20130906160741.06F3F56A17@psf.upfronthosting.co.za> ACTIVITY SUMMARY (2013-08-30 - 2013-09-06) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 4195 (+13) closed 26523 (+47) total 30718 (+60) Open issues with patches: 1915 Issues opened (43) ================== #16826: Don't check for PYTHONCASEOK if interpreter started with -E http://bugs.python.org/issue16826 reopened by meador.inge #18709: SSL module fails to handle NULL bytes inside subjectAltNames g http://bugs.python.org/issue18709 reopened by neologix #18886: BytesGenerator does not handle 'binary' CTE correctly http://bugs.python.org/issue18886 opened by r.david.murray #18887: test_multiprocessing.test_connection failure http://bugs.python.org/issue18887 opened by neologix #18890: Add a raw_data_manager content manager to the email package. http://bugs.python.org/issue18890 opened by r.david.murray #18891: Master patch for content manager addtion to email package. http://bugs.python.org/issue18891 opened by r.david.murray #18893: invalid exception handling in Lib/ctypes/macholib/dyld.py http://bugs.python.org/issue18893 opened by scoder #18894: In unittest.TestResult.failures remove deprecated fail* method http://bugs.python.org/issue18894 opened by py.user #18895: In unittest.TestResult.addError split the sentence http://bugs.python.org/issue18895 opened by py.user #18896: Remove namedtuple 255 arguments restriction http://bugs.python.org/issue18896 opened by valorien #18898: Apply the setobject optimizations to dictionaries http://bugs.python.org/issue18898 opened by rhettinger #18899: make pystone.py Py3 compatible in benchmark suite http://bugs.python.org/issue18899 opened by scoder #18900: Add the random.distrib module http://bugs.python.org/issue18900 opened by serhiy.storchaka #18902: Make ElementTree event handling more modular to allow custom t http://bugs.python.org/issue18902 opened by eli.bendersky #18903: IDLE file-completion is case-sensitive in Windows http://bugs.python.org/issue18903 opened by anikom15 #18904: Unnecessary test in file descriptor inheritance test http://bugs.python.org/issue18904 opened by vajrasky #18905: pydoc -p 0 says the server is available at localhost:0 http://bugs.python.org/issue18905 opened by Wieland.Hoffmann #18906: Create a way to always run tests in subprocesses within regrte http://bugs.python.org/issue18906 opened by eli.bendersky #18907: urllib2.open FTP open times out at 20 secs despite timeout par http://bugs.python.org/issue18907 opened by nagle #18908: Enum docs: sections leak out http://bugs.python.org/issue18908 opened by elazar #18910: IDLE: Unit test for textView.py http://bugs.python.org/issue18910 opened by philwebster #18911: minidom does not encode correctly when calling Document.writex http://bugs.python.org/issue18911 opened by brianvanderburg2 #18913: ssl._ssl._test_decode_cert seems to leak memory with certain c http://bugs.python.org/issue18913 opened by sYnfo #18915: ssl.wrap_socket, pass in certfile and keyfile as PEM strings http://bugs.python.org/issue18915 opened by mpb #18916: Various out-of-date Lock text in 3.2+ http://bugs.python.org/issue18916 opened by tim.peters #18917: python won't display greek characters in apache under windows http://bugs.python.org/issue18917 opened by nickl1 #18918: help('FILES') finds no documentation http://bugs.python.org/issue18918 opened by st.sempert at gmail.com #18919: Unify audio modules tests http://bugs.python.org/issue18919 opened by serhiy.storchaka #18921: In imaplib, cached capabilities may be out of date after login http://bugs.python.org/issue18921 opened by sjmurdoch #18923: Use the new selectors module in the subprocess module http://bugs.python.org/issue18923 opened by haypo #18925: select.poll.modify is not documented http://bugs.python.org/issue18925 opened by giampaolo.rodola #18927: Lock.acquire() docs incorrect about negative timeout http://bugs.python.org/issue18927 opened by tim.peters #18929: inspect.classify_class_attrs ignores metaclass http://bugs.python.org/issue18929 opened by ethan.furman #18930: os.spawnXX functions terminates process if second argument is http://bugs.python.org/issue18930 opened by SSchukat #18931: new selectors module should support devpoll on Solaris http://bugs.python.org/issue18931 opened by giampaolo.rodola #18932: selectors and modify() http://bugs.python.org/issue18932 opened by giampaolo.rodola #18934: multiprocessing: use selectors module http://bugs.python.org/issue18934 opened by neologix #18935: test_regrtest.test_timeout failure http://bugs.python.org/issue18935 opened by neologix #18936: getopt chokes on unicode option names http://bugs.python.org/issue18936 opened by jason.coombs #18937: add unittest assertion for logging http://bugs.python.org/issue18937 opened by pitrou #18943: argparse: default args in mutually exclusive groups http://bugs.python.org/issue18943 opened by arigo #18944: Minor mistake in test_set.py http://bugs.python.org/issue18944 opened by arigo #18945: Name collision handling in tempfile is not covered by tests http://bugs.python.org/issue18945 opened by vlad Most recent 15 issues with no replies (15) ========================================== #18945: Name collision handling in tempfile is not covered by tests http://bugs.python.org/issue18945 #18944: Minor mistake in test_set.py http://bugs.python.org/issue18944 #18943: argparse: default args in mutually exclusive groups http://bugs.python.org/issue18943 #18936: getopt chokes on unicode option names http://bugs.python.org/issue18936 #18935: test_regrtest.test_timeout failure http://bugs.python.org/issue18935 #18932: selectors and modify() http://bugs.python.org/issue18932 #18930: os.spawnXX functions terminates process if second argument is http://bugs.python.org/issue18930 #18927: Lock.acquire() docs incorrect about negative timeout http://bugs.python.org/issue18927 #18915: ssl.wrap_socket, pass in certfile and keyfile as PEM strings http://bugs.python.org/issue18915 #18910: IDLE: Unit test for textView.py http://bugs.python.org/issue18910 #18905: pydoc -p 0 says the server is available at localhost:0 http://bugs.python.org/issue18905 #18904: Unnecessary test in file descriptor inheritance test http://bugs.python.org/issue18904 #18903: IDLE file-completion is case-sensitive in Windows http://bugs.python.org/issue18903 #18895: In unittest.TestResult.addError split the sentence http://bugs.python.org/issue18895 #18894: In unittest.TestResult.failures remove deprecated fail* method http://bugs.python.org/issue18894 Most recent 15 issues waiting for review (15) ============================================= #18944: Minor mistake in test_set.py http://bugs.python.org/issue18944 #18935: test_regrtest.test_timeout failure http://bugs.python.org/issue18935 #18934: multiprocessing: use selectors module http://bugs.python.org/issue18934 #18931: new selectors module should support devpoll on Solaris http://bugs.python.org/issue18931 #18919: Unify audio modules tests http://bugs.python.org/issue18919 #18910: IDLE: Unit test for textView.py http://bugs.python.org/issue18910 #18908: Enum docs: sections leak out http://bugs.python.org/issue18908 #18905: pydoc -p 0 says the server is available at localhost:0 http://bugs.python.org/issue18905 #18904: Unnecessary test in file descriptor inheritance test http://bugs.python.org/issue18904 #18899: make pystone.py Py3 compatible in benchmark suite http://bugs.python.org/issue18899 #18898: Apply the setobject optimizations to dictionaries http://bugs.python.org/issue18898 #18895: In unittest.TestResult.addError split the sentence http://bugs.python.org/issue18895 #18894: In unittest.TestResult.failures remove deprecated fail* method http://bugs.python.org/issue18894 #18893: invalid exception handling in Lib/ctypes/macholib/dyld.py http://bugs.python.org/issue18893 #18891: Master patch for content manager addtion to email package. http://bugs.python.org/issue18891 Top 10 most discussed issues (10) ================================= #18808: Thread.join returns before PyThreadState is destroyed http://bugs.python.org/issue18808 11 msgs #18844: allow weights in random.choice http://bugs.python.org/issue18844 11 msgs #18843: Py_FatalError (msg=0x7f0e3b373232 "bad leading pad byte") at P http://bugs.python.org/issue18843 10 msgs #18693: help() not helpful with enum http://bugs.python.org/issue18693 9 msgs #18726: json functions have too many positional parameters http://bugs.python.org/issue18726 9 msgs #18913: ssl._ssl._test_decode_cert seems to leak memory with certain c http://bugs.python.org/issue18913 8 msgs #18831: importlib.import_module() bypasses builtins.__import__ http://bugs.python.org/issue18831 7 msgs #18709: SSL module fails to handle NULL bytes inside subjectAltNames g http://bugs.python.org/issue18709 6 msgs #18840: Tutorial recommends pickle module without any warning of insec http://bugs.python.org/issue18840 6 msgs #18876: Problems with files opened in append mode with io module http://bugs.python.org/issue18876 6 msgs Issues closed (43) ================== #12037: test_email failures under desktop Windows http://bugs.python.org/issue12037 closed by terry.reedy #15350: {urllib,urllib.parse}.urlencode.__doc__ is unclear http://bugs.python.org/issue15350 closed by orsenthil #16853: add a Selector to the select module http://bugs.python.org/issue16853 closed by pitrou #17224: can not open idle in python 2.7.3 http://bugs.python.org/issue17224 closed by terry.reedy #17930: Search not needed in combinations_with_replacement http://bugs.python.org/issue17930 closed by tim.peters #18105: ElementTree writes invalid files when UTF-16 encoding is speci http://bugs.python.org/issue18105 closed by eli.bendersky #18418: Thread.isAlive() sometimes True after fork http://bugs.python.org/issue18418 closed by neologix #18489: IDLE Unit test for SearchEngine.py http://bugs.python.org/issue18489 closed by terry.reedy #18672: Fix format specifiers for debug output in _sre.c http://bugs.python.org/issue18672 closed by serhiy.storchaka #18720: Switch suitable constants in the socket module to IntEnum http://bugs.python.org/issue18720 closed by python-dev #18738: String formatting (% and str.format) issues with Enum http://bugs.python.org/issue18738 closed by python-dev #18745: Test enum in test_json is ignorant of infinity value http://bugs.python.org/issue18745 closed by python-dev #18750: '' % [1] doesn't fail http://bugs.python.org/issue18750 closed by asvetlov #18756: os.urandom() fails under high load http://bugs.python.org/issue18756 closed by pitrou #18780: SystemError when formatting int subclass http://bugs.python.org/issue18780 closed by python-dev #18826: reversed() requires a sequence - Could work on any iterator? http://bugs.python.org/issue18826 closed by rhettinger #18830: Remove duplicates from a result of getclasstree() http://bugs.python.org/issue18830 closed by serhiy.storchaka #18845: 2.7.5-r2: Fatal Python error: Segmentation fault http://bugs.python.org/issue18845 closed by tim.peters #18849: Failure to try another name for tempfile when directory with c http://bugs.python.org/issue18849 closed by python-dev #18850: xml.etree.ElementTree accepts control chars. http://bugs.python.org/issue18850 closed by eli.bendersky #18851: subprocess's Popen closes stdout/stderr filedescriptors used i http://bugs.python.org/issue18851 closed by pitrou #18870: eval() uses latin-1 to decode str http://bugs.python.org/issue18870 closed by python-dev #18878: Add support of the 'with' statement to sunau.open. http://bugs.python.org/issue18878 closed by serhiy.storchaka #18882: Add threading.main_thread() function http://bugs.python.org/issue18882 closed by asvetlov #18888: Add stdlib support for random sampling with replacement http://bugs.python.org/issue18888 closed by mark.dickinson #18889: test_sax: multiple failures on Windows desktop http://bugs.python.org/issue18889 closed by tim.peters #18892: sqlite3, valued records not persisted, default ones are http://bugs.python.org/issue18892 closed by debewerker #18897: Illegal instruction at Python-2.7.5/Modules/_sre.c:1173 http://bugs.python.org/issue18897 closed by pitrou #18901: sunau.getparams should return a namedtuple http://bugs.python.org/issue18901 closed by serhiy.storchaka #18909: Segfaults on win-amd64 due to corrupt pointer to Tkapp_Interp http://bugs.python.org/issue18909 closed by haypo #18912: Intendation issue in example code in itertools.count documenta http://bugs.python.org/issue18912 closed by python-dev #18914: Confusing documentation in the urllib2 HOWTO http://bugs.python.org/issue18914 closed by michael.foord #18920: argparse module version action http://bugs.python.org/issue18920 closed by eli.bendersky #18922: Output versions of scripts to stdout http://bugs.python.org/issue18922 closed by serhiy.storchaka #18924: Enum members are easily replaced http://bugs.python.org/issue18924 closed by python-dev #18926: plistlib - str converted to bool http://bugs.python.org/issue18926 closed by VertigoRay #18928: Remove misleading documentation for random.shuffle http://bugs.python.org/issue18928 closed by dbenbenn #18933: Add link to source code in logging documentation http://bugs.python.org/issue18933 closed by python-dev #18938: Prepend Is Not A Word http://bugs.python.org/issue18938 closed by benjamin.peterson #18939: Venv docs regarding original python install http://bugs.python.org/issue18939 closed by python-dev #18940: TimedRotatingFileHandler and RotatingFileHandler fail to doRol http://bugs.python.org/issue18940 closed by python-dev #18941: RotatingFileHandler and TimedRotatingFileHandler do not respec http://bugs.python.org/issue18941 closed by python-dev #18942: _debugmallocstats() gibberish output on Windows http://bugs.python.org/issue18942 closed by tim.peters From jcea at jcea.es Fri Sep 6 18:14:26 2013 From: jcea at jcea.es (Jesus Cea) Date: Fri, 06 Sep 2013 18:14:26 +0200 Subject: [Python-Dev] DTRACE support In-Reply-To: <20130906174111.446ea2c4@pitrou.net> References: <5229EFE7.8040702@jcea.es> <20130906174111.446ea2c4@pitrou.net> Message-ID: <5229FF62.9020702@jcea.es> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 06/09/13 17:41, Antoine Pitrou wrote: > You should start by addressing review comments: > http://bugs.python.org/issue13405#msg151751 Antoine, my first step now is to poke Python-DEV about this subject. If the consensus is "DON'T" I will probably maintain this patch by myself. I saw big "BUTs" with this work in the bug tracker, so I rather prefer to ask for actual interest before investing in iron out patch implementation details. If devs actually want it, then next step is to improve current work to make it acceptable in mainline. There are quite a few ugly corners I would love to file out, beside your (really valuable) feedback. That is how I see it. > Right now, I agree with Charles-Fran?ois: your patch is too > intrusive. It is intrusive. Yes. I think it must be, by its own nature. Probably room for improvement and code transparency. But... are Python-DEVs interested in the project?. That is the point :) - -- Jes?s Cea Avi?n _/_/ _/_/_/ _/_/_/ jcea at jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ Twitter: @jcea _/_/ _/_/ _/_/_/_/_/ jabber / xmpp:jcea at jabber.org _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQCVAwUBUin/Yplgi5GaxT1NAQI6tAP+MaGbSm8j3xPBjMui9tePg2engB+G/ckh TzVac8Gz5yhWhixOFLk3cXYySieB6ttjsAG1NZYKxIwyLSLrxcL8pTh4c8oHB8zu /wYfIdXt6leH6HECXJquPLxLM39Z6KIOTt2QgKYSWy6ZHGzVaaeTFvrMhW0N/5rp SCHSJQc7Vn4= =65h4 -----END PGP SIGNATURE----- From ethan at stoneleaf.us Fri Sep 6 17:59:02 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 06 Sep 2013 08:59:02 -0700 Subject: [Python-Dev] inspect and metaclasses In-Reply-To: <20130906154408.BFAAD250845@webabinitio.net> References: <5229D97E.9080103@stoneleaf.us> <CAMSv6X1k30-PAWct_9tDTmwir4mDO0jAs-mbrvvBrJc5XE6VTQ@mail.gmail.com> <5229F141.2050007@stoneleaf.us> <20130906154408.BFAAD250845@webabinitio.net> Message-ID: <5229FBC6.1020708@stoneleaf.us> On 09/06/2013 08:44 AM, R. David Murray wrote: > On Fri, 06 Sep 2013 08:14:09 -0700, Ethan Furman <ethan at stoneleaf.us> wrote: >> On 09/06/2013 07:47 AM, Armin Rigo wrote: >>> >>> Are you suggesting that inspect.getmro(A) would return (A, object, >>> type)? That seems very wrong to me. >> >> Currently, `inspect.getmro(A)` returns `(A, object)`. > > Which matches A.__mro__. EOD, I think. I hope not, because currently this leaves a hole in the introspection of class attributes. Is __mro__ aimed primarily at instances and not classes? That seems to be how it works. In which case, do we need another __mmro__ (or __cmro__ or ...) to handle the mro of classes themselves? For the short term I can restrict the change to inspect.classify_class_attrs(). -- ~Ethan~ From rdmurray at bitdance.com Fri Sep 6 18:37:28 2013 From: rdmurray at bitdance.com (R. David Murray) Date: Fri, 06 Sep 2013 12:37:28 -0400 Subject: [Python-Dev] inspect and metaclasses In-Reply-To: <5229FBC6.1020708@stoneleaf.us> References: <5229D97E.9080103@stoneleaf.us> <CAMSv6X1k30-PAWct_9tDTmwir4mDO0jAs-mbrvvBrJc5XE6VTQ@mail.gmail.com> <5229F141.2050007@stoneleaf.us> <20130906154408.BFAAD250845@webabinitio.net> <5229FBC6.1020708@stoneleaf.us> Message-ID: <20130906163728.ACFAD2507F1@webabinitio.net> On Fri, 06 Sep 2013 08:59:02 -0700, Ethan Furman <ethan at stoneleaf.us> wrote: > On 09/06/2013 08:44 AM, R. David Murray wrote: > > On Fri, 06 Sep 2013 08:14:09 -0700, Ethan Furman <ethan at stoneleaf.us> wrote: > >> On 09/06/2013 07:47 AM, Armin Rigo wrote: > >>> > >>> Are you suggesting that inspect.getmro(A) would return (A, object, > >>> type)? That seems very wrong to me. > >> > >> Currently, `inspect.getmro(A)` returns `(A, object)`. > > > > Which matches A.__mro__. EOD, I think. > > I hope not, because currently this leaves a hole in the introspection of class attributes. > > Is __mro__ aimed primarily at instances and not classes? That seems to be how it works. In which case, do we need > another __mmro__ (or __cmro__ or ...) to handle the mro of classes themselves? Or maybe just a new inspect function? > For the short term I can restrict the change to inspect.classify_class_attrs(). Sounds like the best course. --David From ethan at stoneleaf.us Fri Sep 6 18:18:12 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 06 Sep 2013 09:18:12 -0700 Subject: [Python-Dev] when to fix cross-version bugs? In-Reply-To: <5229EBDA.1030902@stoneleaf.us> References: <5229EBDA.1030902@stoneleaf.us> Message-ID: <522A0044.10302@stoneleaf.us> On 09/06/2013 07:51 AM, Ethan Furman wrote: > > What guidelines determine when a bug is fixed in previous versions? On 09/06/2013 08:29 AM, Brian Curtin wrote: > > If it's a bug in that version and the version is accepting bug fixes, > i.e., not in security mode, go for it. This includes crossing the 2/3 > boundary if applicable. On 09/06/2013 08:31 AM, R. David Murray wrote: > > The basic guideline is: we try very hard not to break currently working > code in a maintenance release. Making that decision very much depends on > the details of each individual case. > > [so] it probably doesn't have much positive impact if it does get > backported, so is it worth the (small) chance of breaking someone's code? And they say never go to the elves for advice! ;) -- ~Ethan~ From solipsis at pitrou.net Fri Sep 6 19:02:47 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 6 Sep 2013 19:02:47 +0200 Subject: [Python-Dev] inspect and metaclasses References: <5229D97E.9080103@stoneleaf.us> <CAMSv6X1k30-PAWct_9tDTmwir4mDO0jAs-mbrvvBrJc5XE6VTQ@mail.gmail.com> <5229F141.2050007@stoneleaf.us> Message-ID: <20130906190247.61d5db99@fsol> On Fri, 06 Sep 2013 08:14:09 -0700 Ethan Furman <ethan at stoneleaf.us> wrote: > On 09/06/2013 07:47 AM, Armin Rigo wrote: > > > > Are you suggesting that inspect.getmro(A) would return (A, object, > > type)? That seems very wrong to me. > > Currently, `inspect.getmro(A)` returns `(A, object)`. > > Considering that Python actually will look in A's metaclass to find a class attribute, I think returning `(A, object, > type(A)` is appropriate: No, I don't think it's appropriate at all. You are conflating two things: - the instance lookup of an attribute (on A, here) - the class lookup when the instance lookup fails (on type, here) Both lookups obey the same MRO rules, it just happens that the second lookup uses a trivial MRO: >>> type.__mro__ (<class 'type'>, <class 'object'>) Regards Antoine. From solipsis at pitrou.net Fri Sep 6 19:05:16 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 6 Sep 2013 19:05:16 +0200 Subject: [Python-Dev] DTRACE support References: <5229EFE7.8040702@jcea.es> <20130906174111.446ea2c4@pitrou.net> <5229FF62.9020702@jcea.es> Message-ID: <20130906190516.471028ba@fsol> On Fri, 06 Sep 2013 18:14:26 +0200 Jesus Cea <jcea at jcea.es> wrote: > > > Right now, I agree with Charles-Fran?ois: your patch is too > > intrusive. > > It is intrusive. Yes. I think it must be, by its own nature. Probably > room for improvement and code transparency. But... are Python-DEVs > interested in the project?. That is the point :) Well, I'm not *personally* interested in anything that only addresses Solaris, OS X and the like :) And, no, it doesn't have to be *that* intrusive. Take a look at Dave Malcolm's systemtap patch, which IIRC takes a much more sensible approach. Regards Antoine. From solipsis at pitrou.net Fri Sep 6 19:06:21 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 6 Sep 2013 19:06:21 +0200 Subject: [Python-Dev] Details of "import lock" in 3.3 References: <5229EFEC.1010503@jcea.es> <20130906174313.63e37ac3@pitrou.net> <5229FC98.9080106@jcea.es> Message-ID: <20130906190621.6b1b8a01@fsol> On Fri, 06 Sep 2013 18:02:32 +0200 Jesus Cea <jcea at jcea.es> wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > On 06/09/13 17:43, Antoine Pitrou wrote: > > Quick summary here: > > http://docs.python.org/3/whatsnew/3.3.html#a-finer-grained-import-lock > > > > Otherwise, I'm afraid the source code has the most information, > > e.g.: > > http://hg.python.org/cpython/file/1d88d04aade2/Lib/importlib/_bootstrap.py#l124 > > Yes, > > > I was depending of the global import lock in my code. I am > evaluating impact under Python 3.3. > > Under Python 3.3, metafinder/loader MUST BE reentrant?. No, they shouldn't have to. Normally, the global import lock is still held at this point. However, it is best to check the code if you want to be sure. Regards Antoine. From cf.natali at gmail.com Fri Sep 6 19:12:46 2013 From: cf.natali at gmail.com (=?ISO-8859-1?Q?Charles=2DFran=E7ois_Natali?=) Date: Fri, 6 Sep 2013 19:12:46 +0200 Subject: [Python-Dev] DTRACE support In-Reply-To: <5229FB08.20603@jcea.es> References: <5229EFE7.8040702@jcea.es> <CAH_1eM2gG2fVsXU88j1T3y8TeGJ+fnDu4BBR2aMVg2hw=37_pg@mail.gmail.com> <5229FB08.20603@jcea.es> Message-ID: <CAH_1eM3ex5PpNY7icxH2zCewB4LZ8nbwKFYcOwmP3iQBuhicBw@mail.gmail.com> > The main value of DTrace is systemwide observability. You can see > something "strange" at kernel level and trace it to a particular line > of code in a random Python script. There is no other tool that can do > that. You have complete transversal observability of ALL the code > running in your computer, kernel or usermode, clean reports with > threads, etc. Don't get me wrong, I'm not saying DTrace is useless. I'm just saying that, as far as I'm concerned, I've never had any trouble debugging/tunning a Python script with non-intrusive tools (strace, gdb, valgrind, and oprofile for profiling). Of course, this includes analysing bug reports. > Maybe the biggest objection would be that most python-devs are running > Linux, and you don't have dtrace support on linux unless you are > running Oracle distribution. But world is larger than linux, and there > are some efforts to port DTrace to Linux itself. DTrace is available > on Solaris and derivatives, MacOS X and FreeBSD. That's true, I might have a different opinion if I used Solaris. But that's not the case, so te me, the cognitive overhead incurred by this large patch isn't worth it. So I'm -1, but that's a personal opinion :-) cf From dcallahan at mozilla.com Fri Sep 6 19:22:49 2013 From: dcallahan at mozilla.com (Dan Callahan) Date: Fri, 06 Sep 2013 12:22:49 -0500 Subject: [Python-Dev] Offtopic: OpenID Providers In-Reply-To: <5228C00F.7000209@jcea.es> References: <5228C00F.7000209@jcea.es> Message-ID: <l0d311$b7n$1@ger.gmane.org> On 9/5/13 12:31 PM, Jesus Cea wrote: > I have big hopes for Mozilla Persona, looking forward > Python infrastructure support :). Hi, I'm the project lead on Persona signin, and I spoke at PyCon earlier this year regarding why and how Mozilla is building Persona. If you'd like some more background, that video [0] is worth a look. Let's pull this discussion up a level: It sounds like many people (Jesus, Donald, Toshio, Barry, Tres, Dirkjan, etc.) are interested in seeing Persona on Python.org properties, and most of the objections coming from a place of "Persona hasn't gone viral, what if this is wasted effort?" We can tackle that from two angles: 1. Dirkjan and I are willing to do the work to make this happen if someone from python-devel is willing to guide us through the contributor process for these systems. 2. There's a seamless migration path away from Persona if we fail: fall back to the pre-existing traditional email/password system using the same email addresses that Persona had previously been in charge of verifying. So let's do this. The open web deserves better than just Google+, Facebook, or Passwords, and visible support from the Python community would be a huge step toward answering the chicken-and-egg objections raised in this thread. At your service, -Callahad PS: Freeform OpenID has utterly failed as a user-empowering authentication system, and the protocol itself is rapidly being supplanted by vendor-specific OAuth[1] systems. If we want to ensure that "you *can* (not *must*) use free and open services to access our resources," then we must provide an option to use something akin to Persona. [0]: http://pyvideo.org/video/1764 [1]: "Google's OpenID service is being replaced by Login with OAuth 2.0." https://developers.google.com/accounts/docs/GettingStarted From ethan at stoneleaf.us Fri Sep 6 19:01:32 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 06 Sep 2013 10:01:32 -0700 Subject: [Python-Dev] inspect and metaclasses In-Reply-To: <20130906163728.ACFAD2507F1@webabinitio.net> References: <5229D97E.9080103@stoneleaf.us> <CAMSv6X1k30-PAWct_9tDTmwir4mDO0jAs-mbrvvBrJc5XE6VTQ@mail.gmail.com> <5229F141.2050007@stoneleaf.us> <20130906154408.BFAAD250845@webabinitio.net> <5229FBC6.1020708@stoneleaf.us> <20130906163728.ACFAD2507F1@webabinitio.net> Message-ID: <522A0A6C.9040401@stoneleaf.us> On 09/06/2013 09:37 AM, R. David Murray wrote: > On Fri, 06 Sep 2013 08:59:02 -0700, Ethan Furman <ethan at stoneleaf.us> wrote: >> >> For the short term I can restrict the change to inspect.classify_class_attrs(). > > Sounds like the best course. There is one other function in inspect that calls getmro(): def getmembers(object, predicate=None): """Return all members of an object as (name, value) pairs sorted by name. Optionally, only return members that satisfy a given predicate.""" if isclass(object): mro = (object,) + getmro(object) Should I add `+ getmro(type(object))` here as well? -- ~Ethan~ From guido at python.org Fri Sep 6 20:01:39 2013 From: guido at python.org (Guido van Rossum) Date: Fri, 6 Sep 2013 11:01:39 -0700 Subject: [Python-Dev] DTRACE support In-Reply-To: <CAH_1eM3ex5PpNY7icxH2zCewB4LZ8nbwKFYcOwmP3iQBuhicBw@mail.gmail.com> References: <5229EFE7.8040702@jcea.es> <CAH_1eM2gG2fVsXU88j1T3y8TeGJ+fnDu4BBR2aMVg2hw=37_pg@mail.gmail.com> <5229FB08.20603@jcea.es> <CAH_1eM3ex5PpNY7icxH2zCewB4LZ8nbwKFYcOwmP3iQBuhicBw@mail.gmail.com> Message-ID: <CAP7+vJKWgsTv_xUB2MZftzeeo0MRfP6oQmS45_9P_brYMFxb=g@mail.gmail.com> I think there are a couple of issues. - In order for the patch to be acceptable, we'd need someone to take responsibility for owning and maintaining it for at least several years. Jesus, are you willing to commit to this? - I think it's not all that important whether any core developer would want to use this for themselves; it's enough if it's useful for others to have this, and it's clear that at least *some* users really like this. Different people like different tools, and that's fine. - There are good reasons (e.g. standardizing a set of probes) for having a somewhat supported version around. - Since this *has* to be done in the form of a patch plus a new configure option, an externally-maintained patch is awkward at least, and likely to be continually out of date -- this is not something that you can just as easily put on PyPI as the recommended 3rd party solution. - So I'm personally inclined to say "yes, Jesus, if you are willing to maintain this code (while it is in the core) for several more years, please invest more of your time, and, assuming you can satisfy the code reviewers' feedback, we want to accept this in the core, hopefully in 3.4 if you get it in an acceptable state before 3.4 beta 1 is released." On Fri, Sep 6, 2013 at 10:12 AM, Charles-Fran?ois Natali <cf.natali at gmail.com> wrote: >> The main value of DTrace is systemwide observability. You can see >> something "strange" at kernel level and trace it to a particular line >> of code in a random Python script. There is no other tool that can do >> that. You have complete transversal observability of ALL the code >> running in your computer, kernel or usermode, clean reports with >> threads, etc. > > Don't get me wrong, I'm not saying DTrace is useless. > I'm just saying that, as far as I'm concerned, I've never had any > trouble debugging/tunning a Python script with non-intrusive tools > (strace, gdb, valgrind, and oprofile for profiling). Of course, this > includes analysing bug reports. > >> Maybe the biggest objection would be that most python-devs are running >> Linux, and you don't have dtrace support on linux unless you are >> running Oracle distribution. But world is larger than linux, and there >> are some efforts to port DTrace to Linux itself. DTrace is available >> on Solaris and derivatives, MacOS X and FreeBSD. > > That's true, I might have a different opinion if I used Solaris. But > that's not the case, so te me, the cognitive overhead incurred by this > large patch isn't worth it. > > So I'm -1, but that's a personal opinion :-) > > cf > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%40python.org -- --Guido van Rossum (python.org/~guido) From tjreedy at udel.edu Fri Sep 6 20:24:57 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 06 Sep 2013 14:24:57 -0400 Subject: [Python-Dev] unicodedata module is out of date In-Reply-To: <CAPMPPoRMBsBbAasu8ewEwqJOJARLAsctJricjKdVYVkXdLy8Tg@mail.gmail.com> References: <CAPMPPoRq1Uuag3BP0OnfsJPhKVsZapN3JS03i+dN2enAj=saYg@mail.gmail.com> <5229F6DB.40000@mrabarnett.plus.com> <CAPMPPoRMBsBbAasu8ewEwqJOJARLAsctJricjKdVYVkXdLy8Tg@mail.gmail.com> Message-ID: <l0d6lj$lnf$1@ger.gmane.org> On 9/6/2013 11:55 AM, Andrew Miller wrote: > I've just checked on Python 2.7.5 and Python 3.3.2 (Win32 versions). > > In Python 3.3.2 unicodedata.unidata_version is set to '6.1.0'. > > In Python 2.7.5 it is set to '5.2.0' so it looks as though this version > is no longer being updated. In general, new features do not go into bugfix releases (x.y.z, where z >= 1). Updating the unidate_version add new features to the unicodedata module. -- Terry Jan Reedy From solipsis at pitrou.net Fri Sep 6 20:33:34 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 6 Sep 2013 20:33:34 +0200 Subject: [Python-Dev] DTRACE support References: <5229EFE7.8040702@jcea.es> <20130906174111.446ea2c4@pitrou.net> <5229FF62.9020702@jcea.es> Message-ID: <20130906203334.1a99c99d@fsol> On Fri, 06 Sep 2013 18:14:26 +0200 Jesus Cea <jcea at jcea.es> wrote: > > It is intrusive. Yes. I think it must be, by its own nature. Probably > room for improvement and code transparency. But... are Python-DEVs > interested in the project?. That is the point :) As a concrete data point: - here are Dave's modifications to ceval.c for systemtap: http://bugs.python.org/review/14776/diff/5177/Python/ceval.c - here are your modifications to ceval.c for dtrace: http://bugs.python.org/review/13405/diff/6151/Python/ceval.c Regards Antoine. From rdmurray at bitdance.com Fri Sep 6 20:34:54 2013 From: rdmurray at bitdance.com (R. David Murray) Date: Fri, 06 Sep 2013 14:34:54 -0400 Subject: [Python-Dev] inspect and metaclasses In-Reply-To: <522A0A6C.9040401@stoneleaf.us> References: <5229D97E.9080103@stoneleaf.us> <CAMSv6X1k30-PAWct_9tDTmwir4mDO0jAs-mbrvvBrJc5XE6VTQ@mail.gmail.com> <5229F141.2050007@stoneleaf.us> <20130906154408.BFAAD250845@webabinitio.net> <5229FBC6.1020708@stoneleaf.us> <20130906163728.ACFAD2507F1@webabinitio.net> <522A0A6C.9040401@stoneleaf.us> Message-ID: <20130906183455.28546250816@webabinitio.net> On Fri, 06 Sep 2013 10:01:32 -0700, Ethan Furman <ethan at stoneleaf.us> wrote: > On 09/06/2013 09:37 AM, R. David Murray wrote: > > On Fri, 06 Sep 2013 08:59:02 -0700, Ethan Furman <ethan at stoneleaf.us> wrote: > >> > >> For the short term I can restrict the change to inspect.classify_class_attrs(). > > > > Sounds like the best course. > > There is one other function in inspect that calls getmro(): > > def getmembers(object, predicate=None): > """Return all members of an object as (name, value) pairs sorted by name. > Optionally, only return members that satisfy a given predicate.""" > if isclass(object): > mro = (object,) + getmro(object) > > Should I add `+ getmro(type(object))` here as well? Not unless you want to void the warranty on the docs :) Note getmembers() does not return metaclass attributes when the argument is a class (this behavior is inherited from the dir() function). --David From donald at stufft.io Fri Sep 6 20:53:00 2013 From: donald at stufft.io (Donald Stufft) Date: Fri, 6 Sep 2013 14:53:00 -0400 Subject: [Python-Dev] Offtopic: OpenID Providers In-Reply-To: <l0d311$b7n$1@ger.gmane.org> References: <5228C00F.7000209@jcea.es> <l0d311$b7n$1@ger.gmane.org> Message-ID: <6DB0A7E3-FADD-4E35-A983-02A7C0D4CAE3@stufft.io> On Sep 6, 2013, at 1:22 PM, Dan Callahan <dcallahan at mozilla.com> wrote: > On 9/5/13 12:31 PM, Jesus Cea wrote: >> I have big hopes for Mozilla Persona, looking forward >> Python infrastructure support :). > > Hi, I'm the project lead on Persona signin, and I spoke at PyCon earlier this year regarding why and how Mozilla is building Persona. If you'd like some more background, that video [0] is worth a look. > > Let's pull this discussion up a level: > > It sounds like many people (Jesus, Donald, Toshio, Barry, Tres, Dirkjan, etc.) are interested in seeing Persona on Python.org properties, and most of the objections coming from a place of "Persona hasn't gone viral, what if this is wasted effort?" > > We can tackle that from two angles: > > 1. Dirkjan and I are willing to do the work to make this happen if someone from python-devel is willing to guide us through the contributor process for these systems. FWIW I'm a maintainer of PyPI and I do plan on enabling Persona there. Mostly blocked because I want to focus my PyPI efforts on the "next gen" code base instead. > > 2. There's a seamless migration path away from Persona if we fail: fall back to the pre-existing traditional email/password system using the same email addresses that Persona had previously been in charge of verifying. > > So let's do this. The open web deserves better than just Google+, Facebook, or Passwords, and visible support from the Python community would be a huge step toward answering the chicken-and-egg objections raised in this thread. > > At your service, > -Callahad > > PS: Freeform OpenID has utterly failed as a user-empowering authentication system, and the protocol itself is rapidly being supplanted by vendor-specific OAuth[1] systems. If we want to ensure that "you *can* (not *must*) use free and open services to access our resources," then we must provide an option to use something akin to Persona. > > [0]: http://pyvideo.org/video/1764 > > [1]: "Google's OpenID service is being replaced by Login with OAuth 2.0." https://developers.google.com/accounts/docs/GettingStarted > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: <http://mail.python.org/pipermail/python-dev/attachments/20130906/d60c5f28/attachment.sig> From rdmurray at bitdance.com Fri Sep 6 21:11:29 2013 From: rdmurray at bitdance.com (R. David Murray) Date: Fri, 06 Sep 2013 15:11:29 -0400 Subject: [Python-Dev] Offtopic: OpenID Providers In-Reply-To: <6DB0A7E3-FADD-4E35-A983-02A7C0D4CAE3@stufft.io> References: <5228C00F.7000209@jcea.es> <l0d311$b7n$1@ger.gmane.org> <6DB0A7E3-FADD-4E35-A983-02A7C0D4CAE3@stufft.io> Message-ID: <20130906191130.B6511250816@webabinitio.net> On Fri, 06 Sep 2013 14:53:00 -0400, Donald Stufft <donald at stufft.io> wrote: > > On Sep 6, 2013, at 1:22 PM, Dan Callahan <dcallahan at mozilla.com> wrote: > > > On 9/5/13 12:31 PM, Jesus Cea wrote: > >> I have big hopes for Mozilla Persona, looking forward > >> Python infrastructure support :). > > > > Hi, I'm the project lead on Persona signin, and I spoke at PyCon > > earlier this year regarding why and how Mozilla is building Persona. > > If you'd like some more background, that video [0] is worth a look. > > > > Let's pull this discussion up a level: > > > > It sounds like many people (Jesus, Donald, Toshio, Barry, Tres, > > Dirkjan, etc.) are interested in seeing Persona on Python.org > > properties, and most of the objections coming from a place of > > "Persona hasn't gone viral, what if this is wasted effort?" > > > > We can tackle that from two angles: > > > > 1. Dirkjan and I are willing to do the work to make this happen if > > someone from python-devel is willing to guide us through the > > contributor process for these systems. Thanks. I'm one of the people with admin access to the bug tracker (I haven't done much maint lately, though, Ezio has done the most). There is information on setting up a replica of our production system here: https://wiki.python.org/moin/TrackerDevelopment If you want to start hacking on a solution, the first step would be to spin up a test setup. If you propose a patch, either I or Ezio should be able to find the time to review and apply it, if you also commit to maintaining it ;) Tracker specific discussion happens on the tracker-discuss mailing list, by the way (very low traffic). > > 2. There's a seamless migration path away from Persona if we fail: > > fall back to the pre-existing traditional email/password system > > using the same email addresses that Persona had previously been in > > charge of verifying. Roundup uses database-derived numeric IDs. An email is associated with each account, but does not participate in authentication or authorization after initial signup. (Except for the email interface...but that is a separate story and you shouldn't need to address that). > > So let's do this. The open web deserves better than just Google+, > > Facebook, or Passwords, and visible support from the Python > > community would be a huge step toward answering the chicken-and-egg > > objections raised in this thread. > > > > At your service, > > -Callahad > > > > PS: Freeform OpenID has utterly failed as a user-empowering > > authentication system, and the protocol itself is rapidly being > > supplanted by vendor-specific OAuth[1] systems. If we want to ensure > > that "you *can* (not *must*) use free and open services to access > > our resources," then we must provide an option to use something akin > > to Persona. IMO, single signon is overrated. Especially if one prefers not to make it easy for various accounts to be automatically associated with one another by various entities who shall remain nameless but have been in the news a lot lately :) --David From donald at stufft.io Fri Sep 6 21:17:12 2013 From: donald at stufft.io (Donald Stufft) Date: Fri, 6 Sep 2013 15:17:12 -0400 Subject: [Python-Dev] Offtopic: OpenID Providers In-Reply-To: <20130906191130.B6511250816@webabinitio.net> References: <5228C00F.7000209@jcea.es> <l0d311$b7n$1@ger.gmane.org> <6DB0A7E3-FADD-4E35-A983-02A7C0D4CAE3@stufft.io> <20130906191130.B6511250816@webabinitio.net> Message-ID: <7087E6B8-1CD4-4002-AD3F-5578B045A3CB@stufft.io> On Sep 6, 2013, at 3:11 PM, "R. David Murray" <rdmurray at bitdance.com> wrote: > IMO, single signon is overrated. Especially if one prefers not to make > it easy for various accounts to be automatically associated with one > another by various entities who shall remain nameless but have been in > the news a lot lately :) If I recall Persona doesn't leak this data like OpenID does, but perhaps Dan can speak to that better than I can. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: <http://mail.python.org/pipermail/python-dev/attachments/20130906/224cd0ea/attachment.sig> From rdmurray at bitdance.com Fri Sep 6 21:34:30 2013 From: rdmurray at bitdance.com (R. David Murray) Date: Fri, 06 Sep 2013 15:34:30 -0400 Subject: [Python-Dev] Offtopic: OpenID Providers In-Reply-To: <7087E6B8-1CD4-4002-AD3F-5578B045A3CB@stufft.io> References: <5228C00F.7000209@jcea.es> <l0d311$b7n$1@ger.gmane.org> <6DB0A7E3-FADD-4E35-A983-02A7C0D4CAE3@stufft.io> <20130906191130.B6511250816@webabinitio.net> <7087E6B8-1CD4-4002-AD3F-5578B045A3CB@stufft.io> Message-ID: <20130906193430.AFA14250816@webabinitio.net> On Fri, 06 Sep 2013 15:17:12 -0400, Donald Stufft <donald at stufft.io> wrote: > On Sep 6, 2013, at 3:11 PM, "R. David Murray" <rdmurray at bitdance.com> wrote: > > > IMO, single signon is overrated. Especially if one prefers not to make > > it easy for various accounts to be automatically associated with one > > another by various entities who shall remain nameless but have been in > > the news a lot lately :) > > If I recall Persona doesn't leak this data like OpenID does, but > perhaps Dan can speak to that better than I can. Note that I said that single signon *itself* was overrated. If you use the same token to authenticate to multiple sites (and here the 'token' is the email address) then your identities on those sites are ipso facto associated with each other. *If* that email address is also never leaked (never displayed, even to other signed on users, all communication with the site encrypted), then you only have to worry if the sites exchange information about their accounts, or if the government comes knocking on their doors.... Yes, I'm paranoid. That doesn't mean they aren't listening. That said, sometimes you *want* identities to be associated, so I'm not saying Persona is a bad thing. Just that single signon is overrated. --David From donald at stufft.io Fri Sep 6 21:40:33 2013 From: donald at stufft.io (Donald Stufft) Date: Fri, 6 Sep 2013 15:40:33 -0400 Subject: [Python-Dev] Offtopic: OpenID Providers In-Reply-To: <20130906193430.AFA14250816@webabinitio.net> References: <5228C00F.7000209@jcea.es> <l0d311$b7n$1@ger.gmane.org> <6DB0A7E3-FADD-4E35-A983-02A7C0D4CAE3@stufft.io> <20130906191130.B6511250816@webabinitio.net> <7087E6B8-1CD4-4002-AD3F-5578B045A3CB@stufft.io> <20130906193430.AFA14250816@webabinitio.net> Message-ID: <ADFAA327-A61E-42AB-8981-87645953FA95@stufft.io> On Sep 6, 2013, at 3:34 PM, "R. David Murray" <rdmurray at bitdance.com> wrote: > On Fri, 06 Sep 2013 15:17:12 -0400, Donald Stufft <donald at stufft.io> wrote: >> On Sep 6, 2013, at 3:11 PM, "R. David Murray" <rdmurray at bitdance.com> wrote: >> >>> IMO, single signon is overrated. Especially if one prefers not to make >>> it easy for various accounts to be automatically associated with one >>> another by various entities who shall remain nameless but have been in >>> the news a lot lately :) >> >> If I recall Persona doesn't leak this data like OpenID does, but >> perhaps Dan can speak to that better than I can. > > Note that I said that single signon *itself* was overrated. If you use > the same token to authenticate to multiple sites (and here the 'token' > is the email address) then your identities on those sites are ipso facto > associated with each other. *If* that email address is also never > leaked (never displayed, even to other signed on users, all communication > with the site encrypted), then you only have to worry if the > sites exchange information about their accounts, or if the government > comes knocking on their doors.... > > Yes, I'm paranoid. That doesn't mean they aren't listening. > > That said, sometimes you *want* identities to be associated, so I'm not > saying Persona is a bad thing. Just that single signon is overrated. Well that's fine to have that opinion but I think you're under estimating just how easy it is to link two disparate accounts especially if you have the cooperation (willing or otherwise) of the site operators. I've personally seen Google do some particularly amazing connections between accounts that I don't believe using the same authentication token is going to make that any easier or harder for them. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: <http://mail.python.org/pipermail/python-dev/attachments/20130906/af163ab9/attachment.sig> From catch-all at masklinn.net Fri Sep 6 21:50:38 2013 From: catch-all at masklinn.net (Xavier Morel) Date: Fri, 6 Sep 2013 21:50:38 +0200 Subject: [Python-Dev] DTRACE support In-Reply-To: <20130906190516.471028ba@fsol> References: <5229EFE7.8040702@jcea.es> <20130906174111.446ea2c4@pitrou.net> <5229FF62.9020702@jcea.es> <20130906190516.471028ba@fsol> Message-ID: <8E3B67D3-367C-440E-9268-87845E44BB15@masklinn.net> On 2013-09-06, at 19:05 , Antoine Pitrou wrote: > On Fri, 06 Sep 2013 18:14:26 +0200 > Jesus Cea <jcea at jcea.es> wrote: >> >>> Right now, I agree with Charles-Fran?ois: your patch is too >>> intrusive. >> >> It is intrusive. Yes. I think it must be, by its own nature. Probably >> room for improvement and code transparency. But... are Python-DEVs >> interested in the project?. That is the point :) > > Well, I'm not *personally* interested in anything that only addresses > Solaris, OS X and the like :) For what it's worth, there's also a linux port and oracle's distro has dtrace support. > And, no, it doesn't have to be *that* intrusive. Take a look at Dave > Malcolm's systemtap patch, which IIRC takes a much more sensible > approach. Is there a possibility of compatibility there, using the same placeholders for a --with-dtrace and --with-systemtap build? Jesus seems to instrument more points than Dave, but the extra points could just be defined to nothing in the systemtap implementation. From jcea at jcea.es Sat Sep 7 05:40:04 2013 From: jcea at jcea.es (Jesus Cea) Date: Sat, 07 Sep 2013 05:40:04 +0200 Subject: [Python-Dev] DTRACE support In-Reply-To: <20130906203334.1a99c99d@fsol> References: <5229EFE7.8040702@jcea.es> <20130906174111.446ea2c4@pitrou.net> <5229FF62.9020702@jcea.es> <20130906203334.1a99c99d@fsol> Message-ID: <522AA014.4060801@jcea.es> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 06/09/13 20:33, Antoine Pitrou wrote: > On Fri, 06 Sep 2013 18:14:26 +0200 Jesus Cea <jcea at jcea.es> wrote: >> >> It is intrusive. Yes. I think it must be, by its own nature. >> Probably room for improvement and code transparency. But... are >> Python-DEVs interested in the project?. That is the point :) > > As a concrete data point: - here are Dave's modifications to > ceval.c for systemtap: > http://bugs.python.org/review/14776/diff/5177/Python/ceval.c - here > are your modifications to ceval.c for dtrace: > http://bugs.python.org/review/13405/diff/6151/Python/ceval.c Unfair, because that code is not doing the same thing. Most of the extra complexity is there to deal with DTRACE ability to provide meaningful stackframes, with Python code instead of CPython evaluation loop. This is kind of magical. Look at this: "" [root at stargate-host /]# ps -lAf|grep -i correo 0 S correo 1371 1 0 40 20 ? 277063 ? Jul 29 ? 90:43 /usr/local/bin/python /export/home/ 0 S root 20397 20373 0 40 20 ? 1294 ? 05:18:16 pts/12 0:00 grep -i correo [root at stargate-host /]# dtrace -n 'python1371:::function-entry {jstack();}' 5 3164 PyEval_EvalFrameEx:function-entry libpython2.7.so.1.0`PyEval_EvalFrameEx+0x89a [ /usr/local/lib/python2.7/logging/__init__.py:1332 (isEnabledFor) ] libpython2.7.so.1.0`PyEval_EvalFrameEx+0x6231 [ /usr/local/lib/python2.7/logging/__init__.py:1202 (log) ] libpython2.7.so.1.0`PyEval_EvalCodeEx+0x7b4 libpython2.7.so.1.0`PyEval_EvalFrameEx+0x59ae [ /home/correo/durus/connection.py:483 (shrink) ] libpython2.7.so.1.0`PyEval_EvalFrameEx+0x6231 [ /home/correo/durus/connection.py:225 (shrink_cache) ] libpython2.7.so.1.0`PyEval_EvalFrameEx+0x6231 [ /home/correo/durus/connection.py:303 (commit2) ] libpython2.7.so.1.0`PyEval_EvalFrameEx+0x6231 [ /home/correo/durus/connection.py:261 (commit) ] libpython2.7.so.1.0`PyEval_EvalFrameEx+0x6231 [ /export/home/correo/lmtp.py:191 (_monitor) ] libpython2.7.so.1.0`PyEval_EvalCodeEx+0x7b4 libpython2.7.so.1.0`PyEval_EvalFrameEx+0x59ae [ /export/home/correo/lmtp.py:125 (process_vacation2) ] libpython2.7.so.1.0`PyEval_EvalCodeEx+0x7b4 libpython2.7.so.1.0`PyEval_EvalFrameEx+0x59ae [ /export/home/correo/lmtp.py:138 (process_vacation) ] libpython2.7.so.1.0`PyEval_EvalCodeEx+0x7b4 libpython2.7.so.1.0`function_call+0x192 libpython2.7.so.1.0`PyObject_Call+0x5c libpython2.7.so.1.0`PyEval_EvalFrameEx+0x2b94 [ /usr/local/lib/python2.7/threading.py: ] libpython2.7.so.1.0`PyEval_EvalFrameEx+0x6231 libpython2.7.so.1.0`PyEval_EvalFrameEx+0x6231 libpython2.7.so.1.0`PyEval_EvalCodeEx+0x7b4 libpython2.7.so.1.0`function_call+0xa4 libpython2.7.so.1.0`PyObject_Call+0x5c libpython2.7.so.1.0`instancemethod_call+0xa1 libpython2.7.so.1.0`PyObject_Call+0x5c libpython2.7.so.1.0`PyEval_CallObjectWithKeywords+0x58 libpython2.7.so.1.0`t_bootstrap+0x52 libc.so.1`_thr_setup+0x4e libc.so.1`_lwp_start ^C """ Lets say I want to know about what codepaths are hitting the OS "sync" features, so I do this: """ [root at stargate-host /]# dtrace -n 'syscall::fdsync:entry/pid==1371/{jstack();}' dtrace: description 'syscall::fdsync:entry' matched 1 probe CPU ID FUNCTION:NAME 3 58344 fdsync:entry libc.so.1`__fdsync+0x15 libdb-6.0.so`__os_fsync+0x91 libdb-6.0.so`__log_flush_int+0x685 libdb-6.0.so`__log_flush+0x6a libdb-6.0.so`__log_flush_pp+0x121 _pybsddb.so`DBEnv_log_flush+0x39 libpython2.7.so.1.0`PyEval_EvalFrameEx+0x6992 [ /export/home/correo/durus-berkeleydbstorage/berkeleydb_storage.py:918 (flush) ] libpython2.7.so.1.0`PyEval_EvalFrameEx+0x6231 [ /export/home/correo/lmtp.py:1005 (lmtp2) ] libpython2.7.so.1.0`PyEval_EvalFrameEx+0x6231 [ /export/home/correo/lmtp.py:763 (lmtp) ] libpython2.7.so.1.0`PyEval_EvalCodeEx+0x7b4 libpython2.7.so.1.0`function_call+0x192 libpython2.7.so.1.0`PyObject_Call+0x5c libpython2.7.so.1.0`PyEval_EvalFrameEx+0x2b94 [ /export/home/correo/lmtp.py:214 (_ignore_socket_errors) ] libpython2.7.so.1.0`PyEval_EvalCodeEx+0x7b4 libpython2.7.so.1.0`function_call+0x192 libpython2.7.so.1.0`PyObject_Call+0x5c libpython2.7.so.1.0`PyEval_EvalFrameEx+0x2b94 [ /usr/local/lib/python2.7/threading.py:504 (run) ] libpython2.7.so.1.0`PyEval_EvalFrameEx+0x6231 [ /usr/local/lib/python2.7/threading.py:551 (__bootstrap_inner) ] libpython2.7.so.1.0`PyEval_EvalFrameEx+0x6231 [ /usr/local/lib/python2.7/threading.py:524 (__bootstrap) ] libpython2.7.so.1.0`PyEval_EvalCodeEx+0x7b4 libpython2.7.so.1.0`function_call+0xa4 libpython2.7.so.1.0`PyObject_Call+0x5c libpython2.7.so.1.0`instancemethod_call+0xa1 libpython2.7.so.1.0`PyObject_Call+0x5c libpython2.7.so.1.0`PyEval_CallObjectWithKeywords+0x58 libpython2.7.so.1.0`t_bootstrap+0x52 libc.so.1`_thr_setup+0x4e libc.so.1`_lwp_start """ I could even aggregate stacktraces and print them by frequency, cost, whatever. Be aware that to provide this information I am not even using the probes I have integrated in Python. The program is in production and I don't need to stop it to make those measurements. They are 100% "live", not interrupting the service and without any kind of support in that Python program. I can even plug to TCP Dtrace providers in kernel and know what exact line of Python code is sending those bytes that are being delayed because the Nagle algorithm <https://en.wikipedia.org/wiki/Nagle%27s_algorithm>. I actually digged this five years ago. what I hunt it was! :-) PS: And it is supporting Unicode :-), another big complexity in the code. - -- Jes?s Cea Avi?n _/_/ _/_/_/ _/_/_/ jcea at jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ Twitter: @jcea _/_/ _/_/ _/_/_/_/_/ jabber / xmpp:jcea at jabber.org _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQCUAwUBUiqgFJlgi5GaxT1NAQLezQP2PRkvn6+9Zev6GlGu14hfYTpwAMie9gcn tEieelaQniisLy22oj/MO2d2j78qlH4RoU3U/vzOTPru6uzt3OKy0UBivnBFqTcc fD/dPUCcfkigZO4nBzcqjAP4lz93wcDkx1Y8RcL3ygxeI6fSe8JKdyZKyXvyRf/9 f+3MfWJVDA== =29tQ -----END PGP SIGNATURE----- From python-dev at masklinn.net Sat Sep 7 08:57:07 2013 From: python-dev at masklinn.net (Xavier Morel) Date: Sat, 7 Sep 2013 08:57:07 +0200 Subject: [Python-Dev] DTRACE support In-Reply-To: <522AA014.4060801@jcea.es> References: <5229EFE7.8040702@jcea.es> <20130906174111.446ea2c4@pitrou.net> <5229FF62.9020702@jcea.es> <20130906203334.1a99c99d@fsol> <522AA014.4060801@jcea.es> Message-ID: <557641DC-8E6B-450C-88A0-A3E2941678C7@masklinn.net> On 2013-09-07, at 05:40 , Jesus Cea wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > On 06/09/13 20:33, Antoine Pitrou wrote: >> On Fri, 06 Sep 2013 18:14:26 +0200 Jesus Cea <jcea at jcea.es> wrote: >>> >>> It is intrusive. Yes. I think it must be, by its own nature. >>> Probably room for improvement and code transparency. But... are >>> Python-DEVs interested in the project?. That is the point :) >> >> As a concrete data point: - here are Dave's modifications to >> ceval.c for systemtap: >> http://bugs.python.org/review/14776/diff/5177/Python/ceval.c - here >> are your modifications to ceval.c for dtrace: >> http://bugs.python.org/review/13405/diff/6151/Python/ceval.c > > Unfair, because that code is not doing the same thing. > > Most of the extra complexity is there to deal with DTRACE ability to > provide meaningful stackframes, with Python code instead of CPython > evaluation loop. This is kind of magical. Antoine will correct me if I'm wrong, but I believe his point is less about the complexity of dtrace provision and more about how much of it lives in ceval.c: the systemtap provision also takes quite a bit of code, but almost all of that code is extracted into a separate file and only a pair of calls live in ceval.c You patch, because it adds quite a bit of complexity to ceval.c, makes reading it significantly more difficult (especially for people who don't care for probe implementations). Dave's more or less doesn't change the complexity of that file (people who don't care for probes don't have to follow the macros to know what they do). From solipsis at pitrou.net Sat Sep 7 11:06:13 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 7 Sep 2013 11:06:13 +0200 Subject: [Python-Dev] DTRACE support References: <5229EFE7.8040702@jcea.es> <20130906174111.446ea2c4@pitrou.net> <5229FF62.9020702@jcea.es> <20130906203334.1a99c99d@fsol> <522AA014.4060801@jcea.es> <557641DC-8E6B-450C-88A0-A3E2941678C7@masklinn.net> Message-ID: <20130907110613.71a9a98d@fsol> On Sat, 7 Sep 2013 08:57:07 +0200 Xavier Morel <python-dev at masklinn.net> wrote: > > On 2013-09-07, at 05:40 , Jesus Cea wrote: > > > -----BEGIN PGP SIGNED MESSAGE----- > > Hash: SHA1 > > > > On 06/09/13 20:33, Antoine Pitrou wrote: > >> On Fri, 06 Sep 2013 18:14:26 +0200 Jesus Cea <jcea at jcea.es> wrote: > >>> > >>> It is intrusive. Yes. I think it must be, by its own nature. > >>> Probably room for improvement and code transparency. But... are > >>> Python-DEVs interested in the project?. That is the point :) > >> > >> As a concrete data point: - here are Dave's modifications to > >> ceval.c for systemtap: > >> http://bugs.python.org/review/14776/diff/5177/Python/ceval.c - here > >> are your modifications to ceval.c for dtrace: > >> http://bugs.python.org/review/13405/diff/6151/Python/ceval.c > > > > Unfair, because that code is not doing the same thing. > > > > Most of the extra complexity is there to deal with DTRACE ability to > > provide meaningful stackframes, with Python code instead of CPython > > evaluation loop. This is kind of magical. > > Antoine will correct me if I'm wrong, but I believe his point is less > about the complexity of dtrace provision and more about how much of it > lives in ceval.c: the systemtap provision also takes quite a bit of > code, but almost all of that code is extracted into a separate file and > only a pair of calls live in ceval.c Yes, that's exactly my point. There can be an arbitrarily complex ceval-dtrace.h for all I care :-) Thanks for the examples, by the way. Regards Antoine. From ncoghlan at gmail.com Sat Sep 7 13:01:36 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 7 Sep 2013 21:01:36 +1000 Subject: [Python-Dev] when to fix cross-version bugs? In-Reply-To: <522A0044.10302@stoneleaf.us> References: <5229EBDA.1030902@stoneleaf.us> <522A0044.10302@stoneleaf.us> Message-ID: <CADiSq7fy1to0quWH8v_2-RaC645PxJb4C32_u7yjPJkT7gMV=Q@mail.gmail.com> On 7 Sep 2013 02:43, "Ethan Furman" <ethan at stoneleaf.us> wrote: > > On 09/06/2013 07:51 AM, Ethan Furman wrote: >> >> >> What guidelines determine when a bug is fixed in previous versions? > > > On 09/06/2013 08:29 AM, Brian Curtin wrote: >> >> >> If it's a bug in that version and the version is accepting bug fixes, >> i.e., not in security mode, go for it. This includes crossing the 2/3 >> boundary if applicable. > > > On 09/06/2013 08:31 AM, R. David Murray wrote: >> >> >> The basic guideline is: we try very hard not to break currently working >> code in a maintenance release. Making that decision very much depends on >> the details of each individual case. >> >> [so] it probably doesn't have much positive impact if it does get >> >> backported, so is it worth the (small) chance of breaking someone's code? > > > And they say never go to the elves for advice! ;) Fixes that risk breaking current doctests are rarely worth backporting, so in this case, I'd say only fixing it in 3.4 is the better option. Definitely the kind of borderline case worth asking about, though :) Cheers, Nick. > > > -- > ~Ethan~ > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130907/d8ca9aff/attachment.html> From ncoghlan at gmail.com Sat Sep 7 16:14:01 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 8 Sep 2013 00:14:01 +1000 Subject: [Python-Dev] DTRACE support In-Reply-To: <20130907110613.71a9a98d@fsol> References: <5229EFE7.8040702@jcea.es> <20130906174111.446ea2c4@pitrou.net> <5229FF62.9020702@jcea.es> <20130906203334.1a99c99d@fsol> <522AA014.4060801@jcea.es> <557641DC-8E6B-450C-88A0-A3E2941678C7@masklinn.net> <20130907110613.71a9a98d@fsol> Message-ID: <CADiSq7epwhPEO+X16SuTOkH3wETVprbWx5h4f1g9miK9sNQhuQ@mail.gmail.com> On 7 September 2013 19:06, Antoine Pitrou <solipsis at pitrou.net> wrote: > On Sat, 7 Sep 2013 08:57:07 +0200 > Xavier Morel <python-dev at masklinn.net> wrote: >> >> On 2013-09-07, at 05:40 , Jesus Cea wrote: >> >> > -----BEGIN PGP SIGNED MESSAGE----- >> > Hash: SHA1 >> > >> > On 06/09/13 20:33, Antoine Pitrou wrote: >> >> On Fri, 06 Sep 2013 18:14:26 +0200 Jesus Cea <jcea at jcea.es> wrote: >> >>> >> >>> It is intrusive. Yes. I think it must be, by its own nature. >> >>> Probably room for improvement and code transparency. But... are >> >>> Python-DEVs interested in the project?. That is the point :) >> >> >> >> As a concrete data point: - here are Dave's modifications to >> >> ceval.c for systemtap: >> >> http://bugs.python.org/review/14776/diff/5177/Python/ceval.c - here >> >> are your modifications to ceval.c for dtrace: >> >> http://bugs.python.org/review/13405/diff/6151/Python/ceval.c >> > >> > Unfair, because that code is not doing the same thing. >> > >> > Most of the extra complexity is there to deal with DTRACE ability to >> > provide meaningful stackframes, with Python code instead of CPython >> > evaluation loop. This is kind of magical. >> >> Antoine will correct me if I'm wrong, but I believe his point is less >> about the complexity of dtrace provision and more about how much of it >> lives in ceval.c: the systemtap provision also takes quite a bit of >> code, but almost all of that code is extracted into a separate file and >> only a pair of calls live in ceval.c > > Yes, that's exactly my point. There can be an arbitrarily complex > ceval-dtrace.h for all I care :-) > > Thanks for the examples, by the way. Yep, complex stuff inline should be abstracted out so it doesn't affect the code flow for those of us that don't care :) Between Jesus, the Mac OS X using core devs and the Solaris and Mac OS X buildbots it should be a maintainable addition. However, it seems worthwhile to write up a short PEP (akin to the tracemalloc PEP) to scope out the suggestion. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ericsnowcurrently at gmail.com Sun Sep 8 00:22:36 2013 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Sat, 7 Sep 2013 16:22:36 -0600 Subject: [Python-Dev] Pre-PEP: Redesigning extension modules In-Reply-To: <CADiSq7c+R92r17c8g3ygW0T4p=VG8LiSf5XSGuOJsFHrOtOJDg@mail.gmail.com> References: <kv77o2$f96$1@ger.gmane.org> <CALeMXf5oxi0oX13zSfHwgVi1eXF_z_coY23MfBP-YOK8WVkWsg@mail.gmail.com> <kvctoj$qsv$1@ger.gmane.org> <kvstdj$jo0$1@ger.gmane.org> <CADiSq7c5xtEcPBXGPWqQosfgGcb2n+2AnHiAWFoATeYfQTUcEQ@mail.gmail.com> <20130902101642.04b69588@pitrou.net> <CADiSq7c+R92r17c8g3ygW0T4p=VG8LiSf5XSGuOJsFHrOtOJDg@mail.gmail.com> Message-ID: <CALFfu7C46gXvF37_=n2usyihCo1yUoRHS6QSx5OBChm9hb7dAQ@mail.gmail.com> On Mon, Sep 2, 2013 at 7:02 AM, Nick Coghlan <ncoghlan at gmail.com> wrote: > The hook API I currently have in mind is a two step initialisation: > > PyImport_PrepareNAME (optional) > PyImport_ExecNAME > Should we also look at an API change for the initfunc() of PyImport_Inittab entries? Currently the function takes a module name, which doesn't jive with loader.exec_module() taking a module. I noticed this while adding an exec_module() to BuiltinImporter. I suppose the same thing goes for PyImport_ImportFrozenModuleObject(). -eric -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130907/73f9cab0/attachment.html> From stefan_ml at behnel.de Sun Sep 8 09:09:05 2013 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 08 Sep 2013 09:09:05 +0200 Subject: [Python-Dev] Pre-PEP: Redesigning extension modules In-Reply-To: <CALFfu7C46gXvF37_=n2usyihCo1yUoRHS6QSx5OBChm9hb7dAQ@mail.gmail.com> References: <kv77o2$f96$1@ger.gmane.org> <CALeMXf5oxi0oX13zSfHwgVi1eXF_z_coY23MfBP-YOK8WVkWsg@mail.gmail.com> <kvctoj$qsv$1@ger.gmane.org> <kvstdj$jo0$1@ger.gmane.org> <CADiSq7c5xtEcPBXGPWqQosfgGcb2n+2AnHiAWFoATeYfQTUcEQ@mail.gmail.com> <20130902101642.04b69588@pitrou.net> <CADiSq7c+R92r17c8g3ygW0T4p=VG8LiSf5XSGuOJsFHrOtOJDg@mail.gmail.com> <CALFfu7C46gXvF37_=n2usyihCo1yUoRHS6QSx5OBChm9hb7dAQ@mail.gmail.com> Message-ID: <l0h7q9$nm2$1@ger.gmane.org> Eric Snow, 08.09.2013 00:22: > On Mon, Sep 2, 2013 at 7:02 AM, Nick Coghlan wrote: > >> The hook API I currently have in mind is a two step initialisation: >> >> PyImport_PrepareNAME (optional) >> PyImport_ExecNAME >> > > Should we also look at an API change for the initfunc() of PyImport_Inittab > entries? Currently the function takes a module name, which doesn't jive > with loader.exec_module() taking a module. I noticed this while adding an > exec_module() to BuiltinImporter. I suppose the same thing goes > for PyImport_ImportFrozenModuleObject(). Is it still the case that the inittab mechanism only works for the embedding case? It would be nice to have a declarative mechanism for registering a set of modules from a running module init function. Stefan From ethan at stoneleaf.us Sun Sep 8 09:42:20 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Sun, 08 Sep 2013 00:42:20 -0700 Subject: [Python-Dev] metaclasses, classes, instances, and proper nomenclature Message-ID: <522C2A5C.4030708@stoneleaf.us> I've run across two different ways to think about this: 1) the type of the first argument 2) where the method/attribute lives Since attributes don't take a first argument they default to 2: an instance attribute lives in the instance, a class attribute lives in the class, and a metaclass attribute lives in the metaclass. Methods, on the other hand, do take a first argument: an instance method takes itself, a class method takes the class, and a metaclass method takes the metaclass. Going by option 1 above there is only one way to get an instance method, and only one way to get a metaclass method -- calling with the instance (either directly or indirectly via the class), or calling a metaclass method that has been marked as a @classmethod. Therein lies my confusion. class Meta(type): @classmethod def meta_method(mcls): print("I'm a metaclass method!") def cls_method1(cls): print("I'm a class method! Aren't I?") class Class(metaclass=Meta): @classmethod def cls_method2(cls): print("I'm a class method for sure!") def instance_method(self): print("And I'm a regular ol' instance method") So, is Meta.cls_method1 a class method? On the one hand, it takes the class as it's first parameter, on the other hand it lives in the metaclass. And on the third hand you can't get to it from the instance Class(). If you're wondering why this is posted to PyDev, the related question is this: What is the proper role of a metaclass? Should it basically fiddle with the class creation process and then get out of the way? The case in point is, of course, Enum. Currently it has a custom __getattr__, but it lives in the metaclass, EnumMeta. Now this is handy, because it means that Color.red.blue raises an AttributeError, where if __getattr__ lived in Enum itself that would work. It also has the __members__ attribute living in EnumMeta, which means it's not accessible from Color.red. In other words, EnumMeta is not getting out the way, it is still very much involved. On the one hand, that's cool; on the other hand, I hand to think hard to figure out why Color.red.blue was not getting routed through EnumMeta's __getattr__, but was instead getting routed through object.__getattr__. -- ~Ethan~ From ncoghlan at gmail.com Sun Sep 8 12:06:17 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 8 Sep 2013 20:06:17 +1000 Subject: [Python-Dev] metaclasses, classes, instances, and proper nomenclature In-Reply-To: <522C2A5C.4030708@stoneleaf.us> References: <522C2A5C.4030708@stoneleaf.us> Message-ID: <CADiSq7eHZg_dgi3+n=v6452iQ+AwzN1kvr=P6XnBSDem+NdqBA@mail.gmail.com> On 8 Sep 2013 18:38, "Ethan Furman" <ethan at stoneleaf.us> wrote: > > I've run across two different ways to think about this: > > 1) the type of the first argument > > 2) where the method/attribute lives > > Since attributes don't take a first argument they default to 2: an instance attribute lives in the instance, a class attribute lives in the class, and a metaclass attribute lives in the metaclass. > > Methods, on the other hand, do take a first argument: an instance method takes itself, a class method takes the class, and a metaclass method takes the metaclass. No, there's no such thing as a "metaclass method". Metaclass instance methods are equivalent to hidden class methods - they don't appear in dir() and can't be accessed through instances of the class. It's only class methods on the metaclass that receive that rather than the class object (__new__ is technically a static method, but still accepts the metaclass as the first argument). > Going by option 1 above there is only one way to get an instance method, and only one way to get a metaclass method -- calling with the instance (either directly or indirectly via the class), or calling a metaclass method that has been marked as a @classmethod. > > Therein lies my confusion. > > class Meta(type): > > @classmethod > def meta_method(mcls): > print("I'm a metaclass method!") > > def cls_method1(cls): > print("I'm a class method! Aren't I?") > > class Class(metaclass=Meta): > > @classmethod > def cls_method2(cls): > print("I'm a class method for sure!") > > def instance_method(self): > print("And I'm a regular ol' instance method") > > > So, is Meta.cls_method1 a class method? On the one hand, it takes the class as it's first parameter, on the other hand it lives in the metaclass. And on the third hand you can't get to it from the instance Class(). It's a hidden class method. > If you're wondering why this is posted to PyDev, the related question is this: What is the proper role of a metaclass? Should it basically fiddle with the class creation process and then get out of the way? The case in point is, of course, Enum. Currently it has a custom __getattr__, but it lives in the metaclass, EnumMeta. Now this is handy, because it means that Color.red.blue raises an AttributeError, where if __getattr__ lived in Enum itself that would work. It also has the __members__ attribute living in EnumMeta, which means it's not accessible from Color.red. In other words, EnumMeta is not getting out the way, it is still very much involved. On the one hand, that's cool; on the other hand, I hand to think hard to figure out why Color.red.blue was not getting routed through EnumMeta's __getattr__, but was instead getting routed through object.__getattr__. This is exactly how a metaclass is intended to be used - to affect the behaviour of the class without affecting the behaviour of instances. And yes, introspection does get a little interesting when a non-trivial metaclass is in play :) Cheers, Nick. > > -- > ~Ethan~ > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130908/ee9f1ed2/attachment.html> From solipsis at pitrou.net Sun Sep 8 13:23:37 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 8 Sep 2013 13:23:37 +0200 Subject: [Python-Dev] cpython: Issue #18904: test_socket: add inheritance tests using fcntl and FD_CLOEXEC References: <3cXns46D3sz7LjR@mail.python.org> Message-ID: <20130908132337.69634518@fsol> On Sun, 8 Sep 2013 11:54:00 +0200 (CEST) victor.stinner <python-checkins at python.org> wrote: > http://hg.python.org/cpython/rev/b7f6f6f59e91 > changeset: 85619:b7f6f6f59e91 > user: Victor Stinner <victor.stinner at gmail.com> > date: Sun Sep 08 11:53:09 2013 +0200 > summary: > Issue #18904: test_socket: add inheritance tests using fcntl and FD_CLOEXEC > [...] > > + if fcntl: > + def test_get_inheritable_cloexec(self): The right way to do this would be to skip the test if fcntl doesn't exist. Regards Antoine. From steve at pearwood.info Sun Sep 8 14:37:06 2013 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 8 Sep 2013 22:37:06 +1000 Subject: [Python-Dev] PEP 450 adding statistics module In-Reply-To: <52215BDF.2090109@pearwood.info> References: <520C2E23.40405@pearwood.info> <52215BDF.2090109@pearwood.info> Message-ID: <20130908123706.GG26319@ando> <bump> I'd like to get some attention for this please. On Sat, Aug 31, 2013 at 12:58:39PM +1000, Steven D'Aprano wrote: > Hi all, > > > I think that PEP 450 is now ready for a PEP dictator. There have been a > number of code reviews, and feedback has been taken into account. The test > suite passes. I'm not aware of any unanswered issues with the code. At > least two people other than myself think that the implementation is ready > for a dictator, and nobody has objected. > > There is still on-going work on speeding up the implementation for the > statistics.sum function, but that will not effect the interface or the > substantially change the test suite. > > http://bugs.python.org/issue18606 > http://www.python.org/dev/peps/pep-0450/ > > > > > -- > Steven > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/steve%40pearwood.info > From victor.stinner at gmail.com Sun Sep 8 15:38:01 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Sun, 8 Sep 2013 15:38:01 +0200 Subject: [Python-Dev] cpython: Issue #18904: test_socket: add inheritance tests using fcntl and FD_CLOEXEC In-Reply-To: <20130908132337.69634518@fsol> References: <3cXns46D3sz7LjR@mail.python.org> <20130908132337.69634518@fsol> Message-ID: <CAMpsgwa4tdO+jOmH5b=gCDD7fZxgqD6NCWd5jCXtncp1NbJ6YA@mail.gmail.com> 2013/9/8 Antoine Pitrou <solipsis at pitrou.net>: > On Sun, 8 Sep 2013 11:54:00 +0200 (CEST) > victor.stinner <python-checkins at python.org> wrote: >> http://hg.python.org/cpython/rev/b7f6f6f59e91 >> changeset: 85619:b7f6f6f59e91 >> user: Victor Stinner <victor.stinner at gmail.com> >> date: Sun Sep 08 11:53:09 2013 +0200 >> summary: >> Issue #18904: test_socket: add inheritance tests using fcntl and FD_CLOEXEC >> > [...] >> >> + if fcntl: >> + def test_get_inheritable_cloexec(self): > > The right way to do this would be to skip the test if fcntl doesn't > exist. Ok. First, I added two methods to get and set the FD_CLOXEC flag. Later, I inlined these methods because it makes the code more readable. I modified the tests to use @unittest.skipIf decorator: New changeset aea58e1cae75 by Victor Stinner in branch 'default': Issue #18904: test_os and test_socket use unittest.skipIf() to check if fcntl http://hg.python.org/cpython/rev/aea58e1cae75 Victor From victor.stinner at gmail.com Sun Sep 8 16:03:43 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Sun, 8 Sep 2013 16:03:43 +0200 Subject: [Python-Dev] RFC: PEP 454: Add a new tracemalloc module In-Reply-To: <CAMpsgwZTOBdXy2Rcxs0itnPZYZgHU4X4tPNf5syoZrOJ5gYCFw@mail.gmail.com> References: <CAMpsgwZTOBdXy2Rcxs0itnPZYZgHU4X4tPNf5syoZrOJ5gYCFw@mail.gmail.com> Message-ID: <CAMpsgwZ3SoU-FuarRT_73yJ_VfFVoCBQ7phzcRQ+g_oFaYQCfA@mail.gmail.com> 2013/9/4 Victor Stinner <victor.stinner at gmail.com>: > http://www.python.org/dev/peps/pep-0454/ > > PEP: 454 > Title: Add a new tracemalloc module to trace Python memory allocations > Version: $Revision$ > Last-Modified: $Date$ > Author: Victor Stinner <victor.stinner at gmail.com> > Status: Draft > Type: Standards Track > Content-Type: text/x-rst > Created: 3-September-2013 > Python-Version: 3.4 I added a function get_tracemalloc_size() to see how much memory is used by the tracemalloc module itself. Result on the Python test suite: * 1 frame: +52% (+%68%) Python=34 MiB; _tracemalloc=18 MiB, tracemalloc.py=5 MiB * 10 frames: +155% (+170%) Python=34 MiB, _tracemalloc=53 MiB, tracemalloc.py=5 MiB * 100 frames: +1273% (+1283%) Python=30 MiB, _tracemalloc=382 MiB, tracemalloc.py=6 MiB On a small application and a computer with GB of memory, it may not matter. In a big application on an embedded device, it can be a blocker point to use tracemalloc. So I added filters (on the filename and line number) directly in the C module: ``add_filter(include: bool, filename: str, lineno: int=None)`` function: Add a filter. If *include* is ``True``, only trace memory blocks allocated in a file with a name matching *filename*. If *include* is ``False``, don't trace memory blocks allocated in a file with a name matching *filename*. The match is done using *filename* as a prefix. For example, ``'/usr/bin/'`` only matchs files the ``/usr/bin`` directories. The ``.pyc`` and ``.pyo`` suffixes are automatically replaced with ``.py`` when matching the filename. *lineno* is a line number. If *lineno* is ``None`` or lesser than ``1``, it matches any line number. ``clear_filters()`` function: Reset the filter list. ``get_filters()`` function: Get the filters as list of ``(include: bool, filename: str, lineno: int)`` tuples. If *lineno* is ``None``, a filter matchs any line number. By default, the filename of the Python tracemalloc module (``tracemalloc.py``) is excluded. Right now, the match is done using a PyUnicode_Tailmatch(). It is not convinient. I will see if it is possible to implement the joker character "*" matching any string, so the API would be closer to Snapshot.filter_filenames() (which uses fnmatch.fnmatch). Victor From rymg19 at gmail.com Sun Sep 8 15:52:32 2013 From: rymg19 at gmail.com (Ryan) Date: Sun, 08 Sep 2013 08:52:32 -0500 Subject: [Python-Dev] PEP 450 adding statistics module In-Reply-To: <20130908123706.GG26319@ando> References: <520C2E23.40405@pearwood.info> <52215BDF.2090109@pearwood.info> <20130908123706.GG26319@ando> Message-ID: <74b03474-ab18-40c5-8ce6-de5159d34903@email.android.com> ...what's a PEP dictator? Steven D'Aprano <steve at pearwood.info> wrote: ><bump> > >I'd like to get some attention for this please. > > >On Sat, Aug 31, 2013 at 12:58:39PM +1000, Steven D'Aprano wrote: >> Hi all, >> >> >> I think that PEP 450 is now ready for a PEP dictator. There have been >a >> number of code reviews, and feedback has been taken into account. The >test >> suite passes. I'm not aware of any unanswered issues with the code. >At >> least two people other than myself think that the implementation is >ready >> for a dictator, and nobody has objected. >> >> There is still on-going work on speeding up the implementation for >the >> statistics.sum function, but that will not effect the interface or >the >> substantially change the test suite. >> >> http://bugs.python.org/issue18606 >> http://www.python.org/dev/peps/pep-0450/ >> >> >> >> >> -- >> Steven >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> http://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> >http://mail.python.org/mailman/options/python-dev/steve%40pearwood.info >> >_______________________________________________ >Python-Dev mailing list >Python-Dev at python.org >https://mail.python.org/mailman/listinfo/python-dev >Unsubscribe: >https://mail.python.org/mailman/options/python-dev/rymg19%40gmail.com -- Sent from my Android phone with K-9 Mail. Please excuse my brevity. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130908/118e06a9/attachment.html> From solipsis at pitrou.net Sun Sep 8 17:20:11 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 8 Sep 2013 17:20:11 +0200 Subject: [Python-Dev] PEP 450 adding statistics module References: <520C2E23.40405@pearwood.info> <52215BDF.2090109@pearwood.info> Message-ID: <20130908172011.276c19af@fsol> On Sat, 31 Aug 2013 12:58:39 +1000 Steven D'Aprano <steve at pearwood.info> wrote: > Hi all, > > > I think that PEP 450 is now ready for a PEP dictator. Perhaps Mark would like to apply? Regards Antoine. From ethan at stoneleaf.us Sun Sep 8 18:06:29 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Sun, 08 Sep 2013 09:06:29 -0700 Subject: [Python-Dev] PEP 450 adding statistics module In-Reply-To: <74b03474-ab18-40c5-8ce6-de5159d34903@email.android.com> References: <520C2E23.40405@pearwood.info> <52215BDF.2090109@pearwood.info> <20130908123706.GG26319@ando> <74b03474-ab18-40c5-8ce6-de5159d34903@email.android.com> Message-ID: <522CA085.6000601@stoneleaf.us> On 09/08/2013 06:52 AM, Ryan wrote: > > ...what's a PEP dictator? The person tasked with deciding on the fate of an individual PEP. -- ~Ethan~ From guido at python.org Sun Sep 8 19:32:34 2013 From: guido at python.org (Guido van Rossum) Date: Sun, 8 Sep 2013 10:32:34 -0700 Subject: [Python-Dev] PEP 450 adding statistics module In-Reply-To: <522CA085.6000601@stoneleaf.us> References: <520C2E23.40405@pearwood.info> <52215BDF.2090109@pearwood.info> <20130908123706.GG26319@ando> <74b03474-ab18-40c5-8ce6-de5159d34903@email.android.com> <522CA085.6000601@stoneleaf.us> Message-ID: <CAP7+vJ+TTaU+ntaLT3xEU2wvFQhgP4P2-FnZUJ7V2PAAy3SMEA@mail.gmail.com> Going over the open issues: - Parallel arrays or arrays of tuples? I think the API should require an array of tuples. It is trivial to zip up parallel arrays to the required format, while if you have an array of tuples, extracting the parallel arrays is slightly more cumbersome. Also for manipulating of the raw data, an array of tuples makes it easier to do insertions or removals without worrying about losing the correspondence between the arrays. - Requiring concrete sequences as opposed to iterators sounds fine. I'm guessing that good algorithms for doing certain calculations in a single pass, assuming the full input doesn't fit in memory, are quite different from good algorithms for doing the same calculations without having that worry. (Just like you can't expect to use the same code to do a good job of sorting in-memory and on-disk data.) - Postponing some algorithms to Python 3.5 sounds fine. On Sun, Sep 8, 2013 at 9:06 AM, Ethan Furman <ethan at stoneleaf.us> wrote: > On 09/08/2013 06:52 AM, Ryan wrote: >> >> >> ...what's a PEP dictator? > > > The person tasked with deciding on the fate of an individual PEP. > > -- > ~Ethan~ > _______________________________________________ > > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org -- --Guido van Rossum (python.org/~guido) From guido at python.org Sun Sep 8 19:25:22 2013 From: guido at python.org (Guido van Rossum) Date: Sun, 8 Sep 2013 10:25:22 -0700 Subject: [Python-Dev] PEP 450 adding statistics module In-Reply-To: <20130908123706.GG26319@ando> References: <520C2E23.40405@pearwood.info> <52215BDF.2090109@pearwood.info> <20130908123706.GG26319@ando> Message-ID: <CAP7+vJKR=nK1OvHUzYoV-P6aP8pOrsAt+WA6aE+2xvKSLcaJ4A@mail.gmail.com> Steven, I'd like to just approve the PEP, given the amount of discussion that's happened already (though I didn't follow much of it). I quickly glanced through the PEP and didn't find anything I'd personally object to, but then I found your section of open issues, and I realized that you don't actually specify the proposed API in the PEP itself. It's highly unusual to approve a PEP that doesn't contain a specification. What did I miss? On Sun, Sep 8, 2013 at 5:37 AM, Steven D'Aprano <steve at pearwood.info> wrote: > <bump> > > I'd like to get some attention for this please. > > > On Sat, Aug 31, 2013 at 12:58:39PM +1000, Steven D'Aprano wrote: >> Hi all, >> >> >> I think that PEP 450 is now ready for a PEP dictator. There have been a >> number of code reviews, and feedback has been taken into account. The test >> suite passes. I'm not aware of any unanswered issues with the code. At >> least two people other than myself think that the implementation is ready >> for a dictator, and nobody has objected. >> >> There is still on-going work on speeding up the implementation for the >> statistics.sum function, but that will not effect the interface or the >> substantially change the test suite. >> >> http://bugs.python.org/issue18606 >> http://www.python.org/dev/peps/pep-0450/ >> >> >> >> >> -- >> Steven >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> http://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> http://mail.python.org/mailman/options/python-dev/steve%40pearwood.info >> > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%40python.org -- --Guido van Rossum (python.org/~guido) From alexander.belopolsky at gmail.com Sun Sep 8 19:45:32 2013 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Sun, 8 Sep 2013 21:45:32 +0400 Subject: [Python-Dev] PEP 450 adding statistics module In-Reply-To: <CAP7+vJ+TTaU+ntaLT3xEU2wvFQhgP4P2-FnZUJ7V2PAAy3SMEA@mail.gmail.com> References: <520C2E23.40405@pearwood.info> <52215BDF.2090109@pearwood.info> <20130908123706.GG26319@ando> <74b03474-ab18-40c5-8ce6-de5159d34903@email.android.com> <522CA085.6000601@stoneleaf.us> <CAP7+vJ+TTaU+ntaLT3xEU2wvFQhgP4P2-FnZUJ7V2PAAy3SMEA@mail.gmail.com> Message-ID: <CAP7h-xZ+_vmZXWU5yQXH1zKxygVdWGwdsioahQJR_gjzH3Cmpw@mail.gmail.com> On Sun, Sep 8, 2013 at 9:32 PM, Guido van Rossum <guido at python.org> wrote: > - Parallel arrays or arrays of tuples? I think the API should require > an array of tuples. It is trivial to zip up parallel arrays to the > required format, while if you have an array of tuples, extracting the > parallel arrays is slightly more cumbersome. > I agree with your conclusion but not with the rationale: converting between array of tuples and parallel arrays is trivial both ways: >>> at = [(1,2,3), (4,5,6), (7,8,9), (10, 11, 12)] >>> zip(*at) [(1, 4, 7, 10), (2, 5, 8, 11), (3, 6, 9, 12)] >>> zip(*_) [(1, 2, 3), (4, 5, 6), (7, 8, 9), (10, 11, 12)] (Note that zip(*x) is basically transpose(x).) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130908/8d85efc9/attachment.html> From guido at python.org Sun Sep 8 19:51:57 2013 From: guido at python.org (Guido van Rossum) Date: Sun, 8 Sep 2013 10:51:57 -0700 Subject: [Python-Dev] PEP 450 adding statistics module In-Reply-To: <CAP7+vJ+TTaU+ntaLT3xEU2wvFQhgP4P2-FnZUJ7V2PAAy3SMEA@mail.gmail.com> References: <520C2E23.40405@pearwood.info> <52215BDF.2090109@pearwood.info> <20130908123706.GG26319@ando> <74b03474-ab18-40c5-8ce6-de5159d34903@email.android.com> <522CA085.6000601@stoneleaf.us> <CAP7+vJ+TTaU+ntaLT3xEU2wvFQhgP4P2-FnZUJ7V2PAAy3SMEA@mail.gmail.com> Message-ID: <CAP7+vJJ6QxcUwcpdt3Mq6r07Q2dGUQt-VWGdsYGLv4wrRW5dGg@mail.gmail.com> Never mind, I found the patch and the issue. I really think that the *PEP* is ready for inclusion after the open issues are changed into something like Discussion or Future Work, and after adding a more prominent link to the issue with the patch. Then the *patch* can be reviewed some more until it is ready -- it looks very close already. On Sun, Sep 8, 2013 at 10:32 AM, Guido van Rossum <guido at python.org> wrote: > Going over the open issues: > > - Parallel arrays or arrays of tuples? I think the API should require > an array of tuples. It is trivial to zip up parallel arrays to the > required format, while if you have an array of tuples, extracting the > parallel arrays is slightly more cumbersome. Also for manipulating of > the raw data, an array of tuples makes it easier to do insertions or > removals without worrying about losing the correspondence between the > arrays. > > - Requiring concrete sequences as opposed to iterators sounds fine. > I'm guessing that good algorithms for doing certain calculations in a > single pass, assuming the full input doesn't fit in memory, are quite > different from good algorithms for doing the same calculations without > having that worry. (Just like you can't expect to use the same code to > do a good job of sorting in-memory and on-disk data.) > > - Postponing some algorithms to Python 3.5 sounds fine. > > On Sun, Sep 8, 2013 at 9:06 AM, Ethan Furman <ethan at stoneleaf.us> wrote: >> On 09/08/2013 06:52 AM, Ryan wrote: >>> >>> >>> ...what's a PEP dictator? >> >> >> The person tasked with deciding on the fate of an individual PEP. >> >> -- >> ~Ethan~ >> _______________________________________________ >> >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> https://mail.python.org/mailman/options/python-dev/guido%40python.org > > > > -- > --Guido van Rossum (python.org/~guido) -- --Guido van Rossum (python.org/~guido) From guido at python.org Sun Sep 8 19:52:28 2013 From: guido at python.org (Guido van Rossum) Date: Sun, 8 Sep 2013 10:52:28 -0700 Subject: [Python-Dev] PEP 450 adding statistics module In-Reply-To: <CAP7h-xZ+_vmZXWU5yQXH1zKxygVdWGwdsioahQJR_gjzH3Cmpw@mail.gmail.com> References: <520C2E23.40405@pearwood.info> <52215BDF.2090109@pearwood.info> <20130908123706.GG26319@ando> <74b03474-ab18-40c5-8ce6-de5159d34903@email.android.com> <522CA085.6000601@stoneleaf.us> <CAP7+vJ+TTaU+ntaLT3xEU2wvFQhgP4P2-FnZUJ7V2PAAy3SMEA@mail.gmail.com> <CAP7h-xZ+_vmZXWU5yQXH1zKxygVdWGwdsioahQJR_gjzH3Cmpw@mail.gmail.com> Message-ID: <CAP7+vJLCFgd_ndhqTrTccO-tO7SxN-bdVQNM4Fj-CysjVHA8iw@mail.gmail.com> Well, to me zip(*x) is unnatural, and it's inefficient when the arrays are long. On Sun, Sep 8, 2013 at 10:45 AM, Alexander Belopolsky <alexander.belopolsky at gmail.com> wrote: > > On Sun, Sep 8, 2013 at 9:32 PM, Guido van Rossum <guido at python.org> wrote: >> >> - Parallel arrays or arrays of tuples? I think the API should require >> an array of tuples. It is trivial to zip up parallel arrays to the >> required format, while if you have an array of tuples, extracting the >> parallel arrays is slightly more cumbersome. > > > I agree with your conclusion but not with the rationale: converting between > array of tuples and parallel arrays is trivial both ways: > >>>> at = [(1,2,3), (4,5,6), (7,8,9), (10, 11, 12)] >>>> zip(*at) > [(1, 4, 7, 10), (2, 5, 8, 11), (3, 6, 9, 12)] >>>> zip(*_) > [(1, 2, 3), (4, 5, 6), (7, 8, 9), (10, 11, 12)] > > (Note that zip(*x) is basically transpose(x).) -- --Guido van Rossum (python.org/~guido) From steve at pearwood.info Sun Sep 8 21:19:54 2013 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 9 Sep 2013 05:19:54 +1000 Subject: [Python-Dev] PEP 450 adding statistics module In-Reply-To: <CAP7+vJKR=nK1OvHUzYoV-P6aP8pOrsAt+WA6aE+2xvKSLcaJ4A@mail.gmail.com> References: <520C2E23.40405@pearwood.info> <52215BDF.2090109@pearwood.info> <20130908123706.GG26319@ando> <CAP7+vJKR=nK1OvHUzYoV-P6aP8pOrsAt+WA6aE+2xvKSLcaJ4A@mail.gmail.com> Message-ID: <20130908191954.GJ26319@ando> On Sun, Sep 08, 2013 at 10:25:22AM -0700, Guido van Rossum wrote: > Steven, I'd like to just approve the PEP, given the amount of > discussion that's happened already (though I didn't follow much of > it). I quickly glanced through the PEP and didn't find anything I'd > personally object to, but then I found your section of open issues, > and I realized that you don't actually specify the proposed API in the > PEP itself. It's highly unusual to approve a PEP that doesn't contain > a specification. What did I miss? You didn't miss anything, but I may have. Should the PEP go through each public function in the module (there are only 11)? That may be a little repetitive, since most have the same, or almost the same, signatures. Or is it acceptable to just include an overview? I've come up with this: API The initial version of the library will provide univariate (single variable) statistics functions. The general API will be based on a functional model ``function(data, ...) -> result``, where ``data`` is a mandatory iterable of (usually) numeric data. The author expects that lists will be the most common data type used, but any iterable type should be acceptable. Where necessary, functions may convert to lists internally. Where possible, functions are expected to conserve the type of the data values, for example, the mean of a list of Decimals should be a Decimal rather than float. Calculating the mean, median and mode The ``mean``, ``median`` and ``mode`` functions take a single mandatory argument and return the appropriate statistic, e.g.: >>> mean([1, 2, 3]) 2.0 ``mode`` is the sole exception to the rule that the data argument must be numeric. It will also accept an iterable of nominal data, such as strings. Calculating variance and standard deviation In order to be similar to scientific calculators, the statistics module will include separate functions for population and sample variance and standard deviation. All four functions have similar signatures, with a single mandatory argument, an iterable of numeric data, e.g.: >>> variance([1, 2, 2, 2, 3]) 0.5 All four functions also accept a second, optional, argument, the mean of the data. This is modelled on a similar API provided by the GNU Scientific Library[18]. There are three use-cases for using this argument, in no particular order: 1) The value of the mean is known *a priori*. 2) You have already calculated the mean, and wish to avoid calculating it again. 3) You wish to (ab)use the variance functions to calculate the second moment about some given point other than the mean. In each case, it is the caller's responsibility to ensure that given argument is meaningful. Is this satisfactory or do I need to go into more detail? -- Steven From p.f.moore at gmail.com Sun Sep 8 22:14:39 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 8 Sep 2013 21:14:39 +0100 Subject: [Python-Dev] PEP 450 adding statistics module In-Reply-To: <20130908191954.GJ26319@ando> References: <520C2E23.40405@pearwood.info> <52215BDF.2090109@pearwood.info> <20130908123706.GG26319@ando> <CAP7+vJKR=nK1OvHUzYoV-P6aP8pOrsAt+WA6aE+2xvKSLcaJ4A@mail.gmail.com> <20130908191954.GJ26319@ando> Message-ID: <CACac1F8mrqM9aEXkbtzDjbVzF49ofrwuZjaBLgVbKd9myQx2eQ@mail.gmail.com> On 8 September 2013 20:19, Steven D'Aprano <steve at pearwood.info> wrote: [...] > Is this satisfactory or do I need to go into more detail? It describes only 7 functions, and yet you state there are 11. I'd suggest you add a 1-line summary of each function, something like: mean - calculate the (arithmetic) mean of the data median - calculate the median value of the data etc. Paul From oscar.j.benjamin at gmail.com Sun Sep 8 22:48:26 2013 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Sun, 8 Sep 2013 21:48:26 +0100 Subject: [Python-Dev] PEP 450 adding statistics module In-Reply-To: <CAP7+vJ+TTaU+ntaLT3xEU2wvFQhgP4P2-FnZUJ7V2PAAy3SMEA@mail.gmail.com> References: <520C2E23.40405@pearwood.info> <52215BDF.2090109@pearwood.info> <20130908123706.GG26319@ando> <74b03474-ab18-40c5-8ce6-de5159d34903@email.android.com> <522CA085.6000601@stoneleaf.us> <CAP7+vJ+TTaU+ntaLT3xEU2wvFQhgP4P2-FnZUJ7V2PAAy3SMEA@mail.gmail.com> Message-ID: <CAHVvXxRAi_HHFOD2dY8UGsXpbgd-LyFU9rsg9x6Uk_=VuKG4hA@mail.gmail.com> On 8 September 2013 18:32, Guido van Rossum <guido at python.org> wrote: > Going over the open issues: > > - Parallel arrays or arrays of tuples? I think the API should require > an array of tuples. It is trivial to zip up parallel arrays to the > required format, while if you have an array of tuples, extracting the > parallel arrays is slightly more cumbersome. Also for manipulating of > the raw data, an array of tuples makes it easier to do insertions or > removals without worrying about losing the correspondence between the > arrays. For something like this, where there are multiple obvious formats for the input data, I think it's reasonable to just request whatever is convenient for the implementation. Otherwise you're asking at least some of your users to convert data from one format to another just so that you can convert it back again. In any real problem you'll likely have more than two variables, so you'll be writing some code to prepare the data for the function anyway. The most obvious alternative that isn't explicitly mentioned in the PEP is to accept either: def correlation(x, y=None): if y is None: xs = [] ys = [] for x, y in x: xs.append(x) ys.append(y) else: xs = list(x) ys = list(y) assert len(xs) == len(ys) # In reality a helper function does the above. # Now compute stuff This avoids any unnecessary conversions and is as convenient as possible for all users at the expense of having a slightly more complicated API. Oscar From janzert at janzert.com Sun Sep 8 22:52:05 2013 From: janzert at janzert.com (Janzert) Date: Sun, 08 Sep 2013 16:52:05 -0400 Subject: [Python-Dev] RFC: PEP 454: Add a new tracemalloc module In-Reply-To: <CAMpsgwZTOBdXy2Rcxs0itnPZYZgHU4X4tPNf5syoZrOJ5gYCFw@mail.gmail.com> References: <CAMpsgwZTOBdXy2Rcxs0itnPZYZgHU4X4tPNf5syoZrOJ5gYCFw@mail.gmail.com> Message-ID: <l0io1m$cqd$1@ger.gmane.org> It seems like most of this could live on PyPi for a while so the API can get hashed out in use? If that's not the case is it because the PEP 445 API isn't rich enough? Janzert From guido at python.org Sun Sep 8 23:41:35 2013 From: guido at python.org (Guido van Rossum) Date: Sun, 8 Sep 2013 14:41:35 -0700 Subject: [Python-Dev] PEP 450 adding statistics module In-Reply-To: <CAHVvXxRAi_HHFOD2dY8UGsXpbgd-LyFU9rsg9x6Uk_=VuKG4hA@mail.gmail.com> References: <520C2E23.40405@pearwood.info> <52215BDF.2090109@pearwood.info> <20130908123706.GG26319@ando> <74b03474-ab18-40c5-8ce6-de5159d34903@email.android.com> <522CA085.6000601@stoneleaf.us> <CAP7+vJ+TTaU+ntaLT3xEU2wvFQhgP4P2-FnZUJ7V2PAAy3SMEA@mail.gmail.com> <CAHVvXxRAi_HHFOD2dY8UGsXpbgd-LyFU9rsg9x6Uk_=VuKG4hA@mail.gmail.com> Message-ID: <CAP7+vJK0HRT4p0MEqraCr8EY1MN29E6CnkVcxvOks_s1rMka_w@mail.gmail.com> On Sun, Sep 8, 2013 at 1:48 PM, Oscar Benjamin <oscar.j.benjamin at gmail.com> wrote: > On 8 September 2013 18:32, Guido van Rossum <guido at python.org> wrote: >> Going over the open issues: >> >> - Parallel arrays or arrays of tuples? I think the API should require >> an array of tuples. It is trivial to zip up parallel arrays to the >> required format, while if you have an array of tuples, extracting the >> parallel arrays is slightly more cumbersome. Also for manipulating of >> the raw data, an array of tuples makes it easier to do insertions or >> removals without worrying about losing the correspondence between the >> arrays. > > For something like this, where there are multiple obvious formats for > the input data, I think it's reasonable to just request whatever is > convenient for the implementation. Not really. The implementation may change, or its needs may not be obvious to the caller. I would say the right thing to do is request something easy to remember, which often means consistent. In general, Python APIs definitely skew towards lists of tuples rather than parallel arrays, and for good reasons -- that way you benefit most from built-in operations like slices and insert/append. > Otherwise you're asking at least > some of your users to convert data from one format to another just so > that you can convert it back again. In any real problem you'll likely > have more than two variables, so you'll be writing some code to > prepare the data for the function anyway. Yeah, so you might as well prepare it in the form that the API expects. > The most obvious alternative that isn't explicitly mentioned in the > PEP is to accept either: > > def correlation(x, y=None): > if y is None: > xs = [] > ys = [] > for x, y in x: > xs.append(x) > ys.append(y) > else: > xs = list(x) > ys = list(y) > assert len(xs) == len(ys) > # In reality a helper function does the above. > # Now compute stuff > > This avoids any unnecessary conversions and is as convenient as > possible for all users at the expense of having a slightly more > complicated API. I don't think this is really more convenient -- it is more to learn, and can cause surprises (e.g. when a user is only familiar with one format and then sees an example using the other format, they may be unable to understand the example). The one argument I *haven't* heard yet which *might* sway me would be something along the line "every other statistics package that users might be familiar with does it this way" or "all the statistics textbooks do it this way". (Because, frankly, when it comes to statistics I'm a rank amateur and I really want Steven's new module to educate me as much as help me compute specific statistical functions.) -- --Guido van Rossum (python.org/~guido) From victor.stinner at gmail.com Mon Sep 9 01:05:11 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Mon, 9 Sep 2013 01:05:11 +0200 Subject: [Python-Dev] RFC: PEP 454: Add a new tracemalloc module In-Reply-To: <l0io1m$cqd$1@ger.gmane.org> References: <CAMpsgwZTOBdXy2Rcxs0itnPZYZgHU4X4tPNf5syoZrOJ5gYCFw@mail.gmail.com> <l0io1m$cqd$1@ger.gmane.org> Message-ID: <CAMpsgwYUZm00eaunv6cxe1rGMzYYe9JA-eBgwBW988y_7S-Lmg@mail.gmail.com> 2013/9/8 Janzert <janzert at janzert.com>: > It seems like most of this could live on PyPi for a while so the API can get > hashed out in use? The pytracemalloc is available on PyPI since 6 months. The only feedback I had was something trying to compile it on Windows (which is complex because of the dependency to glib, I don't think that it succeed to install it on Windows). I guess that I didn't get more feedback because it requires to patch and recompile Python, which is not trivial. I expect more feedback on python-dev with a working implementation (on hg.python.org) and a PEP. The version available on PyPI works and should be enough for most use cases to be able to identify a memory leak. Gregory P. Smith asked me if it would be possible to get more frames (filename and line number) of the Python traceback, instead of just the one frame (the last frame). I implemented it, but now I have new issues (memory usage of the tracemalloc module itself), so I'm working on filters directly implemented in the C module (_tracemalloc). It was already possible to filter traces from a snapshot read from the disk. I still have some tasks in my TODO list to finish the API and the implementation. When I will be done, I will post post a new version of the PEP on python-dev. > If that's not the case is it because the PEP 445 API > isn't rich enough? The PEP 445 API is only designed to allow to develop new tools like failmalloc or tracemalloc, without adding overhead if such debug tool is not used. The tracemalloc module reads the current Python traceback (filename and line number) which is "not directly" accessible from PyMem_Malloc(). I hope that existing tools like Heapy and Melia will benefit directly from tracemalloc instead of having to develop their own memory allocator hooks to get the same information (Python traceback). Victor From steve at pearwood.info Mon Sep 9 03:59:48 2013 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 9 Sep 2013 11:59:48 +1000 Subject: [Python-Dev] PEP 450 adding statistics module In-Reply-To: <CAP7+vJK0HRT4p0MEqraCr8EY1MN29E6CnkVcxvOks_s1rMka_w@mail.gmail.com> References: <520C2E23.40405@pearwood.info> <52215BDF.2090109@pearwood.info> <20130908123706.GG26319@ando> <74b03474-ab18-40c5-8ce6-de5159d34903@email.android.com> <522CA085.6000601@stoneleaf.us> <CAP7+vJ+TTaU+ntaLT3xEU2wvFQhgP4P2-FnZUJ7V2PAAy3SMEA@mail.gmail.com> <CAHVvXxRAi_HHFOD2dY8UGsXpbgd-LyFU9rsg9x6Uk_=VuKG4hA@mail.gmail.com> <CAP7+vJK0HRT4p0MEqraCr8EY1MN29E6CnkVcxvOks_s1rMka_w@mail.gmail.com> Message-ID: <20130909015948.GK26319@ando> On Sun, Sep 08, 2013 at 02:41:35PM -0700, Guido van Rossum wrote: > On Sun, Sep 8, 2013 at 1:48 PM, Oscar Benjamin > <oscar.j.benjamin at gmail.com> wrote: > > The most obvious alternative that isn't explicitly mentioned in the > > PEP is to accept either: > > > > def correlation(x, y=None): > > if y is None: > > xs = [] > > ys = [] > > for x, y in x: > > xs.append(x) > > ys.append(y) > > else: > > xs = list(x) > > ys = list(y) > > assert len(xs) == len(ys) > > # In reality a helper function does the above. > > # Now compute stuff > > > > This avoids any unnecessary conversions and is as convenient as > > possible for all users at the expense of having a slightly more > > complicated API. The PEP does mention that, as "some combination of the above". The PEP also mentions that the decision of what API to use for multivariate stats is deferred until 3.5, so there's plenty of time for people to bike-shed this :-) > I don't think this is really more convenient -- it is more to learn, > and can cause surprises (e.g. when a user is only familiar with one > format and then sees an example using the other format, they may be > unable to understand the example). > > The one argument I *haven't* heard yet which *might* sway me would be > something along the line "every other statistics package that users > might be familiar with does it this way" or "all the statistics > textbooks do it this way". (Because, frankly, when it comes to > statistics I'm a rank amateur and I really want Steven's new module to > educate me as much as help me compute specific statistical functions.) I don't think that there is one common API for multivariate stats packages. It partially depends on whether the package is aimed at basic use or advanced use. I haven't done a systematic comparison of the most common, but here are a few examples: - The Casio Classpad graphing calculator has a spreadsheet-like interface, which I consider equivalent to func(xdata, ydata). - The HP-48G series of calculators uses a fixed global variable holding a matrix, and a second global variable specifying which columns to use. - The R "cor" (correlation coefficient) function takes either a pair of vectors (lists), and calculates a single value, or a matrix, in which case it calculates the correlation matrix. - numpy.corrcoeff takes one or two array arguments, and a third argument specifying whether to treat rows or columns as variables, and like R returns either a single value or the correlation matrix. - Minitab expects two seperate vector arguments, and returns the correlation coefficient between them. - If I'm reading the below page correctly, the SAS corr procedure takes anything up to 27 arguments. http://support.sas.com/documentation/cdl/en/procstat/63104/HTML/default/procstat_corr_sect004.htm I don't suggest we follow that API :-) Quite frankly, I consider the majority of stats APIs to be confusing with a steep learning curve. -- Steven From stephen at xemacs.org Mon Sep 9 04:57:50 2013 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Mon, 09 Sep 2013 11:57:50 +0900 Subject: [Python-Dev] PEP 450 adding statistics module In-Reply-To: <CAP7+vJK0HRT4p0MEqraCr8EY1MN29E6CnkVcxvOks_s1rMka_w@mail.gmail.com> References: <520C2E23.40405@pearwood.info> <52215BDF.2090109@pearwood.info> <20130908123706.GG26319@ando> <74b03474-ab18-40c5-8ce6-de5159d34903@email.android.com> <522CA085.6000601@stoneleaf.us> <CAP7+vJ+TTaU+ntaLT3xEU2wvFQhgP4P2-FnZUJ7V2PAAy3SMEA@mail.gmail.com> <CAHVvXxRAi_HHFOD2dY8UGsXpbgd-LyFU9rsg9x6Uk_=VuKG4hA@mail.gmail.com> <CAP7+vJK0HRT4p0MEqraCr8EY1MN29E6CnkVcxvOks_s1rMka_w@mail.gmail.com> Message-ID: <87ppsi8zwx.fsf@uwakimon.sk.tsukuba.ac.jp> Guido van Rossum writes: > On Sun, Sep 8, 2013 at 1:48 PM, Oscar Benjamin > <oscar.j.benjamin at gmail.com> wrote: > > On 8 September 2013 18:32, Guido van Rossum <guido at python.org> wrote: > >> Going over the open issues: > >> > >> - Parallel arrays or arrays of tuples? I think the API should require > >> an array of tuples. It is trivial to zip up parallel arrays to the > >> required format, while if you have an array of tuples, extracting the > >> parallel arrays is slightly more cumbersome. > >> > >> Also for manipulating of the raw data, an array of tuples makes > >> it easier to do insertions or removals without worrying about > >> losing the correspondence between the arrays. I don't necessarily find this persuasive. It's more common when working with existing databases that you add variables than add observations. This is going to require attention to the correspondence in any case. Observations aren't added, and they're "removed" temporarily for statistics on subsets by slicing. If you use the same slice for all variables, you're not going to make a mistake. > Not really. The implementation may change, or its needs may not be > obvious to the caller. I would say the right thing to do is request > something easy to remember, which often means consistent. In general, > Python APIs definitely skew towards lists of tuples rather than > parallel arrays, and for good reasons -- that way you benefit most > from built-in operations like slices and insert/append. However, it's common in economic statistics to have a rectangular array, and extract both certain rows (tuples of observations on variables) and certain columns (variables). For example you might have data on populations of American states from 1900 to 2012, and extract the data on New England states from 1946 to 2012 for analysis. > The one argument I *haven't* heard yet which *might* sway me would be > something along the line "every other statistics package that users > might be familiar with does it this way" or "all the statistics > textbooks do it this way". (Because, frankly, when it comes to > statistics I'm a rank amateur and I really want Steven's new module to > educate me as much as help me compute specific statistical functions.) In economic statistics, most software traditionally inputs variables in column-major order (ie, parallel arrays). That said, most software nowadays allows input as spreadsheet tables. You pays your money and you takes your choice. I think the example above of state population data shows that rows and columns are pretty symmetric here. Many databases will have "too many" of both, and you'll want to "slice" both to get the sample and variables relevant to your analysis. This is all just for consideration; I am quite familiar with economic statistics and software, but not so much for that used in sociology, psychology, and medical applications. In the end, I think it's best to leave it up to Steven's judgment as to what is convenient for him to maintain. From greg.ewing at canterbury.ac.nz Mon Sep 9 02:26:05 2013 From: greg.ewing at canterbury.ac.nz (Greg) Date: Mon, 09 Sep 2013 12:26:05 +1200 Subject: [Python-Dev] PEP 450 adding statistics module In-Reply-To: <CAP7+vJLCFgd_ndhqTrTccO-tO7SxN-bdVQNM4Fj-CysjVHA8iw@mail.gmail.com> References: <520C2E23.40405@pearwood.info> <52215BDF.2090109@pearwood.info> <20130908123706.GG26319@ando> <74b03474-ab18-40c5-8ce6-de5159d34903@email.android.com> <522CA085.6000601@stoneleaf.us> <CAP7+vJ+TTaU+ntaLT3xEU2wvFQhgP4P2-FnZUJ7V2PAAy3SMEA@mail.gmail.com> <CAP7h-xZ+_vmZXWU5yQXH1zKxygVdWGwdsioahQJR_gjzH3Cmpw@mail.gmail.com> <CAP7+vJLCFgd_ndhqTrTccO-tO7SxN-bdVQNM4Fj-CysjVHA8iw@mail.gmail.com> Message-ID: <522D159D.90200@canterbury.ac.nz> On 9/09/2013 5:52 a.m., Guido van Rossum wrote: > Well, to me zip(*x) is unnatural, and it's inefficient when the arrays are long. Would it be worth having a transpose() function in the stdlib somewhere, that returns a view instead of copying the data? -- Greg From guido at python.org Mon Sep 9 05:16:19 2013 From: guido at python.org (Guido van Rossum) Date: Sun, 8 Sep 2013 20:16:19 -0700 Subject: [Python-Dev] PEP 450 adding statistics module In-Reply-To: <87ppsi8zwx.fsf@uwakimon.sk.tsukuba.ac.jp> References: <520C2E23.40405@pearwood.info> <52215BDF.2090109@pearwood.info> <20130908123706.GG26319@ando> <74b03474-ab18-40c5-8ce6-de5159d34903@email.android.com> <522CA085.6000601@stoneleaf.us> <CAP7+vJ+TTaU+ntaLT3xEU2wvFQhgP4P2-FnZUJ7V2PAAy3SMEA@mail.gmail.com> <CAHVvXxRAi_HHFOD2dY8UGsXpbgd-LyFU9rsg9x6Uk_=VuKG4hA@mail.gmail.com> <CAP7+vJK0HRT4p0MEqraCr8EY1MN29E6CnkVcxvOks_s1rMka_w@mail.gmail.com> <87ppsi8zwx.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <CAP7+vJL2NZPUm0mO_moVYdkvL4YAKkS-CWgTkg9goLyLtCSNxA@mail.gmail.com> Yeah, so this and Steven's review of various other APIs suggests that the field of statistics hasn't really reached the object-oriented age (or perhaps the OO view isn't suitable for the field), and people really think of their data as a matrix of some sort. We should respect that. Now, if this was NumPy, it would *still* make sense to require a single argument, to be interpreted in the usual fashion. So I'm using that as a kind of leverage to still recommend taking a list of pairs instead of a pair of lists. Also, it's quite likely that at least *some* of the users of the new statistics module will be more familiar with OO programming (e.g. the Python DB API , PEP 249) than they are with other statistics packages. On Sun, Sep 8, 2013 at 7:57 PM, Stephen J. Turnbull <stephen at xemacs.org>wrote: > Guido van Rossum writes: > > On Sun, Sep 8, 2013 at 1:48 PM, Oscar Benjamin > > <oscar.j.benjamin at gmail.com> wrote: > > > On 8 September 2013 18:32, Guido van Rossum <guido at python.org> wrote: > > >> Going over the open issues: > > >> > > >> - Parallel arrays or arrays of tuples? I think the API should require > > >> an array of tuples. It is trivial to zip up parallel arrays to the > > >> required format, while if you have an array of tuples, extracting the > > >> parallel arrays is slightly more cumbersome. > > >> > > >> Also for manipulating of the raw data, an array of tuples makes > > >> it easier to do insertions or removals without worrying about > > >> losing the correspondence between the arrays. > > I don't necessarily find this persuasive. It's more common when > working with existing databases that you add variables than add > observations. This is going to require attention to the > correspondence in any case. Observations aren't added, and they're > "removed" temporarily for statistics on subsets by slicing. If you > use the same slice for all variables, you're not going to make a > mistake. > > > Not really. The implementation may change, or its needs may not be > > obvious to the caller. I would say the right thing to do is request > > something easy to remember, which often means consistent. In general, > > Python APIs definitely skew towards lists of tuples rather than > > parallel arrays, and for good reasons -- that way you benefit most > > from built-in operations like slices and insert/append. > > However, it's common in economic statistics to have a rectangular > array, and extract both certain rows (tuples of observations on > variables) and certain columns (variables). For example you might > have data on populations of American states from 1900 to 2012, and > extract the data on New England states from 1946 to 2012 for analysis. > > > The one argument I *haven't* heard yet which *might* sway me would be > > something along the line "every other statistics package that users > > might be familiar with does it this way" or "all the statistics > > textbooks do it this way". (Because, frankly, when it comes to > > statistics I'm a rank amateur and I really want Steven's new module to > > educate me as much as help me compute specific statistical functions.) > > In economic statistics, most software traditionally inputs variables > in column-major order (ie, parallel arrays). That said, most software > nowadays allows input as spreadsheet tables. You pays your money and > you takes your choice. > > I think the example above of state population data shows that rows and > columns are pretty symmetric here. Many databases will have "too many" > of both, and you'll want to "slice" both to get the sample and > variables relevant to your analysis. > > This is all just for consideration; I am quite familiar with economic > statistics and software, but not so much for that used in sociology, > psychology, and medical applications. In the end, I think it's best > to leave it up to Steven's judgment as to what is convenient for him > to maintain. > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130908/054b9fc5/attachment.html> From guido at python.org Mon Sep 9 05:18:46 2013 From: guido at python.org (Guido van Rossum) Date: Sun, 8 Sep 2013 20:18:46 -0700 Subject: [Python-Dev] PEP 450 adding statistics module In-Reply-To: <522D159D.90200@canterbury.ac.nz> References: <520C2E23.40405@pearwood.info> <52215BDF.2090109@pearwood.info> <20130908123706.GG26319@ando> <74b03474-ab18-40c5-8ce6-de5159d34903@email.android.com> <522CA085.6000601@stoneleaf.us> <CAP7+vJ+TTaU+ntaLT3xEU2wvFQhgP4P2-FnZUJ7V2PAAy3SMEA@mail.gmail.com> <CAP7h-xZ+_vmZXWU5yQXH1zKxygVdWGwdsioahQJR_gjzH3Cmpw@mail.gmail.com> <CAP7+vJLCFgd_ndhqTrTccO-tO7SxN-bdVQNM4Fj-CysjVHA8iw@mail.gmail.com> <522D159D.90200@canterbury.ac.nz> Message-ID: <CAP7+vJ+L0g+C5fszy7eub5cLn0By+Qe+afe9-6-CfRpKpeBu+g@mail.gmail.com> On Sun, Sep 8, 2013 at 5:26 PM, Greg <greg.ewing at canterbury.ac.nz> wrote: > On 9/09/2013 5:52 a.m., Guido van Rossum wrote: > >> Well, to me zip(*x) is unnatural, and it's inefficient when the arrays >> are long. >> > > Would it be worth having a transpose() function in the stdlib > somewhere, that returns a view instead of copying the data? I'd be hesitant to add just that one function, given that there's hardly any support for multi-dimensional arrays in the stdlib. (NumPy of course has a transpose(), and that's where it arguably belongs.) -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130908/022d1b16/attachment.html> From steve at pearwood.info Mon Sep 9 05:27:22 2013 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 9 Sep 2013 13:27:22 +1000 Subject: [Python-Dev] PEP 450 adding statistics module In-Reply-To: <CACac1F8mrqM9aEXkbtzDjbVzF49ofrwuZjaBLgVbKd9myQx2eQ@mail.gmail.com> References: <520C2E23.40405@pearwood.info> <52215BDF.2090109@pearwood.info> <20130908123706.GG26319@ando> <CAP7+vJKR=nK1OvHUzYoV-P6aP8pOrsAt+WA6aE+2xvKSLcaJ4A@mail.gmail.com> <20130908191954.GJ26319@ando> <CACac1F8mrqM9aEXkbtzDjbVzF49ofrwuZjaBLgVbKd9myQx2eQ@mail.gmail.com> Message-ID: <20130909032722.GL26319@ando> On Sun, Sep 08, 2013 at 09:14:39PM +0100, Paul Moore wrote: > On 8 September 2013 20:19, Steven D'Aprano <steve at pearwood.info> wrote: > [...] > > Is this satisfactory or do I need to go into more detail? > > It describes only 7 functions, and yet you state there are 11. I'd > suggest you add a 1-line summary of each function, something like: > > mean - calculate the (arithmetic) mean of the data > median - calculate the median value of the data > etc. Thanks Paul, will do. I think PEP 1 needs to be a bit clearer about this part of the process. For instance, if I had a module with 100 functions and methods, would I need to document all of them in the PEP? I expect not, but then I didn't expect I needed to document all 11 either :-) -- Steven From steve at pearwood.info Mon Sep 9 05:43:59 2013 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 9 Sep 2013 13:43:59 +1000 Subject: [Python-Dev] PEP 450 adding statistics module In-Reply-To: <522D159D.90200@canterbury.ac.nz> References: <520C2E23.40405@pearwood.info> <52215BDF.2090109@pearwood.info> <20130908123706.GG26319@ando> <74b03474-ab18-40c5-8ce6-de5159d34903@email.android.com> <522CA085.6000601@stoneleaf.us> <CAP7+vJ+TTaU+ntaLT3xEU2wvFQhgP4P2-FnZUJ7V2PAAy3SMEA@mail.gmail.com> <CAP7h-xZ+_vmZXWU5yQXH1zKxygVdWGwdsioahQJR_gjzH3Cmpw@mail.gmail.com> <CAP7+vJLCFgd_ndhqTrTccO-tO7SxN-bdVQNM4Fj-CysjVHA8iw@mail.gmail.com> <522D159D.90200@canterbury.ac.nz> Message-ID: <20130909034359.GM26319@ando> On Mon, Sep 09, 2013 at 12:26:05PM +1200, Greg wrote: > On 9/09/2013 5:52 a.m., Guido van Rossum wrote: > >Well, to me zip(*x) is unnatural, and it's inefficient when the arrays are > >long. > > Would it be worth having a transpose() function in the stdlib > somewhere, that returns a view instead of copying the data? I've intentionally left out multivariate statistics from the initial version of statistics.py so there will be plenty of time to get feedback from users before deciding on an API before 3.5. If there was a transpose function in the std lib, the obvious place would be the statistics module itself. There is precedent: R includes a transpose function, and presumably the creators of R expect it to be used frequently because they've given it a single-letter name. http://stat.ethz.ch/R-manual/R-devel/library/base/html/t.html -- Steven From tjreedy at udel.edu Mon Sep 9 09:06:24 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 09 Sep 2013 03:06:24 -0400 Subject: [Python-Dev] PEP 450 adding statistics module In-Reply-To: <CAP7+vJK0HRT4p0MEqraCr8EY1MN29E6CnkVcxvOks_s1rMka_w@mail.gmail.com> References: <520C2E23.40405@pearwood.info> <52215BDF.2090109@pearwood.info> <20130908123706.GG26319@ando> <74b03474-ab18-40c5-8ce6-de5159d34903@email.android.com> <522CA085.6000601@stoneleaf.us> <CAP7+vJ+TTaU+ntaLT3xEU2wvFQhgP4P2-FnZUJ7V2PAAy3SMEA@mail.gmail.com> <CAHVvXxRAi_HHFOD2dY8UGsXpbgd-LyFU9rsg9x6Uk_=VuKG4hA@mail.gmail.com> <CAP7+vJK0HRT4p0MEqraCr8EY1MN29E6CnkVcxvOks_s1rMka_w@mail.gmail.com> Message-ID: <l0js19$nuf$1@ger.gmane.org> On 9/8/2013 5:41 PM, Guido van Rossum wrote: > On Sun, Sep 8, 2013 at 1:48 PM, Oscar Benjamin > <oscar.j.benjamin at gmail.com> wrote: >> On 8 September 2013 18:32, Guido van Rossum <guido at python.org> wrote: >>> Going over the open issues: >>> >>> - Parallel arrays or arrays of tuples? I think the API should require >>> an array of tuples. It is trivial to zip up parallel arrays to the >>> required format, while if you have an array of tuples, extracting the >>> parallel arrays is slightly more cumbersome. Also for manipulating of >>> the raw data, an array of tuples makes it easier to do insertions or >>> removals without worrying about losing the correspondence between the >>> arrays. >> >> For something like this, where there are multiple obvious formats for >> the input data, I think it's reasonable to just request whatever is >> convenient for the implementation. > > Not really. The implementation may change, or its needs may not be > obvious to the caller. I would say the right thing to do is request > something easy to remember, which often means consistent. In general, > Python APIs definitely skew towards lists of tuples rather than > parallel arrays, and for good reasons -- that way you benefit most > from built-in operations like slices and insert/append. This question has been discussed in the statistical software community for decades, going back to when storage was on magnetic tape, where contiguity was even more important than cache locality. In my experience with multiple packages, the most common format for input is tables where rows represent cases, samples, or whatever, which translates as lists of records (or tuples), just as with relational databases. Columns then represent a 'variable'. So I think we should go with that. Some packages might transpose the data internally, but that is an internal matter. The tradeoff is that storing by cases makes adding a new case easier, while storing by variables makes adding a new variable easier. -- Terry Jan Reedy From tjreedy at udel.edu Mon Sep 9 09:30:15 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 09 Sep 2013 03:30:15 -0400 Subject: [Python-Dev] PEP 450 adding statistics module In-Reply-To: <87ppsi8zwx.fsf@uwakimon.sk.tsukuba.ac.jp> References: <520C2E23.40405@pearwood.info> <52215BDF.2090109@pearwood.info> <20130908123706.GG26319@ando> <74b03474-ab18-40c5-8ce6-de5159d34903@email.android.com> <522CA085.6000601@stoneleaf.us> <CAP7+vJ+TTaU+ntaLT3xEU2wvFQhgP4P2-FnZUJ7V2PAAy3SMEA@mail.gmail.com> <CAHVvXxRAi_HHFOD2dY8UGsXpbgd-LyFU9rsg9x6Uk_=VuKG4hA@mail.gmail.com> <CAP7+vJK0HRT4p0MEqraCr8EY1MN29E6CnkVcxvOks_s1rMka_w@mail.gmail.com> <87ppsi8zwx.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <l0jte0$7bi$1@ger.gmane.org> On 9/8/2013 10:57 PM, Stephen J. Turnbull wrote: > I don't necessarily find this persuasive. It's more common when > working with existing databases that you add variables than add > observations. My experience with general scientific research is the opposite. One decides on the variables to measure and then adds rows (records) of data as you measure each experimental or observational subject. New calculated variables may be added (and often are) after the data collection is complete (at least for the moment). Time series analysis is a distinct and specialized subfield of statistics. The corresponding data collections is often different: one may start with a fixed set of subjects (50 US states for instance) and add 'variables' (population in year X) indefinitely. Much economic statistics is in this category. A third category is interaction analysis, where the data form a true matrix where both rows and columns represent subjects and entries represent interaction (how many times John emailed Joe, for instance). -- Terry Jan Reedy From paul at colomiets.name Mon Sep 9 09:51:31 2013 From: paul at colomiets.name (Paul Colomiets) Date: Mon, 9 Sep 2013 10:51:31 +0300 Subject: [Python-Dev] PEP 450 adding statistics module In-Reply-To: <CAP7+vJ+TTaU+ntaLT3xEU2wvFQhgP4P2-FnZUJ7V2PAAy3SMEA@mail.gmail.com> References: <520C2E23.40405@pearwood.info> <52215BDF.2090109@pearwood.info> <20130908123706.GG26319@ando> <74b03474-ab18-40c5-8ce6-de5159d34903@email.android.com> <522CA085.6000601@stoneleaf.us> <CAP7+vJ+TTaU+ntaLT3xEU2wvFQhgP4P2-FnZUJ7V2PAAy3SMEA@mail.gmail.com> Message-ID: <CAA0gF6rFbwwU2cK_gEcApaWdb9GOeSy6gWMytma+Dt_wkX7cDQ@mail.gmail.com> Hi Guido, On Sun, Sep 8, 2013 at 8:32 PM, Guido van Rossum <guido at python.org> wrote: > Going over the open issues: > > - Parallel arrays or arrays of tuples? I think the API should require > an array of tuples. It is trivial to zip up parallel arrays to the > required format, while if you have an array of tuples, extracting the > parallel arrays is slightly more cumbersome. Also for manipulating of > the raw data, an array of tuples makes it easier to do insertions or > removals without worrying about losing the correspondence between the > arrays. I think there is a big reason to use parallel arrays that might be overlooked. You can feed an array.array('f') to the function, which may save a lot of memory. -- Paul From storchaka at gmail.com Mon Sep 9 10:02:05 2013 From: storchaka at gmail.com (Serhiy Storchaka) Date: Mon, 09 Sep 2013 11:02:05 +0300 Subject: [Python-Dev] PEP 450 adding statistics module In-Reply-To: <CAP7+vJLCFgd_ndhqTrTccO-tO7SxN-bdVQNM4Fj-CysjVHA8iw@mail.gmail.com> References: <520C2E23.40405@pearwood.info> <52215BDF.2090109@pearwood.info> <20130908123706.GG26319@ando> <74b03474-ab18-40c5-8ce6-de5159d34903@email.android.com> <522CA085.6000601@stoneleaf.us> <CAP7+vJ+TTaU+ntaLT3xEU2wvFQhgP4P2-FnZUJ7V2PAAy3SMEA@mail.gmail.com> <CAP7h-xZ+_vmZXWU5yQXH1zKxygVdWGwdsioahQJR_gjzH3Cmpw@mail.gmail.com> <CAP7+vJLCFgd_ndhqTrTccO-tO7SxN-bdVQNM4Fj-CysjVHA8iw@mail.gmail.com> Message-ID: <l0jv92$oqm$1@ger.gmane.org> 08.09.13 20:52, Guido van Rossum ???????(??): > Well, to me zip(*x) is unnatural, and it's inefficient when the arrays are long. Perhaps we need zip.from_iterable()? From oscar.j.benjamin at gmail.com Mon Sep 9 10:36:23 2013 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Mon, 9 Sep 2013 09:36:23 +0100 Subject: [Python-Dev] PEP 450 adding statistics module In-Reply-To: <l0jv92$oqm$1@ger.gmane.org> References: <520C2E23.40405@pearwood.info> <52215BDF.2090109@pearwood.info> <20130908123706.GG26319@ando> <74b03474-ab18-40c5-8ce6-de5159d34903@email.android.com> <522CA085.6000601@stoneleaf.us> <CAP7+vJ+TTaU+ntaLT3xEU2wvFQhgP4P2-FnZUJ7V2PAAy3SMEA@mail.gmail.com> <CAP7h-xZ+_vmZXWU5yQXH1zKxygVdWGwdsioahQJR_gjzH3Cmpw@mail.gmail.com> <CAP7+vJLCFgd_ndhqTrTccO-tO7SxN-bdVQNM4Fj-CysjVHA8iw@mail.gmail.com> <l0jv92$oqm$1@ger.gmane.org> Message-ID: <CAHVvXxSnF1fAP6dNxMvdB9_NqPMMhXQu+70tdDJ6BEieMZOU=g@mail.gmail.com> On 9 September 2013 09:02, Serhiy Storchaka <storchaka at gmail.com> wrote: > 08.09.13 20:52, Guido van Rossum ???????(??): > >> Well, to me zip(*x) is unnatural, and it's inefficient when the arrays are >> long. > > Perhaps we need zip.from_iterable()? I would prefer it if chain.from_iterable were named something like flatten (as it is in the itertools recipes). Similarly transpose is a better name than zip.from_iterable. Oscar From oscar.j.benjamin at gmail.com Mon Sep 9 11:00:24 2013 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Mon, 9 Sep 2013 10:00:24 +0100 Subject: [Python-Dev] PEP 450 adding statistics module In-Reply-To: <CAP7+vJL2NZPUm0mO_moVYdkvL4YAKkS-CWgTkg9goLyLtCSNxA@mail.gmail.com> References: <520C2E23.40405@pearwood.info> <52215BDF.2090109@pearwood.info> <20130908123706.GG26319@ando> <74b03474-ab18-40c5-8ce6-de5159d34903@email.android.com> <522CA085.6000601@stoneleaf.us> <CAP7+vJ+TTaU+ntaLT3xEU2wvFQhgP4P2-FnZUJ7V2PAAy3SMEA@mail.gmail.com> <CAHVvXxRAi_HHFOD2dY8UGsXpbgd-LyFU9rsg9x6Uk_=VuKG4hA@mail.gmail.com> <CAP7+vJK0HRT4p0MEqraCr8EY1MN29E6CnkVcxvOks_s1rMka_w@mail.gmail.com> <87ppsi8zwx.fsf@uwakimon.sk.tsukuba.ac.jp> <CAP7+vJL2NZPUm0mO_moVYdkvL4YAKkS-CWgTkg9goLyLtCSNxA@mail.gmail.com> Message-ID: <CAHVvXxTyn2hcUL0KPrwcVwD1mfqmN4ypj=UiUziRHhALkaQPJA@mail.gmail.com> On 9 September 2013 04:16, Guido van Rossum <guido at python.org> wrote: > > Yeah, so this and Steven's review of various other APIs suggests that the > field of statistics hasn't really reached the object-oriented age (or > perhaps the OO view isn't suitable for the field), and people really think > of their data as a matrix of some sort. We should respect that. Now, if this > was NumPy, it would *still* make sense to require a single argument, to be > interpreted in the usual fashion. So I'm using that as a kind of leverage to > still recommend taking a list of pairs instead of a pair of lists. Also, > it's quite likely that at least *some* of the users of the new statistics > module will be more familiar with OO programming (e.g. the Python DB API , > PEP 249) than they are with other statistics packages. I'm not sure if I understand what you mean by this. Numpy has built everything on top of a core ndarray class whose methods make the issues about multivariate stats APIs trivial. The transpose of an array A is simply the attribute A.T which is both convenient and cheap since it's just an alternate view on the underlying buffer. Also numpy provides record arrays that enable you to use names instead of numeric indices: >>> import numpy as np >>> dt = np.dtype([('Year', int), ('Arizona', float), ('Dakota', float)]) >>> a = np.array([(2001, 123., 456.), (2002, 234., 345), (2003, 345., 567)], dt) >>> a array([(2001, 123.0, 456.0), (2002, 234.0, 345.0), (2003, 345.0, 567.0)], dtype=[('Year', '<i4'), ('Arizona', '<f8'), ('Dakota', '<f8')]) >>> a['Year'] array([2001, 2002, 2003]) >>> a['Arizona'] array([ 123., 234., 345.]) >>> np.corrcoef(a['Arizona'], a['Dakota']) array([[ 1. , 0.5], [ 0.5, 1. ]]) >>> included = a[a['Year'] > 2001] >>> included array([(2002, 234.0, 345.0), (2003, 345.0, 567.0)], dtype=[('Year', '<i4'), ('Arizona', '<f8'), ('Dakota', '<f8')]) >>> np.corrcoef(included['Arizona'], included['Dakota']) array([[ 1., 1.], [ 1., 1.]]) So perhaps the statistics module could have a similar NameTupleArray type that can be easily loaded and saved from a csv file and makes it easy to put your data in whatever form is required. Oscar From stuart at stuartbishop.net Mon Sep 9 12:38:50 2013 From: stuart at stuartbishop.net (Stuart Bishop) Date: Mon, 9 Sep 2013 17:38:50 +0700 Subject: [Python-Dev] Offtopic: OpenID Providers In-Reply-To: <52293667.8030609@jcea.es> References: <5228C00F.7000209@jcea.es> <CAF-Rda_aajDr71ZU=xJ8m1diBLuG_jr__TO6tR8kAMS625q7hg@mail.gmail.com> <52293667.8030609@jcea.es> Message-ID: <CADmi=6PFXSB2226W4U7vbrjepwjjbCBuj8ypmpgE0JARV4OMYA@mail.gmail.com> On Fri, Sep 6, 2013 at 8:56 AM, Jesus Cea <jcea at jcea.es> wrote: > Sorry, Google, Facebook, Twitter, etc., are not acceptable OpenID > providers for me. I should have made that point in my original email. > My excuses. > > Any other suggestion? As Barry mentioned earlier, launchpad.net. Look for the 'lp' icon on pypi, bugs.python.org etc. -- Stuart Bishop <stuart at stuartbishop.net> http://www.stuartbishop.net/ From skip at pobox.com Mon Sep 9 12:44:43 2013 From: skip at pobox.com (Skip Montanaro) Date: Mon, 9 Sep 2013 05:44:43 -0500 Subject: [Python-Dev] PEP 450 adding statistics module In-Reply-To: <87ppsi8zwx.fsf@uwakimon.sk.tsukuba.ac.jp> References: <520C2E23.40405@pearwood.info> <52215BDF.2090109@pearwood.info> <20130908123706.GG26319@ando> <74b03474-ab18-40c5-8ce6-de5159d34903@email.android.com> <522CA085.6000601@stoneleaf.us> <CAP7+vJ+TTaU+ntaLT3xEU2wvFQhgP4P2-FnZUJ7V2PAAy3SMEA@mail.gmail.com> <CAHVvXxRAi_HHFOD2dY8UGsXpbgd-LyFU9rsg9x6Uk_=VuKG4hA@mail.gmail.com> <CAP7+vJK0HRT4p0MEqraCr8EY1MN29E6CnkVcxvOks_s1rMka_w@mail.gmail.com> <87ppsi8zwx.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <CANc-5UxZ5M7MXxCPYY6Tfn0rpjRUT+yjyOWnGgFO8ROSyWfrHQ@mail.gmail.com> > However, it's common in economic statistics to have a rectangular > array, and extract both certain rows (tuples of observations on > variables) and certain columns (variables). For example you might > have data on populations of American states from 1900 to 2012, and > extract the data on New England states from 1946 to 2012 for analysis. When Steven first brought up this PEP on comp.lang.python, my main concern was basically, "we have SciPy, why do we need this?" Steven's response, which I have come to accept, is that there are uses for basic statistics for which SciPy's stats module would be overkill. However, once you start slicing your data structure along more than one axis, I think you very quickly will find that you need numpy arrays for performance reasons, at which point you might as go "all the way" and install SciPy. I don't think slicing along multiple dimensions should be a significant concern for this package. Alternatively, I thought there was discussion a long time ago about getting numpy's (or even further back, numeric's?) array type into the core. Python has an array type which I don't think gets a lot of use (or love). Might it be worthwhile to make sure the PEP 450 package works with that? Then extend it to multiple dimensions? Or just bite the bullet and get numpy's array type into the Python core once and for all? Sort of Tulip for arrays... Skip From martin at v.loewis.de Mon Sep 9 13:43:08 2013 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 09 Sep 2013 13:43:08 +0200 Subject: [Python-Dev] unicodedata module is out of date In-Reply-To: <CAPMPPoRMBsBbAasu8ewEwqJOJARLAsctJricjKdVYVkXdLy8Tg@mail.gmail.com> References: <CAPMPPoRq1Uuag3BP0OnfsJPhKVsZapN3JS03i+dN2enAj=saYg@mail.gmail.com> <5229F6DB.40000@mrabarnett.plus.com> <CAPMPPoRMBsBbAasu8ewEwqJOJARLAsctJricjKdVYVkXdLy8Tg@mail.gmail.com> Message-ID: <522DB44C.8050802@v.loewis.de> Am 06.09.13 17:55, schrieb Andrew Miller: > Are there plans to add the extra data from the other UCD files to this > module? At the moment I am using a module from > https://gist.github.com/anonymous/2204527 to obtain the script of a > character but it would be nice if this was available from the standard > library. Well, it is available, and new versions of the UCD are added to new Python releases. Please consider Python 2 as dead wrt. Unicode support. Regards, Martin From martin at v.loewis.de Mon Sep 9 13:41:21 2013 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 09 Sep 2013 13:41:21 +0200 Subject: [Python-Dev] unicodedata module is out of date In-Reply-To: <l0d6lj$lnf$1@ger.gmane.org> References: <CAPMPPoRq1Uuag3BP0OnfsJPhKVsZapN3JS03i+dN2enAj=saYg@mail.gmail.com> <5229F6DB.40000@mrabarnett.plus.com> <CAPMPPoRMBsBbAasu8ewEwqJOJARLAsctJricjKdVYVkXdLy8Tg@mail.gmail.com> <l0d6lj$lnf$1@ger.gmane.org> Message-ID: <522DB3E1.30105@v.loewis.de> Am 06.09.13 20:24, schrieb Terry Reedy: >> In Python 2.7.5 it is set to '5.2.0' so it looks as though this version >> is no longer being updated. > > In general, new features do not go into bugfix releases (x.y.z, where z >>= 1). Updating the unidate_version add new features to the unicodedata > module. > One might argue that an update of the UCD data is within the scope of 2.7, since it's just data, not code that is being changed. I'd argue against that, since this specific change has a chance of breaking existing tests that people might have. Of course, it is up the the release manager of 2.7 to decide on that if such a change would be proposed. Regards, Martin From martin at v.loewis.de Mon Sep 9 13:38:09 2013 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 09 Sep 2013 13:38:09 +0200 Subject: [Python-Dev] DTRACE support In-Reply-To: <CAP7+vJJsPVn_EHjfYeYZhK=nd3VfBZ3h_b6tTsShTt6gUSVNaw@mail.gmail.com> References: <5229EFE7.8040702@jcea.es> <CAP7+vJJsPVn_EHjfYeYZhK=nd3VfBZ3h_b6tTsShTt6gUSVNaw@mail.gmail.com> Message-ID: <522DB321.1080905@v.loewis.de> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Am 06.09.13 17:14, schrieb Guido van Rossum: > I've heard good things about DTRACE but never used it myself. > > Do I understand correctly that you have to build a separate Python > executable with it turned on? My understanding is that you would normally have dtrace support built on systems that support it. The claim of the dtrace authors is that there is virtually no overhead in having trace points integrated that are not active; it achieves that by patching trace instructions over the NOPs that have been put there during compile time. Regards, Martin -----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.18 (Darwin) Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlItsyEACgkQavBT8H2dyNLhNgCfXfgXakq9wgvNGADrYNMvvAJu 05YAn0SxHrOCgoW8k9wjW+ajj+0jtpg9 =E/GT -----END PGP SIGNATURE----- From larry at hastings.org Mon Sep 9 14:02:27 2013 From: larry at hastings.org (Larry Hastings) Date: Mon, 09 Sep 2013 21:02:27 +0900 Subject: [Python-Dev] [RELEASED] Python 3.4.0a2 Message-ID: <522DB8D3.1030803@hastings.org> On behalf of the Python development team, I'm chuffed to announce the second alpha release of Python 3.4. This is a preview release, and its use is not recommended for production settings. Python 3.4 includes a range of improvements of the 3.x series, including hundreds of small improvements and bug fixes. Major new features and changes in the 3.4 release series so far include: * PEP 435, a standardized "enum" module * PEP 442, improved semantics for object finalization * PEP 443, adding single-dispatch generic functions to the standard library * PEP 445, a new C API for implementing custom memory allocators * PEP 446, changing file descriptors to not be inherited by default in subprocesses * PEP 447, a new magic method for metaclasses (``__typelookup__``) * PEP 448, making automatic sequence unpacking more general To download Python 3.4.0a2 visit: http://www.python.org/download/releases/3.4.0/ Please consider trying Python 3.4.0a2 with your code and reporting any issues you notice to: http://bugs.python.org/ Enjoy! -- Larry Hastings, Release Manager larry at hastings.org (on behalf of the entire python-dev team and 3.4's contributors) From steve at pearwood.info Mon Sep 9 14:07:58 2013 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 9 Sep 2013 22:07:58 +1000 Subject: [Python-Dev] PEP 450 adding statistics module In-Reply-To: <CANc-5UxZ5M7MXxCPYY6Tfn0rpjRUT+yjyOWnGgFO8ROSyWfrHQ@mail.gmail.com> References: <520C2E23.40405@pearwood.info> <52215BDF.2090109@pearwood.info> <20130908123706.GG26319@ando> <74b03474-ab18-40c5-8ce6-de5159d34903@email.android.com> <522CA085.6000601@stoneleaf.us> <CAP7+vJ+TTaU+ntaLT3xEU2wvFQhgP4P2-FnZUJ7V2PAAy3SMEA@mail.gmail.com> <CAHVvXxRAi_HHFOD2dY8UGsXpbgd-LyFU9rsg9x6Uk_=VuKG4hA@mail.gmail.com> <CAP7+vJK0HRT4p0MEqraCr8EY1MN29E6CnkVcxvOks_s1rMka_w@mail.gmail.com> <87ppsi8zwx.fsf@uwakimon.sk.tsukuba.ac.jp> <CANc-5UxZ5M7MXxCPYY6Tfn0rpjRUT+yjyOWnGgFO8ROSyWfrHQ@mail.gmail.com> Message-ID: <20130909120758.GO26319@ando> On Mon, Sep 09, 2013 at 05:44:43AM -0500, Skip Montanaro wrote: > > However, it's common in economic statistics to have a rectangular > > array, and extract both certain rows (tuples of observations on > > variables) and certain columns (variables). For example you might > > have data on populations of American states from 1900 to 2012, and > > extract the data on New England states from 1946 to 2012 for analysis. > > When Steven first brought up this PEP on comp.lang.python, my main concern > was basically, "we have SciPy, why do we need this?" Steven's response, which > I have come to accept, is that there are uses for basic statistics for > which SciPy's > stats module would be overkill. > > However, once you start slicing your data structure along more than one axis, I > think you very quickly will find that you need numpy arrays for performance > reasons, at which point you might as go "all the way" and install SciPy. I don't > think slicing along multiple dimensions should be a significant concern for this > package. I agree. I'm not interested in trying to compete with numpy in areas where numpy is best. That's a fight any pure-Python module is going to lose :-) > Alternatively, I thought there was discussion a long time ago about > getting numpy's > (or even further back, numeric's?) array type into the core. Python > has an array type > which I don't think gets a lot of use (or love). Might it be > worthwhile to make sure the > PEP 450 package works with that? Then extend it to multiple dimensions? Or just > bite the bullet and get numpy's array type into the Python core once > and for all? I haven't tested PEP 450 statistics with numpy array, but any sequence type ought to work. While I haven't done extensive testing on the array.array type, basic testing shows that it works as expected: py> import array py> import statistics py> data = array.array('f', range(1, 101)) py> statistics.mean(data) 50.5 py> statistics.variance(data) 841.6666666666666 -- Steven From brett at python.org Mon Sep 9 14:16:06 2013 From: brett at python.org (Brett Cannon) Date: Mon, 9 Sep 2013 08:16:06 -0400 Subject: [Python-Dev] [python-committers] [RELEASED] Python 3.4.0a2 In-Reply-To: <522DB8D3.1030803@hastings.org> References: <522DB8D3.1030803@hastings.org> Message-ID: <CAP1=2W4HNkPodjskLXtT=RvaC0DXhtmQbggRJYgQX_4VG-axXw@mail.gmail.com> On Mon, Sep 9, 2013 at 8:02 AM, Larry Hastings <larry at hastings.org> wrote: > > On behalf of the Python development team, I'm chuffed to announce the > second alpha release of Python 3.4. > > This is a preview release, and its use is not recommended for > production settings. > > Python 3.4 includes a range of improvements of the 3.x series, including > hundreds of small improvements and bug fixes. Major new features and > changes in the 3.4 release series so far include: > > * PEP 435, a standardized "enum" module > * PEP 442, improved semantics for object finalization > * PEP 443, adding single-dispatch generic functions to the standard library > * PEP 445, a new C API for implementing custom memory allocators > * PEP 446, changing file descriptors to not be inherited by default > in subprocesses > * PEP 447, a new magic method for metaclasses (``__typelookup__``) > * PEP 448, making automatic sequence unpacking more general > Those last two PEPs are still in draft form and not accepted nor have any committed code yet. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130909/4ccde394/attachment.html> From ncoghlan at gmail.com Mon Sep 9 13:56:05 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 9 Sep 2013 21:56:05 +1000 Subject: [Python-Dev] PEP 450 adding statistics module In-Reply-To: <CANc-5UxZ5M7MXxCPYY6Tfn0rpjRUT+yjyOWnGgFO8ROSyWfrHQ@mail.gmail.com> References: <520C2E23.40405@pearwood.info> <52215BDF.2090109@pearwood.info> <20130908123706.GG26319@ando> <74b03474-ab18-40c5-8ce6-de5159d34903@email.android.com> <522CA085.6000601@stoneleaf.us> <CAP7+vJ+TTaU+ntaLT3xEU2wvFQhgP4P2-FnZUJ7V2PAAy3SMEA@mail.gmail.com> <CAHVvXxRAi_HHFOD2dY8UGsXpbgd-LyFU9rsg9x6Uk_=VuKG4hA@mail.gmail.com> <CAP7+vJK0HRT4p0MEqraCr8EY1MN29E6CnkVcxvOks_s1rMka_w@mail.gmail.com> <87ppsi8zwx.fsf@uwakimon.sk.tsukuba.ac.jp> <CANc-5UxZ5M7MXxCPYY6Tfn0rpjRUT+yjyOWnGgFO8ROSyWfrHQ@mail.gmail.com> Message-ID: <CADiSq7fXsYCy125GyX9HenHkzXMs_2HZu4Smrc+q5_J-tWyv1A@mail.gmail.com> On 9 Sep 2013 20:46, "Skip Montanaro" <skip at pobox.com> wrote: > > > However, it's common in economic statistics to have a rectangular > > array, and extract both certain rows (tuples of observations on > > variables) and certain columns (variables). For example you might > > have data on populations of American states from 1900 to 2012, and > > extract the data on New England states from 1946 to 2012 for analysis. > > When Steven first brought up this PEP on comp.lang.python, my main concern > was basically, "we have SciPy, why do we need this?" Steven's response, which > I have come to accept, is that there are uses for basic statistics for > which SciPy's > stats module would be overkill. > > However, once you start slicing your data structure along more than one axis, I > think you very quickly will find that you need numpy arrays for performance > reasons, at which point you might as go "all the way" and install SciPy. I don't > think slicing along multiple dimensions should be a significant concern for this > package. > > Alternatively, I thought there was discussion a long time ago about > getting numpy's > (or even further back, numeric's?) array type into the core. Python > has an array type > which I don't think gets a lot of use (or love). Might it be > worthwhile to make sure the > PEP 450 package works with that? Then extend it to multiple dimensions? Or just > bite the bullet and get numpy's array type into the Python core once > and for all? > > Sort of Tulip for arrays... Aka memoryview :) Stefan Krah already fixed most of the multidimensional support issues in 3.3 (including the "cast" method to reinterpret the contents in a different format). The main missing API elements are multidimensional slicing and the ability to export them from types defined in Python. Cheers, Nick. > > Skip > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130909/72a66d88/attachment.html> From victor.stinner at gmail.com Mon Sep 9 14:30:50 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Mon, 9 Sep 2013 14:30:50 +0200 Subject: [Python-Dev] [python-committers] [RELEASED] Python 3.4.0a2 In-Reply-To: <522DB8D3.1030803@hastings.org> References: <522DB8D3.1030803@hastings.org> Message-ID: <CAMpsgwZ3Ltsc4dKRnN5zBVvcSF4WFX4YQRcG9oaq_iMvx0NjvQ@mail.gmail.com> 2013/9/9 Larry Hastings <larry at hastings.org>: > Python 3.4 includes a range of improvements of the 3.x series, including > hundreds of small improvements and bug fixes. Major new features and > changes in the 3.4 release series so far include: > > * PEP 446, changing file descriptors to not be inherited by default > in subprocesses The title of the PEP is "Make newly created file descriptors non-inheritable". It has an impact on all functions creating files and sockets not only the subprocess module. You can also add a link to the nice What?s New In Python 3.4 document: http://docs.python.org/dev/whatsnew/3.4.html Victor From oscar.j.benjamin at gmail.com Mon Sep 9 14:57:50 2013 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Mon, 9 Sep 2013 13:57:50 +0100 Subject: [Python-Dev] PEP 450 adding statistics module In-Reply-To: <CADiSq7fXsYCy125GyX9HenHkzXMs_2HZu4Smrc+q5_J-tWyv1A@mail.gmail.com> References: <520C2E23.40405@pearwood.info> <52215BDF.2090109@pearwood.info> <20130908123706.GG26319@ando> <74b03474-ab18-40c5-8ce6-de5159d34903@email.android.com> <522CA085.6000601@stoneleaf.us> <CAP7+vJ+TTaU+ntaLT3xEU2wvFQhgP4P2-FnZUJ7V2PAAy3SMEA@mail.gmail.com> <CAHVvXxRAi_HHFOD2dY8UGsXpbgd-LyFU9rsg9x6Uk_=VuKG4hA@mail.gmail.com> <CAP7+vJK0HRT4p0MEqraCr8EY1MN29E6CnkVcxvOks_s1rMka_w@mail.gmail.com> <87ppsi8zwx.fsf@uwakimon.sk.tsukuba.ac.jp> <CANc-5UxZ5M7MXxCPYY6Tfn0rpjRUT+yjyOWnGgFO8ROSyWfrHQ@mail.gmail.com> <CADiSq7fXsYCy125GyX9HenHkzXMs_2HZu4Smrc+q5_J-tWyv1A@mail.gmail.com> Message-ID: <CAHVvXxTy83-=GJvq3ud6Vee2pF5FuR5n_C3Ac_+aCCjgDwZOkw@mail.gmail.com> On 9 September 2013 12:56, Nick Coghlan <ncoghlan at gmail.com> wrote: >> Alternatively, I thought there was discussion a long time ago about >> getting numpy's >> (or even further back, numeric's?) array type into the core. Python >> has an array type >> which I don't think gets a lot of use (or love). Might it be >> worthwhile to make sure the >> PEP 450 package works with that? Then extend it to multiple dimensions? Or >> just >> bite the bullet and get numpy's array type into the Python core once >> and for all? >> >> Sort of Tulip for arrays... > > Aka memoryview :) > > Stefan Krah already fixed most of the multidimensional support issues in 3.3 > (including the "cast" method to reinterpret the contents in a different > format). The main missing API elements are multidimensional slicing and the > ability to export them from types defined in Python. Being very familiar with numpy's ndarrays and not so much with memoryviews this prompted me to go and have a look at them. How exactly are you supposed to create a multidimensional array using memoryviews? The best I could come up with was something like: $ py -3.3 Python 3.3.2 (v3.3.2:d047928ae3f6, May 16 2013, 00:03:43) [MSC v.1600 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> import array >>> v = memoryview(array.array('b', [1, 2, 3, 4])).cast('b', (2, 2)) >>> v.shape (2, 2) However I don't seem to be able to access the elements: >>> v[0, 1] Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: memoryview: invalid slice key >>> v[0] Traceback (most recent call last): File "<stdin>", line 1, in <module> NotImplementedError: multi-dimensional sub-views are not implemented >>> v[0:1] <memory at 0x00DF8B70> >>> v[0:1].shape (1, 2) >>> v[0:1][0] Traceback (most recent call last): File "<stdin>", line 1, in <module> NotImplementedError: multi-dimensional sub-views are not implemented >>> v[1] Traceback (most recent call last): File "<stdin>", line 1, in <module> NotImplementedError: multi-dimensional sub-views are not implemented >>> v[:, 1] Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: memoryview: invalid slice key >>> v[1, :] Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: memoryview: invalid slice key And the .cast method bails if you try to use a more useful type code: >>> v = memoryview(array.array('q', [1, 2, 3, 4])).cast('q', (2, 2)) Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: memoryview: cannot cast between two non-byte formats Oscar From ncoghlan at gmail.com Mon Sep 9 16:15:25 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 10 Sep 2013 00:15:25 +1000 Subject: [Python-Dev] PEP 450 adding statistics module In-Reply-To: <CAHVvXxTy83-=GJvq3ud6Vee2pF5FuR5n_C3Ac_+aCCjgDwZOkw@mail.gmail.com> References: <520C2E23.40405@pearwood.info> <52215BDF.2090109@pearwood.info> <20130908123706.GG26319@ando> <74b03474-ab18-40c5-8ce6-de5159d34903@email.android.com> <522CA085.6000601@stoneleaf.us> <CAP7+vJ+TTaU+ntaLT3xEU2wvFQhgP4P2-FnZUJ7V2PAAy3SMEA@mail.gmail.com> <CAHVvXxRAi_HHFOD2dY8UGsXpbgd-LyFU9rsg9x6Uk_=VuKG4hA@mail.gmail.com> <CAP7+vJK0HRT4p0MEqraCr8EY1MN29E6CnkVcxvOks_s1rMka_w@mail.gmail.com> <87ppsi8zwx.fsf@uwakimon.sk.tsukuba.ac.jp> <CANc-5UxZ5M7MXxCPYY6Tfn0rpjRUT+yjyOWnGgFO8ROSyWfrHQ@mail.gmail.com> <CADiSq7fXsYCy125GyX9HenHkzXMs_2HZu4Smrc+q5_J-tWyv1A@mail.gmail.com> <CAHVvXxTy83-=GJvq3ud6Vee2pF5FuR5n_C3Ac_+aCCjgDwZOkw@mail.gmail.com> Message-ID: <CADiSq7czgXzgTFUr-AANWGUx_umRggwKbOvGR3ixTMv2odhs_w@mail.gmail.com> On 9 Sep 2013 22:58, "Oscar Benjamin" <oscar.j.benjamin at gmail.com> wrote: > > On 9 September 2013 12:56, Nick Coghlan <ncoghlan at gmail.com> wrote: > >> Alternatively, I thought there was discussion a long time ago about > >> getting numpy's > >> (or even further back, numeric's?) array type into the core. Python > >> has an array type > >> which I don't think gets a lot of use (or love). Might it be > >> worthwhile to make sure the > >> PEP 450 package works with that? Then extend it to multiple dimensions? Or > >> just > >> bite the bullet and get numpy's array type into the Python core once > >> and for all? > >> > >> Sort of Tulip for arrays... > > > > Aka memoryview :) > > > > Stefan Krah already fixed most of the multidimensional support issues in 3.3 > > (including the "cast" method to reinterpret the contents in a different > > format). The main missing API elements are multidimensional slicing and the > > ability to export them from types defined in Python. > > Being very familiar with numpy's ndarrays and not so much with > memoryviews this prompted me to go and have a look at them. > > How exactly are you supposed to create a multidimensional array using > memoryviews? The best I could come up with was something like: > > $ py -3.3 > Python 3.3.2 (v3.3.2:d047928ae3f6, May 16 2013, 00:03:43) [MSC v.1600 > 32 bit (Intel)] on win32 > Type "help", "copyright", "credits" or "license" for more information. > >>> import array > >>> v = memoryview(array.array('b', [1, 2, 3, 4])).cast('b', (2, 2)) > >>> v.shape > (2, 2) > > However I don't seem to be able to access the elements: > > >>> v[0, 1] > Traceback (most recent call last): > File "<stdin>", line 1, in <module> > TypeError: memoryview: invalid slice key > >>> v[0] > Traceback (most recent call last): > File "<stdin>", line 1, in <module> > NotImplementedError: multi-dimensional sub-views are not implemented > >>> v[0:1] > <memory at 0x00DF8B70> > >>> v[0:1].shape > (1, 2) > >>> v[0:1][0] > Traceback (most recent call last): > File "<stdin>", line 1, in <module> > NotImplementedError: multi-dimensional sub-views are not implemented > >>> v[1] > Traceback (most recent call last): > File "<stdin>", line 1, in <module> > NotImplementedError: multi-dimensional sub-views are not implemented > >>> v[:, 1] > Traceback (most recent call last): > File "<stdin>", line 1, in <module> > TypeError: memoryview: invalid slice key > >>> v[1, :] > Traceback (most recent call last): > File "<stdin>", line 1, in <module> > TypeError: memoryview: invalid slice key > > And the .cast method bails if you try to use a more useful type code: > > >>> v = memoryview(array.array('q', [1, 2, 3, 4])).cast('q', (2, 2)) > Traceback (most recent call last): > File "<stdin>", line 1, in <module> > TypeError: memoryview: cannot cast between two non-byte formats Oops, forgot the type casting restrictions, too. My main point was that PEP 3118 is already intended as the tulip equivalent for multi-dimensional arrays, and memoryview is the stdlib API for that. It's just incomplete, since most serious multi-dimensional use cases involve skipping memoryview and go straight to NumPy or one of the image libraries. As far as I am aware, there's no opposition to fixing the multi-dimensional support in memoryview *per se*, just the usual concerns about maintainability and a question of volunteers with the time to actually resolve the relevant open issues on the bug tracker. The fairly extensive 3.3 changes focused on fixing stuff that was previously outright broken, but several limitations remain, often because the best API wasn't clear, or because it reached the point where "just use NumPy" seemed like a justifiable answer. Cheers, Nick. > > > Oscar -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130910/54651c2a/attachment.html> From ncoghlan at gmail.com Mon Sep 9 16:20:54 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 10 Sep 2013 00:20:54 +1000 Subject: [Python-Dev] [Python-checkins] cpython: Post-3.4.0a2-release fixups. In-Reply-To: <3cYSwh0rHdz7Lkm@mail.python.org> References: <3cYSwh0rHdz7Lkm@mail.python.org> Message-ID: <CADiSq7fVsbLCks=oK4UxgJNnzngTmEYLmOW_HEj_OXHMfJwUow@mail.gmail.com> On 9 Sep 2013 22:15, "larry.hastings" <python-checkins at python.org> wrote: > > http://hg.python.org/cpython/rev/6b211a0c8042 > changeset: 85645:6b211a0c8042 > user: Larry Hastings <larry at hastings.org> > date: Mon Sep 09 21:08:52 2013 +0900 > summary: > Post-3.4.0a2-release fixups. > > files: > Include/patchlevel.h | 2 +- > Misc/NEWS | 14 +++++++++++++- > 2 files changed, 14 insertions(+), 2 deletions(-) > > > diff --git a/Include/patchlevel.h b/Include/patchlevel.h > --- a/Include/patchlevel.h > +++ b/Include/patchlevel.h > @@ -23,7 +23,7 @@ > #define PY_RELEASE_SERIAL 2 > > /* Version as a string */ > -#define PY_VERSION "3.4.0a2" > +#define PY_VERSION "3.4.0a2+" > /*--end constants--*/ > > /* Version as a single 4-byte hex number, e.g. 0x010502B2 == 1.5.2b2. > diff --git a/Misc/NEWS b/Misc/NEWS > --- a/Misc/NEWS > +++ b/Misc/NEWS > @@ -2,10 +2,22 @@ > Python News > +++++++++++ > > +What's New in Python 3.4.0 Alpha 3? > +=================================== > + > +Projected Release date: 2013-09-29 > + > +Core and Builtins > +----------------- > + > +Library > +------- > + > + I had already pushed alpha 3 entries in NEWS, so something seems to have gone wrong here. Perhaps, if RMs are preparing the release out of tree, we could get the NEWS file headings on default updated immediately after the last included commit? Cheers, Nick. > What's New in Python 3.4.0 Alpha 2? > =================================== > > -Projected Release date: 2013-09-08 > +Release date: 2013-09-09 > > Core and Builtins > ----------------- > > -- > Repository URL: http://hg.python.org/cpython > > _______________________________________________ > Python-checkins mailing list > Python-checkins at python.org > https://mail.python.org/mailman/listinfo/python-checkins > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130910/02037892/attachment.html> From ethan at stoneleaf.us Mon Sep 9 16:30:12 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 09 Sep 2013 07:30:12 -0700 Subject: [Python-Dev] PEP 447: add type.__locallookup__ In-Reply-To: <7D6D4488-2855-41F8-A9F2-1FF23B9E3A56@mac.com> References: <75030FAC-6918-4E94-95DA-67A88D53E6F5@mac.com> <7D6D4488-2855-41F8-A9F2-1FF23B9E3A56@mac.com> Message-ID: <522DDB74.5080609@stoneleaf.us> On 07/30/2013 11:17 PM, Ronald Oussoren wrote: > > And something I forgot to ask: is anyone willing to be the BDFL-Delegate for > PEP 447? *Bump*. It would be nice if this could make into 3.4. -- ~Ethan~ From rdmurray at bitdance.com Mon Sep 9 16:54:22 2013 From: rdmurray at bitdance.com (R. David Murray) Date: Mon, 09 Sep 2013 10:54:22 -0400 Subject: [Python-Dev] [python-committers] [RELEASED] Python 3.4.0a2 In-Reply-To: <20130909144551.76b84640@pitrou.net> References: <522DB8D3.1030803@hastings.org> <CAMpsgwZ3Ltsc4dKRnN5zBVvcSF4WFX4YQRcG9oaq_iMvx0NjvQ@mail.gmail.com> <20130909144551.76b84640@pitrou.net> Message-ID: <20130909145422.B5689250166@webabinitio.net> On Mon, 09 Sep 2013 14:45:51 +0200, Antoine Pitrou <solipsis at pitrou.net> wrote: > Le Mon, 9 Sep 2013 14:30:50 +0200, > Victor Stinner <victor.stinner at gmail.com> a ??crit : > > 2013/9/9 Larry Hastings <larry at hastings.org>: > > > Python 3.4 includes a range of improvements of the 3.x series, > > > including hundreds of small improvements and bug fixes. Major new > > > features and changes in the 3.4 release series so far include: > > > > > > * PEP 446, changing file descriptors to not be inherited by default > > > in subprocesses > > > > The title of the PEP is "Make newly created file descriptors > > non-inheritable". It has an impact on all functions creating files and > > sockets not only the subprocess module. > > I don't think Larry's description is wrong. "Non-inheritable" is a > shorthand for "non-inheritable in subprocesses" with "subprocesses" > taken in the general sense (i.e. not only created with the subprocess > module). Not wrong, but definitely confusing. It is worth clarifying *somehow* that this does not apply only to the subprocess module, which is what a naive (or fast) reader will assume. --David From jcea at jcea.es Mon Sep 9 17:11:21 2013 From: jcea at jcea.es (Jesus Cea) Date: Mon, 09 Sep 2013 17:11:21 +0200 Subject: [Python-Dev] Offtopic: OpenID Providers In-Reply-To: <20130906193430.AFA14250816@webabinitio.net> References: <5228C00F.7000209@jcea.es> <l0d311$b7n$1@ger.gmane.org> <6DB0A7E3-FADD-4E35-A983-02A7C0D4CAE3@stufft.io> <20130906191130.B6511250816@webabinitio.net> <7087E6B8-1CD4-4002-AD3F-5578B045A3CB@stufft.io> <20130906193430.AFA14250816@webabinitio.net> Message-ID: <522DE519.7080909@jcea.es> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 06/09/13 21:34, R. David Murray wrote: > Note that I said that single signon *itself* was overrated. If you > use the same token to authenticate to multiple sites (and here the > 'token' is the email address) then your identities on those sites > are ipso facto associated with each other. *If* that email address > is also never leaked (never displayed, even to other signed on > users, all communication with the site encrypted), then you only > have to worry if the sites exchange information about their > accounts, or if the government comes knocking on their doors.... > > Yes, I'm paranoid. That doesn't mean they aren't listening. Being paranoid is good. Fix for this is actually trivial: Use different emails for different "personalities". If you are doing things you really NEED to hide, virtual machines and TOR is the way to go. - -- Jes?s Cea Avi?n _/_/ _/_/_/ _/_/_/ jcea at jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ Twitter: @jcea _/_/ _/_/ _/_/_/_/_/ jabber / xmpp:jcea at jabber.org _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQCVAwUBUi3lGZlgi5GaxT1NAQLykAQAi8WDuWmEAAX7bP1glDT8iLrMRpKlu+Vh WndX9ObB/os2D9RZkL7DZB01EDMRvfjGlWFm3gQV0CbM9smkgGkhJNLuxYzA2fpK PQlbO4KUKCoQG7qm413TA1g0xxOGzG2n9g2kJisFBCNu2Y2PXroUhm6p41CbTd89 ovwZLLUcPZU= =EQu4 -----END PGP SIGNATURE----- From guido at python.org Mon Sep 9 17:27:41 2013 From: guido at python.org (Guido van Rossum) Date: Mon, 9 Sep 2013 08:27:41 -0700 Subject: [Python-Dev] PEP 447: add type.__locallookup__ In-Reply-To: <522DDB74.5080609@stoneleaf.us> References: <75030FAC-6918-4E94-95DA-67A88D53E6F5@mac.com> <7D6D4488-2855-41F8-A9F2-1FF23B9E3A56@mac.com> <522DDB74.5080609@stoneleaf.us> Message-ID: <CAP7+vJJTR=+cRY+iQgHY4ujTsC3_QyWn83dnK96rWO6bV9vFLQ@mail.gmail.com> Let's just accept this PEP. It looks like a nice addition to the metaclass machinery and I don't think we'll get much more useful feedback by waiting. On Mon, Sep 9, 2013 at 7:30 AM, Ethan Furman <ethan at stoneleaf.us> wrote: > On 07/30/2013 11:17 PM, Ronald Oussoren wrote: > >> >> And something I forgot to ask: is anyone willing to be the BDFL-Delegate >> for >> PEP 447? >> > > *Bump*. > > It would be nice if this could make into 3.4. > > -- > ~Ethan~ > > ______________________________**_________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/**mailman/listinfo/python-dev<https://mail.python.org/mailman/listinfo/python-dev> > Unsubscribe: https://mail.python.org/**mailman/options/python-dev/** > guido%40python.org<https://mail.python.org/mailman/options/python-dev/guido%40python.org> > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130909/e78eb7d7/attachment.html> From benjamin at python.org Mon Sep 9 17:32:20 2013 From: benjamin at python.org (Benjamin Peterson) Date: Mon, 9 Sep 2013 11:32:20 -0400 Subject: [Python-Dev] [Python-checkins] cpython: Post-3.4.0a2-release fixups. In-Reply-To: <CADiSq7fVsbLCks=oK4UxgJNnzngTmEYLmOW_HEj_OXHMfJwUow@mail.gmail.com> References: <3cYSwh0rHdz7Lkm@mail.python.org> <CADiSq7fVsbLCks=oK4UxgJNnzngTmEYLmOW_HEj_OXHMfJwUow@mail.gmail.com> Message-ID: <CAPZV6o8jzs_nUkgtJL0u6vxoGgX=3JihF5FXUovnLGK96m1o0Q@mail.gmail.com> Well, it's important for the release manager to make sure what the script is doing is sane. :) 2013/9/9 Nick Coghlan <ncoghlan at gmail.com>: > > On 9 Sep 2013 22:15, "larry.hastings" <python-checkins at python.org> wrote: >> >> http://hg.python.org/cpython/rev/6b211a0c8042 >> changeset: 85645:6b211a0c8042 >> user: Larry Hastings <larry at hastings.org> >> date: Mon Sep 09 21:08:52 2013 +0900 >> summary: >> Post-3.4.0a2-release fixups. >> >> files: >> Include/patchlevel.h | 2 +- >> Misc/NEWS | 14 +++++++++++++- >> 2 files changed, 14 insertions(+), 2 deletions(-) >> >> >> diff --git a/Include/patchlevel.h b/Include/patchlevel.h >> --- a/Include/patchlevel.h >> +++ b/Include/patchlevel.h >> @@ -23,7 +23,7 @@ >> #define PY_RELEASE_SERIAL 2 >> >> /* Version as a string */ >> -#define PY_VERSION "3.4.0a2" >> +#define PY_VERSION "3.4.0a2+" >> /*--end constants--*/ >> >> /* Version as a single 4-byte hex number, e.g. 0x010502B2 == 1.5.2b2. >> diff --git a/Misc/NEWS b/Misc/NEWS >> --- a/Misc/NEWS >> +++ b/Misc/NEWS >> @@ -2,10 +2,22 @@ >> Python News >> +++++++++++ >> >> +What's New in Python 3.4.0 Alpha 3? >> +=================================== >> + >> +Projected Release date: 2013-09-29 >> + >> +Core and Builtins >> +----------------- >> + >> +Library >> +------- >> + >> + > > I had already pushed alpha 3 entries in NEWS, so something seems to have > gone wrong here. > > Perhaps, if RMs are preparing the release out of tree, we could get the NEWS > file headings on default updated immediately after the last included commit? > > Cheers, > Nick. > >> What's New in Python 3.4.0 Alpha 2? >> =================================== >> >> -Projected Release date: 2013-09-08 >> +Release date: 2013-09-09 >> >> Core and Builtins >> ----------------- >> >> -- >> Repository URL: http://hg.python.org/cpython >> >> _______________________________________________ >> Python-checkins mailing list >> Python-checkins at python.org >> https://mail.python.org/mailman/listinfo/python-checkins >> > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/benjamin%40python.org > -- Regards, Benjamin From rdmurray at bitdance.com Mon Sep 9 17:36:32 2013 From: rdmurray at bitdance.com (R. David Murray) Date: Mon, 09 Sep 2013 11:36:32 -0400 Subject: [Python-Dev] Offtopic: paranoia In-Reply-To: <522DE519.7080909@jcea.es> References: <5228C00F.7000209@jcea.es> <l0d311$b7n$1@ger.gmane.org> <6DB0A7E3-FADD-4E35-A983-02A7C0D4CAE3@stufft.io> <20130906191130.B6511250816@webabinitio.net> <7087E6B8-1CD4-4002-AD3F-5578B045A3CB@stufft.io> <20130906193430.AFA14250816@webabinitio.net> <522DE519.7080909@jcea.es> Message-ID: <20130909153632.EDACD250166@webabinitio.net> On Mon, 09 Sep 2013 17:11:21 +0200, Jesus Cea <jcea at jcea.es> wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > On 06/09/13 21:34, R. David Murray wrote: > > Note that I said that single signon *itself* was overrated. If you > > use the same token to authenticate to multiple sites (and here the > > 'token' is the email address) then your identities on those sites > > are ipso facto associated with each other. *If* that email address > > is also never leaked (never displayed, even to other signed on > > users, all communication with the site encrypted), then you only > > have to worry if the sites exchange information about their > > accounts, or if the government comes knocking on their doors.... > > > > Yes, I'm paranoid. That doesn't mean they aren't listening. > > Being paranoid is good. Fix for this is actually trivial: Use > different emails for different "personalities". Yes, that's exactly my point. > If you are doing things you really NEED to hide, virtual machines and > TOR is the way to go. Well, it would helpful if a lot more people started routing traffic through TOR even when they didn't NEED to. I plan to start doing so soon. --David From mark at hotpy.org Mon Sep 9 17:43:31 2013 From: mark at hotpy.org (Mark Shannon) Date: Mon, 09 Sep 2013 16:43:31 +0100 Subject: [Python-Dev] PEP 447: add type.__locallookup__ In-Reply-To: <CAP7+vJJTR=+cRY+iQgHY4ujTsC3_QyWn83dnK96rWO6bV9vFLQ@mail.gmail.com> References: <75030FAC-6918-4E94-95DA-67A88D53E6F5@mac.com> <7D6D4488-2855-41F8-A9F2-1FF23B9E3A56@mac.com> <522DDB74.5080609@stoneleaf.us> <CAP7+vJJTR=+cRY+iQgHY4ujTsC3_QyWn83dnK96rWO6bV9vFLQ@mail.gmail.com> Message-ID: <522DECA3.2030709@hotpy.org> I would like time to investigate this further, but at the moment I think it will either make attribute lookup poorly defined or slow. Of the top of my head, the problem as a I see it is basically this: Currently, type.__getattribute__() is a fixed point in the lookup of attributes. The proposal means that a fixed point is not reached until the cls parameter of type.__getattribute__() is either object or type, otherwise type.__getattribute__() and type.__locallookup__ must bounce back and forth. This will slow down *every* attribute lookup for what is a fairly obscure use case. Cheers, Mark. On 09/09/13 16:27, Guido van Rossum wrote: > Let's just accept this PEP. It looks like a nice addition to the metaclass machinery and I don't think we'll get much more useful feedback by waiting. > > > On Mon, Sep 9, 2013 at 7:30 AM, Ethan Furman <ethan at stoneleaf.us <mailto:ethan at stoneleaf.us>> wrote: > > On 07/30/2013 11:17 PM, Ronald Oussoren wrote: > > > And something I forgot to ask: is anyone willing to be the BDFL-Delegate for > PEP 447? > > > *Bump*. > > It would be nice if this could make into 3.4. > > -- > ~Ethan~ > > _________________________________________________ > Python-Dev mailing list > Python-Dev at python.org <mailto:Python-Dev at python.org> > https://mail.python.org/__mailman/listinfo/python-dev <https://mail.python.org/mailman/listinfo/python-dev> > Unsubscribe: https://mail.python.org/__mailman/options/python-dev/__guido%40python.org <https://mail.python.org/mailman/options/python-dev/guido%40python.org> > > > > > -- > --Guido van Rossum (python.org/~guido <http://python.org/~guido>) > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/mark%40hotpy.org > From guido at python.org Mon Sep 9 18:03:23 2013 From: guido at python.org (Guido van Rossum) Date: Mon, 9 Sep 2013 09:03:23 -0700 Subject: [Python-Dev] PEP 447: add type.__locallookup__ In-Reply-To: <522DECA3.2030709@hotpy.org> References: <75030FAC-6918-4E94-95DA-67A88D53E6F5@mac.com> <7D6D4488-2855-41F8-A9F2-1FF23B9E3A56@mac.com> <522DDB74.5080609@stoneleaf.us> <CAP7+vJJTR=+cRY+iQgHY4ujTsC3_QyWn83dnK96rWO6bV9vFLQ@mail.gmail.com> <522DECA3.2030709@hotpy.org> Message-ID: <CAP7+vJ+thJvYpk0d0rnFO6bm88sTJ_t1QdYFB0SdPOGNDABuag@mail.gmail.com> OK, how much time do you need? --Guido van Rossum (sent from Android phone) On Sep 9, 2013 8:44 AM, "Mark Shannon" <mark at hotpy.org> wrote: > I would like time to investigate this further, but at the moment I think > it will either make attribute lookup poorly defined or slow. > > Of the top of my head, the problem as a I see it is basically this: > Currently, type.__getattribute__() is a fixed point in the lookup of > attributes. > The proposal means that a fixed point is not reached until the cls > parameter of type.__getattribute__() is either object or type, > otherwise type.__getattribute__() and type.__locallookup__ must bounce > back and forth. > > This will slow down *every* attribute lookup for what is a fairly obscure > use case. > > Cheers, > Mark. > > On 09/09/13 16:27, Guido van Rossum wrote: > >> Let's just accept this PEP. It looks like a nice addition to the >> metaclass machinery and I don't think we'll get much more useful feedback >> by waiting. >> >> >> On Mon, Sep 9, 2013 at 7:30 AM, Ethan Furman <ethan at stoneleaf.us <mailto: >> ethan at stoneleaf.us>> wrote: >> >> On 07/30/2013 11:17 PM, Ronald Oussoren wrote: >> >> >> And something I forgot to ask: is anyone willing to be the >> BDFL-Delegate for >> PEP 447? >> >> >> *Bump*. >> >> It would be nice if this could make into 3.4. >> >> -- >> ~Ethan~ >> >> ______________________________**___________________ >> Python-Dev mailing list >> Python-Dev at python.org <mailto:Python-Dev at python.org> >> https://mail.python.org/__**mailman/listinfo/python-dev<https://mail.python.org/__mailman/listinfo/python-dev>< >> https://mail.python.org/**mailman/listinfo/python-dev<https://mail.python.org/mailman/listinfo/python-dev> >> > >> Unsubscribe: https://mail.python.org/__** >> mailman/options/python-dev/__**guido%40python.org<https://mail.python.org/__mailman/options/python-dev/__guido%40python.org>< >> https://mail.python.org/**mailman/options/python-dev/**guido%40python.org<https://mail.python.org/mailman/options/python-dev/guido%40python.org> >> > >> >> >> >> >> -- >> --Guido van Rossum (python.org/~guido <http://python.org/~guido>) >> >> >> ______________________________**_________________ >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/**mailman/listinfo/python-dev<https://mail.python.org/mailman/listinfo/python-dev> >> Unsubscribe: https://mail.python.org/**mailman/options/python-dev/** >> mark%40hotpy.org<https://mail.python.org/mailman/options/python-dev/mark%40hotpy.org> >> >> ______________________________**_________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/**mailman/listinfo/python-dev<https://mail.python.org/mailman/listinfo/python-dev> > Unsubscribe: https://mail.python.org/**mailman/options/python-dev/** > guido%40python.org<https://mail.python.org/mailman/options/python-dev/guido%40python.org> > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130909/22254236/attachment-0001.html> From mark at hotpy.org Mon Sep 9 18:06:28 2013 From: mark at hotpy.org (Mark Shannon) Date: Mon, 09 Sep 2013 17:06:28 +0100 Subject: [Python-Dev] PEP 447: add type.__locallookup__ In-Reply-To: <CAP7+vJ+thJvYpk0d0rnFO6bm88sTJ_t1QdYFB0SdPOGNDABuag@mail.gmail.com> References: <75030FAC-6918-4E94-95DA-67A88D53E6F5@mac.com> <7D6D4488-2855-41F8-A9F2-1FF23B9E3A56@mac.com> <522DDB74.5080609@stoneleaf.us> <CAP7+vJJTR=+cRY+iQgHY4ujTsC3_QyWn83dnK96rWO6bV9vFLQ@mail.gmail.com> <522DECA3.2030709@hotpy.org> <CAP7+vJ+thJvYpk0d0rnFO6bm88sTJ_t1QdYFB0SdPOGNDABuag@mail.gmail.com> Message-ID: <522DF204.7040109@hotpy.org> I'll look into it this evening. On 09/09/13 17:03, Guido van Rossum wrote: > OK, how much time do you need? > > --Guido van Rossum (sent from Android phone) > > On Sep 9, 2013 8:44 AM, "Mark Shannon" <mark at hotpy.org <mailto:mark at hotpy.org>> wrote: > > I would like time to investigate this further, but at the moment I think it will either make attribute lookup poorly defined or slow. > > Of the top of my head, the problem as a I see it is basically this: > Currently, type.__getattribute__() is a fixed point in the lookup of attributes. > The proposal means that a fixed point is not reached until the cls parameter of type.__getattribute__() is either object or type, > otherwise type.__getattribute__() and type.__locallookup__ must bounce back and forth. > > This will slow down *every* attribute lookup for what is a fairly obscure use case. > > Cheers, > Mark. > > On 09/09/13 16:27, Guido van Rossum wrote: > > Let's just accept this PEP. It looks like a nice addition to the metaclass machinery and I don't think we'll get much more useful feedback by waiting. > > > On Mon, Sep 9, 2013 at 7:30 AM, Ethan Furman <ethan at stoneleaf.us <mailto:ethan at stoneleaf.us> <mailto:ethan at stoneleaf.us <mailto:ethan at stoneleaf.us>>> wrote: > > On 07/30/2013 11:17 PM, Ronald Oussoren wrote: > > > And something I forgot to ask: is anyone willing to be the BDFL-Delegate for > PEP 447? > > > *Bump*. > > It would be nice if this could make into 3.4. > > -- > ~Ethan~ > > ___________________________________________________ > Python-Dev mailing list > Python-Dev at python.org <mailto:Python-Dev at python.org> <mailto:Python-Dev at python.org <mailto:Python-Dev at python.org>> > https://mail.python.org/____mailman/listinfo/python-dev <https://mail.python.org/__mailman/listinfo/python-dev> <https://mail.python.org/__mailman/listinfo/python-dev > <https://mail.python.org/mailman/listinfo/python-dev>> > Unsubscribe: https://mail.python.org/____mailman/options/python-dev/____guido%40python.org <https://mail.python.org/__mailman/options/python-dev/__guido%40python.org> > <https://mail.python.org/__mailman/options/python-dev/__guido%40python.org <https://mail.python.org/mailman/options/python-dev/guido%40python.org>> > > > > > -- > --Guido van Rossum (python.org/~guido <http://python.org/~guido> <http://python.org/~guido>) > > > _________________________________________________ > Python-Dev mailing list > Python-Dev at python.org <mailto:Python-Dev at python.org> > https://mail.python.org/__mailman/listinfo/python-dev <https://mail.python.org/mailman/listinfo/python-dev> > Unsubscribe: https://mail.python.org/__mailman/options/python-dev/__mark%40hotpy.org <https://mail.python.org/mailman/options/python-dev/mark%40hotpy.org> > > _________________________________________________ > Python-Dev mailing list > Python-Dev at python.org <mailto:Python-Dev at python.org> > https://mail.python.org/__mailman/listinfo/python-dev <https://mail.python.org/mailman/listinfo/python-dev> > Unsubscribe: https://mail.python.org/__mailman/options/python-dev/__guido%40python.org <https://mail.python.org/mailman/options/python-dev/guido%40python.org> > From ncoghlan at gmail.com Mon Sep 9 18:12:27 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 10 Sep 2013 02:12:27 +1000 Subject: [Python-Dev] [Python-checkins] cpython: Post-3.4.0a2-release fixups. In-Reply-To: <CAPZV6o8jzs_nUkgtJL0u6vxoGgX=3JihF5FXUovnLGK96m1o0Q@mail.gmail.com> References: <3cYSwh0rHdz7Lkm@mail.python.org> <CADiSq7fVsbLCks=oK4UxgJNnzngTmEYLmOW_HEj_OXHMfJwUow@mail.gmail.com> <CAPZV6o8jzs_nUkgtJL0u6vxoGgX=3JihF5FXUovnLGK96m1o0Q@mail.gmail.com> Message-ID: <CADiSq7f_fSk8kUZuRp9zX7yCfiQyHKT5GJnFve1TT_WY7Rar8A@mail.gmail.com> On 10 Sep 2013 01:32, "Benjamin Peterson" <benjamin at python.org> wrote: > > Well, it's important for the release manager to make sure what the > script is doing is sane. :) Sure, preparing out of tree is fine and sensible. But we should either freeze the tree or update the NEWS headers immediately, otherwise we're going to have updates going into the wrong section. Cheers, Nick. > > 2013/9/9 Nick Coghlan <ncoghlan at gmail.com>: > > > > On 9 Sep 2013 22:15, "larry.hastings" <python-checkins at python.org> wrote: > >> > >> http://hg.python.org/cpython/rev/6b211a0c8042 > >> changeset: 85645:6b211a0c8042 > >> user: Larry Hastings <larry at hastings.org> > >> date: Mon Sep 09 21:08:52 2013 +0900 > >> summary: > >> Post-3.4.0a2-release fixups. > >> > >> files: > >> Include/patchlevel.h | 2 +- > >> Misc/NEWS | 14 +++++++++++++- > >> 2 files changed, 14 insertions(+), 2 deletions(-) > >> > >> > >> diff --git a/Include/patchlevel.h b/Include/patchlevel.h > >> --- a/Include/patchlevel.h > >> +++ b/Include/patchlevel.h > >> @@ -23,7 +23,7 @@ > >> #define PY_RELEASE_SERIAL 2 > >> > >> /* Version as a string */ > >> -#define PY_VERSION "3.4.0a2" > >> +#define PY_VERSION "3.4.0a2+" > >> /*--end constants--*/ > >> > >> /* Version as a single 4-byte hex number, e.g. 0x010502B2 == 1.5.2b2. > >> diff --git a/Misc/NEWS b/Misc/NEWS > >> --- a/Misc/NEWS > >> +++ b/Misc/NEWS > >> @@ -2,10 +2,22 @@ > >> Python News > >> +++++++++++ > >> > >> +What's New in Python 3.4.0 Alpha 3? > >> +=================================== > >> + > >> +Projected Release date: 2013-09-29 > >> + > >> +Core and Builtins > >> +----------------- > >> + > >> +Library > >> +------- > >> + > >> + > > > > I had already pushed alpha 3 entries in NEWS, so something seems to have > > gone wrong here. > > > > Perhaps, if RMs are preparing the release out of tree, we could get the NEWS > > file headings on default updated immediately after the last included commit? > > > > Cheers, > > Nick. > > > >> What's New in Python 3.4.0 Alpha 2? > >> =================================== > >> > >> -Projected Release date: 2013-09-08 > >> +Release date: 2013-09-09 > >> > >> Core and Builtins > >> ----------------- > >> > >> -- > >> Repository URL: http://hg.python.org/cpython > >> > >> _______________________________________________ > >> Python-checkins mailing list > >> Python-checkins at python.org > >> https://mail.python.org/mailman/listinfo/python-checkins > >> > > > > > > _______________________________________________ > > Python-Dev mailing list > > Python-Dev at python.org > > https://mail.python.org/mailman/listinfo/python-dev > > Unsubscribe: > > https://mail.python.org/mailman/options/python-dev/benjamin%40python.org > > > > > > -- > Regards, > Benjamin -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130910/8ae74a09/attachment.html> From ethan at stoneleaf.us Mon Sep 9 18:05:57 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 09 Sep 2013 09:05:57 -0700 Subject: [Python-Dev] PEP 447: add type.__locallookup__ In-Reply-To: <522DECA3.2030709@hotpy.org> References: <75030FAC-6918-4E94-95DA-67A88D53E6F5@mac.com> <7D6D4488-2855-41F8-A9F2-1FF23B9E3A56@mac.com> <522DDB74.5080609@stoneleaf.us> <CAP7+vJJTR=+cRY+iQgHY4ujTsC3_QyWn83dnK96rWO6bV9vFLQ@mail.gmail.com> <522DECA3.2030709@hotpy.org> Message-ID: <522DF1E5.5050309@stoneleaf.us> On 09/09/2013 08:43 AM, Mark Shannon wrote: > I would like time to investigate this further, but at the moment I think it will either make attribute lookup poorly > defined or slow. > > Of the top of my head, the problem as a I see it is basically this: > Currently, type.__getattribute__() is a fixed point in the lookup of attributes. > The proposal means that a fixed point is not reached until the cls parameter of type.__getattribute__() is either object > or type, > otherwise type.__getattribute__() and type.__locallookup__ must bounce back and forth. > > This will slow down *every* attribute lookup for what is a fairly obscure use case. Looks like there's a patch we can try at http://bugs.python.org/issue18181. Here are Ronald's last timings: ------------------------------------------------------------------------------- PYBENCH 2.1 ------------------------------------------------------------------------------- * using CPython 3.4.0a0 (default, Jul 29 2013, 13:01:34) [GCC 4.4.7 20120313 (Red Hat 4.4.7-3)] * disabled garbage collection * system check interval set to maximum: 2147483647 * using timer: time.perf_counter * timer: resolution=1e-09, implementation=clock_gettime(CLOCK_MONOTONIC) ------------------------------------------------------------------------------- Benchmark: pep447.pybench ------------------------------------------------------------------------------- Rounds: 10 Warp: 10 Timer: time.perf_counter Machine Details: Platform ID: Linux-2.6.32-358.114.1.openstack.el6.x86_64-x86_64-with-centos-6.4-Final Processor: x86_64 Python: Implementation: CPython Executable: /tmp/default-pep447/bin/python3 Version: 3.4.0a0 Compiler: GCC 4.4.7 20120313 (Red Hat 4.4.7-3) Bits: 64bit Build: Jul 29 2013 14:09:12 (#default) Unicode: UCS4 ------------------------------------------------------------------------------- Comparing with: default.pybench ------------------------------------------------------------------------------- Rounds: 10 Warp: 10 Timer: time.perf_counter Machine Details: Platform ID: Linux-2.6.32-358.114.1.openstack.el6.x86_64-x86_64-with-centos-6.4-Final Processor: x86_64 Python: Implementation: CPython Executable: /tmp/default/bin/python3 Version: 3.4.0a0 Compiler: GCC 4.4.7 20120313 (Red Hat 4.4.7-3) Bits: 64bit Build: Jul 29 2013 13:01:34 (#default) Unicode: UCS4 Test minimum run-time average run-time this other diff this other diff ------------------------------------------------------------------------------- BuiltinFunctionCalls: 45ms 44ms +1.3% 45ms 44ms +1.3% BuiltinMethodLookup: 26ms 27ms -2.4% 27ms 27ms -2.2% CompareFloats: 33ms 34ms -0.7% 33ms 34ms -1.1% CompareFloatsIntegers: 66ms 67ms -0.9% 66ms 67ms -0.8% CompareIntegers: 51ms 50ms +0.9% 51ms 50ms +0.8% CompareInternedStrings: 34ms 33ms +0.4% 34ms 34ms -0.4% CompareLongs: 29ms 29ms -0.1% 29ms 29ms -0.0% CompareStrings: 43ms 44ms -1.8% 44ms 44ms -1.8% ComplexPythonFunctionCalls: 44ms 42ms +3.9% 44ms 42ms +4.1% ConcatStrings: 33ms 33ms -0.4% 33ms 33ms -1.0% CreateInstances: 47ms 48ms -2.9% 47ms 49ms -3.4% CreateNewInstances: 35ms 36ms -2.5% 36ms 36ms -2.5% CreateStringsWithConcat: 69ms 70ms -0.7% 69ms 70ms -0.9% DictCreation: 52ms 50ms +3.1% 52ms 50ms +3.0% DictWithFloatKeys: 40ms 44ms -10.1% 43ms 45ms -5.8% DictWithIntegerKeys: 32ms 36ms -11.2% 35ms 37ms -4.6% DictWithStringKeys: 29ms 34ms -15.7% 35ms 40ms -11.0% ForLoops: 30ms 29ms +2.2% 30ms 29ms +2.2% IfThenElse: 38ms 41ms -6.7% 38ms 41ms -6.9% ListSlicing: 36ms 36ms -0.7% 36ms 37ms -1.3% NestedForLoops: 43ms 45ms -3.1% 43ms 45ms -3.2% NestedListComprehensions: 39ms 40ms -1.7% 39ms 40ms -2.1% NormalClassAttribute: 86ms 82ms +5.1% 86ms 82ms +5.0% NormalInstanceAttribute: 42ms 42ms +0.3% 42ms 42ms +0.0% PythonFunctionCalls: 39ms 38ms +3.5% 39ms 38ms +2.8% PythonMethodCalls: 51ms 49ms +3.0% 51ms 50ms +2.8% Recursion: 67ms 68ms -1.4% 67ms 68ms -1.4% SecondImport: 41ms 36ms +12.5% 41ms 36ms +12.6% SecondPackageImport: 45ms 40ms +13.1% 45ms 40ms +13.2% SecondSubmoduleImport: 92ms 95ms -2.4% 95ms 98ms -3.6% SimpleComplexArithmetic: 28ms 28ms -0.1% 28ms 28ms -0.2% SimpleDictManipulation: 57ms 57ms -1.0% 57ms 58ms -1.0% SimpleFloatArithmetic: 29ms 28ms +4.7% 29ms 28ms +4.9% SimpleIntFloatArithmetic: 37ms 41ms -8.5% 37ms 41ms -8.7% SimpleIntegerArithmetic: 37ms 41ms -9.4% 37ms 42ms -10.2% SimpleListComprehensions: 33ms 33ms -1.9% 33ms 34ms -2.9% SimpleListManipulation: 28ms 30ms -4.3% 29ms 30ms -4.1% SimpleLongArithmetic: 26ms 26ms +0.5% 26ms 26ms +0.5% SmallLists: 40ms 40ms +0.1% 40ms 40ms +0.1% SmallTuples: 46ms 47ms -2.4% 46ms 48ms -3.0% SpecialClassAttribute: 126ms 120ms +4.7% 126ms 121ms +4.4% SpecialInstanceAttribute: 42ms 42ms +0.6% 42ms 42ms +0.8% StringMappings: 94ms 91ms +3.9% 94ms 91ms +3.8% StringPredicates: 48ms 49ms -1.7% 48ms 49ms -2.1% StringSlicing: 45ms 45ms +1.4% 46ms 45ms +1.5% TryExcept: 23ms 22ms +4.9% 23ms 22ms +4.8% TryFinally: 32ms 32ms -0.1% 32ms 32ms +0.1% TryRaiseExcept: 17ms 17ms +0.9% 17ms 17ms +0.5% TupleSlicing: 49ms 48ms +1.1% 49ms 49ms +1.0% WithFinally: 48ms 47ms +2.3% 48ms 47ms +2.4% WithRaiseExcept: 45ms 44ms +0.8% 45ms 45ms +0.5% ------------------------------------------------------------------------------- Totals: 2284ms 2287ms -0.1% 2306ms 2308ms -0.1% (this=pep447.pybench, other=default.pybench) From a.badger at gmail.com Mon Sep 9 19:39:11 2013 From: a.badger at gmail.com (Toshio Kuratomi) Date: Mon, 9 Sep 2013 10:39:11 -0700 Subject: [Python-Dev] Offtopic: OpenID Providers In-Reply-To: <87hadyah89.fsf@uwakimon.sk.tsukuba.ac.jp> References: <5228C00F.7000209@jcea.es> <20130905181231.GA25306@iskra.aviel.ru> <8593FDCE-E3E0-4AD4-897E-2FB1D751086C@stufft.io> <20130905182522.GA26090@iskra.aviel.ru> <2C429087-4E06-4F0F-B524-25D0F6FB4FFF@stufft.io> <20130905184337.GA26926@iskra.aviel.ru> <667454FE-47D1-46BB-99B6-4C0B10986269@stufft.io> <CANc-5UxNJQp46dA2b54Ad-rifUGv8OCL1H2Yky1gr8EqYxcABA@mail.gmail.com> <20130905203651.GB30331@iskra.aviel.ru> <20130905165318.2fa40bab@anarchist> <87hadyah89.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <CABVPEKpCgeAH5uLThNCMNLLN6quViRWu2orC_386q09sPiMDyg@mail.gmail.com> On Thu, Sep 5, 2013 at 6:09 PM, Stephen J. Turnbull <stephen at xemacs.org> wrote: > Barry Warsaw writes: > > We're open source, and I think it benefits our mission to support open, > > decentralized, and free systems like OpenID and Persona. > > Thus speaks an employee of yet another Provider-That-Won't-Accept-My- > Third-Party-Credentials. Sorry, Barry, but you see the problem: > Unfortunately, we can't do it alone. What needs to happen is there > needs to be a large network of sites that support login via O-D-F > systems like OpenID and Persona. Too many of the sites I use (news > sources, GMail, etc) don't support them and my browser manages my > logins to most of them, so why bother learning OpenID, and then > setting it up site by site? > [snipped lots of observations that I generally agree with] There's been a lot of negativity towards OpenID in this thread -- I'd like to say that in Fedora Infrastructure we've found OpenID to be very very good -- but not at addressing the problem that most people are after here. As you've observed being an OpenID provider is a relatively easy to swallow proposition; accepting OpenID from third parties is another thing entirely. As you've also observed, this has to do with trust. A site can trust their own account system and practices and issue OpenID based on those. It is much riskier for the site to trust someone else's account system and practices when deciding whether a user is actually the owner of the account that they claim. So OpenID fails as a truly generic SSO method across sites on the internet... what have we found it good for then? SSO within our site. More and more apps support OpenID out of the box. Many web frameworks have modules for the code you write to authenticate against an OpenID server. A site configures these apps and modules to only trust the site's OpenID service and then deploys them with less custom code. Sites also get a choice about how much risk they consider compromised accounts to a particular application. If they run a web forum and a build system for instance, they might constrain the build system to only their OpenID service but allow the forum to allow OpenID from other providers. And finally, having an openid service lets their users sign into more trusting sites like python.org properties (unlike say, LDAP) :-) -Toshio From phd at phdru.name Mon Sep 9 19:46:58 2013 From: phd at phdru.name (Oleg Broytman) Date: Mon, 9 Sep 2013 21:46:58 +0400 Subject: [Python-Dev] Offtopic: OpenID Providers In-Reply-To: <CABVPEKpCgeAH5uLThNCMNLLN6quViRWu2orC_386q09sPiMDyg@mail.gmail.com> References: <8593FDCE-E3E0-4AD4-897E-2FB1D751086C@stufft.io> <20130905182522.GA26090@iskra.aviel.ru> <2C429087-4E06-4F0F-B524-25D0F6FB4FFF@stufft.io> <20130905184337.GA26926@iskra.aviel.ru> <667454FE-47D1-46BB-99B6-4C0B10986269@stufft.io> <CANc-5UxNJQp46dA2b54Ad-rifUGv8OCL1H2Yky1gr8EqYxcABA@mail.gmail.com> <20130905203651.GB30331@iskra.aviel.ru> <20130905165318.2fa40bab@anarchist> <87hadyah89.fsf@uwakimon.sk.tsukuba.ac.jp> <CABVPEKpCgeAH5uLThNCMNLLN6quViRWu2orC_386q09sPiMDyg@mail.gmail.com> Message-ID: <20130909174658.GA8795@iskra.aviel.ru> On Mon, Sep 09, 2013 at 10:39:11AM -0700, Toshio Kuratomi <a.badger at gmail.com> wrote: > So OpenID fails as a truly generic SSO method across sites on the > internet... what have we found it good for then? SSO within our site. I.e., OpenID could be good for core developers (using @python.org email adresses as IDs) but not for general public to login to pydotorg sites, right? Oleg. -- Oleg Broytman http://phdru.name/ phd at phdru.name Programmers don't die, they just GOSUB without RETURN. From solipsis at pitrou.net Mon Sep 9 19:49:59 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 9 Sep 2013 19:49:59 +0200 Subject: [Python-Dev] PEP 447: add type.__locallookup__ References: <75030FAC-6918-4E94-95DA-67A88D53E6F5@mac.com> <7D6D4488-2855-41F8-A9F2-1FF23B9E3A56@mac.com> <522DDB74.5080609@stoneleaf.us> <CAP7+vJJTR=+cRY+iQgHY4ujTsC3_QyWn83dnK96rWO6bV9vFLQ@mail.gmail.com> <522DECA3.2030709@hotpy.org> <522DF1E5.5050309@stoneleaf.us> Message-ID: <20130909194959.1e8115ef@fsol> On Mon, 09 Sep 2013 09:05:57 -0700 Ethan Furman <ethan at stoneleaf.us> wrote: > On 09/09/2013 08:43 AM, Mark Shannon wrote: > > I would like time to investigate this further, but at the moment I think it will either make attribute lookup poorly > > defined or slow. > > > > Of the top of my head, the problem as a I see it is basically this: > > Currently, type.__getattribute__() is a fixed point in the lookup of attributes. > > The proposal means that a fixed point is not reached until the cls parameter of type.__getattribute__() is either object > > or type, > > otherwise type.__getattribute__() and type.__locallookup__ must bounce back and forth. > > > > This will slow down *every* attribute lookup for what is a fairly obscure use case. > > Looks like there's a patch we can try at http://bugs.python.org/issue18181. > > Here are Ronald's last timings: Thanks but can you run a benchmark that actually exercises the feature? (I don't know enough about it to know what that would be, but I suppose it has to do with lookups on classes, rather than on instances) Regards Antoine. From phd at phdru.name Mon Sep 9 20:08:00 2013 From: phd at phdru.name (Oleg Broytman) Date: Mon, 9 Sep 2013 22:08:00 +0400 Subject: [Python-Dev] Offtopic: OpenID Providers In-Reply-To: <20130909174658.GA8795@iskra.aviel.ru> References: <20130905182522.GA26090@iskra.aviel.ru> <2C429087-4E06-4F0F-B524-25D0F6FB4FFF@stufft.io> <20130905184337.GA26926@iskra.aviel.ru> <667454FE-47D1-46BB-99B6-4C0B10986269@stufft.io> <CANc-5UxNJQp46dA2b54Ad-rifUGv8OCL1H2Yky1gr8EqYxcABA@mail.gmail.com> <20130905203651.GB30331@iskra.aviel.ru> <20130905165318.2fa40bab@anarchist> <87hadyah89.fsf@uwakimon.sk.tsukuba.ac.jp> <CABVPEKpCgeAH5uLThNCMNLLN6quViRWu2orC_386q09sPiMDyg@mail.gmail.com> <20130909174658.GA8795@iskra.aviel.ru> Message-ID: <20130909180800.GB8795@iskra.aviel.ru> On Mon, Sep 09, 2013 at 09:46:58PM +0400, Oleg Broytman <phd at phdru.name> wrote: > On Mon, Sep 09, 2013 at 10:39:11AM -0700, Toshio Kuratomi <a.badger at gmail.com> wrote: > > So OpenID fails as a truly generic SSO method across sites on the > > internet... what have we found it good for then? SSO within our site. > > I.e., OpenID could be good for core developers (using @python.org > email adresses as IDs) but not for general public to login to pydotorg > sites, right? Oops, completely messed OpenID URLs and Persona emails. Oleg. -- Oleg Broytman http://phdru.name/ phd at phdru.name Programmers don't die, they just GOSUB without RETURN. From zuo at chopin.edu.pl Mon Sep 9 20:23:40 2013 From: zuo at chopin.edu.pl (Jan Kaliszewski) Date: Mon, 09 Sep 2013 20:23:40 +0200 Subject: [Python-Dev] =?utf-8?q?PEP_447=3A_add_type=2E=5F=5Flocallookup=5F?= =?utf-8?q?=5F?= In-Reply-To: <522DF204.7040109@hotpy.org> References: <75030FAC-6918-4E94-95DA-67A88D53E6F5@mac.com> <7D6D4488-2855-41F8-A9F2-1FF23B9E3A56@mac.com> <522DDB74.5080609@stoneleaf.us> <CAP7+vJJTR=+cRY+iQgHY4ujTsC3_QyWn83dnK96rWO6bV9vFLQ@mail.gmail.com> <522DECA3.2030709@hotpy.org> <CAP7+vJ+thJvYpk0d0rnFO6bm88sTJ_t1QdYFB0SdPOGNDABuag@mail.gmail.com> <522DF204.7040109@hotpy.org> Message-ID: <0752376899dd7436f6115bfc03594574@chopin.edu.pl> Is '__locallookup__' a really good name? In Python, *local* -- especially in context of *lookups* -- usually associates with locals() i.e. a namespace of a function/method execution frame or a namespace of a class, during *definition* of that class... So '__locallookup__' can be confusing. Why not just '__getclassattribute__' or '__classlookup__', or '__classattribute__'...? Cheers. *j From nad at acm.org Mon Sep 9 21:29:27 2013 From: nad at acm.org (Ned Deily) Date: Mon, 09 Sep 2013 12:29:27 -0700 Subject: [Python-Dev] [RELEASED] Python 3.4.0a2 References: <522DB8D3.1030803@hastings.org> Message-ID: <nad-2495B7.12292709092013@news.gmane.org> In article <522DB8D3.1030803 at hastings.org>, Larry Hastings <larry at hastings.org> wrote: > On behalf of the Python development team, I'm chuffed to announce the > second alpha release of Python 3.4. Yay! 3.4.0a2 also contains a new batteries-included feature for OS X users. The python.org 64-bit/32-bit installer now includes its own private version of Tcl/Tk 8.5.14 so it is finally no longer necessary to install a third-party version of Tcl/Tk 8.5 to workaround the problematic system versions shipped in OS X 10.6+. More improvements to come. -- Ned Deily, nad at acm.org From victor.stinner at gmail.com Mon Sep 9 21:51:46 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Mon, 9 Sep 2013 21:51:46 +0200 Subject: [Python-Dev] [python-committers] [RELEASED] Python 3.4.0a2 In-Reply-To: <20130909144551.76b84640@pitrou.net> References: <522DB8D3.1030803@hastings.org> <CAMpsgwZ3Ltsc4dKRnN5zBVvcSF4WFX4YQRcG9oaq_iMvx0NjvQ@mail.gmail.com> <20130909144551.76b84640@pitrou.net> Message-ID: <CAMpsgwZFuAKe+VKCXaXf6aU1wz_3+p3s=0-QMLEwgBEBLgVf0g@mail.gmail.com> 2013/9/9 Antoine Pitrou <solipsis at pitrou.net>: > Le Mon, 9 Sep 2013 14:30:50 +0200, > Victor Stinner <victor.stinner at gmail.com> a ?crit : >> 2013/9/9 Larry Hastings <larry at hastings.org>: >> > Python 3.4 includes a range of improvements of the 3.x series, >> > including hundreds of small improvements and bug fixes. Major new >> > features and changes in the 3.4 release series so far include: >> > >> > * PEP 446, changing file descriptors to not be inherited by default >> > in subprocesses >> >> The title of the PEP is "Make newly created file descriptors >> non-inheritable". It has an impact on all functions creating files and >> sockets not only the subprocess module. > > I don't think Larry's description is wrong. "Non-inheritable" is a > shorthand for "non-inheritable in subprocesses" with "subprocesses" > taken in the general sense (i.e. not only created with the subprocess > module). Oh, I misunderstood "in subprocesses", I read "in the subprocess module". The definition of FD inheritance is tricky. For example, on UNIX "non-inheritable" file descriptors are still inherited at fork :-) I hope that the documentation is explicit enough: http://docs.python.org/dev/library/os.html#inheritance-of-file-descriptors Victor From yselivanov.ml at gmail.com Mon Sep 9 22:01:47 2013 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Mon, 9 Sep 2013 16:01:47 -0400 Subject: [Python-Dev] PEP 447: add type.__locallookup__ In-Reply-To: <0752376899dd7436f6115bfc03594574@chopin.edu.pl> References: <75030FAC-6918-4E94-95DA-67A88D53E6F5@mac.com> <7D6D4488-2855-41F8-A9F2-1FF23B9E3A56@mac.com> <522DDB74.5080609@stoneleaf.us> <CAP7+vJJTR=+cRY+iQgHY4ujTsC3_QyWn83dnK96rWO6bV9vFLQ@mail.gmail.com> <522DECA3.2030709@hotpy.org> <CAP7+vJ+thJvYpk0d0rnFO6bm88sTJ_t1QdYFB0SdPOGNDABuag@mail.gmail.com> <522DF204.7040109@hotpy.org> <0752376899dd7436f6115bfc03594574@chopin.edu.pl> Message-ID: <FA8F15D0-95C8-43D1-ACB6-0499CBE10F95@gmail.com> Yes, I don't like the 'local' prefix too. How about '__dictlookup__'? It's just more self-describing. Yury On 2013-09-09, at 2:23 PM, Jan Kaliszewski <zuo at chopin.edu.pl> wrote: > Is '__locallookup__' a really good name? In Python, *local* -- especially in context of *lookups* -- usually associates with locals() i.e. a namespace of a function/method execution frame or a namespace of a class, during *definition* of that class... So '__locallookup__' can be confusing. > > Why not just '__getclassattribute__' or '__classlookup__', or '__classattribute__'...? > > Cheers. > *j > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/yselivanov.ml%40gmail.com From benjamin at python.org Mon Sep 9 22:16:33 2013 From: benjamin at python.org (Benjamin Peterson) Date: Mon, 9 Sep 2013 16:16:33 -0400 Subject: [Python-Dev] PEP 447: add type.__locallookup__ In-Reply-To: <522DDB74.5080609@stoneleaf.us> References: <75030FAC-6918-4E94-95DA-67A88D53E6F5@mac.com> <7D6D4488-2855-41F8-A9F2-1FF23B9E3A56@mac.com> <522DDB74.5080609@stoneleaf.us> Message-ID: <CAPZV6o8oKepeSgHq13CUHXY8q36tmnAJ5dqEeR2QFO0Q6M81Dg@mail.gmail.com> Since the main problem is super(), maybe we can just add a __super__ method to get a custom super implementation? 2013/9/9 Ethan Furman <ethan at stoneleaf.us>: > On 07/30/2013 11:17 PM, Ronald Oussoren wrote: >> >> >> And something I forgot to ask: is anyone willing to be the BDFL-Delegate >> for >> PEP 447? > > > *Bump*. > > It would be nice if this could make into 3.4. > > -- > ~Ethan~ > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/benjamin%40python.org -- Regards, Benjamin From mark at hotpy.org Mon Sep 9 23:18:38 2013 From: mark at hotpy.org (Mark Shannon) Date: Mon, 09 Sep 2013 22:18:38 +0100 Subject: [Python-Dev] PEP 447: add type.__locallookup__ In-Reply-To: <522DDB74.5080609@stoneleaf.us> References: <75030FAC-6918-4E94-95DA-67A88D53E6F5@mac.com> <7D6D4488-2855-41F8-A9F2-1FF23B9E3A56@mac.com> <522DDB74.5080609@stoneleaf.us> Message-ID: <522E3B2E.6050605@hotpy.org> On 09/09/13 15:30, Ethan Furman wrote: > On 07/30/2013 11:17 PM, Ronald Oussoren wrote: >> >> And something I forgot to ask: is anyone willing to be the >> BDFL-Delegate for >> PEP 447? > > *Bump*. > > It would be nice if this could make into 3.4. > IMO, there are some issues that need to be addressed before PEP 447 should be accepted. 1. Is there even a problem at all, or is this just a bug in super? Why doesn't super() respect the __getattribute__ method of the superclass? 2. Is this the best way to solve the problem (if there is a problem)? Would a __super__ special method be sufficient and less intrusive. 3. Are the proposed semantics OK? I think they are, but very low level changes such as these can have unforeseen consequences. For example, PEP 3135 and issue 12370. 4. What is the performance impact. pybench really doesn't count as a benchmark. 5. Other implementations. What do the Jython/IronPython/PyPy developers think? Cheers, Mark. p.s. Apologies for top-posting earlier From benjamin at python.org Mon Sep 9 23:25:14 2013 From: benjamin at python.org (Benjamin Peterson) Date: Mon, 9 Sep 2013 17:25:14 -0400 Subject: [Python-Dev] PEP 447: add type.__locallookup__ In-Reply-To: <522E3B2E.6050605@hotpy.org> References: <75030FAC-6918-4E94-95DA-67A88D53E6F5@mac.com> <7D6D4488-2855-41F8-A9F2-1FF23B9E3A56@mac.com> <522DDB74.5080609@stoneleaf.us> <522E3B2E.6050605@hotpy.org> Message-ID: <CAPZV6o903HwDnbyOaWm5nTV8xYBHxDvnG0NGkeVOhwrSTBJnig@mail.gmail.com> 2013/9/9 Mark Shannon <mark at hotpy.org>: > On 09/09/13 15:30, Ethan Furman wrote: >> >> On 07/30/2013 11:17 PM, Ronald Oussoren wrote: >>> >>> >>> And something I forgot to ask: is anyone willing to be the >>> BDFL-Delegate for >>> PEP 447? >> >> >> *Bump*. >> >> It would be nice if this could make into 3.4. >> > > IMO, there are some issues that need to be addressed before PEP 447 should > be accepted. > > 1. Is there even a problem at all, or is this just a bug in super? > Why doesn't super() respect the __getattribute__ method of the superclass? You want to be looking things up on the class, not an instance. -- Regards, Benjamin From mark at hotpy.org Tue Sep 10 00:36:15 2013 From: mark at hotpy.org (Mark Shannon) Date: Mon, 09 Sep 2013 23:36:15 +0100 Subject: [Python-Dev] PEP 447: add type.__locallookup__ In-Reply-To: <CAPZV6o903HwDnbyOaWm5nTV8xYBHxDvnG0NGkeVOhwrSTBJnig@mail.gmail.com> References: <75030FAC-6918-4E94-95DA-67A88D53E6F5@mac.com> <7D6D4488-2855-41F8-A9F2-1FF23B9E3A56@mac.com> <522DDB74.5080609@stoneleaf.us> <522E3B2E.6050605@hotpy.org> <CAPZV6o903HwDnbyOaWm5nTV8xYBHxDvnG0NGkeVOhwrSTBJnig@mail.gmail.com> Message-ID: <522E4D5F.4050505@hotpy.org> On 09/09/13 22:25, Benjamin Peterson wrote: > 2013/9/9 Mark Shannon <mark at hotpy.org>: >> On 09/09/13 15:30, Ethan Furman wrote: >>> >>> On 07/30/2013 11:17 PM, Ronald Oussoren wrote: >>>> >>>> >>>> And something I forgot to ask: is anyone willing to be the >>>> BDFL-Delegate for >>>> PEP 447? >>> >>> >>> *Bump*. >>> >>> It would be nice if this could make into 3.4. >>> >> >> IMO, there are some issues that need to be addressed before PEP 447 should >> be accepted. >> >> 1. Is there even a problem at all, or is this just a bug in super? >> Why doesn't super() respect the __getattribute__ method of the superclass? > > You want to be looking things up on the class, not an instance. > Sorry, I meant 'type of the superclass' rather than 'superclass'. I was suggesting that super().m should be type(type(self).__mro__[1]).__getattribute__('m') rather than type(self).__mro__[1].__dict__['m'] (ignoring descriptor __get__ calls) Unfortunately this brings its own problems, due to __getattribute__ doing its own traversal of the mro. So, scratch point 1. Cheers, Mark. From rymg19 at gmail.com Tue Sep 10 02:28:15 2013 From: rymg19 at gmail.com (Ryan) Date: Mon, 09 Sep 2013 19:28:15 -0500 Subject: [Python-Dev] [RELEASED] Python 3.4.0a2 In-Reply-To: <nad-2495B7.12292709092013@news.gmane.org> References: <522DB8D3.1030803@hastings.org> <nad-2495B7.12292709092013@news.gmane.org> Message-ID: <610dd75a-ada8-4ae1-9810-7320cbeda9ce@email.android.com> HALLELUJAH!!! Ned Deily <nad at acm.org> wrote: >In article <522DB8D3.1030803 at hastings.org>, > Larry Hastings <larry at hastings.org> wrote: > >> On behalf of the Python development team, I'm chuffed to announce the >> second alpha release of Python 3.4. > >Yay! 3.4.0a2 also contains a new batteries-included feature for OS X >users. The python.org 64-bit/32-bit installer now includes its own >private version of Tcl/Tk 8.5.14 so it is finally no longer necessary >to >install a third-party version of Tcl/Tk 8.5 to workaround the >problematic system versions shipped in OS X 10.6+. More improvements >to >come. > >-- > Ned Deily, > nad at acm.org > >_______________________________________________ >Python-Dev mailing list >Python-Dev at python.org >https://mail.python.org/mailman/listinfo/python-dev >Unsubscribe: >https://mail.python.org/mailman/options/python-dev/rymg19%40gmail.com -- Sent from my Android phone with K-9 Mail. Please excuse my brevity. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130909/44236030/attachment.html> From stefan_ml at behnel.de Tue Sep 10 07:48:51 2013 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 10 Sep 2013 07:48:51 +0200 Subject: [Python-Dev] [RELEASED] Python 3.4.0a2 In-Reply-To: <nad-2495B7.12292709092013@news.gmane.org> References: <522DB8D3.1030803@hastings.org> <nad-2495B7.12292709092013@news.gmane.org> Message-ID: <l0mbrr$ofo$1@ger.gmane.org> Ned Deily, 09.09.2013 21:29: > 3.4.0a2 also contains a new batteries-included feature for OS X > users. The python.org 64-bit/32-bit installer now includes its own > private version of Tcl/Tk 8.5.14 so it is finally no longer necessary to > install a third-party version of Tcl/Tk 8.5 to workaround the > problematic system versions shipped in OS X 10.6+. More improvements to > come. With MS Windows on steep decline, I'm so happy we have at least MacOS-X rising to keep us busy. Stefan From solipsis at pitrou.net Tue Sep 10 08:08:46 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 10 Sep 2013 08:08:46 +0200 Subject: [Python-Dev] cpython (2.7): Clarify mmap.close method behavior. Addresses issue #18815 References: <3cYw7T6lKwz7LjZ@mail.python.org> Message-ID: <20130910080846.578093a7@fsol> On Tue, 10 Sep 2013 07:40:21 +0200 (CEST) senthil.kumaran <python-checkins at python.org> wrote: > http://hg.python.org/cpython/rev/443d12b61e5b > changeset: 85653:443d12b61e5b > branch: 2.7 > parent: 85639:740bd510a888 > user: Senthil Kumaran <senthil at uthcode.com> > date: Mon Sep 09 22:38:58 2013 -0700 > summary: > Clarify mmap.close method behavior. Addresses issue #18815 > Patch contributed by Anoop Thomas Mathew. > > files: > Doc/library/mmap.rst | 5 +++-- > 1 files changed, 3 insertions(+), 2 deletions(-) > > > diff --git a/Doc/library/mmap.rst b/Doc/library/mmap.rst > --- a/Doc/library/mmap.rst > +++ b/Doc/library/mmap.rst > @@ -152,8 +152,9 @@ > > .. method:: close() > > - Close the file. Subsequent calls to other methods of the object will > - result in an exception being raised. > + Closes the mmap. Subsequent calls to other methods of the object will > + result in a ValueError exception being raised. This will not close > + the open file. I think this could be improved. Anonymous mmap objects don't have an "open file". Also, it would be clearer if it read "the underlying file" rather than "the underlying file object" rather than "the open file", because an mmap object is a file as well (it has read(), write(), seek(), etc.). Regards Antoine. From v+python at g.nevcal.com Tue Sep 10 09:21:28 2013 From: v+python at g.nevcal.com (Glenn Linderman) Date: Tue, 10 Sep 2013 00:21:28 -0700 Subject: [Python-Dev] Offtopic: OpenID Providers In-Reply-To: <l0d311$b7n$1@ger.gmane.org> References: <5228C00F.7000209@jcea.es> <l0d311$b7n$1@ger.gmane.org> Message-ID: <522EC878.6080409@g.nevcal.com> On 9/6/2013 10:22 AM, Dan Callahan wrote: > On 9/5/13 12:31 PM, Jesus Cea wrote: >> I have big hopes for Mozilla Persona, looking forward >> Python infrastructure support :). > > Hi, I'm the project lead on Persona signin, and I spoke at PyCon > earlier this year regarding why and how Mozilla is building Persona. > If you'd like some more background, that video [0] is worth a look. > > Let's pull this discussion up a level: > > It sounds like many people (Jesus, Donald, Toshio, Barry, Tres, > Dirkjan, etc.) are interested in seeing Persona on Python.org > properties, and most of the objections coming from a place of "Persona > hasn't gone viral, what if this is wasted effort?" OK, let's pull this discussion down a level: testing it out. So I tried to login to the crossword.thetimes.co.uk -- I used an email address persona had never seen, it asked me for a password, and sent me a confirmation message, containing a link that I clicked on. However, as I was reading clues and filling in blanks, I got a popup that said "login failure [object Object]". And crossword told me it was saving locally, and to login to save to the server. And the Log in button stayed displayed, rather than a Log out button, which I assume it might get replaced with if I ever get successfully logged in. Firefox 23.0.1, Windows 7 64-bit with autoupdates. Need any other info? Write me privately if you want the email address I used (not the one I use here), or the password. I used new ones, so I can share for testing, and then discard them and use different ones "for real". If the system actually works. Hey, the video demos looked great... Glenn -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130910/04d2b36a/attachment.html> From solipsis at pitrou.net Tue Sep 10 10:10:42 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 10 Sep 2013 10:10:42 +0200 Subject: [Python-Dev] Offtopic: OpenID Providers References: <5228C00F.7000209@jcea.es> <l0d311$b7n$1@ger.gmane.org> <522EC878.6080409@g.nevcal.com> Message-ID: <20130910101042.7c2b66c7@pitrou.net> Ok, can this discussion go off python-dev, please? This has been terribly off-topic for a long time (arguably from the beginning, actually). Thank you Antoine. Le Tue, 10 Sep 2013 00:21:28 -0700, Glenn Linderman <v+python at g.nevcal.com> a ?crit : > On 9/6/2013 10:22 AM, Dan Callahan wrote: > > On 9/5/13 12:31 PM, Jesus Cea wrote: > >> I have big hopes for Mozilla Persona, looking forward > >> Python infrastructure support :). > > > > Hi, I'm the project lead on Persona signin, and I spoke at PyCon > > earlier this year regarding why and how Mozilla is building > > Persona. If you'd like some more background, that video [0] is > > worth a look. > > > > Let's pull this discussion up a level: > > > > It sounds like many people (Jesus, Donald, Toshio, Barry, Tres, > > Dirkjan, etc.) are interested in seeing Persona on Python.org > > properties, and most of the objections coming from a place of > > "Persona hasn't gone viral, what if this is wasted effort?" > > OK, let's pull this discussion down a level: testing it out. > > So I tried to login to the crossword.thetimes.co.uk -- I used an > email address persona had never seen, it asked me for a password, and > sent me a confirmation message, containing a link that I clicked on. > > However, as I was reading clues and filling in blanks, I got a popup > that said "login failure [object Object]". And crossword told me it > was saving locally, and to login to save to the server. And the Log > in button stayed displayed, rather than a Log out button, which I > assume it might get replaced with if I ever get successfully logged > in. > > Firefox 23.0.1, Windows 7 64-bit with autoupdates. Need any other > info? Write me privately if you want the email address I used (not > the one I use here), or the password. I used new ones, so I can share > for testing, and then discard them and use different ones "for real". > If the system actually works. Hey, the video demos looked great... > > Glenn > From solipsis at pitrou.net Tue Sep 10 11:28:32 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 10 Sep 2013 11:28:32 +0200 Subject: [Python-Dev] Add a "transformdict" to collections Message-ID: <20130910112832.75ed6c8b@pitrou.net> Hello, In http://bugs.python.org/issue18986 I proposed adding a new mapping type to the collections module. The original use case is quite common in network programming and elsewhere (Eric Snow on the tracker mentioned an application with stock symbols). You want to have an associative container which matches keys case-insensitively but also preserves the original casing (e.g. for presentation). It is a commonly reimplemented container. It is also an instance of a more general pattern: match keys after applying a derivation (or coercion) function, but at the same time keep track of the original key. Note that the derivation function needn't be (and generally won't be) bijective, otherwise it's too simple. Therefore I propose adding the general pattern. Simple example: >>> d = transformdict(str.lower) >>> d['Foo'] = 5 >>> d['foo'] 5 >>> d['FOO'] 5 >>> list(d) ['Foo'] (case-insensitive but case-preserving, as the best filesystems are ;-)) On the tracker issue, it seems everyone agreed on the principle. There is some bikeshedding left to do, though. So here are the reasonable naming proposals so far: - transformkeydict - coercekeydict - transformdict - coercedict I have a sweet spot for "transformdict" myself. Regards Antoine. From skip at pobox.com Tue Sep 10 11:42:46 2013 From: skip at pobox.com (Skip Montanaro) Date: Tue, 10 Sep 2013 04:42:46 -0500 Subject: [Python-Dev] Add a "transformdict" to collections In-Reply-To: <20130910112832.75ed6c8b@pitrou.net> References: <20130910112832.75ed6c8b@pitrou.net> Message-ID: <CANc-5Uzgd2nwbxN39ykzUv-1YZzg1BEuvVZObQrnrdWd8ofTbA@mail.gmail.com> > (case-insensitive but case-preserving, as the best filesystems are ;-)) > I have a sweet spot for "transformdict" myself. Antoine, "Transform" does not remind me of "case-insensitive but case-preserving". If this is important enough to put into the collections module (I'm skeptical), shouldn't that behavior be more strongly hinted at in the name? Lot's of things are transformations. You're interested in a very specific one. Skip From solipsis at pitrou.net Tue Sep 10 12:00:12 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 10 Sep 2013 12:00:12 +0200 Subject: [Python-Dev] Add a "transformdict" to collections References: <20130910112832.75ed6c8b@pitrou.net> <CANc-5Uzgd2nwbxN39ykzUv-1YZzg1BEuvVZObQrnrdWd8ofTbA@mail.gmail.com> Message-ID: <20130910120012.6628c98b@pitrou.net> Le Tue, 10 Sep 2013 04:42:46 -0500, Skip Montanaro <skip at pobox.com> a ?crit : > > (case-insensitive but case-preserving, as the best filesystems > > are ;-)) > > > I have a sweet spot for "transformdict" myself. > > Antoine, > > "Transform" does not remind me of "case-insensitive but > case-preserving". That was only describing the example, not the proposal. (and I hoped it would be obvious that that sentence was a joke ;-)) Regards Antoine. From victor.stinner at gmail.com Tue Sep 10 12:04:51 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Tue, 10 Sep 2013 12:04:51 +0200 Subject: [Python-Dev] Add a "transformdict" to collections In-Reply-To: <20130910112832.75ed6c8b@pitrou.net> References: <20130910112832.75ed6c8b@pitrou.net> Message-ID: <CAMpsgwbbCOx9tGBkz-Pj5hwJaNP+Dp2UPfov75PksD72+5FVfg@mail.gmail.com> 2013/9/10 Antoine Pitrou <solipsis at pitrou.net>: > In http://bugs.python.org/issue18986 I proposed adding a new mapping > type to the collections module. > > The original use case is quite common in network programming and > elsewhere (Eric Snow on the tracker mentioned an application with stock > symbols). You want to have an associative container which matches keys > case-insensitively but also preserves the original casing (e.g. for > presentation). It is a commonly reimplemented container. If it is commonly reimplemented, what is the most common name? :-) The http.client and email.message modules convert headers to lower case, but keep the original case. > - transformkeydict Do you know a use case where values need also to be transformed? If not, I prefer the transformdict name. > - coercekeydict > - coercedict I only read "coerce" in old Python documentation, not in other languages. I prefer the more common and generic term "tranform". Victor From oscar.j.benjamin at gmail.com Tue Sep 10 12:34:54 2013 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Tue, 10 Sep 2013 11:34:54 +0100 Subject: [Python-Dev] Add a "transformdict" to collections In-Reply-To: <20130910112832.75ed6c8b@pitrou.net> References: <20130910112832.75ed6c8b@pitrou.net> Message-ID: <CAHVvXxQRO-V7002jhVPb5n46yS0b1MeDfNYVbMsVowTZm39NHQ@mail.gmail.com> On 10 September 2013 10:28, Antoine Pitrou <solipsis at pitrou.net> wrote: > On the tracker issue, it seems everyone agreed on the principle. There > is some bikeshedding left to do, though. So here are the reasonable > naming proposals so far: > > - transformkeydict > - coercekeydict > - transformdict > - coercedict > > I have a sweet spot for "transformdict" myself. Could you not use a name with some sense of purpose like approxmatchdict? Oscar From dirkjan at ochtman.nl Tue Sep 10 12:46:33 2013 From: dirkjan at ochtman.nl (Dirkjan Ochtman) Date: Tue, 10 Sep 2013 12:46:33 +0200 Subject: [Python-Dev] Add a "transformdict" to collections In-Reply-To: <20130910112832.75ed6c8b@pitrou.net> References: <20130910112832.75ed6c8b@pitrou.net> Message-ID: <CAKmKYaAYRNNkqOLdi7m72h6_ja+eBeca-fnNa85LLj2ZzXGbfA@mail.gmail.com> On Tue, Sep 10, 2013 at 11:28 AM, Antoine Pitrou <solipsis at pitrou.net> wrote: > On the tracker issue, it seems everyone agreed on the principle. There > is some bikeshedding left to do, though. So here are the reasonable > naming proposals so far: > > - transformkeydict > - coercekeydict > - transformdict > - coercedict > > I have a sweet spot for "transformdict" myself. Given OrderedDict, it seems like this should definitely be TransformDict rather than transformdict. Cheers, Dirkjan From martin at v.loewis.de Tue Sep 10 13:56:56 2013 From: martin at v.loewis.de (=?ISO-8859-15?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 10 Sep 2013 13:56:56 +0200 Subject: [Python-Dev] Offtopic: OpenID Providers In-Reply-To: <5228C00F.7000209@jcea.es> References: <5228C00F.7000209@jcea.es> Message-ID: <522F0908.9030303@v.loewis.de> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Am 05.09.13 19:31, schrieb Jesus Cea: > What are you using?. bugs.python.org admins could share some data? Most users use one of the large services: https://www.google.com 3326 https://login.launchpad.net 335 https://*.myopenid.com 253 https://launchpad.net 23 https://*.id.fedoraproject.org 11 The remaining ones are mostly private URLs. Of those, a majority again delegates to providers, namely (ignoring providers with less than 3 users) www.startssl.com 3 www.clavid.com 3 openid.stackexchange.com 4 openid.claimid.com 4 login.launchpad.net 7 openid.yandex.ru 8 www.google.com 14 pip.verisignlabs.com 20 www.myopenid.com 41 Regards, Martin -----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.18 (Darwin) Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlIvCQgACgkQavBT8H2dyNIX/gCeL9As+Z/aRAggbP+bjwUxFtGo wZYAn2YkEEf/iv+bPGDBEnV+UWZmrf9M =B5pE -----END PGP SIGNATURE----- From ncoghlan at gmail.com Tue Sep 10 14:00:37 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 10 Sep 2013 22:00:37 +1000 Subject: [Python-Dev] Add a "transformdict" to collections In-Reply-To: <CAHVvXxQRO-V7002jhVPb5n46yS0b1MeDfNYVbMsVowTZm39NHQ@mail.gmail.com> References: <20130910112832.75ed6c8b@pitrou.net> <CAHVvXxQRO-V7002jhVPb5n46yS0b1MeDfNYVbMsVowTZm39NHQ@mail.gmail.com> Message-ID: <CADiSq7ec=+GGe7wQREozLyKZTR6pR=yxb71FG1Zv9b3oYWRrqw@mail.gmail.com> Is this just syntactic sugar for recursive lookup of a transformed version in __missing__? Or a way of supplying a custom "key" function to a dictionary? Any such proposal should consider how it composes with other dict variants like defaultdict, OrderedDict and counter. Cheers, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130910/eb002faa/attachment.html> From solipsis at pitrou.net Tue Sep 10 14:22:58 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 10 Sep 2013 14:22:58 +0200 Subject: [Python-Dev] Add a "transformdict" to collections References: <20130910112832.75ed6c8b@pitrou.net> <CAHVvXxQRO-V7002jhVPb5n46yS0b1MeDfNYVbMsVowTZm39NHQ@mail.gmail.com> <CADiSq7ec=+GGe7wQREozLyKZTR6pR=yxb71FG1Zv9b3oYWRrqw@mail.gmail.com> Message-ID: <20130910142258.02c148dd@pitrou.net> Le Tue, 10 Sep 2013 22:00:37 +1000, Nick Coghlan <ncoghlan at gmail.com> a ?crit : > Is this just syntactic sugar for recursive lookup of a transformed > version in __missing__? Nope. For one, it doesn't use __missing__ at all. I think using __missing__ would be... missing the point, because it wouldn't working properly if you have e.g. X != Y such that transform(X) == Y and transform(Y) == X. (a simple example being transform = str.swapcase :-)) > Or a way of supplying a custom "key" function > to a dictionary? Probably, although I'm not entirely sure what you mean by that :-) > Any such proposal should consider how it composes with other dict > variants like defaultdict, OrderedDict and counter. Well, one sticking point here is that those variants don't compose with each other already :-) But, yes, I may make another proposal with composition in mind. Regards Antoine. From p.f.moore at gmail.com Tue Sep 10 14:24:29 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 10 Sep 2013 13:24:29 +0100 Subject: [Python-Dev] Add a "transformdict" to collections In-Reply-To: <CADiSq7ec=+GGe7wQREozLyKZTR6pR=yxb71FG1Zv9b3oYWRrqw@mail.gmail.com> References: <20130910112832.75ed6c8b@pitrou.net> <CAHVvXxQRO-V7002jhVPb5n46yS0b1MeDfNYVbMsVowTZm39NHQ@mail.gmail.com> <CADiSq7ec=+GGe7wQREozLyKZTR6pR=yxb71FG1Zv9b3oYWRrqw@mail.gmail.com> Message-ID: <CACac1F_Hr8U_4n27NvwH1=rN0ggFPx8MqJLbAh=GDXy_VNFLbA@mail.gmail.com> On 10 September 2013 13:00, Nick Coghlan <ncoghlan at gmail.com> wrote: > Is this just syntactic sugar for recursive lookup of a transformed version > in __missing__? Or a way of supplying a custom "key" function to a > dictionary? Not quite, because the dict should preserve the originally entered key. >>> td['FOO'] = 42 >>> td['bar'] = 1 >>> td['foo'] 42 >>> td['BAR'] 1 >>> list(td.keys()) ['FOO', 'bar'] This actually prompts the question, what should the following produce: >>> td['FOO'] = 42 >>> td['foo'] = 32 >>> list(td.keys()) ['FOO'] or ['foo']? Both answers are justifiable. Both are possibly even useful depending on context... Paul From p.f.moore at gmail.com Tue Sep 10 14:25:51 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 10 Sep 2013 13:25:51 +0100 Subject: [Python-Dev] Add a "transformdict" to collections In-Reply-To: <CACac1F_Hr8U_4n27NvwH1=rN0ggFPx8MqJLbAh=GDXy_VNFLbA@mail.gmail.com> References: <20130910112832.75ed6c8b@pitrou.net> <CAHVvXxQRO-V7002jhVPb5n46yS0b1MeDfNYVbMsVowTZm39NHQ@mail.gmail.com> <CADiSq7ec=+GGe7wQREozLyKZTR6pR=yxb71FG1Zv9b3oYWRrqw@mail.gmail.com> <CACac1F_Hr8U_4n27NvwH1=rN0ggFPx8MqJLbAh=GDXy_VNFLbA@mail.gmail.com> Message-ID: <CACac1F9-OhKM-m+Vh1ZMz2a=NXRXNYMtA-FQCmOihxBdL_T34Q@mail.gmail.com> On 10 September 2013 13:24, Paul Moore <p.f.moore at gmail.com> wrote: > On 10 September 2013 13:00, Nick Coghlan <ncoghlan at gmail.com> wrote: >> Is this just syntactic sugar for recursive lookup of a transformed version >> in __missing__? Or a way of supplying a custom "key" function to a >> dictionary? > > Not quite, because the dict should preserve the originally entered key. > >>>> td['FOO'] = 42 >>>> td['bar'] = 1 >>>> td['foo'] > 42 >>>> td['BAR'] > 1 >>>> list(td.keys()) > ['FOO', 'bar'] > > This actually prompts the question, what should the following produce: > >>>> td['FOO'] = 42 >>>> td['foo'] = 32 >>>> list(td.keys()) > > ['FOO'] or ['foo']? Both answers are justifiable. Both are possibly > even useful depending on context... I'm using the case where the transform is str.lower here - sorry I wasn't explicit. Paul From solipsis at pitrou.net Tue Sep 10 14:35:58 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 10 Sep 2013 14:35:58 +0200 Subject: [Python-Dev] Add a "transformdict" to collections References: <20130910112832.75ed6c8b@pitrou.net> <CAHVvXxQRO-V7002jhVPb5n46yS0b1MeDfNYVbMsVowTZm39NHQ@mail.gmail.com> <CADiSq7ec=+GGe7wQREozLyKZTR6pR=yxb71FG1Zv9b3oYWRrqw@mail.gmail.com> <CACac1F_Hr8U_4n27NvwH1=rN0ggFPx8MqJLbAh=GDXy_VNFLbA@mail.gmail.com> Message-ID: <20130910143558.5677b55a@pitrou.net> Le Tue, 10 Sep 2013 13:24:29 +0100, Paul Moore <p.f.moore at gmail.com> a ?crit : > On 10 September 2013 13:00, Nick Coghlan <ncoghlan at gmail.com> wrote: > > Is this just syntactic sugar for recursive lookup of a transformed > > version in __missing__? Or a way of supplying a custom "key" > > function to a dictionary? > > Not quite, because the dict should preserve the originally entered > key. > > >>> td['FOO'] = 42 > >>> td['bar'] = 1 > >>> td['foo'] > 42 > >>> td['BAR'] > 1 > >>> list(td.keys()) > ['FOO', 'bar'] > > This actually prompts the question, what should the following produce: > > >>> td['FOO'] = 42 > >>> td['foo'] = 32 > >>> list(td.keys()) > > ['FOO'] or ['foo']? Both answers are justifiable. Both are possibly > even useful depending on context... I think it would be best to leave it as an implementation detail, because whichever is easiest to implement depends on the exact implementation choices (e.g. C vs. Python). (my prototypes right now conserve the last __setitem__ key, not the first) Regards Antoine. From martin at v.loewis.de Tue Sep 10 14:44:01 2013 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Tue, 10 Sep 2013 14:44:01 +0200 Subject: [Python-Dev] Add a "transformdict" to collections In-Reply-To: <20130910143558.5677b55a@pitrou.net> References: <20130910112832.75ed6c8b@pitrou.net> <CAHVvXxQRO-V7002jhVPb5n46yS0b1MeDfNYVbMsVowTZm39NHQ@mail.gmail.com> <CADiSq7ec=+GGe7wQREozLyKZTR6pR=yxb71FG1Zv9b3oYWRrqw@mail.gmail.com> <CACac1F_Hr8U_4n27NvwH1=rN0ggFPx8MqJLbAh=GDXy_VNFLbA@mail.gmail.com> <20130910143558.5677b55a@pitrou.net> Message-ID: <522F1411.8010109@v.loewis.de> Am 10.09.13 14:35, schrieb Antoine Pitrou: >> ['FOO'] or ['foo']? Both answers are justifiable. Both are possibly >> even useful depending on context... > > I think it would be best to leave it as an implementation detail, > because whichever is easiest to implement depends on the exact > implementation choices (e.g. C vs. Python). I think this is the point where the datatype is *not* clearly straight-forward, and thus deserves a PEP. I think it would be a flaw to have this detail implementation-defined. This would be like saying that it is implementation-defined which of A,B,C is returned from "A and B and C" if all are true. Regards, Martin From hrvoje.niksic at avl.com Tue Sep 10 15:09:56 2013 From: hrvoje.niksic at avl.com (Hrvoje Niksic) Date: Tue, 10 Sep 2013 15:09:56 +0200 Subject: [Python-Dev] Add a "transformdict" to collections In-Reply-To: <CACac1F_Hr8U_4n27NvwH1=rN0ggFPx8MqJLbAh=GDXy_VNFLbA@mail.gmail.com> References: <20130910112832.75ed6c8b@pitrou.net> <CAHVvXxQRO-V7002jhVPb5n46yS0b1MeDfNYVbMsVowTZm39NHQ@mail.gmail.com> <CADiSq7ec=+GGe7wQREozLyKZTR6pR=yxb71FG1Zv9b3oYWRrqw@mail.gmail.com> <CACac1F_Hr8U_4n27NvwH1=rN0ggFPx8MqJLbAh=GDXy_VNFLbA@mail.gmail.com> Message-ID: <522F1A24.5030808@avl.com> On 09/10/2013 02:24 PM, Paul Moore wrote: >>>> td['FOO'] = 42 >>>> td['foo'] = 32 >>>> list(td.keys()) > > ['FOO'] or ['foo']? Both answers are justifiable. Note that the same question can be reasonably asked for dict itself: >>> d = {} >>> d[1.0] = 'foo' >>> d[1] = 'bar' >>> d {1.0: 'bar'} So, dict.__setitem__ only replaces the value, leaving the original key in place. transformdict should probably do the same, returning 'FOO' in your example. From shibturn at gmail.com Tue Sep 10 15:42:39 2013 From: shibturn at gmail.com (Richard Oudkerk) Date: Tue, 10 Sep 2013 14:42:39 +0100 Subject: [Python-Dev] Add a "transformdict" to collections In-Reply-To: <20130910112832.75ed6c8b@pitrou.net> References: <20130910112832.75ed6c8b@pitrou.net> Message-ID: <522F21CF.4060609@gmail.com> On 10/09/2013 10:28am, Antoine Pitrou wrote: > Therefore I propose adding the general pattern. Simple example: > > >>> d = transformdict(str.lower) > >>> d['Foo'] = 5 > >>> d['foo'] > 5 > >>> d['FOO'] > 5 > >>> list(d) > ['Foo'] I guess another example is creating an "identity dict" (see http://code.activestate.com/lists/python-ideas/7161/) by doing d = transformdict(id) -- Richard From barry at python.org Tue Sep 10 15:49:28 2013 From: barry at python.org (Barry Warsaw) Date: Tue, 10 Sep 2013 09:49:28 -0400 Subject: [Python-Dev] Add a "transformdict" to collections In-Reply-To: <CAMpsgwbbCOx9tGBkz-Pj5hwJaNP+Dp2UPfov75PksD72+5FVfg@mail.gmail.com> References: <20130910112832.75ed6c8b@pitrou.net> <CAMpsgwbbCOx9tGBkz-Pj5hwJaNP+Dp2UPfov75PksD72+5FVfg@mail.gmail.com> Message-ID: <20130910094928.1cad3f53@anarchist> On Sep 10, 2013, at 12:04 PM, Victor Stinner wrote: >The http.client and email.message modules convert headers to lower >case, but keep the original case. As RDM pointed out on the tracker, email headers aren't a great use case for this because they aren't really dictionaries. They're lists with some dict-like syntax. -Barry From solipsis at pitrou.net Tue Sep 10 15:57:40 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 10 Sep 2013 15:57:40 +0200 Subject: [Python-Dev] Add a "transformdict" to collections References: <20130910112832.75ed6c8b@pitrou.net> <CAMpsgwbbCOx9tGBkz-Pj5hwJaNP+Dp2UPfov75PksD72+5FVfg@mail.gmail.com> <20130910094928.1cad3f53@anarchist> Message-ID: <20130910155740.1aeab97a@pitrou.net> Le Tue, 10 Sep 2013 09:49:28 -0400, Barry Warsaw <barry at python.org> a ?crit : > On Sep 10, 2013, at 12:04 PM, Victor Stinner wrote: > > >The http.client and email.message modules convert headers to lower > >case, but keep the original case. > > As RDM pointed out on the tracker, email headers aren't a great use > case for this because they aren't really dictionaries. They're lists > with some dict-like syntax. defaultdict(list)? From barry at python.org Tue Sep 10 16:06:00 2013 From: barry at python.org (Barry Warsaw) Date: Tue, 10 Sep 2013 10:06:00 -0400 Subject: [Python-Dev] Add a "transformdict" to collections In-Reply-To: <20130910155740.1aeab97a@pitrou.net> References: <20130910112832.75ed6c8b@pitrou.net> <CAMpsgwbbCOx9tGBkz-Pj5hwJaNP+Dp2UPfov75PksD72+5FVfg@mail.gmail.com> <20130910094928.1cad3f53@anarchist> <20130910155740.1aeab97a@pitrou.net> Message-ID: <20130910100600.0078b29a@anarchist> On Sep 10, 2013, at 03:57 PM, Antoine Pitrou wrote: >Le Tue, 10 Sep 2013 09:49:28 -0400, >Barry Warsaw <barry at python.org> a ?crit : >> On Sep 10, 2013, at 12:04 PM, Victor Stinner wrote: >> >> >The http.client and email.message modules convert headers to lower >> >case, but keep the original case. >> >> As RDM pointed out on the tracker, email headers aren't a great use >> case for this because they aren't really dictionaries. They're lists >> with some dict-like syntax. > >defaultdict(list)? Not really. Email headers support duplicates, although the dict-syntax will only return the first such header. Alternative syntax like .get_all() gives you all of them. But anyway, don't let this stop you! I like your robots-in-disguise[1] dicts no matter what you call them, and even if email can't use them. -Barry [1] http://www.youtube.com/watch?v=59QeKkjishs From arigo at tunes.org Tue Sep 10 16:09:16 2013 From: arigo at tunes.org (Armin Rigo) Date: Tue, 10 Sep 2013 16:09:16 +0200 Subject: [Python-Dev] PEP 447: add type.__locallookup__ Message-ID: <CAMSv6X205ONcjSZx75pyCErc7E6UeXAJ51BU-1hgEQKueNX1iw@mail.gmail.com> Hi Mark, On Mon, Sep 9, 2013 at 11:18 PM, Mark Shannon <mark at hotpy.org> wrote: > 5. Other implementations. What do the Jython/IronPython/PyPy developers > think? Thanks for asking :-) I'm fine with staying out of language design issues like this one, and I believe it's the general concensus in PyPy. Whatever gets decided will probably be easy to port to PyPy and have no measurable performance impact: the extra checks on the type, if any, are all constant-folded by the JIT. A bient?t, Armin. From arigo at tunes.org Tue Sep 10 16:15:56 2013 From: arigo at tunes.org (Armin Rigo) Date: Tue, 10 Sep 2013 16:15:56 +0200 Subject: [Python-Dev] Add a "transformdict" to collections In-Reply-To: <522F21CF.4060609@gmail.com> References: <20130910112832.75ed6c8b@pitrou.net> <522F21CF.4060609@gmail.com> Message-ID: <CAMSv6X2MmJwg2E-kom7VF2_+U3btqRQfdEj1ioxg3AZAi5T61g@mail.gmail.com> Hi Richard, On Tue, Sep 10, 2013 at 3:42 PM, Richard Oudkerk <shibturn at gmail.com> wrote: > I guess another example is creating an "identity dict" (see > http://code.activestate.com/lists/python-ideas/7161/) by doing > > d = transformdict(id) This is bogus, because only the id will be stored, and the original key object will be forgotten (and its id probably reused). A bient?t, Armin. From eliben at gmail.com Tue Sep 10 16:18:41 2013 From: eliben at gmail.com (Eli Bendersky) Date: Tue, 10 Sep 2013 07:18:41 -0700 Subject: [Python-Dev] Add a "transformdict" to collections In-Reply-To: <20130910142258.02c148dd@pitrou.net> References: <20130910112832.75ed6c8b@pitrou.net> <CAHVvXxQRO-V7002jhVPb5n46yS0b1MeDfNYVbMsVowTZm39NHQ@mail.gmail.com> <CADiSq7ec=+GGe7wQREozLyKZTR6pR=yxb71FG1Zv9b3oYWRrqw@mail.gmail.com> <20130910142258.02c148dd@pitrou.net> Message-ID: <CAF-Rda9j0T0O=SmpOn0FD4CzrRi5qtkLHfqsadTQ=8q5MWpseQ@mail.gmail.com> On Tue, Sep 10, 2013 at 5:22 AM, Antoine Pitrou <solipsis at pitrou.net> wrote: > Le Tue, 10 Sep 2013 22:00:37 +1000, > Nick Coghlan <ncoghlan at gmail.com> a ?crit : > > Is this just syntactic sugar for recursive lookup of a transformed > > version in __missing__? > > Nope. For one, it doesn't use __missing__ at all. I think > using __missing__ would be... missing the point, because it wouldn't > working properly if you have e.g. X != Y such that transform(X) == Y > and transform(Y) == X. > > (a simple example being transform = str.swapcase :-)) > > > Or a way of supplying a custom "key" function > > to a dictionary? > > Probably, although I'm not entirely sure what you mean by that :-) > > > Any such proposal should consider how it composes with other dict > > variants like defaultdict, OrderedDict and counter. > > Well, one sticking point here is that those variants don't compose with > each other already :-) > But, yes, I may make another proposal with composition in mind. > Does anyone have an idea how to make the existing variants more composable? It would be nice to get a better understanding of this before we add another variant. Conceptually, composability makes a lot of sense (what if we want this transformdict but also in insertion order...) Eli -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130910/eb84e9b3/attachment.html> From solipsis at pitrou.net Tue Sep 10 16:37:19 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 10 Sep 2013 16:37:19 +0200 Subject: [Python-Dev] Add a "transformdict" to collections References: <20130910112832.75ed6c8b@pitrou.net> <CAHVvXxQRO-V7002jhVPb5n46yS0b1MeDfNYVbMsVowTZm39NHQ@mail.gmail.com> <CADiSq7ec=+GGe7wQREozLyKZTR6pR=yxb71FG1Zv9b3oYWRrqw@mail.gmail.com> <20130910142258.02c148dd@pitrou.net> <CAF-Rda9j0T0O=SmpOn0FD4CzrRi5qtkLHfqsadTQ=8q5MWpseQ@mail.gmail.com> Message-ID: <20130910163719.0c07be69@pitrou.net> Le Tue, 10 Sep 2013 07:18:41 -0700, Eli Bendersky <eliben at gmail.com> a ?crit : > On Tue, Sep 10, 2013 at 5:22 AM, Antoine Pitrou <solipsis at pitrou.net> > wrote: > > > Le Tue, 10 Sep 2013 22:00:37 +1000, > > Nick Coghlan <ncoghlan at gmail.com> a ?crit : > > > Is this just syntactic sugar for recursive lookup of a transformed > > > version in __missing__? > > > > Nope. For one, it doesn't use __missing__ at all. I think > > using __missing__ would be... missing the point, because it wouldn't > > working properly if you have e.g. X != Y such that transform(X) == Y > > and transform(Y) == X. > > > > (a simple example being transform = str.swapcase :-)) > > > > > Or a way of supplying a custom "key" function > > > to a dictionary? > > > > Probably, although I'm not entirely sure what you mean by that :-) > > > > > Any such proposal should consider how it composes with other dict > > > variants like defaultdict, OrderedDict and counter. > > > > Well, one sticking point here is that those variants don't compose > > with each other already :-) > > But, yes, I may make another proposal with composition in mind. > > > > Does anyone have an idea how to make the existing variants more > composable? It would be nice to get a better understanding of this > before we add another variant. Conceptually, composability makes a > lot of sense (what if we want this transformdict but also in > insertion order...) AFAICT, the main problem with composability is that the implementation is less simple and it removes many opportunities for optimization. It may or may not be a problem, depending on your use cases. (I'm playing a bit with a TransformMap which is a composable version of transformdict, and it's clear the implementation is less amenable to optimization tweaks and shortcuts: in fact, I have to *add* some logic to e.g. cater with defaultdict) Regards Antoine. From solipsis at pitrou.net Tue Sep 10 16:37:57 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 10 Sep 2013 16:37:57 +0200 Subject: [Python-Dev] Add a "transformdict" to collections References: <20130910112832.75ed6c8b@pitrou.net> <522F21CF.4060609@gmail.com> <CAMSv6X2MmJwg2E-kom7VF2_+U3btqRQfdEj1ioxg3AZAi5T61g@mail.gmail.com> Message-ID: <20130910163757.30288824@pitrou.net> Le Tue, 10 Sep 2013 16:15:56 +0200, Armin Rigo <arigo at tunes.org> a ?crit : > Hi Richard, > > On Tue, Sep 10, 2013 at 3:42 PM, Richard Oudkerk <shibturn at gmail.com> > wrote: > > I guess another example is creating an "identity dict" (see > > http://code.activestate.com/lists/python-ideas/7161/) by doing > > > > d = transformdict(id) > > This is bogus, because only the id will be stored, and the original > key object will be forgotten (and its id probably reused). No, the original key is kept as well. Regards Antoine. From ethan at stoneleaf.us Tue Sep 10 16:00:17 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 10 Sep 2013 07:00:17 -0700 Subject: [Python-Dev] Add a "transformdict" to collections In-Reply-To: <522F1A24.5030808@avl.com> References: <20130910112832.75ed6c8b@pitrou.net> <CAHVvXxQRO-V7002jhVPb5n46yS0b1MeDfNYVbMsVowTZm39NHQ@mail.gmail.com> <CADiSq7ec=+GGe7wQREozLyKZTR6pR=yxb71FG1Zv9b3oYWRrqw@mail.gmail.com> <CACac1F_Hr8U_4n27NvwH1=rN0ggFPx8MqJLbAh=GDXy_VNFLbA@mail.gmail.com> <522F1A24.5030808@avl.com> Message-ID: <522F25F1.4030104@stoneleaf.us> On 09/10/2013 06:09 AM, Hrvoje Niksic wrote: > On 09/10/2013 02:24 PM, Paul Moore wrote: >>>>> td['FOO'] = 42 >>>>> td['foo'] = 32 >>>>> list(td.keys()) >> >> ['FOO'] or ['foo']? Both answers are justifiable. > > Note that the same question can be reasonably asked for dict itself: > >>>> d = {} >>>> d[1.0] = 'foo' >>>> d[1] = 'bar' >>>> d > {1.0: 'bar'} > > So, dict.__setitem__ only replaces the value, leaving the original key in place. transformdict should probably do the > same, returning 'FOO' in your example. +1 From solipsis at pitrou.net Tue Sep 10 16:54:38 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 10 Sep 2013 16:54:38 +0200 Subject: [Python-Dev] Add a "transformdict" to collections References: <20130910112832.75ed6c8b@pitrou.net> <CAHVvXxQRO-V7002jhVPb5n46yS0b1MeDfNYVbMsVowTZm39NHQ@mail.gmail.com> <CADiSq7ec=+GGe7wQREozLyKZTR6pR=yxb71FG1Zv9b3oYWRrqw@mail.gmail.com> <CACac1F_Hr8U_4n27NvwH1=rN0ggFPx8MqJLbAh=GDXy_VNFLbA@mail.gmail.com> <522F1A24.5030808@avl.com> Message-ID: <20130910165438.297d71bc@pitrou.net> Le Tue, 10 Sep 2013 15:09:56 +0200, Hrvoje Niksic <hrvoje.niksic at avl.com> a ?crit : > On 09/10/2013 02:24 PM, Paul Moore wrote: > >>>> td['FOO'] = 42 > >>>> td['foo'] = 32 > >>>> list(td.keys()) > > > > ['FOO'] or ['foo']? Both answers are justifiable. > > Note that the same question can be reasonably asked for dict itself: > > >>> d = {} > >>> d[1.0] = 'foo' > >>> d[1] = 'bar' > >>> d > {1.0: 'bar'} > > So, dict.__setitem__ only replaces the value, leaving the original > key in place. transformdict should probably do the same, returning > 'FOO' in your example. It's not that obvious. It's not common to rely on that aspect of dict semantics, because you will usually lookup using the exact same type, not a compatible one. I would expect very little code, if any, to rely on this. (also, I would intuitively expect the latest key to be held, not the first one, since that's what happens for values.) Regards Antoine. From corbellini.andrea at gmail.com Tue Sep 10 11:37:01 2013 From: corbellini.andrea at gmail.com (Andrea Corbellini) Date: Tue, 10 Sep 2013 11:37:01 +0200 Subject: [Python-Dev] Add a "transformdict" to collections In-Reply-To: <20130910112832.75ed6c8b@pitrou.net> References: <20130910112832.75ed6c8b@pitrou.net> Message-ID: <CAD8LpNACA_XS3aOW7N86XT+TKVTRJWg0o_2vK7XLe3gXi3997Q@mail.gmail.com> On Tue, Sep 10, 2013 at 11:28 AM, Antoine Pitrou <solipsis at pitrou.net> wrote: > Therefore I propose adding the general pattern. Simple example: > > >>> d = transformdict(str.lower) > >>> d['Foo'] = 5 > >>> d['foo'] > 5 > >>> d['FOO'] > 5 > >>> list(d) > ['Foo'] > > (case-insensitive but case-preserving, as the best filesystems are ;-)) Just a nitpick: if this example (or a similar one) gets into the documentation, it would be better to replace str.lower() with str.casefold(). From shibturn at gmail.com Tue Sep 10 15:42:39 2013 From: shibturn at gmail.com (Richard Oudkerk) Date: Tue, 10 Sep 2013 14:42:39 +0100 Subject: [Python-Dev] Add a "transformdict" to collections In-Reply-To: <20130910112832.75ed6c8b@pitrou.net> References: <20130910112832.75ed6c8b@pitrou.net> Message-ID: <522F21CF.4060609@gmail.com> On 10/09/2013 10:28am, Antoine Pitrou wrote: > Therefore I propose adding the general pattern. Simple example: > > >>> d = transformdict(str.lower) > >>> d['Foo'] = 5 > >>> d['foo'] > 5 > >>> d['FOO'] > 5 > >>> list(d) > ['Foo'] I guess another example is creating an "identity dict" (see http://code.activestate.com/lists/python-ideas/7161/) by doing d = transformdict(id) -- Richard From shibturn at gmail.com Tue Sep 10 16:40:55 2013 From: shibturn at gmail.com (Richard Oudkerk) Date: Tue, 10 Sep 2013 15:40:55 +0100 Subject: [Python-Dev] Add a "transformdict" to collections In-Reply-To: <CAMSv6X2MmJwg2E-kom7VF2_+U3btqRQfdEj1ioxg3AZAi5T61g@mail.gmail.com> References: <20130910112832.75ed6c8b@pitrou.net> <522F21CF.4060609@gmail.com> <CAMSv6X2MmJwg2E-kom7VF2_+U3btqRQfdEj1ioxg3AZAi5T61g@mail.gmail.com> Message-ID: <522F2F77.9020805@gmail.com> On 10/09/2013 3:15pm, Armin Rigo wrote: > Hi Richard, > > On Tue, Sep 10, 2013 at 3:42 PM, Richard Oudkerk <shibturn at gmail.com> wrote: >> I guess another example is creating an "identity dict" (see >> http://code.activestate.com/lists/python-ideas/7161/) by doing >> >> d = transformdict(id) > > This is bogus, because only the id will be stored, and the original > key object will be forgotten (and its id probably reused). Seems to work for me: >>> import collections >>> d = collections.transformdict(id) >>> L = [1,2,3] >>> d[L] = None >>> L in d True >>> [1,2,3] in d False >>> print(d[L]) None >>> d._data {41444136: ([1, 2, 3], None)} >>> list(d) [[1, 2, 3]] However __repr__() is broken: >>> d Traceback (most recent call last): File "<stdin>", line 1, in <module> File "C:\Repos\cpython-dirty\lib\collections\abc.py", line 444, in __repr__ return '{0.__class__.__name__}({0._mapping!r})'.format(self) File "C:\Repos\cpython-dirty\lib\collections\__init__.py", line 944, in __repr__ self._transform, repr(dict(self))) TypeError: unhashable type: 'list' -- Richard From zhangpeipei812 at outlook.com Tue Sep 10 12:58:32 2013 From: zhangpeipei812 at outlook.com (=?gb2312?B?1cXF5cXl?=) Date: Tue, 10 Sep 2013 18:58:32 +0800 Subject: [Python-Dev] IOCP (I/O Completion Port) in Python ? Message-ID: <BLU402-EAS15259F82C4E6C2CEBFC83789D380@phx.gbl> Hello: I wondering why there is no standard IOCP module in Python ? As I know: Python3 have support epoll in linux and kqueue in freebsd. Is there a plan to add IOCP module for Windows ? Regards! peipei From guido at python.org Tue Sep 10 17:08:20 2013 From: guido at python.org (Guido van Rossum) Date: Tue, 10 Sep 2013 08:08:20 -0700 Subject: [Python-Dev] Offtopic: OpenID Providers In-Reply-To: <522EC878.6080409@g.nevcal.com> References: <5228C00F.7000209@jcea.es> <l0d311$b7n$1@ger.gmane.org> <522EC878.6080409@g.nevcal.com> Message-ID: <CAP7+vJJ0oG+QFZQKc4fLvOHwiu315WVoozv45dBdOqv59fw=Gw@mail.gmail.com> Why do several posts in this thread have an Unsubscribe link that tries to unsubscribe me from the list? (I saw one by Glen, and another one by Donald Stufft.) (Come to think of it, what's the point of having an Unbub link in ever message that goes out?) On Tue, Sep 10, 2013 at 12:21 AM, Glenn Linderman <v+python at g.nevcal.com>wrote: > On 9/6/2013 10:22 AM, Dan Callahan wrote: > > On 9/5/13 12:31 PM, Jesus Cea wrote: > > I have big hopes for Mozilla Persona, looking forward > Python infrastructure support :). > > > Hi, I'm the project lead on Persona signin, and I spoke at PyCon earlier > this year regarding why and how Mozilla is building Persona. If you'd like > some more background, that video [0] is worth a look. > > Let's pull this discussion up a level: > > It sounds like many people (Jesus, Donald, Toshio, Barry, Tres, Dirkjan, > etc.) are interested in seeing Persona on Python.org properties, and most > of the objections coming from a place of "Persona hasn't gone viral, what > if this is wasted effort?" > > > OK, let's pull this discussion down a level: testing it out. > > So I tried to login to the crossword.thetimes.co.uk -- I used an email > address persona had never seen, it asked me for a password, and sent me a > confirmation message, containing a link that I clicked on. > > However, as I was reading clues and filling in blanks, I got a popup that > said "login failure [object Object]". And crossword told me it was saving > locally, and to login to save to the server. And the Log in button stayed > displayed, rather than a Log out button, which I assume it might get > replaced with if I ever get successfully logged in. > > Firefox 23.0.1, Windows 7 64-bit with autoupdates. Need any other info? > Write me privately if you want the email address I used (not the one I use > here), or the password. I used new ones, so I can share for testing, and > then discard them and use different ones "for real". If the system actually > works. Hey, the video demos looked great... > > Glenn > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130910/f2b0491e/attachment-0001.html> From guido at python.org Tue Sep 10 17:11:05 2013 From: guido at python.org (Guido van Rossum) Date: Tue, 10 Sep 2013 08:11:05 -0700 Subject: [Python-Dev] IOCP (I/O Completion Port) in Python ? In-Reply-To: <BLU402-EAS15259F82C4E6C2CEBFC83789D380@phx.gbl> References: <BLU402-EAS15259F82C4E6C2CEBFC83789D380@phx.gbl> Message-ID: <CAP7+vJLrMFY5kZjBR4hR5nZo+pWapTeWfjKKewBhWqJO9BLzrQ@mail.gmail.com> One will come with Tulip (http://code.google.com/p/tulip/), assuming we can get PEP 3156 accepted at least as "experimental" and the core Tulip code in the stdlib. At the moment the IOCP code there is pretty specialized for use with Tulip -- but if you want to help extracting the IOCP code from there for inclusion into the stdlib that would be great! On Tue, Sep 10, 2013 at 3:58 AM, ??? <zhangpeipei812 at outlook.com> wrote: > Hello: > I wondering why there is no standard IOCP module in Python ? > As I know: Python3 have support epoll in linux and kqueue in freebsd. > Is there a plan to add IOCP module for Windows ? > > Regards! > peipei > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130910/15ddf52a/attachment.html> From v+python at g.nevcal.com Tue Sep 10 17:29:52 2013 From: v+python at g.nevcal.com (Glenn Linderman) Date: Tue, 10 Sep 2013 08:29:52 -0700 Subject: [Python-Dev] Offtopic: OpenID Providers In-Reply-To: <CAP7+vJJ0oG+QFZQKc4fLvOHwiu315WVoozv45dBdOqv59fw=Gw@mail.gmail.com> References: <5228C00F.7000209@jcea.es> <l0d311$b7n$1@ger.gmane.org> <522EC878.6080409@g.nevcal.com> <CAP7+vJJ0oG+QFZQKc4fLvOHwiu315WVoozv45dBdOqv59fw=Gw@mail.gmail.com> Message-ID: <522F3AF0.8020300@g.nevcal.com> On 9/10/2013 8:08 AM, Guido van Rossum wrote: > Why do several posts in this thread have an Unsubscribe link that > tries to unsubscribe me from the list? (I saw one by Glen, and another > one by Donald Stufft.) Seems to be in all of them. Probably added by the mailing list software. Why don't you always see it? Possibly a combination of being directly listed as a recipient, so that you get a copy not sent through the mailing list, together with an email client that suppresses the duplicates (if the mailing list software doesn't change the Message-Id:). But since heretofore you haven't participated in this thread, if you don't see the Unsub link in all of this thread, I doubt this is the explanation. You message to which I'm replying is the first in this thread that came directly to me and which doesn't have the link, because you addressed me directly. Use Reply-List rather than Reply-All if your MUA supports that. > (Come to think of it, what's the point of having an Unbub link in ever > message that goes out?) To avoid people that accidentally subscribe from asking how to unsubscribe, perhaps. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130910/059114e6/attachment.html> From barry at python.org Tue Sep 10 17:41:34 2013 From: barry at python.org (Barry Warsaw) Date: Tue, 10 Sep 2013 11:41:34 -0400 Subject: [Python-Dev] Offtopic: OpenID Providers In-Reply-To: <CAP7+vJJ0oG+QFZQKc4fLvOHwiu315WVoozv45dBdOqv59fw=Gw@mail.gmail.com> References: <5228C00F.7000209@jcea.es> <l0d311$b7n$1@ger.gmane.org> <522EC878.6080409@g.nevcal.com> <CAP7+vJJ0oG+QFZQKc4fLvOHwiu315WVoozv45dBdOqv59fw=Gw@mail.gmail.com> Message-ID: <20130910114134.1015365d@anarchist> On Sep 10, 2013, at 08:08 AM, Guido van Rossum wrote: >Why do several posts in this thread have an Unsubscribe link that tries to >unsubscribe me from the list? (I saw one by Glen, and another one by Donald >Stufft.) This is way off topic, but I suspect your original response didn't trim your little unsub footer and they didn't trim it from their responses. Looking at my list copy of *this* message, I see my own unsub footer, but my MUA automatically trims it in my response buffer. OTOH, I always try to trim my responses anyway, which I think is good netiquette, and which is easy for me with Emacs as my edit/composer of messages. >(Come to think of it, what's the point of having an Unbub link in ever >message that goes out?) As a general Mailman feature, it decreases the likelihood that laypeople who want off a mailing list will do bad things to unsub because they can't figure out how to do it, like complain back to the list or the admins, spam block, or worse, report the list as spam. All of which and more we've seen in the wild. Adding the footer with the unsub block doesn't eliminate this, because some people are lazy, stressed, mean, or frustrated, but it does reduce the incidences. Now, whether it's appropriate for a highly technical list like python-dev is up for discussion (but maybe not here?). While messages are personalized on this list (meaning, you get a unique copy of it for bounce tracking, and yes unsub personalization), those unsub stanzas cannot currently be disabled on a per-user basis. -Barry From ethan at stoneleaf.us Tue Sep 10 16:59:22 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 10 Sep 2013 07:59:22 -0700 Subject: [Python-Dev] Add a "transformdict" to collections In-Reply-To: <20130910165438.297d71bc@pitrou.net> References: <20130910112832.75ed6c8b@pitrou.net> <CAHVvXxQRO-V7002jhVPb5n46yS0b1MeDfNYVbMsVowTZm39NHQ@mail.gmail.com> <CADiSq7ec=+GGe7wQREozLyKZTR6pR=yxb71FG1Zv9b3oYWRrqw@mail.gmail.com> <CACac1F_Hr8U_4n27NvwH1=rN0ggFPx8MqJLbAh=GDXy_VNFLbA@mail.gmail.com> <522F1A24.5030808@avl.com> <20130910165438.297d71bc@pitrou.net> Message-ID: <522F33CA.6000909@stoneleaf.us> On 09/10/2013 07:54 AM, Antoine Pitrou wrote: > Le Tue, 10 Sep 2013 15:09:56 +0200, > Hrvoje Niksic <hrvoje.niksic at avl.com> a ?crit : >> On 09/10/2013 02:24 PM, Paul Moore wrote: >>>>>> td['FOO'] = 42 >>>>>> td['foo'] = 32 >>>>>> list(td.keys()) >>> >>> ['FOO'] or ['foo']? Both answers are justifiable. >> >> Note that the same question can be reasonably asked for dict itself: >> >> >>> d = {} >> >>> d[1.0] = 'foo' >> >>> d[1] = 'bar' >> >>> d >> {1.0: 'bar'} >> >> So, dict.__setitem__ only replaces the value, leaving the original >> key in place. transformdict should probably do the same, returning >> 'FOO' in your example. > > It's not that obvious. It's not common to rely on that aspect of dict > semantics, because you will usually lookup using the exact same type, > not a compatible one. I would expect very little code, if any, to rely > on this. > > (also, I would intuitively expect the latest key to be held, not the > first one, since that's what happens for values.) Whether there is a bunch of code that relies on this behavior is irrelevant. That behavior is already in place, and having another dict that is just the opposite will only lead to more confusion, not less. -- ~Ethan~ From rymg19 at gmail.com Tue Sep 10 17:26:00 2013 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Tue, 10 Sep 2013 10:26:00 -0500 Subject: [Python-Dev] [RELEASED] Python 3.4.0a2 In-Reply-To: <l0mbrr$ofo$1@ger.gmane.org> References: <522DB8D3.1030803@hastings.org> <nad-2495B7.12292709092013@news.gmane.org> <l0mbrr$ofo$1@ger.gmane.org> Message-ID: <CAO41-mPnC-Rdk=WANFk9dqaWufLeoH-GZM5nP-Q1oc6xgUtecA@mail.gmail.com> MS Windows is on a steep decline? I mean, I know Windows 8 isn't the most popular thing on the planet, but...Windows itself? If anything would be rising, I'd still prefer it to be either Linux or Haiku. On Tue, Sep 10, 2013 at 12:48 AM, Stefan Behnel <stefan_ml at behnel.de> wrote: > Ned Deily, 09.09.2013 21:29: > > 3.4.0a2 also contains a new batteries-included feature for OS X > > users. The python.org 64-bit/32-bit installer now includes its own > > private version of Tcl/Tk 8.5.14 so it is finally no longer necessary to > > install a third-party version of Tcl/Tk 8.5 to workaround the > > problematic system versions shipped in OS X 10.6+. More improvements to > > come. > > With MS Windows on steep decline, I'm so happy we have at least MacOS-X > rising to keep us busy. > > Stefan > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/rymg19%40gmail.com > -- Ryan -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130910/6753146e/attachment.html> From donald.stufft at gmail.com Tue Sep 10 17:28:25 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Tue, 10 Sep 2013 11:28:25 -0400 Subject: [Python-Dev] Offtopic: OpenID Providers In-Reply-To: <CAP7+vJJ0oG+QFZQKc4fLvOHwiu315WVoozv45dBdOqv59fw=Gw@mail.gmail.com> References: <5228C00F.7000209@jcea.es> <l0d311$b7n$1@ger.gmane.org> <522EC878.6080409@g.nevcal.com> <CAP7+vJJ0oG+QFZQKc4fLvOHwiu315WVoozv45dBdOqv59fw=Gw@mail.gmail.com> Message-ID: <EF440AAC-D5D9-4A31-966A-8C093A345F10@gmail.com> On Sep 10, 2013, at 11:08 AM, Guido van Rossum <guido at python.org> wrote: > Why do several posts in this thread have an Unsubscribe link that tries to unsubscribe me from the list? (I saw one by Glen, and another one by Donald Stufft.) > > (Come to think of it, what's the point of having an Unbub link in ever message that goes out?) All posts should have that. It's possible your mail client is collapsing it in some cases though because it looks like quoted text. > > > On Tue, Sep 10, 2013 at 12:21 AM, Glenn Linderman <v+python at g.nevcal.com> wrote: > On 9/6/2013 10:22 AM, Dan Callahan wrote: >> On 9/5/13 12:31 PM, Jesus Cea wrote: >>> I have big hopes for Mozilla Persona, looking forward >>> Python infrastructure support :). >> >> Hi, I'm the project lead on Persona signin, and I spoke at PyCon earlier this year regarding why and how Mozilla is building Persona. If you'd like some more background, that video [0] is worth a look. >> >> Let's pull this discussion up a level: >> >> It sounds like many people (Jesus, Donald, Toshio, Barry, Tres, Dirkjan, etc.) are interested in seeing Persona on Python.org properties, and most of the objections coming from a place of "Persona hasn't gone viral, what if this is wasted effort?" > > OK, let's pull this discussion down a level: testing it out. > > So I tried to login to the crossword.thetimes.co.uk -- I used an email address persona had never seen, it asked me for a password, and sent me a confirmation message, containing a link that I clicked on. > > However, as I was reading clues and filling in blanks, I got a popup that said "login failure [object Object]". And crossword told me it was saving locally, and to login to save to the server. And the Log in button stayed displayed, rather than a Log out button, which I assume it might get replaced with if I ever get successfully logged in. > > Firefox 23.0.1, Windows 7 64-bit with autoupdates. Need any other info? Write me privately if you want the email address I used (not the one I use here), or the password. I used new ones, so I can share for testing, and then discard them and use different ones "for real". If the system actually works. Hey, the video demos looked great... > > Glenn > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%40python.org > > > > > -- > --Guido van Rossum (python.org/~guido) ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130910/a650beeb/attachment.html> -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: <http://mail.python.org/pipermail/python-dev/attachments/20130910/a650beeb/attachment.sig> From janzert at janzert.com Tue Sep 10 17:54:20 2013 From: janzert at janzert.com (Janzert) Date: Tue, 10 Sep 2013 11:54:20 -0400 Subject: [Python-Dev] Add a "transformdict" to collections In-Reply-To: <20130910165438.297d71bc@pitrou.net> References: <20130910112832.75ed6c8b@pitrou.net> <CAHVvXxQRO-V7002jhVPb5n46yS0b1MeDfNYVbMsVowTZm39NHQ@mail.gmail.com> <CADiSq7ec=+GGe7wQREozLyKZTR6pR=yxb71FG1Zv9b3oYWRrqw@mail.gmail.com> <CACac1F_Hr8U_4n27NvwH1=rN0ggFPx8MqJLbAh=GDXy_VNFLbA@mail.gmail.com> <522F1A24.5030808@avl.com> <20130910165438.297d71bc@pitrou.net> Message-ID: <l0nfbc$tp1$1@ger.gmane.org> On 9/10/2013 10:54 AM, Antoine Pitrou wrote: > (also, I would intuitively expect the latest key to be held, not the > first one, since that's what happens for values.) > > Regards > > Antoine. > I intuitively expected, and I think most often would want, the first key to be held. The reason being that I would except most often the initial insertion to be the canonical form. Whichever way is adopted, I think even if you say it's implementation specific, people will end up relying on it in their code. Janzert From phd at phdru.name Tue Sep 10 18:03:35 2013 From: phd at phdru.name (Oleg Broytman) Date: Tue, 10 Sep 2013 20:03:35 +0400 Subject: [Python-Dev] Offtopic: OpenID Providers In-Reply-To: <EF440AAC-D5D9-4A31-966A-8C093A345F10@gmail.com> References: <5228C00F.7000209@jcea.es> <l0d311$b7n$1@ger.gmane.org> <522EC878.6080409@g.nevcal.com> <CAP7+vJJ0oG+QFZQKc4fLvOHwiu315WVoozv45dBdOqv59fw=Gw@mail.gmail.com> <EF440AAC-D5D9-4A31-966A-8C093A345F10@gmail.com> Message-ID: <20130910160335.GA3093@iskra.aviel.ru> On Tue, Sep 10, 2013 at 11:28:25AM -0400, Donald Stufft <donald.stufft at gmail.com> wrote: > On Sep 10, 2013, at 11:08 AM, Guido van Rossum <guido at python.org> wrote: > > Why do several posts in this thread have an Unsubscribe link that tries to unsubscribe me from the list? (I saw one by Glen, and another one by Donald Stufft.) > > > > (Come to think of it, what's the point of having an Unbub link in ever message that goes out?) > > All posts should have that. It's possible your mail client is collapsing it in some cases though because it looks like quoted text. When a mail message is all plain/text Mailman simply inline the stance into the text. When a message is a complex tree of MIME parts (like the message I'm replying to) Mailman append the stance to the end of the message as another MIME part. Some MUAs could ignore that part -- they neither show the part nor include it in replies. Mine (mutt) shows it and includes in replies anyway, I always trim it myself. Oleg. -- Oleg Broytman http://phdru.name/ phd at phdru.name Programmers don't die, they just GOSUB without RETURN. From v+python at g.nevcal.com Tue Sep 10 17:58:43 2013 From: v+python at g.nevcal.com (Glenn Linderman) Date: Tue, 10 Sep 2013 08:58:43 -0700 Subject: [Python-Dev] Add a "transformdict" to collections In-Reply-To: <l0nfbc$tp1$1@ger.gmane.org> References: <20130910112832.75ed6c8b@pitrou.net> <CAHVvXxQRO-V7002jhVPb5n46yS0b1MeDfNYVbMsVowTZm39NHQ@mail.gmail.com> <CADiSq7ec=+GGe7wQREozLyKZTR6pR=yxb71FG1Zv9b3oYWRrqw@mail.gmail.com> <CACac1F_Hr8U_4n27NvwH1=rN0ggFPx8MqJLbAh=GDXy_VNFLbA@mail.gmail.com> <522F1A24.5030808@avl.com> <20130910165438.297d71bc@pitrou.net> <l0nfbc$tp1$1@ger.gmane.org> Message-ID: <522F41B3.9020608@g.nevcal.com> On 9/10/2013 8:54 AM, Janzert wrote: > > I intuitively expected, and I think most often would want, the first > key to be held. My intuition matches yours, but my thoughts are that it should be changeable by specific request. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130910/239efdb8/attachment.html> From eric at trueblade.com Tue Sep 10 18:12:43 2013 From: eric at trueblade.com (Eric V. Smith) Date: Tue, 10 Sep 2013 12:12:43 -0400 Subject: [Python-Dev] Add a "transformdict" to collections In-Reply-To: <l0nfbc$tp1$1@ger.gmane.org> References: <20130910112832.75ed6c8b@pitrou.net> <CAHVvXxQRO-V7002jhVPb5n46yS0b1MeDfNYVbMsVowTZm39NHQ@mail.gmail.com> <CADiSq7ec=+GGe7wQREozLyKZTR6pR=yxb71FG1Zv9b3oYWRrqw@mail.gmail.com> <CACac1F_Hr8U_4n27NvwH1=rN0ggFPx8MqJLbAh=GDXy_VNFLbA@mail.gmail.com> <522F1A24.5030808@avl.com> <20130910165438.297d71bc@pitrou.net> <l0nfbc$tp1$1@ger.gmane.org> Message-ID: <522F44FB.8040408@trueblade.com> On 09/10/2013 11:54 AM, Janzert wrote: > On 9/10/2013 10:54 AM, Antoine Pitrou wrote: >> (also, I would intuitively expect the latest key to be held, not the >> first one, since that's what happens for values.) >> >> Regards >> >> Antoine. >> > > I intuitively expected, and I think most often would want, the first key > to be held. The reason being that I would except most often the initial > insertion to be the canonical form. My stock ticker example (but from a real problem I had a few days ago) would want to have this example: I want the first key, because I'm pre-seeding the dict with a variety of keys that are of the canonical form. > Whichever way is adopted, I think even if you say it's implementation > specific, people will end up relying on it in their code. Agreed. Eric. From nigel at nigelsmall.com Tue Sep 10 17:21:18 2013 From: nigel at nigelsmall.com (Nigel Small) Date: Tue, 10 Sep 2013 16:21:18 +0100 Subject: [Python-Dev] Add a "transformdict" to collections In-Reply-To: <522F2F77.9020805@gmail.com> References: <20130910112832.75ed6c8b@pitrou.net> <522F21CF.4060609@gmail.com> <CAMSv6X2MmJwg2E-kom7VF2_+U3btqRQfdEj1ioxg3AZAi5T61g@mail.gmail.com> <522F2F77.9020805@gmail.com> Message-ID: <CAFaXQDo1ifjvNrzohbsDV_m38fbpKfd5G2Q0yZfbJp9HB6FzPQ@mail.gmail.com> Could a more generic variant of this class work? In the same way that `sorted` can accept a comparison function, similar could be done for a dictionary-like class: d = transformdict(key=str.lower) Strictly speaking, this would provide case-insensitive but not case-preserving behaviour. For any given use case, though, a function could instead be supplied to "normalise" the key (upper, lower, title case, etc) in a way that fits that case. I can't think of many real cases where multiple types of capitalisation would be useful within the same dictionary. Nigel On 10 September 2013 15:40, Richard Oudkerk <shibturn at gmail.com> wrote: > On 10/09/2013 3:15pm, Armin Rigo wrote: > >> Hi Richard, >> >> On Tue, Sep 10, 2013 at 3:42 PM, Richard Oudkerk <shibturn at gmail.com> >> wrote: >> >>> I guess another example is creating an "identity dict" (see >>> http://code.activestate.com/**lists/python-ideas/7161/<http://code.activestate.com/lists/python-ideas/7161/>) >>> by doing >>> >>> d = transformdict(id) >>> >> >> This is bogus, because only the id will be stored, and the original >> key object will be forgotten (and its id probably reused). >> > > Seems to work for me: > > >>> import collections > >>> d = collections.transformdict(id) > >>> L = [1,2,3] > >>> d[L] = None > >>> L in d > True > >>> [1,2,3] in d > False > >>> print(d[L]) > None > >>> d._data > {41444136: ([1, 2, 3], None)} > >>> list(d) > [[1, 2, 3]] > > However __repr__() is broken: > > >>> d > Traceback (most recent call last): > File "<stdin>", line 1, in <module> > File "C:\Repos\cpython-dirty\lib\**collections\abc.py", line 444, in > __repr__ > return '{0.__class__.__name__}({0._**mapping!r})'.format(self) > File "C:\Repos\cpython-dirty\lib\**collections\__init__.py", line 944, > in __repr__ > self._transform, repr(dict(self))) > TypeError: unhashable type: 'list' > > -- > Richard > > ______________________________**_________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/**mailman/listinfo/python-dev<https://mail.python.org/mailman/listinfo/python-dev> > Unsubscribe: https://mail.python.org/**mailman/options/python-dev/** > nigel%40nigelsmall.com<https://mail.python.org/mailman/options/python-dev/nigel%40nigelsmall.com> > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130910/5107ee06/attachment.html> From shibturn at gmail.com Tue Sep 10 19:09:30 2013 From: shibturn at gmail.com (Richard Oudkerk) Date: Tue, 10 Sep 2013 18:09:30 +0100 Subject: [Python-Dev] IOCP (I/O Completion Port) in Python ? In-Reply-To: <BLU402-EAS15259F82C4E6C2CEBFC83789D380@phx.gbl> References: <BLU402-EAS15259F82C4E6C2CEBFC83789D380@phx.gbl> Message-ID: <522F524A.6000800@gmail.com> On 10/09/2013 11:58am, ??? wrote: > Hello: > I wondering why there is no standard IOCP module in Python ? > As I know: Python3 have support epoll in linux and kqueue in freebsd. > Is there a plan to add IOCP module for Windows ? _winapi does have some undocumented support for IOCP (used only by multiprocessing), but the APIs are not very nice and should not be considered stable. (Also _winapi.ReadFile(), _winapi.WriteFile() only support pipes or sockets, not disk-based files where you also have to control file-position.) -- Richard From shibturn at gmail.com Tue Sep 10 19:09:30 2013 From: shibturn at gmail.com (Richard Oudkerk) Date: Tue, 10 Sep 2013 18:09:30 +0100 Subject: [Python-Dev] IOCP (I/O Completion Port) in Python ? In-Reply-To: <BLU402-EAS15259F82C4E6C2CEBFC83789D380@phx.gbl> References: <BLU402-EAS15259F82C4E6C2CEBFC83789D380@phx.gbl> Message-ID: <522F524A.6000800@gmail.com> On 10/09/2013 11:58am, ??? wrote: > Hello: > I wondering why there is no standard IOCP module in Python ? > As I know: Python3 have support epoll in linux and kqueue in freebsd. > Is there a plan to add IOCP module for Windows ? _winapi does have some undocumented support for IOCP (used only by multiprocessing), but the APIs are not very nice and should not be considered stable. (Also _winapi.ReadFile(), _winapi.WriteFile() only support pipes or sockets, not disk-based files where you also have to control file-position.) -- Richard From rdmurray at bitdance.com Tue Sep 10 20:28:00 2013 From: rdmurray at bitdance.com (R. David Murray) Date: Tue, 10 Sep 2013 14:28:00 -0400 Subject: [Python-Dev] Add a "transformdict" to collections In-Reply-To: <CAFaXQDo1ifjvNrzohbsDV_m38fbpKfd5G2Q0yZfbJp9HB6FzPQ@mail.gmail.com> References: <20130910112832.75ed6c8b@pitrou.net> <522F21CF.4060609@gmail.com> <CAMSv6X2MmJwg2E-kom7VF2_+U3btqRQfdEj1ioxg3AZAi5T61g@mail.gmail.com> <522F2F77.9020805@gmail.com> <CAFaXQDo1ifjvNrzohbsDV_m38fbpKfd5G2Q0yZfbJp9HB6FzPQ@mail.gmail.com> Message-ID: <20130910182800.6A0112507A2@webabinitio.net> On Tue, 10 Sep 2013 16:21:18 +0100, Nigel Small <nigel at nigelsmall.com> wrote: > Could a more generic variant of this class work? In the same way that > `sorted` can accept a comparison function, similar could be done for a > dictionary-like class: > > d = transformdict(key=str.lower) > > Strictly speaking, this would provide case-insensitive but not > case-preserving behaviour. For any given use case, though, a function could > instead be supplied to "normalise" the key (upper, lower, title case, etc) > in a way that fits that case. I can't think of many real cases where > multiple types of capitalisation would be useful within the same dictionary. I can: MIME-Version Message-ID Content-Type For network protocols you really do want to *preserve* the case. --David From solipsis at pitrou.net Tue Sep 10 20:31:40 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 10 Sep 2013 20:31:40 +0200 Subject: [Python-Dev] Add a "transformdict" to collections In-Reply-To: <522F1411.8010109@v.loewis.de> References: <20130910112832.75ed6c8b@pitrou.net> <CAHVvXxQRO-V7002jhVPb5n46yS0b1MeDfNYVbMsVowTZm39NHQ@mail.gmail.com> <CADiSq7ec=+GGe7wQREozLyKZTR6pR=yxb71FG1Zv9b3oYWRrqw@mail.gmail.com> <CACac1F_Hr8U_4n27NvwH1=rN0ggFPx8MqJLbAh=GDXy_VNFLbA@mail.gmail.com> <20130910143558.5677b55a@pitrou.net> <522F1411.8010109@v.loewis.de> Message-ID: <20130910203140.1e8c1008@fsol> On Tue, 10 Sep 2013 14:44:01 +0200 "Martin v. L?wis" <martin at v.loewis.de> wrote: > Am 10.09.13 14:35, schrieb Antoine Pitrou: > >> ['FOO'] or ['foo']? Both answers are justifiable. Both are possibly > >> even useful depending on context... > > > > I think it would be best to leave it as an implementation detail, > > because whichever is easiest to implement depends on the exact > > implementation choices (e.g. C vs. Python). > > I think this is the point where the datatype is *not* clearly > straight-forward, and thus deserves a PEP. Not saying you're necessary wrong, but a few data points: - defaultdict was added without a PEP: http://bugs.python.org/issue1433928 - Counter was added without a PEP: http://bugs.python.org/issue1696199 - ChainMap was added without a PEP: http://bugs.python.org/issue11089 http://bugs.python.org/issue11297 - namedtuple was added without a PEP: https://mail.python.org/pipermail/python-dev/2007-February/071196.html (no tracker reference for its addition) OrderedDict, however, came with a PEP: http://www.python.org/dev/peps/pep-0372/ > I think it would be a flaw to have this detail implementation-defined. > This would be like saying that it is implementation-defined which > of A,B,C is returned from "A and B and C" if all are true. Ok, it seems everyone (except me :-)) agrees that it should return the first key value, so that's how it will be. Regards Antoine. From p.f.moore at gmail.com Tue Sep 10 21:08:56 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 10 Sep 2013 20:08:56 +0100 Subject: [Python-Dev] Add a "transformdict" to collections In-Reply-To: <20130910203140.1e8c1008@fsol> References: <20130910112832.75ed6c8b@pitrou.net> <CAHVvXxQRO-V7002jhVPb5n46yS0b1MeDfNYVbMsVowTZm39NHQ@mail.gmail.com> <CADiSq7ec=+GGe7wQREozLyKZTR6pR=yxb71FG1Zv9b3oYWRrqw@mail.gmail.com> <CACac1F_Hr8U_4n27NvwH1=rN0ggFPx8MqJLbAh=GDXy_VNFLbA@mail.gmail.com> <20130910143558.5677b55a@pitrou.net> <522F1411.8010109@v.loewis.de> <20130910203140.1e8c1008@fsol> Message-ID: <CACac1F_mm97XUSde62vWtD9K643QkpUCedYi0YAhKJvOG39MgQ@mail.gmail.com> On 10 September 2013 19:31, Antoine Pitrou <solipsis at pitrou.net> wrote: >> I think it would be a flaw to have this detail implementation-defined. >> This would be like saying that it is implementation-defined which >> of A,B,C is returned from "A and B and C" if all are true. > > Ok, it seems everyone (except me :-)) agrees that it should return the > first key value, so that's how it will be. If you retain the first key value, it's easy enough for the application to implement "retain the last" semantics: try: del d[k] finally: d[k] = v If you provide "retain the last", I can't see any obvious way of implementing "retain the first" in application code without in effect reimplementing the class. Paul From python at mrabarnett.plus.com Tue Sep 10 21:59:56 2013 From: python at mrabarnett.plus.com (MRAB) Date: Tue, 10 Sep 2013 20:59:56 +0100 Subject: [Python-Dev] Add a "transformdict" to collections In-Reply-To: <CACac1F_mm97XUSde62vWtD9K643QkpUCedYi0YAhKJvOG39MgQ@mail.gmail.com> References: <20130910112832.75ed6c8b@pitrou.net> <CAHVvXxQRO-V7002jhVPb5n46yS0b1MeDfNYVbMsVowTZm39NHQ@mail.gmail.com> <CADiSq7ec=+GGe7wQREozLyKZTR6pR=yxb71FG1Zv9b3oYWRrqw@mail.gmail.com> <CACac1F_Hr8U_4n27NvwH1=rN0ggFPx8MqJLbAh=GDXy_VNFLbA@mail.gmail.com> <20130910143558.5677b55a@pitrou.net> <522F1411.8010109@v.loewis.de> <20130910203140.1e8c1008@fsol> <CACac1F_mm97XUSde62vWtD9K643QkpUCedYi0YAhKJvOG39MgQ@mail.gmail.com> Message-ID: <522F7A3C.1040701@mrabarnett.plus.com> On 10/09/2013 20:08, Paul Moore wrote: > On 10 September 2013 19:31, Antoine Pitrou <solipsis at pitrou.net> wrote: >>> I think it would be a flaw to have this detail implementation-defined. >>> This would be like saying that it is implementation-defined which >>> of A,B,C is returned from "A and B and C" if all are true. >> >> Ok, it seems everyone (except me :-)) agrees that it should return the >> first key value, so that's how it will be. > > If you retain the first key value, it's easy enough for the > application to implement "retain the last" semantics: > > try: > del d[k] > finally: > d[k] = v > That would raise a KeyError is the key was missing. A better way is: d.pop(k, None) d[k] = v > If you provide "retain the last", I can't see any obvious way of > implementing "retain the first" in application code without in effect > reimplementing the class. > "Retain the first" does feel more natural to me. From lukas.lueg at gmail.com Tue Sep 10 22:36:18 2013 From: lukas.lueg at gmail.com (Lukas Lueg) Date: Tue, 10 Sep 2013 22:36:18 +0200 Subject: [Python-Dev] Add a "transformdict" to collections In-Reply-To: <522F7A3C.1040701@mrabarnett.plus.com> References: <20130910112832.75ed6c8b@pitrou.net> <CAHVvXxQRO-V7002jhVPb5n46yS0b1MeDfNYVbMsVowTZm39NHQ@mail.gmail.com> <CADiSq7ec=+GGe7wQREozLyKZTR6pR=yxb71FG1Zv9b3oYWRrqw@mail.gmail.com> <CACac1F_Hr8U_4n27NvwH1=rN0ggFPx8MqJLbAh=GDXy_VNFLbA@mail.gmail.com> <20130910143558.5677b55a@pitrou.net> <522F1411.8010109@v.loewis.de> <20130910203140.1e8c1008@fsol> <CACac1F_mm97XUSde62vWtD9K643QkpUCedYi0YAhKJvOG39MgQ@mail.gmail.com> <522F7A3C.1040701@mrabarnett.plus.com> Message-ID: <CAJF-kYn6aX4cx7z4ERGdXTnuZcj0tcSmt6F6jBgu_qG_ZLZxpw@mail.gmail.com> Should'nt the key'ing behaviour be controlled by the type of the key instead of the type of the container? 2013/9/10 MRAB <python at mrabarnett.plus.com> > On 10/09/2013 20:08, Paul Moore wrote: > >> On 10 September 2013 19:31, Antoine Pitrou <solipsis at pitrou.net> wrote: >> >>> I think it would be a flaw to have this detail implementation-defined. >>>> This would be like saying that it is implementation-defined which >>>> of A,B,C is returned from "A and B and C" if all are true. >>>> >>> >>> Ok, it seems everyone (except me :-)) agrees that it should return the >>> first key value, so that's how it will be. >>> >> >> If you retain the first key value, it's easy enough for the >> application to implement "retain the last" semantics: >> >> try: >> del d[k] >> finally: >> d[k] = v >> >> That would raise a KeyError is the key was missing. A better way is: > > d.pop(k, None) > > d[k] = v > > If you provide "retain the last", I can't see any obvious way of >> implementing "retain the first" in application code without in effect >> reimplementing the class. >> >> "Retain the first" does feel more natural to me. > > ______________________________**_________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/**mailman/listinfo/python-dev<https://mail.python.org/mailman/listinfo/python-dev> > Unsubscribe: https://mail.python.org/**mailman/options/python-dev/** > lukas.lueg%40gmail.com<https://mail.python.org/mailman/options/python-dev/lukas.lueg%40gmail.com> > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130910/7810d8bf/attachment.html> From jsbueno at python.org.br Tue Sep 10 22:38:26 2013 From: jsbueno at python.org.br (Joao S. O. Bueno) Date: Tue, 10 Sep 2013 17:38:26 -0300 Subject: [Python-Dev] Add a "transformdict" to collections In-Reply-To: <CACac1F_mm97XUSde62vWtD9K643QkpUCedYi0YAhKJvOG39MgQ@mail.gmail.com> References: <20130910112832.75ed6c8b@pitrou.net> <CAHVvXxQRO-V7002jhVPb5n46yS0b1MeDfNYVbMsVowTZm39NHQ@mail.gmail.com> <CADiSq7ec=+GGe7wQREozLyKZTR6pR=yxb71FG1Zv9b3oYWRrqw@mail.gmail.com> <CACac1F_Hr8U_4n27NvwH1=rN0ggFPx8MqJLbAh=GDXy_VNFLbA@mail.gmail.com> <20130910143558.5677b55a@pitrou.net> <522F1411.8010109@v.loewis.de> <20130910203140.1e8c1008@fsol> <CACac1F_mm97XUSde62vWtD9K643QkpUCedYi0YAhKJvOG39MgQ@mail.gmail.com> Message-ID: <CAH0mxTQBPy4ARJDXYNCCyGs5Oqjb0puL5gtL38FpjhZVcRy6LQ@mail.gmail.com> On 10 September 2013 16:08, Paul Moore <p.f.moore at gmail.com> wrote: > If you provide "retain the last", I can't see any obvious way of > implementing "retain the first" in application code without in effect > reimplementing the class. Which reminds one - this class should obviously have a method for retrivieng the original key value, given a matching key - d.canonical('foo') -> 'Foo' From solipsis at pitrou.net Tue Sep 10 23:06:07 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 10 Sep 2013 23:06:07 +0200 Subject: [Python-Dev] Add a "transformdict" to collections In-Reply-To: <CAH0mxTQBPy4ARJDXYNCCyGs5Oqjb0puL5gtL38FpjhZVcRy6LQ@mail.gmail.com> References: <20130910112832.75ed6c8b@pitrou.net> <CAHVvXxQRO-V7002jhVPb5n46yS0b1MeDfNYVbMsVowTZm39NHQ@mail.gmail.com> <CADiSq7ec=+GGe7wQREozLyKZTR6pR=yxb71FG1Zv9b3oYWRrqw@mail.gmail.com> <CACac1F_Hr8U_4n27NvwH1=rN0ggFPx8MqJLbAh=GDXy_VNFLbA@mail.gmail.com> <20130910143558.5677b55a@pitrou.net> <522F1411.8010109@v.loewis.de> <20130910203140.1e8c1008@fsol> <CACac1F_mm97XUSde62vWtD9K643QkpUCedYi0YAhKJvOG39MgQ@mail.gmail.com> <CAH0mxTQBPy4ARJDXYNCCyGs5Oqjb0puL5gtL38FpjhZVcRy6LQ@mail.gmail.com> Message-ID: <20130910230607.701bba16@fsol> On Tue, 10 Sep 2013 17:38:26 -0300 "Joao S. O. Bueno" <jsbueno at python.org.br> wrote: > On 10 September 2013 16:08, Paul Moore <p.f.moore at gmail.com> wrote: > > If you provide "retain the last", I can't see any obvious way of > > implementing "retain the first" in application code without in effect > > reimplementing the class. > > Which reminds one - this class should obviously have a method for > retrivieng the original key value, given a matching key - > > d.canonical('foo') -> 'Foo' I don't know. Is there any use case? (sure, it is trivially implemented) Regards Antoine. From p.f.moore at gmail.com Tue Sep 10 23:12:51 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 10 Sep 2013 22:12:51 +0100 Subject: [Python-Dev] Add a "transformdict" to collections In-Reply-To: <522F7A3C.1040701@mrabarnett.plus.com> References: <20130910112832.75ed6c8b@pitrou.net> <CAHVvXxQRO-V7002jhVPb5n46yS0b1MeDfNYVbMsVowTZm39NHQ@mail.gmail.com> <CADiSq7ec=+GGe7wQREozLyKZTR6pR=yxb71FG1Zv9b3oYWRrqw@mail.gmail.com> <CACac1F_Hr8U_4n27NvwH1=rN0ggFPx8MqJLbAh=GDXy_VNFLbA@mail.gmail.com> <20130910143558.5677b55a@pitrou.net> <522F1411.8010109@v.loewis.de> <20130910203140.1e8c1008@fsol> <CACac1F_mm97XUSde62vWtD9K643QkpUCedYi0YAhKJvOG39MgQ@mail.gmail.com> <522F7A3C.1040701@mrabarnett.plus.com> Message-ID: <CACac1F9UgGGOMxfmV2E4H89UH8eYqvLOccDZ7t=9FRz+pOwayQ@mail.gmail.com> On 10 September 2013 20:59, MRAB <python at mrabarnett.plus.com> wrote: >> try: >> del d[k] >> finally: >> d[k] = v >> > That would raise a KeyError is the key was missing. A better way is: > > d.pop(k, None) Sorry, I was thinking of try...except: pass. But pop is indeed better. Teach me to code stuff on the fly without testing :-) Paul From ethan at stoneleaf.us Tue Sep 10 23:17:24 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 10 Sep 2013 14:17:24 -0700 Subject: [Python-Dev] Add a "transformdict" to collections In-Reply-To: <CAJF-kYn6aX4cx7z4ERGdXTnuZcj0tcSmt6F6jBgu_qG_ZLZxpw@mail.gmail.com> References: <20130910112832.75ed6c8b@pitrou.net> <CAHVvXxQRO-V7002jhVPb5n46yS0b1MeDfNYVbMsVowTZm39NHQ@mail.gmail.com> <CADiSq7ec=+GGe7wQREozLyKZTR6pR=yxb71FG1Zv9b3oYWRrqw@mail.gmail.com> <CACac1F_Hr8U_4n27NvwH1=rN0ggFPx8MqJLbAh=GDXy_VNFLbA@mail.gmail.com> <20130910143558.5677b55a@pitrou.net> <522F1411.8010109@v.loewis.de> <20130910203140.1e8c1008@fsol> <CACac1F_mm97XUSde62vWtD9K643QkpUCedYi0YAhKJvOG39MgQ@mail.gmail.com> <522F7A3C.1040701@mrabarnett.plus.com> <CAJF-kYn6aX4cx7z4ERGdXTnuZcj0tcSmt6F6jBgu_qG_ZLZxpw@mail.gmail.com> Message-ID: <522F8C64.1080005@stoneleaf.us> On 09/10/2013 01:36 PM, Lukas Lueg wrote: > > Should'nt the key'ing behaviour be controlled by the type of the key instead of the type of the container? That would be up to the key function. -- ~Ethan~ From jsbueno at python.org.br Tue Sep 10 23:44:20 2013 From: jsbueno at python.org.br (Joao S. O. Bueno) Date: Tue, 10 Sep 2013 18:44:20 -0300 Subject: [Python-Dev] Add a "transformdict" to collections In-Reply-To: <20130910230607.701bba16@fsol> References: <20130910112832.75ed6c8b@pitrou.net> <CAHVvXxQRO-V7002jhVPb5n46yS0b1MeDfNYVbMsVowTZm39NHQ@mail.gmail.com> <CADiSq7ec=+GGe7wQREozLyKZTR6pR=yxb71FG1Zv9b3oYWRrqw@mail.gmail.com> <CACac1F_Hr8U_4n27NvwH1=rN0ggFPx8MqJLbAh=GDXy_VNFLbA@mail.gmail.com> <20130910143558.5677b55a@pitrou.net> <522F1411.8010109@v.loewis.de> <20130910203140.1e8c1008@fsol> <CACac1F_mm97XUSde62vWtD9K643QkpUCedYi0YAhKJvOG39MgQ@mail.gmail.com> <CAH0mxTQBPy4ARJDXYNCCyGs5Oqjb0puL5gtL38FpjhZVcRy6LQ@mail.gmail.com> <20130910230607.701bba16@fsol> Message-ID: <CAH0mxTSTzEkhgXuuzZG-XMn+SV6rqEQNGpp9Xyf96zRBSBh4+A@mail.gmail.com> On 10 September 2013 18:06, Antoine Pitrou <solipsis at pitrou.net> wrote: > On Tue, 10 Sep 2013 17:38:26 -0300 > "Joao S. O. Bueno" <jsbueno at python.org.br> wrote: >> On 10 September 2013 16:08, Paul Moore <p.f.moore at gmail.com> wrote: >> > If you provide "retain the last", I can't see any obvious way of >> > implementing "retain the first" in application code without in effect >> > reimplementing the class. >> >> Which reminds one - this class should obviously have a method for >> retrivieng the original key value, given a matching key - >> >> d.canonical('foo') -> 'Foo' > > I don't know. Is there any use case? > (sure, it is trivially implemented) Well, I'd expect it to simply be there. I had not thought of other usecases for the transformdict itself - but if I would use it and would need the original key without such a method it would not be trivial to get it. For example, in latim languages it is common to want accented letters to match their unaccented counterparts - pick my own first name "Jo?o" - if I'd use a transform to strip the diactriticals, and have an user input "joao" - it would match, as intended - but I would not be able to retrieve the accented version without re-implementing the class behavior. js -><- > > Regards > > Antoine. > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/jsbueno%40python.org.br From solipsis at pitrou.net Tue Sep 10 23:46:59 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 10 Sep 2013 23:46:59 +0200 Subject: [Python-Dev] Add a "transformdict" to collections In-Reply-To: <CAH0mxTSTzEkhgXuuzZG-XMn+SV6rqEQNGpp9Xyf96zRBSBh4+A@mail.gmail.com> References: <20130910112832.75ed6c8b@pitrou.net> <CAHVvXxQRO-V7002jhVPb5n46yS0b1MeDfNYVbMsVowTZm39NHQ@mail.gmail.com> <CADiSq7ec=+GGe7wQREozLyKZTR6pR=yxb71FG1Zv9b3oYWRrqw@mail.gmail.com> <CACac1F_Hr8U_4n27NvwH1=rN0ggFPx8MqJLbAh=GDXy_VNFLbA@mail.gmail.com> <20130910143558.5677b55a@pitrou.net> <522F1411.8010109@v.loewis.de> <20130910203140.1e8c1008@fsol> <CACac1F_mm97XUSde62vWtD9K643QkpUCedYi0YAhKJvOG39MgQ@mail.gmail.com> <CAH0mxTQBPy4ARJDXYNCCyGs5Oqjb0puL5gtL38FpjhZVcRy6LQ@mail.gmail.com> <20130910230607.701bba16@fsol> <CAH0mxTSTzEkhgXuuzZG-XMn+SV6rqEQNGpp9Xyf96zRBSBh4+A@mail.gmail.com> Message-ID: <20130910234659.39284dab@fsol> On Tue, 10 Sep 2013 18:44:20 -0300 "Joao S. O. Bueno" <jsbueno at python.org.br> wrote: > On 10 September 2013 18:06, Antoine Pitrou <solipsis at pitrou.net> wrote: > > On Tue, 10 Sep 2013 17:38:26 -0300 > > "Joao S. O. Bueno" <jsbueno at python.org.br> wrote: > >> On 10 September 2013 16:08, Paul Moore <p.f.moore at gmail.com> wrote: > >> > If you provide "retain the last", I can't see any obvious way of > >> > implementing "retain the first" in application code without in effect > >> > reimplementing the class. > >> > >> Which reminds one - this class should obviously have a method for > >> retrivieng the original key value, given a matching key - > >> > >> d.canonical('foo') -> 'Foo' > > > > I don't know. Is there any use case? > > (sure, it is trivially implemented) > > > Well, I'd expect it to simply be there. I had not thought of > other usecases for the transformdict itself - Well, it is not here for dict, set, etc. > For example, in latim languages it is common to want > accented letters to match their unaccented counterparts > - pick my own first name "Jo?o" - if I'd use a transform to strip > the diactriticals, and have an user input "joao" - it would match, > as intended - but I would not be able to retrieve the accented version > without re-implementing the class behavior. Interesting example, thanks. Regards Antoine. From python at mrabarnett.plus.com Wed Sep 11 00:12:56 2013 From: python at mrabarnett.plus.com (MRAB) Date: Tue, 10 Sep 2013 23:12:56 +0100 Subject: [Python-Dev] Add a "transformdict" to collections In-Reply-To: <20130910234659.39284dab@fsol> References: <20130910112832.75ed6c8b@pitrou.net> <CAHVvXxQRO-V7002jhVPb5n46yS0b1MeDfNYVbMsVowTZm39NHQ@mail.gmail.com> <CADiSq7ec=+GGe7wQREozLyKZTR6pR=yxb71FG1Zv9b3oYWRrqw@mail.gmail.com> <CACac1F_Hr8U_4n27NvwH1=rN0ggFPx8MqJLbAh=GDXy_VNFLbA@mail.gmail.com> <20130910143558.5677b55a@pitrou.net> <522F1411.8010109@v.loewis.de> <20130910203140.1e8c1008@fsol> <CACac1F_mm97XUSde62vWtD9K643QkpUCedYi0YAhKJvOG39MgQ@mail.gmail.com> <CAH0mxTQBPy4ARJDXYNCCyGs5Oqjb0puL5gtL38FpjhZVcRy6LQ@mail.gmail.com> <20130910230607.701bba16@fsol> <CAH0mxTSTzEkhgXuuzZG-XMn+SV6rqEQNGpp9Xyf96zRBSBh4+A@mail.gmail.com> <20130910234659.39284dab@fsol> Message-ID: <522F9968.5030108@mrabarnett.plus.com> On 10/09/2013 22:46, Antoine Pitrou wrote: > On Tue, 10 Sep 2013 18:44:20 -0300 > "Joao S. O. Bueno" <jsbueno at python.org.br> wrote: >> On 10 September 2013 18:06, Antoine Pitrou <solipsis at pitrou.net> wrote: >> > On Tue, 10 Sep 2013 17:38:26 -0300 >> > "Joao S. O. Bueno" <jsbueno at python.org.br> wrote: >> >> On 10 September 2013 16:08, Paul Moore <p.f.moore at gmail.com> wrote: >> >> > If you provide "retain the last", I can't see any obvious way of >> >> > implementing "retain the first" in application code without in effect >> >> > reimplementing the class. >> >> >> >> Which reminds one - this class should obviously have a method for >> >> retrivieng the original key value, given a matching key - >> >> >> >> d.canonical('foo') -> 'Foo' >> > >> > I don't know. Is there any use case? >> > (sure, it is trivially implemented) >> >> Well, I'd expect it to simply be there. I had not thought of >> other usecases for the transformdict itself - > I had the same thought. > Well, it is not here for dict, set, etc. > In those cases the key in the dict == the key you're looking for. >> For example, in latim languages it is common to want >> accented letters to match their unaccented counterparts >> - pick my own first name "Jo?o" - if I'd use a transform to strip >> the diactriticals, and have an user input "joao" - it would match, >> as intended - but I would not be able to retrieve the accented version >> without re-implementing the class behavior. > > Interesting example, thanks. > From ethan at stoneleaf.us Wed Sep 11 00:18:05 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 10 Sep 2013 15:18:05 -0700 Subject: [Python-Dev] Add a "transformdict" to collections In-Reply-To: <522F9968.5030108@mrabarnett.plus.com> References: <20130910112832.75ed6c8b@pitrou.net> <CAHVvXxQRO-V7002jhVPb5n46yS0b1MeDfNYVbMsVowTZm39NHQ@mail.gmail.com> <CADiSq7ec=+GGe7wQREozLyKZTR6pR=yxb71FG1Zv9b3oYWRrqw@mail.gmail.com> <CACac1F_Hr8U_4n27NvwH1=rN0ggFPx8MqJLbAh=GDXy_VNFLbA@mail.gmail.com> <20130910143558.5677b55a@pitrou.net> <522F1411.8010109@v.loewis.de> <20130910203140.1e8c1008@fsol> <CACac1F_mm97XUSde62vWtD9K643QkpUCedYi0YAhKJvOG39MgQ@mail.gmail.com> <CAH0mxTQBPy4ARJDXYNCCyGs5Oqjb0puL5gtL38FpjhZVcRy6LQ@mail.gmail.com> <20130910230607.701bba16@fsol> <CAH0mxTSTzEkhgXuuzZG-XMn+SV6rqEQNGpp9Xyf96zRBSBh4+A@mail.gmail.com> <20130910234659.39284dab@fsol> <522F9968.5030108@mrabarnett.plus.com> Message-ID: <522F9A9D.9010008@stoneleaf.us> On 09/10/2013 03:12 PM, MRAB wrote: > On 10/09/2013 22:46, Antoine Pitrou wrote: >> On Tue, 10 Sep 2013 18:44:20 -0300 >> "Joao S. O. Bueno" <jsbueno at python.org.br> wrote: >>> On 10 September 2013 18:06, Antoine Pitrou <solipsis at pitrou.net> wrote: >>> > On Tue, 10 Sep 2013 17:38:26 -0300 >>> > "Joao S. O. Bueno" <jsbueno at python.org.br> wrote: >>> >> On 10 September 2013 16:08, Paul Moore <p.f.moore at gmail.com> wrote: >>> >> > If you provide "retain the last", I can't see any obvious way of >>> >> > implementing "retain the first" in application code without in effect >>> >> > reimplementing the class. >>> >> >>> >> Which reminds one - this class should obviously have a method for >>> >> retrivieng the original key value, given a matching key - >>> >> >>> >> d.canonical('foo') -> 'Foo' >>> > >>> > I don't know. Is there any use case? >>> > (sure, it is trivially implemented) >>> >>> Well, I'd expect it to simply be there. I had not thought of >>> other usecases for the transformdict itself - >> > I had the same thought. > >> Well, it is not here for dict, set, etc. >> > In those cases the key in the dict == the key you're looking for. With the exception of numbers, of course (float vs int vs Decimal, etc.). But if that distinction was useful for someone they could use this new TransformDict. :) -- ~Ethan~ From eric at trueblade.com Wed Sep 11 02:26:55 2013 From: eric at trueblade.com (Eric V. Smith) Date: Tue, 10 Sep 2013 20:26:55 -0400 Subject: [Python-Dev] Add a "transformdict" to collections In-Reply-To: <522F9A9D.9010008@stoneleaf.us> References: <20130910112832.75ed6c8b@pitrou.net> <CAHVvXxQRO-V7002jhVPb5n46yS0b1MeDfNYVbMsVowTZm39NHQ@mail.gmail.com> <CADiSq7ec=+GGe7wQREozLyKZTR6pR=yxb71FG1Zv9b3oYWRrqw@mail.gmail.com> <CACac1F_Hr8U_4n27NvwH1=rN0ggFPx8MqJLbAh=GDXy_VNFLbA@mail.gmail.com> <20130910143558.5677b55a@pitrou.net> <522F1411.8010109@v.loewis.de> <20130910203140.1e8c1008@fsol> <CACac1F_mm97XUSde62vWtD9K643QkpUCedYi0YAhKJvOG39MgQ@mail.gmail.com> <CAH0mxTQBPy4ARJDXYNCCyGs5Oqjb0puL5gtL38FpjhZVcRy6LQ@mail.gmail.com> <20130910230607.701bba16@fsol> <CAH0mxTSTzEkhgXuuzZG-XMn+SV6rqEQNGpp9Xyf96zRBSBh4+A@mail.gmail.com> <20130910234659.39284dab@fsol> <522F9968.5030108@mrabarnett.plus.com> <522F9A9D.9010008@stoneleaf.us> Message-ID: <522FB8CF.5020400@trueblade.com> On 9/10/2013 6:18 PM, Ethan Furman wrote: > On 09/10/2013 03:12 PM, MRAB wrote: >> On 10/09/2013 22:46, Antoine Pitrou wrote: >>> On Tue, 10 Sep 2013 18:44:20 -0300 >>> "Joao S. O. Bueno" <jsbueno at python.org.br> wrote: >>>> On 10 September 2013 18:06, Antoine Pitrou <solipsis at pitrou.net> wrote: >>>> > On Tue, 10 Sep 2013 17:38:26 -0300 >>>> > "Joao S. O. Bueno" <jsbueno at python.org.br> wrote: >>>> >> On 10 September 2013 16:08, Paul Moore <p.f.moore at gmail.com> wrote: >>>> >> > If you provide "retain the last", I can't see any obvious way of >>>> >> > implementing "retain the first" in application code without in >>>> effect >>>> >> > reimplementing the class. >>>> >> >>>> >> Which reminds one - this class should obviously have a method for >>>> >> retrivieng the original key value, given a matching key - >>>> >> >>>> >> d.canonical('foo') -> 'Foo' >>>> > >>>> > I don't know. Is there any use case? >>>> > (sure, it is trivially implemented) >>>> >>>> Well, I'd expect it to simply be there. I had not thought of >>>> other usecases for the transformdict itself - >>> >> I had the same thought. >> >>> Well, it is not here for dict, set, etc. >>> >> In those cases the key in the dict == the key you're looking for. > > With the exception of numbers, of course (float vs int vs Decimal, etc.). They'd still be ==, wouldn't they? -- Eric. From v+python at g.nevcal.com Wed Sep 11 03:15:33 2013 From: v+python at g.nevcal.com (Glenn Linderman) Date: Tue, 10 Sep 2013 18:15:33 -0700 Subject: [Python-Dev] Add a "transformdict" to collections In-Reply-To: <20130910234659.39284dab@fsol> References: <20130910112832.75ed6c8b@pitrou.net> <CAHVvXxQRO-V7002jhVPb5n46yS0b1MeDfNYVbMsVowTZm39NHQ@mail.gmail.com> <CADiSq7ec=+GGe7wQREozLyKZTR6pR=yxb71FG1Zv9b3oYWRrqw@mail.gmail.com> <CACac1F_Hr8U_4n27NvwH1=rN0ggFPx8MqJLbAh=GDXy_VNFLbA@mail.gmail.com> <20130910143558.5677b55a@pitrou.net> <522F1411.8010109@v.loewis.de> <20130910203140.1e8c1008@fsol> <CACac1F_mm97XUSde62vWtD9K643QkpUCedYi0YAhKJvOG39MgQ@mail.gmail.com> <CAH0mxTQBPy4ARJDXYNCCyGs5Oqjb0puL5gtL38FpjhZVcRy6LQ@mail.gmail.com> <20130910230607.701bba16@fsol> <CAH0mxTSTzEkhgXuuzZG-XMn+SV6rqEQNGpp9Xyf96zRBSBh4+A@mail.gmail.com> <20130910234659.39284dab@fsol> Message-ID: <522FC435.20405@g.nevcal.com> On 9/10/2013 2:46 PM, Antoine Pitrou wrote: >>>> > >>Which reminds one - this class should obviously have a method for >>>> > >>retrivieng the original key value, given a matching key - >>>> > >> >>>> > >>d.canonical('foo') -> 'Foo' >>> > > >>> > >I don't know. Is there any use case? >>> > >(sure, it is trivially implemented) >> > >> > >> >Well, I'd expect it to simply be there. I had not thought of >> >other usecases for the transformdict itself - > Well, it is not here for dict, set, etc. But they don't change the keys (although numbers have different representations on occasion). One use of transformdict might be to allow use of non-hashable items as keys, by extracting an actual key from the internals of the non-hashable item. The key may be sufficiently unique to enable use of the dict structure for lookups, but it would certainly be handy to obtain the actual item again. Without a canonical lookup feature, one would be forced to also include the key as part of the value, or some such hack. I also thought Jo?o's example was a very practical reason to have the canonical lookup feature, by some name or another. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130910/5dde4a29/attachment.html> From ethan at stoneleaf.us Wed Sep 11 03:19:20 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 10 Sep 2013 18:19:20 -0700 Subject: [Python-Dev] Add a "transformdict" to collections In-Reply-To: <522FB8CF.5020400@trueblade.com> References: <20130910112832.75ed6c8b@pitrou.net> <CAHVvXxQRO-V7002jhVPb5n46yS0b1MeDfNYVbMsVowTZm39NHQ@mail.gmail.com> <CADiSq7ec=+GGe7wQREozLyKZTR6pR=yxb71FG1Zv9b3oYWRrqw@mail.gmail.com> <CACac1F_Hr8U_4n27NvwH1=rN0ggFPx8MqJLbAh=GDXy_VNFLbA@mail.gmail.com> <20130910143558.5677b55a@pitrou.net> <522F1411.8010109@v.loewis.de> <20130910203140.1e8c1008@fsol> <CACac1F_mm97XUSde62vWtD9K643QkpUCedYi0YAhKJvOG39MgQ@mail.gmail.com> <CAH0mxTQBPy4ARJDXYNCCyGs5Oqjb0puL5gtL38FpjhZVcRy6LQ@mail.gmail.com> <20130910230607.701bba16@fsol> <CAH0mxTSTzEkhgXuuzZG-XMn+SV6rqEQNGpp9Xyf96zRBSBh4+A@mail.gmail.com> <20130910234659.39284dab@fsol> <522F9968.5030108@mrabarnett.plus.com> <522F9A9D.9010008@stoneleaf.us> <522FB8CF.5020400@trueblade.com> Message-ID: <522FC518.6060604@stoneleaf.us> On 09/10/2013 05:26 PM, Eric V. Smith wrote: > On 9/10/2013 6:18 PM, Ethan Furman wrote: >> On 09/10/2013 03:12 PM, MRAB wrote: >>> On 10/09/2013 22:46, Antoine Pitrou wrote: >>>> On Tue, 10 Sep 2013 18:44:20 -0300 >>>> "Joao S. O. Bueno" <jsbueno at python.org.br> wrote: >>>>> On 10 September 2013 18:06, Antoine Pitrou <solipsis at pitrou.net> wrote: >>>>>> On Tue, 10 Sep 2013 17:38:26 -0300 >>>>>> "Joao S. O. Bueno" <jsbueno at python.org.br> wrote: >>>>>>> On 10 September 2013 16:08, Paul Moore <p.f.moore at gmail.com> wrote: >>>>>>>> If you provide "retain the last", I can't see any obvious way of >>>>>>>> implementing "retain the first" in application code without in >>>>> effect >>>>>>>> reimplementing the class. >>>>>>> >>>>>>> Which reminds one - this class should obviously have a method for >>>>>>> retrivieng the original key value, given a matching key - >>>>>>> >>>>>>> d.canonical('foo') -> 'Foo' >>>>>> >>>>>> I don't know. Is there any use case? >>>>>> (sure, it is trivially implemented) >>>>> >>>>> Well, I'd expect it to simply be there. I had not thought of >>>>> other usecases for the transformdict itself - >>>> >>> I had the same thought. >>> >>>> Well, it is not here for dict, set, etc. >>>> >>> In those cases the key in the dict == the key you're looking for. >> >> With the exception of numbers, of course (float vs int vs Decimal, etc.). > > They'd still be ==, wouldn't they? Yes, but for presentation purposes not identical. -- ~Ethan~ From jsbueno at python.org.br Wed Sep 11 04:25:03 2013 From: jsbueno at python.org.br (Joao S. O. Bueno) Date: Tue, 10 Sep 2013 23:25:03 -0300 Subject: [Python-Dev] Add a "transformdict" to collections In-Reply-To: <20130910234659.39284dab@fsol> References: <20130910112832.75ed6c8b@pitrou.net> <CAHVvXxQRO-V7002jhVPb5n46yS0b1MeDfNYVbMsVowTZm39NHQ@mail.gmail.com> <CADiSq7ec=+GGe7wQREozLyKZTR6pR=yxb71FG1Zv9b3oYWRrqw@mail.gmail.com> <CACac1F_Hr8U_4n27NvwH1=rN0ggFPx8MqJLbAh=GDXy_VNFLbA@mail.gmail.com> <20130910143558.5677b55a@pitrou.net> <522F1411.8010109@v.loewis.de> <20130910203140.1e8c1008@fsol> <CACac1F_mm97XUSde62vWtD9K643QkpUCedYi0YAhKJvOG39MgQ@mail.gmail.com> <CAH0mxTQBPy4ARJDXYNCCyGs5Oqjb0puL5gtL38FpjhZVcRy6LQ@mail.gmail.com> <20130910230607.701bba16@fsol> <CAH0mxTSTzEkhgXuuzZG-XMn+SV6rqEQNGpp9Xyf96zRBSBh4+A@mail.gmail.com> <20130910234659.39284dab@fsol> Message-ID: <CAH0mxTQ6aF5HKdpKtSmfYNLiEJ3ZX-CTEfTRjcsVZVcWKPJUMg@mail.gmail.com> On 10 September 2013 18:46, Antoine Pitrou <solipsis at pitrou.net> wrote: > On Tue, 10 Sep 2013 18:44:20 -0300 > "Joao S. O. Bueno" <jsbueno at python.org.br> wrote: >> On 10 September 2013 18:06, Antoine Pitrou <solipsis at pitrou.net> wrote: >> > On Tue, 10 Sep 2013 17:38:26 -0300 >> > "Joao S. O. Bueno" <jsbueno at python.org.br> wrote: >> >> On 10 September 2013 16:08, Paul Moore <p.f.moore at gmail.com> wrote: >> >> > If you provide "retain the last", I can't see any obvious way of >> >> > implementing "retain the first" in application code without in effect >> >> > reimplementing the class. >> >> >> >> Which reminds one - this class should obviously have a method for >> >> retrivieng the original key value, given a matching key - >> >> >> >> d.canonical('foo') -> 'Foo' >> > >> > I don't know. Is there any use case? >> > (sure, it is trivially implemented) >> >> >> Well, I'd expect it to simply be there. I had not thought of >> other usecases for the transformdict itself - > > Well, it is not here for dict, set, etc. For the simple motive that once you retrieve or find that an element is contained in one of these classes, you already have the canonical key. :-) > >> For example, in latim languages it is common to want >> accented letters to match their unaccented counterparts >> - pick my own first name "Jo?o" - if I'd use a transform to strip >> the diactriticals, and have an user input "joao" - it would match, >> as intended - but I would not be able to retrieve the accented version >> without re-implementing the class behavior. > > Interesting example, thanks. > > Regards > > Antoine. > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/jsbueno%40python.org.br From raymond.hettinger at gmail.com Wed Sep 11 04:40:36 2013 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Tue, 10 Sep 2013 21:40:36 -0500 Subject: [Python-Dev] Add a "transformdict" to collections In-Reply-To: <20130910112832.75ed6c8b@pitrou.net> References: <20130910112832.75ed6c8b@pitrou.net> Message-ID: <011DB128-7308-40C0-BE93-A691A26B2C1B@gmail.com> On Sep 10, 2013, at 4:28 AM, Antoine Pitrou <solipsis at pitrou.net> wrote: > In http://bugs.python.org/issue18986 I proposed adding a new mapping > type to the collections module. I would *really* like for this to start outside the standard library. It needs to mature with user feedback before being dumped in the collections module (which was never intended to be a giant pile of every collection a person could think of). Adding yet more dictionary variants is an example of way-too-many-ways-to-do-it. Raymond -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130910/edc1a2fa/attachment-0001.html> From solipsis at pitrou.net Wed Sep 11 10:27:32 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 11 Sep 2013 10:27:32 +0200 Subject: [Python-Dev] Add a "transformdict" to collections References: <20130910112832.75ed6c8b@pitrou.net> <011DB128-7308-40C0-BE93-A691A26B2C1B@gmail.com> Message-ID: <20130911102732.2cb268ca@pitrou.net> Le Tue, 10 Sep 2013 21:40:36 -0500, Raymond Hettinger <raymond.hettinger at gmail.com> a ?crit : > > On Sep 10, 2013, at 4:28 AM, Antoine Pitrou <solipsis at pitrou.net> > wrote: > > > In http://bugs.python.org/issue18986 I proposed adding a new mapping > > type to the collections module. > > I would *really* like for this to start outside the standard library. From a quick search: - case-insensitive dicts (use cases and implementation attempts): http://twistedmatrix.com/documents/current/api/twisted.python.util.InsensitiveDict.html https://mail.python.org/pipermail/python-list/2013-May/647243.html https://mail.python.org/pipermail/python-list/2005-April/296208.html https://mail.python.org/pipermail/python-list/2004-June/241748.html http://bugs.python.org/msg197376 http://stackoverflow.com/a/2082169 http://stackoverflow.com/a/3296782 http://code.activestate.com/recipes/66315-case-insensitive-dictionary/ https://gist.github.com/babakness/3901174 http://www.wikier.org/blog/key-insensitive-dictionary-in-python http://en.sharejs.com/python/14534 http://www.voidspace.org.uk/python/archive.shtml#caseless - identity dicts: https://mail.python.org/pipermail/python-ideas/2010-May/007235.html http://www.gossamer-threads.com/lists/python/python/209527 Python's own pickle module: http://hg.python.org/cpython/file/0e70bf1f32a3/Lib/pickle.py#l234 > It needs to mature with user feedback before being dumped > in the collections module (which was never intended to be a > giant pile of every collection a person could think of). Well, thanks for the reminder, I was *indeed* going to dump all the collections I could think of in the collections module :-) (that would have been embarassing!) Seriously, I'm curious: what needs to mature, according to you? The proposed collection is a plain MutableMapping with the single addition of transforming the key on lookup. The use cases are well-defined and well-known. If you have any concrete questions or concerns then please offer them here. > Adding yet more dictionary variants is an example of > way-too-many-ways-to-do-it. So what is your proposal for what is definitely (see examples above, and this thread's and the tracker's responses) a very common need? Keep letting people write suboptimal, incomplete, buggy versions of the same thing? Thanks Antoine. From storchaka at gmail.com Wed Sep 11 11:38:13 2013 From: storchaka at gmail.com (Serhiy Storchaka) Date: Wed, 11 Sep 2013 12:38:13 +0300 Subject: [Python-Dev] Add a "transformdict" to collections In-Reply-To: <20130910112832.75ed6c8b@pitrou.net> References: <20130910112832.75ed6c8b@pitrou.net> Message-ID: <l0pdl9$sc2$1@ger.gmane.org> There is a question about specifying the transform function. There are three ways to do this: 1. Positional argument of the constructor. d = TransformDict(str.casefold, Foo=5) 2. Subclassing. class CaseInsensitiveDict(TransformDict): def transform(self, key): return key.casefold() d = CaseInsensitiveDict(Foo=5) 3. Type generator. d = TransformDict(str.casefold)(Foo=5) Second method looks a little simpler to me from implementation side (What if you call __init__ for already initialized object? What if you used the object before calling __init__?). Third method allows you to customize other aspects of dict behavior (combine OrderedDict, defaultdict,..). First method looks less cumbersome from user side at first look. But how many different transform functions you use? Aren't they deserve named classes? From solipsis at pitrou.net Wed Sep 11 11:52:59 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 11 Sep 2013 11:52:59 +0200 Subject: [Python-Dev] Add a "transformdict" to collections References: <20130910112832.75ed6c8b@pitrou.net> <l0pdl9$sc2$1@ger.gmane.org> Message-ID: <20130911115259.57be1a82@pitrou.net> Le Wed, 11 Sep 2013 12:38:13 +0300, Serhiy Storchaka <storchaka at gmail.com> a ?crit : > 2. Subclassing. > > class CaseInsensitiveDict(TransformDict): > def transform(self, key): > return key.casefold() > d = CaseInsensitiveDict(Foo=5) I thought about this first, and then I remembered that python-dev isn't generally very keen on subclassing-based APIs :-) > 3. Type generator. > > d = TransformDict(str.casefold)(Foo=5) > [...] > > Third method allows you to customize other aspects of dict behavior > (combine OrderedDict, defaultdict,..). Well, no, it's not that easy. Especially since OrderedDict and defaultdict weren't written with combination in mind. Regards Antoine. From storchaka at gmail.com Wed Sep 11 12:09:36 2013 From: storchaka at gmail.com (Serhiy Storchaka) Date: Wed, 11 Sep 2013 13:09:36 +0300 Subject: [Python-Dev] Need testing audio files Message-ID: <l0pfg4$i7l$1@ger.gmane.org> I work on enhancement of audio modules testing [1], and I need free (in both senses) small sample audio files in different formats. We already have audiotest.au (mono, and sunau has a bug in processing multichannel files [2]) and Sine-1000Hz-300ms.aif, but this is not enough. I have generated a pack of files like Sine-1000Hz-300ms.aif by Python, it is enough for regression testing but only if current implementation is correct. I found some collections of sample files at [3], but I'm not sure about copyright, and perhaps they are a little too big. In ideal it should be one high-quality (float64?) multichannel (5+1?) but short master file and it's lower-quality copies made by third-party tools. In ideal the content should be related to Python. [1] http://bugs.python.org/issue18919 [2] http://bugs.python.org/issue18950 [3] http://www-mmsp.ece.mcgill.ca/documents/AudioFormats/ From storchaka at gmail.com Wed Sep 11 12:37:53 2013 From: storchaka at gmail.com (Serhiy Storchaka) Date: Wed, 11 Sep 2013 13:37:53 +0300 Subject: [Python-Dev] Add a "transformdict" to collections In-Reply-To: <20130911115259.57be1a82@pitrou.net> References: <20130910112832.75ed6c8b@pitrou.net> <l0pdl9$sc2$1@ger.gmane.org> <20130911115259.57be1a82@pitrou.net> Message-ID: <l0pi51$hkq$1@ger.gmane.org> 11.09.13 12:52, Antoine Pitrou ???????(??): > Le Wed, 11 Sep 2013 12:38:13 +0300, > Serhiy Storchaka <storchaka at gmail.com> a ?crit : >> 2. Subclassing. >> >> class CaseInsensitiveDict(TransformDict): >> def transform(self, key): >> return key.casefold() >> d = CaseInsensitiveDict(Foo=5) > > I thought about this first, and then I remembered that python-dev isn't > generally very keen on subclassing-based APIs :-) Why? This way looks most natural to me. >> 3. Type generator. >> >> d = TransformDict(str.casefold)(Foo=5) >> > [...] >> >> Third method allows you to customize other aspects of dict behavior >> (combine OrderedDict, defaultdict,..). > > Well, no, it's not that easy. Especially since OrderedDict and > defaultdict weren't written with combination in mind. They can be rewritten. Actually the defaultdict is just simple wrapper around existing functionality of the __missing__ method. We can add the __transform__ method directly in the dict class. I think it will significant (2-3x) decrease a size of needed C code (no need in defining new type with constructors/destructors/etc, only lookup_transform). From solipsis at pitrou.net Wed Sep 11 13:07:15 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 11 Sep 2013 13:07:15 +0200 Subject: [Python-Dev] Add a "transformdict" to collections References: <20130910112832.75ed6c8b@pitrou.net> <l0pdl9$sc2$1@ger.gmane.org> <20130911115259.57be1a82@pitrou.net> <l0pi51$hkq$1@ger.gmane.org> Message-ID: <20130911130715.2f834a21@pitrou.net> Le Wed, 11 Sep 2013 13:37:53 +0300, Serhiy Storchaka <storchaka at gmail.com> a ?crit : > > Actually the defaultdict is just simple wrapper around existing > functionality of the __missing__ method. We can add the __transform__ > method directly in the dict class. I think it will significant (2-3x) > decrease a size of needed C code (no need in defining new type with > constructors/destructors/etc, only lookup_transform). Right now I am not proposing any C implementation of TransformDict. That could be added, but it's not what I'm pushing for here :-) But I don't think the "type generator" API would be easier to implement in C, anyway. Regards Antoine. From skip at pobox.com Wed Sep 11 13:08:25 2013 From: skip at pobox.com (Skip Montanaro) Date: Wed, 11 Sep 2013 06:08:25 -0500 Subject: [Python-Dev] Add a "transformdict" to collections In-Reply-To: <20130911102732.2cb268ca@pitrou.net> References: <20130910112832.75ed6c8b@pitrou.net> <011DB128-7308-40C0-BE93-A691A26B2C1B@gmail.com> <20130911102732.2cb268ca@pitrou.net> Message-ID: <CANc-5Uw-66ZPYC=mOu4z+vHU74qHY4QDWq2bCLDCFBku+cAqcg@mail.gmail.com> > Seriously, I'm curious: what needs to mature, according to you? In my mind, its availability on PyPI along with demonstrated use in the wild (plus corresponding votes to demonstrate that people use/like it) would help. That you can find several implementations at this doesn't mean it's necessarily worth adding to the std lib. Once in, it is very difficult to evict something that is later deemed not to have belonged in the std lib, so I think some extra scrutiny is worthwhile. Is there some obvious advantage to having a single API for this available to all Python applications? Are the semantics well-defined (do all the implementations you cited offer basically the same semantics)? The discussion so far here suggest that the best semantics might not be completely settled. (I still don't care for the name. "Transform" != "case folding" in my mind. A quick scan of your links suggests most people think something like "cidict" or "CaseInsensitiveDict" would be more descriptive.) Skip From storchaka at gmail.com Wed Sep 11 13:23:21 2013 From: storchaka at gmail.com (Serhiy Storchaka) Date: Wed, 11 Sep 2013 14:23:21 +0300 Subject: [Python-Dev] Add a "transformdict" to collections In-Reply-To: <20130911130715.2f834a21@pitrou.net> References: <20130910112832.75ed6c8b@pitrou.net> <l0pdl9$sc2$1@ger.gmane.org> <20130911115259.57be1a82@pitrou.net> <l0pi51$hkq$1@ger.gmane.org> <20130911130715.2f834a21@pitrou.net> Message-ID: <l0pjq9$423$1@ger.gmane.org> 11.09.13 14:07, Antoine Pitrou ???????(??): > But I don't think the "type generator" API would be easier to implement > in C, anyway. No, I mean subclassing approach. From rdmurray at bitdance.com Wed Sep 11 13:57:06 2013 From: rdmurray at bitdance.com (R. David Murray) Date: Wed, 11 Sep 2013 07:57:06 -0400 Subject: [Python-Dev] Add a "transformdict" to collections In-Reply-To: <CANc-5Uw-66ZPYC=mOu4z+vHU74qHY4QDWq2bCLDCFBku+cAqcg@mail.gmail.com> References: <20130910112832.75ed6c8b@pitrou.net> <011DB128-7308-40C0-BE93-A691A26B2C1B@gmail.com> <20130911102732.2cb268ca@pitrou.net> <CANc-5Uw-66ZPYC=mOu4z+vHU74qHY4QDWq2bCLDCFBku+cAqcg@mail.gmail.com> Message-ID: <20130911115706.C2D98250925@webabinitio.net> On Wed, 11 Sep 2013 06:08:25 -0500, Skip Montanaro <skip at pobox.com> wrote: > > Seriously, I'm curious: what needs to mature, according to you? > > In my mind, its availability on PyPI along with demonstrated use in > the wild (plus corresponding votes to demonstrate that people use/like > it) would help. That you can find several implementations at this > doesn't mean it's necessarily worth adding to the std lib. Once in, it > is very difficult to evict something that is later deemed not to have > belonged in the std lib, so I think some extra scrutiny is worthwhile. The problem is that it is hard to get traction on pypi for a "small" feature like this. Most people will just implement a special purpose version that solves their specific need (possibly in a somewhat broken fashion) rather than add yet another dependency. Our approach in cases like this has (I think) been to learn from the existing implementations and build our own based on that (isn't that what happened with OrderedDict?) > Is there some obvious advantage to having a single API for this > available to all Python applications? Are the semantics well-defined > (do all the implementations you cited offer basically the same > semantics)? The discussion so far here suggest that the best semantics > might not be completely settled. I think the only question was what happens to the key on assignment, and that has been settled. Well, I guess there's also a question of how you create one, but that's not a question a pypi version would be any help in resolving: we'd still bikeshed it upon addition to the stdlib. In fact, I would not be surprised if *all* of the bikeshedding we are doing now (except this bit :) would take place if an existing pypi module for this were proposed for addition. I think the bigger question is composition of different dict types, and that is again something that isn't going to be solved by a pypi module. > (I still don't care for the name. "Transform" != "case folding" in my > mind. A quick scan of your links suggests most people think something > like "cidict" or "CaseInsensitiveDict" would be more descriptive.) Except it is wider than that: the transform function can be anything, not just case folding. I suggested surjectiondict or ontodict, but Antoine didn't like those :) (I had to look up the terms...it's been a long time since I studied math.) --David From solipsis at pitrou.net Wed Sep 11 14:45:53 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 11 Sep 2013 14:45:53 +0200 Subject: [Python-Dev] Need testing audio files References: <l0pfg4$i7l$1@ger.gmane.org> Message-ID: <20130911144553.17909e76@pitrou.net> Le Wed, 11 Sep 2013 13:09:36 +0300, Serhiy Storchaka <storchaka at gmail.com> a ?crit : > I work on enhancement of audio modules testing [1], and I need free > (in both senses) small sample audio files in different formats. We > already have audiotest.au (mono, and sunau has a bug in processing > multichannel files [2]) and Sine-1000Hz-300ms.aif, but this is not > enough. I have generated a pack of files like Sine-1000Hz-300ms.aif > by Python, it is enough for regression testing but only if current > implementation is correct. If you want to edit, shorten, convert sounds between different formats, you can try Audacity, a free sound editor: http://audacity.sourceforge.net/ Regards Antoine. From victor.stinner at gmail.com Wed Sep 11 14:46:06 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 11 Sep 2013 14:46:06 +0200 Subject: [Python-Dev] Need testing audio files In-Reply-To: <l0pfg4$i7l$1@ger.gmane.org> References: <l0pfg4$i7l$1@ger.gmane.org> Message-ID: <CAMpsgwbpOLM8ghr91YYA+uVDGwqNjQdjoP70c0QNqBg4FEVxXQ@mail.gmail.com> Use your microphone, say "python" and save the file in your favorite file format. Try for example Audacity. I suppose that you don't need specific audio content and you don't need a huge file. Victor 2013/9/11 Serhiy Storchaka <storchaka at gmail.com>: > I work on enhancement of audio modules testing [1], and I need free (in both > senses) small sample audio files in different formats. We already have > audiotest.au (mono, and sunau has a bug in processing multichannel files > [2]) and Sine-1000Hz-300ms.aif, but this is not enough. I have generated a > pack of files like Sine-1000Hz-300ms.aif by Python, it is enough for > regression testing but only if current implementation is correct. > > I found some collections of sample files at [3], but I'm not sure about > copyright, and perhaps they are a little too big. > > In ideal it should be one high-quality (float64?) multichannel (5+1?) but > short master file and it's lower-quality copies made by third-party tools. > In ideal the content should be related to Python. > > [1] http://bugs.python.org/issue18919 > [2] http://bugs.python.org/issue18950 > [3] http://www-mmsp.ece.mcgill.ca/documents/AudioFormats/ > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com From ncoghlan at gmail.com Wed Sep 11 14:47:12 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 11 Sep 2013 22:47:12 +1000 Subject: [Python-Dev] Add a "transformdict" to collections In-Reply-To: <20130911115706.C2D98250925@webabinitio.net> References: <20130910112832.75ed6c8b@pitrou.net> <011DB128-7308-40C0-BE93-A691A26B2C1B@gmail.com> <20130911102732.2cb268ca@pitrou.net> <CANc-5Uw-66ZPYC=mOu4z+vHU74qHY4QDWq2bCLDCFBku+cAqcg@mail.gmail.com> <20130911115706.C2D98250925@webabinitio.net> Message-ID: <CADiSq7cjF6RGdrobJh1=f0etG3rVXARRUSCOJXzina0u1rkJTQ@mail.gmail.com> On 11 September 2013 21:57, R. David Murray <rdmurray at bitdance.com> wrote: > Except it is wider than that: the transform function can be anything, > not just case folding. > > I suggested surjectiondict or ontodict, but Antoine didn't like those :) > (I had to look up the terms...it's been a long time since I studied > math.) I'll join the chorus requesting that this live on PyPI for a while first. I think this is a case similar to what happened with contextlib.ExitStack: I'm not sure if anyone actually *used* contextlib2 for anything significant (if they did, they didn't tell me about it), but just going through the process of properly documenting, publishing and testing it as a relatively independent project forced *me* to think through the design. Add in a couple of interesting conversation with Brandon Rhodes and others about the API design, and the end result was a *vast* improvement over what originally went up on PyPI. The barriers to making use of PyPI libraries are also falling with time, so a solid implementation may still see adoption, even if it's in the form of copy-and-paste programming rather than actual dependencies. I think there are also additional API decisions to be made beyond just the one about how to declare the mapping function, related to how to get the mapped key values *out*, as well as how to find out whether or not two potential key values map to the same actual key. As my preferred bikeshed colour, I'm going to suggest "MappedKeyDict" (using a similar naming style to OrderedDict), since the purpose of the container is to map the domain of the supplied keys to a different range before doing the value lookup. Suggested additional methods: md.map_key(key) # Applies the mapping function to the supplied key md.mapped_keys() # Like keys(), but with the key mapping function applied md.mapped_items() # Like items(), but with the key mapping function applied Another (more dubious) possible method: md.same_key(key1, key2) # "md.map_key(key1) == md.map_key(key2)" Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From solipsis at pitrou.net Wed Sep 11 15:04:23 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 11 Sep 2013 15:04:23 +0200 Subject: [Python-Dev] Add a "transformdict" to collections References: <20130910112832.75ed6c8b@pitrou.net> <011DB128-7308-40C0-BE93-A691A26B2C1B@gmail.com> <20130911102732.2cb268ca@pitrou.net> <CANc-5Uw-66ZPYC=mOu4z+vHU74qHY4QDWq2bCLDCFBku+cAqcg@mail.gmail.com> <20130911115706.C2D98250925@webabinitio.net> <CADiSq7cjF6RGdrobJh1=f0etG3rVXARRUSCOJXzina0u1rkJTQ@mail.gmail.com> Message-ID: <20130911150423.62f77bf2@pitrou.net> Le Wed, 11 Sep 2013 22:47:12 +1000, Nick Coghlan <ncoghlan at gmail.com> a ?crit : > > I'll join the chorus requesting that this live on PyPI for a while > first. > > I think this is a case similar to what happened with > contextlib.ExitStack: I'm not sure if anyone actually *used* > contextlib2 for anything significant (if they did, they didn't tell me > about it), but just going through the process of properly documenting, > publishing and testing it as a relatively independent project forced > *me* to think through the design. ExitStack was quite a new thing, API-wise. The proposal here is to generalize something which already exists in various forms all over the Internet, and respecting a well-known API, MutableMapping. There are not many possible APIs to create case-insensitive dicts, or identity dicts. The only API contention until now has been about whether the first or last key would be retained (which settled on the former), and Serhiy's three instantiation schemes. The additional methods you suggest may be nice to have, but they are not necessary and (deliberately) not part of the original proposal. That is, they can be grafted later on. > As my preferred bikeshed colour, I'm going to suggest "MappedKeyDict" > (using a similar naming style to OrderedDict), Ha, another colour! Thanks :-) Regards Antoine. From steve at pearwood.info Wed Sep 11 15:40:32 2013 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 11 Sep 2013 23:40:32 +1000 Subject: [Python-Dev] Add a "transformdict" to collections In-Reply-To: <CANc-5Uw-66ZPYC=mOu4z+vHU74qHY4QDWq2bCLDCFBku+cAqcg@mail.gmail.com> References: <20130910112832.75ed6c8b@pitrou.net> <011DB128-7308-40C0-BE93-A691A26B2C1B@gmail.com> <20130911102732.2cb268ca@pitrou.net> <CANc-5Uw-66ZPYC=mOu4z+vHU74qHY4QDWq2bCLDCFBku+cAqcg@mail.gmail.com> Message-ID: <20130911134032.GA16820@ando> On Wed, Sep 11, 2013 at 06:08:25AM -0500, Skip Montanaro wrote: > (I still don't care for the name. "Transform" != "case folding" in my > mind. A quick scan of your links suggests most people think something > like "cidict" or "CaseInsensitiveDict" would be more descriptive.) But the proposal is not for a case-insensitive dict. It is more general than that, with case-insensitivity just one specific use-case for such a transformative dict. Arguably the most natural, or at least obvious, such transformation, but there are others. I have code that does something like this: MAPPING = {'spam': 23, 'ham': 42, 'eggs': 17} result = MAPPING[key.strip()] # later... answer = MAPPING[key] # Oops, forgot to strip! This is broken. Using Antoine's proposal: MAPPING = TransformDict(str.strip) MAPPING.update({'spam': 23, 'ham': 42, 'eggs': 17}) result = MAPPING[key] # later... answer = MAPPING[key] # Fine now. so that the mapping handles stripping the keys, not the caller. This isn't just about strings, and certainly not just case-insensitivity. -- Steven From stephen at xemacs.org Wed Sep 11 15:50:08 2013 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Wed, 11 Sep 2013 22:50:08 +0900 Subject: [Python-Dev] Add a "transformdict" to collections In-Reply-To: <20130911150423.62f77bf2@pitrou.net> References: <20130910112832.75ed6c8b@pitrou.net> <011DB128-7308-40C0-BE93-A691A26B2C1B@gmail.com> <20130911102732.2cb268ca@pitrou.net> <CANc-5Uw-66ZPYC=mOu4z+vHU74qHY4QDWq2bCLDCFBku+cAqcg@mail.gmail.com> <20130911115706.C2D98250925@webabinitio.net> <CADiSq7cjF6RGdrobJh1=f0etG3rVXARRUSCOJXzina0u1rkJTQ@mail.gmail.com> <20130911150423.62f77bf2@pitrou.net> Message-ID: <87txhr79in.fsf@uwakimon.sk.tsukuba.ac.jp> Antoine Pitrou writes: > ExitStack was quite a new thing, API-wise. The proposal here is to > generalize something which already exists in various forms all over the > Internet, and respecting a well-known API, MutableMapping. What Nick said was "I was too close to the design, and time and a very few good comments greatly improved the product." That's worth considering as generic advice rather than for the specifics of the case he faced compared to yours. Which modules in the stdlib would benefit from rewriting using "transformdict"? How about on PyPI? From brett at python.org Wed Sep 11 15:54:44 2013 From: brett at python.org (Brett Cannon) Date: Wed, 11 Sep 2013 09:54:44 -0400 Subject: [Python-Dev] Need testing audio files In-Reply-To: <CAMpsgwbpOLM8ghr91YYA+uVDGwqNjQdjoP70c0QNqBg4FEVxXQ@mail.gmail.com> References: <l0pfg4$i7l$1@ger.gmane.org> <CAMpsgwbpOLM8ghr91YYA+uVDGwqNjQdjoP70c0QNqBg4FEVxXQ@mail.gmail.com> Message-ID: <CAP1=2W4a=Mzt=DVj4qeBZG8gxQ8iaJ7RUPhRb6-YvfTC2TbYfA@mail.gmail.com> On Wed, Sep 11, 2013 at 8:46 AM, Victor Stinner <victor.stinner at gmail.com>wrote: > Use your microphone, say "python" and save the file in your favorite > file format. Try for example Audacity. I suppose that you don't need > specific audio content and you don't need a huge file. > And if you don't want your voice in the test suite forever you can see if you can convince Guido to say "Python" into a microphone. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130911/236cf5f2/attachment-0001.html> From steve at pearwood.info Wed Sep 11 15:56:01 2013 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 11 Sep 2013 23:56:01 +1000 Subject: [Python-Dev] Add a "transformdict" to collections In-Reply-To: <CADiSq7cjF6RGdrobJh1=f0etG3rVXARRUSCOJXzina0u1rkJTQ@mail.gmail.com> References: <20130910112832.75ed6c8b@pitrou.net> <011DB128-7308-40C0-BE93-A691A26B2C1B@gmail.com> <20130911102732.2cb268ca@pitrou.net> <CANc-5Uw-66ZPYC=mOu4z+vHU74qHY4QDWq2bCLDCFBku+cAqcg@mail.gmail.com> <20130911115706.C2D98250925@webabinitio.net> <CADiSq7cjF6RGdrobJh1=f0etG3rVXARRUSCOJXzina0u1rkJTQ@mail.gmail.com> Message-ID: <20130911135601.GB16820@ando> On Wed, Sep 11, 2013 at 10:47:12PM +1000, Nick Coghlan wrote: > I'll join the chorus requesting that this live on PyPI for a while first. Another alternative would be ActiveState's Python recipes: http://code.activestate.com/recipes/langs/python which is a lot lighter weight than creating a package on PyPI. I believe that ChainMap, namedtuple, groupby, lru_cache and even math.fsum started life on ActiveState. -- Steven From victor.stinner at gmail.com Wed Sep 11 15:58:35 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 11 Sep 2013 15:58:35 +0200 Subject: [Python-Dev] Add a "transformdict" to collections In-Reply-To: <20130911134032.GA16820@ando> References: <20130910112832.75ed6c8b@pitrou.net> <011DB128-7308-40C0-BE93-A691A26B2C1B@gmail.com> <20130911102732.2cb268ca@pitrou.net> <CANc-5Uw-66ZPYC=mOu4z+vHU74qHY4QDWq2bCLDCFBku+cAqcg@mail.gmail.com> <20130911134032.GA16820@ando> Message-ID: <CAMpsgwbWF_7-EDKoQZz61BNk3mVNieMkVCquwweMuG=HSK4RAQ@mail.gmail.com> 2013/9/11 Steven D'Aprano <steve at pearwood.info>: > But the proposal is not for a case-insensitive dict. It is more general > than that, with case-insensitivity just one specific use-case for such > a transformative dict. Arguably the most natural, or at least obvious, > such transformation, but there are others. > > I have code that does something like this: > > MAPPING = {'spam': 23, 'ham': 42, 'eggs': 17} > result = MAPPING[key.strip()] > # later... > answer = MAPPING[key] # Oops, forgot to strip! This is broken. For this use case, you should not keep the key unchanged, but it's better to store the stripped key (so MAPPING.keys() gives you the expected result). The transformdict type proposed by Antoine cannot be used for this use case. The os.environ mapping uses a subprocess of MutableMapping which accepts 4 functions: encoder/decoder for the key and encoder/decoder for the value. Such type is even more generic. transformdict cannot replace os._Environ. (Or do you prefer to store the ' eggs' key?) Victor From barry at python.org Wed Sep 11 16:10:53 2013 From: barry at python.org (Barry Warsaw) Date: Wed, 11 Sep 2013 10:10:53 -0400 Subject: [Python-Dev] Need testing audio files In-Reply-To: <l0pfg4$i7l$1@ger.gmane.org> References: <l0pfg4$i7l$1@ger.gmane.org> Message-ID: <20130911101053.71deed3b@anarchist> On Sep 11, 2013, at 01:09 PM, Serhiy Storchaka wrote: >In ideal it should be one high-quality (float64?) multichannel (5+1?) but >short master file and it's lower-quality copies made by third-party tools. In >ideal the content should be related to Python. I have some pro-audio recording capabilities and would be happy to generate some copyright-donated clips for Python. Please contact me off-list if needed. -Barry From phd at phdru.name Wed Sep 11 16:10:22 2013 From: phd at phdru.name (Oleg Broytman) Date: Wed, 11 Sep 2013 18:10:22 +0400 Subject: [Python-Dev] Need testing audio files In-Reply-To: <CAP1=2W4a=Mzt=DVj4qeBZG8gxQ8iaJ7RUPhRb6-YvfTC2TbYfA@mail.gmail.com> References: <l0pfg4$i7l$1@ger.gmane.org> <CAMpsgwbpOLM8ghr91YYA+uVDGwqNjQdjoP70c0QNqBg4FEVxXQ@mail.gmail.com> <CAP1=2W4a=Mzt=DVj4qeBZG8gxQ8iaJ7RUPhRb6-YvfTC2TbYfA@mail.gmail.com> Message-ID: <20130911141022.GA30682@iskra.aviel.ru> On Wed, Sep 11, 2013 at 09:54:44AM -0400, Brett Cannon <brett at python.org> wrote: > On Wed, Sep 11, 2013 at 8:46 AM, Victor Stinner <victor.stinner at gmail.com>wrote: > And if you don't want your voice in the test suite forever you can see if > you can convince Guido to say "Python" into a microphone. Wouldn't his name be enough? http://www.python.org/~guido/guido.au Oleg. -- Oleg Broytman http://phdru.name/ phd at phdru.name Programmers don't die, they just GOSUB without RETURN. From ethan at stoneleaf.us Wed Sep 11 16:04:17 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 11 Sep 2013 07:04:17 -0700 Subject: [Python-Dev] importance of dir Message-ID: <52307861.6010102@stoneleaf.us> http://docs.python.org/3/library/functions.html#dir: > > Note: Because dir() is supplied primarily as a convenience for > use at an interactive prompt [...] I suspect this comment is out of date, as there are two functions in the inspect module that rely on dir(), which also means that help indirectly relies on dir(). And we also now have the __dir__ method that we can use to customize what gets returned. > [...] more than it tries to supply a rigorously or consistently > defined set of names, [...] If this is still true, should inspect be relying on dir()? -- ~Ethan~ From solipsis at pitrou.net Wed Sep 11 16:32:23 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 11 Sep 2013 16:32:23 +0200 Subject: [Python-Dev] Add a "transformdict" to collections References: <20130910112832.75ed6c8b@pitrou.net> <011DB128-7308-40C0-BE93-A691A26B2C1B@gmail.com> <20130911102732.2cb268ca@pitrou.net> <CANc-5Uw-66ZPYC=mOu4z+vHU74qHY4QDWq2bCLDCFBku+cAqcg@mail.gmail.com> <20130911115706.C2D98250925@webabinitio.net> <CADiSq7cjF6RGdrobJh1=f0etG3rVXARRUSCOJXzina0u1rkJTQ@mail.gmail.com> <20130911150423.62f77bf2@pitrou.net> <87txhr79in.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <20130911163223.1334c00d@pitrou.net> Le Wed, 11 Sep 2013 22:50:08 +0900, "Stephen J. Turnbull" <stephen at xemacs.org> a ?crit : > Antoine Pitrou writes: > > > ExitStack was quite a new thing, API-wise. The proposal here is to > > generalize something which already exists in various forms all > > over the Internet, and respecting a well-known API, MutableMapping. > > What Nick said was "I was too close to the design, and time and a very > few good comments greatly improved the product." That's worth > considering as generic advice rather than for the specifics of the > case he faced compared to yours. "Time and comments" is why I'm posting a discussion thread here and waiting for responses. There are currently 34517 packages on PyPI. A new 34518th one won't magically get a lot of attention :-) (for example, I got very few comments when posting pathlib on PyPI - despite making several actual releases there -, compared to what I got from the PEP 428 discussion on python-ideas) Regards Antoine. From ethan at stoneleaf.us Wed Sep 11 16:34:32 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 11 Sep 2013 07:34:32 -0700 Subject: [Python-Dev] Add a "transformdict" to collections In-Reply-To: <CADiSq7cjF6RGdrobJh1=f0etG3rVXARRUSCOJXzina0u1rkJTQ@mail.gmail.com> References: <20130910112832.75ed6c8b@pitrou.net> <011DB128-7308-40C0-BE93-A691A26B2C1B@gmail.com> <20130911102732.2cb268ca@pitrou.net> <CANc-5Uw-66ZPYC=mOu4z+vHU74qHY4QDWq2bCLDCFBku+cAqcg@mail.gmail.com> <20130911115706.C2D98250925@webabinitio.net> <CADiSq7cjF6RGdrobJh1=f0etG3rVXARRUSCOJXzina0u1rkJTQ@mail.gmail.com> Message-ID: <52307F78.7050303@stoneleaf.us> On 09/11/2013 05:47 AM, Nick Coghlan wrote: > On 11 September 2013 21:57, R. David Murray <rdmurray at bitdance.com> wrote: >> Except it is wider than that: the transform function can be anything, >> not just case folding. >> >> I suggested surjectiondict or ontodict, but Antoine didn't like those :) >> (I had to look up the terms...it's been a long time since I studied >> math.) > > I'll join the chorus requesting that this live on PyPI for a while first. Personally, I would not add a PyPI dependency for a single object like transformdict. If we are that nervous about it we can make it provisional, but I don't see that as necessary. > I think this is a case similar to what happened with > contextlib.ExitStack: I'm not sure if anyone actually *used* > contextlib2 for anything significant (if they did, they didn't tell me > about it), but just going through the process of properly documenting, > publishing and testing it as a relatively independent project forced > *me* to think through the design. I had the opposite experience. Going through PyDev to get Enum was *far* more educational, informative, and useful than creating an independent package. Plus, transformdict is not that new or different from existing dicts, and most decisions (which key to keep? how to initialize?) can be answered by simply being consistent with what we already have. -- ~Ethan~ From ethan at stoneleaf.us Wed Sep 11 16:17:29 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 11 Sep 2013 07:17:29 -0700 Subject: [Python-Dev] Add a "transformdict" to collections In-Reply-To: <l0pdl9$sc2$1@ger.gmane.org> References: <20130910112832.75ed6c8b@pitrou.net> <l0pdl9$sc2$1@ger.gmane.org> Message-ID: <52307B79.1080002@stoneleaf.us> On 09/11/2013 02:38 AM, Serhiy Storchaka wrote: > There is a question about specifying the transform function. > > There are three ways to do this: > > 1. Positional argument of the constructor. > > d = TransformDict(str.casefold, Foo=5) This method follows the precedent of defaultdict: --> from collections import defaultdict --> l = defaultdict(list, foo=['ham','eggs']) --> l defaultdict(<class 'list'>, {'foo': ['ham', 'eggs']}) Without a compelling reason to change, we should keep it consistent. -- ~Ethan~ From ethan at stoneleaf.us Wed Sep 11 16:27:44 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 11 Sep 2013 07:27:44 -0700 Subject: [Python-Dev] Add a "transformdict" to collections In-Reply-To: <CAMpsgwbWF_7-EDKoQZz61BNk3mVNieMkVCquwweMuG=HSK4RAQ@mail.gmail.com> References: <20130910112832.75ed6c8b@pitrou.net> <011DB128-7308-40C0-BE93-A691A26B2C1B@gmail.com> <20130911102732.2cb268ca@pitrou.net> <CANc-5Uw-66ZPYC=mOu4z+vHU74qHY4QDWq2bCLDCFBku+cAqcg@mail.gmail.com> <20130911134032.GA16820@ando> <CAMpsgwbWF_7-EDKoQZz61BNk3mVNieMkVCquwweMuG=HSK4RAQ@mail.gmail.com> Message-ID: <52307DE0.9040208@stoneleaf.us> On 09/11/2013 06:58 AM, Victor Stinner wrote: > 2013/9/11 Steven D'Aprano <steve at pearwood.info>: >> But the proposal is not for a case-insensitive dict. It is more general >> than that, with case-insensitivity just one specific use-case for such >> a transformative dict. Arguably the most natural, or at least obvious, >> such transformation, but there are others. >> >> I have code that does something like this: >> >> MAPPING = {'spam': 23, 'ham': 42, 'eggs': 17} >> result = MAPPING[key.strip()] >> # later... >> answer = MAPPING[key] # Oops, forgot to strip! This is broken. > > For this use case, you should not keep the key unchanged, but it's > better to store the stripped key (so MAPPING.keys() gives you the > expected result). He isn't keeping the key unchanged (notice no white space in MAPPING), he's merely providing a function that will automatically strip the whitespace from key lookups. > The transformdict type proposed by Antoine cannot be > used for this use case. Yes, it can. --> from collections import transformdict --> MAPPING = transformdict(str.strip) --> MAPPING.update({'spam': 23, 'ham': 42, 'eggs': 17}) --> MAPPING transformdict(<method 'strip' of 'str' objects>, {'ham': 42, 'spam': 23, 'eggs': 17}) --> MAPPING[' eggs '] 17 -- ~Ethan~ From skip at pobox.com Wed Sep 11 17:17:37 2013 From: skip at pobox.com (Skip Montanaro) Date: Wed, 11 Sep 2013 10:17:37 -0500 Subject: [Python-Dev] importance of dir In-Reply-To: <52307861.6010102@stoneleaf.us> References: <52307861.6010102@stoneleaf.us> Message-ID: <CANc-5UzZbCjQwY=LUK+3a_4U_aLO4xfbXOxWJUYSt+yifzyVMw@mail.gmail.com> >> Note: Because dir() is supplied primarily as a convenience for >> use at an interactive prompt [...] This was always my interpretation of its intent. In fact, I use a customized dir() for my own needs which would probably break inspect (elides _-prefixed functions by default, notes modules or packages within other packages which haven't been imported yet). I never realized that module used it. Skip From ethan at stoneleaf.us Wed Sep 11 16:48:56 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 11 Sep 2013 07:48:56 -0700 Subject: [Python-Dev] Add a "transformdict" to collections In-Reply-To: <CAMpsgwbWF_7-EDKoQZz61BNk3mVNieMkVCquwweMuG=HSK4RAQ@mail.gmail.com> References: <20130910112832.75ed6c8b@pitrou.net> <011DB128-7308-40C0-BE93-A691A26B2C1B@gmail.com> <20130911102732.2cb268ca@pitrou.net> <CANc-5Uw-66ZPYC=mOu4z+vHU74qHY4QDWq2bCLDCFBku+cAqcg@mail.gmail.com> <20130911134032.GA16820@ando> <CAMpsgwbWF_7-EDKoQZz61BNk3mVNieMkVCquwweMuG=HSK4RAQ@mail.gmail.com> Message-ID: <523082D8.2040607@stoneleaf.us> On 09/11/2013 06:58 AM, Victor Stinner wrote: > > The os.environ mapping uses a subclass of MutableMapping which > accepts 4 functions: encoder/decoder for the key and encoder/decoder > for the value. Such type is even more generic. transformdict cannot > replace os._Environ. True, it's more generic -- but is it used more often? Or often enough to have those four functions be part of transformdict instead of just the one encodekey function? (I mean the idea of os._Environ, not that specific implementation.) Personally, I wouldn't mind having all four; for one thing, the name 'transformdict' would then be entirely appropriate. ;) -- ~Ethan~ From solipsis at pitrou.net Wed Sep 11 17:48:09 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 11 Sep 2013 17:48:09 +0200 Subject: [Python-Dev] Add a "transformdict" to collections References: <20130910112832.75ed6c8b@pitrou.net> <011DB128-7308-40C0-BE93-A691A26B2C1B@gmail.com> <20130911102732.2cb268ca@pitrou.net> <CANc-5Uw-66ZPYC=mOu4z+vHU74qHY4QDWq2bCLDCFBku+cAqcg@mail.gmail.com> <20130911134032.GA16820@ando> <CAMpsgwbWF_7-EDKoQZz61BNk3mVNieMkVCquwweMuG=HSK4RAQ@mail.gmail.com> <523082D8.2040607@stoneleaf.us> Message-ID: <20130911174809.2c12a7d0@pitrou.net> Le Wed, 11 Sep 2013 07:48:56 -0700, Ethan Furman <ethan at stoneleaf.us> a ?crit : > On 09/11/2013 06:58 AM, Victor Stinner wrote: > > > > The os.environ mapping uses a subclass of MutableMapping which > > accepts 4 functions: encoder/decoder for the key and encoder/decoder > > for the value. Such type is even more generic. transformdict cannot > > replace os._Environ. > > True, it's more generic -- but is it used more often? Or often > enough to have those four functions be part of transformdict instead > of just the one encodekey function? (I mean the idea of os._Environ, > not that specific implementation.) > > Personally, I wouldn't mind having all four; for one thing, the name > 'transformdict' would then be entirely appropriate. ;) The key decoder function is quite useless since the original key is retained. As for the value encoder/decoder, I don't really see the point: just store whichever value you want to retrieve later. The main point of transformdict is that the *lookup* is done using a different key than the user-provided key. Regards Antoine. From victor.stinner at gmail.com Wed Sep 11 17:49:25 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 11 Sep 2013 17:49:25 +0200 Subject: [Python-Dev] Add a "transformdict" to collections In-Reply-To: <52307DE0.9040208@stoneleaf.us> References: <20130910112832.75ed6c8b@pitrou.net> <011DB128-7308-40C0-BE93-A691A26B2C1B@gmail.com> <20130911102732.2cb268ca@pitrou.net> <CANc-5Uw-66ZPYC=mOu4z+vHU74qHY4QDWq2bCLDCFBku+cAqcg@mail.gmail.com> <20130911134032.GA16820@ando> <CAMpsgwbWF_7-EDKoQZz61BNk3mVNieMkVCquwweMuG=HSK4RAQ@mail.gmail.com> <52307DE0.9040208@stoneleaf.us> Message-ID: <CAMpsgwb1iB7fMKjYX5T3HR0E+3rmt4H055z2cjyJ-fMEjoEe6Q@mail.gmail.com> 2013/9/11 Ethan Furman <ethan at stoneleaf.us>: > He isn't keeping the key unchanged (notice no white space in MAPPING), he's > merely providing a function that will automatically strip the whitespace > from key lookups. transformdict keeps the key unchanged, see the first message: >>> d = transformdict(str.lower) >>> d['Foo'] = 5 >>> d['foo'] 5 >>> d['FOO'] 5 >>> list(d) ['Foo'] 'Foo' is stored as 'Foo', not as 'foo'. So for stripped keys: d=transformdict(str.strip); d[' abc ']; print(list(d)) should print "[' abc ']", not "['abc']". Is it the expected result? Victor From ethan at stoneleaf.us Wed Sep 11 18:03:18 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 11 Sep 2013 09:03:18 -0700 Subject: [Python-Dev] Add a "transformdict" to collections In-Reply-To: <CAMpsgwb1iB7fMKjYX5T3HR0E+3rmt4H055z2cjyJ-fMEjoEe6Q@mail.gmail.com> References: <20130910112832.75ed6c8b@pitrou.net> <011DB128-7308-40C0-BE93-A691A26B2C1B@gmail.com> <20130911102732.2cb268ca@pitrou.net> <CANc-5Uw-66ZPYC=mOu4z+vHU74qHY4QDWq2bCLDCFBku+cAqcg@mail.gmail.com> <20130911134032.GA16820@ando> <CAMpsgwbWF_7-EDKoQZz61BNk3mVNieMkVCquwweMuG=HSK4RAQ@mail.gmail.com> <52307DE0.9040208@stoneleaf.us> <CAMpsgwb1iB7fMKjYX5T3HR0E+3rmt4H055z2cjyJ-fMEjoEe6Q@mail.gmail.com> Message-ID: <52309446.6070809@stoneleaf.us> On 09/11/2013 08:49 AM, Victor Stinner wrote: > 2013/9/11 Ethan Furman <ethan at stoneleaf.us>: >> He isn't keeping the key unchanged (notice no white space in MAPPING), he's >> merely providing a function that will automatically strip the whitespace >> from key lookups. > > transformdict keeps the key unchanged, see the first message: > > >>> d = transformdict(str.lower) > >>> d['Foo'] = 5 > >>> d['foo'] > 5 > >>> d['FOO'] > 5 > >>> list(d) > ['Foo'] > > 'Foo' is stored as 'Foo', not as 'foo'. So for stripped keys: > > d=transformdict(str.strip); d[' abc ']; print(list(d)) > > should print "[' abc ']", not "['abc']". And indeed it does: Python 3.4.0a1+ (default:833246d42825+, Aug 31 2013, 14:17:59) [GCC 4.7.3] on linux Type "help", "copyright", "credits" or "license" for more information. --> from collections import transformdict --> d=transformdict(str.strip); d[' abc '] = 42; print(list(d)) [' abc '] > Is it the expected result? Yup! :) -- ~Ethan~ From ethan at stoneleaf.us Wed Sep 11 18:04:25 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 11 Sep 2013 09:04:25 -0700 Subject: [Python-Dev] Add a "transformdict" to collections In-Reply-To: <20130911174809.2c12a7d0@pitrou.net> References: <20130910112832.75ed6c8b@pitrou.net> <011DB128-7308-40C0-BE93-A691A26B2C1B@gmail.com> <20130911102732.2cb268ca@pitrou.net> <CANc-5Uw-66ZPYC=mOu4z+vHU74qHY4QDWq2bCLDCFBku+cAqcg@mail.gmail.com> <20130911134032.GA16820@ando> <CAMpsgwbWF_7-EDKoQZz61BNk3mVNieMkVCquwweMuG=HSK4RAQ@mail.gmail.com> <523082D8.2040607@stoneleaf.us> <20130911174809.2c12a7d0@pitrou.net> Message-ID: <52309489.6000008@stoneleaf.us> On 09/11/2013 08:48 AM, Antoine Pitrou wrote: > Le Wed, 11 Sep 2013 07:48:56 -0700, > Ethan Furman <ethan at stoneleaf.us> a ?crit : >> >> Personally, I wouldn't mind having all four; for one thing, the name >> 'transformdict' would then be entirely appropriate. ;) > > The key decoder function is quite useless since the original key is > retained. As for the value encoder/decoder, I don't really see the > point: just store whichever value you want to retrieve later. The main > point of transformdict is that the *lookup* is done using a different > key than the user-provided key. Very good points. I am convinced. -- ~Ethan~ From storchaka at gmail.com Wed Sep 11 17:37:24 2013 From: storchaka at gmail.com (Serhiy Storchaka) Date: Wed, 11 Sep 2013 18:37:24 +0300 Subject: [Python-Dev] Add a "transformdict" to collections In-Reply-To: <87txhr79in.fsf@uwakimon.sk.tsukuba.ac.jp> References: <20130910112832.75ed6c8b@pitrou.net> <011DB128-7308-40C0-BE93-A691A26B2C1B@gmail.com> <20130911102732.2cb268ca@pitrou.net> <CANc-5Uw-66ZPYC=mOu4z+vHU74qHY4QDWq2bCLDCFBku+cAqcg@mail.gmail.com> <20130911115706.C2D98250925@webabinitio.net> <CADiSq7cjF6RGdrobJh1=f0etG3rVXARRUSCOJXzina0u1rkJTQ@mail.gmail.com> <20130911150423.62f77bf2@pitrou.net> <87txhr79in.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <l0q655$ac0$1@ger.gmane.org> 11.09.13 16:50, Stephen J. Turnbull ???????(??): > Which modules in the stdlib would benefit from rewriting using > "transformdict"? How about on PyPI? At least _threading_local, cProfile, doctest, json, and perhaps future implementations of __sizeof__ for some classes would benefit from rewriting using IdentityDict. Unfortunately copy and pickle expose their replacements of IdentityDict to public and can't change them. I don't known anything about general TransformDict use cases. From storchaka at gmail.com Wed Sep 11 18:34:35 2013 From: storchaka at gmail.com (Serhiy Storchaka) Date: Wed, 11 Sep 2013 19:34:35 +0300 Subject: [Python-Dev] Need testing audio files In-Reply-To: <20130911101053.71deed3b@anarchist> References: <l0pfg4$i7l$1@ger.gmane.org> <20130911101053.71deed3b@anarchist> Message-ID: <l0q667$ac0$5@ger.gmane.org> 11.09.13 17:10, Barry Warsaw ???????(??): > I have some pro-audio recording capabilities and would be happy to generate > some copyright-donated clips for Python. Please contact me off-list if > needed. Thank you. From storchaka at gmail.com Wed Sep 11 18:00:43 2013 From: storchaka at gmail.com (Serhiy Storchaka) Date: Wed, 11 Sep 2013 19:00:43 +0300 Subject: [Python-Dev] Need testing audio files In-Reply-To: <20130911144553.17909e76@pitrou.net> References: <l0pfg4$i7l$1@ger.gmane.org> <20130911144553.17909e76@pitrou.net> Message-ID: <l0q65f$ac0$4@ger.gmane.org> 11.09.13 15:45, Antoine Pitrou ???????(??): > If you want to edit, shorten, convert sounds between different formats, > you can try Audacity, a free sound editor: > http://audacity.sourceforge.net/ Yes, Audacity is great. From storchaka at gmail.com Wed Sep 11 17:45:25 2013 From: storchaka at gmail.com (Serhiy Storchaka) Date: Wed, 11 Sep 2013 18:45:25 +0300 Subject: [Python-Dev] Need testing audio files In-Reply-To: <CAMpsgwbpOLM8ghr91YYA+uVDGwqNjQdjoP70c0QNqBg4FEVxXQ@mail.gmail.com> References: <l0pfg4$i7l$1@ger.gmane.org> <CAMpsgwbpOLM8ghr91YYA+uVDGwqNjQdjoP70c0QNqBg4FEVxXQ@mail.gmail.com> Message-ID: <l0q658$ac0$2@ger.gmane.org> 11.09.13 15:46, Victor Stinner ???????(??): > Use your microphone, say "python" and save the file in your favorite > file format. Try for example Audacity. I suppose that you don't need > specific audio content and you don't need a huge file. My voice is even more ugly than my English. I don't want perpetuate it in Python test suite. From storchaka at gmail.com Wed Sep 11 17:47:18 2013 From: storchaka at gmail.com (Serhiy Storchaka) Date: Wed, 11 Sep 2013 18:47:18 +0300 Subject: [Python-Dev] Need testing audio files In-Reply-To: <20130911141022.GA30682@iskra.aviel.ru> References: <l0pfg4$i7l$1@ger.gmane.org> <CAMpsgwbpOLM8ghr91YYA+uVDGwqNjQdjoP70c0QNqBg4FEVxXQ@mail.gmail.com> <CAP1=2W4a=Mzt=DVj4qeBZG8gxQ8iaJ7RUPhRb6-YvfTC2TbYfA@mail.gmail.com> <20130911141022.GA30682@iskra.aviel.ru> Message-ID: <l0q65b$ac0$3@ger.gmane.org> 11.09.13 17:10, Oleg Broytman ???????(??): > Wouldn't his name be enough? http://www.python.org/~guido/guido.au It is Lib/test/audiotest.au. 1-channel and 8-bit. No, it wouldn't enough. From martin at v.loewis.de Wed Sep 11 19:31:56 2013 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Wed, 11 Sep 2013 19:31:56 +0200 Subject: [Python-Dev] Add a "transformdict" to collections In-Reply-To: <20130911150423.62f77bf2@pitrou.net> References: <20130910112832.75ed6c8b@pitrou.net> <011DB128-7308-40C0-BE93-A691A26B2C1B@gmail.com> <20130911102732.2cb268ca@pitrou.net> <CANc-5Uw-66ZPYC=mOu4z+vHU74qHY4QDWq2bCLDCFBku+cAqcg@mail.gmail.com> <20130911115706.C2D98250925@webabinitio.net> <CADiSq7cjF6RGdrobJh1=f0etG3rVXARRUSCOJXzina0u1rkJTQ@mail.gmail.com> <20130911150423.62f77bf2@pitrou.net> Message-ID: <5230A90C.3090102@v.loewis.de> Am 11.09.13 15:04, schrieb Antoine Pitrou: > There are not many possible APIs to create case-insensitive dicts, or > identity dicts. That is certainly not true. Most obviously, you have the choice of a specialized case-mapping dict, or a generalized type that can be used for case mapping also. Does any of the referenced use cases actually use the API that you are proposing (i.e. with a key transformation function)? FWIW, I can think of yet another API for this: write a Proxy class class TransformedKey: # untested def __init__(self, original): self.original = original self.transformed = self.transform(original) def __hash__(self): return hash(self.transformed) def __eq__(self, other): return self.transformed == other.transformed def transform(self, key): raise NotImplementedError If there is then disciplined use in the form d[TransformedKey(key)] == value then you can use the existing dictionary type. Notice that this can do both case-insensitive and identity dicts (plus there are multiple choices of getting the transform function into it, as well as variations of the __eq__ implementation). There is evidence in this thread that people "grasp" case-insensitive more easily than the generalized API. For the record, I initially typed two responses into the tracker which I found to be incorrect before posting them, so I ended up posting neither. The "transformdict" type is not at all "natural", even though it may be useful. If you really want to "push" this API into 3.4, I think you will need to write a PEP, and find a PEP dictator who is willing to approve it. As you seem to dislike the idea of writing a PEP, I suggest to follow the idea of publishing it on PyPI now, and then proposing it for inclusion into 3.5. Regards, Martin From solipsis at pitrou.net Wed Sep 11 19:49:35 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 11 Sep 2013 19:49:35 +0200 Subject: [Python-Dev] Add a "transformdict" to collections In-Reply-To: <5230A90C.3090102@v.loewis.de> References: <20130910112832.75ed6c8b@pitrou.net> <011DB128-7308-40C0-BE93-A691A26B2C1B@gmail.com> <20130911102732.2cb268ca@pitrou.net> <CANc-5Uw-66ZPYC=mOu4z+vHU74qHY4QDWq2bCLDCFBku+cAqcg@mail.gmail.com> <20130911115706.C2D98250925@webabinitio.net> <CADiSq7cjF6RGdrobJh1=f0etG3rVXARRUSCOJXzina0u1rkJTQ@mail.gmail.com> <20130911150423.62f77bf2@pitrou.net> <5230A90C.3090102@v.loewis.de> Message-ID: <20130911194935.1b116266@fsol> On Wed, 11 Sep 2013 19:31:56 +0200 "Martin v. L?wis" <martin at v.loewis.de> wrote: > Am 11.09.13 15:04, schrieb Antoine Pitrou: > > There are not many possible APIs to create case-insensitive dicts, or > > identity dicts. > > That is certainly not true. Most obviously, you have the choice of a > specialized case-mapping dict, or a generalized type that can be used > for case mapping also. Does any of the referenced use cases actually > use the API that you are proposing (i.e. with a key transformation > function)? Well, when you have a specialized need, you don't implement a generic version (*), so I don't think anybody implemented it ;-). The point of a generic API is to help replace all specialized implementations at once, rather than a small subset of them. (*) even the Twisted people don't seem to go that far > FWIW, I can think of yet another API for this: write a Proxy class > > class TransformedKey: # untested > def __init__(self, original): > self.original = original > self.transformed = self.transform(original) > def __hash__(self): > return hash(self.transformed) > def __eq__(self, other): > return self.transformed == other.transformed > def transform(self, key): > raise NotImplementedError > > If there is then disciplined use in the form > > d[TransformedKey(key)] == value The problem is "disciplined use" here. This means any consumer of your API has to know about that quirk, so it's an inferior solution. (TransformedKey could be an internal detail of the implementation, as with weak dictionaries; but as a public API it doesn't seem very desirable) > The "transformdict" type > is not at all "natural", even though it may be useful. By that token, nothing is "natural" ;-) defaultdict isn't natural, yet it went in without a PEP. However, to all persons who already had that need and responded to the proposal, the generic version *does* seem natural enough, AFAICT. Only people who've never had such a need seem to fail to "grasp" it (and I only counted one such person, but my memory may be failing me). You know, I'm not against having "only" a case-insensitive dict. But that would be less useful all the while not being significantly easier to use, so I'm not really seeing the point. > If you really want to "push" this API into 3.4, I think you will need > to write a PEP, and find a PEP dictator who is willing to approve it. > As you seem to dislike the idea of writing a PEP, I suggest to follow > the idea of publishing it on PyPI now, and then proposing it for > inclusion into 3.5. What I dislike is the idea of doing additional work because some barriers are imposed ;-). PEP or PyPI are on a similar scale here. At least a PEP would help record the various discussion details, so I'd favour that over the PyPI path. Regards Antoine. From ethan at stoneleaf.us Wed Sep 11 20:07:55 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 11 Sep 2013 11:07:55 -0700 Subject: [Python-Dev] Add a "transformdict" to collections In-Reply-To: <20130911194935.1b116266@fsol> References: <20130910112832.75ed6c8b@pitrou.net> <011DB128-7308-40C0-BE93-A691A26B2C1B@gmail.com> <20130911102732.2cb268ca@pitrou.net> <CANc-5Uw-66ZPYC=mOu4z+vHU74qHY4QDWq2bCLDCFBku+cAqcg@mail.gmail.com> <20130911115706.C2D98250925@webabinitio.net> <CADiSq7cjF6RGdrobJh1=f0etG3rVXARRUSCOJXzina0u1rkJTQ@mail.gmail.com> <20130911150423.62f77bf2@pitrou.net> <5230A90C.3090102@v.loewis.de> <20130911194935.1b116266@fsol> Message-ID: <5230B17B.3000607@stoneleaf.us> On 09/11/2013 10:49 AM, Antoine Pitrou wrote: > > What I dislike is the idea of doing additional work because some > barriers are imposed ;-). PEP or PyPI are on a similar scale here. At > least a PEP would help record the various discussion details, so I'd > favour that over the PyPI path. I would think a good summary on the bug tracker would preserve the detail just fine. -- ~Ethan~ From tjreedy at udel.edu Wed Sep 11 22:04:45 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 11 Sep 2013 16:04:45 -0400 Subject: [Python-Dev] Need testing audio files In-Reply-To: <20130911101053.71deed3b@anarchist> References: <l0pfg4$i7l$1@ger.gmane.org> <20130911101053.71deed3b@anarchist> Message-ID: <l0qicn$vs2$1@ger.gmane.org> On 9/11/2013 10:10 AM, Barry Warsaw wrote: > On Sep 11, 2013, at 01:09 PM, Serhiy Storchaka wrote: > >> In ideal it should be one high-quality (float64?) multichannel (5+1?) but >> short master file and it's lower-quality copies made by third-party tools. In >> ideal the content should be related to Python. How about "Python is a great programming language." Long enough? > I have some pro-audio recording capabilities and would be happy to generate > some copyright-donated clips for Python. Please contact me off-list if > needed. -- Terry Jan Reedy From timothy.c.delaney at gmail.com Wed Sep 11 23:39:31 2013 From: timothy.c.delaney at gmail.com (Tim Delaney) Date: Thu, 12 Sep 2013 07:39:31 +1000 Subject: [Python-Dev] Add a "transformdict" to collections In-Reply-To: <52309446.6070809@stoneleaf.us> References: <20130910112832.75ed6c8b@pitrou.net> <011DB128-7308-40C0-BE93-A691A26B2C1B@gmail.com> <20130911102732.2cb268ca@pitrou.net> <CANc-5Uw-66ZPYC=mOu4z+vHU74qHY4QDWq2bCLDCFBku+cAqcg@mail.gmail.com> <20130911134032.GA16820@ando> <CAMpsgwbWF_7-EDKoQZz61BNk3mVNieMkVCquwweMuG=HSK4RAQ@mail.gmail.com> <52307DE0.9040208@stoneleaf.us> <CAMpsgwb1iB7fMKjYX5T3HR0E+3rmt4H055z2cjyJ-fMEjoEe6Q@mail.gmail.com> <52309446.6070809@stoneleaf.us> Message-ID: <CAN8CLg=MzvW77F9XKCa3g+26Ufby6EwGyO5yTQbw+--A0iQhYA@mail.gmail.com> On 12 September 2013 02:03, Ethan Furman <ethan at stoneleaf.us> wrote: > On 09/11/2013 08:49 AM, Victor Stinner wrote: > >> 2013/9/11 Ethan Furman <ethan at stoneleaf.us>: >> >>> He isn't keeping the key unchanged (notice no white space in MAPPING), >>> he's >>> merely providing a function that will automatically strip the whitespace >>> from key lookups. >>> >> >> transformdict keeps the key unchanged, see the first message: >> >> >>> d = transformdict(str.lower) >> >>> d['Foo'] = 5 >> >>> d['foo'] >> 5 >> >>> d['FOO'] >> 5 >> >>> list(d) >> ['Foo'] >> > That seems backwards to me. I would think that retrieving the keys from the dict would return the transformed keys (I'd call them canonical keys). That way there's no question about which key is stored - it's *always* the transformed key. In fact, I think this might get more traction if it were referred to as a canonicalising dictionary (bikeshedding, I know). Tim Delaney -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130912/33424550/attachment.html> From ethan at stoneleaf.us Wed Sep 11 23:56:22 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 11 Sep 2013 14:56:22 -0700 Subject: [Python-Dev] Add a "transformdict" to collections In-Reply-To: <CAN8CLg=MzvW77F9XKCa3g+26Ufby6EwGyO5yTQbw+--A0iQhYA@mail.gmail.com> References: <20130910112832.75ed6c8b@pitrou.net> <011DB128-7308-40C0-BE93-A691A26B2C1B@gmail.com> <20130911102732.2cb268ca@pitrou.net> <CANc-5Uw-66ZPYC=mOu4z+vHU74qHY4QDWq2bCLDCFBku+cAqcg@mail.gmail.com> <20130911134032.GA16820@ando> <CAMpsgwbWF_7-EDKoQZz61BNk3mVNieMkVCquwweMuG=HSK4RAQ@mail.gmail.com> <52307DE0.9040208@stoneleaf.us> <CAMpsgwb1iB7fMKjYX5T3HR0E+3rmt4H055z2cjyJ-fMEjoEe6Q@mail.gmail.com> <52309446.6070809@stoneleaf.us> <CAN8CLg=MzvW77F9XKCa3g+26Ufby6EwGyO5yTQbw+--A0iQhYA@mail.gmail.com> Message-ID: <5230E706.2070103@stoneleaf.us> On 09/11/2013 02:39 PM, Tim Delaney wrote: > On 12 September 2013 02:03, Ethan Furman <ethan at stoneleaf.us <mailto:ethan at stoneleaf.us>> wrote: > > On 09/11/2013 08:49 AM, Victor Stinner wrote: > > 2013/9/11 Ethan Furman <ethan at stoneleaf.us <mailto:ethan at stoneleaf.us>>: > > He isn't keeping the key unchanged (notice no white space in MAPPING), he's > merely providing a function that will automatically strip the whitespace > from key lookups. > > > transformdict keeps the key unchanged, see the first message: > > >>> d = transformdict(str.lower) > >>> d['Foo'] = 5 > >>> d['foo'] > 5 > >>> d['FOO'] > 5 > >>> list(d) > ['Foo'] > > That seems backwards to me. I would think that retrieving the keys from the dict would return the transformed keys (I'd > call them canonical keys). That way there's no question about which key is stored - it's *always* the transformed key. At this point there is still no question: it's the first version of the key seen. For a stupid example: --> d = transformdict(str.lower) --> d['ThePyramid'] = 'Game Show' --> d['AtOnce'] = now() --> for k, v in d.items(): ... print(k, v) Imagine writing a function to get that capitalization right. > In fact, I think this might get more traction if it were referred to as a canonicalising dictionary (bikeshedding, I know). Whoa, that's way harder to spell! ;) Drop the 'ising', though, and I'm in. -- ~Ethan~ From ben+python at benfinney.id.au Thu Sep 12 01:12:36 2013 From: ben+python at benfinney.id.au (Ben Finney) Date: Thu, 12 Sep 2013 09:12:36 +1000 Subject: [Python-Dev] Need testing audio files References: <l0pfg4$i7l$1@ger.gmane.org> Message-ID: <7w7gen2brv.fsf@benfinney.id.au> Serhiy Storchaka <storchaka at gmail.com> writes: > I work on enhancement of audio modules testing [1], and I need free > (in both senses) small sample audio files in different formats. The Internet Archive <URL:https://archive.org/> has a wide selection of free-software media, many of which have free license terms that would be suitable for inclusion in Python. I haven't used it, but <URL:http://www.pdsounds.org/> appears to be a collection of sounds in the public domain. -- \ ?The whole area of [treating source code as intellectual | `\ property] is almost assuring a customer that you are not going | _o__) to do any innovation in the future.? ?Gary Barnett | Ben Finney From solipsis at pitrou.net Thu Sep 12 08:14:23 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 12 Sep 2013 08:14:23 +0200 Subject: [Python-Dev] cpython (merge 3.3 -> default): merge from 3.3 References: <3cb8QF2vy7z7Ljh@mail.python.org> Message-ID: <20130912081423.4607ea82@fsol> On Thu, 12 Sep 2013 07:57:25 +0200 (CEST) senthil.kumaran <python-checkins at python.org> wrote: > > +<<<<<<< local > Optional argument random is a 0-argument function returning a > random float in [0.0, 1.0); if it is the default None, the > standard random.random will be used. > +======= > + Optional arg random is a 0-argument function returning a random > + float in [0.0, 1.0); by default, the standard random.random. > + > + Do not supply the 'int' argument. > +>>>>>>> other Can you fix this? Thanks Antoine. From ethan at stoneleaf.us Thu Sep 12 16:08:47 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 12 Sep 2013 07:08:47 -0700 Subject: [Python-Dev] Add a "transformdict" to collections In-Reply-To: <CAN8CLg=MzvW77F9XKCa3g+26Ufby6EwGyO5yTQbw+--A0iQhYA@mail.gmail.com> References: <20130910112832.75ed6c8b@pitrou.net> <011DB128-7308-40C0-BE93-A691A26B2C1B@gmail.com> <20130911102732.2cb268ca@pitrou.net> <CANc-5Uw-66ZPYC=mOu4z+vHU74qHY4QDWq2bCLDCFBku+cAqcg@mail.gmail.com> <20130911134032.GA16820@ando> <CAMpsgwbWF_7-EDKoQZz61BNk3mVNieMkVCquwweMuG=HSK4RAQ@mail.gmail.com> <52307DE0.9040208@stoneleaf.us> <CAMpsgwb1iB7fMKjYX5T3HR0E+3rmt4H055z2cjyJ-fMEjoEe6Q@mail.gmail.com> <52309446.6070809@stoneleaf.us> <CAN8CLg=MzvW77F9XKCa3g+26Ufby6EwGyO5yTQbw+--A0iQhYA@mail.gmail.com> Message-ID: <5231CAEF.1080500@stoneleaf.us> On 09/11/2013 02:39 PM, Tim Delaney wrote: > > I would think that retrieving the keys from the dict would return the transformed keys (I'd > call them canonical keys). The more I think about this the more I agree. A canonicaldict with a key function that simply stored the transformed key and it's value would seem to be a lot simpler: - no need to store a separate "presentation" key - no confusion about which of the first key/last key seen is stored - no mistakes with the "first" key not being added before real data and getting the presentation key wrong Further, in order to store the non-canonical keys a separate list must be kept of the keys to preseed the canonicaldict; if we store the canonical keys a separate list must be kept for presentation purposes -- so worst case scenario we're keeping the same amount of information and best-case scenario the presentation of the keys doesn't matter and we just saved ourselves an extra data structure. -- ~Ethan~ From ronaldoussoren at mac.com Thu Sep 12 16:42:39 2013 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Thu, 12 Sep 2013 16:42:39 +0200 Subject: [Python-Dev] PEP 447: add type.__locallookup__ In-Reply-To: <0752376899dd7436f6115bfc03594574@chopin.edu.pl> References: <75030FAC-6918-4E94-95DA-67A88D53E6F5@mac.com> <7D6D4488-2855-41F8-A9F2-1FF23B9E3A56@mac.com> <522DDB74.5080609@stoneleaf.us> <CAP7+vJJTR=+cRY+iQgHY4ujTsC3_QyWn83dnK96rWO6bV9vFLQ@mail.gmail.com> <522DECA3.2030709@hotpy.org> <CAP7+vJ+thJvYpk0d0rnFO6bm88sTJ_t1QdYFB0SdPOGNDABuag@mail.gmail.com> <522DF204.7040109@hotpy.org> <0752376899dd7436f6115bfc03594574@chopin.edu.pl> Message-ID: <8F0FD1DE-A8ED-40D7-878A-9A005ABDF70C@mac.com> On 9 Sep, 2013, at 20:23, Jan Kaliszewski <zuo at chopin.edu.pl> wrote: > Is '__locallookup__' a really good name? In Python, *local* -- especially in context of *lookups* -- usually associates with locals() i.e. a namespace of a function/method execution frame or a namespace of a class, during *definition* of that class... So '__locallookup__' can be confusing. > > Why not just '__getclassattribute__' or '__classlookup__', or '__classattribute__'...? I don't particularly like __locallookup__ either, but haven't found a better name yet. "__lookup_in_class__" was the best alternative I could come up with, and that feels different than other special methods. The name in the PEP is more or less derived from _PyType_Lookup, with "local" meaning "only in this class, don't recurse in the rest of the MRO". Ronald From solipsis at pitrou.net Thu Sep 12 16:43:00 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 12 Sep 2013 16:43:00 +0200 Subject: [Python-Dev] Add a "transformdict" to collections References: <20130910112832.75ed6c8b@pitrou.net> <011DB128-7308-40C0-BE93-A691A26B2C1B@gmail.com> <20130911102732.2cb268ca@pitrou.net> <CANc-5Uw-66ZPYC=mOu4z+vHU74qHY4QDWq2bCLDCFBku+cAqcg@mail.gmail.com> <20130911134032.GA16820@ando> <CAMpsgwbWF_7-EDKoQZz61BNk3mVNieMkVCquwweMuG=HSK4RAQ@mail.gmail.com> <52307DE0.9040208@stoneleaf.us> <CAMpsgwb1iB7fMKjYX5T3HR0E+3rmt4H055z2cjyJ-fMEjoEe6Q@mail.gmail.com> <52309446.6070809@stoneleaf.us> <CAN8CLg=MzvW77F9XKCa3g+26Ufby6EwGyO5yTQbw+--A0iQhYA@mail.gmail.com> <5231CAEF.1080500@stoneleaf.us> Message-ID: <20130912164300.5cd27a1a@pitrou.net> Le Thu, 12 Sep 2013 07:08:47 -0700, Ethan Furman <ethan at stoneleaf.us> a ?crit : > On 09/11/2013 02:39 PM, Tim Delaney wrote: > > > > I would think that retrieving the keys from the dict would return > > the transformed keys (I'd call them canonical keys). > > The more I think about this the more I agree. A canonicaldict with a > key function that simply stored the transformed key and it's value > would seem to be a lot simpler: And it wouldn't solve the use cases. What's the point? > Further, in order to store the non-canonical keys a separate list > must be kept of the keys to preseed the canonicaldict; Yeah, so this is totally silly. What you're basically saying is "we don't need TransformDict since people can re-implement it themselves". Regards Antoine. From duda.piotr at gmail.com Thu Sep 12 16:51:26 2013 From: duda.piotr at gmail.com (Piotr Duda) Date: Thu, 12 Sep 2013 16:51:26 +0200 Subject: [Python-Dev] Add a "transformdict" to collections In-Reply-To: <5231CAEF.1080500@stoneleaf.us> References: <20130910112832.75ed6c8b@pitrou.net> <011DB128-7308-40C0-BE93-A691A26B2C1B@gmail.com> <20130911102732.2cb268ca@pitrou.net> <CANc-5Uw-66ZPYC=mOu4z+vHU74qHY4QDWq2bCLDCFBku+cAqcg@mail.gmail.com> <20130911134032.GA16820@ando> <CAMpsgwbWF_7-EDKoQZz61BNk3mVNieMkVCquwweMuG=HSK4RAQ@mail.gmail.com> <52307DE0.9040208@stoneleaf.us> <CAMpsgwb1iB7fMKjYX5T3HR0E+3rmt4H055z2cjyJ-fMEjoEe6Q@mail.gmail.com> <52309446.6070809@stoneleaf.us> <CAN8CLg=MzvW77F9XKCa3g+26Ufby6EwGyO5yTQbw+--A0iQhYA@mail.gmail.com> <5231CAEF.1080500@stoneleaf.us> Message-ID: <CAJ1Wxn2GOodq_USD_L3ret73cfq7WEPtaGXftuQprOUVQ+FzMA@mail.gmail.com> 2013/9/12 Ethan Furman <ethan at stoneleaf.us>: > On 09/11/2013 02:39 PM, Tim Delaney wrote: >> >> >> I would think that retrieving the keys from the dict would return the >> transformed keys (I'd >> call them canonical keys). > > > The more I think about this the more I agree. A canonicaldict with a key > function that simply stored the transformed key and it's value would seem to > be a lot simpler: > > - no need to store a separate "presentation" key > - no confusion about which of the first key/last key seen is stored > - no mistakes with the "first" key not being added before real data > and getting the presentation key wrong If original keys aren't stored then d = TransformDict(transfunc, ...) for k in d: dosomething(k, d[k]) will break if transfunc(transfunc(x)) != transfunc(x) -- ???????? ?????? From ronaldoussoren at mac.com Thu Sep 12 16:25:56 2013 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Thu, 12 Sep 2013 16:25:56 +0200 Subject: [Python-Dev] PEP 447: add type.__locallookup__ In-Reply-To: <522DECA3.2030709@hotpy.org> References: <75030FAC-6918-4E94-95DA-67A88D53E6F5@mac.com> <7D6D4488-2855-41F8-A9F2-1FF23B9E3A56@mac.com> <522DDB74.5080609@stoneleaf.us> <CAP7+vJJTR=+cRY+iQgHY4ujTsC3_QyWn83dnK96rWO6bV9vFLQ@mail.gmail.com> <522DECA3.2030709@hotpy.org> Message-ID: <D280AEE7-B847-45B6-9F6B-359384971CBA@mac.com> On 9 Sep, 2013, at 17:43, Mark Shannon <mark at hotpy.org> wrote: > I would like time to investigate this further, but at the moment I think it will either make attribute lookup poorly defined or slow. > > Of the top of my head, the problem as a I see it is basically this: > Currently, type.__getattribute__() is a fixed point in the lookup of attributes. > The proposal means that a fixed point is not reached until the cls parameter of type.__getattribute__() is either object or type, > otherwise type.__getattribute__() and type.__locallookup__ must bounce back and forth. > > This will slow down *every* attribute lookup for what is a fairly obscure use case. I did a benchmark run (see the pep for details) and that seems to indicate that the performance impact is very small, possibly because the patch keeps the attribute lookup cache used by _PyType_Lookup. Anyway, I'm glad that there is now some real discussion on the proposal. Not unsurprisingly I'd love to have this, or something simular, in 3.4. I had hoped to repost the PEP a while back with more information on how the API would affect PyObjC (code removal, performance impact), but haven't had time to move forward on that front :-( Ronald > > Cheers, > Mark. > > On 09/09/13 16:27, Guido van Rossum wrote: >> Let's just accept this PEP. It looks like a nice addition to the metaclass machinery and I don't think we'll get much more useful feedback by waiting. >> >> >> On Mon, Sep 9, 2013 at 7:30 AM, Ethan Furman <ethan at stoneleaf.us <mailto:ethan at stoneleaf.us>> wrote: >> >> On 07/30/2013 11:17 PM, Ronald Oussoren wrote: >> >> >> And something I forgot to ask: is anyone willing to be the BDFL-Delegate for >> PEP 447? >> >> >> *Bump*. >> >> It would be nice if this could make into 3.4. >> >> -- >> ~Ethan~ >> >> _________________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org <mailto:Python-Dev at python.org> >> https://mail.python.org/__mailman/listinfo/python-dev <https://mail.python.org/mailman/listinfo/python-dev> >> Unsubscribe: https://mail.python.org/__mailman/options/python-dev/__guido%40python.org <https://mail.python.org/mailman/options/python-dev/guido%40python.org> >> >> >> >> >> -- >> --Guido van Rossum (python.org/~guido <http://python.org/~guido>) >> >> >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: https://mail.python.org/mailman/options/python-dev/mark%40hotpy.org >> > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ronaldoussoren%40mac.com From ethan at stoneleaf.us Thu Sep 12 17:05:44 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 12 Sep 2013 08:05:44 -0700 Subject: [Python-Dev] Add a "transformdict" to collections In-Reply-To: <20130912164300.5cd27a1a@pitrou.net> References: <20130910112832.75ed6c8b@pitrou.net> <011DB128-7308-40C0-BE93-A691A26B2C1B@gmail.com> <20130911102732.2cb268ca@pitrou.net> <CANc-5Uw-66ZPYC=mOu4z+vHU74qHY4QDWq2bCLDCFBku+cAqcg@mail.gmail.com> <20130911134032.GA16820@ando> <CAMpsgwbWF_7-EDKoQZz61BNk3mVNieMkVCquwweMuG=HSK4RAQ@mail.gmail.com> <52307DE0.9040208@stoneleaf.us> <CAMpsgwb1iB7fMKjYX5T3HR0E+3rmt4H055z2cjyJ-fMEjoEe6Q@mail.gmail.com> <52309446.6070809@stoneleaf.us> <CAN8CLg=MzvW77F9XKCa3g+26Ufby6EwGyO5yTQbw+--A0iQhYA@mail.gmail.com> <5231CAEF.1080500@stoneleaf.us> <20130912164300.5cd27a1a@pitrou.net> Message-ID: <5231D848.1090709@stoneleaf.us> On 09/12/2013 07:43 AM, Antoine Pitrou wrote: > > Yeah, so this is totally silly. What you're basically saying is "we > don't need TransformDict since people can re-implement it themselves". No, what I'm saying is that the "case-preserving" aspect of transformdict is silly. The main point of transformdict is to enable, for example, 'IBM', 'Ibm', and 'ibm' to all match up as the same key. But why? Because you don't trust the user data. And if you don't trust the user data you have to add the correct version of the key yourself before you ever process that data, which means you already have the correct version stored somewhere. -- ~Ethan~ From rdmurray at bitdance.com Thu Sep 12 17:30:49 2013 From: rdmurray at bitdance.com (R. David Murray) Date: Thu, 12 Sep 2013 11:30:49 -0400 Subject: [Python-Dev] PEP 447: add type.__locallookup__ In-Reply-To: <8F0FD1DE-A8ED-40D7-878A-9A005ABDF70C@mac.com> References: <75030FAC-6918-4E94-95DA-67A88D53E6F5@mac.com> <7D6D4488-2855-41F8-A9F2-1FF23B9E3A56@mac.com> <522DDB74.5080609@stoneleaf.us> <CAP7+vJJTR=+cRY+iQgHY4ujTsC3_QyWn83dnK96rWO6bV9vFLQ@mail.gmail.com> <522DECA3.2030709@hotpy.org> <CAP7+vJ+thJvYpk0d0rnFO6bm88sTJ_t1QdYFB0SdPOGNDABuag@mail.gmail.com> <522DF204.7040109@hotpy.org> <0752376899dd7436f6115bfc03594574@chopin.edu.pl> <8F0FD1DE-A8ED-40D7-878A-9A005ABDF70C@mac.com> Message-ID: <20130912153050.3CBC0250843@webabinitio.net> On Thu, 12 Sep 2013 16:42:39 +0200, Ronald Oussoren <ronaldoussoren at mac.com> wrote: > > On 9 Sep, 2013, at 20:23, Jan Kaliszewski <zuo at chopin.edu.pl> wrote: > > > Is '__locallookup__' a really good name? In Python, *local* -- > > especially in context of *lookups* -- usually associates with > > locals() i.e. a namespace of a function/method execution frame or a > > namespace of a class, during *definition* of that class... So > > '__locallookup__' can be confusing. > > > > Why not just '__getclassattribute__' or '__classlookup__', or > > '__classattribute__'...? > > I don't particularly like __locallookup__ either, but haven't found a > better name yet. "__lookup_in_class__" was the best alternative I > could come up with, and that feels different than other special > methods. The name in the PEP is more or less derived from > _PyType_Lookup, with "local" meaning "only in this class, don't > recurse in the rest of the MRO". Why is __getclassattribute__ worse than __locallookup__? --David From ronaldoussoren at mac.com Thu Sep 12 16:33:12 2013 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Thu, 12 Sep 2013 16:33:12 +0200 Subject: [Python-Dev] PEP 447: add type.__locallookup__ In-Reply-To: <522E3B2E.6050605@hotpy.org> References: <75030FAC-6918-4E94-95DA-67A88D53E6F5@mac.com> <7D6D4488-2855-41F8-A9F2-1FF23B9E3A56@mac.com> <522DDB74.5080609@stoneleaf.us> <522E3B2E.6050605@hotpy.org> Message-ID: <B4E5C214-4D51-4C0C-A906-01DC717B69DB@mac.com> On 9 Sep, 2013, at 23:18, Mark Shannon <mark at hotpy.org> wrote: > On 09/09/13 15:30, Ethan Furman wrote: >> On 07/30/2013 11:17 PM, Ronald Oussoren wrote: >>> >>> And something I forgot to ask: is anyone willing to be the >>> BDFL-Delegate for >>> PEP 447? >> >> *Bump*. >> >> It would be nice if this could make into 3.4. >> > > IMO, there are some issues that need to be addressed before PEP 447 should be accepted. > > 1. Is there even a problem at all, or is this just a bug in super? > Why doesn't super() respect the __getattribute__ method of the superclass? Because __getattribute__ looks in the instance __dict__ before walking the MRO, while super does not. > > 2. Is this the best way to solve the problem (if there is a problem)? > Would a __super__ special method be sufficient and less intrusive. One reason for the __locallookup__ method is to make normal and super attribute lookup more simular, adding a __super__ special method would lead to code duplication: both __getattribute__ and __super__ would either contain simular code, or would call out to a shared method anyway. > > 3. Are the proposed semantics OK? > I think they are, but very low level changes such as these can have unforeseen consequences. For example, PEP 3135 and issue 12370. > > 4. What is the performance impact. pybench really doesn't count as a benchmark. What kind of benchmark would you like to see? BTW. I ran more than pybench, I also ran the part of the performance benchmark that worked on py3k at the time. Ronald From solipsis at pitrou.net Thu Sep 12 17:40:31 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 12 Sep 2013 17:40:31 +0200 Subject: [Python-Dev] Add a "transformdict" to collections References: <20130910112832.75ed6c8b@pitrou.net> <011DB128-7308-40C0-BE93-A691A26B2C1B@gmail.com> <20130911102732.2cb268ca@pitrou.net> <CANc-5Uw-66ZPYC=mOu4z+vHU74qHY4QDWq2bCLDCFBku+cAqcg@mail.gmail.com> <20130911134032.GA16820@ando> <CAMpsgwbWF_7-EDKoQZz61BNk3mVNieMkVCquwweMuG=HSK4RAQ@mail.gmail.com> <52307DE0.9040208@stoneleaf.us> <CAMpsgwb1iB7fMKjYX5T3HR0E+3rmt4H055z2cjyJ-fMEjoEe6Q@mail.gmail.com> <52309446.6070809@stoneleaf.us> <CAN8CLg=MzvW77F9XKCa3g+26Ufby6EwGyO5yTQbw+--A0iQhYA@mail.gmail.com> <5231CAEF.1080500@stoneleaf.us> <20130912164300.5cd27a1a@pitrou.net> <5231D848.1090709@stoneleaf.us> Message-ID: <20130912174031.5971338f@pitrou.net> Le Thu, 12 Sep 2013 08:05:44 -0700, Ethan Furman <ethan at stoneleaf.us> a ?crit : > On 09/12/2013 07:43 AM, Antoine Pitrou wrote: > > > > Yeah, so this is totally silly. What you're basically saying is "we > > don't need TransformDict since people can re-implement it > > themselves". > > No, what I'm saying is that the "case-preserving" aspect of > transformdict is silly. The main point of transformdict is to > enable, for example, 'IBM', 'Ibm', and 'ibm' to all match up as the > same key. But why? Because you don't trust the user data. And if > you don't trust the user data you have to add the correct version of > the key yourself before you ever process that data, which means you > already have the correct version stored somewhere. That's assuming there is an a priori "correct" version. But there might not be any. Keeping the original key is important for different reasons depending on the use case: - for case-insensitive dicts, you want to keep the original key for presentation, logging and debugging purposes (*) - for identity dicts, the original key is mandatory because the id() value in itself is completely useless, it's just used for matching (*) For a well-known example of such behaviour, think about Windows filesystems. Regards Antoine. From rdmurray at bitdance.com Thu Sep 12 17:46:11 2013 From: rdmurray at bitdance.com (R. David Murray) Date: Thu, 12 Sep 2013 11:46:11 -0400 Subject: [Python-Dev] Add a "transformdict" to collections In-Reply-To: <5231D848.1090709@stoneleaf.us> References: <20130910112832.75ed6c8b@pitrou.net> <011DB128-7308-40C0-BE93-A691A26B2C1B@gmail.com> <20130911102732.2cb268ca@pitrou.net> <CANc-5Uw-66ZPYC=mOu4z+vHU74qHY4QDWq2bCLDCFBku+cAqcg@mail.gmail.com> <20130911134032.GA16820@ando> <CAMpsgwbWF_7-EDKoQZz61BNk3mVNieMkVCquwweMuG=HSK4RAQ@mail.gmail.com> <52307DE0.9040208@stoneleaf.us> <CAMpsgwb1iB7fMKjYX5T3HR0E+3rmt4H055z2cjyJ-fMEjoEe6Q@mail.gmail.com> <52309446.6070809@stoneleaf.us> <CAN8CLg=MzvW77F9XKCa3g+26Ufby6EwGyO5yTQbw+--A0iQhYA@mail.gmail.com> <5231CAEF.1080500@stoneleaf.us> <20130912164300.5cd27a1a@pitrou.net> <5231D848.1090709@stoneleaf.us> Message-ID: <20130912154611.CEDD4250843@webabinitio.net> On Thu, 12 Sep 2013 08:05:44 -0700, Ethan Furman <ethan at stoneleaf.us> wrote: > On 09/12/2013 07:43 AM, Antoine Pitrou wrote: > > > > Yeah, so this is totally silly. What you're basically saying is "we > > don't need TransformDict since people can re-implement it themselves". > > No, what I'm saying is that the "case-preserving" aspect of > transformdict is silly. The main point of transformdict is to enable, > for example, 'IBM', 'Ibm', and 'ibm' to all match up as the same key. > But why? Because you don't trust the user data. And if you don't > trust the user data you have to add the correct version of the key > yourself before you ever process that data, which means you already > have the correct version stored somewhere. No, in the *(original use case* (and many other use cases, such as the identity dict) we *do* trust the user data. It is more trustworthy than the key forms used to look up the data inside the program. That's the whole point. (Don't be distracted by Eric's particular use case.) But even for the general case...well, let me repost here what I said on the tracker: In our view, this data structure is for cases where the original key is the most important piece of information (about the keys). The transformation in the lookup process is entirely in the service of looking up the value paired with that original key when there is more than one possible representation of that key. It is the original key that is critical when re-serializing the data or otherwise making use of the keys for anything other than lookup. So this is about making the data structure succinctly model the problem domain, which is what OO is supposed to be good at :) --David From ethan at stoneleaf.us Thu Sep 12 17:48:22 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 12 Sep 2013 08:48:22 -0700 Subject: [Python-Dev] Add a "transformdict" to collections In-Reply-To: <20130912174031.5971338f@pitrou.net> References: <20130910112832.75ed6c8b@pitrou.net> <011DB128-7308-40C0-BE93-A691A26B2C1B@gmail.com> <20130911102732.2cb268ca@pitrou.net> <CANc-5Uw-66ZPYC=mOu4z+vHU74qHY4QDWq2bCLDCFBku+cAqcg@mail.gmail.com> <20130911134032.GA16820@ando> <CAMpsgwbWF_7-EDKoQZz61BNk3mVNieMkVCquwweMuG=HSK4RAQ@mail.gmail.com> <52307DE0.9040208@stoneleaf.us> <CAMpsgwb1iB7fMKjYX5T3HR0E+3rmt4H055z2cjyJ-fMEjoEe6Q@mail.gmail.com> <52309446.6070809@stoneleaf.us> <CAN8CLg=MzvW77F9XKCa3g+26Ufby6EwGyO5yTQbw+--A0iQhYA@mail.gmail.com> <5231CAEF.1080500@stoneleaf.us> <20130912164300.5cd27a1a@pitrou.net> <5231D848.1090709@stoneleaf.us> <20130912174031.5971338f@pitrou.net> Message-ID: <5231E246.2020507@stoneleaf.us> On 09/12/2013 08:40 AM, Antoine Pitrou wrote: > Le Thu, 12 Sep 2013 08:05:44 -0700, > Ethan Furman <ethan at stoneleaf.us> a ?crit : >> On 09/12/2013 07:43 AM, Antoine Pitrou wrote: >>> >>> Yeah, so this is totally silly. What you're basically saying is "we >>> don't need TransformDict since people can re-implement it >>> themselves". >> >> No, what I'm saying is that the "case-preserving" aspect of >> transformdict is silly. The main point of transformdict is to >> enable, for example, 'IBM', 'Ibm', and 'ibm' to all match up as the >> same key. But why? Because you don't trust the user data. And if >> you don't trust the user data you have to add the correct version of >> the key yourself before you ever process that data, which means you >> already have the correct version stored somewhere. > > That's assuming there is an a priori "correct" version. But there might > not be any. Keeping the original key is important for different reasons > depending on the use case: > > - for case-insensitive dicts, you want to keep the original key for > presentation, logging and debugging purposes (*) > > - for identity dicts, the original key is mandatory because the id() > value in itself is completely useless, it's just used for matching > > (*) For a well-known example of such behaviour, think about Windows > filesystems. Ah, I see. Thank you for explaining. -- ~Ethan~ From ronaldoussoren at mac.com Thu Sep 12 20:12:18 2013 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Thu, 12 Sep 2013 20:12:18 +0200 Subject: [Python-Dev] PEP 447: add type.__locallookup__ In-Reply-To: <20130912153050.3CBC0250843@webabinitio.net> References: <75030FAC-6918-4E94-95DA-67A88D53E6F5@mac.com> <7D6D4488-2855-41F8-A9F2-1FF23B9E3A56@mac.com> <522DDB74.5080609@stoneleaf.us> <CAP7+vJJTR=+cRY+iQgHY4ujTsC3_QyWn83dnK96rWO6bV9vFLQ@mail.gmail.com> <522DECA3.2030709@hotpy.org> <CAP7+vJ+thJvYpk0d0rnFO6bm88sTJ_t1QdYFB0SdPOGNDABuag@mail.gmail.com> <522DF204.7040109@hotpy.org> <0752376899dd7436f6115bfc03594574@chopin.edu.pl> <8F0FD1DE-A8ED-40D7-878A-9A005ABDF70C@mac.com> <20130912153050.3CBC0250843@webabinitio.net> Message-ID: <35B9C3A0-3039-49CA-90D6-319890611618@mac.com> > On 12 sep. 2013, at 17:30, "R. David Murray" <rdmurray at bitdance.com> wrote: > >> On Thu, 12 Sep 2013 16:42:39 +0200, Ronald Oussoren <ronaldoussoren at mac.com> wrote: >> >>> On 9 Sep, 2013, at 20:23, Jan Kaliszewski <zuo at chopin.edu.pl> wrote: >>> >>> Is '__locallookup__' a really good name? In Python, *local* -- >>> especially in context of *lookups* -- usually associates with >>> locals() i.e. a namespace of a function/method execution frame or a >>> namespace of a class, during *definition* of that class... So >>> '__locallookup__' can be confusing. >>> >>> Why not just '__getclassattribute__' or '__classlookup__', or >>> '__classattribute__'...? >> >> I don't particularly like __locallookup__ either, but haven't found a >> better name yet. "__lookup_in_class__" was the best alternative I >> could come up with, and that feels different than other special >> methods. The name in the PEP is more or less derived from >> _PyType_Lookup, with "local" meaning "only in this class, don't >> recurse in the rest of the MRO". > > Why is __getclassattribute__ worse than __locallookup__? Getclassattribute feels like it is related to classmethod, or fetches an attribute of a class. The method does however fetch a value from the class that is transformed to the actual attribute value through the descriptor protocol. Ronald > > --David > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ronaldoussoren%40mac.com From v+python at g.nevcal.com Thu Sep 12 21:15:14 2013 From: v+python at g.nevcal.com (Glenn Linderman) Date: Thu, 12 Sep 2013 12:15:14 -0700 Subject: [Python-Dev] Add a "transformdict" to collections In-Reply-To: <20130912174031.5971338f@pitrou.net> References: <20130910112832.75ed6c8b@pitrou.net> <011DB128-7308-40C0-BE93-A691A26B2C1B@gmail.com> <20130911102732.2cb268ca@pitrou.net> <CANc-5Uw-66ZPYC=mOu4z+vHU74qHY4QDWq2bCLDCFBku+cAqcg@mail.gmail.com> <20130911134032.GA16820@ando> <CAMpsgwbWF_7-EDKoQZz61BNk3mVNieMkVCquwweMuG=HSK4RAQ@mail.gmail.com> <52307DE0.9040208@stoneleaf.us> <CAMpsgwb1iB7fMKjYX5T3HR0E+3rmt4H055z2cjyJ-fMEjoEe6Q@mail.gmail.com> <52309446.6070809@stoneleaf.us> <CAN8CLg=MzvW77F9XKCa3g+26Ufby6EwGyO5yTQbw+--A0iQhYA@mail.gmail.com> <5231CAEF.1080500@stoneleaf.us> <20130912164300.5cd27a1a@pitrou.net> <5231D848.1090709@stoneleaf.us> <20130912174031.5971338f@pitrou.net> Message-ID: <523212C2.6040705@g.nevcal.com> On 9/12/2013 8:40 AM, Antoine Pitrou wrote: > Le Thu, 12 Sep 2013 08:05:44 -0700, > Ethan Furman <ethan at stoneleaf.us> a ?crit : >> On 09/12/2013 07:43 AM, Antoine Pitrou wrote: >>> Yeah, so this is totally silly. What you're basically saying is "we >>> don't need TransformDict since people can re-implement it >>> themselves". >> No, what I'm saying is that the "case-preserving" aspect of >> transformdict is silly. The main point of transformdict is to >> enable, for example, 'IBM', 'Ibm', and 'ibm' to all match up as the >> same key. But why? Because you don't trust the user data. And if >> you don't trust the user data you have to add the correct version of >> the key yourself before you ever process that data, which means you >> already have the correct version stored somewhere. > That's assuming there is an a priori "correct" version. But there might > not be any. Keeping the original key is important for different reasons > depending on the use case: > > - for case-insensitive dicts, you want to keep the original key for > presentation, logging and debugging purposes (*) > > - for identity dicts, the original key is mandatory because the id() > value in itself is completely useless, it's just used for matching - for dict with non-hashable key, but a transform function that can derive a hashable key from it, the presentation key value may be much more complex than the actual key value. > > (*) For a well-known example of such behaviour, think about Windows > filesystems. > > Regards > > Antoine. > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130912/89fcb0a4/attachment.html> From xdegaye at gmail.com Thu Sep 12 22:20:04 2013 From: xdegaye at gmail.com (Xavier de Gaye) Date: Thu, 12 Sep 2013 22:20:04 +0200 Subject: [Python-Dev] f_lineno accessors Message-ID: <523221F4.8020406@gmail.com> In issues 7238 [1], 16482 [2], 17697 [3] and 17277 [4], the line number may be incorrect when the global trace function has been removed but not the frame f_trace function. A simple test (see below) in issue 17288 [5] crashes the interpreter when setting f_lineno in a generator from a return trace function. All those issues are few months old. There is a patch at issue 17277 [4] that fixes the first 4 issues. There is also a patch for issue 17288 [5]. ###### Setting f_lineno in a generator ###### $ cat jump.py def gen(): for i in range(1): yield i lineno = 4 for i in gen(): pass $ python -m pdb jump.py > /tmp/jump.py(1)<module>() -> def gen(): (Pdb) import sys; print(sys.version) 3.4.0a1+ (default:975d1e180689, Sep 6 2013, 09:26:12) [GCC 4.3.2] (Pdb) break 3 Breakpoint 1 at /tmp/jump.py:3 (Pdb) continue > /tmp/jump.py(3)gen() -> yield i (Pdb) step --Return-- > /tmp/jump.py(3)gen()->0 -> yield i (Pdb) jump 2 > /tmp/jump.py(2)gen()->0 -> for i in range(1): (Pdb) continue Segmentation fault [1] http://bugs.python.org/issue7238 [2] http://bugs.python.org/issue16482 [3] http://bugs.python.org/issue17697 [4] http://bugs.python.org/issue17277 [5] http://bugs.python.org/issue17288 Xavier From timothy.c.delaney at gmail.com Thu Sep 12 23:29:30 2013 From: timothy.c.delaney at gmail.com (Tim Delaney) Date: Fri, 13 Sep 2013 07:29:30 +1000 Subject: [Python-Dev] Add a "transformdict" to collections In-Reply-To: <20130912174031.5971338f@pitrou.net> References: <20130910112832.75ed6c8b@pitrou.net> <011DB128-7308-40C0-BE93-A691A26B2C1B@gmail.com> <20130911102732.2cb268ca@pitrou.net> <CANc-5Uw-66ZPYC=mOu4z+vHU74qHY4QDWq2bCLDCFBku+cAqcg@mail.gmail.com> <20130911134032.GA16820@ando> <CAMpsgwbWF_7-EDKoQZz61BNk3mVNieMkVCquwweMuG=HSK4RAQ@mail.gmail.com> <52307DE0.9040208@stoneleaf.us> <CAMpsgwb1iB7fMKjYX5T3HR0E+3rmt4H055z2cjyJ-fMEjoEe6Q@mail.gmail.com> <52309446.6070809@stoneleaf.us> <CAN8CLg=MzvW77F9XKCa3g+26Ufby6EwGyO5yTQbw+--A0iQhYA@mail.gmail.com> <5231CAEF.1080500@stoneleaf.us> <20130912164300.5cd27a1a@pitrou.net> <5231D848.1090709@stoneleaf.us> <20130912174031.5971338f@pitrou.net> Message-ID: <CAN8CLgk3eG_uf7M-f0AuQX0XXq4t2B7JOjS2DL40NC94QioabA@mail.gmail.com> On 13 September 2013 01:40, Antoine Pitrou <solipsis at pitrou.net> wrote: > Le Thu, 12 Sep 2013 08:05:44 -0700, > Ethan Furman <ethan at stoneleaf.us> a ?crit : > > On 09/12/2013 07:43 AM, Antoine Pitrou wrote: > > > > > > Yeah, so this is totally silly. What you're basically saying is "we > > > don't need TransformDict since people can re-implement it > > > themselves". > > > > No, what I'm saying is that the "case-preserving" aspect of > > transformdict is silly. The main point of transformdict is to > > enable, for example, 'IBM', 'Ibm', and 'ibm' to all match up as the > > same key. But why? Because you don't trust the user data. And if > > you don't trust the user data you have to add the correct version of > > the key yourself before you ever process that data, which means you > > already have the correct version stored somewhere. > > That's assuming there is an a priori "correct" version. But there might > not be any. Keeping the original key is important for different reasons > depending on the use case: > > - for case-insensitive dicts, you want to keep the original key for > presentation, logging and debugging purposes (*) > > - for identity dicts, the original key is mandatory because the id() > value in itself is completely useless, it's just used for matching > > (*) For a well-known example of such behaviour, think about Windows > filesystems. > In this case though, there are two pieces of information: 1. A canonical key (which may or may not equal the original key); 2. The original key. It seems to me then that TransformDict is a specialised case of CanonicalDict, where the canonical key is defined to be the first key inserted. It would in fact be possible (though inefficient) to implement that using a canonicalising callable that maintained state - something like (untested): class OriginalKeys: def __init__(self):: self.keys = CanonicalDict(str.lower) def __call__(self, key): return self.keys.setdefault(key, key) class OriginalKeyDict(CanonicalDict): def __init__(self):: super().__init__(OriginalKeys()) Tim Delaney -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130913/ab28b609/attachment.html> From timothy.c.delaney at gmail.com Thu Sep 12 23:34:15 2013 From: timothy.c.delaney at gmail.com (Tim Delaney) Date: Fri, 13 Sep 2013 07:34:15 +1000 Subject: [Python-Dev] Add a "transformdict" to collections In-Reply-To: <CAN8CLgk3eG_uf7M-f0AuQX0XXq4t2B7JOjS2DL40NC94QioabA@mail.gmail.com> References: <20130910112832.75ed6c8b@pitrou.net> <011DB128-7308-40C0-BE93-A691A26B2C1B@gmail.com> <20130911102732.2cb268ca@pitrou.net> <CANc-5Uw-66ZPYC=mOu4z+vHU74qHY4QDWq2bCLDCFBku+cAqcg@mail.gmail.com> <20130911134032.GA16820@ando> <CAMpsgwbWF_7-EDKoQZz61BNk3mVNieMkVCquwweMuG=HSK4RAQ@mail.gmail.com> <52307DE0.9040208@stoneleaf.us> <CAMpsgwb1iB7fMKjYX5T3HR0E+3rmt4H055z2cjyJ-fMEjoEe6Q@mail.gmail.com> <52309446.6070809@stoneleaf.us> <CAN8CLg=MzvW77F9XKCa3g+26Ufby6EwGyO5yTQbw+--A0iQhYA@mail.gmail.com> <5231CAEF.1080500@stoneleaf.us> <20130912164300.5cd27a1a@pitrou.net> <5231D848.1090709@stoneleaf.us> <20130912174031.5971338f@pitrou.net> <CAN8CLgk3eG_uf7M-f0AuQX0XXq4t2B7JOjS2DL40NC94QioabA@mail.gmail.com> Message-ID: <CAN8CLg=L8+URU-JB0-hA+yk4U9M+U4dVgkeDR7t-Fes=5BfHEg@mail.gmail.com> On 13 September 2013 07:29, Tim Delaney <timothy.c.delaney at gmail.com> wrote: > > In this case though, there are two pieces of information: > > 1. A canonical key (which may or may not equal the original key); > > 2. The original key. > > It seems to me then that TransformDict is a specialised case of > CanonicalDict, where the canonical key is defined to be the first key > inserted. It would in fact be possible (though inefficient) to implement > that using a canonicalising callable that maintained state - something like > (untested): > > class OriginalKeys: > def __init__(self):: > self.keys = CanonicalDict(str.lower) > > def __call__(self, key): > return self.keys.setdefault(key, key) > > class OriginalKeyDict(CanonicalDict): > def __init__(self):: > super().__init__(OriginalKeys()) > Bah - got myself mixed up with original key and case preserving there ... try this: class OriginalKeys: def __init__(self, func):: self.keys = CanonicalDict(func) def __call__(self, key): return self.keys.setdefault(key, key) class OriginalKeyDict(CanonicalDict): def __init__(self, func):: super().__init__(OriginalKeys(func)) class IdentityDict(OriginalKeyDict): def __init__(self): super().__init__(id) class CasePreservingDict(OriginalKeyDict): def __init__(self): super().__init__(str.lower) Tim Delaney -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130913/ede12be5/attachment.html> From steve at pearwood.info Fri Sep 13 01:08:21 2013 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 13 Sep 2013 09:08:21 +1000 Subject: [Python-Dev] PEP 447: add type.__locallookup__ In-Reply-To: <8F0FD1DE-A8ED-40D7-878A-9A005ABDF70C@mac.com> References: <75030FAC-6918-4E94-95DA-67A88D53E6F5@mac.com> <7D6D4488-2855-41F8-A9F2-1FF23B9E3A56@mac.com> <522DDB74.5080609@stoneleaf.us> <CAP7+vJJTR=+cRY+iQgHY4ujTsC3_QyWn83dnK96rWO6bV9vFLQ@mail.gmail.com> <522DECA3.2030709@hotpy.org> <CAP7+vJ+thJvYpk0d0rnFO6bm88sTJ_t1QdYFB0SdPOGNDABuag@mail.gmail.com> <522DF204.7040109@hotpy.org> <0752376899dd7436f6115bfc03594574@chopin.edu.pl> <8F0FD1DE-A8ED-40D7-878A-9A005ABDF70C@mac.com> Message-ID: <20130912230820.GC16820@ando> On Thu, Sep 12, 2013 at 04:42:39PM +0200, Ronald Oussoren wrote: > I don't particularly like __locallookup__ either, but haven't found a > better name yet. "__lookup_in_class__" was the best alternative I > could come up with, and that feels different than other special > methods. The name in the PEP is more or less derived from > _PyType_Lookup, with "local" meaning "only in this class, don't > recurse in the rest of the MRO". How about __typelookup__ ? Surely that's the obvious name to derive from _PyType_Lookup :-) -- Steven From Steve.Dower at microsoft.com Fri Sep 13 05:04:49 2013 From: Steve.Dower at microsoft.com (Steve Dower) Date: Fri, 13 Sep 2013 03:04:49 +0000 Subject: [Python-Dev] PEP 447: add type.__locallookup__ In-Reply-To: <20130912230820.GC16820@ando> References: <75030FAC-6918-4E94-95DA-67A88D53E6F5@mac.com> <7D6D4488-2855-41F8-A9F2-1FF23B9E3A56@mac.com> <522DDB74.5080609@stoneleaf.us> <CAP7+vJJTR=+cRY+iQgHY4ujTsC3_QyWn83dnK96rWO6bV9vFLQ@mail.gmail.com> <522DECA3.2030709@hotpy.org> <CAP7+vJ+thJvYpk0d0rnFO6bm88sTJ_t1QdYFB0SdPOGNDABuag@mail.gmail.com> <522DF204.7040109@hotpy.org> <0752376899dd7436f6115bfc03594574@chopin.edu.pl> <8F0FD1DE-A8ED-40D7-878A-9A005ABDF70C@mac.com>,<20130912230820.GC16820@ando> Message-ID: <51c253d972c94acbace9e46fc43b922a@BLUPR03MB199.namprd03.prod.outlook.com> What about __getlocalattribute__ or __getattributenorecurse__? Long, but this isn't going to be used often. Putting "type" or "class" in the name would be misleading. It's an instance method (that is most useful when implemented on a metaclass). (Apologies for the top post.) Sent from my Windows Phone ________________________________ From: Steven D'Aprano<mailto:steve at pearwood.info> Sent: ?9/?12/?2013 16:09 To: python-dev at python.org<mailto:python-dev at python.org> Subject: Re: [Python-Dev] PEP 447: add type.__locallookup__ On Thu, Sep 12, 2013 at 04:42:39PM +0200, Ronald Oussoren wrote: > I don't particularly like __locallookup__ either, but haven't found a > better name yet. "__lookup_in_class__" was the best alternative I > could come up with, and that feels different than other special > methods. The name in the PEP is more or less derived from > _PyType_Lookup, with "local" meaning "only in this class, don't > recurse in the rest of the MRO". How about __typelookup__ ? Surely that's the obvious name to derive from _PyType_Lookup :-) -- Steven _______________________________________________ Python-Dev mailing list Python-Dev at python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/steve.dower%40microsoft.com -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130913/a7be4a71/attachment.html> From steve at pearwood.info Fri Sep 13 05:59:17 2013 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 13 Sep 2013 13:59:17 +1000 Subject: [Python-Dev] PEP 447: add type.__locallookup__ In-Reply-To: <51c253d972c94acbace9e46fc43b922a@BLUPR03MB199.namprd03.prod.outlook.com> References: <75030FAC-6918-4E94-95DA-67A88D53E6F5@mac.com> <7D6D4488-2855-41F8-A9F2-1FF23B9E3A56@mac.com> <522DDB74.5080609@stoneleaf.us> <CAP7+vJJTR=+cRY+iQgHY4ujTsC3_QyWn83dnK96rWO6bV9vFLQ@mail.gmail.com> <522DECA3.2030709@hotpy.org> <CAP7+vJ+thJvYpk0d0rnFO6bm88sTJ_t1QdYFB0SdPOGNDABuag@mail.gmail.com> <522DF204.7040109@hotpy.org> <0752376899dd7436f6115bfc03594574@chopin.edu.pl> <51c253d972c94acbace9e46fc43b922a@BLUPR03MB199.namprd03.prod.outlook.com> Message-ID: <20130913035917.GE16820@ando> On Fri, Sep 13, 2013 at 03:04:49AM +0000, Steve Dower wrote: > What about __getlocalattribute__ or __getattributenorecurse__? Long, > but this isn't going to be used often. This has nothing to do with locals, nor does it have anything to do with recursion, so both those names are misleading. > Putting "type" or "class" in the name would be misleading. It's an > instance method (that is most useful when implemented on a metaclass). Regardless of whether it is an instance method or not, by default it performs the lookup on the type. Hence the C function _PyType_Lookup and hence my suggestion __typelookup__. But I think that __typelookup__ does describe quite well what the method does. It looks up on the type. The PEP is fairly clear on how this is supposed to work, e.g. the default type.__<whatever>__ method will look up in the class/type dict. PEP 447 includes an example of how you might implement this in Python: class MetaType(type): def __locallookup__(cls, name): try: return cls.__dict__[name] except KeyError: raise AttributeError(name) from None "local lookup" doesn't even come close to describing what the method does or why you would use it. It suggests something to do with locals, which is not the case. Neither does __getattributenorecurse__, which suggests looking up an attribute on an object without following the inheritance hierarchy, e.g. looking in the instance __dict__ but not the class __dict__. So the complete opposite of what it actually does. -- Steven From Steve.Dower at microsoft.com Fri Sep 13 06:26:06 2013 From: Steve.Dower at microsoft.com (Steve Dower) Date: Fri, 13 Sep 2013 04:26:06 +0000 Subject: [Python-Dev] PEP 447: add type.__locallookup__ In-Reply-To: <20130913035917.GE16820@ando> References: <75030FAC-6918-4E94-95DA-67A88D53E6F5@mac.com> <7D6D4488-2855-41F8-A9F2-1FF23B9E3A56@mac.com> <522DDB74.5080609@stoneleaf.us> <CAP7+vJJTR=+cRY+iQgHY4ujTsC3_QyWn83dnK96rWO6bV9vFLQ@mail.gmail.com> <522DECA3.2030709@hotpy.org> <CAP7+vJ+thJvYpk0d0rnFO6bm88sTJ_t1QdYFB0SdPOGNDABuag@mail.gmail.com> <522DF204.7040109@hotpy.org> <0752376899dd7436f6115bfc03594574@chopin.edu.pl> <51c253d972c94acbace9e46fc43b922a@BLUPR03MB199.namprd03.prod.outlook.com>, <20130913035917.GE16820@ando> Message-ID: <392015291ec843afb1bc85632055729d@BLUPR03MB199.namprd03.prod.outlook.com> Last I checked, looking up in the instance dict us exactly what it does. Even the example you posted is doing that. And the only difference from __getattribute__ is that it throws instead of following the MRO, which is intended to allow base classes (via super, and another call to this method) to dynamically respond to a getattr without the cooperation of subclasses. Consider class A, which knows it has a method F, but will not create it until the first __getattribute__ call. Now class B derives from A, and someone calls super(B).F(obj). Currently, super only looks in __dict__ for F, which will fail to invoke A.__getattribute__. Because super is used to provide MRO traversal, it can't rely on B.__getattribute__ to perform the traversal, so it currently has no choice. The PEP was originally adding a special class method to provide a __getattribute__ equivalent that would not traverse the MRO, so that super could use it and people can create dynamic classes that can act as base classes. I pointed out that this could be an instance method (thereby avoid automatic-classmethod magic) and implemented on a metaclass for the class behavior. Unless the PEP has changed recently, this is still an instance method that will look up members defined directly on the type and not on base classes. There may still be valid questions to answer (such as, should overrides if this method on base classes be inherited), but whether it is a type/class method is no longer one of those. Cheers, Steve Sent from my Windows Phone ________________________________ From: Steven D'Aprano<mailto:steve at pearwood.info> Sent: ?9/?12/?2013 21:00 To: python-dev at python.org<mailto:python-dev at python.org> Subject: Re: [Python-Dev] PEP 447: add type.__locallookup__ On Fri, Sep 13, 2013 at 03:04:49AM +0000, Steve Dower wrote: > What about __getlocalattribute__ or __getattributenorecurse__? Long, > but this isn't going to be used often. This has nothing to do with locals, nor does it have anything to do with recursion, so both those names are misleading. > Putting "type" or "class" in the name would be misleading. It's an > instance method (that is most useful when implemented on a metaclass). Regardless of whether it is an instance method or not, by default it performs the lookup on the type. Hence the C function _PyType_Lookup and hence my suggestion __typelookup__. But I think that __typelookup__ does describe quite well what the method does. It looks up on the type. The PEP is fairly clear on how this is supposed to work, e.g. the default type.__<whatever>__ method will look up in the class/type dict. PEP 447 includes an example of how you might implement this in Python: class MetaType(type): def __locallookup__(cls, name): try: return cls.__dict__[name] except KeyError: raise AttributeError(name) from None "local lookup" doesn't even come close to describing what the method does or why you would use it. It suggests something to do with locals, which is not the case. Neither does __getattributenorecurse__, which suggests looking up an attribute on an object without following the inheritance hierarchy, e.g. looking in the instance __dict__ but not the class __dict__. So the complete opposite of what it actually does. -- Steven _______________________________________________ Python-Dev mailing list Python-Dev at python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/steve.dower%40microsoft.com -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130913/fec6fd3c/attachment-0001.html> From steve at pearwood.info Fri Sep 13 08:52:38 2013 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 13 Sep 2013 16:52:38 +1000 Subject: [Python-Dev] PEP 447: add type.__locallookup__ In-Reply-To: <392015291ec843afb1bc85632055729d@BLUPR03MB199.namprd03.prod.outlook.com> References: <75030FAC-6918-4E94-95DA-67A88D53E6F5@mac.com> <7D6D4488-2855-41F8-A9F2-1FF23B9E3A56@mac.com> <522DDB74.5080609@stoneleaf.us> <CAP7+vJJTR=+cRY+iQgHY4ujTsC3_QyWn83dnK96rWO6bV9vFLQ@mail.gmail.com> <522DECA3.2030709@hotpy.org> <CAP7+vJ+thJvYpk0d0rnFO6bm88sTJ_t1QdYFB0SdPOGNDABuag@mail.gmail.com> <522DF204.7040109@hotpy.org> <0752376899dd7436f6115bfc03594574@chopin.edu.pl> <392015291ec843afb1bc85632055729d@BLUPR03MB199.namprd03.prod.outlook.com> Message-ID: <20130913065236.GF16820@ando> On Fri, Sep 13, 2013 at 04:26:06AM +0000, Steve Dower wrote: > Last I checked, looking up in the instance dict us exactly what it > does. Even the example you posted is doing that. The example from the PEP shows: return cls.__dict__[name] not "self.__dict__[name]". It is true that "the instance" in this case refers to it being an instance of the metaclass, but that instance is, in fact, a class/type. That's why we normally call it "cls" in a metaclass method rather than "self". I was reacting to your statement that [quote]Putting "type" or "class" in the name would be misleading[end quote]. I don't believe it is misleading. "type lookup" is exactly what it does: it does a lookup on a type. Take your example below: > Consider class A, which knows it has a method F, but will not create > it until the first __getattribute__ call. Now class B derives from A, > and someone calls super(B).F(obj). [Aside: I'm not sure why you are using an unbound super object super(B) instead of the more usual super(B, obj).F(). Have I missed something?] As I understand it, that ends up calling type(B).__<whatever>__(B, 'F'). So the objects being used are: - the metaclass type(B); - the class B but not the instance self = B(), even though F is a regular instance method on A that ends up seeing self as the first argument. Given this, I believe that "lookup on the type" is exactly what the method does, whether you interpret "the type" as the metaclass (the __<whatever>__ method is called on the metaclass) or the class B (which ends up as the first argument to the <whatever> method). By the way, I think the PEP should have a more complex example. The SillyObject example is nice and easy to understand, but it doesn't really help with the motivating use-case "dynamic classes that can grow new methods on demand". Ronald, if you're reading this, can you add such an example please? Also, there's a typo in the SillyObject M method ("fourtytwo" should not have a U in it). [...] > There may still be valid questions to answer (such as, should > overrides if this method on base classes be inherited), but whether it > is a type/class method is no longer one of those. I don't believe that anyone is arguing that it should be a class method. I'm certainly not. But even if somebody is, that doesn't have anything to do with the name. We write dict.fromkeys(), not dict.typefromkeys(). -- Steven From ncoghlan at gmail.com Fri Sep 13 12:42:46 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 13 Sep 2013 20:42:46 +1000 Subject: [Python-Dev] PEP 447: add type.__locallookup__ In-Reply-To: <20130913065236.GF16820@ando> References: <75030FAC-6918-4E94-95DA-67A88D53E6F5@mac.com> <7D6D4488-2855-41F8-A9F2-1FF23B9E3A56@mac.com> <522DDB74.5080609@stoneleaf.us> <CAP7+vJJTR=+cRY+iQgHY4ujTsC3_QyWn83dnK96rWO6bV9vFLQ@mail.gmail.com> <522DECA3.2030709@hotpy.org> <CAP7+vJ+thJvYpk0d0rnFO6bm88sTJ_t1QdYFB0SdPOGNDABuag@mail.gmail.com> <522DF204.7040109@hotpy.org> <0752376899dd7436f6115bfc03594574@chopin.edu.pl> <392015291ec843afb1bc85632055729d@BLUPR03MB199.namprd03.prod.outlook.com> <20130913065236.GF16820@ando> Message-ID: <CADiSq7fNhAgGUE9-1EdASM5MH0wx7W-mJTi6pRv=45r3K6LvmA@mail.gmail.com> Perhaps "__getdescriptor__" would work as the method name? Yes, it can technically return a non-descriptor, but the *primary* purpose is to customise the retrieval of objects that will be checked to see if they're descriptors. It *won't* be invoked when looking for ordinary attributes in an instance dict, but *will* be invoked when looking on the class object. Cheers, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130913/ef922aa2/attachment.html> From steve at pearwood.info Fri Sep 13 14:23:28 2013 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 13 Sep 2013 22:23:28 +1000 Subject: [Python-Dev] PEP 447: add type.__locallookup__ In-Reply-To: <CADiSq7fNhAgGUE9-1EdASM5MH0wx7W-mJTi6pRv=45r3K6LvmA@mail.gmail.com> References: <7D6D4488-2855-41F8-A9F2-1FF23B9E3A56@mac.com> <522DDB74.5080609@stoneleaf.us> <CAP7+vJJTR=+cRY+iQgHY4ujTsC3_QyWn83dnK96rWO6bV9vFLQ@mail.gmail.com> <522DECA3.2030709@hotpy.org> <CAP7+vJ+thJvYpk0d0rnFO6bm88sTJ_t1QdYFB0SdPOGNDABuag@mail.gmail.com> <522DF204.7040109@hotpy.org> <0752376899dd7436f6115bfc03594574@chopin.edu.pl> <392015291ec843afb1bc85632055729d@BLUPR03MB199.namprd03.prod.outlook.com> <20130913065236.GF16820@ando> <CADiSq7fNhAgGUE9-1EdASM5MH0wx7W-mJTi6pRv=45r3K6LvmA@mail.gmail.com> Message-ID: <20130913122328.GI16820@ando> On Fri, Sep 13, 2013 at 08:42:46PM +1000, Nick Coghlan wrote: > Perhaps "__getdescriptor__" would work as the method name? Yes, it can > technically return a non-descriptor, So technically that name is, um, what's the term... oh yes, "a lie". :-) > but the *primary* purpose is to > customise the retrieval of objects that will be checked to see if they're > descriptors. If that's the case, the PEP should make that clear. [Aside: the PEP states that the method shouldn't invoke descriptors. What's the reason for that? If I take the statement literally, doesn't it mean that the method mustn't use any other methods at all? Surely that can't be what is intended, but I'm not sure what is intended.] > It *won't* be invoked when looking for ordinary attributes in > an instance dict, but *will* be invoked when looking on the class object. Just to be clear, if I have: instance = MyClass() x = instance.name and "name" is found in instance.__dict__, then this special method will not be invoked. But if "name" is not found in the instance dict, then "name" will be looked up on the class object MyClass, which may invoke this special method. Am I correct? -- Steven From eric at trueblade.com Fri Sep 13 16:49:05 2013 From: eric at trueblade.com (Eric V. Smith) Date: Fri, 13 Sep 2013 10:49:05 -0400 Subject: [Python-Dev] [Python-checkins] cpython (merge 3.3 -> default): Fix http.server's request handling case on trailing '/'. In-Reply-To: <3cbpGS1zZSz7Lk4@mail.python.org> References: <3cbpGS1zZSz7Lk4@mail.python.org> Message-ID: <523325E1.70608@trueblade.com> On 9/13/2013 3:22 AM, senthil.kumaran wrote: > http://hg.python.org/cpython/rev/b85c9d2a5227 > changeset: 85672:b85c9d2a5227 > parent: 85668:66ec8431032d > parent: 85671:1fcccbbe15e2 > user: Senthil Kumaran <senthil at uthcode.com> > date: Fri Sep 13 00:22:45 2013 -0700 > summary: > Fix http.server's request handling case on trailing '/'. > > Patch contributed by Vajrasky Kok. Addresses Issue #17324 > + trailing_slash = True if path.rstrip().endswith('/') else False Wouldn't this be better just as: trailing_slash = path.rstrip().endswith('/') -- Eric. From status at bugs.python.org Fri Sep 13 18:07:33 2013 From: status at bugs.python.org (Python tracker) Date: Fri, 13 Sep 2013 18:07:33 +0200 (CEST) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20130913160733.5F38B56A5E@psf.upfronthosting.co.za> ACTIVITY SUMMARY (2013-09-06 - 2013-09-13) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 4212 (+17) closed 26571 (+48) total 30783 (+65) Open issues with patches: 1935 Issues opened (46) ================== #18939: Venv docs regarding original python install http://bugs.python.org/issue18939 reopened by gwideman #18948: deliberately crashing tests should prevent core dumps http://bugs.python.org/issue18948 opened by pitrou #18950: Miscellaneous fixes for the sunau module http://bugs.python.org/issue18950 opened by serhiy.storchaka #18951: In unittest.TestCase.assertRegex change "re" and "regex" to "r http://bugs.python.org/issue18951 opened by py.user #18955: Confusing documentation in Lib/importlib/util.py http://bugs.python.org/issue18955 opened by vajrasky #18956: Document useful functions in ???pydoc??? module http://bugs.python.org/issue18956 opened by bignose #18958: Exception('No JSON object could be decoded') when parsing a va http://bugs.python.org/issue18958 opened by Gallaecio #18959: Create a "Superseded modules" section in standard library ToC http://bugs.python.org/issue18959 opened by ncoghlan #18960: First line can be executed twice http://bugs.python.org/issue18960 opened by serhiy.storchaka #18961: Non-UTF8 encoding line http://bugs.python.org/issue18961 opened by serhiy.storchaka #18965: 2to3 can produce illegal bytes literals http://bugs.python.org/issue18965 opened by serhiy.storchaka #18966: Threads within multiprocessing Process terminate early http://bugs.python.org/issue18966 opened by pietvo #18967: Find a less conflict prone approach to Misc/NEWS http://bugs.python.org/issue18967 opened by ncoghlan #18968: Find a way to detect incorrectly skipped tests http://bugs.python.org/issue18968 opened by ncoghlan #18969: test suite: enable faulthandler timeout in assert_python http://bugs.python.org/issue18969 opened by neologix #18970: run_setup() behavior differs from cli invocation of setup.py http://bugs.python.org/issue18970 opened by l #18971: Use argparse in the profile/cProfile modules http://bugs.python.org/issue18971 opened by serhiy.storchaka #18972: Use argparse in email example scripts http://bugs.python.org/issue18972 opened by serhiy.storchaka #18973: Use argparse in the calendar module http://bugs.python.org/issue18973 opened by serhiy.storchaka #18974: Use argparse in the diff script http://bugs.python.org/issue18974 opened by serhiy.storchaka #18975: timeit: Use thousands separators and print number of loops per http://bugs.python.org/issue18975 opened by jstasiak #18976: distutils/command/build_ext passes wrong linker flags http://bugs.python.org/issue18976 opened by Benedikt.Morbach #18977: The -t option has no effect in for uu command-line http://bugs.python.org/issue18977 opened by serhiy.storchaka #18978: Allow urllib.request.Request subclasses to override method http://bugs.python.org/issue18978 opened by jason.coombs #18979: Use argparse in the uu module http://bugs.python.org/issue18979 opened by serhiy.storchaka #18981: Typo in the ctypes tests http://bugs.python.org/issue18981 opened by Anoop.Thomas.Mathew #18982: Add tests for CLI of the calendar module http://bugs.python.org/issue18982 opened by serhiy.storchaka #18983: Specify time unit for timeit CLI http://bugs.python.org/issue18983 opened by serhiy.storchaka #18985: Improve the documentation in fcntl module http://bugs.python.org/issue18985 opened by vajrasky #18986: Add a case-insensitive case-preserving dict http://bugs.python.org/issue18986 opened by pitrou #18987: distutils.utils.get_platform() for 32-bit Python on a 64-bit m http://bugs.python.org/issue18987 opened by sferencik #18989: reuse of enum names in class creation inconsistent http://bugs.python.org/issue18989 opened by ethan.furman #18990: Remove unnecessary API inconsistency from ElementTree.XMLPullP http://bugs.python.org/issue18990 opened by scoder #18993: There is an overshadowed and invalid test in testmock.py http://bugs.python.org/issue18993 opened by vajrasky #18994: Inside fcntl module, we does not check the return code of all_ http://bugs.python.org/issue18994 opened by vajrasky #18995: Enum does not work with reversed http://bugs.python.org/issue18995 opened by vajrasky #18996: unittest: more helpful truncating long strings http://bugs.python.org/issue18996 opened by serhiy.storchaka #18998: iter() not working in ElementTree http://bugs.python.org/issue18998 opened by Kronuz #18999: Robustness issues in multiprocessing.{get,set}_start_method http://bugs.python.org/issue18999 opened by larsmans #19001: test_gdb fails on Fedora buildbot http://bugs.python.org/issue19001 opened by pitrou #19003: email.generator.BytesGenerator corrupts data by changing line http://bugs.python.org/issue19003 opened by Alexander.Kruppa #19005: PyIter_Next crashes if passed a non-iterator http://bugs.python.org/issue19005 opened by abacabadabacaba #19006: UnitTest docs should have a single list of assertions http://bugs.python.org/issue19006 opened by roysmith #19007: precise time.time() under Windows 8 http://bugs.python.org/issue19007 opened by pitrou #19009: Enhance HTTPResponse.readline() performance http://bugs.python.org/issue19009 opened by kristjan.jonsson #19010: Make XMLPullParser in ElementTree inherit from XMLParser http://bugs.python.org/issue19010 opened by scoder Most recent 15 issues with no replies (15) ========================================== #19010: Make XMLPullParser in ElementTree inherit from XMLParser http://bugs.python.org/issue19010 #19005: PyIter_Next crashes if passed a non-iterator http://bugs.python.org/issue19005 #19003: email.generator.BytesGenerator corrupts data by changing line http://bugs.python.org/issue19003 #19001: test_gdb fails on Fedora buildbot http://bugs.python.org/issue19001 #18996: unittest: more helpful truncating long strings http://bugs.python.org/issue18996 #18994: Inside fcntl module, we does not check the return code of all_ http://bugs.python.org/issue18994 #18985: Improve the documentation in fcntl module http://bugs.python.org/issue18985 #18982: Add tests for CLI of the calendar module http://bugs.python.org/issue18982 #18981: Typo in the ctypes tests http://bugs.python.org/issue18981 #18979: Use argparse in the uu module http://bugs.python.org/issue18979 #18977: The -t option has no effect in for uu command-line http://bugs.python.org/issue18977 #18976: distutils/command/build_ext passes wrong linker flags http://bugs.python.org/issue18976 #18974: Use argparse in the diff script http://bugs.python.org/issue18974 #18972: Use argparse in email example scripts http://bugs.python.org/issue18972 #18971: Use argparse in the profile/cProfile modules http://bugs.python.org/issue18971 Most recent 15 issues waiting for review (15) ============================================= #19009: Enhance HTTPResponse.readline() performance http://bugs.python.org/issue19009 #18999: Robustness issues in multiprocessing.{get,set}_start_method http://bugs.python.org/issue18999 #18996: unittest: more helpful truncating long strings http://bugs.python.org/issue18996 #18995: Enum does not work with reversed http://bugs.python.org/issue18995 #18994: Inside fcntl module, we does not check the return code of all_ http://bugs.python.org/issue18994 #18993: There is an overshadowed and invalid test in testmock.py http://bugs.python.org/issue18993 #18990: Remove unnecessary API inconsistency from ElementTree.XMLPullP http://bugs.python.org/issue18990 #18989: reuse of enum names in class creation inconsistent http://bugs.python.org/issue18989 #18986: Add a case-insensitive case-preserving dict http://bugs.python.org/issue18986 #18985: Improve the documentation in fcntl module http://bugs.python.org/issue18985 #18982: Add tests for CLI of the calendar module http://bugs.python.org/issue18982 #18981: Typo in the ctypes tests http://bugs.python.org/issue18981 #18979: Use argparse in the uu module http://bugs.python.org/issue18979 #18978: Allow urllib.request.Request subclasses to override method http://bugs.python.org/issue18978 #18977: The -t option has no effect in for uu command-line http://bugs.python.org/issue18977 Top 10 most discussed issues (10) ================================= #18986: Add a case-insensitive case-preserving dict http://bugs.python.org/issue18986 46 msgs #18943: argparse: default args in mutually exclusive groups http://bugs.python.org/issue18943 12 msgs #18999: Robustness issues in multiprocessing.{get,set}_start_method http://bugs.python.org/issue18999 12 msgs #17797: Visual C++ 11.0 reports fileno(stdin) == 0 for non-console pro http://bugs.python.org/issue17797 10 msgs #18967: Find a less conflict prone approach to Misc/NEWS http://bugs.python.org/issue18967 10 msgs #18975: timeit: Use thousands separators and print number of loops per http://bugs.python.org/issue18975 10 msgs #18987: distutils.utils.get_platform() for 32-bit Python on a 64-bit m http://bugs.python.org/issue18987 10 msgs #5815: locale.getdefaultlocale() missing corner case http://bugs.python.org/issue5815 9 msgs #18829: csv produces confusing error message when passed a non-string http://bugs.python.org/issue18829 8 msgs #1565525: tracebacks eat up memory by holding references to locals and g http://bugs.python.org/issue1565525 8 msgs Issues closed (46) ================== #14927: add "Do not supply 'int' argument" to random.shuffle docstring http://bugs.python.org/issue14927 closed by orsenthil #14971: (unittest) loadTestsFromName does not work on method with a de http://bugs.python.org/issue14971 closed by python-dev #17324: SimpleHTTPServer serves files even if the URL has a trailing s http://bugs.python.org/issue17324 closed by orsenthil #18033: Example for Profile Module shows incorrect method http://bugs.python.org/issue18033 closed by orsenthil #18206: license url in site.py should always use X.Y.Z form of version http://bugs.python.org/issue18206 closed by orsenthil #18301: In itertools.chain.from_iterable() there is no cls argument http://bugs.python.org/issue18301 closed by rhettinger #18438: Obsolete url in comment inside decimal module http://bugs.python.org/issue18438 closed by orsenthil #18553: os.isatty() is not Unix only http://bugs.python.org/issue18553 closed by orsenthil #18623: Factor out the _SuppressCoreFiles context manager http://bugs.python.org/issue18623 closed by pitrou #18752: Make chain.from_iterable an alias for a new chain_iterable. http://bugs.python.org/issue18752 closed by rhettinger #18784: minor uuid.py loading optimization http://bugs.python.org/issue18784 closed by serhiy.storchaka #18800: Document Fraction's numerator and denominator properties http://bugs.python.org/issue18800 closed by orsenthil #18808: Thread.join returns before PyThreadState is destroyed http://bugs.python.org/issue18808 closed by pitrou #18815: DOCUMENTATION: "mmap .close()" doesn't close the underlying fi http://bugs.python.org/issue18815 closed by orsenthil #18818: Empty PYTHONIOENCODING is not the same as nonexistent http://bugs.python.org/issue18818 closed by serhiy.storchaka #18821: Add .lastitem attribute to takewhile instances http://bugs.python.org/issue18821 closed by rhettinger #18852: site.py does not handle readline.__doc__ being None http://bugs.python.org/issue18852 closed by r.david.murray #18894: In unittest.TestResult.failures remove deprecated fail* method http://bugs.python.org/issue18894 closed by ezio.melotti #18895: In unittest.TestResult.addError split the sentence http://bugs.python.org/issue18895 closed by ezio.melotti #18904: Unnecessary test in file descriptor inheritance test http://bugs.python.org/issue18904 closed by haypo #18908: Enum docs: sections leak out http://bugs.python.org/issue18908 closed by python-dev #18915: ssl.wrap_socket, pass in certfile and keyfile as PEM strings http://bugs.python.org/issue18915 closed by asvetlov #18917: python won't display greek characters in apache under windows http://bugs.python.org/issue18917 closed by nickl1 #18934: multiprocessing: use selectors module http://bugs.python.org/issue18934 closed by neologix #18935: test_regrtest.test_timeout failure http://bugs.python.org/issue18935 closed by neologix #18944: Minor mistake in test_set.py http://bugs.python.org/issue18944 closed by tim.peters #18946: HTMLParser should ignore errors when parsing text in <script> http://bugs.python.org/issue18946 closed by ezio.melotti #18947: Bug in module netaddr http://bugs.python.org/issue18947 closed by r.david.murray #18949: codeop possible flow error http://bugs.python.org/issue18949 closed by serhiy.storchaka #18952: Fix test.support.open_urlresource (support package regression) http://bugs.python.org/issue18952 closed by python-dev #18953: Typo in NEWS about fixed format specifiers for Py_ssize_t in d http://bugs.python.org/issue18953 closed by python-dev #18954: Grammatically incorrect sentences in Python/fileutils.c http://bugs.python.org/issue18954 closed by python-dev #18957: PYTHONFAULTHANDLER enables faulthandler when the variable is e http://bugs.python.org/issue18957 closed by python-dev #18962: Add special case for single iterator in heapq.merge function http://bugs.python.org/issue18962 closed by rhettinger #18963: test_selectors test_above_fd_setsize cases fail on OS X due to http://bugs.python.org/issue18963 closed by neologix #18964: test_tcl fails when _tkinter linked with Tcl 8.4 http://bugs.python.org/issue18964 closed by python-dev #18980: Docs: enum - ReST format http://bugs.python.org/issue18980 closed by python-dev #18984: Remove .stopped Event from Thread internals http://bugs.python.org/issue18984 closed by tim.peters #18988: Tab key doesn't work sometimes. http://bugs.python.org/issue18988 closed by serhiy.storchaka #18991: Overriding OrderedDict.__getitem__() doesn't work http://bugs.python.org/issue18991 closed by alex #18992: test_sax fails on Windows under 3.4.0a2 http://bugs.python.org/issue18992 closed by loewis #18997: Crash when using pickle and ElementTree http://bugs.python.org/issue18997 closed by eli.bendersky #19000: Alt + <Num> Does Not Work in IDLE. http://bugs.python.org/issue19000 closed by Howitzer21 #19002: ``dir`` function does not work correctly with classes. http://bugs.python.org/issue19002 closed by r.david.murray #19004: datetime: Read in isoformat() output http://bugs.python.org/issue19004 closed by r.david.murray #19008: tkinter: UnboundLocalError: local variable 'sys' referenced be http://bugs.python.org/issue19008 closed by Arfrever From Steve.Dower at microsoft.com Fri Sep 13 18:19:19 2013 From: Steve.Dower at microsoft.com (Steve Dower) Date: Fri, 13 Sep 2013 16:19:19 +0000 Subject: [Python-Dev] PEP 447: add type.__locallookup__ In-Reply-To: <20130913065236.GF16820@ando> References: <75030FAC-6918-4E94-95DA-67A88D53E6F5@mac.com> <7D6D4488-2855-41F8-A9F2-1FF23B9E3A56@mac.com> <522DDB74.5080609@stoneleaf.us> <CAP7+vJJTR=+cRY+iQgHY4ujTsC3_QyWn83dnK96rWO6bV9vFLQ@mail.gmail.com> <522DECA3.2030709@hotpy.org> <CAP7+vJ+thJvYpk0d0rnFO6bm88sTJ_t1QdYFB0SdPOGNDABuag@mail.gmail.com> <522DF204.7040109@hotpy.org> <0752376899dd7436f6115bfc03594574@chopin.edu.pl> <392015291ec843afb1bc85632055729d@BLUPR03MB199.namprd03.prod.outlook.com> <20130913065236.GF16820@ando> Message-ID: <f482c1d7f04f49d99616f3ca3c3b82da@BLUPR03MB199.namprd03.prod.outlook.com> From: Steven D'Aprano > On Fri, Sep 13, 2013 at 04:26:06AM +0000, Steve Dower wrote: > >> Last I checked, looking up in the instance dict us exactly what it >> does. Even the example you posted is doing that. > > The example from the PEP shows: > > return cls.__dict__[name] > > not "self.__dict__[name]". It is true that "the instance" in this case refers to > it being an instance of the metaclass, but that instance is, in fact, a > class/type. That's why we normally call it "cls" in a metaclass method rather > than "self". Right, but where's the difference between these two? class A: def __tdb__(self, name): if name == 'some_attribute_on_my_instance_of_A': return its_value try: return self.__dict__[name] except KeyError: raise AttributeError(name) class MetaB: def __tdb__(cls, name): if name == 'some_attribute_on_my_class_B': return its_value try: return cls.__dict__[name] except KeyError: raise AttributeError(name) (Yes, either of these could be written with __getattribute__, but that function cannot be called by super().) As I see it, there are two (correct) ways to interpret what this method is for, which influences what it should be called. 1. It directly replaces obj.__dict__[name] everywhere that is done, including internally in the interpreter. 2. It is the same as __getattribute__ without the final call to object.__getattribute__ I guess it's also correct to view it as a special helper for super(), but it is more generally applicable than that. [...] > By the way, I think the PEP should have a more complex example. The SillyObject > example is nice and easy to understand, but it doesn't really help with the > motivating use-case "dynamic classes that can grow new methods on demand". > Ronald, if you're reading this, can you add such an example please? Also, > there's a typo in the SillyObject M method ("fourtytwo" should not have a U in > it). Agreed. No harm in more examples. Here's a quick example of code that does not behave correctly at present. class A: def __getattribute__(self, name): if name == 'foo': return 'A.foo' return object.__getattribute__(self, name) class B(A): def get_foo(self): return super().foo >>> B().get_foo() # skips A.__getattribute__ Traceback (most recent call last): File "<stdin>", line 1, in <module> File "<stdin>", line 3, in get_foo AttributeError: 'super' object has no attribute 'foo' >>> B().foo A.foo After changing to use the __tbd__ method: class A: def __getattribute__(self, name): '''Not strictly necessary for this example, but I would expect that most types overriding __tbd__ also want to override __getattribute__ to use it. Or maybe object.__getattribute__ should be changed to use __tbd__ too...?''' try: return self.__tbd__(name) except AttributeError: return object.__getattribute__(self, name) def __tbd__(self, name): # CHANGED if name == 'foo': return 'A.foo' try: return self.__dict__[name] except KeyError: raise AttributeError(name) # CHANGED class B(A): def get_foo(self): return super().foo >>> B().get_foo() # does not skip A.__tbd__ A.foo # hopefully this is the result :) >>> B().foo A.foo A full example of where this may realistically be needed is longer and certainly involves metaclasses, but fundamentally it's just the same as __getattribute__ with slightly different semantics. Cheers, Steve From solipsis at pitrou.net Fri Sep 13 20:40:58 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 13 Sep 2013 20:40:58 +0200 Subject: [Python-Dev] PEP 455: TransformDict Message-ID: <20130913204058.518f43d3@fsol> Hello, Following the python-dev discussion, I've written a PEP to recap the proposal and the various arguments. It's inlined below, and it will probably appear soon at http://www.python.org/dev/peps/pep-0455/, too. Regards Antoine. PEP: 455 Title: Adding a key-transforming dictionary to collections Version: $Revision$ Last-Modified: $Date$ Author: Antoine Pitrou <solipsis at pitrou.net> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 13-Sep-2013 Python-Version: 3.4 Post-History: Abstract ======== This PEP proposes a new data structure for the ``collections`` module, called "TransformDict" in this PEP. This structure is a mutable mapping which transforms the key using a given function when doing a lookup, but retains the original key when reading. Rationale ========= Numerous specialized versions of this pattern exist. The most common is a case-insensitive case-preserving dict, i.e. a dict-like container which matches keys in a case-insensitive fashion but retains the original casing. It is a very common need in network programming, as many protocols feature some arrays of "key / value" properties in their messages, where the keys are textual strings whose casing isn't relevant. Another common request is an identity dict, where keys are matched according to their respective id()s instead of normal matching. Both are instances of a more general pattern, where a given transformation function is applied to keys when looking them up: that function being ``str.lower`` in the former example and the built-in ``id`` function in the latter. (it can be said that the pattern *projects* keys from the user-visible set onto the internal lookup set, hence this PEP's title) Semantics ========= TransformDict is a ``MutableMapping`` implementation: it faithfully implements the well-known API of mutable mappings, as ``dict`` itself and other dict-like classes in the standard library. Therefore, this PEP won't rehash the semantics of most TransformDict methods. The transformation function needn't be bijective, it can be strictly surjective as in the case-insensitive example:: >>> d = TransformDict(str.lower) >>> d['SomeKey'] = 5 >>> d['somekey'] 5 >>> d['SOMEKEY'] 5 TransformDict retains the first key used when creating an entry:: >>> d = TransformDict(str.lower) >>> d['SomeKey'] = 1 >>> d['somekey'] = 2 >>> list(d.items()) [('SomeKey', 2)] The original keys needn't be hashable, as long as the transformation function returns a hashable one:: >>> d = TransformDict(id) >>> l = [None] >>> d[l] = 5 >>> l in d True Constructor ----------- As shown in the example aboves, creating a TransformDict requires passing the key transformation function as the first argument (much like creating a ``defaultdict`` requires passing the factory function as first argument). The constructor also takes other optional arguments which can be used to initialize the TransformDict with certain key-value pairs. Those optional arguments are the same as in the ``dict`` and ``defaultdict`` constructors:: >>> d = TransformDict(str.lower, [('Foo': 1)], Bar=2) >>> sorted(d.items()) [('Bar', 2), ('Foo', 1)] Alternative proposals and questions =================================== Retaining the last original key ------------------------------- Most python-dev respondents found retaining the first user-supplied key more intuitive than retaining the last. Also, it matches the dict object's own behaviour when using different but equal keys:: >>> d = {} >>> d[1] = 'hello' >>> d[1.0] = 'world' >>> d {1: 'world'} Furthermore, explicitly retaining the last key in a first-key-retaining scheme is still possible using the following approach:: d.pop(key, None) d[key] = value while the converse (retaining the first key in a last-key-retaining scheme) doesn't look possible without rewriting part of the container's code. Using an encoder / decoder pair ------------------------------- Using a function pair isn't necessary, since the original key is retained by the container. Moreover, an encoder / decoder pair would require the transformation to be bijective, which prevents important use cases like case-insensitive matching. Providing a transformation function for values ---------------------------------------------- Dictionary values are not used for lookup, their semantics are totally irrelevant to the container's operation. Therefore, there is no point in having both an "original" and a "transformed" value: the transformed value wouldn't be used for anything. Providing a specialized container, not generic ---------------------------------------------- It was asked why we would provide the generic TransformDict construct rather than a specialized case-insensitive dict variant. The answer is that it's nearly as cheap (code-wise and performance-wise) to provide the generic construct, and it can fill more use cases. Implementation ============== A patch for the collections module is tracked on the bug tracker at http://bugs.python.org/issue18986. Existing work ============= Case-insensitive dicts are a popular request: * http://twistedmatrix.com/documents/current/api/twisted.python.util.InsensitiveDict.html * https://mail.python.org/pipermail/python-list/2013-May/647243.html * https://mail.python.org/pipermail/python-list/2005-April/296208.html * https://mail.python.org/pipermail/python-list/2004-June/241748.html * http://bugs.python.org/msg197376 * http://stackoverflow.com/a/2082169 * http://stackoverflow.com/a/3296782 * http://code.activestate.com/recipes/66315-case-insensitive-dictionary/ * https://gist.github.com/babakness/3901174 * http://www.wikier.org/blog/key-insensitive-dictionary-in-python * http://en.sharejs.com/python/14534 * http://www.voidspace.org.uk/python/archive.shtml#caseless Identity dicts have been requested too: * https://mail.python.org/pipermail/python-ideas/2010-May/007235.html * http://www.gossamer-threads.com/lists/python/python/209527 Python's own pickle module uses identity lookups for object memoization: http://hg.python.org/cpython/file/0e70bf1f32a3/Lib/pickle.py#l234 Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: From storchaka at gmail.com Fri Sep 13 21:31:02 2013 From: storchaka at gmail.com (Serhiy Storchaka) Date: Fri, 13 Sep 2013 22:31:02 +0300 Subject: [Python-Dev] PEP 455: TransformDict In-Reply-To: <20130913204058.518f43d3@fsol> References: <20130913204058.518f43d3@fsol> Message-ID: <l0vp4p$3lj$1@ger.gmane.org> 13.09.13 21:40, Antoine Pitrou ???????(??): > Both are instances of a more general pattern, where a given > transformation function is applied to keys when looking them up: that > function being ``str.lower`` in the former example and the built-in > ``id`` function in the latter. Please use str.casefold in examples. > >>> d = TransformDict(str.lower, [('Foo': 1)], Bar=2) {'Foo': 1} or [('Foo', 1)]. > Providing a specialized container, not generic > ---------------------------------------------- > > It was asked why we would provide the generic TransformDict construct > rather than a specialized case-insensitive dict variant. The answer > is that it's nearly as cheap (code-wise and performance-wise) to provide > the generic construct, and it can fill more use cases. Except lightweight IdentityDict which can be implemented more efficient than TransformDict(id). It doesn't need in calling the transform function, computing the hash of transformed key, comparing keys. But perhaps in many cases TransformDict(id) is enough. > Python's own pickle module uses identity lookups for object > memoization: > http://hg.python.org/cpython/file/0e70bf1f32a3/Lib/pickle.py#l234 Also copy, json, cProfile, doctest and _threading_local. From ethan at stoneleaf.us Fri Sep 13 20:49:28 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 13 Sep 2013 11:49:28 -0700 Subject: [Python-Dev] PEP 455: TransformDict In-Reply-To: <20130913204058.518f43d3@fsol> References: <20130913204058.518f43d3@fsol> Message-ID: <52335E38.5010309@stoneleaf.us> +1 :) From solipsis at pitrou.net Fri Sep 13 21:37:25 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 13 Sep 2013 21:37:25 +0200 Subject: [Python-Dev] PEP 455: TransformDict References: <20130913204058.518f43d3@fsol> <l0vp4p$3lj$1@ger.gmane.org> Message-ID: <20130913213725.19dd8817@fsol> On Fri, 13 Sep 2013 22:31:02 +0300 Serhiy Storchaka <storchaka at gmail.com> wrote: > 13.09.13 21:40, Antoine Pitrou ???????(??): > > Both are instances of a more general pattern, where a given > > transformation function is applied to keys when looking them up: that > > function being ``str.lower`` in the former example and the built-in > > ``id`` function in the latter. > > Please use str.casefold in examples. > > > >>> d = TransformDict(str.lower, [('Foo': 1)], Bar=2) > > {'Foo': 1} or [('Foo', 1)]. Ok, thanks. > > Providing a specialized container, not generic > > ---------------------------------------------- > > > > It was asked why we would provide the generic TransformDict construct > > rather than a specialized case-insensitive dict variant. The answer > > is that it's nearly as cheap (code-wise and performance-wise) to provide > > the generic construct, and it can fill more use cases. > > Except lightweight IdentityDict which can be implemented more efficient > than TransformDict(id). It doesn't need in calling the transform > function, computing the hash of transformed key, comparing keys. > But > perhaps in many cases TransformDict(id) is enough. That's true. But it's only important if TransformDict is the bottleneck. I doubt the memoizing dictionary is a bottleneck in e.g. the pure Python implementation of pickle or json. > > Python's own pickle module uses identity lookups for object > > memoization: > > http://hg.python.org/cpython/file/0e70bf1f32a3/Lib/pickle.py#l234 > > Also copy, json, cProfile, doctest and _threading_local. Thanks, will add them. Regards Antoine. From jsbueno at python.org.br Fri Sep 13 21:54:01 2013 From: jsbueno at python.org.br (Joao S. O. Bueno) Date: Fri, 13 Sep 2013 16:54:01 -0300 Subject: [Python-Dev] PEP 455: TransformDict In-Reply-To: <20130913213725.19dd8817@fsol> References: <20130913204058.518f43d3@fsol> <l0vp4p$3lj$1@ger.gmane.org> <20130913213725.19dd8817@fsol> Message-ID: <CAH0mxTRZ7v5H-7MudrYSc32BbRTW8bu4thznas52LcXKW7HDUg@mail.gmail.com> I see the PEP does not contemplate a way to retrieve the original key - like we've talked about somewhere along the thread. On 13 September 2013 16:37, Antoine Pitrou <solipsis at pitrou.net> wrote: > On Fri, 13 Sep 2013 22:31:02 +0300 > Serhiy Storchaka <storchaka at gmail.com> wrote: >> 13.09.13 21:40, Antoine Pitrou ???????(??): >> > Both are instances of a more general pattern, where a given >> > transformation function is applied to keys when looking them up: that >> > function being ``str.lower`` in the former example and the built-in >> > ``id`` function in the latter. >> >> Please use str.casefold in examples. >> >> > >>> d = TransformDict(str.lower, [('Foo': 1)], Bar=2) >> >> {'Foo': 1} or [('Foo', 1)]. > > Ok, thanks. > >> > Providing a specialized container, not generic >> > ---------------------------------------------- >> > >> > It was asked why we would provide the generic TransformDict construct >> > rather than a specialized case-insensitive dict variant. The answer >> > is that it's nearly as cheap (code-wise and performance-wise) to provide >> > the generic construct, and it can fill more use cases. >> >> Except lightweight IdentityDict which can be implemented more efficient >> than TransformDict(id). It doesn't need in calling the transform >> function, computing the hash of transformed key, comparing keys. >> But >> perhaps in many cases TransformDict(id) is enough. > > That's true. But it's only important if TransformDict is the > bottleneck. I doubt the memoizing dictionary is a bottleneck in > e.g. the pure Python implementation of pickle or json. > >> > Python's own pickle module uses identity lookups for object >> > memoization: >> > http://hg.python.org/cpython/file/0e70bf1f32a3/Lib/pickle.py#l234 >> >> Also copy, json, cProfile, doctest and _threading_local. > > Thanks, will add them. > > Regards > > Antoine. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/jsbueno%40python.org.br From solipsis at pitrou.net Fri Sep 13 22:02:36 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 13 Sep 2013 22:02:36 +0200 Subject: [Python-Dev] PEP 455: TransformDict In-Reply-To: <CAH0mxTRZ7v5H-7MudrYSc32BbRTW8bu4thznas52LcXKW7HDUg@mail.gmail.com> References: <20130913204058.518f43d3@fsol> <l0vp4p$3lj$1@ger.gmane.org> <20130913213725.19dd8817@fsol> <CAH0mxTRZ7v5H-7MudrYSc32BbRTW8bu4thznas52LcXKW7HDUg@mail.gmail.com> Message-ID: <20130913220236.1f27b73f@fsol> On Fri, 13 Sep 2013 16:54:01 -0300 "Joao S. O. Bueno" <jsbueno at python.org.br> wrote: > I see the PEP does not contemplate a way to retrieve the original key > - like we've talked about somewhere along the thread. Indeed. If that's important I can add it. I was hoping to keep very close to the MutableMapping API, to make the PEP as few controversial as possible. Regards Antoine. From storchaka at gmail.com Fri Sep 13 22:06:05 2013 From: storchaka at gmail.com (Serhiy Storchaka) Date: Fri, 13 Sep 2013 23:06:05 +0300 Subject: [Python-Dev] PEP 455: TransformDict In-Reply-To: <20130913213725.19dd8817@fsol> References: <20130913204058.518f43d3@fsol> <l0vp4p$3lj$1@ger.gmane.org> <20130913213725.19dd8817@fsol> Message-ID: <l0vr6i$aso$1@ger.gmane.org> 13.09.13 22:37, Antoine Pitrou ???????(??): > That's true. But it's only important if TransformDict is the > bottleneck. I doubt the memoizing dictionary is a bottleneck in > e.g. the pure Python implementation of pickle or json. It can be used in the C implementation. From rdmurray at bitdance.com Fri Sep 13 22:09:10 2013 From: rdmurray at bitdance.com (R. David Murray) Date: Fri, 13 Sep 2013 16:09:10 -0400 Subject: [Python-Dev] PEP 455: TransformDict In-Reply-To: <20130913204058.518f43d3@fsol> References: <20130913204058.518f43d3@fsol> Message-ID: <20130913200910.697972502EA@webabinitio.net> On Fri, 13 Sep 2013 20:40:58 +0200, Antoine Pitrou <solipsis at pitrou.net> wrote: > Rationale > ========= > > Numerous specialized versions of this pattern exist. The most common > is a case-insensitive case-preserving dict, i.e. a dict-like container > which matches keys in a case-insensitive fashion but retains the > original casing. It is a very common need in network programming, as > many protocols feature some arrays of "key / value" properties in their > messages, where the keys are textual strings whose casing isn't > relevant. This motivation would be stronger if the last phrase was something like "where the keys are textual strings whose case is specified to be ignored on receipt but by either specification or custom is to be preserved or non-trivially canonicalized when retransmitted." > (it can be said that the pattern *projects* keys from the user-visible > set onto the internal lookup set, hence this PEP's title) Not clear what "projects" has to do with the PEP title. > Constructor > ----------- > > As shown in the example aboves, creating a TransformDict requires > passing the key transformation function as the first argument (much > like creating a ``defaultdict`` requires passing the factory function > as first argument). > > The constructor also takes other optional arguments which can be used > to initialize the TransformDict with certain key-value pairs. Those > optional arguments are the same as in the ``dict`` and ``defaultdict`` > constructors:: > > >>> d = TransformDict(str.lower, [('Foo': 1)], Bar=2) > >>> sorted(d.items()) > [('Bar', 2), ('Foo', 1)] > > > Alternative proposals and questions > =================================== You don't mention the alternate constructor discussion, or the rationale for following the defaultdict pattern (ie: following the established pattern :) --David From storchaka at gmail.com Fri Sep 13 22:16:10 2013 From: storchaka at gmail.com (Serhiy Storchaka) Date: Fri, 13 Sep 2013 23:16:10 +0300 Subject: [Python-Dev] PEP 455: TransformDict In-Reply-To: <20130913204058.518f43d3@fsol> References: <20130913204058.518f43d3@fsol> Message-ID: <l0vrpf$cso$1@ger.gmane.org> 13.09.13 21:40, Antoine Pitrou ???????(??): > Alternative proposals and questions > =================================== Yet one alternative proposal is to add the dict.__transform__() method. Actually this not contradict TransformDict, but generalize it. TransformDict will be just handly interface to __transform__() as defaultdict to __missing__(). It provides only constructor, repr and pickling. My second patch for http://bugs.python.org/issue18986 shows that as implementation of dict.__transform__(), so implementation of TransformDict using dict.__transform__() are simple enough. From storchaka at gmail.com Fri Sep 13 22:18:39 2013 From: storchaka at gmail.com (Serhiy Storchaka) Date: Fri, 13 Sep 2013 23:18:39 +0300 Subject: [Python-Dev] PEP 455: TransformDict In-Reply-To: <20130913220236.1f27b73f@fsol> References: <20130913204058.518f43d3@fsol> <l0vp4p$3lj$1@ger.gmane.org> <20130913213725.19dd8817@fsol> <CAH0mxTRZ7v5H-7MudrYSc32BbRTW8bu4thznas52LcXKW7HDUg@mail.gmail.com> <20130913220236.1f27b73f@fsol> Message-ID: <l0vrtv$cso$2@ger.gmane.org> 13.09.13 23:02, Antoine Pitrou ???????(??): > On Fri, 13 Sep 2013 16:54:01 -0300 > "Joao S. O. Bueno" <jsbueno at python.org.br> wrote: >> I see the PEP does not contemplate a way to retrieve the original key >> - like we've talked about somewhere along the thread. > > Indeed. If that's important I can add it. I was hoping to keep very > close to the MutableMapping API, to make the PEP as few > controversial as possible. I think that's important. As OrderectDict has additional methods besides the MutableMapping API, so TransformDict should provide useful specialized methods. From solipsis at pitrou.net Fri Sep 13 22:21:40 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 13 Sep 2013 22:21:40 +0200 Subject: [Python-Dev] PEP 455: TransformDict References: <20130913204058.518f43d3@fsol> <l0vrpf$cso$1@ger.gmane.org> Message-ID: <20130913222140.1aaa5742@fsol> On Fri, 13 Sep 2013 23:16:10 +0300 Serhiy Storchaka <storchaka at gmail.com> wrote: > 13.09.13 21:40, Antoine Pitrou ???????(??): > > Alternative proposals and questions > > =================================== > > Yet one alternative proposal is to add the dict.__transform__() method. > Actually this not contradict TransformDict, but generalize it. > TransformDict will be just handly interface to __transform__() as > defaultdict to __missing__(). It provides only constructor, repr and > pickling. Is it an alternative proposal or is it compatible with the PEP? The PEP specifies the API, not the implementation. Regards Antoine. From solipsis at pitrou.net Fri Sep 13 22:22:51 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 13 Sep 2013 22:22:51 +0200 Subject: [Python-Dev] PEP 455: TransformDict References: <20130913204058.518f43d3@fsol> <20130913200910.697972502EA@webabinitio.net> Message-ID: <20130913222251.292d60ca@fsol> On Fri, 13 Sep 2013 16:09:10 -0400 "R. David Murray" <rdmurray at bitdance.com> wrote: > On Fri, 13 Sep 2013 20:40:58 +0200, Antoine Pitrou <solipsis at pitrou.net> wrote: > > Rationale > > ========= > > > > Numerous specialized versions of this pattern exist. The most common > > is a case-insensitive case-preserving dict, i.e. a dict-like container > > which matches keys in a case-insensitive fashion but retains the > > original casing. It is a very common need in network programming, as > > many protocols feature some arrays of "key / value" properties in their > > messages, where the keys are textual strings whose casing isn't > > relevant. > > This motivation would be stronger if the last phrase was something like > "where the keys are textual strings whose case is specified to be ignored > on receipt but by either specification or custom is to be preserved > or non-trivially canonicalized when retransmitted." Thanks, will add. > > (it can be said that the pattern *projects* keys from the user-visible > > set onto the internal lookup set, hence this PEP's title) > > Not clear what "projects" has to do with the PEP title. The PEP was originally titled "a key-projecting dictionary" and I've changed it as it was too obscure. I've now also removed the obsolete part of the sentence above :-) > > Alternative proposals and questions > > =================================== > > You don't mention the alternate constructor discussion, or the > rationale for following the defaultdict pattern (ie: following the > established pattern :) Ok, will do that too. Regards Antoine. From storchaka at gmail.com Fri Sep 13 23:05:27 2013 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sat, 14 Sep 2013 00:05:27 +0300 Subject: [Python-Dev] PEP 455: TransformDict In-Reply-To: <20130913222140.1aaa5742@fsol> References: <20130913204058.518f43d3@fsol> <l0vrpf$cso$1@ger.gmane.org> <20130913222140.1aaa5742@fsol> Message-ID: <l0vuls$lp9$1@ger.gmane.org> 13.09.13 23:21, Antoine Pitrou ???????(??): > On Fri, 13 Sep 2013 23:16:10 +0300 > Serhiy Storchaka <storchaka at gmail.com> wrote: >> 13.09.13 21:40, Antoine Pitrou ???????(??): >>> Alternative proposals and questions >>> =================================== >> >> Yet one alternative proposal is to add the dict.__transform__() method. >> Actually this not contradict TransformDict, but generalize it. >> TransformDict will be just handly interface to __transform__() as >> defaultdict to __missing__(). It provides only constructor, repr and >> pickling. > > Is it an alternative proposal or is it compatible with the PEP? > The PEP specifies the API, not the implementation. Both. On one side, with this proposition TransformDict itself doesn't deserve PEP. It will be trivial and obvious thing. Not very important. On other side, TransformDict can be implemented even without dict.__transform__(), on pure Python. From solipsis at pitrou.net Fri Sep 13 23:20:25 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 13 Sep 2013 23:20:25 +0200 Subject: [Python-Dev] PEP 455: TransformDict References: <20130913204058.518f43d3@fsol> <l0vrpf$cso$1@ger.gmane.org> <20130913222140.1aaa5742@fsol> <l0vuls$lp9$1@ger.gmane.org> Message-ID: <20130913232025.781fe9a8@fsol> On Sat, 14 Sep 2013 00:05:27 +0300 Serhiy Storchaka <storchaka at gmail.com> wrote: > 13.09.13 23:21, Antoine Pitrou ???????(??): > > On Fri, 13 Sep 2013 23:16:10 +0300 > > Serhiy Storchaka <storchaka at gmail.com> wrote: > >> 13.09.13 21:40, Antoine Pitrou ???????(??): > >>> Alternative proposals and questions > >>> =================================== > >> > >> Yet one alternative proposal is to add the dict.__transform__() method. > >> Actually this not contradict TransformDict, but generalize it. > >> TransformDict will be just handly interface to __transform__() as > >> defaultdict to __missing__(). It provides only constructor, repr and > >> pickling. > > > > Is it an alternative proposal or is it compatible with the PEP? > > The PEP specifies the API, not the implementation. > > Both. On one side, with this proposition TransformDict itself doesn't > deserve PEP. It will be trivial and obvious thing. Well, TransformDict would still be the user-visible API, not __transform__; like defaultdict is the user-visible API, not __missing__. Regards Antoine. From rymg19 at gmail.com Fri Sep 13 23:24:22 2013 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Fri, 13 Sep 2013 16:24:22 -0500 Subject: [Python-Dev] PEP 455: TransformDict In-Reply-To: <20130913204058.518f43d3@fsol> References: <20130913204058.518f43d3@fsol> Message-ID: <CAO41-mOx8bGJ+LqnExxwSb8FrBN6UoKaL_gsYRvTy-JRQ5jwCg@mail.gmail.com> You know, I can think up several use cases for the case-insensitive idea. Not sure if I missed something, but there should be a function to retrieve the original key from the modified. i.e.: >>> mydict['SomeStr'] = 5 >>> mydict.get_key('somestr') 'SomeStr' >>> On Fri, Sep 13, 2013 at 1:40 PM, Antoine Pitrou <solipsis at pitrou.net> wrote: > > Hello, > > Following the python-dev discussion, I've written a PEP to recap the > proposal and the various arguments. It's inlined below, and it will > probably appear soon at http://www.python.org/dev/peps/pep-0455/, too. > > Regards > > Antoine. > > > PEP: 455 > Title: Adding a key-transforming dictionary to collections > Version: $Revision$ > Last-Modified: $Date$ > Author: Antoine Pitrou <solipsis at pitrou.net> > Status: Draft > Type: Standards Track > Content-Type: text/x-rst > Created: 13-Sep-2013 > Python-Version: 3.4 > Post-History: > > > Abstract > ======== > > This PEP proposes a new data structure for the ``collections`` module, > called "TransformDict" in this PEP. This structure is a mutable mapping > which transforms the key using a given function when doing a lookup, but > retains the original key when reading. > > > Rationale > ========= > > Numerous specialized versions of this pattern exist. The most common > is a case-insensitive case-preserving dict, i.e. a dict-like container > which matches keys in a case-insensitive fashion but retains the > original casing. It is a very common need in network programming, as > many protocols feature some arrays of "key / value" properties in their > messages, where the keys are textual strings whose casing isn't > relevant. > > Another common request is an identity dict, where keys are matched > according to their respective id()s instead of normal matching. > > Both are instances of a more general pattern, where a given > transformation function is applied to keys when looking them up: that > function being ``str.lower`` in the former example and the built-in > ``id`` function in the latter. > > (it can be said that the pattern *projects* keys from the user-visible > set onto the internal lookup set, hence this PEP's title) > > > Semantics > ========= > > TransformDict is a ``MutableMapping`` implementation: it faithfully > implements the well-known API of mutable mappings, as ``dict`` itself > and other dict-like classes in the standard library. Therefore, this > PEP won't rehash the semantics of most TransformDict methods. > > The transformation function needn't be bijective, it can be strictly > surjective as in the case-insensitive example:: > > >>> d = TransformDict(str.lower) > >>> d['SomeKey'] = 5 > >>> d['somekey'] > 5 > >>> d['SOMEKEY'] > 5 > > TransformDict retains the first key used when creating an entry:: > > >>> d = TransformDict(str.lower) > >>> d['SomeKey'] = 1 > >>> d['somekey'] = 2 > >>> list(d.items()) > [('SomeKey', 2)] > > The original keys needn't be hashable, as long as the transformation > function returns a hashable one:: > > >>> d = TransformDict(id) > >>> l = [None] > >>> d[l] = 5 > >>> l in d > True > > Constructor > ----------- > > As shown in the example aboves, creating a TransformDict requires > passing the key transformation function as the first argument (much > like creating a ``defaultdict`` requires passing the factory function > as first argument). > > The constructor also takes other optional arguments which can be used > to initialize the TransformDict with certain key-value pairs. Those > optional arguments are the same as in the ``dict`` and ``defaultdict`` > constructors:: > > >>> d = TransformDict(str.lower, [('Foo': 1)], Bar=2) > >>> sorted(d.items()) > [('Bar', 2), ('Foo', 1)] > > > Alternative proposals and questions > =================================== > > Retaining the last original key > ------------------------------- > > Most python-dev respondents found retaining the first user-supplied key > more intuitive than retaining the last. Also, it matches the dict > object's own behaviour when using different but equal keys:: > > >>> d = {} > >>> d[1] = 'hello' > >>> d[1.0] = 'world' > >>> d > {1: 'world'} > > Furthermore, explicitly retaining the last key in a first-key-retaining > scheme is still possible using the following approach:: > > d.pop(key, None) > d[key] = value > > while the converse (retaining the first key in a last-key-retaining > scheme) doesn't look possible without rewriting part of the container's > code. > > Using an encoder / decoder pair > ------------------------------- > > Using a function pair isn't necessary, since the original key is > retained by the container. Moreover, an encoder / decoder pair would > require the transformation to be bijective, which prevents important > use cases like case-insensitive matching. > > Providing a transformation function for values > ---------------------------------------------- > > Dictionary values are not used for lookup, their semantics are totally > irrelevant to the container's operation. Therefore, there is no point > in having both an "original" and a "transformed" value: the transformed > value wouldn't be used for anything. > > Providing a specialized container, not generic > ---------------------------------------------- > > It was asked why we would provide the generic TransformDict construct > rather than a specialized case-insensitive dict variant. The answer > is that it's nearly as cheap (code-wise and performance-wise) to provide > the generic construct, and it can fill more use cases. > > > Implementation > ============== > > A patch for the collections module is tracked on the bug tracker at > http://bugs.python.org/issue18986. > > > Existing work > ============= > > Case-insensitive dicts are a popular request: > > * > > http://twistedmatrix.com/documents/current/api/twisted.python.util.InsensitiveDict.html > * https://mail.python.org/pipermail/python-list/2013-May/647243.html > * https://mail.python.org/pipermail/python-list/2005-April/296208.html > * https://mail.python.org/pipermail/python-list/2004-June/241748.html > * http://bugs.python.org/msg197376 > * http://stackoverflow.com/a/2082169 > * http://stackoverflow.com/a/3296782 > * http://code.activestate.com/recipes/66315-case-insensitive-dictionary/ > * https://gist.github.com/babakness/3901174 > * http://www.wikier.org/blog/key-insensitive-dictionary-in-python > * http://en.sharejs.com/python/14534 > * http://www.voidspace.org.uk/python/archive.shtml#caseless > > Identity dicts have been requested too: > > * https://mail.python.org/pipermail/python-ideas/2010-May/007235.html > * http://www.gossamer-threads.com/lists/python/python/209527 > > Python's own pickle module uses identity lookups for object > memoization: > http://hg.python.org/cpython/file/0e70bf1f32a3/Lib/pickle.py#l234 > > > Copyright > ========= > > This document has been placed in the public domain. > > > .. > Local Variables: > mode: indented-text > indent-tabs-mode: nil > sentence-end-double-space: t > fill-column: 70 > coding: utf-8 > End: > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/rymg19%40gmail.com > -- Ryan -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130913/6bda162d/attachment-0001.html> From solipsis at pitrou.net Fri Sep 13 23:45:16 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 13 Sep 2013 23:45:16 +0200 Subject: [Python-Dev] PEP 455: TransformDict References: <20130913204058.518f43d3@fsol> <l0vp4p$3lj$1@ger.gmane.org> <20130913213725.19dd8817@fsol> <CAH0mxTRZ7v5H-7MudrYSc32BbRTW8bu4thznas52LcXKW7HDUg@mail.gmail.com> <20130913220236.1f27b73f@fsol> <l0vrtv$cso$2@ger.gmane.org> Message-ID: <20130913234516.10661214@fsol> On Fri, 13 Sep 2013 23:18:39 +0300 Serhiy Storchaka <storchaka at gmail.com> wrote: > 13.09.13 23:02, Antoine Pitrou ???????(??): > > On Fri, 13 Sep 2013 16:54:01 -0300 > > "Joao S. O. Bueno" <jsbueno at python.org.br> wrote: > >> I see the PEP does not contemplate a way to retrieve the original key > >> - like we've talked about somewhere along the thread. > > > > Indeed. If that's important I can add it. I was hoping to keep very > > close to the MutableMapping API, to make the PEP as few > > controversial as possible. > > I think that's important. As OrderectDict has additional methods besides > the MutableMapping API, so TransformDict should provide useful > specialized methods. Ok, I have a better (IMO) proposal: >>> d = TransformDict(str.casefold, {'Foo': 1}) >>> d.getitem('foo') ('Foo', 1) >>> d.getitem('bar') Traceback (most recent call last): File "<stdin>", line 1, in <module> KeyError: 'bar' Regards Antoine. From ethan at stoneleaf.us Sat Sep 14 00:32:55 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 13 Sep 2013 15:32:55 -0700 Subject: [Python-Dev] PEP 455: TransformDict In-Reply-To: <20130913234516.10661214@fsol> References: <20130913204058.518f43d3@fsol> <l0vp4p$3lj$1@ger.gmane.org> <20130913213725.19dd8817@fsol> <CAH0mxTRZ7v5H-7MudrYSc32BbRTW8bu4thznas52LcXKW7HDUg@mail.gmail.com> <20130913220236.1f27b73f@fsol> <l0vrtv$cso$2@ger.gmane.org> <20130913234516.10661214@fsol> Message-ID: <52339297.3070203@stoneleaf.us> On 09/13/2013 02:45 PM, Antoine Pitrou wrote: > Serhiy Storchaka wrote: >> >> I think that's important. As OrderectDict has additional methods besides >> the MutableMapping API, so TransformDict should provide useful >> specialized methods. > > Ok, I have a better (IMO) proposal: > > >>> d = TransformDict(str.casefold, {'Foo': 1}) > >>> d.getitem('foo') > ('Foo', 1) > >>> d.getitem('bar') > Traceback (most recent call last): > File "<stdin>", line 1, in <module> > KeyError: 'bar' -1 When I want the canonical key, I'm not going to care about the value; and if I do, it's easy enough to get. -- ~Ethan~ From steve at pearwood.info Sat Sep 14 02:49:52 2013 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 14 Sep 2013 10:49:52 +1000 Subject: [Python-Dev] PEP 455: TransformDict In-Reply-To: <20130913234516.10661214@fsol> References: <20130913204058.518f43d3@fsol> <l0vp4p$3lj$1@ger.gmane.org> <20130913213725.19dd8817@fsol> <CAH0mxTRZ7v5H-7MudrYSc32BbRTW8bu4thznas52LcXKW7HDUg@mail.gmail.com> <20130913220236.1f27b73f@fsol> <l0vrtv$cso$2@ger.gmane.org> <20130913234516.10661214@fsol> Message-ID: <20130914004952.GJ16820@ando> On Fri, Sep 13, 2013 at 11:45:16PM +0200, Antoine Pitrou wrote: > Ok, I have a better (IMO) proposal: > > >>> d = TransformDict(str.casefold, {'Foo': 1}) > >>> d.getitem('foo') > ('Foo', 1) > >>> d.getitem('bar') > Traceback (most recent call last): > File "<stdin>", line 1, in <module> > KeyError: 'bar' Is it more common to want both the canonical key and value at the same time, or to just want the canonical key? My gut feeling is that I'm likely to have code like this: d = TransformDict(...) for key in data: key = d.get_canonical(key) value = d[key] print("{}: {}".format(key, value)) in which case having a single call to return both will be great: for key in data: key, value = d.getitem(key) print("{}: {}".format(key, value)) but I'm really not sure. Maybe Ethan is right. I think these sorts of associated questions are why some people (Raymond, Nick) want to see the proposal live outside of the standard library for a while first. The general idea is great, but we're going to bike shed the API without having much idea of how it will actually be used. So, my suggestion is this: - Let's add __transform__ to dicts for 3.4, similar to __missing__, and see how people end up using it in the real world. - Then, in 3.5, we can make a much better informed decision about the best API for a TransformedDict front-end to __transform__. +1 on __transform__ method on dicts. +0 on TransformedDict in 3.4 +1 on waiting for 3.5 based on experience on using __transform__. -- Steven From steve at pearwood.info Sat Sep 14 02:59:21 2013 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 14 Sep 2013 10:59:21 +1000 Subject: [Python-Dev] PEP 450 adding statistics module In-Reply-To: <CAP7+vJJ6QxcUwcpdt3Mq6r07Q2dGUQt-VWGdsYGLv4wrRW5dGg@mail.gmail.com> References: <520C2E23.40405@pearwood.info> <52215BDF.2090109@pearwood.info> <20130908123706.GG26319@ando> <74b03474-ab18-40c5-8ce6-de5159d34903@email.android.com> <522CA085.6000601@stoneleaf.us> <CAP7+vJ+TTaU+ntaLT3xEU2wvFQhgP4P2-FnZUJ7V2PAAy3SMEA@mail.gmail.com> <CAP7+vJJ6QxcUwcpdt3Mq6r07Q2dGUQt-VWGdsYGLv4wrRW5dGg@mail.gmail.com> Message-ID: <20130914005921.GK16820@ando> On Sun, Sep 08, 2013 at 10:51:57AM -0700, Guido van Rossum wrote: > Never mind, I found the patch and the issue. I really think that the > *PEP* is ready for inclusion after the open issues are changed into > something like Discussion or Future Work, and after adding a more > prominent link to the issue with the patch. Then the *patch* can be > reviewed some more until it is ready -- it looks very close already. I've updated the PEP as requested. Is there anything further that needs to be done to have it approved? http://www.python.org/dev/peps/pep-0450/ -- Steven From solipsis at pitrou.net Sat Sep 14 03:02:52 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 14 Sep 2013 03:02:52 +0200 Subject: [Python-Dev] PEP 455: TransformDict References: <20130913204058.518f43d3@fsol> <l0vp4p$3lj$1@ger.gmane.org> <20130913213725.19dd8817@fsol> <CAH0mxTRZ7v5H-7MudrYSc32BbRTW8bu4thznas52LcXKW7HDUg@mail.gmail.com> <20130913220236.1f27b73f@fsol> <l0vrtv$cso$2@ger.gmane.org> <20130913234516.10661214@fsol> <20130914004952.GJ16820@ando> Message-ID: <20130914030252.1b9f4378@fsol> On Sat, 14 Sep 2013 10:49:52 +1000 Steven D'Aprano <steve at pearwood.info> wrote: > > Is it more common to want both the canonical key and value at the same > time, or to just want the canonical key? My gut feeling is that I'm > likely to have code like this: > > d = TransformDict(...) > for key in data: > key = d.get_canonical(key) > value = d[key] > print("{}: {}".format(key, value)) > > in which case having a single call to return both will be great: > > for key in data: > key, value = d.getitem(key) > print("{}: {}".format(key, value)) > > but I'm really not sure. Maybe Ethan is right. I don't think it really matters. From getitem() it's trivial to extract the key alone. > I think these sorts of associated questions are why some people > (Raymond, Nick) want to see the proposal live outside of the standard > library for a while first. The general idea is great, but we're going to > bike shed the API without having much idea of how it will actually be > used. Even after it's used, it will still be bikeshedded: such is how proposals are generally discussed. There really isn't very much to decide here, and whether we only return a key or a (key, value) pair is almost cosmetic: both APIs are reasonably convenient. > So, my suggestion is this: > > - Let's add __transform__ to dicts for 3.4, similar to __missing__, and > see how people end up using it in the real world. Well, __missing__ isn't used very much, I think. People use defaultdict. (note that I'm not even proposing __transform__ right now :-)) > +1 on __transform__ method on dicts. > > +0 on TransformedDict in 3.4 > > +1 on waiting for 3.5 based on experience on using __transform__. Ok, thanks. Regards Antoine. From solipsis at pitrou.net Sat Sep 14 03:03:34 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 14 Sep 2013 03:03:34 +0200 Subject: [Python-Dev] PEP 455: TransformDict References: <20130913204058.518f43d3@fsol> <l0vp4p$3lj$1@ger.gmane.org> <20130913213725.19dd8817@fsol> <CAH0mxTRZ7v5H-7MudrYSc32BbRTW8bu4thznas52LcXKW7HDUg@mail.gmail.com> <20130913220236.1f27b73f@fsol> <l0vrtv$cso$2@ger.gmane.org> <20130913234516.10661214@fsol> <52339297.3070203@stoneleaf.us> Message-ID: <20130914030334.344cec78@fsol> On Fri, 13 Sep 2013 15:32:55 -0700 Ethan Furman <ethan at stoneleaf.us> wrote: > On 09/13/2013 02:45 PM, Antoine Pitrou wrote: > > Serhiy Storchaka wrote: > >> > >> I think that's important. As OrderectDict has additional methods besides > >> the MutableMapping API, so TransformDict should provide useful > >> specialized methods. > > > > Ok, I have a better (IMO) proposal: > > > > >>> d = TransformDict(str.casefold, {'Foo': 1}) > > >>> d.getitem('foo') > > ('Foo', 1) > > >>> d.getitem('bar') > > Traceback (most recent call last): > > File "<stdin>", line 1, in <module> > > KeyError: 'bar' > > -1 > > When I want the canonical key, I'm not going to care about the value; and if I do, it's easy enough to get. But if you don't care about the value, the key is enough to get ;-) From python at mrabarnett.plus.com Sat Sep 14 03:25:42 2013 From: python at mrabarnett.plus.com (MRAB) Date: Sat, 14 Sep 2013 02:25:42 +0100 Subject: [Python-Dev] PEP 455: TransformDict In-Reply-To: <20130914004952.GJ16820@ando> References: <20130913204058.518f43d3@fsol> <l0vp4p$3lj$1@ger.gmane.org> <20130913213725.19dd8817@fsol> <CAH0mxTRZ7v5H-7MudrYSc32BbRTW8bu4thznas52LcXKW7HDUg@mail.gmail.com> <20130913220236.1f27b73f@fsol> <l0vrtv$cso$2@ger.gmane.org> <20130913234516.10661214@fsol> <20130914004952.GJ16820@ando> Message-ID: <5233BB16.8050201@mrabarnett.plus.com> On 14/09/2013 01:49, Steven D'Aprano wrote: > On Fri, Sep 13, 2013 at 11:45:16PM +0200, Antoine Pitrou wrote: > >> Ok, I have a better (IMO) proposal: >> >> >>> d = TransformDict(str.casefold, {'Foo': 1}) >> >>> d.getitem('foo') >> ('Foo', 1) >> >>> d.getitem('bar') >> Traceback (most recent call last): >> File "<stdin>", line 1, in <module> >> KeyError: 'bar' > > > Is it more common to want both the canonical key and value at the same > time, or to just want the canonical key? My gut feeling is that I'm > likely to have code like this: > > > d = TransformDict(...) > for key in data: > key = d.get_canonical(key) > value = d[key] > print("{}: {}".format(key, value)) > I think must be missing something. I thought that iterating over the dict would yield the original keys, so if you wanted the original key and value you would write: for key, value in data.items(): print("{}: {}".format(key, value)) and if you wanted the transformed key you would apply the transform function to the key. > > in which case having a single call to return both will be great: > > for key in data: > key, value = d.getitem(key) > print("{}: {}".format(key, value)) > > > but I'm really not sure. Maybe Ethan is right. > > I think these sorts of associated questions are why some people > (Raymond, Nick) want to see the proposal live outside of the standard > library for a while first. The general idea is great, but we're going to > bike shed the API without having much idea of how it will actually be > used. > > So, my suggestion is this: > > - Let's add __transform__ to dicts for 3.4, similar to __missing__, and > see how people end up using it in the real world. > > - Then, in 3.5, we can make a much better informed decision about the > best API for a TransformedDict front-end to __transform__. > > > +1 on __transform__ method on dicts. > > +0 on TransformedDict in 3.4 > > +1 on waiting for 3.5 based on experience on using __transform__. > From ethan at stoneleaf.us Sat Sep 14 03:00:18 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 13 Sep 2013 18:00:18 -0700 Subject: [Python-Dev] PEP 455: TransformDict In-Reply-To: <20130914004952.GJ16820@ando> References: <20130913204058.518f43d3@fsol> <l0vp4p$3lj$1@ger.gmane.org> <20130913213725.19dd8817@fsol> <CAH0mxTRZ7v5H-7MudrYSc32BbRTW8bu4thznas52LcXKW7HDUg@mail.gmail.com> <20130913220236.1f27b73f@fsol> <l0vrtv$cso$2@ger.gmane.org> <20130913234516.10661214@fsol> <20130914004952.GJ16820@ando> Message-ID: <5233B522.7070104@stoneleaf.us> On 09/13/2013 05:49 PM, Steven D'Aprano wrote: > > +1 on __transform__ method on dicts. What would __transform__ do? Just canonicalize the key, or also save the original key? How much front-end work will each user have to do to actually use this new magic method to good effect? Personally, if there's a bunch of push-back against just adding TransformDict directly, why don't we make it provisional? I thought that was what provisional was for (meaning: we're going to add it, PyPI is not really appropriate, there may be some API changes). -- ~Ethan~ From ethan at stoneleaf.us Sat Sep 14 03:40:06 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 13 Sep 2013 18:40:06 -0700 Subject: [Python-Dev] PEP 455: TransformDict In-Reply-To: <5233BB16.8050201@mrabarnett.plus.com> References: <20130913204058.518f43d3@fsol> <l0vp4p$3lj$1@ger.gmane.org> <20130913213725.19dd8817@fsol> <CAH0mxTRZ7v5H-7MudrYSc32BbRTW8bu4thznas52LcXKW7HDUg@mail.gmail.com> <20130913220236.1f27b73f@fsol> <l0vrtv$cso$2@ger.gmane.org> <20130913234516.10661214@fsol> <20130914004952.GJ16820@ando> <5233BB16.8050201@mrabarnett.plus.com> Message-ID: <5233BE76.2040206@stoneleaf.us> On 09/13/2013 06:25 PM, MRAB wrote: > On 14/09/2013 01:49, Steven D'Aprano wrote: >> >> Is it more common to want both the canonical key and value at the same >> time, or to just want the canonical key? My gut feeling is that I'm >> likely to have code like this: >> >> >> d = TransformDict(...) >> for key in data: >> key = d.get_canonical(key) >> value = d[key] >> print("{}: {}".format(key, value)) >> > I think I must be missing something. I thought that iterating over the > dict would yield the original keys, so if you wanted the original key > and value you would write: > > for key, value in data.items(): > print("{}: {}".format(key, value)) Well, that's certainly how I would do it. ;) > and if you wanted the transformed key you would apply the transform > function to the key. Indeed. The question is: how? It is entirely possible that your function has a TransformDict alone, and no memory of the transform function used to create the dict... If the key transform function were saved directly on the TransformDict instance as, say, .transform_key, then problem solved. -- ~Ethan~ From python at mrabarnett.plus.com Sat Sep 14 04:39:18 2013 From: python at mrabarnett.plus.com (MRAB) Date: Sat, 14 Sep 2013 03:39:18 +0100 Subject: [Python-Dev] PEP 455: TransformDict In-Reply-To: <5233BE76.2040206@stoneleaf.us> References: <20130913204058.518f43d3@fsol> <l0vp4p$3lj$1@ger.gmane.org> <20130913213725.19dd8817@fsol> <CAH0mxTRZ7v5H-7MudrYSc32BbRTW8bu4thznas52LcXKW7HDUg@mail.gmail.com> <20130913220236.1f27b73f@fsol> <l0vrtv$cso$2@ger.gmane.org> <20130913234516.10661214@fsol> <20130914004952.GJ16820@ando> <5233BB16.8050201@mrabarnett.plus.com> <5233BE76.2040206@stoneleaf.us> Message-ID: <5233CC56.8070608@mrabarnett.plus.com> On 14/09/2013 02:40, Ethan Furman wrote: > On 09/13/2013 06:25 PM, MRAB wrote: >> On 14/09/2013 01:49, Steven D'Aprano wrote: >>> >>> Is it more common to want both the canonical key and value at the same >>> time, or to just want the canonical key? My gut feeling is that I'm >>> likely to have code like this: >>> >>> >>> d = TransformDict(...) >>> for key in data: >>> key = d.get_canonical(key) >>> value = d[key] >>> print("{}: {}".format(key, value)) >>> >> I think I must be missing something. I thought that iterating over the >> dict would yield the original keys, so if you wanted the original key >> and value you would write: >> >> for key, value in data.items(): >> print("{}: {}".format(key, value)) > > Well, that's certainly how I would do it. ;) > > >> and if you wanted the transformed key you would apply the transform >> function to the key. > > Indeed. The question is: how? It is entirely possible that your function has a TransformDict alone, and no memory of > the transform function used to create the dict... > > If the key transform function were saved directly on the TransformDict instance as, say, .transform_key, then problem > solved. > defaultdict has .default_factory, so having something like .transform_key would have the added advantage of consistency. From steve at pearwood.info Sat Sep 14 04:44:52 2013 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 14 Sep 2013 12:44:52 +1000 Subject: [Python-Dev] PEP 455: TransformDict In-Reply-To: <5233B522.7070104@stoneleaf.us> References: <20130913204058.518f43d3@fsol> <l0vp4p$3lj$1@ger.gmane.org> <20130913213725.19dd8817@fsol> <CAH0mxTRZ7v5H-7MudrYSc32BbRTW8bu4thznas52LcXKW7HDUg@mail.gmail.com> <20130913220236.1f27b73f@fsol> <l0vrtv$cso$2@ger.gmane.org> <20130913234516.10661214@fsol> <20130914004952.GJ16820@ando> <5233B522.7070104@stoneleaf.us> Message-ID: <20130914024452.GL16820@ando> On Fri, Sep 13, 2013 at 06:00:18PM -0700, Ethan Furman wrote: > Personally, if there's a bunch of push-back against just adding > TransformDict directly, why don't we make it provisional? I thought that > was what provisional was for (meaning: we're going to add it, PyPI is not > really appropriate, there may be some API changes). Not according to PEP 411. It implies that only modules/packages can be provisional, not individual functions, and states that "most packages" are expected to be provisional. So either PEP 411 doesn't apply to TransformDict at all, or it applies by default. The PEP doesn't say. http://www.python.org/dev/peps/pep-0411/ Everything below the line is about PEP 411, not TransformDict. If you don't care about PEP 411, you can stop reading now. ========================== Personally, I think it's a poor PEP. It doesn't document opposition to the idea, and if I recall the discussion at the time correctly, there was plenty of opposition. - Since people cannot rely on provisional features still being available in the future, if they care the slightest about forward compatibility, they dare not use them. - If people do use them, and we care about backwards compatibility, we dare not remove provisional packages without going through the same deprecation process as for ordinary packages. - It relies on people reading the documentation and noticing that a package is marked "provisional". Like that's going to happen. None of these arguments are documented in the PEP, let alone refuted. Even if the decision to approve the PEP ends up being vindicated, I think it was poor form for the PEP to ignore arguments against. I don't think that "provisional" helps end users at all. If anything, it hurts them -- it means more uncertainty and doubt. Packages may languish in provisional status indefinitely. Speaking as an end user, I used to know that once a feature hit the std lib, it was stable and wouldn't be removed without going through a lengthy period of deprecation (except under truly remarkable circumstances). But for provisional packages, that promise doesn't apply, and although PEP 411 says that packages won't be gratuitously removed, I have to assume that any provisional package in version 3.x could be gone without notice or warning in 3.x+1. I don't see how that helps me. It just means that for some indefinite period, there's a package in the std lib doing exactly what I want that I dare not use in production software because there are no stability guarantees. The PEP has a lengthy section that claims to be about how it will help end users. It actually isn't -- it is about how inclusion in the standard library helps end users. Not surprisingly, there's nothing about why reserving the right to rip out a package without warning is good for end uers, since it isn't. End of rant. -- Steven From steve at pearwood.info Sat Sep 14 05:18:12 2013 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 14 Sep 2013 13:18:12 +1000 Subject: [Python-Dev] PEP 455: TransformDict In-Reply-To: <5233BB16.8050201@mrabarnett.plus.com> References: <20130913204058.518f43d3@fsol> <l0vp4p$3lj$1@ger.gmane.org> <20130913213725.19dd8817@fsol> <CAH0mxTRZ7v5H-7MudrYSc32BbRTW8bu4thznas52LcXKW7HDUg@mail.gmail.com> <20130913220236.1f27b73f@fsol> <l0vrtv$cso$2@ger.gmane.org> <20130913234516.10661214@fsol> <20130914004952.GJ16820@ando> <5233BB16.8050201@mrabarnett.plus.com> Message-ID: <20130914031812.GM16820@ando> On Sat, Sep 14, 2013 at 02:25:42AM +0100, MRAB wrote: > On 14/09/2013 01:49, Steven D'Aprano wrote: > >d = TransformDict(...) > >for key in data: > > key = d.get_canonical(key) > > value = d[key] > > print("{}: {}".format(key, value)) > > > I think must be missing something. I thought that iterating over the > dict would yield the original keys, so if you wanted the original key > and value you would write: > > for key, value in data.items(): > print("{}: {}".format(key, value)) You're missing that I'm not iterating over the entire dict, just some subset ("data") that I got from elsewhere. In real code, of course I would have to handle missing keys. My gut feeling is that normally I would want both the canonical key and the value, not just any valid key: # not this for key in data: value = d[key] print("{}: {}".format(key, value)) # but this for key in data: key, value = d.getitem(key) print("{}: {}".format(key, value)) but that's just a gut feeling, and I daresay that it would depend on the application. > and if you wanted the transformed key you would apply the transform > function to the key. The caller may not know or care what the transformation function is. The caller may only care about one or two things: - does this key match a key in the dict? - what is the canonical version of this key? For example, in a case-preserving file system, the canonical version of a file name is whatever the user first typed when they created the file: create file "SoMe FiLe" so the canonical version is, "SoMe FiLe". But any of these may be expected to match: some file SOME FILE Some File some%20file etc. It is *not* simply the case that the canonical version is "some file". Certainly that's not how case-preserving file systems work. -- Steven From steve at pearwood.info Sat Sep 14 05:25:47 2013 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 14 Sep 2013 13:25:47 +1000 Subject: [Python-Dev] PEP 455: TransformDict In-Reply-To: <5233B522.7070104@stoneleaf.us> References: <20130913204058.518f43d3@fsol> <l0vp4p$3lj$1@ger.gmane.org> <20130913213725.19dd8817@fsol> <CAH0mxTRZ7v5H-7MudrYSc32BbRTW8bu4thznas52LcXKW7HDUg@mail.gmail.com> <20130913220236.1f27b73f@fsol> <l0vrtv$cso$2@ger.gmane.org> <20130913234516.10661214@fsol> <20130914004952.GJ16820@ando> <5233B522.7070104@stoneleaf.us> Message-ID: <20130914032546.GN16820@ando> On Fri, Sep 13, 2013 at 06:00:18PM -0700, Ethan Furman wrote: > On 09/13/2013 05:49 PM, Steven D'Aprano wrote: > > > >+1 on __transform__ method on dicts. > > What would __transform__ do? Just canonicalize the key, or also save the > original key? How much front-end work will each user have to do to > actually use this new magic method to good effect? Sorry, I missed replying to this part. See Serhiy's post: https://mail.python.org/pipermail/python-dev/2013-September/128633.html and suggested patch here: http://bugs.python.org/issue18986 -- Steven From steve at pearwood.info Sat Sep 14 05:33:51 2013 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 14 Sep 2013 13:33:51 +1000 Subject: [Python-Dev] PEP 455: TransformDict In-Reply-To: <5233BE76.2040206@stoneleaf.us> References: <20130913204058.518f43d3@fsol> <l0vp4p$3lj$1@ger.gmane.org> <20130913213725.19dd8817@fsol> <CAH0mxTRZ7v5H-7MudrYSc32BbRTW8bu4thznas52LcXKW7HDUg@mail.gmail.com> <20130913220236.1f27b73f@fsol> <l0vrtv$cso$2@ger.gmane.org> <20130913234516.10661214@fsol> <20130914004952.GJ16820@ando> <5233BB16.8050201@mrabarnett.plus.com> <5233BE76.2040206@stoneleaf.us> Message-ID: <20130914033351.GO16820@ando> On Fri, Sep 13, 2013 at 06:40:06PM -0700, Ethan Furman wrote: > >and if you wanted the transformed key you would apply the transform > >function to the key. > > Indeed. The question is: how? It is entirely possible that your function > has a TransformDict alone, and no memory of the transform function used to > create the dict... > > If the key transform function were saved directly on the TransformDict > instance as, say, .transform_key, then problem solved. While I think it is good and proper to have the transformation function available, if you're manually applying the transformation function then IMO chances are good that you've missed the whole point of the transformdict. Think of defaultdict. The purpose of defaultdict is to avoid needing to check whether the key is missing or not. If I suggested that the way to use a defaultdict was to write code like this: value = d[key] if key in d else d.default_factory() I'd be missing the point. Likewise, the point of transformdict is to avoid needing to care about the transformation function 99% of the time. -- Steven From ethan at stoneleaf.us Sat Sep 14 06:45:48 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 13 Sep 2013 21:45:48 -0700 Subject: [Python-Dev] PEP 455: TransformDict In-Reply-To: <20130914033351.GO16820@ando> References: <20130913204058.518f43d3@fsol> <l0vp4p$3lj$1@ger.gmane.org> <20130913213725.19dd8817@fsol> <CAH0mxTRZ7v5H-7MudrYSc32BbRTW8bu4thznas52LcXKW7HDUg@mail.gmail.com> <20130913220236.1f27b73f@fsol> <l0vrtv$cso$2@ger.gmane.org> <20130913234516.10661214@fsol> <20130914004952.GJ16820@ando> <5233BB16.8050201@mrabarnett.plus.com> <5233BE76.2040206@stoneleaf.us> <20130914033351.GO16820@ando> Message-ID: <5233E9FC.10509@stoneleaf.us> On 09/13/2013 08:33 PM, Steven D'Aprano wrote: > > Likewise, the point of transformdict is to avoid needing to care > about the transformation function 99% of the time. No one's arguing against that point. It's the 1% of the time that being able to get the canonical name back is a Good Thing, and the best way to do that is to have the transform_key function easily available. -- ~Ethan~ From jsbueno at python.org.br Sat Sep 14 06:53:20 2013 From: jsbueno at python.org.br (Joao S. O. Bueno) Date: Sat, 14 Sep 2013 01:53:20 -0300 Subject: [Python-Dev] PEP 455: TransformDict In-Reply-To: <5233BE76.2040206@stoneleaf.us> References: <20130913204058.518f43d3@fsol> <l0vp4p$3lj$1@ger.gmane.org> <20130913213725.19dd8817@fsol> <CAH0mxTRZ7v5H-7MudrYSc32BbRTW8bu4thznas52LcXKW7HDUg@mail.gmail.com> <20130913220236.1f27b73f@fsol> <l0vrtv$cso$2@ger.gmane.org> <20130913234516.10661214@fsol> <20130914004952.GJ16820@ando> <5233BB16.8050201@mrabarnett.plus.com> <5233BE76.2040206@stoneleaf.us> Message-ID: <CAH0mxTQOkNm8zMEXD-jtLxnoru-ik7pRJ92sRe7dD463-Eh3gg@mail.gmail.com> On 13 September 2013 22:40, Ethan Furman <ethan at stoneleaf.us> wrote: > On 09/13/2013 06:25 PM, MRAB wrote: >> >> On 14/09/2013 01:49, Steven D'Aprano wrote: >>> >>> >>> Is it more common to want both the canonical key and value at the same >>> time, or to just want the canonical key? My gut feeling is that I'm >>> likely to have code like this: >>> >>> >>> d = TransformDict(...) >>> for key in data: >>> key = d.get_canonical(key) >>> value = d[key] >>> print("{}: {}".format(key, value)) >>> >> I think I must be missing something. I thought that iterating over the >> >> dict would yield the original keys, so if you wanted the original key >> and value you would write: >> >> for key, value in data.items(): >> print("{}: {}".format(key, value)) > > > Well, that's certainly how I would do it. ;) I hope you are aware that this pattern does not help when one wants _one_ canonical key having a non-canonical one, besides having to linearly walk through all keys and check the "__transform__" to each one. I mean - given no function to retrieve the canonical key, one would have to resort to: my_key = data.__transform__(given_key) for key, value in data.items(): if data.__transform__(key) == my_key: .... WHich would defeat not only the purpose of a Transform dict, but the purpose of a dict alltogether. (of course, the one obvious way to do it would be to have the original key stored along the value - which is just a bit less silly than the example above) OTOH, having a `get_canonical_key` method or similar seens trivial enough- if the only provided method retrieves the value as well, it would not be that bad. > > > >> and if you wanted the transformed key you would apply the transform >> function to the key. > > > Indeed. The question is: how? It is entirely possible that your function > has a TransformDict alone, and no memory of the transform function used to > create the dict... > > If the key transform function were saved directly on the TransformDict > instance as, say, .transform_key, then problem solved. > > -- > ~Ethan~ > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/jsbueno%40python.org.br From g.brandl at gmx.net Sat Sep 14 07:21:00 2013 From: g.brandl at gmx.net (Georg Brandl) Date: Sat, 14 Sep 2013 07:21:00 +0200 Subject: [Python-Dev] PEP 455: TransformDict In-Reply-To: <20130913234516.10661214@fsol> References: <20130913204058.518f43d3@fsol> <l0vp4p$3lj$1@ger.gmane.org> <20130913213725.19dd8817@fsol> <CAH0mxTRZ7v5H-7MudrYSc32BbRTW8bu4thznas52LcXKW7HDUg@mail.gmail.com> <20130913220236.1f27b73f@fsol> <l0vrtv$cso$2@ger.gmane.org> <20130913234516.10661214@fsol> Message-ID: <l10rme$75n$1@ger.gmane.org> On 09/13/2013 11:45 PM, Antoine Pitrou wrote: > On Fri, 13 Sep 2013 23:18:39 +0300 > Serhiy Storchaka <storchaka at gmail.com> wrote: >> 13.09.13 23:02, Antoine Pitrou ???????(??): >> > On Fri, 13 Sep 2013 16:54:01 -0300 >> > "Joao S. O. Bueno" <jsbueno at python.org.br> wrote: >> >> I see the PEP does not contemplate a way to retrieve the original key >> >> - like we've talked about somewhere along the thread. >> > >> > Indeed. If that's important I can add it. I was hoping to keep very >> > close to the MutableMapping API, to make the PEP as few >> > controversial as possible. >> >> I think that's important. As OrderectDict has additional methods besides >> the MutableMapping API, so TransformDict should provide useful >> specialized methods. > > Ok, I have a better (IMO) proposal: > > >>> d = TransformDict(str.casefold, {'Foo': 1}) > >>> d.getitem('foo') > ('Foo', 1) > >>> d.getitem('bar') > Traceback (most recent call last): > File "<stdin>", line 1, in <module> > KeyError: 'bar' Except that getitem is very close to __getitem__, which will be confusing. (Although it would be the correct name, and __getitem__ is the wrong one). cheers, Georg From g.brandl at gmx.net Sat Sep 14 07:24:27 2013 From: g.brandl at gmx.net (Georg Brandl) Date: Sat, 14 Sep 2013 07:24:27 +0200 Subject: [Python-Dev] PEP 455: TransformDict In-Reply-To: <20130913200910.697972502EA@webabinitio.net> References: <20130913204058.518f43d3@fsol> <20130913200910.697972502EA@webabinitio.net> Message-ID: <l10rst$899$1@ger.gmane.org> On 09/13/2013 10:09 PM, R. David Murray wrote: > On Fri, 13 Sep 2013 20:40:58 +0200, Antoine Pitrou <solipsis at pitrou.net> wrote: >> Rationale >> ========= >> >> Numerous specialized versions of this pattern exist. The most common >> is a case-insensitive case-preserving dict, i.e. a dict-like container >> which matches keys in a case-insensitive fashion but retains the >> original casing. It is a very common need in network programming, as >> many protocols feature some arrays of "key / value" properties in their >> messages, where the keys are textual strings whose casing isn't >> relevant. > > This motivation would be stronger if the last phrase was something like > "where the keys are textual strings whose case is specified to be ignored > on receipt but by either specification or custom is to be preserved > or non-trivially canonicalized when retransmitted." > >> (it can be said that the pattern *projects* keys from the user-visible >> set onto the internal lookup set, hence this PEP's title) > > Not clear what "projects" has to do with the PEP title. Clearly it should be called ProjectionDict! cheers, Georg From g.brandl at gmx.net Sat Sep 14 07:31:01 2013 From: g.brandl at gmx.net (Georg Brandl) Date: Sat, 14 Sep 2013 07:31:01 +0200 Subject: [Python-Dev] PEP 455: TransformDict In-Reply-To: <20130913204058.518f43d3@fsol> References: <20130913204058.518f43d3@fsol> Message-ID: <l10s98$9mk$1@ger.gmane.org> On 09/13/2013 08:40 PM, Antoine Pitrou wrote: > > Hello, > > Following the python-dev discussion, I've written a PEP to recap the > proposal and the various arguments. It's inlined below, and it will > probably appear soon at http://www.python.org/dev/peps/pep-0455/, too. > Looks good to me. Georg From larry at hastings.org Sat Sep 14 07:33:56 2013 From: larry at hastings.org (Larry Hastings) Date: Sat, 14 Sep 2013 14:33:56 +0900 Subject: [Python-Dev] PEP 455: TransformDict In-Reply-To: <20130913204058.518f43d3@fsol> References: <20130913204058.518f43d3@fsol> Message-ID: <5233F544.7030604@hastings.org> On 09/14/2013 03:40 AM, Antoine Pitrou wrote: > Hello, > > Following the python-dev discussion, I've written a PEP to recap the > proposal and the various arguments. It's inlined below, and it will > probably appear soon at http://www.python.org/dev/peps/pep-0455/, too. Whenever I read a discussion about the dict, I always wonder whether the same thing applies to a set. Have you considered the utility of a TransformSet? Or is it YAGNI? Cheers, //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130914/c20f74ce/attachment.html> From ethan at stoneleaf.us Sat Sep 14 06:47:25 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 13 Sep 2013 21:47:25 -0700 Subject: [Python-Dev] PEP 455: TransformDict In-Reply-To: <20130914031812.GM16820@ando> References: <20130913204058.518f43d3@fsol> <l0vp4p$3lj$1@ger.gmane.org> <20130913213725.19dd8817@fsol> <CAH0mxTRZ7v5H-7MudrYSc32BbRTW8bu4thznas52LcXKW7HDUg@mail.gmail.com> <20130913220236.1f27b73f@fsol> <l0vrtv$cso$2@ger.gmane.org> <20130913234516.10661214@fsol> <20130914004952.GJ16820@ando> <5233BB16.8050201@mrabarnett.plus.com> <20130914031812.GM16820@ando> Message-ID: <5233EA5D.6060500@stoneleaf.us> On 09/13/2013 08:18 PM, Steven D'Aprano wrote: > > You're missing that I'm not iterating over the entire dict, just some > subset ("data") that I got from elsewhere. Ah, okay. Between you and Antoine I am convinced that .getitem() is a good thing. So have that and .transform_key and we're golden! :) -- ~Ethan~ From ncoghlan at gmail.com Sat Sep 14 08:30:54 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 14 Sep 2013 16:30:54 +1000 Subject: [Python-Dev] PEP 447: add type.__locallookup__ In-Reply-To: <20130913122328.GI16820@ando> References: <7D6D4488-2855-41F8-A9F2-1FF23B9E3A56@mac.com> <522DDB74.5080609@stoneleaf.us> <CAP7+vJJTR=+cRY+iQgHY4ujTsC3_QyWn83dnK96rWO6bV9vFLQ@mail.gmail.com> <522DECA3.2030709@hotpy.org> <CAP7+vJ+thJvYpk0d0rnFO6bm88sTJ_t1QdYFB0SdPOGNDABuag@mail.gmail.com> <522DF204.7040109@hotpy.org> <0752376899dd7436f6115bfc03594574@chopin.edu.pl> <392015291ec843afb1bc85632055729d@BLUPR03MB199.namprd03.prod.outlook.com> <20130913065236.GF16820@ando> <CADiSq7fNhAgGUE9-1EdASM5MH0wx7W-mJTi6pRv=45r3K6LvmA@mail.gmail.com> <20130913122328.GI16820@ando> Message-ID: <CADiSq7cdSEHryP+LMppRdabTuUG-GzMd56ub-oWpFNKPHNzsHA@mail.gmail.com> On 13 September 2013 22:23, Steven D'Aprano <steve at pearwood.info> wrote: > On Fri, Sep 13, 2013 at 08:42:46PM +1000, Nick Coghlan wrote: >> Perhaps "__getdescriptor__" would work as the method name? Yes, it can >> technically return a non-descriptor, > > So technically that name is, um, what's the term... oh yes, "a lie". > > :-) In this case, "__getdescriptor__" means "I am looking for a descriptor, please don't invoke the descriptor methods or traverse the MRO, just return the raw object", not "you *must* give me a descriptor". The name suggestion comes from the fact that name bindings on the instance and names bindings on the class are *different*, in that only the latter participate in the descriptor protocol: >>> class A: ... def m(self): pass ... >>> A.m <function A.m at 0x7fb51ad63320> >>> a = A() >>> a.f = A.m >>> a.m <bound method A.m of <__main__.A object at 0x7fb51ad60550>> >>> a.f <function A.m at 0x7fb51ad63320> It's the same function object underneath, so what's going on? The trick is that only *types* get to play the descriptor game, where the interpreter looks for __get__, __set__ and __delete__ on the returned object and invokes them with found. Ordinary instances don't have that behaviour - instead, they go through type(self) to look for descriptors, and anything they find in the instance dictionary is returned unaltered. This difference in how attribute lookups are handled is actually the most fundamental difference between normal class instances and classes themselves (which are instances of metaclasses). >> but the *primary* purpose is to >> customise the retrieval of objects that will be checked to see if they're >> descriptors. > > If that's the case, the PEP should make that clear. Technically, that's what "Currently object.__getattribute__ and super.__getattribute__ peek in the __dict__ of classes on the MRO for a class when looking for an attribute." means. However, I agree the current wording only conveys that to the handful of people that already know exactly when in the attribute lookup sequence that step occurs, which is a rather niche audience :) It's also why I like __getdescriptor__ as a name - it's based on *why* we're doing the lookup, rather than *how* we expect it to be done. > [Aside: the PEP states that the method shouldn't invoke descriptors. > What's the reason for that? If I take the statement literally, doesn't > it mean that the method mustn't use any other methods at all? Surely > that can't be what is intended, but I'm not sure what is intended.] It means it shouldn't invoke __get__, __set__ or __delete__ on the returned object, since that's the responsibility of the caller. For example, there are times when a descriptor will be retrieved to check for the presence of a __set__ or __delete__ method, but never actually have its methods invoked because it is shadowed in the instance dictionary: >>> class Shadowable: ... def __get__(self, *args): ... print("Shadowable.__get__ called!") ... >>> class Enforced: ... def __get__(self, *args): ... print("Enforced.__get__ called!") ... def __set__(self, *args): ... print("Enforced.__set__ called!") ... >>> class Example: ... s = Shadowable() ... e = Enforced() ... def __getattribute__(self, attr): ... print("Retrieving {} from class".format(attr)) ... return super().__getattribute__(attr) ... >>> x = Example() >>> x.s Retrieving s from class Shadowable.__get__ called! >>> x.s = 1 >>> x.s Retrieving s from class 1 This is the key line: we retrieved 's' from the class, but *didn't* invoke the __get__ method because it was shadowed in the instance dictionary. It works this way because *if* the descriptor defines __set__ or __delete__, then Python will *ignore* the instance variable: >>> x.e Retrieving e from class Enforced.__get__ called! >>> x.__dict__["e"] = 1 Retrieving __dict__ from class >>> x.e Retrieving e from class Enforced.__get__ called! So my proposed name is based on the idea that what Ronald is after with the PEP is a hook that *only* gets invoked when the interpreter is doing this hunt for descriptors, but *not* for ordinary attribute lookups. >> It *won't* be invoked when looking for ordinary attributes in >> an instance dict, but *will* be invoked when looking on the class object. > > Just to be clear, if I have: > > instance = MyClass() > x = instance.name > > and "name" is found in instance.__dict__, then this special method will > not be invoked. But if "name" is not found in the instance dict, then > "name" will be looked up on the class object MyClass, which may invoke > this special method. Am I correct? Well, that's *my* proposal. While re-reading the current PEP, I realised my suggested change actually goes quite a bit further than just proposing a different name: unlike the current PEP, my advice is that the new hook should NOT be invoked for instance attribute lookups and should *not* replace looking directly into the class dict. Instead, it would be the descriptor lookup counterpart to __getattr__: whereas __getattr__ only fires for lookups on instances of the class, __getdescriptor__ would only fire for lookups on the class itself (so, almost exactly equivalent to defining __getattr__ on the metaclass, but without needing a custom metaclass). A class that wanted to affect both would be able to define __getdescriptor__ for the fallback lookup on the class and then call it from __getattr__. This also suggests to me that __getdescriptor__ should be an implicit static method like __new__ (it would also be possible to make it an implicit classmethod, but an implicit staticmethod would be more consistent with the way __new__ works, so if we're going to have implicit magic, it may as well be *consistent* implicit magic): >>> class C: ... def __new__(cls): ... print(cls) ... return super().__new__(cls) ... >>> C() <class '__main__.C'> <__main__.C object at 0x7fb51ad60c50> >>> class D(C): pass ... >>> D() <class '__main__.D'> <__main__.D object at 0x7fb51ad60c90> >>> C.__new__ <function C.__new__ at 0x7fb51ad638c0> >>> C().__new__ <class '__main__.C'> <function C.__new__ at 0x7fb51ad638c0> Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ethan at stoneleaf.us Sat Sep 14 06:59:11 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 13 Sep 2013 21:59:11 -0700 Subject: [Python-Dev] PEP 455: TransformDict In-Reply-To: <CAH0mxTQOkNm8zMEXD-jtLxnoru-ik7pRJ92sRe7dD463-Eh3gg@mail.gmail.com> References: <20130913204058.518f43d3@fsol> <l0vp4p$3lj$1@ger.gmane.org> <20130913213725.19dd8817@fsol> <CAH0mxTRZ7v5H-7MudrYSc32BbRTW8bu4thznas52LcXKW7HDUg@mail.gmail.com> <20130913220236.1f27b73f@fsol> <l0vrtv$cso$2@ger.gmane.org> <20130913234516.10661214@fsol> <20130914004952.GJ16820@ando> <5233BB16.8050201@mrabarnett.plus.com> <5233BE76.2040206@stoneleaf.us> <CAH0mxTQOkNm8zMEXD-jtLxnoru-ik7pRJ92sRe7dD463-Eh3gg@mail.gmail.com> Message-ID: <5233ED1F.9020808@stoneleaf.us> On 09/13/2013 09:53 PM, Joao S. O. Bueno wrote: > On 13 September 2013 22:40, Ethan Furman <ethan at stoneleaf.us> wrote: >> On 09/13/2013 06:25 PM, MRAB wrote: >>> >>> On 14/09/2013 01:49, Steven D'Aprano wrote: >>>> >>>> >>>> Is it more common to want both the canonical key and value at the same >>>> time, or to just want the canonical key? My gut feeling is that I'm >>>> likely to have code like this: >>>> >>>> >>>> d = TransformDict(...) >>>> for key in data: >>>> key = d.get_canonical(key) >>>> value = d[key] >>>> print("{}: {}".format(key, value)) >>>> >>> I think I must be missing something. I thought that iterating over the >>> >>> dict would yield the original keys, so if you wanted the original key >>> and value you would write: >>> >>> for key, value in data.items(): >>> print("{}: {}".format(key, value)) >> >> >> Well, that's certainly how I would do it. ;) > > I hope you are aware that this pattern does not help when one wants > _one_ canonical key having a non-canonical one [...] True, but I was thinking Steve was talking about printing the entire dict, in which case that is, indeed, how I would do it. > I mean - given no function to retrieve the canonical key, > one would have to resort to: > > my_key = data.__transform__(given_key) > for key, value in data.items(): > if data.__transform__(key) == my_key: > .... Which is exactly why I, and others, would like to have the transform function easily available. Besides being able to use it to get a canonical key, one could use it to get the function itself. Yay, introspection! -- ~Ethan~ From stephen at xemacs.org Sat Sep 14 08:59:33 2013 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sat, 14 Sep 2013 15:59:33 +0900 Subject: [Python-Dev] "Provisional" considered harmful? [was: PEP 455: TransformDict] In-Reply-To: <20130914024452.GL16820@ando> References: <20130913204058.518f43d3@fsol> <l0vp4p$3lj$1@ger.gmane.org> <20130913213725.19dd8817@fsol> <CAH0mxTRZ7v5H-7MudrYSc32BbRTW8bu4thznas52LcXKW7HDUg@mail.gmail.com> <20130913220236.1f27b73f@fsol> <l0vrtv$cso$2@ger.gmane.org> <20130913234516.10661214@fsol> <20130914004952.GJ16820@ando> <5233B522.7070104@stoneleaf.us> <20130914024452.GL16820@ando> Message-ID: <87fvt77usq.fsf@uwakimon.sk.tsukuba.ac.jp> As will become evident, I disagree with Steven at almost every point. However, I think his point about users not reading documentation is well-taken. Due to hyperlinking, users are very likely to skip past module docstrings. I suggest two (perhaps informal) additions to the documentation policy in PEP 411: (1) Provisional packages should be explicitly described as such in release announcements. I think this is done already, cf. "Improvements in email" in the Release announcement and What's New for Python 3.3.0.[1] However the policy should be explicit and the word "provisional" should be emphasized and linked to PEP 411. (2) Individual APIs in those packages should be documented as provisional. Steven D'Aprano writes: > - Since people cannot rely on provisional features still being available > in the future, if they care the slightest about forward compatibility, > they dare not use them. You exaggerate to the extent that you harm your argument. People make tradeoffs. Some will choose the extreme you describe here, others will take more risk. Your real argument is that the benefits are small because you expect that provisional stdlib packages will not get much if any additional use over PyPI packages. Can't know without trying ... thus, the PEP. Similar considerations apply to your other arguments. > None of these arguments are documented in the PEP, let alone refuted. They're not real arguments, as stated. If you rephrase them as actual arguments, they turn out to merely be the "half-empty" phrasing of the "half-full" arguments used in favor. Can't know without trying. > I don't think that "provisional" helps end users at all. If anything, it > hurts them -- it means more uncertainty and doubt. True, it *increases* the uncertainty *inside* the stdlib. On the flip side, it decreases uncertainty *overall* by announcing that the core developers *want* this feature in stdlib *now*, and probably the API won't change *at all*, and very probably the API won't change "much". > But for provisional packages, that promise doesn't apply, and > although PEP 411 says that packages won't be gratuitously removed, > I have to assume that any provisional package in version 3.x could > be gone without notice or warning in 3.x+1. It's unlikely it will be "gone", it will end up on PyPI. > I don't see how that helps me. Since you evidently demand absolute certainty, you're right, it doesn't help you with respect to any given provisional package. I'm more flexible, and I greatly value having the core developers evaluate the modules on PyPI (or the modules that have only seen the flourescent lights of Google Labs up to now) for me, and putting their Good Codesmithing Stamp of Approval on some of them more quickly. And if in fact the policy works in the sense that more code gets into the stdlib (non-provisionally) faster than without the PEP, in the long run *you* benefit even with if you adopt a policy of using only non-provisional APIs. > The PEP has a lengthy section that claims to be about how it will > help end users. It actually isn't -- it is about how inclusion in > the standard library helps end users. Right. And this PEP is about how to get good code into the stdlib faster. It's a means to that intermediate end, and transitively a means to the real end of helping end users. > Not surprisingly, there's nothing about why reserving the right to > rip out a package without warning is good for end uers, since it > isn't. What it really comes down to is you're saying you fear you will frequently disagree with the judgment of python-dev with regard to the options provided by "provisional". You want them bound by hard and fast rules. That's fine; everybody has their own requirements. Nevertheless, it's logically possible that as long as you pay attention to "provisional" markings this PEP will be a win for you. I agree with the PEP proponents that "logically possible" can be strengthened to "probably true", but of course YMMV. Footnotes: [1] http://www.python.org/download/releases/3.3.0/ http://docs.python.org/3.3/whatsnew/3.3.html From solipsis at pitrou.net Sat Sep 14 12:27:06 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 14 Sep 2013 12:27:06 +0200 Subject: [Python-Dev] PEP 455: TransformDict References: <20130913204058.518f43d3@fsol> <l0vp4p$3lj$1@ger.gmane.org> <20130913213725.19dd8817@fsol> <CAH0mxTRZ7v5H-7MudrYSc32BbRTW8bu4thznas52LcXKW7HDUg@mail.gmail.com> <20130913220236.1f27b73f@fsol> <l0vrtv$cso$2@ger.gmane.org> <20130913234516.10661214@fsol> <20130914004952.GJ16820@ando> <5233BB16.8050201@mrabarnett.plus.com> <5233BE76.2040206@stoneleaf.us> <CAH0mxTQOkNm8zMEXD-jtLxnoru-ik7pRJ92sRe7dD463-Eh3gg@mail.gmail.com> <5233ED1F.9020808@stoneleaf.us> Message-ID: <20130914122706.6c1efbc7@fsol> On Fri, 13 Sep 2013 21:59:11 -0700 Ethan Furman <ethan at stoneleaf.us> wrote: > > > I mean - given no function to retrieve the canonical key, > > one would have to resort to: > > > > my_key = data.__transform__(given_key) > > for key, value in data.items(): > > if data.__transform__(key) == my_key: > > .... > > Which is exactly why I, and others, would like to have the transform function easily available. Besides being able to > use it to get a canonical key, one could use it to get the function itself. Yay, introspection! Well, no, you misunderstand :) The transform function takes an original key (perhaps "canonical") and returns the transformed key, it can't do the reverse which is what getitem() does. i.e.: >>> d = TransformDict(str.lower) >>> d['Foo'] = 5 >>> d._transform_func('Foo') 'foo' >>> d.getitem('foo') [('Foo', 5)] What getitem() does is make the surjection bijective by restricting its input domain to the set of stored keys. Regards Antoine. From solipsis at pitrou.net Sat Sep 14 12:30:46 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 14 Sep 2013 12:30:46 +0200 Subject: [Python-Dev] PEP 455: TransformDict References: <20130913204058.518f43d3@fsol> <5233F544.7030604@hastings.org> Message-ID: <20130914123046.4f053e6a@fsol> On Sat, 14 Sep 2013 14:33:56 +0900 Larry Hastings <larry at hastings.org> wrote: > On 09/14/2013 03:40 AM, Antoine Pitrou wrote: > > Hello, > > > > Following the python-dev discussion, I've written a PEP to recap the > > proposal and the various arguments. It's inlined below, and it will > > probably appear soon at http://www.python.org/dev/peps/pep-0455/, too. > > Whenever I read a discussion about the dict, I always wonder whether the > same thing applies to a set. Have you considered the utility of a > TransformSet? Or is it YAGNI? Well, a TransformSet is like a normal dict, you just need to call the transformation function yourself when inserting the keys. i.e.: d = TransformSet(str.lower) d.add('Foo') is the same conceptually as: d = {} d['Foo'.lower()] = 'Foo' d['foo'] # gets the original key Regards Antoine. From ncoghlan at gmail.com Sat Sep 14 18:25:46 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 15 Sep 2013 02:25:46 +1000 Subject: [Python-Dev] PEP 455: TransformDict In-Reply-To: <20130914024452.GL16820@ando> References: <20130913204058.518f43d3@fsol> <l0vp4p$3lj$1@ger.gmane.org> <20130913213725.19dd8817@fsol> <CAH0mxTRZ7v5H-7MudrYSc32BbRTW8bu4thznas52LcXKW7HDUg@mail.gmail.com> <20130913220236.1f27b73f@fsol> <l0vrtv$cso$2@ger.gmane.org> <20130913234516.10661214@fsol> <20130914004952.GJ16820@ando> <5233B522.7070104@stoneleaf.us> <20130914024452.GL16820@ando> Message-ID: <CADiSq7euQnhBdQAMJkv-c6-t+scevGNUA0Qg9ammZvthAnU6fA@mail.gmail.com> On 14 September 2013 12:44, Steven D'Aprano <steve at pearwood.info> wrote: > On Fri, Sep 13, 2013 at 06:00:18PM -0700, Ethan Furman wrote: > >> Personally, if there's a bunch of push-back against just adding >> TransformDict directly, why don't we make it provisional? I thought that >> was what provisional was for (meaning: we're going to add it, PyPI is not >> really appropriate, there may be some API changes). > > Not according to PEP 411. It implies that only modules/packages can be > provisional, not individual functions, and states that "most packages" > are expected to be provisional. So either PEP 411 doesn't apply to > TransformDict at all, or it applies by default. The PEP doesn't say. > > http://www.python.org/dev/peps/pep-0411/ > > > Everything below the line is about PEP 411, not TransformDict. If you > don't care about PEP 411, you can stop reading now. > > > ========================== > > > Personally, I think it's a poor PEP. It doesn't document opposition to > the idea, and if I recall the discussion at the time correctly, there > was plenty of opposition. > > - Since people cannot rely on provisional features still being available > in the future, if they care the slightest about forward compatibility, > they dare not use them. Correct, that's exactly what they should do. > - If people do use them, and we care about backwards compatibility, we > dare not remove provisional packages without going through the same > deprecation process as for ordinary packages. > - It relies on people reading the documentation and noticing that a > package is marked "provisional". Like that's going to happen. > > None of these arguments are documented in the PEP, let alone refuted. > Even if the decision to approve the PEP ends up being vindicated, I > think it was poor form for the PEP to ignore arguments against. > > I don't think that "provisional" helps end users at all. If anything, it > hurts them -- it means more uncertainty and doubt. Packages may languish > in provisional status indefinitely. Speaking as an end user, I used to > know that once a feature hit the std lib, it was stable and wouldn't be > removed without going through a lengthy period of deprecation (except > under truly remarkable circumstances). But for provisional packages, > that promise doesn't apply, and although PEP 411 says that packages > won't be gratuitously removed, I have to assume that any provisional > package in version 3.x could be gone without notice or warning in 3.x+1. > I don't see how that helps me. It just means that for some indefinite > period, there's a package in the std lib doing exactly what I want that > I dare not use in production software because there are no stability > guarantees. Correct, it doesn't help you until it graduates from provisional status. Less risk averse users may choose to use it sooner (since any removed modules would almost certainly find a home on PyPI in fairly short order) > The PEP has a lengthy section that claims to be about how it will help > end users. It actually isn't -- it is about how inclusion in the > standard library helps end users. Not surprisingly, there's nothing > about why reserving the right to rip out a package without warning is > good for end uers, since it isn't. "We expect that most of the API of most provisional packages will be unchanged at graduation. Withdrawals are expected to be rare." Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Sat Sep 14 18:27:32 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 15 Sep 2013 02:27:32 +1000 Subject: [Python-Dev] PEP 455: TransformDict In-Reply-To: <20130914024452.GL16820@ando> References: <20130913204058.518f43d3@fsol> <l0vp4p$3lj$1@ger.gmane.org> <20130913213725.19dd8817@fsol> <CAH0mxTRZ7v5H-7MudrYSc32BbRTW8bu4thznas52LcXKW7HDUg@mail.gmail.com> <20130913220236.1f27b73f@fsol> <l0vrtv$cso$2@ger.gmane.org> <20130913234516.10661214@fsol> <20130914004952.GJ16820@ando> <5233B522.7070104@stoneleaf.us> <20130914024452.GL16820@ando> Message-ID: <CADiSq7f22svN0UGpBATM-RyKQ3EV6nnE=QOtpdVd4pBdS+_QCw@mail.gmail.com> On 14 September 2013 12:44, Steven D'Aprano <steve at pearwood.info> wrote: > On Fri, Sep 13, 2013 at 06:00:18PM -0700, Ethan Furman wrote: > >> Personally, if there's a bunch of push-back against just adding >> TransformDict directly, why don't we make it provisional? I thought that >> was what provisional was for (meaning: we're going to add it, PyPI is not >> really appropriate, there may be some API changes). > > Not according to PEP 411. It implies that only modules/packages can be > provisional, not individual functions, and states that "most packages" > are expected to be provisional. So either PEP 411 doesn't apply to > TransformDict at all, or it applies by default. The PEP doesn't say. > > http://www.python.org/dev/peps/pep-0411/ > > > Everything below the line is about PEP 411, not TransformDict. If you > don't care about PEP 411, you can stop reading now. > > > ========================== > > > Personally, I think it's a poor PEP. It doesn't document opposition to > the idea, and if I recall the discussion at the time correctly, there > was plenty of opposition. Oops, meant to reply to this part, too. PEP 411 is an Informational PEP, not a standards track PEP. It's there to describe the policy, not to make the case for *having* the policy. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From python at mrabarnett.plus.com Sat Sep 14 19:16:29 2013 From: python at mrabarnett.plus.com (MRAB) Date: Sat, 14 Sep 2013 18:16:29 +0100 Subject: [Python-Dev] PEP 455: TransformDict In-Reply-To: <5233EA5D.6060500@stoneleaf.us> References: <20130913204058.518f43d3@fsol> <l0vp4p$3lj$1@ger.gmane.org> <20130913213725.19dd8817@fsol> <CAH0mxTRZ7v5H-7MudrYSc32BbRTW8bu4thznas52LcXKW7HDUg@mail.gmail.com> <20130913220236.1f27b73f@fsol> <l0vrtv$cso$2@ger.gmane.org> <20130913234516.10661214@fsol> <20130914004952.GJ16820@ando> <5233BB16.8050201@mrabarnett.plus.com> <20130914031812.GM16820@ando> <5233EA5D.6060500@stoneleaf.us> Message-ID: <523499ED.4030301@mrabarnett.plus.com> On 14/09/2013 05:47, Ethan Furman wrote: > On 09/13/2013 08:18 PM, Steven D'Aprano wrote: >> >> You're missing that I'm not iterating over the entire dict, just some >> subset ("data") that I got from elsewhere. > > Ah, okay. Between you and Antoine I am convinced that .getitem() is a good thing. So have that and .transform_key and > we're golden! :) > +1 From ethan at stoneleaf.us Sat Sep 14 18:43:13 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Sat, 14 Sep 2013 09:43:13 -0700 Subject: [Python-Dev] PEP 455: TransformDict In-Reply-To: <20130914122706.6c1efbc7@fsol> References: <20130913204058.518f43d3@fsol> <l0vp4p$3lj$1@ger.gmane.org> <20130913213725.19dd8817@fsol> <CAH0mxTRZ7v5H-7MudrYSc32BbRTW8bu4thznas52LcXKW7HDUg@mail.gmail.com> <20130913220236.1f27b73f@fsol> <l0vrtv$cso$2@ger.gmane.org> <20130913234516.10661214@fsol> <20130914004952.GJ16820@ando> <5233BB16.8050201@mrabarnett.plus.com> <5233BE76.2040206@stoneleaf.us> <CAH0mxTQOkNm8zMEXD-jtLxnoru-ik7pRJ92sRe7dD463-Eh3gg@mail.gmail.com> <5233ED1F.9020808@stoneleaf.us> <20130914122706.6c1efbc7@fsol> Message-ID: <52349221.10808@stoneleaf.us> On 09/14/2013 03:27 AM, Antoine Pitrou wrote: > On Fri, 13 Sep 2013 21:59:11 -0700 > Ethan Furman <ethan at stoneleaf.us> wrote: >> >>> I mean - given no function to retrieve the canonical key, >>> one would have to resort to: >>> >>> my_key = data.__transform__(given_key) >>> for key, value in data.items(): >>> if data.__transform__(key) == my_key: >>> .... >> >> Which is exactly why I, and others, would like to have the transform function easily available. Besides being able to >> use it to get a canonical key, one could use it to get the function itself. Yay, introspection! > > Well, no, you misunderstand :) The transform function takes an > original key (perhaps "canonical") and returns the transformed key, it > can't do the reverse which is what getitem() does. i.e.: Argh, of course you are right. Still, I think it would be useful to expose the transform function. Any good reason not to? -- ~Ethan~ From solipsis at pitrou.net Sat Sep 14 19:41:31 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 14 Sep 2013 19:41:31 +0200 Subject: [Python-Dev] PEP 455: TransformDict References: <20130913204058.518f43d3@fsol> <l0vp4p$3lj$1@ger.gmane.org> <20130913213725.19dd8817@fsol> <CAH0mxTRZ7v5H-7MudrYSc32BbRTW8bu4thznas52LcXKW7HDUg@mail.gmail.com> <20130913220236.1f27b73f@fsol> <l0vrtv$cso$2@ger.gmane.org> <20130913234516.10661214@fsol> <20130914004952.GJ16820@ando> <5233BB16.8050201@mrabarnett.plus.com> <5233BE76.2040206@stoneleaf.us> <CAH0mxTQOkNm8zMEXD-jtLxnoru-ik7pRJ92sRe7dD463-Eh3gg@mail.gmail.com> <5233ED1F.9020808@stoneleaf.us> <20130914122706.6c1efbc7@fsol> <52349221.10808@stoneleaf.us> Message-ID: <20130914194131.232668c3@fsol> On Sat, 14 Sep 2013 09:43:13 -0700 Ethan Furman <ethan at stoneleaf.us> wrote: > On 09/14/2013 03:27 AM, Antoine Pitrou wrote: > > On Fri, 13 Sep 2013 21:59:11 -0700 > > Ethan Furman <ethan at stoneleaf.us> wrote: > >> > >>> I mean - given no function to retrieve the canonical key, > >>> one would have to resort to: > >>> > >>> my_key = data.__transform__(given_key) > >>> for key, value in data.items(): > >>> if data.__transform__(key) == my_key: > >>> .... > >> > >> Which is exactly why I, and others, would like to have the transform function easily available. Besides being able to > >> use it to get a canonical key, one could use it to get the function itself. Yay, introspection! > > > > Well, no, you misunderstand :) The transform function takes an > > original key (perhaps "canonical") and returns the transformed key, it > > can't do the reverse which is what getitem() does. i.e.: > > Argh, of course you are right. > > Still, I think it would be useful to expose the transform function. > Any good reason not to? No good reason. What's the name? transform_func? Regards Antoine. From ethan at stoneleaf.us Sat Sep 14 20:11:24 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Sat, 14 Sep 2013 11:11:24 -0700 Subject: [Python-Dev] PEP 455: TransformDict In-Reply-To: <20130914194131.232668c3@fsol> References: <20130913204058.518f43d3@fsol> <l0vp4p$3lj$1@ger.gmane.org> <20130913213725.19dd8817@fsol> <CAH0mxTRZ7v5H-7MudrYSc32BbRTW8bu4thznas52LcXKW7HDUg@mail.gmail.com> <20130913220236.1f27b73f@fsol> <l0vrtv$cso$2@ger.gmane.org> <20130913234516.10661214@fsol> <20130914004952.GJ16820@ando> <5233BB16.8050201@mrabarnett.plus.com> <5233BE76.2040206@stoneleaf.us> <CAH0mxTQOkNm8zMEXD-jtLxnoru-ik7pRJ92sRe7dD463-Eh3gg@mail.gmail.com> <5233ED1F.9020808@stoneleaf.us> <20130914122706.6c1efbc7@fsol> <52349221.10808@stoneleaf.us> <20130914194131.232668c3@fsol> Message-ID: <5234A6CC.1010900@stoneleaf.us> On 09/14/2013 10:41 AM, Antoine Pitrou wrote: > On Sat, 14 Sep 2013 09:43:13 -0700 Ethan Furman wrote: >> >> Still, I think it would be useful to expose the transform function. >> Any good reason not to? > > No good reason. What's the name? transform_func? I had originally thought transform_key, but transform_func is also good, and possibly less confusing. -- ~Ethan~ From storchaka at gmail.com Sat Sep 14 21:07:50 2013 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sat, 14 Sep 2013 22:07:50 +0300 Subject: [Python-Dev] PEP 455: TransformDict In-Reply-To: <20130914194131.232668c3@fsol> References: <20130913204058.518f43d3@fsol> <l0vp4p$3lj$1@ger.gmane.org> <20130913213725.19dd8817@fsol> <CAH0mxTRZ7v5H-7MudrYSc32BbRTW8bu4thznas52LcXKW7HDUg@mail.gmail.com> <20130913220236.1f27b73f@fsol> <l0vrtv$cso$2@ger.gmane.org> <20130913234516.10661214@fsol> <20130914004952.GJ16820@ando> <5233BB16.8050201@mrabarnett.plus.com> <5233BE76.2040206@stoneleaf.us> <CAH0mxTQOkNm8zMEXD-jtLxnoru-ik7pRJ92sRe7dD463-Eh3gg@mail.gmail.com> <5233ED1F.9020808@stoneleaf.us> <20130914122706.6c1efbc7@fsol> <52349221.10808@stoneleaf.us> <20130914194131.232668c3@fsol> Message-ID: <l12c5a$m72$1@ger.gmane.org> 14.09.13 20:41, Antoine Pitrou ???????(??): > On Sat, 14 Sep 2013 09:43:13 -0700 > Ethan Furman <ethan at stoneleaf.us> wrote: >> Still, I think it would be useful to expose the transform function. >> Any good reason not to? > > No good reason. What's the name? transform_func? There is one reason -- serialization. For example pickle saves the transform function as an argument for TransformDict constructor. Repr exposes the transform function too (in evaluable representation). Other serializers need the transform function too. My implementations expose it as public property (transform), your -- as private attribute (_transform). Or perhaps I misunderstood you? From solipsis at pitrou.net Sat Sep 14 21:12:25 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 14 Sep 2013 21:12:25 +0200 Subject: [Python-Dev] PEP 455: TransformDict References: <20130913204058.518f43d3@fsol> <l0vp4p$3lj$1@ger.gmane.org> <20130913213725.19dd8817@fsol> <CAH0mxTRZ7v5H-7MudrYSc32BbRTW8bu4thznas52LcXKW7HDUg@mail.gmail.com> <20130913220236.1f27b73f@fsol> <l0vrtv$cso$2@ger.gmane.org> <20130913234516.10661214@fsol> <20130914004952.GJ16820@ando> <5233BB16.8050201@mrabarnett.plus.com> <5233BE76.2040206@stoneleaf.us> <CAH0mxTQOkNm8zMEXD-jtLxnoru-ik7pRJ92sRe7dD463-Eh3gg@mail.gmail.com> <5233ED1F.9020808@stoneleaf.us> <20130914122706.6c1efbc7@fsol> <52349221.10808@stoneleaf.us> <20130914194131.232668c3@fsol> <l12c5a$m72$1@ger.gmane.org> Message-ID: <20130914211225.2d99fcb9@fsol> On Sat, 14 Sep 2013 22:07:50 +0300 Serhiy Storchaka <storchaka at gmail.com> wrote: > 14.09.13 20:41, Antoine Pitrou ???????(??): > > On Sat, 14 Sep 2013 09:43:13 -0700 > > Ethan Furman <ethan at stoneleaf.us> wrote: > >> Still, I think it would be useful to expose the transform function. > >> Any good reason not to? > > > > No good reason. What's the name? transform_func? > > There is one reason -- serialization. For example pickle saves the > transform function as an argument for TransformDict constructor. Repr > exposes the transform function too (in evaluable representation). Other > serializers need the transform function too. My implementations expose > it as public property (transform), your -- as private attribute > (_transform). > > Or perhaps I misunderstood you? "No good reason" not to expose it. It was a double negation, sorry ;-) Regards Antoine. From tjreedy at udel.edu Sat Sep 14 20:14:10 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 14 Sep 2013 14:14:10 -0400 Subject: [Python-Dev] [Python-checkins] cpython: Issue #18937: Add an assertLogs() context manager to unittest.TestCase to In-Reply-To: <3cch2p3mLCz7LjT@mail.python.org> References: <3cch2p3mLCz7LjT@mail.python.org> Message-ID: <5234A772.4000800@udel.edu> On 9/14/2013 1:45 PM, antoine.pitrou wrote: > http://hg.python.org/cpython/rev/4f5815747f58 > changeset: 85701:4f5815747f58 > user: Antoine Pitrou <solipsis at pitrou.net> > date: Sat Sep 14 19:45:47 2013 +0200 > summary: > Issue #18937: Add an assertLogs() context manager to unittest.TestCase to ensure that a block of code emits a message using the logging module. > > files: > Doc/library/unittest.rst | 41 +++++++ > Lib/unittest/case.py | 109 +++++++++++++++++++- > Lib/unittest/test/test_case.py | 96 ++++++++++++++++++ > Misc/NEWS | 2 + > 4 files changed, 242 insertions(+), 6 deletions(-) > > > diff --git a/Doc/library/unittest.rst b/Doc/library/unittest.rst > --- a/Doc/library/unittest.rst > +++ b/Doc/library/unittest.rst > @@ -1031,6 +1031,47 @@ > .. versionchanged:: 3.3 > Added the *msg* keyword argument when used as a context manager. assertWarnsRegex > + .. method:: assertLogs(logger=None, level=None) > + > + A context manager to test that at least one message is logged on > + the *logger* or one of its children, with at least the given > + *level*. > + > + If given, *logger* should be a :class:`logging.Logger` object or a > + :class:`str` giving the name of a logger. The default is the root > + logger, which will catch all messages. > + > + If given, *level* should be either a numeric logging level or > + its string equivalent (for example either ``"ERROR"`` or > + :attr:`logging.ERROR`). The default is :attr:`logging.INFO`. > + > + The test passes if at least one message emitted inside the ``with`` > + block matches the *logger* and *level* conditions, otherwise it fails. > + > + The object returned by the context manager is a recording helper > + which keeps tracks of the matching log messages. It has two > + attributes: > + > + .. attribute:: records > + > + A list of :class:`logging.LogRecord` objects of the matching > + log messages. > + > + .. attribute:: output > + > + A list of :class:`str` objects with the formatted output of > + matching messages. > + > + Example:: > + > + with self.assertLogs('foo', level='INFO') as cm: > + logging.getLogger('foo').info('first message') > + logging.getLogger('foo.bar').error('second message') > + self.assertEqual(cm.output, ['INFO:foo:first message', > + 'ERROR:foo.bar:second message']) > + > + .. versionadded:: 3.4 > + > > There are also other methods used to perform more specific checks, such as: The new method should be added to the table of raises/warns methods that starts this group. The table intro sentence "It is also possible to check that exceptions and warnings are raised using the following methods:" should be modified to something like "It is also possible to check the production of exception, warning, and log messages using the following methods:" Terry From guido at python.org Sat Sep 14 21:31:36 2013 From: guido at python.org (Guido van Rossum) Date: Sat, 14 Sep 2013 12:31:36 -0700 Subject: [Python-Dev] PEP 455: TransformDict In-Reply-To: <20130913204058.518f43d3@fsol> References: <20130913204058.518f43d3@fsol> Message-ID: <CAP7+vJJzzCOLz0WCX23=aZFDP4-_OWum1weV_-KjZs3=2CMC5A@mail.gmail.com> On Fri, Sep 13, 2013 at 11:40 AM, Antoine Pitrou <solipsis at pitrou.net>wrote: > Following the python-dev discussion, I've written a PEP to recap the > proposal and the various arguments. It's inlined below, and it will > probably appear soon at http://www.python.org/dev/peps/pep-0455/, too. > Thanks, Antoine! Raymond Hettinger has volunteered to be the PEP dictator (is that the word we use?) for this PEP. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130914/1dcc7fa9/attachment-0001.html> From raymond.hettinger at gmail.com Sat Sep 14 21:44:17 2013 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Sat, 14 Sep 2013 12:44:17 -0700 Subject: [Python-Dev] Compiler for the Mac OS X version of Python 3.4 Message-ID: <70C99F87-E9A5-4838-A1E9-4739FBF2E6A5@gmail.com> I was exercising the alpha two release of 3.4 and noticed that it is still being built under GCC 4.2.1. Is there any reason we have to use an old compiler? I would like to see it built under the latest version of Clang (like the other tools on the Mac) or under GCC 4.8.1. I've better using the latter for many months. It works great and produces *much* better code (in terms of readability for those of us who look at assembly output and in terms of speed (approx 20-25% better for cpu bound code)). I'm wondering whether is some issue that forces us to use GCC 4.2.1 or whether that is just an artifact of continuing to do what we've always been doing. Raymond From nad at acm.org Sat Sep 14 22:32:04 2013 From: nad at acm.org (Ned Deily) Date: Sat, 14 Sep 2013 13:32:04 -0700 Subject: [Python-Dev] Compiler for the Mac OS X version of Python 3.4 References: <70C99F87-E9A5-4838-A1E9-4739FBF2E6A5@gmail.com> Message-ID: <nad-EC5D57.13320314092013@news.gmane.org> In article <70C99F87-E9A5-4838-A1E9-4739FBF2E6A5 at gmail.com>, Raymond Hettinger <raymond.hettinger at gmail.com> wrote: > I was exercising the alpha two release of 3.4 and noticed that > it is still being built under GCC 4.2.1. > > Is there any reason we have to use an old compiler? Yes, kinda. It's because the 64-bit/32-bit installer we supply is designed to run on multiple versions of OS X: 10.6 (Snow Leopard), 10.7, 10.8, and the upcoming 10.9. The safest way to ensure that the bits produced (executables, shared libs, and .so bundles) are linked with compatible versions of system shared libraries and frameworks is to build on the lowest supported system, 10.6, or - carefully - with a 10.6 SDK on a newer system. The standard compiler in the Apple Developer Tools for 10.6 is gcc 4.2. Clang was very immature at that point. The most recent Developer Tools for 10.8 and 10.7 systems, Xcode 4.6.x, have a mature clang but do not provide a 10.6 SDK. Even with using an SDK, it's still possible to end up inadvertently linking with the wrong versions of system libraries. We have been burned by that in the past. > I would like to see it built under the latest version of Clang > (like the other tools on the Mac) or under GCC 4.8.1. > I've better using the latter for many months. It works great > and produces *much* better code (in terms of readability > for those of us who look at assembly output and in terms > of speed (approx 20-25% better for cpu bound code)). > > I'm wondering whether is some issue that forces us to use > GCC 4.2.1 or whether that is just an artifact of continuing > to do what we've always been doing. I'd like to move to Apple's latest clang, as well. (Apple is dropping gcc totally from the next release of Xcode.) There are a couple of approaches that may work and still reliably support multiple versions of OS X. Investigating this is on my list of things to do prior to the next alpha. I've opened a bug tracker issue for this (Issue19019). -- Ned Deily, nad at acm.org From solipsis at pitrou.net Sat Sep 14 22:36:09 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 14 Sep 2013 22:36:09 +0200 Subject: [Python-Dev] PEP 455: TransformDict In-Reply-To: <CAP7+vJJzzCOLz0WCX23=aZFDP4-_OWum1weV_-KjZs3=2CMC5A@mail.gmail.com> References: <20130913204058.518f43d3@fsol> <CAP7+vJJzzCOLz0WCX23=aZFDP4-_OWum1weV_-KjZs3=2CMC5A@mail.gmail.com> Message-ID: <20130914223609.4deceb6a@fsol> On Sat, 14 Sep 2013 12:31:36 -0700 Guido van Rossum <guido at python.org> wrote: > On Fri, Sep 13, 2013 at 11:40 AM, Antoine Pitrou <solipsis at pitrou.net>wrote: > > > Following the python-dev discussion, I've written a PEP to recap the > > proposal and the various arguments. It's inlined below, and it will > > probably appear soon at http://www.python.org/dev/peps/pep-0455/, too. > > > > Thanks, Antoine! > > Raymond Hettinger has volunteered to be the PEP dictator (is that the word > we use?) for this PEP. The discussion has settled down for the most part, and there is a consensus amongst participants on the desireability of the feature and its characteristics. The implementation is straightforward pure Python (Serhiy's C proposal should probably be a separate enhancement request on the tracker). I think the proposal will soon be ready for a pronouncement - unless other concrete questions and suggestions are recorded. Thanks Antoine. From raymond.hettinger at gmail.com Sat Sep 14 23:29:37 2013 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Sat, 14 Sep 2013 14:29:37 -0700 Subject: [Python-Dev] Compiler for the Mac OS X version of Python 3.4 In-Reply-To: <nad-EC5D57.13320314092013@news.gmane.org> References: <70C99F87-E9A5-4838-A1E9-4739FBF2E6A5@gmail.com> <nad-EC5D57.13320314092013@news.gmane.org> Message-ID: <B3293155-E4D5-4389-A555-C31BC49CE539@gmail.com> On Sep 14, 2013, at 1:32 PM, Ned Deily <nad at acm.org> wrote: > The > most recent Developer Tools for 10.8 and 10.7 systems, Xcode 4.6.x, have > a mature clang but do not provide a 10.6 SDK. Even with using an SDK, > it's still possible to end up inadvertently linking with the wrong > versions of system libraries. We have been burned by that in the past. I think we should offer a separate Mac build just for 10.6 (much like we do for the 32-bit PPC option for 10.5). Otherwise, were trapped in an old world and using the same reasoning as companies that still mandate Internet Explorer 6; forgoing the benefits of newer browsers just to make sure a few pieces of legacy code can be supported. > > I'd like to move to Apple's latest clang, as well. (Apple is dropping > gcc totally from the next release of Xcode.) That is an excellent reason for us to move as well. Otherwise, we risk getting out of sync with the rest of the ecosystem. > I've opened a bug tracker issue for this (Issue19019). Thank you. One thing that can be done is to take advantage of the 3.4 alphas and betas to experiment with having both a legacy version and fresh build with better tools. That would let us find out early whether the worries and fears are real and will manifest themselves in practice. I've been building under GCC 4.8.1 for months and have encountered no problems at all with the popular third-party modules or with linking to existing SO files. Raymond From storchaka at gmail.com Sat Sep 14 23:55:35 2013 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sun, 15 Sep 2013 00:55:35 +0300 Subject: [Python-Dev] PEP 455: TransformDict In-Reply-To: <20130914194131.232668c3@fsol> References: <20130913204058.518f43d3@fsol> <l0vp4p$3lj$1@ger.gmane.org> <20130913213725.19dd8817@fsol> <CAH0mxTRZ7v5H-7MudrYSc32BbRTW8bu4thznas52LcXKW7HDUg@mail.gmail.com> <20130913220236.1f27b73f@fsol> <l0vrtv$cso$2@ger.gmane.org> <20130913234516.10661214@fsol> <20130914004952.GJ16820@ando> <5233BB16.8050201@mrabarnett.plus.com> <5233BE76.2040206@stoneleaf.us> <CAH0mxTQOkNm8zMEXD-jtLxnoru-ik7pRJ92sRe7dD463-Eh3gg@mail.gmail.com> <5233ED1F.9020808@stoneleaf.us> <20130914122706.6c1efbc7@fsol> <52349221.10808@stoneleaf.us> <20130914194131.232668c3@fsol> Message-ID: <l12m0b$9f6$1@ger.gmane.org> 14.09.13 20:41, Antoine Pitrou ???????(??): > No good reason. What's the name? transform_func? transform_func looks... truncated. Why not transform_function or trans_func? From solipsis at pitrou.net Sat Sep 14 23:58:44 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 14 Sep 2013 23:58:44 +0200 Subject: [Python-Dev] PEP 455: TransformDict References: <20130913204058.518f43d3@fsol> <l0vp4p$3lj$1@ger.gmane.org> <20130913213725.19dd8817@fsol> <CAH0mxTRZ7v5H-7MudrYSc32BbRTW8bu4thznas52LcXKW7HDUg@mail.gmail.com> <20130913220236.1f27b73f@fsol> <l0vrtv$cso$2@ger.gmane.org> <20130913234516.10661214@fsol> <20130914004952.GJ16820@ando> <5233BB16.8050201@mrabarnett.plus.com> <5233BE76.2040206@stoneleaf.us> <CAH0mxTQOkNm8zMEXD-jtLxnoru-ik7pRJ92sRe7dD463-Eh3gg@mail.gmail.com> <5233ED1F.9020808@stoneleaf.us> <20130914122706.6c1efbc7@fsol> <52349221.10808@stoneleaf.us> <20130914194131.232668c3@fsol> <l12m0b$9f6$1@ger.gmane.org> Message-ID: <20130914235844.08eacd96@fsol> On Sun, 15 Sep 2013 00:55:35 +0300 Serhiy Storchaka <storchaka at gmail.com> wrote: > 14.09.13 20:41, Antoine Pitrou ???????(??): > > No good reason. What's the name? transform_func? > > transform_func looks... truncated. Why not transform_function or trans_func? The stdlib commonly uses "func" rather than "function". For example "functools", where partial has an attribute named "func". I suppose it's like "int" vs. "integer". Regards Antoine. From eliben at gmail.com Sun Sep 15 00:31:38 2013 From: eliben at gmail.com (Eli Bendersky) Date: Sat, 14 Sep 2013 15:31:38 -0700 Subject: [Python-Dev] PEP 455: TransformDict In-Reply-To: <l12m0b$9f6$1@ger.gmane.org> References: <20130913204058.518f43d3@fsol> <l0vp4p$3lj$1@ger.gmane.org> <20130913213725.19dd8817@fsol> <CAH0mxTRZ7v5H-7MudrYSc32BbRTW8bu4thznas52LcXKW7HDUg@mail.gmail.com> <20130913220236.1f27b73f@fsol> <l0vrtv$cso$2@ger.gmane.org> <20130913234516.10661214@fsol> <20130914004952.GJ16820@ando> <5233BB16.8050201@mrabarnett.plus.com> <5233BE76.2040206@stoneleaf.us> <CAH0mxTQOkNm8zMEXD-jtLxnoru-ik7pRJ92sRe7dD463-Eh3gg@mail.gmail.com> <5233ED1F.9020808@stoneleaf.us> <20130914122706.6c1efbc7@fsol> <52349221.10808@stoneleaf.us> <20130914194131.232668c3@fsol> <l12m0b$9f6$1@ger.gmane.org> Message-ID: <CAF-Rda_efmsrrU3+fws7orO6WpndE4XXS397HRb+OeXprGfY+A@mail.gmail.com> On Sat, Sep 14, 2013 at 2:55 PM, Serhiy Storchaka <storchaka at gmail.com>wrote: > 14.09.13 20:41, Antoine Pitrou ???????(??): > >> No good reason. What's the name? transform_func? >> > > transform_func looks... truncated. Why not transform_function or > trans_func? transform_? -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130914/ffcb6223/attachment.html> From mark at hotpy.org Sun Sep 15 00:54:58 2013 From: mark at hotpy.org (Mark Shannon) Date: Sat, 14 Sep 2013 23:54:58 +0100 Subject: [Python-Dev] PEP 455: TransformDict In-Reply-To: <CAF-Rda_efmsrrU3+fws7orO6WpndE4XXS397HRb+OeXprGfY+A@mail.gmail.com> References: <20130913204058.518f43d3@fsol> <l0vp4p$3lj$1@ger.gmane.org> <20130913213725.19dd8817@fsol> <CAH0mxTRZ7v5H-7MudrYSc32BbRTW8bu4thznas52LcXKW7HDUg@mail.gmail.com> <20130913220236.1f27b73f@fsol> <l0vrtv$cso$2@ger.gmane.org> <20130913234516.10661214@fsol> <20130914004952.GJ16820@ando> <5233BB16.8050201@mrabarnett.plus.com> <5233BE76.2040206@stoneleaf.us> <CAH0mxTQOkNm8zMEXD-jtLxnoru-ik7pRJ92sRe7dD463-Eh3gg@mail.gmail.com> <5233ED1F.9020808@stoneleaf.us> <20130914122706.6c1efbc7@fsol> <52349221.10808@stoneleaf.us> <20130914194131.232668c3@fsol> <l12m0b$9f6$1@ger.gmane.org> <CAF-Rda_efmsrrU3+fws7orO6WpndE4XXS397HRb+OeXprGfY+A@mail.gmail.com> Message-ID: <5234E942.7090801@hotpy.org> On 14/09/13 23:31, Eli Bendersky wrote: > > > > On Sat, Sep 14, 2013 at 2:55 PM, Serhiy Storchaka <storchaka at gmail.com > <mailto:storchaka at gmail.com>> wrote: > > 14.09.13 20:41, Antoine Pitrou ???????(??): > > No good reason. What's the name? transform_func? > > > transform_func looks... truncated. Why not transform_function or > trans_func? > > > transform_? > Shouldn't that be transform_?? (PEP 3117) From rymg19 at gmail.com Sun Sep 15 00:56:06 2013 From: rymg19 at gmail.com (Ryan) Date: Sat, 14 Sep 2013 17:56:06 -0500 Subject: [Python-Dev] Compiler for the Mac OS X version of Python 3.4 In-Reply-To: <B3293155-E4D5-4389-A555-C31BC49CE539@gmail.com> References: <70C99F87-E9A5-4838-A1E9-4739FBF2E6A5@gmail.com> <nad-EC5D57.13320314092013@news.gmane.org> <B3293155-E4D5-4389-A555-C31BC49CE539@gmail.com> Message-ID: <534e7dd6-e7e2-49cd-b719-8a070852b623@email.android.com> +1. A 10.6-only build makes sense. If you aren't having problems with GCC 4.8, then Clang shouldn't give any trouble. Honestly, I still think Clang should be a compiler option in Windows distutils... Raymond Hettinger <raymond.hettinger at gmail.com> wrote: > >On Sep 14, 2013, at 1:32 PM, Ned Deily <nad at acm.org> wrote: >> The >> most recent Developer Tools for 10.8 and 10.7 systems, Xcode 4.6.x, >have >> a mature clang but do not provide a 10.6 SDK. Even with using an >SDK, >> it's still possible to end up inadvertently linking with the wrong >> versions of system libraries. We have been burned by that in the >past. > >I think we should offer a separate Mac build just for 10.6 >(much like we do for the 32-bit PPC option for 10.5). > >Otherwise, were trapped in an old world and using the same >reasoning as companies that still mandate Internet Explorer 6; >forgoing the benefits of newer browsers just to make sure >a few pieces of legacy code can be supported. > >> >> I'd like to move to Apple's latest clang, as well. (Apple is >dropping >> gcc totally from the next release of Xcode.) > >That is an excellent reason for us to move as well. >Otherwise, we risk getting out of sync with the rest of the ecosystem. > >> I've opened a bug tracker issue for this (Issue19019). > >Thank you. > >One thing that can be done is to take advantage of the 3.4 alphas >and betas to experiment with having both a legacy version and >fresh build with better tools. That would let us find out early >whether >the worries and fears are real and will manifest themselves in >practice. > >I've been building under GCC 4.8.1 for months and have encountered >no problems at all with the popular third-party modules or with linking >to existing SO files. > > >Raymond >_______________________________________________ >Python-Dev mailing list >Python-Dev at python.org >https://mail.python.org/mailman/listinfo/python-dev >Unsubscribe: >https://mail.python.org/mailman/options/python-dev/rymg19%40gmail.com -- Sent from my Android phone with K-9 Mail. Please excuse my brevity. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130914/2bdb2b26/attachment.html> From storchaka at gmail.com Sun Sep 15 02:32:05 2013 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sun, 15 Sep 2013 03:32:05 +0300 Subject: [Python-Dev] PEP 455: TransformDict In-Reply-To: <20130914235844.08eacd96@fsol> References: <20130913204058.518f43d3@fsol> <l0vp4p$3lj$1@ger.gmane.org> <20130913213725.19dd8817@fsol> <CAH0mxTRZ7v5H-7MudrYSc32BbRTW8bu4thznas52LcXKW7HDUg@mail.gmail.com> <20130913220236.1f27b73f@fsol> <l0vrtv$cso$2@ger.gmane.org> <20130913234516.10661214@fsol> <20130914004952.GJ16820@ando> <5233BB16.8050201@mrabarnett.plus.com> <5233BE76.2040206@stoneleaf.us> <CAH0mxTQOkNm8zMEXD-jtLxnoru-ik7pRJ92sRe7dD463-Eh3gg@mail.gmail.com> <5233ED1F.9020808@stoneleaf.us> <20130914122706.6c1efbc7@fsol> <52349221.10808@stoneleaf.us> <20130914194131.232668c3@fsol> <l12m0b$9f6$1@ger.gmane.org> <20130914235844.08eacd96@fsol> Message-ID: <l12v58$gi7$1@ger.gmane.org> 15.09.13 00:58, Antoine Pitrou ???????(??): > On Sun, 15 Sep 2013 00:55:35 +0300 > Serhiy Storchaka <storchaka at gmail.com> wrote: >> 14.09.13 20:41, Antoine Pitrou ???????(??): >>> No good reason. What's the name? transform_func? >> >> transform_func looks... truncated. Why not transform_function or trans_func? > > The stdlib commonly uses "func" rather than "function". For example > "functools", where partial has an attribute named "func". > I suppose it's like "int" vs. "integer". The same functools has decorating_function() with the user_function argument. Also find_function, search_function, create_function, isfunction, isgeneratorfunction, from_function, has_function, copy_function, pickle_function, register_function, emit_function, and print_function. This is not counting tests, IDLE and private names. From ethan at stoneleaf.us Sun Sep 15 02:57:18 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Sat, 14 Sep 2013 17:57:18 -0700 Subject: [Python-Dev] PEP 455: TransformDict In-Reply-To: <l12v58$gi7$1@ger.gmane.org> References: <20130913204058.518f43d3@fsol> <l0vp4p$3lj$1@ger.gmane.org> <20130913213725.19dd8817@fsol> <CAH0mxTRZ7v5H-7MudrYSc32BbRTW8bu4thznas52LcXKW7HDUg@mail.gmail.com> <20130913220236.1f27b73f@fsol> <l0vrtv$cso$2@ger.gmane.org> <20130913234516.10661214@fsol> <20130914004952.GJ16820@ando> <5233BB16.8050201@mrabarnett.plus.com> <5233BE76.2040206@stoneleaf.us> <CAH0mxTQOkNm8zMEXD-jtLxnoru-ik7pRJ92sRe7dD463-Eh3gg@mail.gmail.com> <5233ED1F.9020808@stoneleaf.us> <20130914122706.6c1efbc7@fsol> <52349221.10808@stoneleaf.us> <20130914194131.232668c3@fsol> <l12m0b$9f6$1@ger.gmane.org> <20130914235844.08eacd96@fsol> <l12v58$gi7$1@ger.gmane.org> Message-ID: <523505EE.1060306@stoneleaf.us> On 09/14/2013 05:32 PM, Serhiy Storchaka wrote: > 15.09.13 00:58, Antoine Pitrou ???????(??): >> On Sun, 15 Sep 2013 00:55:35 +0300 >> Serhiy Storchaka <storchaka at gmail.com> wrote: >>> 14.09.13 20:41, Antoine Pitrou ???????(??): >>>> No good reason. What's the name? transform_func? >>> >>> transform_func looks... truncated. Why not transform_function or trans_func? >> >> The stdlib commonly uses "func" rather than "function". For example >> "functools", where partial has an attribute named "func". >> I suppose it's like "int" vs. "integer". > > The same functools has decorating_function() with the user_function argument. > > Also find_function, search_function, create_function, isfunction, isgeneratorfunction, from_function, has_function, > copy_function, pickle_function, register_function, emit_function, and print_function. This is not counting tests, IDLE > and private names. The name of the parameter in __init__ is 'transform'. We could just call it that. -- ~Ethan~ From larry at hastings.org Sun Sep 15 06:56:26 2013 From: larry at hastings.org (Larry Hastings) Date: Sun, 15 Sep 2013 13:56:26 +0900 Subject: [Python-Dev] PEP 455: TransformDict In-Reply-To: <20130914123046.4f053e6a@fsol> References: <20130913204058.518f43d3@fsol> <5233F544.7030604@hastings.org> <20130914123046.4f053e6a@fsol> Message-ID: <52353DFA.9060804@hastings.org> On 09/14/2013 07:30 PM, Antoine Pitrou wrote: > On Sat, 14 Sep 2013 14:33:56 +0900 > Larry Hastings <larry at hastings.org> wrote: >> Whenever I read a discussion about the dict, I always wonder whether the >> same thing applies to a set. Have you considered the utility of a >> TransformSet? Or is it YAGNI? > Well, a TransformSet is like a normal dict, you just need to call the > transformation function yourself when inserting the keys. s/normal dict/normal set/ But, then, a TransformDict is like a normal dict, you just need to call the transformation function yourself when inserting the keys. Yet a TransformDict is a useful enough concept that it is being proposed for addition; I was wondering if a TransformSet would be useful, too. But I suppose we should take baby steps here. //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130915/0b8d4b83/attachment-0001.html> From g.brandl at gmx.net Sun Sep 15 10:34:04 2013 From: g.brandl at gmx.net (Georg Brandl) Date: Sun, 15 Sep 2013 10:34:04 +0200 Subject: [Python-Dev] cpython: Issue #18571: Implementation of the PEP 446: file descriptors and file handles In-Reply-To: <3cPmKh2nMPz7LlL@mail.python.org> References: <3cPmKh2nMPz7LlL@mail.python.org> Message-ID: <l13rcd$23b$1@ger.gmane.org> On 08/28/2013 01:20 AM, victor.stinner wrote: > http://hg.python.org/cpython/rev/ef889c3d5dc6 > changeset: 85420:ef889c3d5dc6 > user: Victor Stinner <victor.stinner at gmail.com> > date: Wed Aug 28 00:53:59 2013 +0200 > summary: > Issue #18571: Implementation of the PEP 446: file descriptors and file handles > are now created non-inheritable; add functions os.get/set_inheritable(), > os.get/set_handle_inheritable() and socket.socket.get/set_inheritable(). > +.. _fd_inheritance: > + > +Inheritance of File Descriptors > +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > + > +A file descriptor has a inheritable flag which indicates if the file descriptor > +can be inherited or not in child processes. Since Python 3.4, file descriptors > +created by Python are non-inheritable by default. > + > +On UNIX, non-inheritable file descriptors are closed in child processes at the > +execution of a new program, other file descriptors are inherited. > + > +On Windows, non-inheritable handles and file descriptors are closed in child > +processes, except standard streams (file descriptors 0, 1 and 2: stdin, stdout > +and stderr) which are always inherited. Using :func:`os.spawn*` functions, > +all inheritable handles and all inheritable file descriptors are inherited. > +Using the :mod:`subprocess` module, all file descriptors except standard > +streams are closed, inheritable handles are only inherited if the *close_fds* > +parameter is ``False``. > + > +.. versionadded:: 3.4 > + > +.. function:: get_inheritable(fd) > + > + Get the `inheritable flag <fd_inheritance>`_ of the specified file > + descriptor. Return a :class:`bool`. > + > +.. function:: set_inheritable(fd, inheritable) > + > + Set the `inheritable flag <fd_inheritance>`_ of the specified file descriptor. > + > +.. function:: get_handle_inheritable(handle) > + > + Get the `inheritable flag <fd_inheritance>`_ of the specified handle. Return a :class:`bool`. > + > + Availability: Windows. > + > +.. function:: set_handle_inheritable(handle, inheritable) > + > + Set the `inheritable flag <fd_inheritance>`_ of the specified handle. > + > + Availability: Windows. > + "Handle" is used nowhere else in the os module documentation. Do you think it would make sense to have a brief paragraph on what are possible handles under Windows (and that file descriptors aren't handles)? Georg From solipsis at pitrou.net Sun Sep 15 13:23:29 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 15 Sep 2013 13:23:29 +0200 Subject: [Python-Dev] PEP 455: TransformDict References: <20130913204058.518f43d3@fsol> <5233F544.7030604@hastings.org> <20130914123046.4f053e6a@fsol> <52353DFA.9060804@hastings.org> Message-ID: <20130915132329.5ea6adf6@fsol> On Sun, 15 Sep 2013 13:56:26 +0900 Larry Hastings <larry at hastings.org> wrote: > On 09/14/2013 07:30 PM, Antoine Pitrou wrote: > > On Sat, 14 Sep 2013 14:33:56 +0900 > > Larry Hastings <larry at hastings.org> wrote: > >> Whenever I read a discussion about the dict, I always wonder whether the > >> same thing applies to a set. Have you considered the utility of a > >> TransformSet? Or is it YAGNI? > > Well, a TransformSet is like a normal dict, you just need to call the > > transformation function yourself when inserting the keys. > > s/normal dict/normal set/ No, read my example. From storchaka at gmail.com Sun Sep 15 15:14:24 2013 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sun, 15 Sep 2013 16:14:24 +0300 Subject: [Python-Dev] PEP 455: TransformDict In-Reply-To: <20130915132329.5ea6adf6@fsol> References: <20130913204058.518f43d3@fsol> <5233F544.7030604@hastings.org> <20130914123046.4f053e6a@fsol> <52353DFA.9060804@hastings.org> <20130915132329.5ea6adf6@fsol> Message-ID: <l14bqq$cdk$1@ger.gmane.org> 15.09.13 14:23, Antoine Pitrou ???????(??): > On Sun, 15 Sep 2013 13:56:26 +0900 > Larry Hastings <larry at hastings.org> wrote: >> On 09/14/2013 07:30 PM, Antoine Pitrou wrote: >>> On Sat, 14 Sep 2013 14:33:56 +0900 >>> Larry Hastings <larry at hastings.org> wrote: >>>> Whenever I read a discussion about the dict, I always wonder whether the >>>> same thing applies to a set. Have you considered the utility of a >>>> TransformSet? Or is it YAGNI? >>> Well, a TransformSet is like a normal dict, you just need to call the >>> transformation function yourself when inserting the keys. >> >> s/normal dict/normal set/ > > No, read my example. A TransformDict is like a normal dict, you just need to call the transformation function yourself when inserting the keys and insert a pair of (orig_key, value), or is like two normal dicts, you just need to call the transformation function yourself when inserting the keys and update parallel mapping from transformed keys to original keys. From solipsis at pitrou.net Sun Sep 15 15:57:07 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 15 Sep 2013 15:57:07 +0200 Subject: [Python-Dev] PEP 455: TransformDict References: <20130913204058.518f43d3@fsol> <5233F544.7030604@hastings.org> <20130914123046.4f053e6a@fsol> <52353DFA.9060804@hastings.org> <20130915132329.5ea6adf6@fsol> <l14bqq$cdk$1@ger.gmane.org> Message-ID: <20130915155707.6ee9790d@fsol> On Sun, 15 Sep 2013 16:14:24 +0300 Serhiy Storchaka <storchaka at gmail.com> wrote: > 15.09.13 14:23, Antoine Pitrou ???????(??): > > On Sun, 15 Sep 2013 13:56:26 +0900 > > Larry Hastings <larry at hastings.org> wrote: > >> On 09/14/2013 07:30 PM, Antoine Pitrou wrote: > >>> On Sat, 14 Sep 2013 14:33:56 +0900 > >>> Larry Hastings <larry at hastings.org> wrote: > >>>> Whenever I read a discussion about the dict, I always wonder whether the > >>>> same thing applies to a set. Have you considered the utility of a > >>>> TransformSet? Or is it YAGNI? > >>> Well, a TransformSet is like a normal dict, you just need to call the > >>> transformation function yourself when inserting the keys. > >> > >> s/normal dict/normal set/ > > > > No, read my example. > > A TransformDict is like a normal dict, you just need to call the > transformation function yourself when inserting the keys and insert a > pair of (orig_key, value), or is like two normal dicts, you just need to > call the transformation function yourself when inserting the keys and > update parallel mapping from transformed keys to original keys. I don't really care. What's the point in the end? TransformDict is non-trivial to implement, while the so-called "TransformSet" is just a dict with a different API. From victor.stinner at gmail.com Sun Sep 15 16:04:45 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Sun, 15 Sep 2013 16:04:45 +0200 Subject: [Python-Dev] cpython: Issue #18571: Implementation of the PEP 446: file descriptors and file handles In-Reply-To: <l13rcd$23b$1@ger.gmane.org> References: <3cPmKh2nMPz7LlL@mail.python.org> <l13rcd$23b$1@ger.gmane.org> Message-ID: <CAMpsgwb+W_fOtjvPjHcGT-VgwJ34ZEHvMRiFEybSkQ18pnvtnw@mail.gmail.com> Yes, but I'm not interested to write such doc. Victor Le 15 sept. 2013 10:34, "Georg Brandl" <g.brandl at gmx.net> a ?crit : > On 08/28/2013 01:20 AM, victor.stinner wrote: > > http://hg.python.org/cpython/rev/ef889c3d5dc6 > > changeset: 85420:ef889c3d5dc6 > > user: Victor Stinner <victor.stinner at gmail.com> > > date: Wed Aug 28 00:53:59 2013 +0200 > > summary: > > Issue #18571: Implementation of the PEP 446: file descriptors and file > handles > > are now created non-inheritable; add functions os.get/set_inheritable(), > > os.get/set_handle_inheritable() and socket.socket.get/set_inheritable(). > > > +.. _fd_inheritance: > > + > > +Inheritance of File Descriptors > > +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > > + > > +A file descriptor has a inheritable flag which indicates if the file > descriptor > > +can be inherited or not in child processes. Since Python 3.4, file > descriptors > > +created by Python are non-inheritable by default. > > + > > +On UNIX, non-inheritable file descriptors are closed in child processes > at the > > +execution of a new program, other file descriptors are inherited. > > + > > +On Windows, non-inheritable handles and file descriptors are closed in > child > > +processes, except standard streams (file descriptors 0, 1 and 2: stdin, > stdout > > +and stderr) which are always inherited. Using :func:`os.spawn*` > functions, > > +all inheritable handles and all inheritable file descriptors are > inherited. > > +Using the :mod:`subprocess` module, all file descriptors except standard > > +streams are closed, inheritable handles are only inherited if the > *close_fds* > > +parameter is ``False``. > > + > > +.. versionadded:: 3.4 > > + > > +.. function:: get_inheritable(fd) > > + > > + Get the `inheritable flag <fd_inheritance>`_ of the specified file > > + descriptor. Return a :class:`bool`. > > + > > +.. function:: set_inheritable(fd, inheritable) > > + > > + Set the `inheritable flag <fd_inheritance>`_ of the specified file > descriptor. > > + > > +.. function:: get_handle_inheritable(handle) > > + > > + Get the `inheritable flag <fd_inheritance>`_ of the specified > handle. Return a :class:`bool`. > > + > > + Availability: Windows. > > + > > +.. function:: set_handle_inheritable(handle, inheritable) > > + > > + Set the `inheritable flag <fd_inheritance>`_ of the specified handle. > > + > > + Availability: Windows. > > + > > "Handle" is used nowhere else in the os module documentation. Do you > think it > would make sense to have a brief paragraph on what are possible handles > under > Windows (and that file descriptors aren't handles)? > > Georg > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130915/62cb1131/attachment.html> From ethan at stoneleaf.us Sun Sep 15 15:46:08 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Sun, 15 Sep 2013 06:46:08 -0700 Subject: [Python-Dev] PEP 428: Pathlib Message-ID: <5235BA20.8010105@stoneleaf.us> I see PEP 428 is both targeted at 3.4 and still in draft status. What remains to be done to ask for pronouncement? -- ~Ethan~ From ethan at stoneleaf.us Sun Sep 15 16:19:32 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Sun, 15 Sep 2013 07:19:32 -0700 Subject: [Python-Dev] PEP 422: Simpler customisation of class creation Message-ID: <5235C1F4.5070804@stoneleaf.us> Three questions: Would the new __initclass__ function be usable with actual full-blown metaclasses? The PEP says this: > > If present on the created object, this new hook will be called by the class creation machinery after the __class__ reference has been initialised. Given that statement, and this example: class A: def __initclass__(cls): # do stuff class B(A): pass am I correct that A's __initclass__ will not run when B is created? Where are we at with being ready for pronouncement? -- ~Ethan~ From ncoghlan at gmail.com Sun Sep 15 16:40:48 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 16 Sep 2013 00:40:48 +1000 Subject: [Python-Dev] PEP 422: Simpler customisation of class creation In-Reply-To: <5235C1F4.5070804@stoneleaf.us> References: <5235C1F4.5070804@stoneleaf.us> Message-ID: <CADiSq7f6kSYHdZqPuUsOSk8PR3bOf+TQxWQ8augBJjbx1PrDnA@mail.gmail.com> On 16 September 2013 00:19, Ethan Furman <ethan at stoneleaf.us> wrote: > Three questions: > > > Would the new __initclass__ function be usable with actual full-blown > metaclasses? Sure (as long as the metaclass in question called it at the appropriate time, which those inheriting from type would do automatically)). > The PEP says this: >> If present on the created object, this new hook will be called by the >> class creation machinery after the __class__ reference has been initialised. > > > Given that statement, and this example: > > class A: > def __initclass__(cls): > # do stuff > > class B(A): > pass > > am I correct that A's __initclass__ will not run when B is created? No, that would be pointless (since that's the way class decorators already behave). "present on the created object" means "hasattr(cls, '__initclass__') returns True. > Where are we at with being ready for pronouncement? The PEP needs to be updated to incorporate the feedback from the last time I proposed it for inclusion. That's not going to happen for 3.4, I just forgot to mark it as Deferred - fixed now. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From storchaka at gmail.com Sun Sep 15 16:45:00 2013 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sun, 15 Sep 2013 17:45:00 +0300 Subject: [Python-Dev] PEP 455: TransformDict In-Reply-To: <20130915155707.6ee9790d@fsol> References: <20130913204058.518f43d3@fsol> <5233F544.7030604@hastings.org> <20130914123046.4f053e6a@fsol> <52353DFA.9060804@hastings.org> <20130915132329.5ea6adf6@fsol> <l14bqq$cdk$1@ger.gmane.org> <20130915155707.6ee9790d@fsol> Message-ID: <l14h4d$5lq$1@ger.gmane.org> 15.09.13 16:57, Antoine Pitrou ???????(??): > I don't really care. What's the point in the end? TransformDict is > non-trivial to implement, while the so-called "TransformSet" is just a > dict with a different API. I don't see a difference. To me TransformDict is just a dict (or two). From solipsis at pitrou.net Sun Sep 15 16:52:07 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 15 Sep 2013 16:52:07 +0200 Subject: [Python-Dev] PEP 455: TransformDict References: <20130913204058.518f43d3@fsol> <5233F544.7030604@hastings.org> <20130914123046.4f053e6a@fsol> <52353DFA.9060804@hastings.org> <20130915132329.5ea6adf6@fsol> <l14bqq$cdk$1@ger.gmane.org> <20130915155707.6ee9790d@fsol> <l14h4d$5lq$1@ger.gmane.org> Message-ID: <20130915165207.6e4733bb@fsol> On Sun, 15 Sep 2013 17:45:00 +0300 Serhiy Storchaka <storchaka at gmail.com> wrote: > 15.09.13 16:57, Antoine Pitrou ???????(??): > > I don't really care. What's the point in the end? TransformDict is > > non-trivial to implement, while the so-called "TransformSet" is just a > > dict with a different API. > > I don't see a difference. To me TransformDict is just a dict (or two). Well, feel free to propose a TransformSet PEP, but please do so in a separate discussion thread. Regards Antoine. From ncoghlan at gmail.com Sun Sep 15 16:55:27 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 16 Sep 2013 00:55:27 +1000 Subject: [Python-Dev] PEP 455: TransformDict In-Reply-To: <l14h4d$5lq$1@ger.gmane.org> References: <20130913204058.518f43d3@fsol> <5233F544.7030604@hastings.org> <20130914123046.4f053e6a@fsol> <52353DFA.9060804@hastings.org> <20130915132329.5ea6adf6@fsol> <l14bqq$cdk$1@ger.gmane.org> <20130915155707.6ee9790d@fsol> <l14h4d$5lq$1@ger.gmane.org> Message-ID: <CADiSq7dayhKrabumYv6WbJrHfv_cL4p49uNXsK-oQ77iQc_F8w@mail.gmail.com> On 16 September 2013 00:45, Serhiy Storchaka <storchaka at gmail.com> wrote: > 15.09.13 16:57, Antoine Pitrou ???????(??): > >> I don't really care. What's the point in the end? TransformDict is >> non-trivial to implement, while the so-called "TransformSet" is just a >> dict with a different API. > > > I don't see a difference. To me TransformDict is just a dict (or two). I think it's best to hold off a TransformSet, since you can trivially emulate it by setting the values in a TransformDict to None (that's how Python itself lived without a set type for so long: people just used dicts with all the values set to None). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From vitaly.murashev at gmail.com Sun Sep 15 22:37:22 2013 From: vitaly.murashev at gmail.com (Vitaly Murashev) Date: Mon, 16 Sep 2013 00:37:22 +0400 Subject: [Python-Dev] Relative path in co_filename for zipped modules In-Reply-To: <CALPW6hzs1NLihojpbwtJE1YKCr6ggHawda8wusbWvNEM=5P2PA@mail.gmail.com> References: <CALPW6hzs1NLihojpbwtJE1YKCr6ggHawda8wusbWvNEM=5P2PA@mail.gmail.com> Message-ID: <CALPW6hwGHwB5Y=aVMmrbmin960awQadAH3fF8Zh2M3n5KE3_ww@mail.gmail.com> This bug has been reported as http://bugs.python.org/issue18307 over a month ago ?nd I've found out the root cause of this issue so far. Unit-test and patch are suggested. Without patch test fails, with patch - passed. On Wed, Jun 26, 2013 at 12:11 AM, Vitaly Murashev <vitaly.murashev at gmail.com > wrote: > Dear Python developers community, > > Recently I found out that it not possible to debug python code if it > is a part of zip-module. > Python version being used is 3.3.0 > > Well known GUI debuggers like Eclipse+PyDev or PyCharm are unable to > start debugging and give the following warning: > --- > pydev debugger: CRITICAL WARNING: This version of python seems to be > incorrectly compiled (internal generated filenames are not absolute) > pydev debugger: The debugger may still function, but it will work > slower and may miss breakpoints. > --- > So I started my own investigation of this issue and results are the > following. > > At first I took traditional python debugger 'pdb' to analyze how it > behaves during debugging of zipped module. > 'pdb' showed me some backtaces and filename part for stack entries > looks malformed. > I expected something like > 'full-path-to-zip-dir/my_zipped_module.zip/subdir/test_module.py' > but realy it looks like 'full-path-to-current-dir/subdir/test_module.py' > > Source code in pdb.py and bdb.py (which one are a part of python > stdlib) gave me the answer why it happens. > > The root cause are inside Bdb.format_stack_entry() + Bdb.canonic() > > Please take a look at the following line inside 'format_stack_entry' > method: > > filename = self.canonic(frame.f_code.co_filename) > > For zipped module variable 'frame.f_code.co_filename' holds _relative_ > file path started from the root of zip archive like > 'subdir/test_module.py' > And as result Bdb.canonic() method gives what we have - > 'full-path-to-current-dir/subdir/test_module.py' > --- > So my final question. > Could anyone confirm that it is a bug in python core subsystem which > one is responsible for loading zipped modules, > or something is wrong with my zipped module ? > > If it is a bug could you please submit it into official python bugtracker ? > I never did it before and afraid to do something wrong. > > Thanks, > Vitaly Murashev > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130916/5a2114c0/attachment.html> From techtonik at gmail.com Mon Sep 16 08:28:28 2013 From: techtonik at gmail.com (anatoly techtonik) Date: Mon, 16 Sep 2013 09:28:28 +0300 Subject: [Python-Dev] TransformDict (PEP 455) Naming Message-ID: <CAPkN8xJzJK2o=5VQ=1X2gvm7u1FiPciDsA3LU5r_bRQkszWXPw@mail.gmail.com> Does anybody know if http://vote.python.org is already operational? I decided to start a separate thread for TransformDict name, because I want to change it. Current implementation of PEP 455 only touches dictionary keys and it is more narrow than the name suggests. I'd reserve TransformDict name for something that is used to transform some other data. For my data transformation theory I have an idea of mapping with annotated fields that is used to change the names of some source data structure to target data structure, converting types and applying custom rules on the way. This is a different, but more intuitive application of such name. Name for the PEP 455 should be less abstract, more specialized. Which transform? What is transformed? Where the transformation is taking place? How it is going to be used? Why do you need a generalized structure? In PEP 455 I see two use cases only. 1. Case-insensitive keys. dictik() - dictionary with insensitive keys 2. Identity dict. dictik() - dictionary with implicit keys The generic name for collections entry can be VagueKeysDict() or DynamicKeysDict() or DynKeyDict(). The name should tell what's going on with the type to people completely unfamiliar with concept. OrderedDict() is a good name - dict with ordered entries, NamedTuple() is so-so - tuple that has a name, but also ok. TransformDict() is just too generic and doesn't tell anything - something is transformed, or dict somehow transformed, or dict for transform. There should be a better name. -- anatoly t. From solipsis at pitrou.net Mon Sep 16 10:15:08 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 16 Sep 2013 10:15:08 +0200 Subject: [Python-Dev] PEP 428: Pathlib References: <5235BA20.8010105@stoneleaf.us> Message-ID: <20130916101508.135926de@pitrou.net> Le Sun, 15 Sep 2013 06:46:08 -0700, Ethan Furman <ethan at stoneleaf.us> a ?crit : > I see PEP 428 is both targeted at 3.4 and still in draft status. > > What remains to be done to ask for pronouncement? I think I have a couple of items left to integrate in the PEP. Mostly it needs me to take a bit of time and finalize the PEP, and then have a PEP delegate (or Guido) pronounce on it. That's unless someone else wants to add it, of course. I don't mind suggestions, as long as they don't deter from the short-/middle-term goal of getting the PEP approved :) Note that the pathlib API has to be provisional. While "OO path objects" have been a common desire for a long time, it is a much less well-paved avenue than e.g. TransformDict :) Regards Antoine. From guido at python.org Mon Sep 16 17:20:30 2013 From: guido at python.org (Guido van Rossum) Date: Mon, 16 Sep 2013 08:20:30 -0700 Subject: [Python-Dev] PEP 428: Pathlib In-Reply-To: <20130916101508.135926de@pitrou.net> References: <5235BA20.8010105@stoneleaf.us> <20130916101508.135926de@pitrou.net> Message-ID: <CAP7+vJK1EXcCoXh8vH4wNDM8-Fp==MtexJi4XXEZAxEN5+XpUQ@mail.gmail.com> I hope there is a volunteer for delegate. --Guido van Rossum (sent from Android phone) On Sep 16, 2013 1:17 AM, "Antoine Pitrou" <solipsis at pitrou.net> wrote: > Le Sun, 15 Sep 2013 06:46:08 -0700, > Ethan Furman <ethan at stoneleaf.us> a ?crit : > > I see PEP 428 is both targeted at 3.4 and still in draft status. > > > > What remains to be done to ask for pronouncement? > > I think I have a couple of items left to integrate in the PEP. > Mostly it needs me to take a bit of time and finalize the PEP, and > then have a PEP delegate (or Guido) pronounce on it. > > That's unless someone else wants to add it, of course. I don't mind > suggestions, as long as they don't deter from the short-/middle-term > goal of getting the PEP approved :) > > Note that the pathlib API has to be provisional. While "OO path > objects" have been a common desire for a long time, it is a much less > well-paved avenue than e.g. TransformDict :) > > Regards > > Antoine. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130916/0db3cc12/attachment.html> From guido at python.org Mon Sep 16 17:42:12 2013 From: guido at python.org (Guido van Rossum) Date: Mon, 16 Sep 2013 08:42:12 -0700 Subject: [Python-Dev] PEP 450 adding statistics module In-Reply-To: <20130914005921.GK16820@ando> References: <520C2E23.40405@pearwood.info> <52215BDF.2090109@pearwood.info> <20130908123706.GG26319@ando> <74b03474-ab18-40c5-8ce6-de5159d34903@email.android.com> <522CA085.6000601@stoneleaf.us> <CAP7+vJ+TTaU+ntaLT3xEU2wvFQhgP4P2-FnZUJ7V2PAAy3SMEA@mail.gmail.com> <CAP7+vJJ6QxcUwcpdt3Mq6r07Q2dGUQt-VWGdsYGLv4wrRW5dGg@mail.gmail.com> <20130914005921.GK16820@ando> Message-ID: <CAP7+vJL81VR=kdO+2rCz3Di3GVNSwhkRCXrw2HAL=7ywT42rvA@mail.gmail.com> I'm ready to accept this PEP. Because I haven't read this entire thread (and 60 messages about random diversions is really too much to try and catch up on) I'll give people 24 hours to remind me of outstanding rejections. I also haven't reviewed the code in any detail, but I believe the code review is going well, so I'm not concerned that the PEP would have to revised based on that alone. On Fri, Sep 13, 2013 at 5:59 PM, Steven D'Aprano <steve at pearwood.info>wrote: > On Sun, Sep 08, 2013 at 10:51:57AM -0700, Guido van Rossum wrote: > > Never mind, I found the patch and the issue. I really think that the > > *PEP* is ready for inclusion after the open issues are changed into > > something like Discussion or Future Work, and after adding a more > > prominent link to the issue with the patch. Then the *patch* can be > > reviewed some more until it is ready -- it looks very close already. > > I've updated the PEP as requested. Is there anything further that needs > to be done to have it approved? > > http://www.python.org/dev/peps/pep-0450/ > > > > -- > Steven > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130916/6a3a9728/attachment.html> From oscar.j.benjamin at gmail.com Mon Sep 16 18:45:45 2013 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Mon, 16 Sep 2013 17:45:45 +0100 Subject: [Python-Dev] PEP 450 adding statistics module In-Reply-To: <CAP7+vJL81VR=kdO+2rCz3Di3GVNSwhkRCXrw2HAL=7ywT42rvA@mail.gmail.com> References: <520C2E23.40405@pearwood.info> <52215BDF.2090109@pearwood.info> <20130908123706.GG26319@ando> <74b03474-ab18-40c5-8ce6-de5159d34903@email.android.com> <522CA085.6000601@stoneleaf.us> <CAP7+vJ+TTaU+ntaLT3xEU2wvFQhgP4P2-FnZUJ7V2PAAy3SMEA@mail.gmail.com> <CAP7+vJJ6QxcUwcpdt3Mq6r07Q2dGUQt-VWGdsYGLv4wrRW5dGg@mail.gmail.com> <20130914005921.GK16820@ando> <CAP7+vJL81VR=kdO+2rCz3Di3GVNSwhkRCXrw2HAL=7ywT42rvA@mail.gmail.com> Message-ID: <CAHVvXxSGBAV9GgEB2fznmA=wuAfGZBBkREgVaPJR5xiir5LmtA@mail.gmail.com> On 16 September 2013 16:42, Guido van Rossum <guido at python.org> wrote: > I'm ready to accept this PEP. Because I haven't read this entire thread (and > 60 messages about random diversions is really too much to try and catch up > on) I'll give people 24 hours to remind me of outstanding rejections. > > I also haven't reviewed the code in any detail, but I believe the code > review is going well, so I'm not concerned that the PEP would have to > revised based on that alone. I think Steven has addressed all of the issues raised. Briefly from memory: 1) There was concern about having an additional sum function. Steven has pointed out that neither of sum/fsum is accurate for all stdlib numeric types as is the intention for the statistics module. It is not possible to modify either of sum/fsum in a backward compatible way that would make them suitable here. 2) The initial names for the median functions were median.low median.high etc. This naming scheme was considered non-standard by some and has been redesigned as median_low, median_high etc. (there was also discussion about the method used to attach the names to the median function but this became irrelevant after the rename). 3) The mode function also provided an algorithm for estimating the mode of a continuous probability distribution from a sample. It was suggested that there is no uniquely good way of doing this and that it is not commonly needed. This was removed and the API for mode() was simplified (it now returns a unique mode or raises an error). 4) Some of the functions (e.g. variance) used different algorithms (and produced different results) when given an iterator instead of a collection. These are now changed to always use the same algorithm and build a collection internally if necessary. 5) It was suggested that it should also be possible to compute the mean of e.g. timedelta objects but it was pointed out that they can be converted to numbers with the timedelta.total_seconds() method. 6) I raised an issue about the way the sum function behaved for decimals but this was changed in a subsequent patch presenting a new sum function that isn't susceptible to accumulated rounding errors with Decimals. Oscar From v+python at g.nevcal.com Mon Sep 16 17:31:58 2013 From: v+python at g.nevcal.com (Glenn Linderman) Date: Mon, 16 Sep 2013 08:31:58 -0700 Subject: [Python-Dev] TransformDict (PEP 455) Naming In-Reply-To: <CAPkN8xJzJK2o=5VQ=1X2gvm7u1FiPciDsA3LU5r_bRQkszWXPw@mail.gmail.com> References: <CAPkN8xJzJK2o=5VQ=1X2gvm7u1FiPciDsA3LU5r_bRQkszWXPw@mail.gmail.com> Message-ID: <5237246E.6060400@g.nevcal.com> On 9/15/2013 11:28 PM, anatoly techtonik wrote: > Does anybody know if http://vote.python.org is already operational? > > I decided to start a separate thread for TransformDict name, because I > want to change it. > Current implementation of PEP 455 only touches dictionary keys and it > is more narrow than the name suggests. I'd reserve TransformDict name > for something that is used to transform some other data. For my data > transformation theory I have an idea of mapping with annotated fields > that is used to change the names of some source data structure to > target data structure, converting types and applying custom rules on > the way. This is a different, but more intuitive application of such > name. The multitude of data transformations that are possible are certainly broader than the scope of TransformDict. However, such transformations have little to do with the operation of a dict ... the key characteristic of a dict is accessing data by key value, and the idea of transformation for a dict is easily understood to be a transformation of that access pattern, rather than a rich transformation of the data. Rich data transformations may be useful, and if possible to abstract a large number of useful data transformations into an API that would become popular, it would seem that such transformations would want to be applied not only to dict, but also to list and other data structures. It would be more of an object-to-object mapping, independent of the container that might hold the object during part of its lifetime. Hence, it seems unlikely to me that "dict" would be part of the name or requirements for such rich data transformations, leaving TransformDict available to be used for exactly what PEP 455 proposes. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130916/2e316906/attachment.html> From cf.natali at gmail.com Mon Sep 16 19:06:37 2013 From: cf.natali at gmail.com (=?ISO-8859-1?Q?Charles=2DFran=E7ois_Natali?=) Date: Mon, 16 Sep 2013 19:06:37 +0200 Subject: [Python-Dev] PEP 428: Pathlib In-Reply-To: <20130916101508.135926de@pitrou.net> References: <5235BA20.8010105@stoneleaf.us> <20130916101508.135926de@pitrou.net> Message-ID: <CAH_1eM1uo0W29xy-pJPOpe-Orfc830M3BUd3coBO1dDdHSxekg@mail.gmail.com> 2013/9/16 Antoine Pitrou <solipsis at pitrou.net>: > Le Sun, 15 Sep 2013 06:46:08 -0700, > Ethan Furman <ethan at stoneleaf.us> a ?crit : >> I see PEP 428 is both targeted at 3.4 and still in draft status. >> >> What remains to be done to ask for pronouncement? > > I think I have a couple of items left to integrate in the PEP. > Mostly it needs me to take a bit of time and finalize the PEP, and > then have a PEP delegate (or Guido) pronounce on it. IIRC, during the last discussion round, we were still debating between implicit stat() result caching - which requires an explicit restat() method - vs a mapping between the stat() method and a stat() syscall. What was the conclusion? From rowen at uw.edu Mon Sep 16 21:31:10 2013 From: rowen at uw.edu (Russell E. Owen) Date: Mon, 16 Sep 2013 12:31:10 -0700 Subject: [Python-Dev] Compiler for the Mac OS X version of Python 3.4 References: <70C99F87-E9A5-4838-A1E9-4739FBF2E6A5@gmail.com> <nad-EC5D57.13320314092013@news.gmane.org> <B3293155-E4D5-4389-A555-C31BC49CE539@gmail.com> Message-ID: <rowen-8E335B.12310916092013@news.gmane.org> In article <B3293155-E4D5-4389-A555-C31BC49CE539 at gmail.com>, Raymond Hettinger <raymond.hettinger at gmail.com> wrote: > On Sep 14, 2013, at 1:32 PM, Ned Deily <nad at acm.org> wrote: > > The > > most recent Developer Tools for 10.8 and 10.7 systems, Xcode 4.6.x, have > > a mature clang but do not provide a 10.6 SDK. Even with using an SDK, > > it's still possible to end up inadvertently linking with the wrong > > versions of system libraries. We have been burned by that in the past. > > I think we should offer a separate Mac build just for 10.6 > (much like we do for the 32-bit PPC option for 10.5). If Apple drops support for gcc in 10.9 I guess we have to go this route, but please be careful. Every time you add a new version of python for MacOS X it means that folks providing binary installers (e.g. for numpy) have to provide another binary, and folks using those installers have another chance of picking the wrong one. If you do make a 10.6-only installer, what is the minimum version of MacOS X the modern compiler would support? 10.7 gives a more measured upgrade path, but 10.8 gives a better compiler. -- Russell From solipsis at pitrou.net Mon Sep 16 21:45:36 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 16 Sep 2013 21:45:36 +0200 Subject: [Python-Dev] PEP 428: Pathlib -> stat caching In-Reply-To: <CAH_1eM1uo0W29xy-pJPOpe-Orfc830M3BUd3coBO1dDdHSxekg@mail.gmail.com> References: <5235BA20.8010105@stoneleaf.us> <20130916101508.135926de@pitrou.net> <CAH_1eM1uo0W29xy-pJPOpe-Orfc830M3BUd3coBO1dDdHSxekg@mail.gmail.com> Message-ID: <20130916214536.55907f19@fsol> On Mon, 16 Sep 2013 19:06:37 +0200 Charles-Fran?ois Natali <cf.natali at gmail.com> wrote: > 2013/9/16 Antoine Pitrou <solipsis at pitrou.net>: > > Le Sun, 15 Sep 2013 06:46:08 -0700, > > Ethan Furman <ethan at stoneleaf.us> a ?crit : > >> I see PEP 428 is both targeted at 3.4 and still in draft status. > >> > >> What remains to be done to ask for pronouncement? > > > > I think I have a couple of items left to integrate in the PEP. > > Mostly it needs me to take a bit of time and finalize the PEP, and > > then have a PEP delegate (or Guido) pronounce on it. > > IIRC, during the last discussion round, we were still debating between > implicit stat() result caching - which requires an explicit restat() > method - vs a mapping between the stat() method and a stat() syscall. > > What was the conclusion? No definite conclusion. You and Nick liked the idea of a rich stat object (returned by os.stat()) with is_dir() methods and the like: https://mail.python.org/pipermail/python-dev/2013-May/125809.html However, nothing was done about that since then ;-) There was also the scandir() proposal to return rich objects with optional stat-like fields, but similarly it didn't get a conclusion: https://mail.python.org/pipermail/python-dev/2013-May/126119.html So I would like to propose the following API change: - Path.stat() (and stat-accessing methods such as get_mtime()...) returns an uncached stat object by default - Path.cache_stat() can be called to return the stat() *and* cache it for future use, such that any future call to stat(), cache_stat() or a stat-accessing function reuses that cached stat In other words, only if you use cache_stat() at least once is the stat() value cached and reused by the Path object. (also, it's a per-Path decision) Regards Antoine. From brett at python.org Mon Sep 16 21:48:54 2013 From: brett at python.org (Brett Cannon) Date: Mon, 16 Sep 2013 15:48:54 -0400 Subject: [Python-Dev] PEP 428: Pathlib -> stat caching In-Reply-To: <20130916214536.55907f19@fsol> References: <5235BA20.8010105@stoneleaf.us> <20130916101508.135926de@pitrou.net> <CAH_1eM1uo0W29xy-pJPOpe-Orfc830M3BUd3coBO1dDdHSxekg@mail.gmail.com> <20130916214536.55907f19@fsol> Message-ID: <CAP1=2W5yq_t2iMiuptkW5w7x7kiowVCFXM4qQUEHug1QVAx6ug@mail.gmail.com> On Mon, Sep 16, 2013 at 3:45 PM, Antoine Pitrou <solipsis at pitrou.net> wrote: > On Mon, 16 Sep 2013 19:06:37 +0200 > Charles-Fran?ois Natali <cf.natali at gmail.com> wrote: > > 2013/9/16 Antoine Pitrou <solipsis at pitrou.net>: > > > Le Sun, 15 Sep 2013 06:46:08 -0700, > > > Ethan Furman <ethan at stoneleaf.us> a ?crit : > > >> I see PEP 428 is both targeted at 3.4 and still in draft status. > > >> > > >> What remains to be done to ask for pronouncement? > > > > > > I think I have a couple of items left to integrate in the PEP. > > > Mostly it needs me to take a bit of time and finalize the PEP, and > > > then have a PEP delegate (or Guido) pronounce on it. > > > > IIRC, during the last discussion round, we were still debating between > > implicit stat() result caching - which requires an explicit restat() > > method - vs a mapping between the stat() method and a stat() syscall. > > > > What was the conclusion? > > No definite conclusion. You and Nick liked the idea of a rich stat > object (returned by os.stat()) with is_dir() methods and the like: > https://mail.python.org/pipermail/python-dev/2013-May/125809.html > > However, nothing was done about that since then ;-) > > There was also the scandir() proposal to return rich objects with > optional stat-like fields, but similarly it didn't get a conclusion: > https://mail.python.org/pipermail/python-dev/2013-May/126119.html > > So I would like to propose the following API change: > > - Path.stat() (and stat-accessing methods such as get_mtime()...) > returns an uncached stat object by default > > - Path.cache_stat() can be called to return the stat() *and* cache it > for future use, such that any future call to stat(), cache_stat() or > a stat-accessing function reuses that cached stat > > In other words, only if you use cache_stat() at least once is the > stat() value cached and reused by the Path object. > (also, it's a per-Path decision) > Any reason why stat() can't get a keyword-only cached=True argument instead? Or have stat() never cache() but stat_cache() always so that people can choose if they want fresh or cached based on API and not whether some library happened to make a decision for them? -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130916/7d9608fb/attachment.html> From victor.stinner at gmail.com Mon Sep 16 22:00:53 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Mon, 16 Sep 2013 22:00:53 +0200 Subject: [Python-Dev] PEP 428: Pathlib -> stat caching In-Reply-To: <CAP1=2W5yq_t2iMiuptkW5w7x7kiowVCFXM4qQUEHug1QVAx6ug@mail.gmail.com> References: <5235BA20.8010105@stoneleaf.us> <20130916101508.135926de@pitrou.net> <CAH_1eM1uo0W29xy-pJPOpe-Orfc830M3BUd3coBO1dDdHSxekg@mail.gmail.com> <20130916214536.55907f19@fsol> <CAP1=2W5yq_t2iMiuptkW5w7x7kiowVCFXM4qQUEHug1QVAx6ug@mail.gmail.com> Message-ID: <CAMpsgwZbyegAMONaXJH=Mi46SpW1X-_YTgzKS+CWThq5qccf9Q@mail.gmail.com> 2013/9/16 Brett Cannon <brett at python.org>: > Any reason why stat() can't get a keyword-only cached=True argument instead? > Or have stat() never cache() but stat_cache() always so that people can > choose if they want fresh or cached based on API and not whether some > library happened to make a decision for them? I also prefer a single function, but only if the default is cached=False. Caching by default can be surprising and unexpected. Victor From janssen at parc.com Mon Sep 16 21:56:42 2013 From: janssen at parc.com (Bill Janssen) Date: Mon, 16 Sep 2013 12:56:42 -0700 Subject: [Python-Dev] Compiler for the Mac OS X version of Python 3.4 In-Reply-To: <rowen-8E335B.12310916092013@news.gmane.org> References: <70C99F87-E9A5-4838-A1E9-4739FBF2E6A5@gmail.com> <nad-EC5D57.13320314092013@news.gmane.org> <B3293155-E4D5-4389-A555-C31BC49CE539@gmail.com> <rowen-8E335B.12310916092013@news.gmane.org> Message-ID: <51403.1379361402@parc.com> Russell E. Owen <rowen at uw.edu> wrote: > In article <B3293155-E4D5-4389-A555-C31BC49CE539 at gmail.com>, > Raymond Hettinger <raymond.hettinger at gmail.com> wrote: > > > On Sep 14, 2013, at 1:32 PM, Ned Deily <nad at acm.org> wrote: > > > The > > > most recent Developer Tools for 10.8 and 10.7 systems, Xcode 4.6.x, have > > > a mature clang but do not provide a 10.6 SDK. Even with using an SDK, > > > it's still possible to end up inadvertently linking with the wrong > > > versions of system libraries. We have been burned by that in the past. > > > > I think we should offer a separate Mac build just for 10.6 > > (much like we do for the 32-bit PPC option for 10.5). > > If Apple drops support for gcc in 10.9 I guess we have to go this route, Could go the Sage route -- Sage first checks for an up-to-date version of gcc, and downloads it and builds it for its own use if necessary. Bill > but please be careful. Every time you add a new version of python for > MacOS X it means that folks providing binary installers (e.g. for numpy) > have to provide another binary, and folks using those installers have > another chance of picking the wrong one. > > If you do make a 10.6-only installer, what is the minimum version of > MacOS X the modern compiler would support? 10.7 gives a more measured > upgrade path, but 10.8 gives a better compiler. > > -- Russell > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/bill%40janssen.org From solipsis at pitrou.net Mon Sep 16 22:05:46 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 16 Sep 2013 22:05:46 +0200 Subject: [Python-Dev] PEP 428: Pathlib -> stat caching In-Reply-To: <CAP1=2W5yq_t2iMiuptkW5w7x7kiowVCFXM4qQUEHug1QVAx6ug@mail.gmail.com> References: <5235BA20.8010105@stoneleaf.us> <20130916101508.135926de@pitrou.net> <CAH_1eM1uo0W29xy-pJPOpe-Orfc830M3BUd3coBO1dDdHSxekg@mail.gmail.com> <20130916214536.55907f19@fsol> <CAP1=2W5yq_t2iMiuptkW5w7x7kiowVCFXM4qQUEHug1QVAx6ug@mail.gmail.com> Message-ID: <20130916220546.341e369f@fsol> On Mon, 16 Sep 2013 15:48:54 -0400 Brett Cannon <brett at python.org> wrote: > > > > So I would like to propose the following API change: > > > > - Path.stat() (and stat-accessing methods such as get_mtime()...) > > returns an uncached stat object by default > > > > - Path.cache_stat() can be called to return the stat() *and* cache it > > for future use, such that any future call to stat(), cache_stat() or > > a stat-accessing function reuses that cached stat > > > > In other words, only if you use cache_stat() at least once is the > > stat() value cached and reused by the Path object. > > (also, it's a per-Path decision) > > > > Any reason why stat() can't get a keyword-only cached=True argument > instead? Or have stat() never cache() but stat_cache() always so that > people can choose if they want fresh or cached based on API and not whether > some library happened to make a decision for them? 1. Because you also want the helper functions (get_mtime(), etc.) to cache the value too. It's not only about stat(). 2. Because of the reverse use case where you want a library to reuse a cached value despite the library not using an explicit caching call. Basically, the rationale is: 1. Caching should be opt-in, which is what this new API achieves. 2. Once you have asked for caching, most always you also want the subsequent accesses to be cached. I realize there should be a third method clear_cache(), though ;-) Regards Antoine. From rdmurray at bitdance.com Mon Sep 16 22:14:43 2013 From: rdmurray at bitdance.com (R. David Murray) Date: Mon, 16 Sep 2013 16:14:43 -0400 Subject: [Python-Dev] PEP 428: Pathlib -> stat caching In-Reply-To: <CAP1=2W5yq_t2iMiuptkW5w7x7kiowVCFXM4qQUEHug1QVAx6ug@mail.gmail.com> References: <5235BA20.8010105@stoneleaf.us> <20130916101508.135926de@pitrou.net> <CAH_1eM1uo0W29xy-pJPOpe-Orfc830M3BUd3coBO1dDdHSxekg@mail.gmail.com> <20130916214536.55907f19@fsol> <CAP1=2W5yq_t2iMiuptkW5w7x7kiowVCFXM4qQUEHug1QVAx6ug@mail.gmail.com> Message-ID: <20130916201444.34AD7250512@webabinitio.net> On Mon, 16 Sep 2013 15:48:54 -0400, Brett Cannon <brett at python.org> wrote: > On Mon, Sep 16, 2013 at 3:45 PM, Antoine Pitrou <solipsis at pitrou.net> wrote: > > So I would like to propose the following API change: > > > > - Path.stat() (and stat-accessing methods such as get_mtime()...) > > returns an uncached stat object by default > > > > - Path.cache_stat() can be called to return the stat() *and* cache it > > for future use, such that any future call to stat(), cache_stat() or > > a stat-accessing function reuses that cached stat > > > > In other words, only if you use cache_stat() at least once is the > > stat() value cached and reused by the Path object. > > (also, it's a per-Path decision) > > > > Any reason why stat() can't get a keyword-only cached=True argument > instead? Or have stat() never cache() but stat_cache() always so that > people can choose if they want fresh or cached based on API and not whether > some library happened to make a decision for them? Well, we tend to avoid single boolean arguments in favor of differently named functions. But here is an alternate API: expose the state by having a 'cache_stat' attribute of the Path that is 'False' by default but can be set 'True'. It could also (or only?) be set via an optional constructor argument. --David From rymg19 at gmail.com Mon Sep 16 22:28:24 2013 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Mon, 16 Sep 2013 15:28:24 -0500 Subject: [Python-Dev] Compiler for the Mac OS X version of Python 3.4 In-Reply-To: <51403.1379361402@parc.com> References: <70C99F87-E9A5-4838-A1E9-4739FBF2E6A5@gmail.com> <nad-EC5D57.13320314092013@news.gmane.org> <B3293155-E4D5-4389-A555-C31BC49CE539@gmail.com> <rowen-8E335B.12310916092013@news.gmane.org> <51403.1379361402@parc.com> Message-ID: <CAO41-mMUQD=K7WbSPcHSZk6jh_K3zv=Aa-YAGL2dRrwckCqcOg@mail.gmail.com> Meh...I hate it when tools download stuff without me noticing. Honestly, a separate 10.6 build would work well. Plus, if a new Clang versions includes some awesome feature that could make Python builds better, you'd be able to take advantage of it better. On Mon, Sep 16, 2013 at 2:56 PM, Bill Janssen <janssen at parc.com> wrote: > Russell E. Owen <rowen at uw.edu> wrote: > > > In article <B3293155-E4D5-4389-A555-C31BC49CE539 at gmail.com>, > > Raymond Hettinger <raymond.hettinger at gmail.com> wrote: > > > > > On Sep 14, 2013, at 1:32 PM, Ned Deily <nad at acm.org> wrote: > > > > The > > > > most recent Developer Tools for 10.8 and 10.7 systems, Xcode 4.6.x, > have > > > > a mature clang but do not provide a 10.6 SDK. Even with using an > SDK, > > > > it's still possible to end up inadvertently linking with the wrong > > > > versions of system libraries. We have been burned by that in the > past. > > > > > > I think we should offer a separate Mac build just for 10.6 > > > (much like we do for the 32-bit PPC option for 10.5). > > > > If Apple drops support for gcc in 10.9 I guess we have to go this route, > > Could go the Sage route -- Sage first checks for an up-to-date version > of gcc, and downloads it and builds it for its own use if necessary. > > Bill > > > but please be careful. Every time you add a new version of python for > > MacOS X it means that folks providing binary installers (e.g. for numpy) > > have to provide another binary, and folks using those installers have > > another chance of picking the wrong one. > > > > If you do make a 10.6-only installer, what is the minimum version of > > MacOS X the modern compiler would support? 10.7 gives a more measured > > upgrade path, but 10.8 gives a better compiler. > > > > -- Russell > > > > _______________________________________________ > > Python-Dev mailing list > > Python-Dev at python.org > > https://mail.python.org/mailman/listinfo/python-dev > > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/bill%40janssen.org > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/rymg19%40gmail.com > -- Ryan -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130916/f2b2a677/attachment.html> From solipsis at pitrou.net Mon Sep 16 22:43:31 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 16 Sep 2013 22:43:31 +0200 Subject: [Python-Dev] PEP 428: Pathlib -> stat caching References: <5235BA20.8010105@stoneleaf.us> <20130916101508.135926de@pitrou.net> <CAH_1eM1uo0W29xy-pJPOpe-Orfc830M3BUd3coBO1dDdHSxekg@mail.gmail.com> <20130916214536.55907f19@fsol> <CAP1=2W5yq_t2iMiuptkW5w7x7kiowVCFXM4qQUEHug1QVAx6ug@mail.gmail.com> <20130916201444.34AD7250512@webabinitio.net> Message-ID: <20130916224331.4a035d6c@fsol> On Mon, 16 Sep 2013 16:14:43 -0400 "R. David Murray" <rdmurray at bitdance.com> wrote: > On Mon, 16 Sep 2013 15:48:54 -0400, Brett Cannon <brett at python.org> wrote: > > On Mon, Sep 16, 2013 at 3:45 PM, Antoine Pitrou <solipsis at pitrou.net> wrote: > > > So I would like to propose the following API change: > > > > > > - Path.stat() (and stat-accessing methods such as get_mtime()...) > > > returns an uncached stat object by default > > > > > > - Path.cache_stat() can be called to return the stat() *and* cache it > > > for future use, such that any future call to stat(), cache_stat() or > > > a stat-accessing function reuses that cached stat > > > > > > In other words, only if you use cache_stat() at least once is the > > > stat() value cached and reused by the Path object. > > > (also, it's a per-Path decision) > > > > > > > Any reason why stat() can't get a keyword-only cached=True argument > > instead? Or have stat() never cache() but stat_cache() always so that > > people can choose if they want fresh or cached based on API and not whether > > some library happened to make a decision for them? > > Well, we tend to avoid single boolean arguments in favor of differently > named functions. > > But here is an alternate API: expose the state by having a 'cache_stat' > attribute of the Path that is 'False' by default but can be set 'True'. Thanks for the suggestion, that's a possibility too. > It could also (or only?) be set via an optional constructor argument. That's impractical if you get the Path object from a library call. Regards Antoine. From tjreedy at udel.edu Mon Sep 16 22:50:41 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 16 Sep 2013 16:50:41 -0400 Subject: [Python-Dev] PEP 428: Pathlib -> stat caching In-Reply-To: <20130916201444.34AD7250512@webabinitio.net> References: <5235BA20.8010105@stoneleaf.us> <20130916101508.135926de@pitrou.net> <CAH_1eM1uo0W29xy-pJPOpe-Orfc830M3BUd3coBO1dDdHSxekg@mail.gmail.com> <20130916214536.55907f19@fsol> <CAP1=2W5yq_t2iMiuptkW5w7x7kiowVCFXM4qQUEHug1QVAx6ug@mail.gmail.com> <20130916201444.34AD7250512@webabinitio.net> Message-ID: <l17qus$l1f$1@ger.gmane.org> On 9/16/2013 4:14 PM, R. David Murray wrote: > Well, we tend to avoid single boolean arguments in favor of differently > named functions. The stdlib has lots of boolean arguments. My impression is that they are to be avoided when they would change the return type or otherwise do something disjointly different. I do not think this would apply here. -- Terry Jan Reedy From ncoghlan at gmail.com Tue Sep 17 01:15:26 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 17 Sep 2013 09:15:26 +1000 Subject: [Python-Dev] PEP 428: Pathlib -> stat caching In-Reply-To: <20130916224331.4a035d6c@fsol> References: <5235BA20.8010105@stoneleaf.us> <20130916101508.135926de@pitrou.net> <CAH_1eM1uo0W29xy-pJPOpe-Orfc830M3BUd3coBO1dDdHSxekg@mail.gmail.com> <20130916214536.55907f19@fsol> <CAP1=2W5yq_t2iMiuptkW5w7x7kiowVCFXM4qQUEHug1QVAx6ug@mail.gmail.com> <20130916201444.34AD7250512@webabinitio.net> <20130916224331.4a035d6c@fsol> Message-ID: <CADiSq7eESmFN7_LOhdgoS0xDwDLZ40OJ5yvqog5yjf+=+-59-Q@mail.gmail.com> On 17 Sep 2013 06:45, "Antoine Pitrou" <solipsis at pitrou.net> wrote: > > On Mon, 16 Sep 2013 16:14:43 -0400 > "R. David Murray" <rdmurray at bitdance.com> wrote: > > On Mon, 16 Sep 2013 15:48:54 -0400, Brett Cannon <brett at python.org> wrote: > > > On Mon, Sep 16, 2013 at 3:45 PM, Antoine Pitrou <solipsis at pitrou.net> wrote: > > > > So I would like to propose the following API change: > > > > > > > > - Path.stat() (and stat-accessing methods such as get_mtime()...) > > > > returns an uncached stat object by default > > > > > > > > - Path.cache_stat() can be called to return the stat() *and* cache it > > > > for future use, such that any future call to stat(), cache_stat() or > > > > a stat-accessing function reuses that cached stat > > > > > > > > In other words, only if you use cache_stat() at least once is the > > > > stat() value cached and reused by the Path object. > > > > (also, it's a per-Path decision) > > > > > > > > > > Any reason why stat() can't get a keyword-only cached=True argument > > > instead? Or have stat() never cache() but stat_cache() always so that > > > people can choose if they want fresh or cached based on API and not whether > > > some library happened to make a decision for them? > > > > Well, we tend to avoid single boolean arguments in favor of differently > > named functions. > > > > But here is an alternate API: expose the state by having a 'cache_stat' > > attribute of the Path that is 'False' by default but can be set 'True'. > > Thanks for the suggestion, that's a possibility too. > > > It could also (or only?) be set via an optional constructor argument. > > That's impractical if you get the Path object from a library call. Given that this is a behavioural state change, I think asking for a possibly *new* path with caching enabled in that case would be a good way to go. If we treat path objects as effectively immutable (aside from the optional internal stat cache), then checking in __new__ if a passed in path object already has the appropriate caching status and returning it directly if so, but otherwise creating a new path object with the cache setting changed would avoid having libraries potentially alter the behaviour of applications' path objects and vice-versa. In effect, the unique "identity" of a path would be a triple representing the type, the filesystem path and whether or not it cached stat results internally. If you wanted to change any of those, you would have to create a new object. Cheers, Nick. > > Regards > > Antoine. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130917/1bf099a6/attachment.html> From victor.stinner at gmail.com Tue Sep 17 01:47:24 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Tue, 17 Sep 2013 01:47:24 +0200 Subject: [Python-Dev] PEP 454: add a new tracemalloc module (second round) Message-ID: <CAMpsgwaEuVQYt1HfLUyvN2Bjgj=116gxWR39kAQgKJPtbSVDwA@mail.gmail.com> Hi, Thanks to the early remarks on the PEP 454, I redesigned and enhanced the API of the new tracemalloc module. Changes: * it is now possibility to record more than 1 frame per memory allocation * add filters on filename and line number * new GroupedStats and StatsDiff class to generate and compare statistics * cumulative statistics * display the traceback of a memory block * almost all the code has unit tests * better documentation HTML version of the documentation: http://www.haypocalc.com/tmp/tracemalloc/library/tracemalloc.html The documentation contains output examples and a short tutorial ("Usage"), but also the documentation of the command line, which are not included in the PEP. HTML version of the PEP: http://www.python.org/dev/peps/pep-0454/ Issue tracking the implementation: http://bugs.python.org/issue18874 PEP: 454 Title: Add a new tracemalloc module to trace Python memory allocations Version: $Revision$ Last-Modified: $Date$ Author: Victor Stinner <victor.stinner at gmail.com> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 3-September-2013 Python-Version: 3.4 Abstract ======== Add a new ``tracemalloc`` module to trace memory blocks allocated by Python. Rationale ========= Common debug tools tracing memory allocations read the C filename and number. Using such tool to analyze Python memory allocations does not help because most memory block are allocated in the same C function, in ``PyMem_Malloc()`` for example. There are debug tools dedicated to the Python language like ``Heapy`` and ``PySizer``. These projects analyze objects type and/or content. These tools are useful when most memory leaks are instances of the same type and this type is only instancied in a few functions. The problem is when the object type is very common like ``str`` or ``tuple``, and it is hard to identify where these objects are instancied. Finding reference cycles is also a difficult problem. There are different tools to draw a diagram of all references. These tools cannot be used on large applications with thousands of objects because the diagram is too huge to be analyzed manually. Proposal ======== Using the PEP 445, it becomes easy to setup an hook on Python memory allocators. The hook can inspect the current Python frame to get the Python filename and line number. This PEP proposes to add a new ``tracemalloc`` module. It is a debug tool to trace memory allocations made by Python. The module provides the following information: * Compute the differences between two snapshots to detect memory leaks * Statistics on allocated memory blocks per filename and per line number: total size, number and average size of allocated memory blocks * For each allocated memory block: its size and the traceback where the block was allocated The API of the tracemalloc module is similar to the API of the faulthandler module: ``enable()``, ``disable()`` and ``is_enabled()`` functions, an environment variable (``PYTHONFAULTHANDLER`` and ``PYTHONTRACEMALLOC``), a ``-X`` command line option (``-X faulthandler`` and ``-X tracemalloc``). See the `documentation of the faulthandler module <http://docs.python.org/dev/library/faulthandler.html>`_. The tracemalloc module has been written for CPython. Other implementations of Python may not provide it. API === To trace most memory blocks allocated by Python, the module should be enabled as early as possible by calling ``tracemalloc.enable()`` function, by setting the ``PYTHONTRACEMALLOC`` environment variable to ``1``, or by using ``-X tracemalloc`` command line option. By default, the ``Trace.traceback`` attribute only stores one ``Frame`` instance per allocated memory block. Use ``set_traceback_limit()`` to store more frames. Functions --------- ``add_filter(filter)`` function: Add a new filter on Python memory allocations, *filter* is a ``Filter`` instance. All inclusive filters are applied at once, a memory allocation is only ignored if no inclusive filter match its trace. A memory allocation is ignored if at least one exclusive filter matchs its trace. The new filter is not applied on already collected traces. Use ``clear_traces()`` to ensure that all traces match the new filter. ``add_include_filter(filename: str, lineno: int=None, traceback: bool=False)`` function: Add an inclusive filter: helper for ``add_filter()`` creating a ``Filter`` instance with ``include`` attribute set to ``True``. Example: ``tracemalloc.add_include_filter(tracemalloc.__file__)`` only includes memory blocks allocated by the ``tracemalloc`` module. ``add_exclude_filter(filename: str, lineno: int=None, traceback: bool=False)`` function: Add an exclusive filter: helper for ``add_filter()`` creating a ``Filter`` instance with ``include`` attribute set to ``False``. Example: ``tracemalloc.add_exclude_filter(tracemalloc.__file__)`` ignores memory blocks allocated by the ``tracemalloc`` module. ``clear_filters()`` function: Reset the filter list. ``clear_traces()`` function: Clear all traces and statistics on Python memory allocations, and reset the ``get_traced_memory()`` counter. ``disable()`` function: Stop tracing Python memory allocations and stop the timer started by ``start_timer()``. See also ``enable()`` and ``is_enabled()`` functions. ``enable()`` function: Start tracing Python memory allocations. See also ``disable()`` and ``is_enabled()`` functions. ``get_filters()`` function: Get the filters on Python memory allocations as list of ``Filter`` instances. ``get_traceback_limit()`` function: Get the maximum number of ``Frame`` instances stored in the ``traceback`` attribute of a ``Trace`` instance. Use ``set_traceback_limit()`` to change the limit. ``get_object_address(obj)`` function: Get the address of the memory block of the specified Python object. ``get_object_trace(obj)`` function: Get the trace of a Python object *obj* as a ``Trace`` instance. The function only returns the trace of the memory block directly holding to object. The ``size`` attribute of the trace is smaller than the total size of the object if the object is composed of more than one memory block. Return ``None`` if the ``tracemalloc`` module did not trace the allocation of the object. See also ``gc.get_referrers()`` and ``sys.getsizeof()`` functions. ``get_process_memory()`` function: Get the memory usage of the current process as a meminfo namedtuple with two attributes: * ``rss``: Resident Set Size in bytes * ``vms``: size of the virtual memory in bytes Return ``None`` if the platform is not supported. ``get_stats()`` function: Get statistics on traced Python memory blocks as a dictionary ``{filename (str): {line_number (int): stats}}`` where *stats* in a ``TraceStats`` instance, *filename* and *line_number* can be ``None``. Return an empty dictionary if the ``tracemalloc`` module is disabled. ``get_traced_memory()`` function: Get the total size of all traced memory blocks allocated by Python. ``get_tracemalloc_size()`` function: Get the memory usage in bytes of the ``tracemalloc`` module. ``get_traces(obj)`` function: Get all traces of Python memory allocations as a dictionary ``{address (int): trace}`` where *trace* is a ``Trace`` instance. Return an empty dictionary if the ``tracemalloc`` module is disabled. ``is_enabled()`` function: ``True`` if the ``tracemalloc`` module is tracing Python memory allocations, ``False`` otherwise. See also ``enable()`` and ``disable()`` functions. ``start_timer(delay: int, func: callable, args: tuple=(), kwargs: dict={})`` function: Start a timer calling ``func(*args, **kwargs)`` every *delay* seconds. Enable the ``tracemalloc`` module if it is disabled. The timer is based on the Python memory allocator, it is not real time. *func* is called after at least *delay* seconds, it is not called exactly after *delay* seconds if no Python memory allocation occurred. The timer has a resolution of 1 second. If the ``start_timer()`` function is called twice, previous parameters are replaced. Call the ``stop_timer()`` function to stop the timer. The ``DisplayTopTask.start()`` and ``TakeSnapshot.start()`` methods use the ``start_timer()`` function to run regulary a task. ``set_traceback_limit(limit: int)`` function: Set the maximum number of ``Frame`` instances stored in the ``traceback`` attribute of a ``Trace`` instance. Clear all traces and statistics on Python memory allocations if the ``tracemalloc`` module is enabled, Storing the traceback of each memory allocation has an important overhead on the memory usage. Example with the Python test suite: tracing all memory allocations increases the memory usage by ``+50%`` when storing only 1 frame and ``+150%`` when storing 10 frames. Use ``get_tracemalloc_size()`` to measure the overhead and ``add_filter()`` to select which memory allocations are traced. Use ``get_traceback_limit()`` to get the current limit. ``stop_timer()`` function: Stop the timer started by ``start_timer()``. DisplayTop class ---------------- ``DisplayTop()`` class: Display the top of allocated memory blocks. ``display_snapshot(snapshot, count=10, group_by="filename_lineno", cumulative=False, file=None)`` method: Display a snapshot of memory blocks allocated by Python, *snapshot* is a ``Snapshot`` instance. ``display_top_diff(top_diff, count=10, file=None)`` method: Display differences between two ``GroupedStats`` instances, *top_diff* is a ``StatsDiff`` instance. ``display_top_stats(top_stats, count=10, file=None)`` method: Display the top of allocated memory blocks grouped by the ``group_by`` attribute of *top_stats*, *top_stats* is a ``GroupedStats`` instance. ``color`` attribute: If ``True``, always use colors. If ``False``, never use colors. The default value is ``None``: use colors if the *file* parameter is a TTY device. ``compare_with_previous`` attribute: If ``True`` (default value), compare with the previous snapshot. If ``False``, compare with the first snapshot. ``filename_parts`` attribute: Number of displayed filename parts (int, default: ``3``). Extra parts are replaced with ``'...'``. ``show_average`` attribute: If ``True`` (default value), display the average size of memory blocks. ``show_count`` attribute: If ``True`` (default value), display the number of allocated memory blocks. ``show_size`` attribute: If ``True`` (default value), display the size of memory blocks. DisplayTopTask class -------------------- ``DisplayTopTask(count=10, group_by="filename_lineno", cumulative=False, file=sys.stdout, user_data_callback=None)`` class: Task taking temporary snapshots and displaying the top *count* memory allocations grouped by *group_by*. Call the ``start()`` method to start the task. ``display()`` method: Take a snapshot and display the top *count* biggest allocated memory blocks grouped by *group_by* using the ``display_top`` attribute. Return the snapshot, a ``Snapshot`` instance. ``start(delay: int)`` method: Start a task using the ``start_timer()`` function calling the ``display()`` method every *delay* seconds. ``stop()`` method: Stop the task started by the ``start()`` method using the ``stop_timer()`` function. ``count`` attribute: Maximum number of displayed memory blocks. ``cumulative`` attribute: If ``True``, cumulate size and count of memory blocks of all frames of each ``Trace`` instance, not only the most recent frame. The default value is ``False``. The option is ignored if the traceback limit is ``1``, see the ``get_traceback_limit()`` function. ``display_top`` attribute: Instance of ``DisplayTop``. ``file`` attribute: The top is written into *file*. ``group_by`` attribute: Determine how memory allocations are grouped: see ``Snapshot.top_by`` for the available values. ``user_data_callback`` attribute: Optional callback collecting user data (callable, default: ``None``). See ``Snapshot.create()``. Filter class ------------ ``Filter(include: bool, pattern: str, lineno: int=None, traceback: bool=False)`` class: Filter to select which memory allocations are traced. Filters can be used to reduce the memory usage of the ``tracemalloc`` module, which can be read using ``get_tracemalloc_size()``. ``match_trace(trace)`` method: Return ``True`` if the ``Trace`` instance must be kept according to the filter, ``False`` otherwise. ``match(filename: str, lineno: int)`` method: Return ``True`` if the filename and line number must be kept according to the filter, ``False`` otherwise. ``match_filename(filename: str)`` method: Return ``True`` if the filename must be kept according to the filter, ``False`` otherwise. ``match_lineno(lineno: int)`` method: Return ``True`` if the line number must be kept according to the filter, ``False`` otherwise. ``include`` attribute: If *include* is ``True``, only trace memory blocks allocated in a file with a name matching filename ``pattern`` at line number ``lineno``. If *include* is ``False``, ignore memory blocks allocated in a file with a name matching filename :attr`pattern` at line number ``lineno``. ``pattern`` attribute: The filename *pattern* can contain one or many ``*`` joker characters which match any substring, including an empty string. The ``.pyc`` and ``.pyo`` suffixes are replaced with ``.py``. On Windows, the comparison is case insensitive and the alternative separator ``/`` is replaced with the standard separator ``\``. ``lineno`` attribute: Line number (``int``). If is is ``None`` or lesser than ``1``, it matches any line number. ``traceback`` attribute: If *traceback* is ``True``, all frames of the ``traceback`` attribute of ``Trace`` instances are checked. If *traceback* is ``False``, only the most recent frame is checked. This attribute only has an effect on the ``match_trace()`` method and only if the traceback limit is greater than ``1``. See the ``get_traceback_limit()`` function. Frame class ----------- ``Frame`` class: Trace of a Python frame, used by ``Trace.traceback`` attribute. ``filename`` attribute: Python filename, ``None`` if unknown. ``lineno`` attribute: Python line number, ``None`` if unknown. GroupedStats class ------------------ ``GroupedStats(stats: dict, group_by: str, cumulative=False, timestamp=None, process_memory=None, tracemalloc_size=None)`` class: Top of allocated memory blocks grouped by on *group_by* as a dictionary. The ``Snapshot.top_by()`` method creates a ``GroupedStats`` instance. ``compare_to(old_stats: GroupedStats=None)`` method: Compare to an older ``GroupedStats`` instance. Return a ``StatsDiff`` instance. ``cumulative`` attribute: If ``True``, cumulate size and count of memory blocks of all frames of ``Trace``, not only the most recent frame. ``group_by`` attribute: Determine how memory allocations were grouped. The type of ``stats`` keys depends on *group_by*: ===================== ======================== ============== group_by description key type ===================== ======================== ============== ``'filename'`` filename ``str`` ``'filename_lineno'`` filename and line number ``(str, str)`` ``'address'`` memory block address ``int`` ===================== ======================== ============== See the *group_by* parameter of the ``Snapshot.top_by()`` method. ``stats`` attribute: Dictionary ``{key: stats}`` where the *key* type depends on the ``group_by`` attribute and *stats* type is ``TraceStats``. ``process_memory`` attribute: Result of the ``get_process_memory()`` function, can be ``None``. ``timestamp`` attribute: Creation date and time of the snapshot, ``datetime.datetime`` instance. ``tracemalloc_size`` attribute: The memory usage in bytes of the ``tracemalloc`` module, result of the ``get_tracemalloc_size()`` function. Snapshot class -------------- ``Snapshot`` class: Snapshot of memory blocks allocated by Python. Use ``TakeSnapshot`` to take regulary snapshots. ``apply_filters(filters)`` method: Apply a list filters on the ``traces`` and ``stats`` dictionaries, *filters* is a list of ``Filter`` instances. ``create(\*, with_traces=False, with_stats=True, user_data_callback=None)`` classmethod: Take a snapshot of traces and/or statistics of allocated memory blocks. If *with_traces* is ``True``, ``get_traces()`` is called and its result is stored in the ``traces`` attribute. This attribute contains more information than ``stats`` and uses more memory and more disk space. If *with_traces* is ``False``, ``traces`` is set to ``None``. If *with_stats* is ``True``, ``get_stats()`` is called and its result is stored in the ``Snapshot.stats`` attribute. If *with_stats* is ``False``, ``Snapshot.stats`` is set to ``None``. *with_traces* and *with_stats* cannot be ``False`` at the same time. *user_data_callback* is an optional callable object. Its result should be serializable by the ``pickle`` module, or ``Snapshot.write()`` would fail. If *user_data_callback* is set, it is called and the result is stored in the ``Snapshot.user_data`` attribute. Otherwise, ``Snapshot.user_data`` is set to ``None``. The ``tracemalloc`` module must be enabled to take a snapshot. See the ``enable()`` function. ``load(filename)`` classmethod: Load a snapshot from a file. ``top_by(group_by: str, cumulative: bool=False)`` method: Compute top statistics grouped by *group_by* as a ``GroupedStats`` instance: ===================== ======================== ============== group_by description key type ===================== ======================== ============== ``'filename'`` filename ``str`` ``'filename_lineno'`` filename and line number ``(str, str)`` ``'address'`` memory block address ``int`` ===================== ======================== ============== If *cumulative* is ``True``, cumulate size and count of memory blocks of all frames of each ``Trace`` instance, not only the most recent frame. The *cumulative* parameter is ignored if *group_by* is ``'address'`` or if the traceback limit is ``1``. See the ``traceback_limit`` attribute. ``write(filename)`` method: Write the snapshot into a file. ``pid`` attribute: Identifier of the process which created the snapshot, result of ``os.getpid()``. ``process_memory`` attribute: Memory usage of the current process, result of the ``get_process_memory()`` function. It can be ``None``. ``stats`` attribute: Statistics on traced Python memory, result of the ``get_stats()`` function, if ``create()`` was called with *with_stats* equals to ``True``, ``None`` otherwise. ``tracemalloc_size`` attribute: The memory usage in bytes of the ``tracemalloc`` module, result of the ``get_tracemalloc_size()`` function. ``traceback_limit`` attribute: The maximum number of frames stored in the ``traceback`` attribute of a ``Trace``, result of the ``get_traceback_limit()`` function. ``traces`` attribute: Traces of Python memory allocations, result of the ``get_traces()`` function, if ``create()`` was called with *with_traces* equals to ``True``, ``None`` otherwise. The ``traceback`` attribute of each ``Trace`` instance is limited to ``traceback_limit`` frames. ``timestamp`` attribute: Creation date and time of the snapshot, ``datetime.datetime`` instance. ``user_data`` attribute: Result of *user_data_callback* called in ``Snapshot.create()`` (default: ``None``). StatsDiff class --------------- ``StatsDiff(differences, old_stats, new_stats)`` class: Differences between two ``GroupedStats`` instances. By default, the ``differences`` list is unsorted: call ``sort()`` to sort it. The ``GroupedStats.compare_to()`` method creates a ``StatsDiff`` instance. ``sort()`` method: Sort the ``differences`` list from the biggest allocation to the smallest. Sort by *size_diff*, *size*, *count_diff*, *count* and then by *key*. ``differences`` attribute: Differences between ``old_stats`` and ``new_stats`` as a list of ``(size_diff, size, count_diff, count, key)`` tuples. *size_diff*, *size*, *count_diff* and *count* are ``int``. The key type depends on the ``group_by`` attribute of ``new_stats``: ===================== ======================== ============== group_by description key type ===================== ======================== ============== ``'filename'`` filename ``str`` ``'filename_lineno'`` filename and line number ``(str, str)`` ``'address'`` memory block address ``int`` ===================== ======================== ============== See the ``group_by`` attribute of the ``GroupedStats`` class. ``old_stats`` attribute: Old ``GroupedStats`` instance, can be ``None``. ``new_stats`` attribute: New ``GroupedStats`` instance. Trace class ----------- ``Trace`` class: Debug information of a memory block allocated by Python. ``size`` attribute: Size in bytes of the memory block. ``traceback`` attribute: Traceback where the memory block was allocated as a list of ``Frame`` instances, most recent first. The list can be empty or incomplete if the ``tracemalloc`` module was unable to retrieve the full traceback. The traceback is limited to ``get_traceback_limit()`` frames. Use ``set_traceback_limit()`` to store more frames. TraceStats class ---------------- ``TraceStats`` class: Statistics on Python memory allocations. ``size`` attribute: Total size in bytes of allocated memory blocks. ``count`` attribute: Number of allocated memory blocks. Links ===== tracemalloc: * `#18874: Add a new tracemalloc module to trace Python memory allocations <http://bugs.python.org/issue18874>`_ * `pytracemalloc on PyPI <https://pypi.python.org/pypi/pytracemalloc>`_ Similar projects: * `Meliae: Python Memory Usage Analyzer <https://pypi.python.org/pypi/meliae>`_ * `Guppy-PE: umbrella package combining Heapy and GSL <http://guppy-pe.sourceforge.net/>`_ * `PySizer <http://pysizer.8325.org/>`_: developed for Python 2.4 * `memory_profiler <https://pypi.python.org/pypi/memory_profiler>`_ * `pympler <http://code.google.com/p/pympler/>`_ * `Dozer <https://pypi.python.org/pypi/Dozer>`_: WSGI Middleware version of the CherryPy memory leak debugger * `objgraph <http://mg.pov.lt/objgraph/>`_ * `caulk <https://github.com/smartfile/caulk/>`_ Copyright ========= This document has been placed into the public domain. Victor From steve at pearwood.info Tue Sep 17 01:59:13 2013 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 17 Sep 2013 09:59:13 +1000 Subject: [Python-Dev] PEP 450 adding statistics module In-Reply-To: <CAP7+vJL81VR=kdO+2rCz3Di3GVNSwhkRCXrw2HAL=7ywT42rvA@mail.gmail.com> References: <520C2E23.40405@pearwood.info> <52215BDF.2090109@pearwood.info> <20130908123706.GG26319@ando> <74b03474-ab18-40c5-8ce6-de5159d34903@email.android.com> <522CA085.6000601@stoneleaf.us> <CAP7+vJ+TTaU+ntaLT3xEU2wvFQhgP4P2-FnZUJ7V2PAAy3SMEA@mail.gmail.com> <CAP7+vJJ6QxcUwcpdt3Mq6r07Q2dGUQt-VWGdsYGLv4wrRW5dGg@mail.gmail.com> <20130914005921.GK16820@ando> <CAP7+vJL81VR=kdO+2rCz3Di3GVNSwhkRCXrw2HAL=7ywT42rvA@mail.gmail.com> Message-ID: <20130916235912.GG19939@ando> On Mon, Sep 16, 2013 at 08:42:12AM -0700, Guido van Rossum wrote: > I'm ready to accept this PEP. Because I haven't read this entire thread > (and 60 messages about random diversions is really too much to try and > catch up on) I'll give people 24 hours to remind me of outstanding > rejections. > > I also haven't reviewed the code in any detail, but I believe the code > review is going well, so I'm not concerned that the PEP would have to > revised based on that alone. There are a couple of outstanding issues that I am aware of, but I don't believe that either of these affect acceptance/rejection of the PEP. Please correct me if I am wrong. 1) Implementation details of the statistics.sum function. Oscar is giving me a lot of very valuable assistance speeding up the implementation of sum. 2) The current implementation has extensive docstrings, but will also need a separate statistics.rst file. I don't recall any other outstanding issues, if I have forgotten any, please remind me. > On Fri, Sep 13, 2013 at 5:59 PM, Steven D'Aprano <steve at pearwood.info>wrote: > > > On Sun, Sep 08, 2013 at 10:51:57AM -0700, Guido van Rossum wrote: > > > Never mind, I found the patch and the issue. I really think that the > > > *PEP* is ready for inclusion after the open issues are changed into > > > something like Discussion or Future Work, and after adding a more > > > prominent link to the issue with the patch. Then the *patch* can be > > > reviewed some more until it is ready -- it looks very close already. > > > > I've updated the PEP as requested. Is there anything further that needs > > to be done to have it approved? > > > > http://www.python.org/dev/peps/pep-0450/ > > > > > > > > -- > > Steven > > _______________________________________________ > > Python-Dev mailing list > > Python-Dev at python.org > > https://mail.python.org/mailman/listinfo/python-dev > > Unsubscribe: > > https://mail.python.org/mailman/options/python-dev/guido%40python.org > > > > > > -- > --Guido van Rossum (python.org/~guido) -- Steven From guido at python.org Tue Sep 17 02:06:55 2013 From: guido at python.org (Guido van Rossum) Date: Mon, 16 Sep 2013 17:06:55 -0700 Subject: [Python-Dev] PEP 450 adding statistics module In-Reply-To: <20130916235912.GG19939@ando> References: <520C2E23.40405@pearwood.info> <52215BDF.2090109@pearwood.info> <20130908123706.GG26319@ando> <74b03474-ab18-40c5-8ce6-de5159d34903@email.android.com> <522CA085.6000601@stoneleaf.us> <CAP7+vJ+TTaU+ntaLT3xEU2wvFQhgP4P2-FnZUJ7V2PAAy3SMEA@mail.gmail.com> <CAP7+vJJ6QxcUwcpdt3Mq6r07Q2dGUQt-VWGdsYGLv4wrRW5dGg@mail.gmail.com> <20130914005921.GK16820@ando> <CAP7+vJL81VR=kdO+2rCz3Di3GVNSwhkRCXrw2HAL=7ywT42rvA@mail.gmail.com> <20130916235912.GG19939@ando> Message-ID: <CAP7+vJKPDCZd4ZQn6mSkiAx7=wjv7WmmP3HjbOj-FLGUY-yNRg@mail.gmail.com> On Mon, Sep 16, 2013 at 4:59 PM, Steven D'Aprano <steve at pearwood.info>wrote: > On Mon, Sep 16, 2013 at 08:42:12AM -0700, Guido van Rossum wrote: > > I'm ready to accept this PEP. Because I haven't read this entire thread > > (and 60 messages about random diversions is really too much to try and > > catch up on) I'll give people 24 hours to remind me of outstanding > > rejections. > > > > I also haven't reviewed the code in any detail, but I believe the code > > review is going well, so I'm not concerned that the PEP would have to > > revised based on that alone. > > There are a couple of outstanding issues that I am aware of, but I don't > believe that either of these affect acceptance/rejection of the PEP. > Please correct me if I am wrong. > > 1) Implementation details of the statistics.sum function. Oscar is > giving me a lot of very valuable assistance speeding up the > implementation of sum. > > 2) The current implementation has extensive docstrings, but will also > need a separate statistics.rst file. > > > I don't recall any other outstanding issues, if I have forgotten any, > please remind me. > Those certainly don't stand in the way of the PEP's acceptance (but they do block the commit of the code :-). The issues that Oscar listed also all seem resolved (though they would make a nice addition to the "Discussion" section in the PEP). -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130916/0fa44e6a/attachment.html> From stephen at xemacs.org Tue Sep 17 02:34:39 2013 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Tue, 17 Sep 2013 09:34:39 +0900 Subject: [Python-Dev] PEP 428: Pathlib -> stat caching In-Reply-To: <l17qus$l1f$1@ger.gmane.org> References: <5235BA20.8010105@stoneleaf.us> <20130916101508.135926de@pitrou.net> <CAH_1eM1uo0W29xy-pJPOpe-Orfc830M3BUd3coBO1dDdHSxekg@mail.gmail.com> <20130916214536.55907f19@fsol> <CAP1=2W5yq_t2iMiuptkW5w7x7kiowVCFXM4qQUEHug1QVAx6ug@mail.gmail.com> <20130916201444.34AD7250512@webabinitio.net> <l17qus$l1f$1@ger.gmane.org> Message-ID: <87eh8o5lr4.fsf@uwakimon.sk.tsukuba.ac.jp> Terry Reedy writes: > On 9/16/2013 4:14 PM, R. David Murray wrote: > > > Well, we tend to avoid single boolean arguments in favor of differently > > named functions. > > The stdlib has lots of boolean arguments. My impression is that they are > to be avoided when they would change the return type or otherwise do > something disjointly different. I do not think this would apply here. I remember reading that the criterion is whether the argument is most often given a literal value. Then "stat_cache()" is preferable to "stat(cache=True)". OTOH, "stat(cache=want_cache)" is better than if want_cache: result = stat_cache() else: result = stat() or "result = stat_cache() if want_cache else stat()". From victor.stinner at gmail.com Tue Sep 17 12:36:03 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Tue, 17 Sep 2013 12:36:03 +0200 Subject: [Python-Dev] PEP 454: add a new tracemalloc module (second round) In-Reply-To: <CAMpsgwaEuVQYt1HfLUyvN2Bjgj=116gxWR39kAQgKJPtbSVDwA@mail.gmail.com> References: <CAMpsgwaEuVQYt1HfLUyvN2Bjgj=116gxWR39kAQgKJPtbSVDwA@mail.gmail.com> Message-ID: <CAMpsgwYL6KpL+R+FquooS+pEu_e9H-m1t40rb6DqOJmww9RerA@mail.gmail.com> 2013/9/17 Victor Stinner <victor.stinner at gmail.com>: > Issue tracking the implementation: > http://bugs.python.org/issue18874 If you want to test the implementation, you can try the following repository: http://hg.python.org/features/tracemalloc Or try the patch attached on the issue #18874 on the Python default version. Compile Python and use "-X tracemalloc" command line option to enable the module at startup. Then you can play with tracemalloc.DisplayTopTask.display() and tracemalloc.TakeSnapshot.take_snapshot(). To create Python snapshots easily, modify Lib/test/regrtest.py to use the following block: take = tracemalloc.TakeSnapshot() # tracemalloc.set_traceback_limit(15); take.with_traces = True take.filename_template = "/tmp/tracemalloc-$pid-$counter.pickle" take.start(10) And then start: ./python -X tracemalloc -m test "take.with_traces = True" is slower but required if you want to use cumulative views, show the traceback, or group traces by address. Use a lower traceback limit if the test suite is too slow. When you get a snapshot file, you can analyze it using: ./python -m tracemalloc /tmp/tracemalloc-1564-0001.pickle Add --help to see all options. I like the --traceback option. > ``get_filters()`` function: > > Get the filters on Python memory allocations as list of ``Filter`` > instances. I hesitate to add a Filters class which would contain a list of filters. The logic to check if list of filters matchs is non-trivial. You have to split inclusive and exclusive filters and take care of empty list of inclusive/exclusive filters. See the code of Snapshot.apply_filters() for example. > ``get_object_trace(obj)`` function: > > Get the trace of a Python object *obj* as a ``Trace`` instance. > > The function only returns the trace of the memory block directly > holding to object. The ``size`` attribute of the trace is smaller > than the total size of the object if the object is composed of more > than one memory block. > > Return ``None`` if the ``tracemalloc`` module did not trace the > allocation of the object. > > See also ``gc.get_referrers()`` and ``sys.getsizeof()`` functions. The function can be see of a lie because it does not count all bytes of a object (as explained in the doc above). The function should maybe be renamed to "get_trace(address)" to avoid the confusion. > DisplayTop class > ---------------- Oh, I forgot to document the new "previous_top_stats" attribute. It is used to compare two snapshots. > DisplayTopTask class > -------------------- > > ``start(delay: int)`` method: > > Start a task using the ``start_timer()`` function calling the > ``display()`` method every *delay* seconds. I should probably repeat here that only one timer can used at the same time. So only one DisplayTopTask or one TakeSnapshot instance can be used at the same time. It's a design choice to keep start_timer() simple, there is no need for a complex scheduler for such simple debug tool. You *can* run the two tasks at the same time by writing your own function: def mytask(top_task, snapshot_task): top_task.display() snapshot_task.take_snapshot() tracemalloc.start_timer(10, mytask, top_task, snapshot_task) > Snapshot class > -------------- > > ``create(\*, with_traces=False, with_stats=True, > user_data_callback=None)`` classmethod: It's the only function using keyword-only parameters. I don't know it's a good practice and should be used on other methods, or if it should be avoided? > *user_data_callback* is an optional callable object. Its result > should be serializable by the ``pickle`` module, or > ``Snapshot.write()`` would fail. If *user_data_callback* is set, it > is called and the result is stored in the ``Snapshot.user_data`` > attribute. Otherwise, ``Snapshot.user_data`` is set to ``None``. The idea is to attach arbitrary data to a snapshot. Examples: * size of Python caches: cache of linecache and re modules * size of the internal Unicode intern dict * gc.get_stats() * gc.get_count() * len(gc.get_objects()) * ("tracemalloc_size" should maybe moved to the user_data) I hesitate to use a dictionary for user_data. The problem is to decice how to display such data in DisplayTop. For example, gc.get_count() is a number whereas tracemalloc_size is size is bytes (should be formatted using kB, MB, etc. suffixes). What do you think? Victor From ncoghlan at gmail.com Tue Sep 17 16:46:01 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 18 Sep 2013 00:46:01 +1000 Subject: [Python-Dev] PEP 453: Explicit bootstrapping of pip Message-ID: <CADiSq7f2yabBwu6_esKtvcs7tnFeifM7qdZCzO5ZAuevxsjo+w@mail.gmail.com> After a couple of rounds of review on distutils-sig, and with Martin agreeing to serve as BDFL-Delegate, it's time for the pip bootstrapping proposal to run the gauntlet of python-dev :) The last round of review showed that there were a few things we were assuming people knew (based on the many, many discussions around this topic on distutils-sig, starting even before Richard wrote PEP 439), so I hope I've managed to cover those better in this version. It also goes into more details on the proposed module API for getpip and the associated API updates in venv. There are still a couple of open questions related to the Windows installers, and there may still be unasked questions affecting the Mac OS X installers. HTML version: http://www.python.org/dev/peps/pep-0453/ For those that reviewed the previous versions on distutils-sig, the latest diff is here: http://hg.python.org/peps/rev/df9e4c301415 Cheers, Nick. ============================== PEP: 453 Title: Explicit bootstrapping of pip in Python installations Version: $Revision$ Last-Modified: $Date$ Author: Donald Stufft <donald at stufft.io>, Nick Coghlan <ncoghlan at gmail.com> BDFL-Delegate: Martin von L?wis Status: Draft Type: Process Content-Type: text/x-rst Created: 10-Aug-2013 Post-History: 30-Aug-2013, 15-Sep-2013, 18-Sep-2013 Abstract ======== This PEP proposes that the `pip`_ package manager be made available by default when installing CPython and when creating virtual environments using the standard library's ``venv`` module (including via the ``pyvenv`` command line utility). To clearly demarcate development responsibilities, and to avoid inadvertently downgrading ``pip`` when updating CPython, the proposed mechanism to achieve this is to include an explicit `pip`_ bootstrapping mechanism in the standard library that is invoked automatically by the CPython installers provided on python.org. The PEP also strongly recommends that CPython redistributors and other Python implementations ensure that ``pip`` is available by default, or at the very least, explicitly document the fact that it is not included. Proposal ======== This PEP proposes the inclusion of a ``getpip`` bootstrapping module in Python 3.4, as well as in the next maintenance releases of Python 3.3 and 2.7. This PEP does *not* propose making pip (or any dependencies) part of the standard library. Instead, pip will be a bundled application provided along with CPython for the convenience of Python users, but subject to its own development life cycle and able to be upgraded independently of the core interpreter and standard library. Rationale ========= Currently, on systems without a platform package manager and repository, installing a third-party Python package into a freshly installed Python requires first identifying an appropriate package manager and then installing it. Even on systems that *do* have a platform package manager, it is unlikely to include every package that is available on the Python Package Index, and even when a desired third-party package is available, the correct name in the platform package manager may not be clear. This means that, to work effectively with the Python Package Index ecosystem, users must know which package manager to install, where to get it, and how to install it. The effect of this is that third-party Python projects are currently required to choose from a variety of undesirable alternatives: * assume the user already has a suitable cross-platform package manager installed * duplicate the instructions and tell their users how to install the package manager * completely forgo the use of dependencies to ease installation concerns for their users All of these available options have significant drawbacks. If a project simply assumes a user already has the tooling then beginning users may get a confusing error message when the installation command doesn't work. Some operating systems may ease this pain by providing a global hook that looks for commands that don't exist and suggest an OS package they can install to make the command work, but that only works on Linux systems with platform package managers. No such assistance is availabe for Windows and Mac OS X users. The challenges of dealing with this problem are a regular feature of feedback the core Python developers receive from professional educators and others introducing new users to Python. If a project chooses to duplicate the installation instructions and tell their users how to install the package manager before telling them how to install their own project then whenever these instructions need updates they need updating by every project that has duplicated them. This is particular problematic when there are multiple competing installation tools available, and different projects recommend different tools. This specific problem can be partially alleviated by strongly promoting ``pip`` as the default installer and recommending that other projects reference `pip's own bootstrapping instructions <http://www.pip-installer.org/en/latest/installing.html>`__ rather than duplicating them. However the user experience created by this approach still isn't good (especially on Windows, where downloading and running the ``get-pip.py`` bootstrap script with the default OS configuration is significantly more painful than downloading and running a binary executable or installer). The situation becomes even more complicated when multiple Python versions are involved (for example, parallel installations of Python 2 and Python 3), since that makes it harder to create and maintain good platform specific ``pip`` installers independently of the CPython installers. The projects that have decided to forgo dependencies altogether are forced to either duplicate the efforts of other projects by inventing their own solutions to problems or are required to simply include the other projects in their own source trees. Both of these options present their own problems either in duplicating maintenance work across the ecosystem or potentially leaving users vulnerable to security issues because the included code or duplicated efforts are not automatically updated when upstream releases a new version. By providing a cross-platform package manager by default it will be easier for users trying to install these third-party packages as well as easier for the people distributing them as they should now be able to safely assume that most users will have the appropriate installation tools available. This is expected to become more important in the future as the Wheel_ package format (deliberately) does not have a built in "installer" in the form of ``setup.py`` so users wishing to install from a wheel file will want an installer even in the simplest cases. Reducing the burden of actually installing a third-party package should also decrease the pressure to add every useful module to the standard library. This will allow additions to the standard library to focus more on why Python should have a particular tool out of the box instead of using the general difficulty of installing third-party packages as justification for inclusion. Providing a standard installation system also helps with bootstrapping alternate build and installer systems, such as ``setuptools``, ``zc.buildout`` and the ``hashdist``/``conda`` combination that is aimed specifically at the scientific community. So long as ``pip install <tool>`` works, then a standard Python-specific installer provides a reasonably secure, cross platform mechanism to get access to these utilities. Why pip? -------- ``pip`` has been chosen as the preferred default installer, as it addresses several design and user experience issues with its predecessor ``easy_install`` (these issues can't readily be fixed in ``easy_install`` itself due to backwards compatibility concerns). ``pip`` is also well suited to working within the bounds of a single Python runtime installation (including associated virtual environments), which is a desirable feature for a tool bundled with CPython. Other tools like ``zc.buildout`` and ``conda`` are more ambitious in their aims (and hence substantially better than ``pip`` at handling external binary dependencies), so it makes sense for the Python ecosystem to treat them more like platform package managers to interoperate with rather than as the default cross-platform installation tool. This relationship is similar to that between ``pip`` and platform package management systems like ``apt`` and ``yum`` (which are also designed to handle arbitrary binary dependencies). Explicit bootstrapping mechanism ================================ An additional module called ``getpip`` will be added to the standard library whose purpose is to install pip and any of its dependencies into the appropriate location (most commonly site-packages). It will expose a single callable named ``bootstrap()`` as well as offer direct execution via ``python -m getpip``. Options for installing it such as index server, installation location (``--user``, ``--root``, etc) will also be available to enable different installation schemes. It is believed that users will want the most recent versions available to be installed so that they can take advantage of the new advances in packaging. Since any particular version of Python has a much longer staying power than a version of pip in order to satisfy a user's desire to have the most recent version the bootstrap will (by default) contact PyPI, find the latest version, download it, and then install it. This process is security sensitive, difficult to get right, and evolves along with the rest of packaging. Instead of attempting to maintain a "mini pip" for the sole purpose of installing pip, the ``getpip`` module will, as an implementation detail, include a private copy of pip and its dependencies which will be used to discover and install pip from PyPI. It is important to stress that this private copy of pip is *only* an implementation detail and it should *not* be relied on or assumed to exist. Not all users will have network access to PyPI whenever they run the bootstrap. In order to ensure that these users will still be able to bootstrap pip the bootstrap will fallback to simply installing the included copy of pip. The pip ``--no-download`` command line option will be supported to force installation of the bundled version, without even attempting to contact PyPI. This presents a balance between giving users the latest version of pip, saving them from needing to immediately upgrade pip after bootstrapping it, and allowing the bootstrap to work offline in situations where users might already have packages downloaded that they wish to install. Proposed CLI ------------ The proposed CLI is based on a subset of the existing ``pip install`` options:: Usage: python -m getpip [options] Download Options: --no-download Install the bundled version, don't attempt to download -i, --index-url <url> Base URL of Python Package Index (default https://pypi.python.org/simple/). --proxy <proxy> Specify a proxy in the form [user:passwd@]proxy.server:port. --timeout <sec> Set the socket timeout (default 15 seconds). --cert <path> Path to alternate CA bundle. Installation Options: -U, --upgrade Upgrade pip and dependencies, even if already installed --user Install using the user scheme. --root <dir> Install everything relative to this alternate root directory. Additional options (such as verbosity and logging options) may also be supported. Proposed module API ------------------- The proposed ``getpip`` module API is a single ``bootstrap`` function with parameter names derived directly from the proposed CLI:: def bootstrap(download=True, upgrade=False, root=None, user=False, index_url=None, cert=None, proxy=None, timeout=15): """Bootstrap pip into the current Python installation (or the given root directory)""" The only changes are to replace the ``--no-download`` opt-out option with the True-by-default ``download`` option and to replace the hyphen in ``index-url`` with an underscore to create a legal Python identifier. Invocation from the CPython installers -------------------------------------- The CPython Windows and Mac OS X installers will each gain two new options: * Install pip (the default Python package management utility)? * Upgrade pip to the latest version (requires network access)? Both options will be checked by default, with the option to upgrade pip being available for selection only if the option to install pip is checked. If both options are checked, then the installer will invoke the following command with the just installed Python:: python -m getpip --upgrade If only the "Install pip" option is checked, then the following command will be invoked:: python -m getpip --upgrade --no-download This ensures that, by default, installing or updating CPython will ensure that either the latest available version of PyPI is installed (directly from PyPI if permitted, otherwise whichever is more recent out of an already installed version and the private copy inside ``getpip``) Installing from source ---------------------- While the prebuilt binary installers will be updated to run ``python -m getpip`` by default, no such change will be made to the ``make install`` and ``make altinstall`` commands of the source distribution. ``getpip`` itself will still be installed normally (as it is a regular part of the standard library), only the implicit installation of pip and its dependencies will be skipped. Keeping the pip bootstrapping as a separate step for ``make``-based installations should minimize the changes CPython redistributors need to make to their build processes. Avoiding the layer of indirection through ``make`` for the ``getpip`` invocation also ensures those installing from a custom source build can easily force an offline installation of pip, install it from a private index server, or skip installing pip entirely. Changes to virtual environments ------------------------------- Python 3.3 included a standard library approach to virtual Python environments through the ``venv`` module. Since it's release it has become clear that very few users have been willing to use this feature directly, in part due to the lack of an installer present by default inside of the virtual environment. They have instead opted to continue using the ``virtualenv`` package which *does* include pip installed by default. To make the ``venv`` more useful to users it will be modified to issue the pip bootstrap by default inside of the new environment while creating it. This will allow people the same convenience inside of the virtual environment as this PEP provides outside of it as well as bringing the ``venv`` module closer to feature parity with the external ``virtualenv`` package, making it a more suitable replacement. To handle cases where a user does not wish to have pip bootstrapped into their virtual environment a ``--without-pip`` option will be added. The ``--no-download`` option will also be supported, to force the use of the bundled ``pip`` rather than retrieving the latest version from PyPI. The ``venv.EnvBuilder`` and ``venv.create`` APIs will be updated to accept two new parameters: ``with_pip`` (defaulting to ``False``) and ``bootstrap_options`` (accepting a dictionary of keyword arguments to pass to ``getpip.bootstrap`` if ``with_pip`` is set, defaulting to ``None``). This particular change will be made only for Python 3.4 and later versions. The third-party ``virtualenv`` project will still be needed to obtain a consistent cross-version experience in Python 3.3 and 2.7. Documentation ------------- The "Installing Python Modules" section of the standard library documentation will be updated to recommend the use of the bootstrapped `pip` installer. It will give a brief description of the most common commands and options, but delegate to the externally maintained ``pip`` documentation for the full details. The existing content of the module installation guide will be retained, but under a new "Invoking distutils directly" subsection. Bundling CA certificates with CPython ------------------------------------- The reference ``getpip`` implementation includes the ``pip`` CA bundle along with the rest of pip. This means CPython effectively includes a CA bundle that is used solely for ``getpip``. This is considered desirable, as it ensures that ``pip`` will behave the same across all supported versions of Python, even those prior to Python 3.4 that cannot access the system certificate store on Windows. Automatic installation of setuptools ------------------------------------ ``pip`` currently depends on ``setuptools`` to handle metadata generation during the build process, along with some other features. While work is ongoing to reduce or eliminate this dependency, it is not clear if that work will be complete for pip 1.5 (which is the version likely to be current when Python 3.4.0 is released). This PEP proposes that, if pip still requires it as a dependency, ``getpip`` will include a private copy of ``setuptools`` (in addition to the private copy of ``pip``). In normal operation, ``python -m getpip`` will then download and install the latest version of ``setuptools`` from PyPI (as a dependency of ``pip``), while ``python -m getpip --no-download`` will install the private copy. However, this behaviour is officially considered an implementation detail. Other projects which explicitly require ``setuptools`` must still provide an appropriate dependency declaration, rather than assuming ``setuptools`` will always be installed alongside ``pip``. Once pip is able to run ``pip install --upgrade pip`` without needing ``setuptools`` installed first, then the private copy of ``setuptools`` will be removed from ``getpip``. Updating the bundled pip ------------------------ In order to keep up with evolutions in packaging as well as providing users who are using the offline installation method with as recent version a possible the ``getpip`` module will be regularly updated to the latest versions of everything it bootstraps. After each new pip release, and again during the preparation for any release of Python (including feature releases), a script, provided as part of this PEP, will be run to ensure the private copies stored in the CPython source repository have been updated to the latest versions. Updating the getpip module API and CLI -------------------------------------- Future security updates for pip and PyPI (for example, automatic verification of package signatures) may also provide desirable security enhancements for the ``getpip`` bootstrapping mechanism. It is desirable that these features be made available in standard library maintenance releases, not just new feature releases. Accordingly, a slight relaxation of the usual "no new features in maintenance releases" rule is proposed for the ``getpip`` module. This relaxation also indirectly affects the new ``bootstrap_options`` parameter in the ``venv`` module APIs. Specifically, new security related flags will be permitted, with the following restrictions: - for compatibility with third-party usage of ``getpip`` module (for example, with a private index server), any such flag must be *off* by default in maintenance releases. It *should* be switched on by default in the next feature release. - the CPython installers and the ``pyvenv`` CLI in the affected maintenance release should explicitly opt-in to the enhanced security features when automatically bootstrapping ``pip`` This means that maintenance releases of the CPython installers will benefit from security enhancements by default, while avoiding breaking customised usage of the bootstrap mechanism. Feature addition in maintenance releases ======================================== Adding a new module to the standard library in Python 2.7 and 3.3 maintenance releases breaks the usual policy of "no new features in maintenance releases". It is being proposed in this case as the current bootstrapping issues for the third-party Python package ecosystem greatly affects the experience of new users, especially on Python 2 where many Python 3 standard library improvements are available as backports on PyPI, but are not included in the Python 2 standard library. By updating Python 2.7, 3.3 and 3.4 to easily bootstrap the PyPI ecosystem, this change should aid the vast majority of current Python users, rather than only those with the freedom to adopt Python 3.4 as soon as it is released. This is also a matter of starting as we mean to continue: as noted above, ``getpip`` will have a limited permanent exemption from the "no new features in maintenance releases" restriction, as it will include (and rely on) upgraded private copies of ``pip`` and ``setuptools`` even in maintenance releases, and may offer new security related options itself. Open Question: Uninstallation ============================= No changes are currently proposed to the uninstallation process. The bootstrapped pip will be installed the same way as any other pip installed packages, and will be handled in the same way as any other post-install additions to the Python environment. At least on Windows, that means the bootstrapped files will be left behind after uninstallation, since those files won't be associated with the Python MSI installer. .. note:: Perhaps the installer should be updated to clobber everything in site-packages and the Scripts directory when uninstalled (treating them as "data directories" from Python's point of view), but I would prefer not to make this PEP conditional on that change. Open Question: Script Execution on Windows ========================================== While the Windows installer was updated in Python 3.3 to optionally make ``python`` available on the PATH, no such change was made to include the Scripts directory. This PEP proposes that this installer option be changed to also add the Scripts directory to PATH (either always, or else as a checked by default suboption). Without this change, the most reliable way to invoke pip on Windows (without tinkering manually with PATH) is actually ``py -m pip`` (or ``py -3 -m pip`` to select the Python 3 version if both Python 2 and 3 are installed) rather than simply calling ``pip``. Adding the scripts directory to the system PATH would mean that ``pip`` works reliably in the "only one Python installation on the system PATH" case, with ``py -m pip`` needed only to select a non-default version in the parallel installation case (and outside a virtual environment). While the script invocations on recent versions of Python will run through the Python launcher for Windows, this shouldn't cause any issues, as long as the Python files in the Scripts directory correctly specify a Python version in their shebang line or have an adjacent Windows executable (as ``easy_install`` and ``pip`` do). Recommendations for Downstream Distributors =========================================== A common source of Python installations are through downstream distributors such as the various Linux Distributions [#ubuntu]_ [#debian]_ [#fedora]_, OSX package managers [#homebrew]_, or Python-specific tools [#conda]_. In order to provide a consistent, user-friendly experience to all users of Python regardless of how they attained Python this PEP recommends and asks that downstream distributors: * Ensure that whenever Python is installed pip is also installed. * This may take the form of separate packages with dependencies on each other so that installing the Python package installs the pip package and installing the pip package installs the Python package. * Do not remove the bundled copy of pip. * This is required for offline installation of pip into a virtual environment by the ``venv`` module. * This is similar to the existing ``virtualenv`` package for which many downstream distributors have already made exception to the common "debundling" policy. * This does mean that if ``pip`` needs to be updated due to a security issue, so does the bundled version in the ``getpip`` bootstrap module * However, altering the bundled version of pip to remove the embedded CA certificate bundle and rely the system CA bundle instead is a reasonable change. * Migrate build systems to utilize `pip`_ and `Wheel`_ instead of directly using ``setup.py``. * This will ensure that downstream packages can more easily utilize the new metadata formats which may not have a ``setup.py``. * Ensure that all features of this PEP continue to work with any modifications made. * Online installation of the latest version of pip into a global or virtual python environment using ``python -m getpip``. * Offline installation of the bundled version of pip into a global or virtual python environment using ``python -m getpip``. * ``pip install --upgrade pip`` in a global installation should not affect any already created virtual environments. * ``pip install --upgrade pip`` in a virtual environment should not affect the global installation. In the event that a Python redistributor chooses *not* to follow these recommendations, we request that they explicitly document this fact and provide their users with suitable guidance on translating upstream ``pip`` based installation instructions into something appropriate for the platform. Other Python implementations are also encouraged to follow these guidelines where applicable. Policies & Governance ===================== The maintainers of the bootstrapped software and the CPython core team will work together in order to address the needs of both. The bootstrapped software will still remain external to CPython and this PEP does not include CPython subsuming the development responsibilities or design decisions of the bootstrapped software. This PEP aims to decrease the burden on end users wanting to use third-party packages and the decisions inside it are pragmatic ones that represent the trust that the Python community has already placed in the Python Packaging Authority as the authors and maintainers of ``pip``, ``setuptools``, PyPI, ``virtualenv`` and other related projects. Backwards Compatibility ----------------------- Except for security enhancements (as noted above), the public API of the ``getpip`` module itself will fall under the typical backwards compatibility policy of Python for its standard library. The externally developed software that this PEP bundles does not. Most importantly, this means that the bootstrapped version of pip may gain new features in CPython maintenance releases, and pip continues to operate on its own 6 month release cycle rather than CPython's 18-24 month cycle. Security Releases ----------------- Any security update that affects the ``getpip`` module will be shared prior to release with the Python Security Response Team (security at python.org). The PSRT will then decide if the reported issue warrants a security release of CPython. Appendix: Rejected Proposals ============================ Implicit bootstrap ------------------ `PEP439`_, the predecessor for this PEP, proposes its own solution. Its solution involves shipping a fake ``pip`` command that when executed would implicitly bootstrap and install pip if it does not already exist. This has been rejected because it is too "magical". It hides from the end user when exactly the pip command will be installed or that it is being installed at all. It also does not provide any recommendations or considerations towards downstream packagers who wish to manage the globally installed pip through the mechanisms typical for their system. The implicit bootstrap mechanism also ran into possible permissions issues, if a user inadvertently attempted to bootstrap pip without write access to the appropriate installation directories. Including pip directly in the standard library ---------------------------------------------- Similar to this PEP is the proposal of just including pip in the standard library. This would ensure that Python always includes pip and fixes all of the end user facing problems with not having pip present by default. This has been rejected because we've learned through the inclusion and history of ``distutils`` in the standard library that losing the ability to update the packaging tools independently can leave the tooling in a state of constant limbo. Making it unable to ever reasonably evolve in a timeframe that actually affects users as any new features will not be available to the general population for *years*. Allowing the packaging tools to progress separately from the Python release and adoption schedules allows the improvements to be used by *all* members of the Python community and not just those able to live on the bleeding edge of Python releases. There have also been issues in the past with the "dual maintenance" problem if a project continues to be maintained externally while *also* having a fork maintained in the standard library. Since external maintenance of ``pip`` will always be needed to support earlier Python versions, the proposed bootstrapping mechanism will becoming the explicit responsibility of the CPython core developers (assisted by the pip developers), while pip issues reported to the CPython tracker will be migrated to the pip issue tracker. There will no doubt still be some user confusion over which tracker to use, but hopefully less than has been seen historically when including complete public copies of third-party projects in the standard library. Finally, the approach described in this PEP avoids some technical issues related to handle CPython maintenance updates when pip has been independently updated to a more recent version. The proposed pip-based bootstrapping mechanism handles that automatically, since pip and the system installer never get into a fight about who owns the pip installation (it is always managed through pip, either directly, or indirectly via the getpip bootstrap module). Defaulting to --user installation --------------------------------- Some consideration was given to bootstrapping pip into the per-user site-packages directory by default. However, this behaviour would be surprising (as it differs from the default behaviour of pip itself) and is also not currently considered reliable (there are some edge cases which are not handled correctly when pip is installed into the user site-packages directory rather than the system site-packages). .. _Wheel: http://www.python.org/dev/peps/pep-0427/ .. _pip: http://www.pip-installer.org .. _setuptools: https://pypi.python.org/pypi/setuptools .. _PEP439: http://www.python.org/dev/peps/pep-0439/ References ========== .. [#ubuntu] `Ubuntu <http://www.ubuntu.com/>` .. [#debian] `Debian <http://www.debian.org>` .. [#fedora] `Fedora <https://fedoraproject.org/>` .. [#homebrew] `Homebrew <http://brew.sh/>` .. [#conda] `Conda <http://www.continuum.io/blog/conda>` Copyright ========= This document has been placed in the public domain. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From tjreedy at udel.edu Tue Sep 17 17:37:48 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 17 Sep 2013 11:37:48 -0400 Subject: [Python-Dev] License() release list is imcomplete; intentional? Message-ID: <l19t08$uj0$1@ger.gmane.org> On 2.7, >>> license() return a text that includes a complete list of releases from 1.6 to 2.7 and stops there Release Derived Year Owner GPL- from compatible? (1) 0.9.0 thru 1.2 1991-1995 CWI yes 1.3 thru 1.5.2 1.2 1995-1999 CNRI yes 1.6 1.5.2 2000 CNRI no 2.0 1.6 2000 BeOpen.com no ... 2.6.5 2.6.4 2010 PSF yes 2.7 2.6 2010 PSF yes Was it intentional to stop with 2.7 and not continue with 2.7.1, etc? On 3.3.2, the 2.x list ends with 2.6.5 and never mentions 2.7. Intentional? It then jumps back to 3.0 and ends with the 'previous' release, 3.3.1. Should 3.3.2 be included in the 3.3.2 list? ... 2.6.4 2.6.3 2009 PSF yes 2.6.5 2.6.4 2010 PSF yes 3.0 2.6 2008 PSF yes 3.0.1 3.0 2009 PSF yes ... 3.2.4 3.2.3 2013 PSF yes 3.3.0 3.2 2012 PSF yes 3.3.1 3.3.0 2013 PSF yes -- Terry Jan Reedy From solipsis at pitrou.net Tue Sep 17 17:47:36 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 17 Sep 2013 17:47:36 +0200 Subject: [Python-Dev] License() release list is imcomplete; intentional? References: <l19t08$uj0$1@ger.gmane.org> Message-ID: <20130917174736.38068e19@pitrou.net> Le Tue, 17 Sep 2013 11:37:48 -0400, Terry Reedy <tjreedy at udel.edu> a ?crit : > On 2.7, >>> license() return a text that includes a complete list of > releases from 1.6 to 2.7 and stops there > Release Derived Year Owner GPL- > from compatible? > (1) > > 0.9.0 thru 1.2 1991-1995 CWI yes > 1.3 thru 1.5.2 1.2 1995-1999 CNRI yes > 1.6 1.5.2 2000 CNRI no > 2.0 1.6 2000 BeOpen.com no > ... > 2.6.5 2.6.4 2010 PSF yes > 2.7 2.6 2010 PSF yes > > Was it intentional to stop with 2.7 and not continue with 2.7.1, etc? > > On 3.3.2, the 2.x list ends with 2.6.5 and never mentions 2.7. > Intentional? It then jumps back to 3.0 and ends with the 'previous' > release, 3.3.1. Should 3.3.2 be included in the 3.3.2 list? > > ... > 2.6.4 2.6.3 2009 PSF yes > 2.6.5 2.6.4 2010 PSF yes > 3.0 2.6 2008 PSF yes > 3.0.1 3.0 2009 PSF yes > ... > 3.2.4 3.2.3 2013 PSF yes > 3.3.0 3.2 2012 PSF yes > 3.3.1 3.3.0 2013 PSF yes I don't really understand why the releases should be manually listed. Is it some kind of defensive coding? Regards Antoine. From python at mrabarnett.plus.com Tue Sep 17 17:48:44 2013 From: python at mrabarnett.plus.com (MRAB) Date: Tue, 17 Sep 2013 16:48:44 +0100 Subject: [Python-Dev] License() release list is imcomplete; intentional? In-Reply-To: <l19t08$uj0$1@ger.gmane.org> References: <l19t08$uj0$1@ger.gmane.org> Message-ID: <523879DC.5070402@mrabarnett.plus.com> On 17/09/2013 16:37, Terry Reedy wrote: > On 2.7, >>> license() return a text that includes a complete list of > releases from 1.6 to 2.7 and stops there > Release Derived Year Owner GPL- > from compatible? (1) > > 0.9.0 thru 1.2 1991-1995 CWI yes > 1.3 thru 1.5.2 1.2 1995-1999 CNRI yes > 1.6 1.5.2 2000 CNRI no > 2.0 1.6 2000 BeOpen.com no > ... > 2.6.5 2.6.4 2010 PSF yes > 2.7 2.6 2010 PSF yes > > Was it intentional to stop with 2.7 and not continue with 2.7.1, etc? > > On 3.3.2, the 2.x list ends with 2.6.5 and never mentions 2.7. Intentional? > It then jumps back to 3.0 and ends with the 'previous' release, 3.3.1. > Should 3.3.2 be included in the 3.3.2 list? > > ... > 2.6.4 2.6.3 2009 PSF yes > 2.6.5 2.6.4 2010 PSF yes > 3.0 2.6 2008 PSF yes > 3.0.1 3.0 2009 PSF yes > ... > 3.2.4 3.2.3 2013 PSF yes > 3.3.0 3.2 2012 PSF yes > 3.3.1 3.3.0 2013 PSF yes > To me it looks like Python 2.7 isn't mentioned in the licence for Python 3 because Python 3 is derived from Python 2.6 and Python 2.7 is on a different branch, so it isn't relevant. From rdmurray at bitdance.com Tue Sep 17 18:25:42 2013 From: rdmurray at bitdance.com (R. David Murray) Date: Tue, 17 Sep 2013 12:25:42 -0400 Subject: [Python-Dev] PEP 453: Explicit bootstrapping of pip In-Reply-To: <CADiSq7f2yabBwu6_esKtvcs7tnFeifM7qdZCzO5ZAuevxsjo+w@mail.gmail.com> References: <CADiSq7f2yabBwu6_esKtvcs7tnFeifM7qdZCzO5ZAuevxsjo+w@mail.gmail.com> Message-ID: <20130917162543.2AC0D250909@webabinitio.net> This PEP looks great to me. Thanks to everyone involved. On Wed, 18 Sep 2013 00:46:01 +1000, Nick Coghlan <ncoghlan at gmail.com> wrote: > Reducing the burden of actually installing a third-party package should > also decrease the pressure to add every useful module to the standard > library. This will allow additions to the standard library to focus more > on why Python should have a particular tool out of the box instead of > using the general difficulty of installing third-party packages as > justification for inclusion. I don't think this PEP changes the inclusion calculus, because I don't think we've given any real weight to that in stdlib inclusion decisions. I think the decision making is already focused on why Python should have a particular tool out of the box. The only context I recall where "installation difficulty" has been considered relevant is the issue of corporate users restricted from installing 3rd party packages...which will not be affected at all by this PEP. --David From donald at stufft.io Tue Sep 17 18:38:31 2013 From: donald at stufft.io (Donald Stufft) Date: Tue, 17 Sep 2013 12:38:31 -0400 Subject: [Python-Dev] PEP 453: Explicit bootstrapping of pip In-Reply-To: <20130917162543.2AC0D250909@webabinitio.net> References: <CADiSq7f2yabBwu6_esKtvcs7tnFeifM7qdZCzO5ZAuevxsjo+w@mail.gmail.com> <20130917162543.2AC0D250909@webabinitio.net> Message-ID: <877B56B4-16B2-4694-8CEF-469B0C16EDB7@stufft.io> On Sep 17, 2013, at 12:25 PM, "R. David Murray" <rdmurray at bitdance.com> wrote: > This PEP looks great to me. Thanks to everyone involved. > > On Wed, 18 Sep 2013 00:46:01 +1000, Nick Coghlan <ncoghlan at gmail.com> wrote: >> Reducing the burden of actually installing a third-party package should >> also decrease the pressure to add every useful module to the standard >> library. This will allow additions to the standard library to focus more >> on why Python should have a particular tool out of the box instead of >> using the general difficulty of installing third-party packages as >> justification for inclusion. > > I don't think this PEP changes the inclusion calculus, because I don't > think we've given any real weight to that in stdlib inclusion decisions. > I think the decision making is already focused on why Python should have > a particular tool out of the box. When I originally wrote this statement I wasn't trying to say that modules were included because of the lack of an easy installer, but instead that a not insignificant number of people had wanted to include stuff and some of their reasoning that I had seen in the past included the difficulty in bootstrapping the installer. So it decreases the *pressure* from others to include stuff for connivence not that it necessarily changes how python-dev itself weighs the decision to do so (but it might for some people?). > > The only context I recall where "installation difficulty" has been > considered relevant is the issue of corporate users restricted from > installing 3rd party packages...which will not be affected at all by > this PEP. > > --David > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: <http://mail.python.org/pipermail/python-dev/attachments/20130917/b13f1454/attachment.sig> From skip at pobox.com Tue Sep 17 18:42:22 2013 From: skip at pobox.com (Skip Montanaro) Date: Tue, 17 Sep 2013 11:42:22 -0500 Subject: [Python-Dev] License() release list is imcomplete; intentional? In-Reply-To: <20130917174736.38068e19@pitrou.net> References: <l19t08$uj0$1@ger.gmane.org> <20130917174736.38068e19@pitrou.net> Message-ID: <CANc-5UzmYe_CwZR+ns5ftK5u74V-N+HB2JZ2VjTByYKXAJ62+Q@mail.gmail.com> > I don't really understand why the releases should be manually listed. > Is it some kind of defensive coding? I think it's to give people who care about such things all the information they need to make informed decisions. As I recall, the 1.6 series was problematic, because it wasn't actually open source. That's why 2.0 is a derivative of 1.5.2. I doubt there are other licensing weirdities lurking, but better to give people everything they might need, especially if it's not difficult. At this point I think the hard work is done. Release managers or their minions just need to remember to tack on the next line. Skip From guido at python.org Tue Sep 17 18:47:48 2013 From: guido at python.org (Guido van Rossum) Date: Tue, 17 Sep 2013 09:47:48 -0700 Subject: [Python-Dev] License() release list is imcomplete; intentional? In-Reply-To: <20130917174736.38068e19@pitrou.net> References: <l19t08$uj0$1@ger.gmane.org> <20130917174736.38068e19@pitrou.net> Message-ID: <CAP7+vJJ7L8KG0zQ-gOzbMAV3KcLu3bnZC3i0tAezy1UNBvsNMA@mail.gmail.com> On Tue, Sep 17, 2013 at 8:47 AM, Antoine Pitrou <solipsis at pitrou.net> wrote: > Le Tue, 17 Sep 2013 11:37:48 -0400, > Terry Reedy <tjreedy at udel.edu> a ?crit : > > > On 2.7, >>> license() return a text that includes a complete list of > > releases from 1.6 to 2.7 and stops there > > Release Derived Year Owner GPL- > > from compatible? > > (1) > > > > 0.9.0 thru 1.2 1991-1995 CWI yes > > 1.3 thru 1.5.2 1.2 1995-1999 CNRI yes > > 1.6 1.5.2 2000 CNRI no > > 2.0 1.6 2000 BeOpen.com no > > ... > > 2.6.5 2.6.4 2010 PSF yes > > 2.7 2.6 2010 PSF yes > > > > Was it intentional to stop with 2.7 and not continue with 2.7.1, etc? > > > > On 3.3.2, the 2.x list ends with 2.6.5 and never mentions 2.7. > > Intentional? It then jumps back to 3.0 and ends with the 'previous' > > release, 3.3.1. Should 3.3.2 be included in the 3.3.2 list? > > > > ... > > 2.6.4 2.6.3 2009 PSF yes > > 2.6.5 2.6.4 2010 PSF yes > > 3.0 2.6 2008 PSF yes > > 3.0.1 3.0 2009 PSF yes > > ... > > 3.2.4 3.2.3 2013 PSF yes > > 3.3.0 3.2 2012 PSF yes > > 3.3.1 3.3.0 2013 PSF yes > > I don't really understand why the releases should be manually listed. > Is it some kind of defensive coding? > Worse, it's superstition based on myth. IIRC this table was added when a few core Python developers including myself left CNRI in 2000. We had a bit of an argument about the license (not too much though -- in the end things came out alright). Some lawyer at CNRI thought it was a good idea to record a release history like this with the license, as a defense against whatever claims of ownership to the code someone else might suddenly come up with. Since all I wanted was to get out of there while causing them minimal upset, I told them I'd comply. But that's over 13 years ago now, and I'm not sure if it ever made sense (the internet is a different place than CNRI's lawyers envisioned). Only the top 10 of so lines of the table are in the least interesting (note that it describes a graph). I propose that we truncate the table and add a note saying that all following releases are owned by the PSF, GPL-compatible, and derived from previous PSF-owned and GPL-compatible releases. That should do until the PSF goes out of business (which I hope will never happen -- this is one reason why I wish the conferences were run by a separate entity, to avoid a conference bankruptcy from risking Python's continued open-source status). -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130917/1a267881/attachment.html> From solipsis at pitrou.net Tue Sep 17 19:39:26 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 17 Sep 2013 19:39:26 +0200 Subject: [Python-Dev] License() release list is imcomplete; intentional? In-Reply-To: <CAP7+vJJ7L8KG0zQ-gOzbMAV3KcLu3bnZC3i0tAezy1UNBvsNMA@mail.gmail.com> References: <l19t08$uj0$1@ger.gmane.org> <20130917174736.38068e19@pitrou.net> <CAP7+vJJ7L8KG0zQ-gOzbMAV3KcLu3bnZC3i0tAezy1UNBvsNMA@mail.gmail.com> Message-ID: <20130917193926.0d180351@fsol> On Tue, 17 Sep 2013 09:47:48 -0700 Guido van Rossum <guido at python.org> wrote: > > IIRC this table was added when a few core Python developers including > myself left CNRI in 2000. We had a bit of an argument about the license > (not too much though -- in the end things came out alright). Some lawyer at > CNRI thought it was a good idea to record a release history like this with > the license, as a defense against whatever claims of ownership to the code > someone else might suddenly come up with. Since all I wanted was to get out > of there while causing them minimal upset, I told them I'd comply. But > that's over 13 years ago now, and I'm not sure if it ever made sense (the > internet is a different place than CNRI's lawyers envisioned). Only the top > 10 of so lines of the table are in the least interesting (note that it > describes a graph). I propose that we truncate the table and add a note > saying that all following releases are owned by the PSF, GPL-compatible, > and derived from previous PSF-owned and GPL-compatible releases. That > should do until the PSF goes out of business (which I hope will never > happen -- this is one reason why I wish the conferences were run by a > separate entity, to avoid a conference bankruptcy from risking Python's > continued open-source status). The PSF isn't technically the copyright holder, so would that pose a significant threat? (at worse the PSF could relicense Python based on the copyright agreements, but Python would still be distributable under the original license) Regards Antoine. From guido at python.org Tue Sep 17 19:51:28 2013 From: guido at python.org (Guido van Rossum) Date: Tue, 17 Sep 2013 10:51:28 -0700 Subject: [Python-Dev] License() release list is imcomplete; intentional? In-Reply-To: <20130917193926.0d180351@fsol> References: <l19t08$uj0$1@ger.gmane.org> <20130917174736.38068e19@pitrou.net> <CAP7+vJJ7L8KG0zQ-gOzbMAV3KcLu3bnZC3i0tAezy1UNBvsNMA@mail.gmail.com> <20130917193926.0d180351@fsol> Message-ID: <CAP7+vJKeTDEZa4kFy18j6vO++6PB-swxf+KNL0pcBj-Erqqzcg@mail.gmail.com> On Tue, Sep 17, 2013 at 10:39 AM, Antoine Pitrou <solipsis at pitrou.net>wrote: > On Tue, 17 Sep 2013 09:47:48 -0700 > Guido van Rossum <guido at python.org> wrote: > > > > IIRC this table was added when a few core Python developers including > > myself left CNRI in 2000. We had a bit of an argument about the license > > (not too much though -- in the end things came out alright). Some lawyer > at > > CNRI thought it was a good idea to record a release history like this > with > > the license, as a defense against whatever claims of ownership to the > code > > someone else might suddenly come up with. Since all I wanted was to get > out > > of there while causing them minimal upset, I told them I'd comply. But > > that's over 13 years ago now, and I'm not sure if it ever made sense (the > > internet is a different place than CNRI's lawyers envisioned). Only the > top > > 10 of so lines of the table are in the least interesting (note that it > > describes a graph). I propose that we truncate the table and add a note > > saying that all following releases are owned by the PSF, GPL-compatible, > > and derived from previous PSF-owned and GPL-compatible releases. That > > should do until the PSF goes out of business (which I hope will never > > happen -- this is one reason why I wish the conferences were run by a > > separate entity, to avoid a conference bankruptcy from risking Python's > > continued open-source status). > > The PSF isn't technically the copyright holder, so would that pose a > significant threat? > I have no idea. If you want a real answer talk to a lawyer. > (at worse the PSF could relicense Python based on the copyright > agreements, but Python would still be distributable under the original > license) > But only the PSF has the list of original contributors and their licenses. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130917/c5f0dfd7/attachment.html> From rdmurray at bitdance.com Tue Sep 17 20:52:33 2013 From: rdmurray at bitdance.com (R. David Murray) Date: Tue, 17 Sep 2013 14:52:33 -0400 Subject: [Python-Dev] PEP 453: Explicit bootstrapping of pip In-Reply-To: <877B56B4-16B2-4694-8CEF-469B0C16EDB7@stufft.io> References: <CADiSq7f2yabBwu6_esKtvcs7tnFeifM7qdZCzO5ZAuevxsjo+w@mail.gmail.com> <20130917162543.2AC0D250909@webabinitio.net> <877B56B4-16B2-4694-8CEF-469B0C16EDB7@stufft.io> Message-ID: <20130917185233.8191E250512@webabinitio.net> On Tue, 17 Sep 2013 12:38:31 -0400, Donald Stufft <donald at stufft.io> wrote: > On Sep 17, 2013, at 12:25 PM, "R. David Murray" <rdmurray at bitdance.com> wrote: > > I don't think this PEP changes the inclusion calculus, because I don't > > think we've given any real weight to that in stdlib inclusion decisions. > > I think the decision making is already focused on why Python should have > > a particular tool out of the box. > > When I originally wrote this statement I wasn't trying to say that modules > were included because of the lack of an easy installer, but instead that > a not insignificant number of people had wanted to include stuff and some > of their reasoning that I had seen in the past included the difficulty in > bootstrapping the installer. > > So it decreases the *pressure* from others to include stuff for connivence > not that it necessarily changes how python-dev itself weighs the decision > to do so (but it might for some people?). I see. Yes, that makes sense. --David From v+python at g.nevcal.com Tue Sep 17 20:45:33 2013 From: v+python at g.nevcal.com (Glenn Linderman) Date: Tue, 17 Sep 2013 11:45:33 -0700 Subject: [Python-Dev] License() release list is imcomplete; intentional? In-Reply-To: <CAP7+vJKeTDEZa4kFy18j6vO++6PB-swxf+KNL0pcBj-Erqqzcg@mail.gmail.com> References: <l19t08$uj0$1@ger.gmane.org> <20130917174736.38068e19@pitrou.net> <CAP7+vJJ7L8KG0zQ-gOzbMAV3KcLu3bnZC3i0tAezy1UNBvsNMA@mail.gmail.com> <20130917193926.0d180351@fsol> <CAP7+vJKeTDEZa4kFy18j6vO++6PB-swxf+KNL0pcBj-Erqqzcg@mail.gmail.com> Message-ID: <5238A34D.6030408@g.nevcal.com> On 9/17/2013 10:51 AM, Guido van Rossum wrote: > But only the PSF has the list of original contributors and their > licenses. So can that list be made public, and available in multiple archives? -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130917/6f12820e/attachment.html> From brett at python.org Tue Sep 17 21:39:25 2013 From: brett at python.org (Brett Cannon) Date: Tue, 17 Sep 2013 15:39:25 -0400 Subject: [Python-Dev] License() release list is imcomplete; intentional? In-Reply-To: <5238A34D.6030408@g.nevcal.com> References: <l19t08$uj0$1@ger.gmane.org> <20130917174736.38068e19@pitrou.net> <CAP7+vJJ7L8KG0zQ-gOzbMAV3KcLu3bnZC3i0tAezy1UNBvsNMA@mail.gmail.com> <20130917193926.0d180351@fsol> <CAP7+vJKeTDEZa4kFy18j6vO++6PB-swxf+KNL0pcBj-Erqqzcg@mail.gmail.com> <5238A34D.6030408@g.nevcal.com> Message-ID: <CAP1=2W4D0_iwq1zhq06Z4PKrNAfEfXVGKoHoB4GRNdR2g0JTfA@mail.gmail.com> On Tue, Sep 17, 2013 at 2:45 PM, Glenn Linderman <v+python at g.nevcal.com>wrote: > On 9/17/2013 10:51 AM, Guido van Rossum wrote: > > But only the PSF has the list of original contributors and their licenses. > > > So can that list be made public, and available in multiple archives? > Technically it already is public based on the * next to your name on bugs.python.org. As for a concise list that's more human readable I'm sure it could if someone chose to put the effort in to write the code to make the list. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130917/881cfe71/attachment-0001.html> From tjreedy at udel.edu Tue Sep 17 21:40:58 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 17 Sep 2013 15:40:58 -0400 Subject: [Python-Dev] License() release list is imcomplete; intentional? In-Reply-To: <523879DC.5070402@mrabarnett.plus.com> References: <l19t08$uj0$1@ger.gmane.org> <523879DC.5070402@mrabarnett.plus.com> Message-ID: <l1ab86$jt7$1@ger.gmane.org> On 9/17/2013 11:48 AM, MRAB wrote: > On 17/09/2013 16:37, Terry Reedy wrote: >> On 2.7, >>> license() return a text that includes a complete list of >> releases from 1.6 to 2.7 and stops there >> Release Derived Year Owner GPL- >> from compatible? (1) >> >> 0.9.0 thru 1.2 1991-1995 CWI yes >> 1.3 thru 1.5.2 1.2 1995-1999 CNRI yes >> 1.6 1.5.2 2000 CNRI no >> 2.0 1.6 2000 BeOpen.com no >> ... >> 2.6.5 2.6.4 2010 PSF yes >> 2.7 2.6 2010 PSF yes >> >> Was it intentional to stop with 2.7 and not continue with 2.7.1, etc? >> >> On 3.3.2, the 2.x list ends with 2.6.5 and never mentions 2.7. >> Intentional? >> It then jumps back to 3.0 and ends with the 'previous' release, 3.3.1. >> Should 3.3.2 be included in the 3.3.2 list? >> >> ... >> 2.6.4 2.6.3 2009 PSF yes >> 2.6.5 2.6.4 2010 PSF yes >> 3.0 2.6 2008 PSF yes >> 3.0.1 3.0 2009 PSF yes >> ... >> 3.2.4 3.2.3 2013 PSF yes >> 3.3.0 3.2 2012 PSF yes >> 3.3.1 3.3.0 2013 PSF yes >> > To me it looks like Python 2.7 isn't mentioned in the licence for > Python 3 because Python 3 is derived from Python 2.6 and Python 2.7 is > on a different branch, so it isn't relevant. I thought of that, but if that were the reason, nothing after 2.6(.0) should be listed. I think we should follow Guido's idea of truncating after n lines, summarizing the rest, and freezing the file until there is a significant change. -- Terry Jan Reedy From rdmurray at bitdance.com Tue Sep 17 22:46:12 2013 From: rdmurray at bitdance.com (R. David Murray) Date: Tue, 17 Sep 2013 16:46:12 -0400 Subject: [Python-Dev] License() release list is imcomplete; intentional? In-Reply-To: <CAP1=2W4D0_iwq1zhq06Z4PKrNAfEfXVGKoHoB4GRNdR2g0JTfA@mail.gmail.com> References: <l19t08$uj0$1@ger.gmane.org> <20130917174736.38068e19@pitrou.net> <CAP7+vJJ7L8KG0zQ-gOzbMAV3KcLu3bnZC3i0tAezy1UNBvsNMA@mail.gmail.com> <20130917193926.0d180351@fsol> <CAP7+vJKeTDEZa4kFy18j6vO++6PB-swxf+KNL0pcBj-Erqqzcg@mail.gmail.com> <5238A34D.6030408@g.nevcal.com> <CAP1=2W4D0_iwq1zhq06Z4PKrNAfEfXVGKoHoB4GRNdR2g0JTfA@mail.gmail.com> Message-ID: <20130917204613.376D8250818@webabinitio.net> On Tue, 17 Sep 2013 15:39:25 -0400, Brett Cannon <brett at python.org> wrote: > On Tue, Sep 17, 2013 at 2:45 PM, Glenn Linderman <v+python at g.nevcal.com>wrote: > > On 9/17/2013 10:51 AM, Guido van Rossum wrote: > > > > But only the PSF has the list of original contributors and their licenses. > > > > So can that list be made public, and available in multiple archives? > > > > Technically it already is public based on the * next to your name on > bugs.python.org. As for a concise list that's more human readable I'm sure > it could if someone chose to put the effort in to write the code to make > the list. The list is public in that sense (but probably not complete). However, whatever is considered to be the legal info, including the license the contributor chose, is not in the tracker. That information is in the hands of the PSF secretary, and to my (outsider, and therefore possibly incorrect :) understanding most of it is currently in the form of physical paper in a filing cabinet. I don't know if there is a project to digitize the archives or not, but it would seem like a sensible thing to do, for backup reasons if nothing else (perhaps there are already physical backups, though). --David From guido at python.org Tue Sep 17 23:21:16 2013 From: guido at python.org (Guido van Rossum) Date: Tue, 17 Sep 2013 14:21:16 -0700 Subject: [Python-Dev] PEP 450 adding statistics module In-Reply-To: <CAP7+vJKPDCZd4ZQn6mSkiAx7=wjv7WmmP3HjbOj-FLGUY-yNRg@mail.gmail.com> References: <520C2E23.40405@pearwood.info> <52215BDF.2090109@pearwood.info> <20130908123706.GG26319@ando> <74b03474-ab18-40c5-8ce6-de5159d34903@email.android.com> <522CA085.6000601@stoneleaf.us> <CAP7+vJ+TTaU+ntaLT3xEU2wvFQhgP4P2-FnZUJ7V2PAAy3SMEA@mail.gmail.com> <CAP7+vJJ6QxcUwcpdt3Mq6r07Q2dGUQt-VWGdsYGLv4wrRW5dGg@mail.gmail.com> <20130914005921.GK16820@ando> <CAP7+vJL81VR=kdO+2rCz3Di3GVNSwhkRCXrw2HAL=7ywT42rvA@mail.gmail.com> <20130916235912.GG19939@ando> <CAP7+vJKPDCZd4ZQn6mSkiAx7=wjv7WmmP3HjbOj-FLGUY-yNRg@mail.gmail.com> Message-ID: <CAP7+vJ+rtwSmKXG4Pwq7jD7Xegdka6dGhGVKW8Lf3PtWLCCpXA@mail.gmail.com> Congrats, I've accepted the PEP. Nice work! Please work with the reviewers on the issue on the code. (Steven or Oscar, if either of you could work Oscar's list of resolved issues into a patch for the PEP I'll happily update it, just mail it to peps at python.org.) On Mon, Sep 16, 2013 at 5:06 PM, Guido van Rossum <guido at python.org> wrote: > On Mon, Sep 16, 2013 at 4:59 PM, Steven D'Aprano <steve at pearwood.info>wrote: > >> On Mon, Sep 16, 2013 at 08:42:12AM -0700, Guido van Rossum wrote: >> > I'm ready to accept this PEP. Because I haven't read this entire thread >> > (and 60 messages about random diversions is really too much to try and >> > catch up on) I'll give people 24 hours to remind me of outstanding >> > rejections. >> > >> > I also haven't reviewed the code in any detail, but I believe the code >> > review is going well, so I'm not concerned that the PEP would have to >> > revised based on that alone. >> >> There are a couple of outstanding issues that I am aware of, but I don't >> believe that either of these affect acceptance/rejection of the PEP. >> Please correct me if I am wrong. >> >> 1) Implementation details of the statistics.sum function. Oscar is >> giving me a lot of very valuable assistance speeding up the >> implementation of sum. >> >> 2) The current implementation has extensive docstrings, but will also >> need a separate statistics.rst file. >> >> >> I don't recall any other outstanding issues, if I have forgotten any, >> please remind me. >> > > Those certainly don't stand in the way of the PEP's acceptance (but they > do block the commit of the code :-). > > The issues that Oscar listed also all seem resolved (though they would > make a nice addition to the "Discussion" section in the PEP). > > -- > --Guido van Rossum (python.org/~guido) > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130917/4b35d7ec/attachment.html> From steve at pearwood.info Wed Sep 18 00:05:55 2013 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 18 Sep 2013 08:05:55 +1000 Subject: [Python-Dev] PEP 450 adding statistics module In-Reply-To: <CAP7+vJ+rtwSmKXG4Pwq7jD7Xegdka6dGhGVKW8Lf3PtWLCCpXA@mail.gmail.com> References: <20130908123706.GG26319@ando> <74b03474-ab18-40c5-8ce6-de5159d34903@email.android.com> <522CA085.6000601@stoneleaf.us> <CAP7+vJ+TTaU+ntaLT3xEU2wvFQhgP4P2-FnZUJ7V2PAAy3SMEA@mail.gmail.com> <CAP7+vJJ6QxcUwcpdt3Mq6r07Q2dGUQt-VWGdsYGLv4wrRW5dGg@mail.gmail.com> <20130914005921.GK16820@ando> <CAP7+vJL81VR=kdO+2rCz3Di3GVNSwhkRCXrw2HAL=7ywT42rvA@mail.gmail.com> <20130916235912.GG19939@ando> <CAP7+vJKPDCZd4ZQn6mSkiAx7=wjv7WmmP3HjbOj-FLGUY-yNRg@mail.gmail.com> <CAP7+vJ+rtwSmKXG4Pwq7jD7Xegdka6dGhGVKW8Lf3PtWLCCpXA@mail.gmail.com> Message-ID: <20130917220554.GH19939@ando> On Tue, Sep 17, 2013 at 02:21:16PM -0700, Guido van Rossum wrote: > Congrats, I've accepted the PEP. Nice work! Please work with the reviewers > on the issue on the code. Thank you, and thanks to everyone who contributed with the discussion. > (Steven or Oscar, if either of you could work Oscar's list of resolved > issues into a patch for the PEP I'll happily update it, just mail it to > peps at python.org.) I will do this. -- Steven From ethan at stoneleaf.us Tue Sep 17 23:40:21 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 17 Sep 2013 14:40:21 -0700 Subject: [Python-Dev] PEP 450 adding statistics module In-Reply-To: <CAP7+vJ+rtwSmKXG4Pwq7jD7Xegdka6dGhGVKW8Lf3PtWLCCpXA@mail.gmail.com> References: <520C2E23.40405@pearwood.info> <52215BDF.2090109@pearwood.info> <20130908123706.GG26319@ando> <74b03474-ab18-40c5-8ce6-de5159d34903@email.android.com> <522CA085.6000601@stoneleaf.us> <CAP7+vJ+TTaU+ntaLT3xEU2wvFQhgP4P2-FnZUJ7V2PAAy3SMEA@mail.gmail.com> <CAP7+vJJ6QxcUwcpdt3Mq6r07Q2dGUQt-VWGdsYGLv4wrRW5dGg@mail.gmail.com> <20130914005921.GK16820@ando> <CAP7+vJL81VR=kdO+2rCz3Di3GVNSwhkRCXrw2HAL=7ywT42rvA@mail.gmail.com> <20130916235912.GG19939@ando> <CAP7+vJKPDCZd4ZQn6mSkiAx7=wjv7WmmP3HjbOj-FLGUY-yNRg@mail.gmail.com> <CAP7+vJ+rtwSmKXG4Pwq7jD7Xegdka6dGhGVKW8Lf3PtWLCCpXA@mail.gmail.com> Message-ID: <5238CC45.2070104@stoneleaf.us> On 09/17/2013 02:21 PM, Guido van Rossum wrote: > Congrats, I've accepted the PEP. Nice work! Please work with the reviewers on the issue on the code. Congratulations, Stephen! -- ~Ethan~ From victor.stinner at gmail.com Wed Sep 18 01:17:21 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 18 Sep 2013 01:17:21 +0200 Subject: [Python-Dev] benchmark Message-ID: <CAMpsgwa=Tq3OtbQVn6ieO3fp77tHX-ywoNqBXa_sY9gxhOMWDw@mail.gmail.com> -------------- next part -------------- A non-text attachment was scrubbed... Name: benchmark.patch Type: application/octet-stream Size: 36048 bytes Desc: not available URL: <http://mail.python.org/pipermail/python-dev/attachments/20130918/871cef31/attachment-0001.obj> From ncoghlan at gmail.com Wed Sep 18 01:44:24 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 18 Sep 2013 09:44:24 +1000 Subject: [Python-Dev] PEP 450 adding statistics module In-Reply-To: <5238CC45.2070104@stoneleaf.us> References: <520C2E23.40405@pearwood.info> <52215BDF.2090109@pearwood.info> <20130908123706.GG26319@ando> <74b03474-ab18-40c5-8ce6-de5159d34903@email.android.com> <522CA085.6000601@stoneleaf.us> <CAP7+vJ+TTaU+ntaLT3xEU2wvFQhgP4P2-FnZUJ7V2PAAy3SMEA@mail.gmail.com> <CAP7+vJJ6QxcUwcpdt3Mq6r07Q2dGUQt-VWGdsYGLv4wrRW5dGg@mail.gmail.com> <20130914005921.GK16820@ando> <CAP7+vJL81VR=kdO+2rCz3Di3GVNSwhkRCXrw2HAL=7ywT42rvA@mail.gmail.com> <20130916235912.GG19939@ando> <CAP7+vJKPDCZd4ZQn6mSkiAx7=wjv7WmmP3HjbOj-FLGUY-yNRg@mail.gmail.com> <CAP7+vJ+rtwSmKXG4Pwq7jD7Xegdka6dGhGVKW8Lf3PtWLCCpXA@mail.gmail.com> <5238CC45.2070104@stoneleaf.us> Message-ID: <CADiSq7e9a5GzUQoEwJ+MGnXVdkGNd5044xx4YN4_z6bWmFRHnw@mail.gmail.com> On 18 Sep 2013 08:36, "Ethan Furman" <ethan at stoneleaf.us> wrote: > > On 09/17/2013 02:21 PM, Guido van Rossum wrote: >> >> Congrats, I've accepted the PEP. Nice work! Please work with the reviewers on the issue on the code. > > > Congratulations, Stephen! Yay! Cheers, Nick. > > -- > ~Ethan~ > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130918/b03541b0/attachment.html> From pjenvey at underboss.org Wed Sep 18 03:10:53 2013 From: pjenvey at underboss.org (Philip Jenvey) Date: Tue, 17 Sep 2013 18:10:53 -0700 Subject: [Python-Dev] PEP 428: Pathlib -> stat caching In-Reply-To: <20130916220546.341e369f@fsol> References: <5235BA20.8010105@stoneleaf.us> <20130916101508.135926de@pitrou.net> <CAH_1eM1uo0W29xy-pJPOpe-Orfc830M3BUd3coBO1dDdHSxekg@mail.gmail.com> <20130916214536.55907f19@fsol> <CAP1=2W5yq_t2iMiuptkW5w7x7kiowVCFXM4qQUEHug1QVAx6ug@mail.gmail.com> <20130916220546.341e369f@fsol> Message-ID: <BEA6DBE1-BB54-44A7-9E83-CD9AA8CB533A@underboss.org> On Sep 16, 2013, at 1:05 PM, Antoine Pitrou wrote: > On Mon, 16 Sep 2013 15:48:54 -0400 > Brett Cannon <brett at python.org> wrote: >>> >>> So I would like to propose the following API change: >>> >>> - Path.stat() (and stat-accessing methods such as get_mtime()...) >>> returns an uncached stat object by default >>> >>> - Path.cache_stat() can be called to return the stat() *and* cache it >>> for future use, such that any future call to stat(), cache_stat() or >>> a stat-accessing function reuses that cached stat >>> >>> In other words, only if you use cache_stat() at least once is the >>> stat() value cached and reused by the Path object. >>> (also, it's a per-Path decision) >>> >> >> Any reason why stat() can't get a keyword-only cached=True argument >> instead? Or have stat() never cache() but stat_cache() always so that >> people can choose if they want fresh or cached based on API and not whether >> some library happened to make a decision for them? > > 1. Because you also want the helper functions (get_mtime(), etc.) to > cache the value too. It's not only about stat(). With the proposed rich stat object the convenience methods living on Path wouldn't result in much added convenience: p.is_dir() vs p.stat().is_dir() Why not move these methods from Path to a rich stat obj and not cache stat results at all? It's easy enough for users to cache them themselves and much more explicit. -- Philip Jenvey From ncoghlan at gmail.com Wed Sep 18 03:32:14 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 18 Sep 2013 11:32:14 +1000 Subject: [Python-Dev] PEP 428: Pathlib -> stat caching In-Reply-To: <BEA6DBE1-BB54-44A7-9E83-CD9AA8CB533A@underboss.org> References: <5235BA20.8010105@stoneleaf.us> <20130916101508.135926de@pitrou.net> <CAH_1eM1uo0W29xy-pJPOpe-Orfc830M3BUd3coBO1dDdHSxekg@mail.gmail.com> <20130916214536.55907f19@fsol> <CAP1=2W5yq_t2iMiuptkW5w7x7kiowVCFXM4qQUEHug1QVAx6ug@mail.gmail.com> <20130916220546.341e369f@fsol> <BEA6DBE1-BB54-44A7-9E83-CD9AA8CB533A@underboss.org> Message-ID: <CADiSq7fuXZMFjaEJKyFd9O1O=z+hzkesAFtxrHVa5m-f2WbOeA@mail.gmail.com> On 18 September 2013 11:10, Philip Jenvey <pjenvey at underboss.org> wrote: > > On Sep 16, 2013, at 1:05 PM, Antoine Pitrou wrote: > >> On Mon, 16 Sep 2013 15:48:54 -0400 >> Brett Cannon <brett at python.org> wrote: >>>> >>>> So I would like to propose the following API change: >>>> >>>> - Path.stat() (and stat-accessing methods such as get_mtime()...) >>>> returns an uncached stat object by default >>>> >>>> - Path.cache_stat() can be called to return the stat() *and* cache it >>>> for future use, such that any future call to stat(), cache_stat() or >>>> a stat-accessing function reuses that cached stat >>>> >>>> In other words, only if you use cache_stat() at least once is the >>>> stat() value cached and reused by the Path object. >>>> (also, it's a per-Path decision) >>>> >>> >>> Any reason why stat() can't get a keyword-only cached=True argument >>> instead? Or have stat() never cache() but stat_cache() always so that >>> people can choose if they want fresh or cached based on API and not whether >>> some library happened to make a decision for them? >> >> 1. Because you also want the helper functions (get_mtime(), etc.) to >> cache the value too. It's not only about stat(). > > With the proposed rich stat object the convenience methods living on Path wouldn't result in much added convenience: > > p.is_dir() vs p.stat().is_dir() > > Why not move these methods from Path to a rich stat obj and not cache stat results at all? It's easy enough for users to cache them themselves and much more explicit. Because that doesn't help iterator based os.walk inspired APIs like walkdir, which would benefit greatly from a path type with implicit caching, but would have to complicate their APIs significantly to pass around separate stat objects. Rewriting walkdir to depend on pathlib has been on my todo list for a while, as it solves a potentially serious walkdir performance problem where chained iterators have to make repeated stat calls to answer questions that were already asked by earlier iterators in the pipeline. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From oscar.j.benjamin at gmail.com Wed Sep 18 04:32:20 2013 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Wed, 18 Sep 2013 03:32:20 +0100 Subject: [Python-Dev] PEP 450 adding statistics module In-Reply-To: <CAP7+vJ+rtwSmKXG4Pwq7jD7Xegdka6dGhGVKW8Lf3PtWLCCpXA@mail.gmail.com> References: <520C2E23.40405@pearwood.info> <52215BDF.2090109@pearwood.info> <20130908123706.GG26319@ando> <74b03474-ab18-40c5-8ce6-de5159d34903@email.android.com> <522CA085.6000601@stoneleaf.us> <CAP7+vJ+TTaU+ntaLT3xEU2wvFQhgP4P2-FnZUJ7V2PAAy3SMEA@mail.gmail.com> <CAP7+vJJ6QxcUwcpdt3Mq6r07Q2dGUQt-VWGdsYGLv4wrRW5dGg@mail.gmail.com> <20130914005921.GK16820@ando> <CAP7+vJL81VR=kdO+2rCz3Di3GVNSwhkRCXrw2HAL=7ywT42rvA@mail.gmail.com> <20130916235912.GG19939@ando> <CAP7+vJKPDCZd4ZQn6mSkiAx7=wjv7WmmP3HjbOj-FLGUY-yNRg@mail.gmail.com> <CAP7+vJ+rtwSmKXG4Pwq7jD7Xegdka6dGhGVKW8Lf3PtWLCCpXA@mail.gmail.com> Message-ID: <CAHVvXxQC7eKnLUvx8vfBsRu7Yhut2XMkUrkTnUi_SPNPqVy0Ew@mail.gmail.com> On 17 September 2013 22:21, Guido van Rossum <guido at python.org> wrote: > Congrats, I've accepted the PEP. Nice work! Please work with the reviewers > on the issue on the code. Good work, Steven! From greg at krypto.org Wed Sep 18 08:43:57 2013 From: greg at krypto.org (Gregory P. Smith) Date: Tue, 17 Sep 2013 23:43:57 -0700 Subject: [Python-Dev] Compiler for the Mac OS X version of Python 3.4 In-Reply-To: <CAO41-mMUQD=K7WbSPcHSZk6jh_K3zv=Aa-YAGL2dRrwckCqcOg@mail.gmail.com> References: <70C99F87-E9A5-4838-A1E9-4739FBF2E6A5@gmail.com> <nad-EC5D57.13320314092013@news.gmane.org> <B3293155-E4D5-4389-A555-C31BC49CE539@gmail.com> <rowen-8E335B.12310916092013@news.gmane.org> <51403.1379361402@parc.com> <CAO41-mMUQD=K7WbSPcHSZk6jh_K3zv=Aa-YAGL2dRrwckCqcOg@mail.gmail.com> Message-ID: <CAGE7PNJYjxhu2YkLyiUcrm8PdhYUmMLOGTYzc1WXm3K2+tqgVQ@mail.gmail.com> Just drop support for 10.6 with Python 3.4. Problem solved. People on that old of a version of the OS can build their own Python 3.4 or do the right thing and upgrade or just install Linux. This isn't Windows. Compiler tool chains are freely available for the legacy platform. We don't need to maintain such a long legacy support tail there ourselves. -gps On Mon, Sep 16, 2013 at 1:28 PM, Ryan Gonzalez <rymg19 at gmail.com> wrote: > Meh...I hate it when tools download stuff without me noticing. > > Honestly, a separate 10.6 build would work well. Plus, if a new Clang > versions includes some awesome feature that could make Python builds > better, you'd be able to take advantage of it better. > > > On Mon, Sep 16, 2013 at 2:56 PM, Bill Janssen <janssen at parc.com> wrote: > >> Russell E. Owen <rowen at uw.edu> wrote: >> >> > In article <B3293155-E4D5-4389-A555-C31BC49CE539 at gmail.com>, >> > Raymond Hettinger <raymond.hettinger at gmail.com> wrote: >> > >> > > On Sep 14, 2013, at 1:32 PM, Ned Deily <nad at acm.org> wrote: >> > > > The >> > > > most recent Developer Tools for 10.8 and 10.7 systems, Xcode 4.6.x, >> have >> > > > a mature clang but do not provide a 10.6 SDK. Even with using an >> SDK, >> > > > it's still possible to end up inadvertently linking with the wrong >> > > > versions of system libraries. We have been burned by that in the >> past. >> > > >> > > I think we should offer a separate Mac build just for 10.6 >> > > (much like we do for the 32-bit PPC option for 10.5). >> > >> > If Apple drops support for gcc in 10.9 I guess we have to go this route, >> >> Could go the Sage route -- Sage first checks for an up-to-date version >> of gcc, and downloads it and builds it for its own use if necessary. >> >> Bill >> >> > but please be careful. Every time you add a new version of python for >> > MacOS X it means that folks providing binary installers (e.g. for numpy) >> > have to provide another binary, and folks using those installers have >> > another chance of picking the wrong one. >> > >> > If you do make a 10.6-only installer, what is the minimum version of >> > MacOS X the modern compiler would support? 10.7 gives a more measured >> > upgrade path, but 10.8 gives a better compiler. >> > >> > -- Russell >> > >> > _______________________________________________ >> > Python-Dev mailing list >> > Python-Dev at python.org >> > https://mail.python.org/mailman/listinfo/python-dev >> > Unsubscribe: >> https://mail.python.org/mailman/options/python-dev/bill%40janssen.org >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> https://mail.python.org/mailman/options/python-dev/rymg19%40gmail.com >> > > > > -- > Ryan > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/greg%40krypto.org > > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130917/bc7b8df1/attachment.html> From victor.stinner at gmail.com Wed Sep 18 10:14:17 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 18 Sep 2013 10:14:17 +0200 Subject: [Python-Dev] benchmark In-Reply-To: <CAMpsgwa=Tq3OtbQVn6ieO3fp77tHX-ywoNqBXa_sY9gxhOMWDw@mail.gmail.com> References: <CAMpsgwa=Tq3OtbQVn6ieO3fp77tHX-ywoNqBXa_sY9gxhOMWDw@mail.gmail.com> Message-ID: <CAMpsgwY7-GhgEfC32nWAX+wyQftCN9ABTM6qESPzQ8iC10DZew@mail.gmail.com> Oops, I wanted to send the patch to myself, not to the python-dev mailing list, sorry :-) (It's a set of patch to try to optimize the tracemalloc module.) Victor 2013/9/18 Victor Stinner <victor.stinner at gmail.com>: > From solipsis at pitrou.net Wed Sep 18 10:26:06 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 18 Sep 2013 10:26:06 +0200 Subject: [Python-Dev] PEP 450 adding statistics module References: <520C2E23.40405@pearwood.info> <52215BDF.2090109@pearwood.info> <20130908123706.GG26319@ando> <74b03474-ab18-40c5-8ce6-de5159d34903@email.android.com> <522CA085.6000601@stoneleaf.us> <CAP7+vJ+TTaU+ntaLT3xEU2wvFQhgP4P2-FnZUJ7V2PAAy3SMEA@mail.gmail.com> <CAP7+vJJ6QxcUwcpdt3Mq6r07Q2dGUQt-VWGdsYGLv4wrRW5dGg@mail.gmail.com> <20130914005921.GK16820@ando> <CAP7+vJL81VR=kdO+2rCz3Di3GVNSwhkRCXrw2HAL=7ywT42rvA@mail.gmail.com> <20130916235912.GG19939@ando> <CAP7+vJKPDCZd4ZQn6mSkiAx7=wjv7WmmP3HjbOj-FLGUY-yNRg@mail.gmail.com> <CAP7+vJ+rtwSmKXG4Pwq7jD7Xegdka6dGhGVKW8Lf3PtWLCCpXA@mail.gmail.com> <5238CC45.2070104@stoneleaf.us> Message-ID: <20130918102606.45e0f175@pitrou.net> Le Tue, 17 Sep 2013 14:40:21 -0700, Ethan Furman <ethan at stoneleaf.us> a ?crit : > On 09/17/2013 02:21 PM, Guido van Rossum wrote: > > Congrats, I've accepted the PEP. Nice work! Please work with the > > reviewers on the issue on the code. > > Congratulations, Stephen! Or Steven even ;) From solipsis at pitrou.net Wed Sep 18 10:26:43 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 18 Sep 2013 10:26:43 +0200 Subject: [Python-Dev] PEP 428: Pathlib -> stat caching References: <5235BA20.8010105@stoneleaf.us> <20130916101508.135926de@pitrou.net> <CAH_1eM1uo0W29xy-pJPOpe-Orfc830M3BUd3coBO1dDdHSxekg@mail.gmail.com> <20130916214536.55907f19@fsol> <CAP1=2W5yq_t2iMiuptkW5w7x7kiowVCFXM4qQUEHug1QVAx6ug@mail.gmail.com> <20130916220546.341e369f@fsol> <BEA6DBE1-BB54-44A7-9E83-CD9AA8CB533A@underboss.org> Message-ID: <20130918102643.7be29a61@pitrou.net> Le Tue, 17 Sep 2013 18:10:53 -0700, Philip Jenvey <pjenvey at underboss.org> a ?crit : > > On Sep 16, 2013, at 1:05 PM, Antoine Pitrou wrote: > > > On Mon, 16 Sep 2013 15:48:54 -0400 > > Brett Cannon <brett at python.org> wrote: > >>> > >>> So I would like to propose the following API change: > >>> > >>> - Path.stat() (and stat-accessing methods such as get_mtime()...) > >>> returns an uncached stat object by default > >>> > >>> - Path.cache_stat() can be called to return the stat() *and* > >>> cache it for future use, such that any future call to stat(), > >>> cache_stat() or a stat-accessing function reuses that cached stat > >>> > >>> In other words, only if you use cache_stat() at least once is the > >>> stat() value cached and reused by the Path object. > >>> (also, it's a per-Path decision) > >>> > >> > >> Any reason why stat() can't get a keyword-only cached=True argument > >> instead? Or have stat() never cache() but stat_cache() always so > >> that people can choose if they want fresh or cached based on API > >> and not whether some library happened to make a decision for them? > > > > 1. Because you also want the helper functions (get_mtime(), etc.) to > > cache the value too. It's not only about stat(). > > With the proposed rich stat object the convenience methods living on > Path wouldn't result in much added convenience: > > p.is_dir() vs p.stat().is_dir() One reason is that the proposed rich stat object doesn't exist yet :-) Regards Antoine. From g.brandl at gmx.net Wed Sep 18 10:54:28 2013 From: g.brandl at gmx.net (Georg Brandl) Date: Wed, 18 Sep 2013 10:54:28 +0200 Subject: [Python-Dev] License() release list is imcomplete; intentional? In-Reply-To: <l19t08$uj0$1@ger.gmane.org> References: <l19t08$uj0$1@ger.gmane.org> Message-ID: <l1bpmn$j54$1@ger.gmane.org> On 09/17/2013 05:37 PM, Terry Reedy wrote: > On 2.7, >>> license() return a text that includes a complete list of > releases from 1.6 to 2.7 and stops there > Release Derived Year Owner GPL- > from compatible? (1) > > 0.9.0 thru 1.2 1991-1995 CWI yes > 1.3 thru 1.5.2 1.2 1995-1999 CNRI yes > 1.6 1.5.2 2000 CNRI no > 2.0 1.6 2000 BeOpen.com no > ... > 2.6.5 2.6.4 2010 PSF yes > 2.7 2.6 2010 PSF yes > > Was it intentional to stop with 2.7 and not continue with 2.7.1, etc? > > On 3.3.2, the 2.x list ends with 2.6.5 and never mentions 2.7. Intentional? > It then jumps back to 3.0 and ends with the 'previous' release, 3.3.1. > Should 3.3.2 be included in the 3.3.2 list? > > ... > 2.6.4 2.6.3 2009 PSF yes > 2.6.5 2.6.4 2010 PSF yes > 3.0 2.6 2008 PSF yes > 3.0.1 3.0 2009 PSF yes > ... > 3.2.4 3.2.3 2013 PSF yes > 3.3.0 3.2 2012 PSF yes > 3.3.1 3.3.0 2013 PSF yes Since there are 3 versions of this table, it's unavoidable that one of them gets out of sync :) * LICENSE * Doc/license.rst * the versions for each release on the website So I wholeheartedly support truncating before 2.2 (which is the first release where all micro versions are PSF owned and GPL compatible) and just saying "2.2 onwards" (or whatever is the best way to say it). cheers, Georg From ethan at stoneleaf.us Wed Sep 18 11:55:10 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 18 Sep 2013 02:55:10 -0700 Subject: [Python-Dev] PEP 450 adding statistics module In-Reply-To: <20130918102606.45e0f175@pitrou.net> References: <520C2E23.40405@pearwood.info> <52215BDF.2090109@pearwood.info> <20130908123706.GG26319@ando> <74b03474-ab18-40c5-8ce6-de5159d34903@email.android.com> <522CA085.6000601@stoneleaf.us> <CAP7+vJ+TTaU+ntaLT3xEU2wvFQhgP4P2-FnZUJ7V2PAAy3SMEA@mail.gmail.com> <CAP7+vJJ6QxcUwcpdt3Mq6r07Q2dGUQt-VWGdsYGLv4wrRW5dGg@mail.gmail.com> <20130914005921.GK16820@ando> <CAP7+vJL81VR=kdO+2rCz3Di3GVNSwhkRCXrw2HAL=7ywT42rvA@mail.gmail.com> <20130916235912.GG19939@ando> <CAP7+vJKPDCZd4ZQn6mSkiAx7=wjv7WmmP3HjbOj-FLGUY-yNRg@mail.gmail.com> <CAP7+vJ+rtwSmKXG4Pwq7jD7Xegdka6dGhGVKW8Lf3PtWLCCpXA@mail.gmail.com> <5238CC45.2070104@stoneleaf.us> <20130918102606.45e0f175@pitrou.net> Message-ID: <5239787E.8080408@stoneleaf.us> On 09/18/2013 01:26 AM, Antoine Pitrou wrote: > Le Tue, 17 Sep 2013 14:40:21 -0700, > Ethan Furman <ethan at stoneleaf.us> a ?crit : >> On 09/17/2013 02:21 PM, Guido van Rossum wrote: >>> Congrats, I've accepted the PEP. Nice work! Please work with the >>> reviewers on the issue on the code. >> >> Congratulations, Stephen! > > Or Steven even ;) *sigh* I used to be a good speller, too! -- ~Ethan~ From martin at v.loewis.de Wed Sep 18 14:54:32 2013 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 18 Sep 2013 14:54:32 +0200 Subject: [Python-Dev] Compiler for the Mac OS X version of Python 3.4 In-Reply-To: <CAGE7PNJYjxhu2YkLyiUcrm8PdhYUmMLOGTYzc1WXm3K2+tqgVQ@mail.gmail.com> References: <70C99F87-E9A5-4838-A1E9-4739FBF2E6A5@gmail.com> <nad-EC5D57.13320314092013@news.gmane.org> <B3293155-E4D5-4389-A555-C31BC49CE539@gmail.com> <rowen-8E335B.12310916092013@news.gmane.org> <51403.1379361402@parc.com> <CAO41-mMUQD=K7WbSPcHSZk6jh_K3zv=Aa-YAGL2dRrwckCqcOg@mail.gmail.com> <CAGE7PNJYjxhu2YkLyiUcrm8PdhYUmMLOGTYzc1WXm3K2+tqgVQ@mail.gmail.com> Message-ID: <5239A288.2050403@v.loewis.de> Am 18.09.13 08:43, schrieb Gregory P. Smith: > Just drop support for 10.6 with Python 3.4. Problem solved. People on > that old of a version of the OS can build their own Python 3.4 or do the > right thing and upgrade or just install Linux. > > This isn't Windows. Compiler tool chains are freely available for the > legacy platform. We don't need to maintain such a long legacy support > tail there ourselves. I don't mind such a decision in principle, but also in principle, I'd prefer if there was a pre-set policy to decide this question, documented in PEP 11. Here a piece of OSX release history: - 10.5: October 2007 * 10.5.8: August 2009 - 10.6: August 2009 * 10.6.8: July 2011 - 10.7: July 2011 * 10.7.5: July 2012 - 10.8: July 2012 So possible policy that would now exclude 10.6 for binary installers would be: - only the two latest feature releases are supported - only feature releases younger than 3 years are supported Note that a separate policy should be added to decide whether support for older versions is actively removed from source code (or equally bug fixes for old versions are not accepted anymore). Regards, Martin From martin at v.loewis.de Wed Sep 18 15:03:19 2013 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 18 Sep 2013 15:03:19 +0200 Subject: [Python-Dev] Compiler for the Mac OS X version of Python 3.4 In-Reply-To: <534e7dd6-e7e2-49cd-b719-8a070852b623@email.android.com> References: <70C99F87-E9A5-4838-A1E9-4739FBF2E6A5@gmail.com> <nad-EC5D57.13320314092013@news.gmane.org> <B3293155-E4D5-4389-A555-C31BC49CE539@gmail.com> <534e7dd6-e7e2-49cd-b719-8a070852b623@email.android.com> Message-ID: <5239A497.4070204@v.loewis.de> Am 15.09.13 00:56, schrieb Ryan: > +1. A 10.6-only build makes sense. I'd like to support Russell's point: this could put a burden on everyone releasing extension modules to also provide two binary releases, which e.g. would then mess up downloads from PyPI. So -1. +0 on dropping 10.6 support from the binary installers as proposed by Greg. Regards, Martin From solipsis at pitrou.net Wed Sep 18 15:36:44 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 18 Sep 2013 15:36:44 +0200 Subject: [Python-Dev] Compiler for the Mac OS X version of Python 3.4 References: <70C99F87-E9A5-4838-A1E9-4739FBF2E6A5@gmail.com> <nad-EC5D57.13320314092013@news.gmane.org> <B3293155-E4D5-4389-A555-C31BC49CE539@gmail.com> <rowen-8E335B.12310916092013@news.gmane.org> <51403.1379361402@parc.com> <CAO41-mMUQD=K7WbSPcHSZk6jh_K3zv=Aa-YAGL2dRrwckCqcOg@mail.gmail.com> <CAGE7PNJYjxhu2YkLyiUcrm8PdhYUmMLOGTYzc1WXm3K2+tqgVQ@mail.gmail.com> <5239A288.2050403@v.loewis.de> Message-ID: <20130918153644.73eae75f@pitrou.net> Le Wed, 18 Sep 2013 14:54:32 +0200, "Martin v. L?wis" <martin at v.loewis.de> a ?crit : > Am 18.09.13 08:43, schrieb Gregory P. Smith: > > Just drop support for 10.6 with Python 3.4. Problem solved. People > > on that old of a version of the OS can build their own Python 3.4 > > or do the right thing and upgrade or just install Linux. > > > > This isn't Windows. Compiler tool chains are freely available for > > the legacy platform. We don't need to maintain such a long legacy > > support tail there ourselves. > > I don't mind such a decision in principle, but also in principle, I'd > prefer if there was a pre-set policy to decide this question, > documented in PEP 11. > > Here a piece of OSX release history: > - 10.5: October 2007 > * 10.5.8: August 2009 > - 10.6: August 2009 > * 10.6.8: July 2011 > - 10.7: July 2011 > * 10.7.5: July 2012 > - 10.8: July 2012 This means that any Mac shipped pre-July 2011 has 10.6 or lower? I know Mac users are fashion victims, but 2011 doesn't sound that old to me ;-) (it's even younger than Barry!) Regards Antoine. From ronaldoussoren at mac.com Wed Sep 18 15:41:56 2013 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Wed, 18 Sep 2013 13:41:56 +0000 (GMT) Subject: [Python-Dev] Compiler for the Mac OS X version of Python 3.4 In-Reply-To: <20130918153644.73eae75f@pitrou.net> Message-ID: <2f235f3a-af73-48b0-9d9f-26047f7175ad@me.com> On Sep 18, 2013, at 03:36 PM, Antoine Pitrou <solipsis at pitrou.net> wrote: Le Wed, 18 Sep 2013 14:54:32 +0200, "Martin v. L?wis" <martin at v.loewis.de> a ?crit : Am 18.09.13 08:43, schrieb Gregory P. Smith: > Just drop support for 10.6 with Python 3.4. Problem solved. People > on that old of a version of the OS can build their own Python 3.4 > or do the right thing and upgrade or just install Linux. > > This isn't Windows. Compiler tool chains are freely available for > the legacy platform. We don't need to maintain such a long legacy > support tail there ourselves. I don't mind such a decision in principle, but also in principle, I'd prefer if there was a pre-set policy to decide this question, documented in PEP 11. Here a piece of OSX release history: - 10.5: October 2007 * 10.5.8: August 2009 - 10.6: August 2009 * 10.6.8: July 2011 - 10.7: July 2011 * 10.7.5: July 2012 - 10.8: July 2012 This means that any Mac shipped pre-July 2011 has 10.6 or lower? I know Mac users are fashion victims, but 2011 doesn't sound that old to me ;-) (it's even younger than Barry!) ? That's correct, and IMHO it is too early to drop support for 10.6 and doubly so when it is only done to use a newer compiler for the binary build. Still-using-10.5-in-production-ly yours, ? ? Ronald Regards Antoine. _______________________________________________ Python-Dev mailing list Python-Dev at python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/ronaldoussoren%40mac.com -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130918/ef1a6b42/attachment.html> From rowen at uw.edu Wed Sep 18 15:58:01 2013 From: rowen at uw.edu (Russell Owen) Date: Wed, 18 Sep 2013 06:58:01 -0700 Subject: [Python-Dev] Compiler for the Mac OS X version of Python 3.4 In-Reply-To: <5239A288.2050403@v.loewis.de> References: <70C99F87-E9A5-4838-A1E9-4739FBF2E6A5@gmail.com> <nad-EC5D57.13320314092013@news.gmane.org> <B3293155-E4D5-4389-A555-C31BC49CE539@gmail.com> <rowen-8E335B.12310916092013@news.gmane.org> <51403.1379361402@parc.com> <CAO41-mMUQD=K7WbSPcHSZk6jh_K3zv=Aa-YAGL2dRrwckCqcOg@mail.gmail.com> <CAGE7PNJYjxhu2YkLyiUcrm8PdhYUmMLOGTYzc1WXm3K2+tqgVQ@mail.gmail.com> <5239A288.2050403@v.loewis.de> Message-ID: <9829E582-9A68-4679-8087-31D42069F20D@uw.edu> I agree that a policy is a good idea, and I suggest it be primarily based on age, since we cannot assume Apple will release new versions of the OS on a given timeline. I personally think too early to drop support for MacOS X 10.6 and am on the edge about 10.5. -- Russell On Sep 18, 2013, at 5:54 AM, "Martin v. L?wis" <martin at v.loewis.de> wrote: > Am 18.09.13 08:43, schrieb Gregory P. Smith: >> Just drop support for 10.6 with Python 3.4. Problem solved. People on >> that old of a version of the OS can build their own Python 3.4 or do the >> right thing and upgrade or just install Linux. >> >> This isn't Windows. Compiler tool chains are freely available for the >> legacy platform. We don't need to maintain such a long legacy support >> tail there ourselves. > > I don't mind such a decision in principle, but also in principle, I'd > prefer if there was a pre-set policy to decide this question, documented > in PEP 11. > > Here a piece of OSX release history: > - 10.5: October 2007 > * 10.5.8: August 2009 > - 10.6: August 2009 > * 10.6.8: July 2011 > - 10.7: July 2011 > * 10.7.5: July 2012 > - 10.8: July 2012 > > So possible policy that would now exclude 10.6 for binary installers > would be: > > - only the two latest feature releases are supported > - only feature releases younger than 3 years are supported > > Note that a separate policy should be added to decide whether support > for older versions is actively removed from source code (or equally > bug fixes for old versions are not accepted anymore). > > Regards, > Martin > From ronaldoussoren at mac.com Wed Sep 18 15:38:48 2013 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Wed, 18 Sep 2013 13:38:48 +0000 (GMT) Subject: [Python-Dev] Compiler for the Mac OS X version of Python 3.4 In-Reply-To: <5239A497.4070204@v.loewis.de> Message-ID: <8279a538-028f-45ae-a762-25302c52e370@me.com> On Sep 18, 2013, at 03:03 PM, "Martin v. L?wis" <martin at v.loewis.de> wrote: Am 15.09.13 00:56, schrieb Ryan: +1. A 10.6-only build makes sense. I'd like to support Russell's point: this could put a burden on everyone releasing extension modules to also provide two binary releases, which e.g. would then mess up downloads from PyPI. So -1. +0 on dropping 10.6 support from the binary installers as proposed by Greg. ? Dropping support for 10.6 isn't really necessary, just build (carefully) on a later release of OSX (preferably the last one to pick up the most recent developer tools). ?? I'll try to provide a more detailed response tonight, but the basic idea is to use OSX 10.8, the current Xcode release and the OSX 10.8 SDK to do the build. This is a supported configuration for building binaries that target older OSX releases, but some care is needed with the configure script for 3th-party libraries: those might pick up features that aren't available in 10.6.? Ronald Regards, Martin _______________________________________________ Python-Dev mailing list Python-Dev at python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/ronaldoussoren%40mac.com -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130918/e50ddff8/attachment.html> From guido at python.org Wed Sep 18 16:57:22 2013 From: guido at python.org (Guido van Rossum) Date: Wed, 18 Sep 2013 07:57:22 -0700 Subject: [Python-Dev] License() release list is imcomplete; intentional? In-Reply-To: <l1bpmn$j54$1@ger.gmane.org> References: <l19t08$uj0$1@ger.gmane.org> <l1bpmn$j54$1@ger.gmane.org> Message-ID: <CAP7+vJJLZ93sEs3nR6NHy7m8A_14e54qOO-8iA9p1=N77t7BhA@mail.gmail.com> OK. Let's do it. On Wed, Sep 18, 2013 at 1:54 AM, Georg Brandl <g.brandl at gmx.net> wrote: > On 09/17/2013 05:37 PM, Terry Reedy wrote: > > On 2.7, >>> license() return a text that includes a complete list of > > releases from 1.6 to 2.7 and stops there > > Release Derived Year Owner GPL- > > from compatible? (1) > > > > 0.9.0 thru 1.2 1991-1995 CWI yes > > 1.3 thru 1.5.2 1.2 1995-1999 CNRI yes > > 1.6 1.5.2 2000 CNRI no > > 2.0 1.6 2000 BeOpen.com no > > ... > > 2.6.5 2.6.4 2010 PSF yes > > 2.7 2.6 2010 PSF yes > > > > Was it intentional to stop with 2.7 and not continue with 2.7.1, etc? > > > > On 3.3.2, the 2.x list ends with 2.6.5 and never mentions 2.7. > Intentional? > > It then jumps back to 3.0 and ends with the 'previous' release, 3.3.1. > > Should 3.3.2 be included in the 3.3.2 list? > > > > ... > > 2.6.4 2.6.3 2009 PSF yes > > 2.6.5 2.6.4 2010 PSF yes > > 3.0 2.6 2008 PSF yes > > 3.0.1 3.0 2009 PSF yes > > ... > > 3.2.4 3.2.3 2013 PSF yes > > 3.3.0 3.2 2012 PSF yes > > 3.3.1 3.3.0 2013 PSF yes > > Since there are 3 versions of this table, it's unavoidable that one of them > gets out of sync :) > > * LICENSE > * Doc/license.rst > * the versions for each release on the website > > So I wholeheartedly support truncating before 2.2 (which is the first > release where all micro versions are PSF owned and GPL compatible) and > just saying "2.2 onwards" (or whatever is the best way to say it). > > cheers, > Georg > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130918/5466a52e/attachment.html> From skip at pobox.com Wed Sep 18 17:04:45 2013 From: skip at pobox.com (Skip Montanaro) Date: Wed, 18 Sep 2013 10:04:45 -0500 Subject: [Python-Dev] Compiler for the Mac OS X version of Python 3.4 In-Reply-To: <8279a538-028f-45ae-a762-25302c52e370@me.com> References: <5239A497.4070204@v.loewis.de> <8279a538-028f-45ae-a762-25302c52e370@me.com> Message-ID: <CANc-5UwCtaFGJ40PVkKzE+PtroRxz=yQ0coQU5irVoALH5Okog@mail.gmail.com> As a MacBook Pro user running Snow Leopard/10.6.8, I would find the lack of a binary release problematic, were it not for the fact that I routinely build from source (and don't do anything Mac-specific with Python). For those not familiar with the platform, it's perhaps worth noting that upgrade is out of my hands. Apple dropped support for my "ancient" MBP (2006?), so 10.6.8 is as up-to-date as I can get. I guess that's what you expect from a company whose main revenues these days come from cell phones and tablets, and addition of pastels to their color pallet warrants announcement in a TV commercial. I suspect other Mac users stuck on Snow Leopard who are not Python developers would rue the lack of binary installers more than me. Skip From janssen at parc.com Wed Sep 18 18:31:09 2013 From: janssen at parc.com (Bill Janssen) Date: Wed, 18 Sep 2013 09:31:09 -0700 Subject: [Python-Dev] Compiler for the Mac OS X version of Python 3.4 In-Reply-To: <9829E582-9A68-4679-8087-31D42069F20D@uw.edu> References: <70C99F87-E9A5-4838-A1E9-4739FBF2E6A5@gmail.com> <nad-EC5D57.13320314092013@news.gmane.org> <B3293155-E4D5-4389-A555-C31BC49CE539@gmail.com> <rowen-8E335B.12310916092013@news.gmane.org> <51403.1379361402@parc.com> <CAO41-mMUQD=K7WbSPcHSZk6jh_K3zv=Aa-YAGL2dRrwckCqcOg@mail.gmail.com> <CAGE7PNJYjxhu2YkLyiUcrm8PdhYUmMLOGTYzc1WXm3K2+tqgVQ@mail.gmail.com> <5239A288.2050403@v.loewis.de> <9829E582-9A68-4679-8087-31D42069F20D@uw.edu> Message-ID: <71841.1379521869@parc.com> Russell Owen <rowen at uw.edu> wrote: > I agree that a policy is a good idea, and I suggest it be primarily based on age, since we cannot assume Apple will release new versions of the OS on a given timeline. > > I personally think too early to drop support for MacOS X 10.6 and am on the edge about 10.5. I run one 10.5 machine, three 10.6 machines, and 5+ 10.7+ machines right now. I have a perfectly fine and fast Mac Pro which can't even be upgraded past 10.7. So using Apple's OS release numbers as the gating factor might be a problem. I'd be happy to drop pre-built binary installers for 10.6, but that's different from dropping all support for it. Bill > -- Russell > > On Sep 18, 2013, at 5:54 AM, "Martin v. L?wis" <martin at v.loewis.de> wrote: > > > Am 18.09.13 08:43, schrieb Gregory P. Smith: > >> Just drop support for 10.6 with Python 3.4. Problem solved. People on > >> that old of a version of the OS can build their own Python 3.4 or do the > >> right thing and upgrade or just install Linux. > >> > >> This isn't Windows. Compiler tool chains are freely available for the > >> legacy platform. We don't need to maintain such a long legacy support > >> tail there ourselves. > > > > I don't mind such a decision in principle, but also in principle, I'd > > prefer if there was a pre-set policy to decide this question, documented > > in PEP 11. > > > > Here a piece of OSX release history: > > - 10.5: October 2007 > > * 10.5.8: August 2009 > > - 10.6: August 2009 > > * 10.6.8: July 2011 > > - 10.7: July 2011 > > * 10.7.5: July 2012 > > - 10.8: July 2012 > > > > So possible policy that would now exclude 10.6 for binary installers > > would be: > > > > - only the two latest feature releases are supported > > - only feature releases younger than 3 years are supported > > > > Note that a separate policy should be added to decide whether support > > for older versions is actively removed from source code (or equally > > bug fixes for old versions are not accepted anymore). > > > > Regards, > > Martin > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/bill%40janssen.org From stephen at xemacs.org Wed Sep 18 18:35:03 2013 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Thu, 19 Sep 2013 01:35:03 +0900 Subject: [Python-Dev] Compiler for the Mac OS X version of Python 3.4 In-Reply-To: <CANc-5UwCtaFGJ40PVkKzE+PtroRxz=yQ0coQU5irVoALH5Okog@mail.gmail.com> References: <5239A497.4070204@v.loewis.de> <8279a538-028f-45ae-a762-25302c52e370@me.com> <CANc-5UwCtaFGJ40PVkKzE+PtroRxz=yQ0coQU5irVoALH5Okog@mail.gmail.com> Message-ID: <87zjra3x6w.fsf@uwakimon.sk.tsukuba.ac.jp> Skip Montanaro writes: > I suspect other Mac users stuck on Snow Leopard who are not Python > developers would rue the lack of binary installers more than me. Perhaps, but if Python current as of now isn't good enough for someone for the foreseeable future, aren't they going to want up-to-date libraries to backup the Python modules that use them, not to mention security fixes and the like? That's why I get my Python (for Snow Leopard) from MacPorts. I wouldn't hesitate to build it myself, but I get timely upgrades from MacPorts so there's no need to bother. I'd be far less likely to upgrade quickly (for bugfix releases) if I had to make a special trip to python.org for installers. The MacPorts build could break at any time, I suppose, but at least for the moment there are alternatives to building it yourself. From skip at pobox.com Wed Sep 18 18:44:12 2013 From: skip at pobox.com (Skip Montanaro) Date: Wed, 18 Sep 2013 11:44:12 -0500 Subject: [Python-Dev] Compiler for the Mac OS X version of Python 3.4 In-Reply-To: <87zjra3x6w.fsf@uwakimon.sk.tsukuba.ac.jp> References: <5239A497.4070204@v.loewis.de> <8279a538-028f-45ae-a762-25302c52e370@me.com> <CANc-5UwCtaFGJ40PVkKzE+PtroRxz=yQ0coQU5irVoALH5Okog@mail.gmail.com> <87zjra3x6w.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <CANc-5UzNGSEdAHOHno7Tx_Z_zjD5WwbLuZ-icdonhzEiCwVNpw@mail.gmail.com> > That's why I get my Python (for Snow Leopard) from MacPorts. Unless things have changed, that probably doesn't support Mac-specific stuff, does it? I was thinking more of non-developer users who are likely to need/want Mac-specific interfaces for tools which are written in Python. That might just shift the burden to the developers of such tools. My guess is that there is a domino effect. Apple stops supporting Snow Leopard, which, all other things considered, forces third-party developers who use Python in their products to stop supporting it, which then pressures their users to buy new hardware so they can run newer versions of those developers' tools. This is good for Apple, but creates headaches and monetary costs for developers and users downstream. S From nad at acm.org Wed Sep 18 19:09:55 2013 From: nad at acm.org (Ned Deily) Date: Wed, 18 Sep 2013 10:09:55 -0700 Subject: [Python-Dev] Compiler for the Mac OS X version of Python 3.4 References: <20130918153644.73eae75f@pitrou.net> <2f235f3a-af73-48b0-9d9f-26047f7175ad@me.com> Message-ID: <nad-87D0F5.10095518092013@news.gmane.org> Thank you all for your comments so far on this subject. I have noted two separate issues raised here: one, how to build the Pythons provided by binary installers to get optimum performance (i.e. use more recent compilers); and, two, what OS X releases should we support with binary installers. As I noted earlier, I've opened Issue19019 and I will update it with concrete proposals when we've complete the necessary testing and fixing of various build configurations, including of the sort Ronald and I outlined. If you are interested in the details of this, please move that discussion to the bug tracker. As to point two, I will put a stake in the ground here and declare that we will continue to support 10.6 with 3.4 batteries-included installers. For various reasons, 10.6 remains surprisingly popular (at a recent Python hackathon meetup in San Francisco, every person I helped who had a Mac was running 10.6) and it is not that old even by Apple standards. There are tradeoffs in how best to provide that support. Among those tradeoffs are the impacts to those who provide binary packages for extension modules and third-party libraries, as Russell notes. Again, after investigating and testing the nitty-gritty details, if it seems that a change in how we provide installers is warranted, we can discuss that on Issue19019 and report back here prior to any final decision. Also, at that time, it would be appropriate to consider a policy for long-term support of OS X releases; it's a bit premature to do so now. -- Ned Deily, nad at acm.org From stephen at xemacs.org Wed Sep 18 19:20:13 2013 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Thu, 19 Sep 2013 02:20:13 +0900 Subject: [Python-Dev] Compiler for the Mac OS X version of Python 3.4 In-Reply-To: <CANc-5UzNGSEdAHOHno7Tx_Z_zjD5WwbLuZ-icdonhzEiCwVNpw@mail.gmail.com> References: <5239A497.4070204@v.loewis.de> <8279a538-028f-45ae-a762-25302c52e370@me.com> <CANc-5UwCtaFGJ40PVkKzE+PtroRxz=yQ0coQU5irVoALH5Okog@mail.gmail.com> <87zjra3x6w.fsf@uwakimon.sk.tsukuba.ac.jp> <CANc-5UzNGSEdAHOHno7Tx_Z_zjD5WwbLuZ-icdonhzEiCwVNpw@mail.gmail.com> Message-ID: <87wqme3v3m.fsf@uwakimon.sk.tsukuba.ac.jp> Skip Montanaro writes: > > That's why I get my Python (for Snow Leopard) from MacPorts. > > Unless things have changed, that probably doesn't support Mac-specific > stuff, does it? You mean in the Python port or in general? MacPorts supports Mac-specific APIs in a number of ports where upstream does. In Python, I don't know. I would assume that anything python.org supports is supported by MacPorts, though. From nad at acm.org Wed Sep 18 20:11:43 2013 From: nad at acm.org (Ned Deily) Date: Wed, 18 Sep 2013 11:11:43 -0700 Subject: [Python-Dev] Compiler for the Mac OS X version of Python 3.4 References: <5239A497.4070204@v.loewis.de> <8279a538-028f-45ae-a762-25302c52e370@me.com> <CANc-5UwCtaFGJ40PVkKzE+PtroRxz=yQ0coQU5irVoALH5Okog@mail.gmail.com> <87zjra3x6w.fsf@uwakimon.sk.tsukuba.ac.jp> <CANc-5UzNGSEdAHOHno7Tx_Z_zjD5WwbLuZ-icdonhzEiCwVNpw@mail.gmail.com> <87wqme3v3m.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <nad-97155F.11114318092013@news.gmane.org> In article <87wqme3v3m.fsf at uwakimon.sk.tsukuba.ac.jp>, "Stephen J. Turnbull" <stephen at xemacs.org> wrote: > Skip Montanaro writes: > > > That's why I get my Python (for Snow Leopard) from MacPorts. > > Unless things have changed, that probably doesn't support Mac-specific > > stuff, does it? > You mean in the Python port or in general? MacPorts supports > Mac-specific APIs in a number of ports where upstream does. > > In Python, I don't know. I would assume that anything python.org > supports is supported by MacPorts, though. Yes, there are no significant differences in feature support between a python.org Python and a MacPorts-supplied Python or a Homebrew-supplied Python or a Fink-supplied Python. If you are doing any sort of a hard-core Python development work that requires the use of third-party C libraries et al, you are probably better off using a complete solution, including Python, from one of the above big-three package managers for OS X. If you need to support multiple versions of OS X, IME, MacPorts does the best job of supporting older versions (I have up-to-date MacPorts installations on OS X 10.4 through 10.9). But the primary mission for python.org installers is somewhat different: provide up-to-date, easy-to-install, batteries-included Pythons for users who are not necessarily experienced with Python itself. People who need development environments with significant third-party libraries not included with Python or OS X itself are often better served by either investing in the time to set up and become familiar with using one of the general open source distributors (MacPorts, Homebrew, et al) or by using a specialized Python distribution (e.g. for scientific users). -- Ned Deily, nad at acm.org From g.brandl at gmx.net Thu Sep 19 03:01:10 2013 From: g.brandl at gmx.net (Georg Brandl) Date: Thu, 19 Sep 2013 03:01:10 +0200 Subject: [Python-Dev] License() release list is imcomplete; intentional? In-Reply-To: <CAP7+vJJLZ93sEs3nR6NHy7m8A_14e54qOO-8iA9p1=N77t7BhA@mail.gmail.com> References: <l19t08$uj0$1@ger.gmane.org> <l1bpmn$j54$1@ger.gmane.org> <CAP7+vJJLZ93sEs3nR6NHy7m8A_14e54qOO-8iA9p1=N77t7BhA@mail.gmail.com> Message-ID: <l1dib8$oii$1@ger.gmane.org> Now tracked at http://bugs.python.org/issue19043 Georg On 09/18/2013 04:57 PM, Guido van Rossum wrote: > OK. Let's do it. > > > On Wed, Sep 18, 2013 at 1:54 AM, Georg Brandl <g.brandl at gmx.net > <mailto:g.brandl at gmx.net>> wrote: > > On 09/17/2013 05:37 PM, Terry Reedy wrote: > > On 2.7, >>> license() return a text that includes a complete list of > > releases from 1.6 to 2.7 and stops there > > Release Derived Year Owner GPL- > > from compatible? (1) > > > > 0.9.0 thru 1.2 1991-1995 CWI yes > > 1.3 thru 1.5.2 1.2 1995-1999 CNRI yes > > 1.6 1.5.2 2000 CNRI no > > 2.0 1.6 2000 BeOpen.com no > > ... > > 2.6.5 2.6.4 2010 PSF yes > > 2.7 2.6 2010 PSF yes > > > > Was it intentional to stop with 2.7 and not continue with 2.7.1, etc? > > > > On 3.3.2, the 2.x list ends with 2.6.5 and never mentions 2.7. Intentional? > > It then jumps back to 3.0 and ends with the 'previous' release, 3.3.1. > > Should 3.3.2 be included in the 3.3.2 list? > > > > ... > > 2.6.4 2.6.3 2009 PSF yes > > 2.6.5 2.6.4 2010 PSF yes > > 3.0 2.6 2008 PSF yes > > 3.0.1 3.0 2009 PSF yes > > ... > > 3.2.4 3.2.3 2013 PSF yes > > 3.3.0 3.2 2012 PSF yes > > 3.3.1 3.3.0 2013 PSF yes > > Since there are 3 versions of this table, it's unavoidable that one of them > gets out of sync :) > > * LICENSE > * Doc/license.rst > * the versions for each release on the website > > So I wholeheartedly support truncating before 2.2 (which is the first > release where all micro versions are PSF owned and GPL compatible) and > just saying "2.2 onwards" (or whatever is the best way to say it). > > cheers, > Georg > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org <mailto:Python-Dev at python.org> > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > > > > > -- > --Guido van Rossum (python.org/~guido <http://python.org/~guido>) > > From barry at python.org Thu Sep 19 03:29:50 2013 From: barry at python.org (Barry Warsaw) Date: Wed, 18 Sep 2013 21:29:50 -0400 Subject: [Python-Dev] Compiler for the Mac OS X version of Python 3.4 In-Reply-To: <CANc-5UwCtaFGJ40PVkKzE+PtroRxz=yQ0coQU5irVoALH5Okog@mail.gmail.com> References: <5239A497.4070204@v.loewis.de> <8279a538-028f-45ae-a762-25302c52e370@me.com> <CANc-5UwCtaFGJ40PVkKzE+PtroRxz=yQ0coQU5irVoALH5Okog@mail.gmail.com> Message-ID: <20130918212950.0abfd469@limelight.wooz.org> On Sep 18, 2013, at 10:04 AM, Skip Montanaro wrote: >As a MacBook Pro user running Snow Leopard/10.6.8, I would find the >lack of a binary release problematic, were it not for the fact that I >routinely build from source (and don't do anything Mac-specific with >Python). For those not familiar with the platform, it's perhaps worth >noting that upgrade is out of my hands. Apple dropped support for my >"ancient" MBP (2006?), so 10.6.8 is as up-to-date as I can get. I >guess that's what you expect from a company whose main revenues these >days come from cell phones and tablets, and addition of pastels to >their color pallet warrants announcement in a TV commercial. I'm in a similar boat with my old MBP, and I am not upgrading my somewhat newer one because of compatibility with other software. But OTOH, I have never installed the binary releases, I just build them from MacPorts, so as long as that still works, I'm happy. (I have a 8mo Air running the latest Lion and will likely update that when Mavericks is out.) -Barry From barry at python.org Thu Sep 19 03:38:48 2013 From: barry at python.org (Barry Warsaw) Date: Wed, 18 Sep 2013 21:38:48 -0400 Subject: [Python-Dev] [Python-checkins] cpython (2.6): #14984: only import pwd on POSIX. In-Reply-To: <3cg1WQ2NDkz7LnP@mail.python.org> References: <3cg1WQ2NDkz7LnP@mail.python.org> Message-ID: <20130918213848.6e943835@limelight.wooz.org> On Sep 18, 2013, at 03:00 PM, r.david.murray wrote: >http://hg.python.org/cpython/rev/fb3ad8a749c8 >changeset: 85749:fb3ad8a749c8 >branch: 2.6 >parent: 85735:1b673e0fd8f3 >user: R David Murray <rdmurray at bitdance.com> >date: Wed Sep 18 08:49:25 2013 -0400 >summary: > #14984: only import pwd on POSIX. > >files: > Lib/netrc.py | 4 +++- > 1 files changed, 3 insertions(+), 1 deletions(-) > > >diff --git a/Lib/netrc.py b/Lib/netrc.py >--- a/Lib/netrc.py >+++ b/Lib/netrc.py >@@ -2,7 +2,9 @@ > > # Module and documentation by Eric S. Raymond, 21 Dec 1998 > >-import os, stat, shlex, pwd >+import os, stat, shlex >+if os.name == 'posix': >+ import pwd Would it make more sense to do: try: import pwd except ImportError: pwd = None ? From rdmurray at bitdance.com Thu Sep 19 06:09:30 2013 From: rdmurray at bitdance.com (R. David Murray) Date: Thu, 19 Sep 2013 00:09:30 -0400 Subject: [Python-Dev] [Python-checkins] cpython (2.6): #14984: only import pwd on POSIX. In-Reply-To: <20130918213848.6e943835@limelight.wooz.org> References: <3cg1WQ2NDkz7LnP@mail.python.org> <20130918213848.6e943835@limelight.wooz.org> Message-ID: <20130919040931.B21C3250925@webabinitio.net> On Wed, 18 Sep 2013 21:38:48 -0400, Barry Warsaw <barry at python.org> wrote: > On Sep 18, 2013, at 03:00 PM, r.david.murray wrote: > > >http://hg.python.org/cpython/rev/fb3ad8a749c8 > >changeset: 85749:fb3ad8a749c8 > >branch: 2.6 > >parent: 85735:1b673e0fd8f3 > >user: R David Murray <rdmurray at bitdance.com> > >date: Wed Sep 18 08:49:25 2013 -0400 > >summary: > > #14984: only import pwd on POSIX. > > > >files: > > Lib/netrc.py | 4 +++- > > 1 files changed, 3 insertions(+), 1 deletions(-) > > > > > >diff --git a/Lib/netrc.py b/Lib/netrc.py > >--- a/Lib/netrc.py > >+++ b/Lib/netrc.py > >@@ -2,7 +2,9 @@ > > > > # Module and documentation by Eric S. Raymond, 21 Dec 1998 > > > >-import os, stat, shlex, pwd > >+import os, stat, shlex > >+if os.name == 'posix': > >+ import pwd > > Would it make more sense to do: > > try: > import pwd > except ImportError: > pwd = None I don't think so. The code that uses pwd is protected by an 'if os.name == 'posix' as well. I think that's clearer than setting pwd to none and guarding that section by 'if pwd'. And in 3.4 I just moved the import into the if block that uses it. --David From greg at krypto.org Thu Sep 19 06:43:11 2013 From: greg at krypto.org (Gregory P. Smith) Date: Wed, 18 Sep 2013 21:43:11 -0700 Subject: [Python-Dev] Compiler for the Mac OS X version of Python 3.4 In-Reply-To: <8279a538-028f-45ae-a762-25302c52e370@me.com> References: <5239A497.4070204@v.loewis.de> <8279a538-028f-45ae-a762-25302c52e370@me.com> Message-ID: <CAGE7PN+QZmoAn=e5oxgAupzpwdmwaa7OktOJymS7s+HRy9vPDg@mail.gmail.com> On Wed, Sep 18, 2013 at 6:38 AM, Ronald Oussoren <ronaldoussoren at mac.com>wrote: > > > On Sep 18, 2013, at 03:03 PM, "Martin v. L?wis" <martin at v.loewis.de> > wrote: > > Am 15.09.13 00:56, schrieb Ryan: > > +1. A 10.6-only build makes sense. > > > I'd like to support Russell's point: this could put a burden on everyone > releasing extension modules to also provide two binary releases, which > e.g. would then mess up downloads from PyPI. So -1. > > +0 on dropping 10.6 support from the binary installers as proposed > by Greg. > > > > Dropping support for 10.6 isn't really necessary, just build (carefully) > on a later release of OSX (preferably the last one to pick up the most > recent developer tools). > > I'll try to provide a more detailed response tonight, but the basic idea > is to use OSX 10.8, the current Xcode release and the OSX 10.8 SDK to do > the build. This is a supported configuration for building binaries that > target older OSX releases, but some care is needed with the configure > script for 3th-party libraries: those might pick up features that aren't > available in 10.6. > > Cool, that sounds like we should _really_ be doing. Best of both worlds and it'll give a much faster to build to everyone. Good luck! -gps -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130918/bce4c59b/attachment.html> From ronaldoussoren at mac.com Thu Sep 19 10:35:22 2013 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Thu, 19 Sep 2013 10:35:22 +0200 Subject: [Python-Dev] PEP 447: add type.__locallookup__ In-Reply-To: <CADiSq7fNhAgGUE9-1EdASM5MH0wx7W-mJTi6pRv=45r3K6LvmA@mail.gmail.com> References: <75030FAC-6918-4E94-95DA-67A88D53E6F5@mac.com> <7D6D4488-2855-41F8-A9F2-1FF23B9E3A56@mac.com> <522DDB74.5080609@stoneleaf.us> <CAP7+vJJTR=+cRY+iQgHY4ujTsC3_QyWn83dnK96rWO6bV9vFLQ@mail.gmail.com> <522DECA3.2030709@hotpy.org> <CAP7+vJ+thJvYpk0d0rnFO6bm88sTJ_t1QdYFB0SdPOGNDABuag@mail.gmail.com> <522DF204.7040109@hotpy.org> <0752376899dd7436f6115bfc03594574@chopin.edu.pl> <392015291ec843afb1bc85632055729d@BLUPR03MB199.namprd03.prod.outlook.com> <20130913065236.GF16820@ando> <CADiSq7fNhAgGUE9-1EdASM5MH0wx7W-mJTi6pRv=45r3K6LvmA@mail.gmail.com> Message-ID: <2251401A-0CCB-42EE-8030-B21DAB416116@mac.com> On 13 Sep, 2013, at 12:42, Nick Coghlan <ncoghlan at gmail.com> wrote: > Perhaps "__getdescriptor__" would work as the method name? Yes, it can technically return a non-descriptor, but the *primary* purpose is to customise the retrieval of objects that will be checked to see if they're descriptors. It *won't* be invoked when looking for ordinary attributes in an instance dict, but *will* be invoked when looking on the class object. __getdescriptor__ would work. The name is not 100% accurate, but a lot clearer than the other alternatives I've seen. Ronald P.S. Sorry about the slow response, its hard to find enough time to seriously participate in the discussion. From ronaldoussoren at mac.com Thu Sep 19 10:50:48 2013 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Thu, 19 Sep 2013 10:50:48 +0200 Subject: [Python-Dev] PEP 447: add type.__locallookup__ In-Reply-To: <20130913122328.GI16820@ando> References: <7D6D4488-2855-41F8-A9F2-1FF23B9E3A56@mac.com> <522DDB74.5080609@stoneleaf.us> <CAP7+vJJTR=+cRY+iQgHY4ujTsC3_QyWn83dnK96rWO6bV9vFLQ@mail.gmail.com> <522DECA3.2030709@hotpy.org> <CAP7+vJ+thJvYpk0d0rnFO6bm88sTJ_t1QdYFB0SdPOGNDABuag@mail.gmail.com> <522DF204.7040109@hotpy.org> <0752376899dd7436f6115bfc03594574@chopin.edu.pl> <392015291ec843afb1bc85632055729d@BLUPR03MB199.namprd03.prod.outlook.com> <20130913065236.GF16820@ando> <CADiSq7fNhAgGUE9-1EdASM5MH0wx7W-mJTi6pRv=45r3K6LvmA@mail.gmail.com> <20130913122328.GI16820@ando> Message-ID: <61609457-2C6A-4860-9AD5-B6C7618A7D13@mac.com> On 13 Sep, 2013, at 14:23, Steven D'Aprano <steve at pearwood.info> wrote: > On Fri, Sep 13, 2013 at 08:42:46PM +1000, Nick Coghlan wrote: >> Perhaps "__getdescriptor__" would work as the method name? Yes, it can >> technically return a non-descriptor, > > So technically that name is, um, what's the term... oh yes, "a lie". > > :-) > > >> but the *primary* purpose is to >> customise the retrieval of objects that will be checked to see if they're >> descriptors. > > If that's the case, the PEP should make that clear. > > [Aside: the PEP states that the method shouldn't invoke descriptors. > What's the reason for that? If I take the statement literally, doesn't > it mean that the method mustn't use any other methods at all? Surely > that can't be what is intended, but I'm not sure what is intended.] What it tries to say is that even if cls.__dict__[name] is a descriptor the __locallookup__ method should not call its __get__ method. That was more relevant when __locallookup__ had access to the object for which method resolution was performed. I'm pretty sure this muddies the water even further, but don't have a clearer description at the moment. > > >> It *won't* be invoked when looking for ordinary attributes in >> an instance dict, but *will* be invoked when looking on the class object. > > Just to be clear, if I have: > > instance = MyClass() > x = instance.name > > and "name" is found in instance.__dict__, then this special method will > not be invoked. But if "name" is not found in the instance dict, then > "name" will be looked up on the class object MyClass, which may invoke > this special method. Am I correct? No. This special method will always be called when "instance.__getattribute__" eventually calls PyObject_GenericGetAttribute (otherwise all bets are off). PyObject_GenericGetAttribute always looks for a candicate descriptor in the class (or one of the other classes on the MRO). When a candidate descriptor is found, and this is a data descriptor the descriptor is used instead of the value from instance.__dict__ (otherwise the value from instance.__dict__ is used). Ronald From ronaldoussoren at mac.com Thu Sep 19 11:00:25 2013 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Thu, 19 Sep 2013 11:00:25 +0200 Subject: [Python-Dev] PEP 447: add type.__locallookup__ In-Reply-To: <f482c1d7f04f49d99616f3ca3c3b82da@BLUPR03MB199.namprd03.prod.outlook.com> References: <75030FAC-6918-4E94-95DA-67A88D53E6F5@mac.com> <7D6D4488-2855-41F8-A9F2-1FF23B9E3A56@mac.com> <522DDB74.5080609@stoneleaf.us> <CAP7+vJJTR=+cRY+iQgHY4ujTsC3_QyWn83dnK96rWO6bV9vFLQ@mail.gmail.com> <522DECA3.2030709@hotpy.org> <CAP7+vJ+thJvYpk0d0rnFO6bm88sTJ_t1QdYFB0SdPOGNDABuag@mail.gmail.com> <522DF204.7040109@hotpy.org> <0752376899dd7436f6115bfc03594574@chopin.edu.pl> <392015291ec843afb1bc85632055729d@BLUPR03MB199.namprd03.prod.outlook.com> <20130913065236.GF16820@ando> <f482c1d7f04f49d99616f3ca3c3b82da@BLUPR03MB199.namprd03.prod.outlook.com> Message-ID: <8AF283AC-3C8E-4D14-BCD8-4DD9002E56C6@mac.com> On 13 Sep, 2013, at 18:19, Steve Dower <Steve.Dower at microsoft.com> wrote: > From: Steven D'Aprano >> On Fri, Sep 13, 2013 at 04:26:06AM +0000, Steve Dower wrote: >> >>> Last I checked, looking up in the instance dict us exactly what it >>> does. Even the example you posted is doing that. >> >> The example from the PEP shows: >> >> return cls.__dict__[name] >> >> not "self.__dict__[name]". It is true that "the instance" in this case refers to >> it being an instance of the metaclass, but that instance is, in fact, a >> class/type. That's why we normally call it "cls" in a metaclass method rather >> than "self". > > Right, but where's the difference between these two? > > class A: > def __tdb__(self, name): > if name == 'some_attribute_on_my_instance_of_A': > return its_value > try: > return self.__dict__[name] > except KeyError: > raise AttributeError(name) > > class MetaB: > def __tdb__(cls, name): > if name == 'some_attribute_on_my_class_B': > return its_value > try: > return cls.__dict__[name] > except KeyError: > raise AttributeError(name) > > (Yes, either of these could be written with __getattribute__, but that function cannot be called by super().) > > As I see it, there are two (correct) ways to interpret what this method is for, which influences what it should be called. > > 1. It directly replaces obj.__dict__[name] everywhere that is done, including internally in the interpreter. > 2. It is the same as __getattribute__ without the final call to object.__getattribute__ > > I guess it's also correct to view it as a special helper for super(), but it is more generally applicable than that. It directly replaces cls.__dict__[name] for object.__getattribute__ and super.__getattribute__. The primary reason for writing a proposal is that __getattribute__ can be used to override attribute lookup on an instance, but there is way to override how super() looks up an attribute. Using the method for both super().__getattribute__ and object.__getattribute__ results in a cleaner model than just having a new hook for super().__getattribute__. > > [...] > >> By the way, I think the PEP should have a more complex example. The SillyObject >> example is nice and easy to understand, but it doesn't really help with the >> motivating use-case "dynamic classes that can grow new methods on demand". >> Ronald, if you're reading this, can you add such an example please? Also, >> there's a typo in the SillyObject M method ("fourtytwo" should not have a U in >> it). > > Agreed. No harm in more examples. [...] > > A full example of where this may realistically be needed is longer and certainly involves metaclasses, but fundamentally it's just the same as __getattribute__ with slightly different semantics. PyObjC will be a truly realistic example, but that involves loads of fairly complex C code and likely won't help to explain anything beyond (hopefully) showing that this proposal can lead to significant code removal in some situations. Ronald From ronaldoussoren at mac.com Thu Sep 19 11:32:17 2013 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Thu, 19 Sep 2013 11:32:17 +0200 Subject: [Python-Dev] PEP 447: add type.__locallookup__ In-Reply-To: <CADiSq7cdSEHryP+LMppRdabTuUG-GzMd56ub-oWpFNKPHNzsHA@mail.gmail.com> References: <7D6D4488-2855-41F8-A9F2-1FF23B9E3A56@mac.com> <522DDB74.5080609@stoneleaf.us> <CAP7+vJJTR=+cRY+iQgHY4ujTsC3_QyWn83dnK96rWO6bV9vFLQ@mail.gmail.com> <522DECA3.2030709@hotpy.org> <CAP7+vJ+thJvYpk0d0rnFO6bm88sTJ_t1QdYFB0SdPOGNDABuag@mail.gmail.com> <522DF204.7040109@hotpy.org> <0752376899dd7436f6115bfc03594574@chopin.edu.pl> <392015291ec843afb1bc85632055729d@BLUPR03MB199.namprd03.prod.outlook.com> <20130913065236.GF16820@ando> <CADiSq7fNhAgGUE9-1EdASM5MH0wx7W-mJTi6pRv=45r3K6LvmA@mail.gmail.com> <20130913122328.GI16820@ando> <CADiSq7cdSEHryP+LMppRdabTuUG-GzMd56ub-oWpFNKPHNzsHA@mail.gmail.com> Message-ID: <9BBAFB63-68C5-441B-8ED4-43604241AA42@mac.com> On 14 Sep, 2013, at 8:30, Nick Coghlan <ncoghlan at gmail.com> wrote: [... interesting text that I'll respond to later ...] > > So my proposed name is based on the idea that what Ronald is after > with the PEP is a hook that *only* gets invoked when the interpreter > is doing this hunt for descriptors, but *not* for ordinary attribute > lookups. I'll describe the usecase that led to my proposal. I'm amongst other the primary author (and sole maintainer) of PyObjC, which is a bridge between the Python and Objective-C class/object models. PyObjC defines a proxy class for every (Objective-C) class in the Objective-C runtime and when an Objective-C object is made available to Python code the PyObjC bridge creates a proxy object that is an instance of this proxy object. This is simular what wxWidgets or PyQt do, but everything is done at runtime by introspecting the Objective-C runtime. The first time a method is called the bridge looks for an Objective-C selector with the same name and adds that to the class dictionary. This works fine for normal method lookups, by overriding __getattribute__, but causes problems with super: super happily ignores __getattribute__ and peeks in the class __dict__ which may not yet contain the name we're looking for and that can result in incorrect results (both incorrect AttributeErrors and totally incorrect results when the name is not yet present in the parent class' __dict__ but is in the grandparent's __dict__). The current release of PyObjC "solves" this problem by providing a subclass of super (objc.super) that should be used instead of the builtin one and that works, although the implementation of this class is a gross hack. Users's of PyObjC are currently oblivious of the problem because I sneak in objc.super as a module's globals()['super'] when it imports PyObjC in the currently prefered way (which is using *-imports: "from Cocoa import *"). I'm currently migrating to deprecating *-imports because those are bad for the usual reasons, but also because framework binding modules are huge and using plain imports makes it possible to delay, and often even avoid, the cost of loading stuff from the framework binding modules. That switch will however make the problem of using __builtin__.super extremely visible for PyObjC users. That, and the implementation hack of objc.super, is why I started looking for a way to cleanly provide a way to hook into the attribute resolution for super(), which ended up as PEP 447. Note that the description is slightly simplified from what PyObjC reall does, Objective-C can (and does) have class and instance methods with the same name, because of that all PyObjC classes have a meta class of the same name and that makes everything even more complicated, but that should not be important for the previous description. PyObjC also enables implementing Objective-C classes in Python, but that's also not relevant for this discussion. Ronald From p.f.moore at gmail.com Thu Sep 19 12:00:01 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 19 Sep 2013 11:00:01 +0100 Subject: [Python-Dev] PEP 447: add type.__locallookup__ In-Reply-To: <9BBAFB63-68C5-441B-8ED4-43604241AA42@mac.com> References: <7D6D4488-2855-41F8-A9F2-1FF23B9E3A56@mac.com> <522DDB74.5080609@stoneleaf.us> <CAP7+vJJTR=+cRY+iQgHY4ujTsC3_QyWn83dnK96rWO6bV9vFLQ@mail.gmail.com> <522DECA3.2030709@hotpy.org> <CAP7+vJ+thJvYpk0d0rnFO6bm88sTJ_t1QdYFB0SdPOGNDABuag@mail.gmail.com> <522DF204.7040109@hotpy.org> <0752376899dd7436f6115bfc03594574@chopin.edu.pl> <392015291ec843afb1bc85632055729d@BLUPR03MB199.namprd03.prod.outlook.com> <20130913065236.GF16820@ando> <CADiSq7fNhAgGUE9-1EdASM5MH0wx7W-mJTi6pRv=45r3K6LvmA@mail.gmail.com> <20130913122328.GI16820@ando> <CADiSq7cdSEHryP+LMppRdabTuUG-GzMd56ub-oWpFNKPHNzsHA@mail.gmail.com> <9BBAFB63-68C5-441B-8ED4-43604241AA42@mac.com> Message-ID: <CACac1F_gT=mDEV3_VwvhjvS2H=JmNodWdP1vqMcEopWKMrPPHA@mail.gmail.com> On 19 September 2013 10:32, Ronald Oussoren <ronaldoussoren at mac.com> wrote: > The first time a method is called the bridge looks for an Objective-C selector > with the same name and adds that to the class dictionary. This works fine for normal > method lookups, by overriding __getattribute__, but causes problems with super: > super happily ignores __getattribute__ and peeks in the class __dict__ which may > not yet contain the name we're looking for and that can result in incorrect results > (both incorrect AttributeErrors and totally incorrect results when the name is > not yet present in the parent class' __dict__ but is in the grandparent's __dict__). As an alternative approach, could you use a custom dict subclass as the class __dict__, and catch the peeking in the class __dict__ that way? Or is this one of those places where only a real dict will do? Paul From ronaldoussoren at mac.com Thu Sep 19 12:04:48 2013 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Thu, 19 Sep 2013 12:04:48 +0200 Subject: [Python-Dev] PEP 447: add type.__locallookup__ In-Reply-To: <CACac1F_gT=mDEV3_VwvhjvS2H=JmNodWdP1vqMcEopWKMrPPHA@mail.gmail.com> References: <7D6D4488-2855-41F8-A9F2-1FF23B9E3A56@mac.com> <522DDB74.5080609@stoneleaf.us> <CAP7+vJJTR=+cRY+iQgHY4ujTsC3_QyWn83dnK96rWO6bV9vFLQ@mail.gmail.com> <522DECA3.2030709@hotpy.org> <CAP7+vJ+thJvYpk0d0rnFO6bm88sTJ_t1QdYFB0SdPOGNDABuag@mail.gmail.com> <522DF204.7040109@hotpy.org> <0752376899dd7436f6115bfc03594574@chopin.edu.pl> <392015291ec843afb1bc85632055729d@BLUPR03MB199.namprd03.prod.outlook.com> <20130913065236.GF16820@ando> <CADiSq7fNhAgGUE9-1EdASM5MH0wx7W-mJTi6pRv=45r3K6LvmA@mail.gmail.com> <20130913122328.GI16820@ando> <CADiSq7cdSEHryP+LMppRdabTuUG-GzMd56ub-oWpFNKPHNzsHA@mail.gmail.com> <9BBAFB63-68C5-441B-8ED4-43604241AA42@mac.com> <CACac1F_gT=mDEV3_VwvhjvS2H=JmNodWdP1vqMcEopWKMrPPHA@mail.gmail.com> Message-ID: <C1DAADB8-E611-4359-B0F4-7914E9579368@mac.com> On 19 Sep, 2013, at 12:00, Paul Moore <p.f.moore at gmail.com> wrote: > On 19 September 2013 10:32, Ronald Oussoren <ronaldoussoren at mac.com> wrote: >> The first time a method is called the bridge looks for an Objective-C selector >> with the same name and adds that to the class dictionary. This works fine for normal >> method lookups, by overriding __getattribute__, but causes problems with super: >> super happily ignores __getattribute__ and peeks in the class __dict__ which may >> not yet contain the name we're looking for and that can result in incorrect results >> (both incorrect AttributeErrors and totally incorrect results when the name is >> not yet present in the parent class' __dict__ but is in the grandparent's __dict__). > > As an alternative approach, could you use a custom dict subclass as > the class __dict__, and catch the peeking in the class __dict__ that > way? Or is this one of those places where only a real dict will do? The C code uses PyDict_GetItem and AFAIK that doesn't look for a __getitem__ implementation in a subclass. Ronald > > Paul From ncoghlan at gmail.com Thu Sep 19 12:12:33 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 19 Sep 2013 20:12:33 +1000 Subject: [Python-Dev] PEP 447: add type.__locallookup__ In-Reply-To: <CACac1F_gT=mDEV3_VwvhjvS2H=JmNodWdP1vqMcEopWKMrPPHA@mail.gmail.com> References: <7D6D4488-2855-41F8-A9F2-1FF23B9E3A56@mac.com> <522DDB74.5080609@stoneleaf.us> <CAP7+vJJTR=+cRY+iQgHY4ujTsC3_QyWn83dnK96rWO6bV9vFLQ@mail.gmail.com> <522DECA3.2030709@hotpy.org> <CAP7+vJ+thJvYpk0d0rnFO6bm88sTJ_t1QdYFB0SdPOGNDABuag@mail.gmail.com> <522DF204.7040109@hotpy.org> <0752376899dd7436f6115bfc03594574@chopin.edu.pl> <392015291ec843afb1bc85632055729d@BLUPR03MB199.namprd03.prod.outlook.com> <20130913065236.GF16820@ando> <CADiSq7fNhAgGUE9-1EdASM5MH0wx7W-mJTi6pRv=45r3K6LvmA@mail.gmail.com> <20130913122328.GI16820@ando> <CADiSq7cdSEHryP+LMppRdabTuUG-GzMd56ub-oWpFNKPHNzsHA@mail.gmail.com> <9BBAFB63-68C5-441B-8ED4-43604241AA42@mac.com> <CACac1F_gT=mDEV3_VwvhjvS2H=JmNodWdP1vqMcEopWKMrPPHA@mail.gmail.com> Message-ID: <CADiSq7eQ457=Vva-_3xbpi8o2EtS9ea8Q3XG=oHe0xBBAtzywQ@mail.gmail.com> On 19 Sep 2013 20:00, "Paul Moore" <p.f.moore at gmail.com> wrote: > > On 19 September 2013 10:32, Ronald Oussoren <ronaldoussoren at mac.com> wrote: > > The first time a method is called the bridge looks for an Objective-C selector > > with the same name and adds that to the class dictionary. This works fine for normal > > method lookups, by overriding __getattribute__, but causes problems with super: > > super happily ignores __getattribute__ and peeks in the class __dict__ which may > > not yet contain the name we're looking for and that can result in incorrect results > > (both incorrect AttributeErrors and totally incorrect results when the name is > > not yet present in the parent class' __dict__ but is in the grandparent's __dict__). > > As an alternative approach, could you use a custom dict subclass as > the class __dict__, and catch the peeking in the class __dict__ that > way? Or is this one of those places where only a real dict will do? Even Python 3 doesn't let you control the *runtime* type of the class dict, only the type used during evaluation of the class body. I've played with changing that - it makes for a rather special interpreter experience :) Cheers, Nick. > > Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130919/05d27e0a/attachment.html> From ronaldoussoren at mac.com Thu Sep 19 12:32:38 2013 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Thu, 19 Sep 2013 12:32:38 +0200 Subject: [Python-Dev] PEP 447: add type.__locallookup__ In-Reply-To: <CADiSq7eQ457=Vva-_3xbpi8o2EtS9ea8Q3XG=oHe0xBBAtzywQ@mail.gmail.com> References: <7D6D4488-2855-41F8-A9F2-1FF23B9E3A56@mac.com> <522DDB74.5080609@stoneleaf.us> <CAP7+vJJTR=+cRY+iQgHY4ujTsC3_QyWn83dnK96rWO6bV9vFLQ@mail.gmail.com> <522DECA3.2030709@hotpy.org> <CAP7+vJ+thJvYpk0d0rnFO6bm88sTJ_t1QdYFB0SdPOGNDABuag@mail.gmail.com> <522DF204.7040109@hotpy.org> <0752376899dd7436f6115bfc03594574@chopin.edu.pl> <392015291ec843afb1bc85632055729d@BLUPR03MB199.namprd03.prod.outlook.com> <20130913065236.GF16820@ando> <CADiSq7fNhAgGUE9-1EdASM5MH0wx7W-mJTi6pRv=45r3K6LvmA@mail.gmail.com> <20130913122328.GI16820@ando> <CADiSq7cdSEHryP+LMppRdabTuUG-GzMd56ub-oWpFNKPHNzsHA@mail.gmail.com> <9BBAFB63-68C5-441B-8ED4-43604241AA42@mac.com> <CACac1F_gT=mDEV3_VwvhjvS2H=JmNodWdP1vqMcEopWKMrPPHA@mail.gmail.com> <CADiSq7eQ457=Vva-_3xbpi8o2EtS9ea8Q3XG=oHe0xBBAtzywQ@mail.gmail.com> Message-ID: <F77A9A16-183E-4927-A96A-948F7494CD1D@mac.com> On 19 Sep, 2013, at 12:12, Nick Coghlan <ncoghlan at gmail.com> wrote: > > On 19 Sep 2013 20:00, "Paul Moore" <p.f.moore at gmail.com> wrote: > > > > On 19 September 2013 10:32, Ronald Oussoren <ronaldoussoren at mac.com> wrote: > > > The first time a method is called the bridge looks for an Objective-C selector > > > with the same name and adds that to the class dictionary. This works fine for normal > > > method lookups, by overriding __getattribute__, but causes problems with super: > > > super happily ignores __getattribute__ and peeks in the class __dict__ which may > > > not yet contain the name we're looking for and that can result in incorrect results > > > (both incorrect AttributeErrors and totally incorrect results when the name is > > > not yet present in the parent class' __dict__ but is in the grandparent's __dict__). > > > > As an alternative approach, could you use a custom dict subclass as > > the class __dict__, and catch the peeking in the class __dict__ that > > way? Or is this one of those places where only a real dict will do? > > Even Python 3 doesn't let you control the *runtime* type of the class dict, only the type used during evaluation of the class body. Changing the class dict type from C is easy enough, but as you wrote below doing this gives you an interesting experience. Changing PyDict_* to call the subclass implementation of the corresponding slot would be easy enough, but that changes the behavior of a core CPython API and I wouldn't look forward to auditing the CPython code base to check if such a change is safe (let alone all other extensions). Some code will use the PyDict_* API instead of the abstract API because the former is faster, but at least some callers for PyDict_GetItem will do this to explicitly get the base class implementation. Ronald > > I've played with changing that - it makes for a rather special interpreter experience :) > > Cheers, > Nick. > > > > > Paul From ronaldoussoren at mac.com Thu Sep 19 11:49:10 2013 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Thu, 19 Sep 2013 11:49:10 +0200 Subject: [Python-Dev] PEP 447: add type.__locallookup__ In-Reply-To: <CADiSq7cdSEHryP+LMppRdabTuUG-GzMd56ub-oWpFNKPHNzsHA@mail.gmail.com> References: <7D6D4488-2855-41F8-A9F2-1FF23B9E3A56@mac.com> <522DDB74.5080609@stoneleaf.us> <CAP7+vJJTR=+cRY+iQgHY4ujTsC3_QyWn83dnK96rWO6bV9vFLQ@mail.gmail.com> <522DECA3.2030709@hotpy.org> <CAP7+vJ+thJvYpk0d0rnFO6bm88sTJ_t1QdYFB0SdPOGNDABuag@mail.gmail.com> <522DF204.7040109@hotpy.org> <0752376899dd7436f6115bfc03594574@chopin.edu.pl> <392015291ec843afb1bc85632055729d@BLUPR03MB199.namprd03.prod.outlook.com> <20130913065236.GF16820@ando> <CADiSq7fNhAgGUE9-1EdASM5MH0wx7W-mJTi6pRv=45r3K6LvmA@mail.gmail.com> <20130913122328.GI16820@ando> <CADiSq7cdSEHryP+LMppRdabTuUG-GzMd56ub-oWpFNKPHNzsHA@mail.gmail.com> Message-ID: <4CE957E3-ECE7-4DD1-AB62-2EA7776B9F85@mac.com> On 14 Sep, 2013, at 8:30, Nick Coghlan <ncoghlan at gmail.com> wrote: > > >>> but the *primary* purpose is to >>> customise the retrieval of objects that will be checked to see if they're >>> descriptors. >> >> If that's the case, the PEP should make that clear. > > Technically, that's what "Currently object.__getattribute__ and > super.__getattribute__ peek in the __dict__ of classes on the MRO for > a class when looking for an attribute." means. > > However, I agree the current wording only conveys that to the handful > of people that already know exactly when in the attribute lookup > sequence that step occurs, which is a rather niche audience :) I've been fooling around with this long enough that I forgot that not everyone knows this :-). I guess I'd better include a clearer and more complete description of the current attribute resolution protocol and how my proposal affects that. A nice readable Python implementation of that protocol would be nice to have regardless of the fate of this PEP. Ronald From donald at stufft.io Thu Sep 19 15:27:24 2013 From: donald at stufft.io (Donald Stufft) Date: Thu, 19 Sep 2013 09:27:24 -0400 Subject: [Python-Dev] PEP 453 Round 4 - Explicit bootstrapping of pip in Python installations Message-ID: <C1831192-CF74-4CBC-BC55-A8CEEDF96D46@stufft.io> We've updated PEP453 based on some of the early feedback we've gotten from -dev and Martin. Major changes: * Removal of the option to fetch pip from PyPI in order not to modify the trust model of the Python installers * Consequently rename the model from ``getpip`` to ``extractpip`` Also available online at: http://www.python.org/dev/peps/pep-0453/ Abstract ======== This PEP proposes that the `pip`_ package manager be made available by default when installing CPython and when creating virtual environments using the standard library's ``venv`` module via the ``pyvenv`` command line utility). To clearly demarcate development responsibilities, and to avoid inadvertently downgrading ``pip`` when updating CPython, the proposed mechanism to achieve this is to include an explicit `pip`_ bootstrapping mechanism in the standard library that is invoked automatically by the CPython installers provided on python.org. The PEP also strongly recommends that CPython redistributors and other Python implementations ensure that ``pip`` is available by default, or at the very least, explicitly document the fact that it is not included. Proposal ======== This PEP proposes the inclusion of an ``extractpip`` bootstrapping module in Python 3.4, as well as in the next maintenance releases of Python 3.3 and 2.7. This PEP does *not* propose making pip (or any dependencies) directly available as part of the standard library. Instead, pip will be a bundled application provided along with CPython for the convenience of Python users, but subject to its own development life cycle and able to be upgraded independently of the core interpreter and standard library. Rationale ========= Currently, on systems without a platform package manager and repository, installing a third-party Python package into a freshly installed Python requires first identifying an appropriate package manager and then installing it. Even on systems that *do* have a platform package manager, it is unlikely to include every package that is available on the Python Package Index, and even when a desired third-party package is available, the correct name in the platform package manager may not be clear. This means that, to work effectively with the Python Package Index ecosystem, users must know which package manager to install, where to get it, and how to install it. The effect of this is that third-party Python projects are currently required to choose from a variety of undesirable alternatives: * Assume the user already has a suitable cross-platform package manager installed. * Duplicate the instructions and tell their users how to install the package manager. * Completely forgo the use of dependencies to ease installation concerns for their users. All of these available options have significant drawbacks. If a project simply assumes a user already has the tooling then beginning users may get a confusing error message when the installation command doesn't work. Some operating systems may ease this pain by providing a global hook that looks for commands that don't exist and suggest an OS package they can install to make the command work, but that only works on systems with platform package managers (such as major Linux distributions). No such assistance is available for Windows and Mac OS X users. The challenges of dealing with this problem are a regular feature of feedback the core Python developers receive from professional educators and others introducing new users to Python. If a project chooses to duplicate the installation instructions and tell their users how to install the package manager before telling them how to install their own project then whenever these instructions need updates they need updating by every project that has duplicated them. This is particular problematic when there are multiple competing installation tools available, and different projects recommend different tools. This specific problem can be partially alleviated by strongly promoting ``pip`` as the default installer and recommending that other projects reference `pip's own bootstrapping instructions <http://www.pip-installer.org/en/latest/installing.html>`__ rather than duplicating them. However the user experience created by this approach still isn't good (especially on Windows, where downloading and running the ``get-pip.py`` bootstrap script with the default OS configuration is significantly more painful than downloading and running a binary executable or installer). The situation becomes even more complicated when multiple Python versions are involved (for example, parallel installations of Python 2 and Python 3), since that makes it harder to create and maintain good platform specific ``pip`` installers independently of the CPython installers. The projects that have decided to forgo dependencies altogether are forced to either duplicate the efforts of other projects by inventing their own solutions to problems or are required to simply include the other projects in their own source trees. Both of these options present their own problems either in duplicating maintenance work across the ecosystem or potentially leaving users vulnerable to security issues because the included code or duplicated efforts are not automatically updated when upstream releases a new version. By providing a cross-platform package manager by default it will be easier for users trying to install these third-party packages as well as easier for the people distributing them as they should now be able to safely assume that most users will have the appropriate installation tools available. This is expected to become more important in the future as the Wheel_ package format (deliberately) does not have a built in "installer" in the form of ``setup.py`` so users wishing to install from a wheel file will want an installer even in the simplest cases. Reducing the burden of actually installing a third-party package should also decrease the pressure to add every useful module to the standard library. This will allow additions to the standard library to focus more on why Python should have a particular tool out of the box, and why it is reasonable for that package to adopt the standard library's 18-24 month feature release cycle, instead of using the general difficulty of installing third-party packages as justification for inclusion. Providing a standard installation system also helps with bootstrapping alternate build and installer systems, such as ``setuptools``, ``zc.buildout`` and the ``hashdist``/``conda`` combination that is aimed specifically at the scientific community. So long as ``pip install <tool>`` works, then a standard Python-specific installer provides a reasonably secure, cross platform mechanism to get access to these utilities. Why pip? -------- ``pip`` has been chosen as the preferred default installer, as it addresses several design and user experience issues with its predecessor ``easy_install`` (these issues can't readily be fixed in ``easy_install`` itself due to backwards compatibility concerns). ``pip`` is also well suited to working within the bounds of a single Python runtime installation (including associated virtual environments), which is a desirable feature for a tool bundled with CPython. Other tools like ``zc.buildout`` and ``conda`` are more ambitious in their aims (and hence substantially better than ``pip`` at handling external binary dependencies), so it makes sense for the Python ecosystem to treat them more like platform package managers to inter operate with rather than as the default cross-platform installation tool. This relationship is similar to that between ``pip`` and platform package management systems like ``apt`` and ``yum`` (which are also designed to handle arbitrary binary dependencies). Explicit bootstrapping mechanism ================================ An additional module called ``extractpip`` will be added to the standard library whose purpose is to install pip and any of its dependencies into the appropriate location (most commonly site-packages). It will expose a callable named ``bootstrap()`` as well as offer direct execution via ``python -m extractpip``. The bootstrap will *not* contact PyPI, but instead rely on a private copy of pip stored inside the standard library. Accordingly, only options related to the installation location will be supported (``--user``, ``--root``, etc). It is considered desirable that users be strongly encouraged to use the latest available version of ``pip``, in order to take advantage of the ongoing efforts to improve the security of the PyPI based ecosystem, as well as benefiting from the efforts to improve the speed, reliability and flexibility of that ecosystem. In order to satisfy this goal of providing the most recent version of ``pip`` by default, the private copy of ``pip`` will be updated in CPython maintenance releases, which should align well with the 6-month cycle used for new ``pip`` releases. Security considerations ----------------------- The design in this PEP has been deliberately chosen to avoid making any significant changes to the trust model of the CPython installers for end users that do not subsequently make use of ``pip``. The installers will contain all the components of a fully functioning version of Python, including the ``pip`` installer. The installation process will *not* require network access, and will *not* rely on trusting the security of the network connection established between ``pip`` and the Python package index. Only users that choose to use ``pip`` directly will need to pay attention to any PyPI related security considerations. Implementation strategy ----------------------- To ensure there is no need for network access when installing Python or creating virtual environments, the ``extractpip`` module will, as an implementation detail, include a complete private copy of pip and its dependencies which will be used to extract pip and install it into the target environment. It is important to stress that this private copy of pip is *only* an implementation detail and it should *not* be relied on or assumed to exist beyond the public capabilities exposed through the ``extractpip`` module (and indirectly through ``venv``). There is not yet a reference ``extractpip`` implementation. The existing ``get-pip.py`` bootstrap script demonstrates an earlier variation of the general concept, but the standard library version would take advantage of the improved distribution capabilities offered by the CPython installers to include private copies of ``pip`` and ``setuptools`` as wheel files (rather than as embedded base64 encoded data), and would not try to contact PyPI (instead installing directly from the private wheel files. Rather than including separate code to handle the bootstrapping, the ``extractpip`` module will manipulate sys.path appropriately to allow the wheel files to be used to install themselves, either into the current Python installation or into a virtual environment (as determined by the options passed to the bootstrap command). Proposed CLI ------------ The proposed CLI is based on a subset of the existing ``pip install`` options:: Usage: python -m extractpip [options] General Options: -h, --help Show help. -v, --verbose Give more output. Option is additive, and can be used up to 3 times. -V, --version Show the pip version that would be extracted and exit. -q, --quiet Give less output. Installation Options: -U, --upgrade Upgrade pip and dependencies, even if already installed --user Install using the user scheme. --root <dir> Install everything relative to this alternate root directory. In most cases, end users won't need to use this CLI directly, as ``pip`` should have been installed automatically when installing Python or when creating a virtual environment. Users that want to retrieve the latest version from PyPI, or otherwise needing more flexibility, should invoke the extracted ``pip`` appropriately. Proposed module API ------------------- The proposed ``extractpip`` module API consists of the following two functions:: def version(): """ Returns a string specifying the bundled version of pip. """ def bootstrap(root=None, upgrade=False, user=False, verbosity=0): """ Bootstrap pip into the current Python installation (or the given root directory). """ Invocation from the CPython installers -------------------------------------- The CPython Windows and Mac OS X installers will each gain a new option: * Install pip (the default Python package management utility)? This option will be checked by default. If the option is checked, then the installer will invoke the following command with the just installed Python:: python -m extractpip --upgrade This ensures that, by default, installing or updating CPython will ensure that the installed version of pip is at least as recent as the one included with that version of CPython. Installing from source ---------------------- While the prebuilt binary installers will be updated to run ``python -m extractpip`` by default, no such change will be made to the ``make install`` and ``make altinstall`` commands of the source distribution. ``extractpip`` itself (including the private copy of ``pip`` and its dependencies) will still be installed normally (as it is a regular part of the standard library), only the implicit installation of pip and its dependencies will be skipped. Keeping the pip bootstrapping as a separate step for ``make``-based installations should minimize the changes CPython redistributors need to make to their build processes. Avoiding the layer of indirection through ``make`` for the ``extractpip`` invocation avoids any challenges associated with determining where to install the extracted ``pip``. Changes to virtual environments ------------------------------- Python 3.3 included a standard library approach to virtual Python environments through the ``venv`` module. Since its release it has become clear that very few users have been willing to use this feature directly, in part due to the lack of an installer present by default inside of the virtual environment. They have instead opted to continue using the ``virtualenv`` package which *does* include pip installed by default. To make the ``venv`` more useful to users it will be modified to issue the pip bootstrap by default inside of the new environment while creating it. This will allow people the same convenience inside of the virtual environment as this PEP provides outside of it as well as bringing the ``venv`` module closer to feature parity with the external ``virtualenv`` package, making it a more suitable replacement. To handle cases where a user does not wish to have pip bootstrapped into their virtual environment a ``--without-pip`` option will be added. The ``venv.EnvBuilder`` and ``venv.create`` APIs will be updated to accept one new parameter: ``with_pip`` (defaulting to ``False``). The new default for the module API is chosen for backwards compatibility with the current behaviour (as it is assumed that most invocation of the ``venv`` module happens through third part tools that likely will not want ``pip`` installed without explicitly requesting it), while the default for the command line interface is chosen to try to ensure ``pip`` is available in most virtual environments without additional action on the part of the end user. This particular change will be made only for Python 3.4 and later versions. The third-party ``virtualenv`` project will still be needed to obtain a consistent cross-version experience in Python 3.3 and 2.7. Documentation ------------- The "Installing Python Modules" section of the standard library documentation will be updated to recommend the use of the bootstrapped `pip` installer. It will give a brief description of the most common commands and options, but delegate to the externally maintained ``pip`` documentation for the full details. The existing content of the module installation guide will be retained, but under a new "Invoking distutils directly" subsection. Bundling CA certificates with CPython ------------------------------------- The ``extractpip`` implementation will include the ``pip`` CA bundle along with the rest of ``pip``. This means CPython effectively includes a CA bundle that is used solely by ``pip`` after it has been extracted. This is considered preferable to relying solely on the system certificate stores, as it ensures that ``pip`` will behave the same across all supported versions of Python, even those prior to Python 3.4 that cannot access the system certificate store on Windows. Automatic installation of setuptools ------------------------------------ ``pip`` currently depends on ``setuptools`` to handle metadata generation during the build process, along with some other features. While work is ongoing to reduce or eliminate this dependency, it is not clear if that work will be complete for pip 1.5 (which is the version likely to be current when Python 3.4.0 is released). This PEP proposes that, if pip still requires it as a dependency, ``extractpip`` will include a private copy of ``setuptools`` (in addition to the private copy of ``extractpip``). ``python -m extractpip`` will then install the private copy in addition to installing ``pip`` itself. However, this behavior is officially considered an implementation detail. Other projects which explicitly require ``setuptools`` must still provide an appropriate dependency declaration, rather than assuming ``setuptools`` will always be installed alongside ``pip``. Once pip is able to run ``pip install --upgrade pip`` without needing ``setuptools`` installed first, then the private copy of ``setuptools`` will be removed from ``extractpip`` in subsequent CPython releases. Updating the private copy of pip -------------------------------- In order to keep up with evolutions in packaging as well as providing users with as recent version a possible the ``extractpip`` module will be regularly updated to the latest versions of everything it bootstraps. After each new ``pip`` release, and again during the preparation for any release of Python (including feature releases), a script, provided as part of this PEP, will be run to ensure the private copies stored in the CPython source repository have been updated to the latest versions. Updating the extractpip module API and CLI ------------------------------------------ Like ``venv`` and ``pyvenv``, the ``extractpip`` module API and CLI will be governed by the normal rules for the standard library: no new features are permitted in maintenance releases. However, the embedded components may be updated as noted above, so the extracted ``pip`` may offer additional functionality in maintenance releases. Feature addition in maintenance releases ======================================== Adding a new module to the standard library in Python 2.7 and 3.3 maintenance releases breaks the usual policy of "no new features in maintenance releases". It is being proposed in this case as the current bootstrapping issues for the third-party Python package ecosystem greatly affects the experience of new users, especially on Python 2 where many Python 3 standard library improvements are available as backports on PyPI, but are not included in the Python 2 standard library. By updating Python 2.7, 3.3 and 3.4 to easily bootstrap the PyPI ecosystem, this change should aid the vast majority of current Python users, rather than only those with the freedom to adopt Python 3.4 as soon as it is released. Open Question: Uninstallation ============================= No changes are currently proposed to the uninstallation process. The bootstrapped pip will be installed the same way as any other pip installed packages, and will be handled in the same way as any other post-install additions to the Python environment. At least on Windows, that means the bootstrapped files will be left behind after uninstallation, since those files won't be associated with the Python MSI installer. .. note:: Perhaps the installer should be updated to clobber everything in site-packages and the Scripts directory when uninstalled (treating them as "data directories" from Python's point of view), but I would prefer not to make this PEP conditional on that change. Open Question: Script Execution on Windows ========================================== .. note:: Perhaps this question should be separated out from the PEP? While the Windows installer was updated in Python 3.3 to optionally make ``python`` available on the PATH, no such change was made to include the Scripts directory. Independently of this PEP, a proposal has also been made to rename the ``Tools\Scripts`` subdirectory to ``bin`` in order to improve consistency with the typical script installation directory names on \*nix systems. Accordingly, in addition to adding the option to extract and install ``pip`` during installation, this PEP proposes that the Windows installer (and ``sysconfig``) in Python 3.4 and later be updated to: - install scripts to PythonXY\bin rather than PythonXY\Tools\Scripts - add PythonXY\bin to the Windows PATH (in addition to PythonXY) when the PATH modification option is enabled during installation For Python 2.7 and 3.3, it is proposed that the only change be the one to bootstrap ``pip`` by default. This means that, for Python 3.3, the most reliable way to invoke pip on Windows (without tinkering manually with PATH) will actually be ``py -m pip`` (or ``py -3 -m pip`` to select the Python 3 version if both Python 2 and 3 are installed) rather than simply calling ``pip``. For Python 2.7 and 3.2, the most reliable mechanism will be to install the standalone Python launcher for Windows and then use ``py -m pip`` as noted above. Adding the scripts directory to the system PATH would mean that ``pip`` works reliably in the "only one Python installation on the system PATH" case, with ``py -m pip``, ``pipX``, or ``pipX.Y`` needed only to select a non-default version in the parallel installation case (and outside a virtual environment). This change should also make the ``pyvenv`` command substantially easier to invoke on Windows, along with all scripts installed by ``pip``, ``easy_install`` and similar tools. While the script invocations on recent versions of Python will run through the Python launcher for Windows, this shouldn't cause any issues, as long as the Python files in the Scripts directory correctly specify a Python version in their shebang line or have an adjacent Windows executable (as ``easy_install`` and ``pip`` do). Recommendations for Downstream Distributors =========================================== A common source of Python installations are through downstream distributors such as the various Linux Distributions [#ubuntu]_ [#debian]_ [#fedora]_, OSX package managers [#homebrew]_, or Python-specific tools [#conda]_. In order to provide a consistent, user-friendly experience to all users of Python regardless of how they attained Python this PEP recommends and asks that downstream distributors: * Ensure that whenever Python is installed pip is also installed. * This may take the form of separate packages with dependencies on each other so that installing the Python package installs the pip package and installing the pip package installs the Python package. * Do not remove the bundled copy of pip. * This is required for installation of pip into a virtual environment by the ``venv`` module. * This is similar to the existing ``virtualenv`` package for which many downstream distributors have already made exception to the common "debundling" policy. * This does mean that if ``pip`` needs to be updated due to a security issue, so does the bundled version in the ``extractpip`` bootstrap module * However, altering the bundled version of pip to remove the embedded CA certificate bundle and rely on the system CA bundle instead is a reasonable change. * Migrate build systems to utilize `pip`_ and `Wheel`_ instead of directly using ``setup.py``. * This will ensure that downstream packages can more easily utilize the new metadata formats which may not have a ``setup.py``. * Ensure that all features of this PEP continue to work with any modifications made to the redistributed version of Python. * Checking the version of pip that will be bootstrapped using ``python -m extractpip --version`` or ``extractpip.version()``. * Installation of the bundled version of pip into a global or virtual python environment using ``python -m extractpip`` or ``extractpip.bootstrap()``. * ``pip install --upgrade pip`` in a global installation should not affect any already created virtual environments (but is permitted to affect future virtual environments, even though it will not do so in the normal case). * ``pip install --upgrade pip`` in a virtual environment should not affect the global installation. In the event that a Python redistributor chooses *not* to follow these recommendations, we request that they explicitly document this fact and provide their users with suitable guidance on translating upstream ``pip`` based installation instructions into something appropriate for the platform. Other Python implementations are also encouraged to follow these guidelines where applicable. Policies & Governance ===================== The maintainers of the bootstrapped software and the CPython core team will work together in order to address the needs of both. The bootstrapped software will still remain external to CPython and this PEP does not include CPython subsuming the development responsibilities or design decisions of the bootstrapped software. This PEP aims to decrease the burden on end users wanting to use third-party packages and the decisions inside it are pragmatic ones that represent the trust that the Python community has already placed in the Python Packaging Authority as the authors and maintainers of ``pip``, ``setuptools``, PyPI, ``virtualenv`` and other related projects. Backwards Compatibility ----------------------- The public API and CLI of the ``extractpip`` module itself will fall under the typical backwards compatibility policy of Python for its standard library. The externally developed software that this PEP bundles does not. Most importantly, this means that the bootstrapped version of pip may gain new features in CPython maintenance releases, and pip continues to operate on its own 6 month release cycle rather than CPython's 18-24 month cycle. Security Releases ----------------- Any security update that affects the ``extractpip`` module will be shared prior to release with the Python Security Response Team (security at python.org). The PSRT will then decide if the reported issue warrants a security release of CPython with an updated private copy of ``pip``. Appendix: Rejected Proposals ============================ Automatically contacting PyPI when bootstrapping pip ---------------------------------------------------- Earlier versions of this PEP called the bootstrapping module ``getpip`` and defaulted to downloading and installing ``pip`` from PyPI, with the private copy used only as a fallback option or when explicitly requested. This resulted in several complex edge cases, along with difficulties in defining a clean API and CLI for the bootstrap module. It also significantly altered the default trust model for the binary installers published on python.org, as end users would need to explicitly *opt-out* of trusting the security of the PyPI ecosystem (rather than opting in to it by explicitly invoking ``pip`` following installation). As a result, the PEP was simplified to the current design, where the bootstrapping *always* uses the private copy of ``pip``. Contacting PyPI is now always an explicit separate step, with direct access to the full pip interface. Implicit bootstrap ------------------ `PEP439`_, the predecessor for this PEP, proposes its own solution. Its solution involves shipping a fake ``pip`` command that when executed would implicitly bootstrap and install pip if it does not already exist. This has been rejected because it is too "magical". It hides from the end user when exactly the pip command will be installed or that it is being installed at all. It also does not provide any recommendations or considerations towards downstream packagers who wish to manage the globally installed pip through the mechanisms typical for their system. The implicit bootstrap mechanism also ran into possible permissions issues, if a user inadvertently attempted to bootstrap pip without write access to the appropriate installation directories. Including pip directly in the standard library ---------------------------------------------- Similar to this PEP is the proposal of just including pip in the standard library. This would ensure that Python always includes pip and fixes all of the end user facing problems with not having pip present by default. This has been rejected because we've learned, through the inclusion and history of ``distutils`` in the standard library, that losing the ability to update the packaging tools independently can leave the tooling in a state of constant limbo. Making it unable to ever reasonably evolve in a time frame that actually affects users as any new features will not be available to the general population for *years*. Allowing the packaging tools to progress separately from the Python release and adoption schedules allows the improvements to be used by *all* members of the Python community and not just those able to live on the bleeding edge of Python releases. There have also been issues in the past with the "dual maintenance" problem if a project continues to be maintained externally while *also* having a fork maintained in the standard library. Since external maintenance of ``pip`` will always be needed to support earlier Python versions, the proposed bootstrapping mechanism will becoming the explicit responsibility of the CPython core developers (assisted by the pip developers), while pip issues reported to the CPython tracker will be migrated to the pip issue tracker. There will no doubt still be some user confusion over which tracker to use, but hopefully less than has been seen historically when including complete public copies of third-party projects in the standard library. Finally, the approach described in this PEP avoids some technical issues related to handle CPython maintenance updates when pip has been independently updated to a more recent version. The proposed pip-based bootstrapping mechanism handles that automatically, since pip and the system installer never get into a fight about who owns the pip installation (it is always managed through pip, either directly, or indirectly via the getpip bootstrap module). Defaulting to --user installation --------------------------------- Some consideration was given to bootstrapping pip into the per-user site-packages directory by default. However, this behavior would be surprising (as it differs from the default behavior of pip itself) and is also not currently considered reliable (there are some edge cases which are not handled correctly when pip is installed into the user site-packages directory rather than the system site-packages). .. _Wheel: http://www.python.org/dev/peps/pep-0427/ .. _pip: http://www.pip-installer.org .. _setuptools: https://pypi.python.org/pypi/setuptools .. _PEP439: http://www.python.org/dev/peps/pep-0439/ References ========== .. [#ubuntu] `Ubuntu <http://www.ubuntu.com/>` .. [#debian] `Debian <http://www.debian.org>` .. [#fedora] `Fedora <https://fedoraproject.org/>` .. [#homebrew] `Homebrew <http://brew.sh/>` .. [#conda] `Conda <http://www.continuum.io/blog/conda>` ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: <http://mail.python.org/pipermail/python-dev/attachments/20130919/c6314883/attachment-0001.sig> From donald at stufft.io Thu Sep 19 15:30:21 2013 From: donald at stufft.io (Donald Stufft) Date: Thu, 19 Sep 2013 09:30:21 -0400 Subject: [Python-Dev] PEP 453 Round 4 - Explicit bootstrapping of pip in Python installations In-Reply-To: <C1831192-CF74-4CBC-BC55-A8CEEDF96D46@stufft.io> References: <C1831192-CF74-4CBC-BC55-A8CEEDF96D46@stufft.io> Message-ID: <4D249E79-D01A-407E-99B8-38E0311A71DF@stufft.io> On Sep 19, 2013, at 9:27 AM, Donald Stufft <donald at stufft.io> wrote: > We've updated PEP453 based on some of the early feedback we've gotten from -dev and Martin. > > Major changes: > > * Removal of the option to fetch pip from PyPI in order not to modify the trust model of the Python installers > * Consequently rename the model from ``getpip`` to ``extractpip`` > > > Also available online at: http://www.python.org/dev/peps/pep-0453/ > > [snip] One thing I don't like is the name ``extractpip``, maybe ``installpip`` would be better? Naming things is hard. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: <http://mail.python.org/pipermail/python-dev/attachments/20130919/fbdfda0d/attachment.sig> From ncoghlan at gmail.com Thu Sep 19 15:40:25 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 19 Sep 2013 23:40:25 +1000 Subject: [Python-Dev] PEP 453 Round 4 - Explicit bootstrapping of pip in Python installations In-Reply-To: <4D249E79-D01A-407E-99B8-38E0311A71DF@stufft.io> References: <C1831192-CF74-4CBC-BC55-A8CEEDF96D46@stufft.io> <4D249E79-D01A-407E-99B8-38E0311A71DF@stufft.io> Message-ID: <CADiSq7fvEif0jdVgQwv+g3tN0+DGfNXshGjHppa7eOctffssiQ@mail.gmail.com> On 19 Sep 2013 23:30, "Donald Stufft" <donald at stufft.io> wrote: > > > On Sep 19, 2013, at 9:27 AM, Donald Stufft <donald at stufft.io> wrote: > > > We've updated PEP453 based on some of the early feedback we've gotten from -dev and Martin. > > > > Major changes: > > > > * Removal of the option to fetch pip from PyPI in order not to modify the trust model of the Python installers > > * Consequently rename the model from ``getpip`` to ``extractpip`` > > > > > > Also available online at: http://www.python.org/dev/peps/pep-0453/ > > > > [snip] > > One thing I don't like is the name ``extractpip``, maybe ``installpip`` would be better? I figured that was too close to "pip install", not to mention the actual phrase "install pip". There are also multiple ways to install pip - the new bootstrap will just be the simplest. "extractpip" is a bit clunky, but it does exactly what it says on the tin and should be a hidden implementation detail the vast majority of the time. I also believe that choice of name may help avoid the problem of users expecting a global pip update to implicitly affect future virtual environments. > Naming things is hard. Yup :) Cheers, Nick. > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130919/5887c9c5/attachment.html> From p.f.moore at gmail.com Thu Sep 19 15:43:09 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 19 Sep 2013 14:43:09 +0100 Subject: [Python-Dev] PEP 453 Round 4 - Explicit bootstrapping of pip in Python installations In-Reply-To: <C1831192-CF74-4CBC-BC55-A8CEEDF96D46@stufft.io> References: <C1831192-CF74-4CBC-BC55-A8CEEDF96D46@stufft.io> Message-ID: <CACac1F8J7H8YZ5HjQDVydqu5PJJ7AF2d+qMKoqMg=Tqumzrf5w@mail.gmail.com> On 19 September 2013 14:27, Donald Stufft <donald at stufft.io> wrote: > Major changes: > > * Removal of the option to fetch pip from PyPI in order not to modify the trust model of the Python installers > * Consequently rename the model from ``getpip`` to ``extractpip`` If extractpip (I agree, I don't like the name, installpip is better) only ever unpacks the bundled pip, and it's always run, why bother? Why not just bundle pip directly into site-packages? The extra step seems to add little or no value. Paul From donald at stufft.io Thu Sep 19 15:43:46 2013 From: donald at stufft.io (Donald Stufft) Date: Thu, 19 Sep 2013 09:43:46 -0400 Subject: [Python-Dev] PEP 453 Round 4 - Explicit bootstrapping of pip in Python installations In-Reply-To: <20130919133601.GA6420@helios.localdomain> References: <C1831192-CF74-4CBC-BC55-A8CEEDF96D46@stufft.io> <20130919133601.GA6420@helios.localdomain> Message-ID: <76C8A128-90AC-453F-8E46-81A0D29D8D56@stufft.io> On Sep 19, 2013, at 9:36 AM, Paul Tagliamonte <paultag at debian.org> wrote: > On Thu, Sep 19, 2013 at 09:27:24AM -0400, Donald Stufft wrote: >> Rationale >> ========= >> >> Currently, on systems without a platform package manager and repository, >> installing a third-party Python package into a freshly installed Python >> requires first identifying an appropriate package manager and then >> installing it. > > Howdy Donald, > > Thanks for helping make Python better. > > However, speaking as a Debian-folk (but not for the Debian-folk, I'll > let barry take care of that), many users of Python software don't even > know what pip is (or even that Python exists) -- as such, I find it very > unlikely that a development tool would be shiped as part of the Python > distribution in Debian. I don't see this changing, even with this pep. > python-pip is still installable, but I don't see it pulled in by > default. That is obviously Debian's right. Hopefully if Debian does *not* pull it in by default they'll be the odd man out and documentation can be updated to say that on Debian based systems additional steps to get the "standard" Python install must be taken. I believe Fedora has +1'd this so if this gets accepted there will likely be that additional difference. > > >> Even on systems that *do* have a platform package manager, it is unlikely to >> include every package that is available on the Python Package Index, and > > Yes. This is true. However, it's not Debian's job to include all of > pypi. Pypi allows anyone to upload, and we have quite a bit of attention > to concerns such as providing source for everything (less of a concern > for Python, but pickles can still contain data not in the prefered form > of modification), and proper free software licensing. > > We also have a concern about stability; so we manage all the package set > together to allow non-technical end-users to install stuff and not worry > about breaking their system. It's not always the case where you can > upgrade a library without breaking API. This statement isn't meant to imply that it's Debian's job to include all of PyPI, it merely calls out the fact that people often do wish to install things that are not available in an OS repository while they are available on PyPI. > > > I'm trimming the rest, since I don't want to get dragged into > side-conversations about pip as a package manager. > > > Cheers, > Paul > > > -- > .''`. Paul Tagliamonte <paultag at debian.org> > : :' : Proud Debian Developer > `. `'` 4096R / 8F04 9AD8 2C92 066C 7352 D28A 7B58 5B30 807C 2A87 > `- http://people.debian.org/~paultag ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: <http://mail.python.org/pipermail/python-dev/attachments/20130919/15eae484/attachment.sig> From ncoghlan at gmail.com Thu Sep 19 15:48:49 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 19 Sep 2013 23:48:49 +1000 Subject: [Python-Dev] PEP 453 Round 4 - Explicit bootstrapping of pip in Python installations In-Reply-To: <20130919133601.GA6420@helios.localdomain> References: <C1831192-CF74-4CBC-BC55-A8CEEDF96D46@stufft.io> <20130919133601.GA6420@helios.localdomain> Message-ID: <CADiSq7eG3ExCmpWUswKoR9Ya9=+if5ToUoQKd7o3yVWpLOjzaQ@mail.gmail.com> On 19 Sep 2013 23:37, "Paul Tagliamonte" <paultag at debian.org> wrote: > > On Thu, Sep 19, 2013 at 09:27:24AM -0400, Donald Stufft wrote: > > Rationale > > ========= > > > > Currently, on systems without a platform package manager and repository, > > installing a third-party Python package into a freshly installed Python > > requires first identifying an appropriate package manager and then > > installing it. > > Howdy Donald, > > Thanks for helping make Python better. > > However, speaking as a Debian-folk (but not for the Debian-folk, I'll > let barry take care of that), many users of Python software don't even > know what pip is (or even that Python exists) -- as such, I find it very > unlikely that a development tool would be shiped as part of the Python > distribution in Debian. I don't see this changing, even with this pep. > python-pip is still installable, but I don't see it pulled in by > default. > > > > Even on systems that *do* have a platform package manager, it is unlikely to > > include every package that is available on the Python Package Index, and > > Yes. This is true. However, it's not Debian's job to include all of > pypi. Pypi allows anyone to upload, and we have quite a bit of attention > to concerns such as providing source for everything (less of a concern > for Python, but pickles can still contain data not in the prefered form > of modification), and proper free software licensing. > > We also have a concern about stability; so we manage all the package set > together to allow non-technical end-users to install stuff and not worry > about breaking their system. It's not always the case where you can > upgrade a library without breaking API. > > > I'm trimming the rest, since I don't want to get dragged into > side-conversations about pip as a package manager. That's OK, especially if running "pip" recommends installing "python-pip" and "pyvenv" and "extractpip" are also separated out so that running "pyvenv" recommends installing a suitable package that depends on both Python and "python-pip". We can add that to the PEP as an alternative approach for redistributors to take that still offers a reasonable end-user experience. Cheers, Nick. > > > Cheers, > Paul > > > -- > .''`. Paul Tagliamonte <paultag at debian.org> > : :' : Proud Debian Developer > `. `'` 4096R / 8F04 9AD8 2C92 066C 7352 D28A 7B58 5B30 807C 2A87 > `- http://people.debian.org/~paultag -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130919/354cad7b/attachment.html> From solipsis at pitrou.net Thu Sep 19 15:50:44 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 19 Sep 2013 15:50:44 +0200 Subject: [Python-Dev] PEP 453 Round 4 - Explicit bootstrapping of pip in Python installations References: <C1831192-CF74-4CBC-BC55-A8CEEDF96D46@stufft.io> Message-ID: <20130919155044.41d992f6@pitrou.net> Le Thu, 19 Sep 2013 09:27:24 -0400, Donald Stufft <donald at stufft.io> a ?crit : > We've updated PEP453 based on some of the early feedback we've gotten > from -dev and Martin. > > Major changes: > > * Removal of the option to fetch pip from PyPI in order not to modify > the trust model of the Python installers > * Consequently rename the model from ``getpip`` to ``extractpip`` "ensurepip" ? What happens if there is already a higher pip version installed? I suppose "extractpip" doesn't do anything in that case? (sorry, perhaps it's mentioned in the PEP and I haven't seen it. The PEP has become so long that I've only skimmed through it.) Regards Antoine. From donald at stufft.io Thu Sep 19 15:52:01 2013 From: donald at stufft.io (Donald Stufft) Date: Thu, 19 Sep 2013 09:52:01 -0400 Subject: [Python-Dev] PEP 453 Round 4 - Explicit bootstrapping of pip in Python installations In-Reply-To: <CACac1F8J7H8YZ5HjQDVydqu5PJJ7AF2d+qMKoqMg=Tqumzrf5w@mail.gmail.com> References: <C1831192-CF74-4CBC-BC55-A8CEEDF96D46@stufft.io> <CACac1F8J7H8YZ5HjQDVydqu5PJJ7AF2d+qMKoqMg=Tqumzrf5w@mail.gmail.com> Message-ID: <C1F08515-8078-4259-906A-8D1B8A6477F4@stufft.io> On Sep 19, 2013, at 9:43 AM, Paul Moore <p.f.moore at gmail.com> wrote: > On 19 September 2013 14:27, Donald Stufft <donald at stufft.io> wrote: >> Major changes: >> >> * Removal of the option to fetch pip from PyPI in order not to modify the trust model of the Python installers >> * Consequently rename the model from ``getpip`` to ``extractpip`` > > If extractpip (I agree, I don't like the name, installpip is better) > only ever unpacks the bundled pip, and it's always run, why bother? > Why not just bundle pip directly into site-packages? The extra step > seems to add little or no value. > > Paul Well it's not always run (the PEP has it as an option in the installers that is checked by default) but even if it were always installed: - Nick has stated something about making it clear in the OSs installer DB which files are owned by Python and which are owned by pip - Upgrading becomes harder, instead of simply using pip's own mechanism the installer needs to take care not to clobber a user installed pip that is even newer than the bundled version. - "Fixing" a broken environment. If someone accidentally uninstalls pip this provides a simple way to reinstall it that doesn't require the old standby of getpip.py ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: <http://mail.python.org/pipermail/python-dev/attachments/20130919/b8d08673/attachment.sig> From donald at stufft.io Thu Sep 19 15:54:08 2013 From: donald at stufft.io (Donald Stufft) Date: Thu, 19 Sep 2013 09:54:08 -0400 Subject: [Python-Dev] PEP 453 Round 4 - Explicit bootstrapping of pip in Python installations In-Reply-To: <20130919155044.41d992f6@pitrou.net> References: <C1831192-CF74-4CBC-BC55-A8CEEDF96D46@stufft.io> <20130919155044.41d992f6@pitrou.net> Message-ID: <E5558EB5-DFF3-4566-8EB4-42A5980667A8@stufft.io> On Sep 19, 2013, at 9:50 AM, Antoine Pitrou <solipsis at pitrou.net> wrote: > Le Thu, 19 Sep 2013 09:27:24 -0400, > Donald Stufft <donald at stufft.io> a ?crit : >> We've updated PEP453 based on some of the early feedback we've gotten >> from -dev and Martin. >> >> Major changes: >> >> * Removal of the option to fetch pip from PyPI in order not to modify >> the trust model of the Python installers >> * Consequently rename the model from ``getpip`` to ``extractpip`` > > "ensurepip" ? > > What happens if there is already a higher pip version installed? > I suppose "extractpip" doesn't do anything in that case? > > (sorry, perhaps it's mentioned in the PEP and I haven't seen it. > The PEP has become so long that I've only skimmed through it.) We could explicitly call this out but it's implied it will do nothing because it's going to just call out to to the privately installed pip to actually install itself, and pip itself won't "upgrade" to an older version. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: <http://mail.python.org/pipermail/python-dev/attachments/20130919/154ea63d/attachment.sig> From donald at stufft.io Thu Sep 19 15:57:02 2013 From: donald at stufft.io (Donald Stufft) Date: Thu, 19 Sep 2013 09:57:02 -0400 Subject: [Python-Dev] PEP 453 Round 4 - Explicit bootstrapping of pip in Python installations In-Reply-To: <20130919135025.GA7393@helios.localdomain> References: <C1831192-CF74-4CBC-BC55-A8CEEDF96D46@stufft.io> <20130919133601.GA6420@helios.localdomain> <CADiSq7eG3ExCmpWUswKoR9Ya9=+if5ToUoQKd7o3yVWpLOjzaQ@mail.gmail.com> <20130919135025.GA7393@helios.localdomain> Message-ID: <1E689A33-EA02-4967-BE7B-FD99BEE1BB40@stufft.io> On Sep 19, 2013, at 9:50 AM, Paul Tagliamonte <paultag at debian.org> wrote: > On Thu, Sep 19, 2013 at 11:48:49PM +1000, Nick Coghlan wrote: >> That's OK, especially if running "pip" recommends installing "python-pip" and >> "pyvenv" and "extractpip" are also separated out so that running "pyvenv" >> recommends installing a suitable package that depends on both Python and >> "python-pip". > > Sounds sane to me! > >> We can add that to the PEP as an alternative approach for redistributors to >> take that still offers a reasonable end-user experience. > > That'd really be great, and I'm sure it'd cut off the small bit of > concern some Debian-folk have about this stuff :) This also sounds reasonable to me. Really the PEP is about reducing user friction, especially for new users, so the ideal method is to ensure it's always installed but it'd be totally OK to do what Nick suggested as that still at leasts lets pypi packages to simply document installing as ``pip install <package>`` and if it's not installed by default on Debian they'll get a good message telling them what they need to do. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: <http://mail.python.org/pipermail/python-dev/attachments/20130919/7fe7863e/attachment-0001.sig> From ncoghlan at gmail.com Thu Sep 19 15:58:58 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 19 Sep 2013 23:58:58 +1000 Subject: [Python-Dev] PEP 453 Round 4 - Explicit bootstrapping of pip in Python installations In-Reply-To: <CACac1F8J7H8YZ5HjQDVydqu5PJJ7AF2d+qMKoqMg=Tqumzrf5w@mail.gmail.com> References: <C1831192-CF74-4CBC-BC55-A8CEEDF96D46@stufft.io> <CACac1F8J7H8YZ5HjQDVydqu5PJJ7AF2d+qMKoqMg=Tqumzrf5w@mail.gmail.com> Message-ID: <CADiSq7eko=mSqo3MtziHyU8dmR-zjQoCVmsCM33Lz2PWq3AzXg@mail.gmail.com> On 19 Sep 2013 23:43, "Paul Moore" <p.f.moore at gmail.com> wrote: > > On 19 September 2013 14:27, Donald Stufft <donald at stufft.io> wrote: > > Major changes: > > > > * Removal of the option to fetch pip from PyPI in order not to modify the trust model of the Python installers > > * Consequently rename the model from ``getpip`` to ``extractpip`` > > If extractpip (I agree, I don't like the name, installpip is better) > only ever unpacks the bundled pip, and it's always run, why bother? > Why not just bundle pip directly into site-packages? The extra step > seems to add little or no value. It's not always run - it's opt-out for the Windows and Mac OS X installers and not invoked implicitly at all if installing from source. The bootstrap also needs to be available for use by "venv". There's also the significant fact that python-dev still has a blanket ban on externally maintained standard library modules (for good reasons) and the clear separation between the bootstrap module and pip itself is what is allowing this proposal to adhere to that guideline. Cheers, Nick. > > Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130919/4106fdeb/attachment.html> From ncoghlan at gmail.com Thu Sep 19 16:05:26 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 20 Sep 2013 00:05:26 +1000 Subject: [Python-Dev] PEP 453 Round 4 - Explicit bootstrapping of pip in Python installations In-Reply-To: <20130919155044.41d992f6@pitrou.net> References: <C1831192-CF74-4CBC-BC55-A8CEEDF96D46@stufft.io> <20130919155044.41d992f6@pitrou.net> Message-ID: <CADiSq7e9gnzttq--i_T-+YqEASHucs1jGERb-AKUF=y9aENUMg@mail.gmail.com> On 19 Sep 2013 23:53, "Antoine Pitrou" <solipsis at pitrou.net> wrote: > > Le Thu, 19 Sep 2013 09:27:24 -0400, > Donald Stufft <donald at stufft.io> a ?crit : > > We've updated PEP453 based on some of the early feedback we've gotten > > from -dev and Martin. > > > > Major changes: > > > > * Removal of the option to fetch pip from PyPI in order not to modify > > the trust model of the Python installers > > * Consequently rename the model from ``getpip`` to ``extractpip`` > > "ensurepip" ? Oh, I like it - describes exactly why the module exists :) > > What happens if there is already a higher pip version installed? > I suppose "extractpip" doesn't do anything in that case? > > (sorry, perhaps it's mentioned in the PEP and I haven't seen it. > The PEP has become so long that I've only skimmed through it.) As Donald said, the fact nothing happens in that case is currently only implied by the fact this is the way pip's "--upgrade" option works. We should make that explicit in the PEP. Cheers, Nick. > > Regards > > Antoine. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130920/1e593cca/attachment.html> From donald at stufft.io Thu Sep 19 16:15:45 2013 From: donald at stufft.io (Donald Stufft) Date: Thu, 19 Sep 2013 10:15:45 -0400 Subject: [Python-Dev] PEP 453 Round 4 - Explicit bootstrapping of pip in Python installations In-Reply-To: <C1831192-CF74-4CBC-BC55-A8CEEDF96D46@stufft.io> References: <C1831192-CF74-4CBC-BC55-A8CEEDF96D46@stufft.io> Message-ID: <28F9A6A3-4758-45F7-8D6F-EEBEB2C73512@stufft.io> I've updated the PEP to include the solution for Debian that Nick mentioned and to clarify the behavior in the upgrade case. Online: http://hg.python.org/peps/rev/6c1b658bc16c diff -r 2899ba3bcef1 pep-0453.txt --- a/pep-0453.txt Thu Sep 19 09:23:03 2013 -0400 +++ b/pep-0453.txt Thu Sep 19 10:12:59 2013 -0400 @@ -294,7 +294,9 @@ This ensures that, by default, installing or updating CPython will ensure Update PEP453 to address Debian's concerns and Antoine's that the installed version of pip is at least as recent as the one included -with that version of CPython. +with that version of CPython. If a newer version of pip has already been +installed then ``python -m extractpip --upgrade`` will simply return without +doing anything. Installing from source @@ -532,6 +534,12 @@ * This may take the form of separate packages with dependencies on each other so that installing the Python package installs the pip package and installing the pip package installs the Python package. + * Another reasonable way to implement this is to package pip separately but + ensure that there is some sort of global hook that will recommend + installing the separate pip package when a user executes ``pip`` without + it being installed. Systems that choose this option should ensure that + the ``pyvenv`` command still installs pip into the virtual environment + by default. * Do not remove the bundled copy of pip. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: <http://mail.python.org/pipermail/python-dev/attachments/20130919/4b41c680/attachment.sig> From paultag at debian.org Thu Sep 19 15:36:01 2013 From: paultag at debian.org (Paul Tagliamonte) Date: Thu, 19 Sep 2013 09:36:01 -0400 Subject: [Python-Dev] PEP 453 Round 4 - Explicit bootstrapping of pip in Python installations In-Reply-To: <C1831192-CF74-4CBC-BC55-A8CEEDF96D46@stufft.io> References: <C1831192-CF74-4CBC-BC55-A8CEEDF96D46@stufft.io> Message-ID: <20130919133601.GA6420@helios.localdomain> On Thu, Sep 19, 2013 at 09:27:24AM -0400, Donald Stufft wrote: > Rationale > ========= > > Currently, on systems without a platform package manager and repository, > installing a third-party Python package into a freshly installed Python > requires first identifying an appropriate package manager and then > installing it. Howdy Donald, Thanks for helping make Python better. However, speaking as a Debian-folk (but not for the Debian-folk, I'll let barry take care of that), many users of Python software don't even know what pip is (or even that Python exists) -- as such, I find it very unlikely that a development tool would be shiped as part of the Python distribution in Debian. I don't see this changing, even with this pep. python-pip is still installable, but I don't see it pulled in by default. > Even on systems that *do* have a platform package manager, it is unlikely to > include every package that is available on the Python Package Index, and Yes. This is true. However, it's not Debian's job to include all of pypi. Pypi allows anyone to upload, and we have quite a bit of attention to concerns such as providing source for everything (less of a concern for Python, but pickles can still contain data not in the prefered form of modification), and proper free software licensing. We also have a concern about stability; so we manage all the package set together to allow non-technical end-users to install stuff and not worry about breaking their system. It's not always the case where you can upgrade a library without breaking API. I'm trimming the rest, since I don't want to get dragged into side-conversations about pip as a package manager. Cheers, Paul -- .''`. Paul Tagliamonte <paultag at debian.org> : :' : Proud Debian Developer `. `'` 4096R / 8F04 9AD8 2C92 066C 7352 D28A 7B58 5B30 807C 2A87 `- http://people.debian.org/~paultag -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: Digital signature URL: <http://mail.python.org/pipermail/python-dev/attachments/20130919/21d80dd8/attachment.sig> From paultag at debian.org Thu Sep 19 15:50:25 2013 From: paultag at debian.org (Paul Tagliamonte) Date: Thu, 19 Sep 2013 09:50:25 -0400 Subject: [Python-Dev] PEP 453 Round 4 - Explicit bootstrapping of pip in Python installations In-Reply-To: <CADiSq7eG3ExCmpWUswKoR9Ya9=+if5ToUoQKd7o3yVWpLOjzaQ@mail.gmail.com> References: <C1831192-CF74-4CBC-BC55-A8CEEDF96D46@stufft.io> <20130919133601.GA6420@helios.localdomain> <CADiSq7eG3ExCmpWUswKoR9Ya9=+if5ToUoQKd7o3yVWpLOjzaQ@mail.gmail.com> Message-ID: <20130919135025.GA7393@helios.localdomain> On Thu, Sep 19, 2013 at 11:48:49PM +1000, Nick Coghlan wrote: > That's OK, especially if running "pip" recommends installing "python-pip" and > "pyvenv" and "extractpip" are also separated out so that running "pyvenv" > recommends installing a suitable package that depends on both Python and > "python-pip". Sounds sane to me! > We can add that to the PEP as an alternative approach for redistributors to > take that still offers a reasonable end-user experience. That'd really be great, and I'm sure it'd cut off the small bit of concern some Debian-folk have about this stuff :) Cheers, Paul -- .''`. Paul Tagliamonte <paultag at debian.org> : :' : Proud Debian Developer `. `'` 4096R / 8F04 9AD8 2C92 066C 7352 D28A 7B58 5B30 807C 2A87 `- http://people.debian.org/~paultag -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: Digital signature URL: <http://mail.python.org/pipermail/python-dev/attachments/20130919/bd096e17/attachment.sig> From ericsnowcurrently at gmail.com Thu Sep 19 22:21:40 2013 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Thu, 19 Sep 2013 14:21:40 -0600 Subject: [Python-Dev] PEP 447: add type.__locallookup__ In-Reply-To: <CADiSq7eQ457=Vva-_3xbpi8o2EtS9ea8Q3XG=oHe0xBBAtzywQ@mail.gmail.com> References: <7D6D4488-2855-41F8-A9F2-1FF23B9E3A56@mac.com> <522DDB74.5080609@stoneleaf.us> <CAP7+vJJTR=+cRY+iQgHY4ujTsC3_QyWn83dnK96rWO6bV9vFLQ@mail.gmail.com> <522DECA3.2030709@hotpy.org> <CAP7+vJ+thJvYpk0d0rnFO6bm88sTJ_t1QdYFB0SdPOGNDABuag@mail.gmail.com> <522DF204.7040109@hotpy.org> <0752376899dd7436f6115bfc03594574@chopin.edu.pl> <392015291ec843afb1bc85632055729d@BLUPR03MB199.namprd03.prod.outlook.com> <20130913065236.GF16820@ando> <CADiSq7fNhAgGUE9-1EdASM5MH0wx7W-mJTi6pRv=45r3K6LvmA@mail.gmail.com> <20130913122328.GI16820@ando> <CADiSq7cdSEHryP+LMppRdabTuUG-GzMd56ub-oWpFNKPHNzsHA@mail.gmail.com> <9BBAFB63-68C5-441B-8ED4-43604241AA42@mac.com> <CACac1F_gT=mDEV3_VwvhjvS2H=JmNodWdP1vqMcEopWKMrPPHA@mail.gmail.com> <CADiSq7eQ457=Vva-_3xbpi8o2EtS9ea8Q3XG=oHe0xBBAtzywQ@mail.gmail.com> Message-ID: <CALFfu7D2U0oG4LjRh0wLKPdaxAJv31-azR8LEG=+=2HgkNKzWA@mail.gmail.com> On Thu, Sep 19, 2013 at 4:12 AM, Nick Coghlan <ncoghlan at gmail.com> wrote: > On 19 Sep 2013 20:00, "Paul Moore" <p.f.moore at gmail.com> wrote: > > > > On 19 September 2013 10:32, Ronald Oussoren <ronaldoussoren at mac.com> > wrote: > > > The first time a method is called the bridge looks for an Objective-C > selector > > > with the same name and adds that to the class dictionary. This works > fine for normal > > > method lookups, by overriding __getattribute__, but causes problems > with super: > > > super happily ignores __getattribute__ and peeks in the class __dict__ > which may > > > not yet contain the name we're looking for and that can result in > incorrect results > > > (both incorrect AttributeErrors and totally incorrect results when the > name is > > > not yet present in the parent class' __dict__ but is in the > grandparent's __dict__). > > > > As an alternative approach, could you use a custom dict subclass as > > the class __dict__, and catch the peeking in the class __dict__ that > > way? Or is this one of those places where only a real dict will do? > > Even Python 3 doesn't let you control the *runtime* type of the class > dict, only the type used during evaluation of the class body. > > I've played with changing that - it makes for a rather special interpreter > experience :) > Same here. :) The PyDict_* API is not your friend for that. It's why I gave up on using a C OrderedDict for tp_dict (opting for a __definition_order__ attribute on classes instead). -eric -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130919/bc4c279e/attachment.html> From ericsnowcurrently at gmail.com Thu Sep 19 22:23:12 2013 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Thu, 19 Sep 2013 14:23:12 -0600 Subject: [Python-Dev] PEP 447: add type.__locallookup__ In-Reply-To: <C1DAADB8-E611-4359-B0F4-7914E9579368@mac.com> References: <7D6D4488-2855-41F8-A9F2-1FF23B9E3A56@mac.com> <522DDB74.5080609@stoneleaf.us> <CAP7+vJJTR=+cRY+iQgHY4ujTsC3_QyWn83dnK96rWO6bV9vFLQ@mail.gmail.com> <522DECA3.2030709@hotpy.org> <CAP7+vJ+thJvYpk0d0rnFO6bm88sTJ_t1QdYFB0SdPOGNDABuag@mail.gmail.com> <522DF204.7040109@hotpy.org> <0752376899dd7436f6115bfc03594574@chopin.edu.pl> <392015291ec843afb1bc85632055729d@BLUPR03MB199.namprd03.prod.outlook.com> <20130913065236.GF16820@ando> <CADiSq7fNhAgGUE9-1EdASM5MH0wx7W-mJTi6pRv=45r3K6LvmA@mail.gmail.com> <20130913122328.GI16820@ando> <CADiSq7cdSEHryP+LMppRdabTuUG-GzMd56ub-oWpFNKPHNzsHA@mail.gmail.com> <9BBAFB63-68C5-441B-8ED4-43604241AA42@mac.com> <CACac1F_gT=mDEV3_VwvhjvS2H=JmNodWdP1vqMcEopWKMrPPHA@mail.gmail.com> <C1DAADB8-E611-4359-B0F4-7914E9579368@mac.com> Message-ID: <CALFfu7CTHJohah-detWO=WoDe6jhLBCXcDgez2xgPLshZ-i13g@mail.gmail.com> On Thu, Sep 19, 2013 at 4:04 AM, Ronald Oussoren <ronaldoussoren at mac.com>wrote: > The C code uses PyDict_GetItem and AFAIK that doesn't look for a > __getitem__ > implementation in a subclass. > Yeah, the PyDict_* API is definitely not subclass friendly. :( -eric -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130919/19f7f5c4/attachment.html> From ericsnowcurrently at gmail.com Thu Sep 19 22:25:20 2013 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Thu, 19 Sep 2013 14:25:20 -0600 Subject: [Python-Dev] PEP 447: add type.__locallookup__ In-Reply-To: <4CE957E3-ECE7-4DD1-AB62-2EA7776B9F85@mac.com> References: <7D6D4488-2855-41F8-A9F2-1FF23B9E3A56@mac.com> <522DDB74.5080609@stoneleaf.us> <CAP7+vJJTR=+cRY+iQgHY4ujTsC3_QyWn83dnK96rWO6bV9vFLQ@mail.gmail.com> <522DECA3.2030709@hotpy.org> <CAP7+vJ+thJvYpk0d0rnFO6bm88sTJ_t1QdYFB0SdPOGNDABuag@mail.gmail.com> <522DF204.7040109@hotpy.org> <0752376899dd7436f6115bfc03594574@chopin.edu.pl> <392015291ec843afb1bc85632055729d@BLUPR03MB199.namprd03.prod.outlook.com> <20130913065236.GF16820@ando> <CADiSq7fNhAgGUE9-1EdASM5MH0wx7W-mJTi6pRv=45r3K6LvmA@mail.gmail.com> <20130913122328.GI16820@ando> <CADiSq7cdSEHryP+LMppRdabTuUG-GzMd56ub-oWpFNKPHNzsHA@mail.gmail.com> <4CE957E3-ECE7-4DD1-AB62-2EA7776B9F85@mac.com> Message-ID: <CALFfu7Bm5gvY+xDrDwpKwhfwhhSUSRYQE=T=pPP_Yo0XB7SioQ@mail.gmail.com> On Thu, Sep 19, 2013 at 3:49 AM, Ronald Oussoren <ronaldoussoren at mac.com>wrote: > On 14 Sep, 2013, at 8:30, Nick Coghlan <ncoghlan at gmail.com> wrote: > > However, I agree the current wording only conveys that to the handful > > of people that already know exactly when in the attribute lookup > > sequence that step occurs, which is a rather niche audience :) > > I've been fooling around with this long enough that I forgot that not > everyone knows this :-). > > I guess I'd better include a clearer and more complete description > of the current attribute resolution protocol and how my proposal affects > that. A nice readable Python implementation of that protocol would be nice > to have regardless of the fate of this PEP. > +1 -eric -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130919/f22ef310/attachment.html> From benhgift at gmail.com Thu Sep 19 22:54:08 2013 From: benhgift at gmail.com (Ben Gift) Date: Thu, 19 Sep 2013 13:54:08 -0700 Subject: [Python-Dev] Use an empty def as a lambda Message-ID: <CAA1uNA=tPhX2y0wP8tdNGfxZ1iV8OJthdd9YpXQsbRHCZDzb_g@mail.gmail.com> I think the lambda keyword is difficult to understand for many people. It would be more pythonic to use an empty def call instead. For instance this: words.sort(key = lambda x: x[2]) could look like this: words.sort(key = def (x): x[2]) It's obvious and explicit that we're creating an unnamed, anonymous function this way. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130919/0b985608/attachment-0001.html> From joe.pinsonault at gmail.com Thu Sep 19 23:00:07 2013 From: joe.pinsonault at gmail.com (Joe Pinsonault) Date: Thu, 19 Sep 2013 14:00:07 -0700 Subject: [Python-Dev] Use an empty def as a lambda In-Reply-To: <CAA1uNA=tPhX2y0wP8tdNGfxZ1iV8OJthdd9YpXQsbRHCZDzb_g@mail.gmail.com> References: <CAA1uNA=tPhX2y0wP8tdNGfxZ1iV8OJthdd9YpXQsbRHCZDzb_g@mail.gmail.com> Message-ID: <CALFC9k-QaxOGBoCKK6Lvx+b5d2rxfaA8qendbhE22+jE=4EoCg@mail.gmail.com> I think it's a great idea personally. It's explicit and obvious. "lamda" is too computer sciencey On Sep 19, 2013 1:55 PM, "Ben Gift" <benhgift at gmail.com> wrote: > I think the lambda keyword is difficult to understand for many people. It > would be more pythonic to use an empty def call instead. > > For instance this: > > words.sort(key = lambda x: x[2]) > > could look like this: > > words.sort(key = def (x): x[2]) > > It's obvious and explicit that we're creating an unnamed, anonymous > function this way. > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/joe.pinsonault%40gmail.com > > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130919/88a252e1/attachment.html> From ncoghlan at gmail.com Thu Sep 19 23:17:58 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 20 Sep 2013 07:17:58 +1000 Subject: [Python-Dev] Use an empty def as a lambda In-Reply-To: <CALFC9k-QaxOGBoCKK6Lvx+b5d2rxfaA8qendbhE22+jE=4EoCg@mail.gmail.com> References: <CAA1uNA=tPhX2y0wP8tdNGfxZ1iV8OJthdd9YpXQsbRHCZDzb_g@mail.gmail.com> <CALFC9k-QaxOGBoCKK6Lvx+b5d2rxfaA8qendbhE22+jE=4EoCg@mail.gmail.com> Message-ID: <CADiSq7dkj6aP-909ifBmt3jWxSwD4i=eM==T+vQK7y1TnLW2TA@mail.gmail.com> On 20 Sep 2013 07:04, "Joe Pinsonault" <joe.pinsonault at gmail.com> wrote: > > I think it's a great idea personally. It's explicit and obvious. "lamda" is too computer sciencey This suggestion has been made many times, occasionally with the associated "must be contained in parentheses when used as an expression" caveat that is needed to avoid making the language grammar ambiguous at the statement level. It mainly runs afoul of two problems: - reusing the same keyword would make the additional syntactic restrictions of the expression form even more confusing. - Guido doesn't particularly like the notion of functions-as-expressions in the first place (which is why lambda was on thin ice when Python 3 was being designed), so doesn't actually mind the fact that people avoid using them because they don't like the keyword. Cheers, Nick. > > On Sep 19, 2013 1:55 PM, "Ben Gift" <benhgift at gmail.com> wrote: >> >> I think the lambda keyword is difficult to understand for many people. It would be more pythonic to use an empty def call instead. >> >> For instance this: >> >> words.sort(key = lambda x: x[2]) >> >> could look like this: >> >> words.sort(key = def (x): x[2]) >> >> It's obvious and explicit that we're creating an unnamed, anonymous function this way. >> >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: https://mail.python.org/mailman/options/python-dev/joe.pinsonault%40gmail.com >> > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130920/290c0ca3/attachment.html> From catch-all at masklinn.net Thu Sep 19 23:21:11 2013 From: catch-all at masklinn.net (Xavier Morel) Date: Thu, 19 Sep 2013 23:21:11 +0200 Subject: [Python-Dev] Use an empty def as a lambda In-Reply-To: <CADiSq7dkj6aP-909ifBmt3jWxSwD4i=eM==T+vQK7y1TnLW2TA@mail.gmail.com> References: <CAA1uNA=tPhX2y0wP8tdNGfxZ1iV8OJthdd9YpXQsbRHCZDzb_g@mail.gmail.com> <CALFC9k-QaxOGBoCKK6Lvx+b5d2rxfaA8qendbhE22+jE=4EoCg@mail.gmail.com> <CADiSq7dkj6aP-909ifBmt3jWxSwD4i=eM==T+vQK7y1TnLW2TA@mail.gmail.com> Message-ID: <B0854DED-7AC6-41E7-B409-E32C0ADC2C03@masklinn.net> On 2013-09-19, at 23:17 , Nick Coghlan wrote: > On 20 Sep 2013 07:04, "Joe Pinsonault" <joe.pinsonault at gmail.com> wrote: >> >> I think it's a great idea personally. It's explicit and obvious. "lamda" > is too computer sciencey > > This suggestion has been made many times, occasionally with the associated > "must be contained in parentheses when used as an expression" caveat that > is needed to avoid making the language grammar ambiguous at the statement > level. Examples of some of these times: https://wiki.python.org/moin/AlternateLambdaSyntax https://mail.python.org/pipermail/python-dev/2006-February/060415.html https://mail.python.org/pipermail/python-dev/2006-February/thread.html#60415 Unless significant new insight is developed or Guido has picked the functional bug at dropbox, merely suggesting a name change from lambda to def (which has already been suggested in the past) probably isn't going to cut it. From alexander.belopolsky at gmail.com Fri Sep 20 00:19:47 2013 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Thu, 19 Sep 2013 18:19:47 -0400 Subject: [Python-Dev] Use an empty def as a lambda In-Reply-To: <CAA1uNA=tPhX2y0wP8tdNGfxZ1iV8OJthdd9YpXQsbRHCZDzb_g@mail.gmail.com> References: <CAA1uNA=tPhX2y0wP8tdNGfxZ1iV8OJthdd9YpXQsbRHCZDzb_g@mail.gmail.com> Message-ID: <CAP7h-xbomsduOfwDMKBGk4bJfgRQBu7g16Db-eMmRhKJFK3usA@mail.gmail.com> On Thu, Sep 19, 2013 at 4:54 PM, Ben Gift <benhgift at gmail.com> wrote: > It would be more pythonic to use an empty def call instead. No, it won't. Python draws a very strong distinction between expressions and statements. This line has been blurred somewhat with the advent of comprehensions and the if-else expression, but it would still require more benefit than three characters in a keyword saving to allow def use in both statements and expressions. The following, for example, does not look pythonic at all: *def* transform(*seq*, *func*=*def*(*x*):*x*): ... (Note that I attempted to emulate syntax highlighting to make my point.) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130919/0ee8e0fd/attachment.html> From rymg19 at gmail.com Fri Sep 20 00:24:22 2013 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Thu, 19 Sep 2013 17:24:22 -0500 Subject: [Python-Dev] Use an empty def as a lambda In-Reply-To: <CAA1uNA=tPhX2y0wP8tdNGfxZ1iV8OJthdd9YpXQsbRHCZDzb_g@mail.gmail.com> References: <CAA1uNA=tPhX2y0wP8tdNGfxZ1iV8OJthdd9YpXQsbRHCZDzb_g@mail.gmail.com> Message-ID: <CAO41-mPj9jigQVBNaWUeLmJPxYkdiyAdV9dbBjDFWWxKq=_zmg@mail.gmail.com> Nice idea, BUT... Not sure how a parser addition that supports it would go. Imagine this: if you did a one-line function: def test(x): print(x) Python could interpret it two ways: `def` `name` `lparen` `name` `rparen` `colon`... OR, it could see it as a lambda-like thingamajig and throw a syntax error. And, if someone accidentally wrote: def (x): print(x) Python should throw a syntax error. But it won't. And it'll take the person a tad bit to realize he forgot the function name. Whoops. And, it just would be odd in general. On Thu, Sep 19, 2013 at 3:54 PM, Ben Gift <benhgift at gmail.com> wrote: > I think the lambda keyword is difficult to understand for many people. It > would be more pythonic to use an empty def call instead. > > For instance this: > > words.sort(key = lambda x: x[2]) > > could look like this: > > words.sort(key = def (x): x[2]) > > It's obvious and explicit that we're creating an unnamed, anonymous > function this way. > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/rymg19%40gmail.com > > -- Ryan -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130919/c3382176/attachment.html> From steve at pearwood.info Fri Sep 20 02:27:07 2013 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 20 Sep 2013 10:27:07 +1000 Subject: [Python-Dev] Use an empty def as a lambda In-Reply-To: <CAA1uNA=tPhX2y0wP8tdNGfxZ1iV8OJthdd9YpXQsbRHCZDzb_g@mail.gmail.com> References: <CAA1uNA=tPhX2y0wP8tdNGfxZ1iV8OJthdd9YpXQsbRHCZDzb_g@mail.gmail.com> Message-ID: <20130920002707.GN19939@ando> On Thu, Sep 19, 2013 at 01:54:08PM -0700, Ben Gift wrote: > I think the lambda keyword is difficult to understand for many people. It > would be more pythonic to use an empty def call instead. Hi Ben, and welcome! Is this your first post? I'm afraid I don't recognise your name. I think this discussion belongs on the python-ideas mailing list rather than here. Proposals for changes to syntax and functionality are normally expected to gather feedback on python-ideas before coming to python-dev for final approval or rejection. https://mail.python.org/mailman/listinfo/python-ideas http://docs.python.org/devguide/communication.html -- Steven From jcea at jcea.es Fri Sep 20 13:19:24 2013 From: jcea at jcea.es (Jesus Cea) Date: Fri, 20 Sep 2013 13:19:24 +0200 Subject: [Python-Dev] sys.intern should work on bytes Message-ID: <523C2F3C.7020605@jcea.es> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 """ sys.intern(b'12121212') Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: must be str, not bytes """ I wonder why. - -- Jes?s Cea Avi?n _/_/ _/_/_/ _/_/_/ jcea at jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ Twitter: @jcea _/_/ _/_/ _/_/_/_/_/ jabber / xmpp:jcea at jabber.org _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQCVAwUBUjwvPJlgi5GaxT1NAQIINQP/ZmEyPBSapa52yZRhQf8QSVSBm5tXpWrC k9MbcvsK/5K6ArRogkulk1GSu1EJPPvuHMAXX8EenNFBXPvDRm0mOxrHkcYw5IKx Ml2ORENm+EEM/ziUJMRtY4aqD5Jp6pXSSl9UmP5OQMDJfuabSrVqs7X2409OOhUj BXeg9HvURxo= =78M+ -----END PGP SIGNATURE----- From victor.stinner at gmail.com Fri Sep 20 14:04:11 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Fri, 20 Sep 2013 14:04:11 +0200 Subject: [Python-Dev] sys.intern should work on bytes In-Reply-To: <523C2F3C.7020605@jcea.es> References: <523C2F3C.7020605@jcea.es> Message-ID: <CAMpsgwammTPe_2UAM+AdqxmXP8c6hnqPiOXwMUKLvyTjFnQL6g@mail.gmail.com> 2013/9/20 Jesus Cea <jcea at jcea.es>: > """ > sys.intern(b'12121212') > Traceback (most recent call last): > File "<stdin>", line 1, in <module> > TypeError: must be str, not bytes > """ > > I wonder why. Intern strings optimize dictionary lookup. In Python 3, most dictionaries use str keys (ex: __dict__ of classes). What would you be the use case of interned bytes objets? Victor From solipsis at pitrou.net Fri Sep 20 14:15:51 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 20 Sep 2013 14:15:51 +0200 Subject: [Python-Dev] sys.intern should work on bytes References: <523C2F3C.7020605@jcea.es> Message-ID: <20130920141551.0ac2c769@pitrou.net> Le Fri, 20 Sep 2013 13:19:24 +0200, Jesus Cea <jcea at jcea.es> a ?crit : > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > """ > sys.intern(b'12121212') > Traceback (most recent call last): > File "<stdin>", line 1, in <module> > TypeError: must be str, not bytes > """ > > I wonder why. From http://docs.python.org/3.3/library/sys.html#sys.intern """sys.intern(string) Enter string in the table of ?interned? strings and return the interned string [...]""" In Python 3 context, "string" means "str". Regards Antoine. From jcea at jcea.es Fri Sep 20 15:14:37 2013 From: jcea at jcea.es (Jesus Cea) Date: Fri, 20 Sep 2013 15:14:37 +0200 Subject: [Python-Dev] sys.intern should work on bytes In-Reply-To: <CAMpsgwammTPe_2UAM+AdqxmXP8c6hnqPiOXwMUKLvyTjFnQL6g@mail.gmail.com> References: <523C2F3C.7020605@jcea.es> <CAMpsgwammTPe_2UAM+AdqxmXP8c6hnqPiOXwMUKLvyTjFnQL6g@mail.gmail.com> Message-ID: <523C4A3D.90901@jcea.es> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 20/09/13 14:04, Victor Stinner wrote: > What would you be the use case of interned bytes objets? Performance and memory. Pickle sizes (my particular issue now). - -- Jes?s Cea Avi?n _/_/ _/_/_/ _/_/_/ jcea at jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ Twitter: @jcea _/_/ _/_/ _/_/_/_/_/ jabber / xmpp:jcea at jabber.org _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQCVAwUBUjxKPZlgi5GaxT1NAQJ6FwQAim2H+OGpRD75KplahNhnKIfm9AUqVnHg CaLakWhADdHBYlit+DxRQsxtv5C7gyhhqMryydyvx97z33VaI2p1RGOOcK/lWdNw ETcetqJo8UswS2PSthJ0e5snOUsIeVJRomhJ48n8sJfIadCxAk6ozdMR75pHP5Y3 lJoUuUgdcJU= =vbuK -----END PGP SIGNATURE----- From solipsis at pitrou.net Fri Sep 20 15:31:12 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 20 Sep 2013 15:31:12 +0200 Subject: [Python-Dev] sys.intern should work on bytes References: <523C2F3C.7020605@jcea.es> <CAMpsgwammTPe_2UAM+AdqxmXP8c6hnqPiOXwMUKLvyTjFnQL6g@mail.gmail.com> <523C4A3D.90901@jcea.es> Message-ID: <20130920153112.5af58729@pitrou.net> Le Fri, 20 Sep 2013 15:14:37 +0200, Jesus Cea <jcea at jcea.es> a ?crit : > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > On 20/09/13 14:04, Victor Stinner wrote: > > What would you be the use case of interned bytes objets? > > Performance and memory. Pickle sizes (my particular issue now). sys.intern is an internal interpreter optimization and should be orthogonal to pickling. If pickle can't detect already-seen bytes object, then you may file an improvement request on the bug tracker. Regards Antoine. From jcea at jcea.es Fri Sep 20 15:33:05 2013 From: jcea at jcea.es (Jesus Cea) Date: Fri, 20 Sep 2013 15:33:05 +0200 Subject: [Python-Dev] sys.intern should work on bytes In-Reply-To: <20130920141551.0ac2c769@pitrou.net> References: <523C2F3C.7020605@jcea.es> <20130920141551.0ac2c769@pitrou.net> Message-ID: <523C4E91.4050905@jcea.es> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 20/09/13 14:15, Antoine Pitrou wrote: > From http://docs.python.org/3.3/library/sys.html#sys.intern > > """sys.intern(string) > > Enter string in the table of ?interned? strings and return the > interned string [...]""" > > > In Python 3 context, "string" means "str". I read that, Antoine. In fact I read the manual, I thought it was a mistake carried over from 2.x documentation, I tried it just in case before reporting the "documentation mistake", and I was surprised it was actually true :-). I know that intern is used for performance reasons internally to the interpreter. But I am thinking about memory usage optimizations. For instance, I have a pickle that is 14MB in size, when "interning" the strings on it (there are a lot of redundancy), the new size is only 3MB and it loads faster. I can do it because most data in the pickle are strings, I could NOT do it if I used bytes. I could do a manual "intern" for hashable objects by hand using an "object:object" dictionary (that would work for integers too), but I wonder if extending builtin "sys.intern" would be something to consider. Anyway, this pattern is easy enough: Instead of object = sys.intern(object) I could do interned = dict() ... object = interned.setdefault(object, object) - -- Jes?s Cea Avi?n _/_/ _/_/_/ _/_/_/ jcea at jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ Twitter: @jcea _/_/ _/_/ _/_/_/_/_/ jabber / xmpp:jcea at jabber.org _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQCVAwUBUjxOkZlgi5GaxT1NAQIOVgQAhN36yRAAQP1YWbDsXGSamgZnhEULTloB penRZYTYz/Ir/VM9l27GoXS7ThGrucAkkYZoJqXnUr2vyP0hq6rsfp+N5lzl61Nf mDJBCtAczzKNdYqQSgMQ+Ugk43KnbEFFX7SB9Y5IkYroWCeWq7+5y6KX3ZKBspXG lmXotLgpvW0= =/RNw -----END PGP SIGNATURE----- From benjamin at python.org Fri Sep 20 15:33:35 2013 From: benjamin at python.org (Benjamin Peterson) Date: Fri, 20 Sep 2013 09:33:35 -0400 Subject: [Python-Dev] sys.intern should work on bytes In-Reply-To: <523C4A3D.90901@jcea.es> References: <523C2F3C.7020605@jcea.es> <CAMpsgwammTPe_2UAM+AdqxmXP8c6hnqPiOXwMUKLvyTjFnQL6g@mail.gmail.com> <523C4A3D.90901@jcea.es> Message-ID: <CAPZV6o_z6gyiGSEwP2i7uDagjxdn8i1p-cUWEHN-JAUXR=+zFw@mail.gmail.com> Well, the pickler should memoize bytes objects if you have lots of the same one in a pickle... 2013/9/20 Jesus Cea <jcea at jcea.es>: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > On 20/09/13 14:04, Victor Stinner wrote: >> What would you be the use case of interned bytes objets? > > Performance and memory. Pickle sizes (my particular issue now). > > - -- > Jes?s Cea Avi?n _/_/ _/_/_/ _/_/_/ > jcea at jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ > Twitter: @jcea _/_/ _/_/ _/_/_/_/_/ > jabber / xmpp:jcea at jabber.org _/_/ _/_/ _/_/ _/_/ _/_/ > "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ > "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ > "El amor es poner tu felicidad en la felicidad de otro" - Leibniz > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.10 (GNU/Linux) > Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ > > iQCVAwUBUjxKPZlgi5GaxT1NAQJ6FwQAim2H+OGpRD75KplahNhnKIfm9AUqVnHg > CaLakWhADdHBYlit+DxRQsxtv5C7gyhhqMryydyvx97z33VaI2p1RGOOcK/lWdNw > ETcetqJo8UswS2PSthJ0e5snOUsIeVJRomhJ48n8sJfIadCxAk6ozdMR75pHP5Y3 > lJoUuUgdcJU= > =vbuK > -----END PGP SIGNATURE----- > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/benjamin%40python.org -- Regards, Benjamin From jcea at jcea.es Fri Sep 20 15:36:45 2013 From: jcea at jcea.es (Jesus Cea) Date: Fri, 20 Sep 2013 15:36:45 +0200 Subject: [Python-Dev] sys.intern should work on bytes In-Reply-To: <20130920153112.5af58729@pitrou.net> References: <523C2F3C.7020605@jcea.es> <CAMpsgwammTPe_2UAM+AdqxmXP8c6hnqPiOXwMUKLvyTjFnQL6g@mail.gmail.com> <523C4A3D.90901@jcea.es> <20130920153112.5af58729@pitrou.net> Message-ID: <523C4F6D.8060602@jcea.es> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 20/09/13 15:31, Antoine Pitrou wrote: > sys.intern is an internal interpreter optimization and should be > orthogonal to pickling. If pickle can't detect already-seen bytes > object, then you may file an improvement request on the bug > tracker. Understood. Thanks for the clarification. Pickle manage correctly "same object" references, but not "different objects but equivalent". That is the issue. But for most uses this is not a problem, and implementing this redundance removal looks like a performance cost that few users would benefice from, but everybody pays. Pickle is already slow enough now :). - -- Jes?s Cea Avi?n _/_/ _/_/_/ _/_/_/ jcea at jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ Twitter: @jcea _/_/ _/_/ _/_/_/_/_/ jabber / xmpp:jcea at jabber.org _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQCVAwUBUjxPbZlgi5GaxT1NAQIyYQQAplpwDnz2/0bNTF7KN7V0PQnXZQknEnvL 0VACm298Y386hs8bJQFlRlTOzfhguulaaEwdqLaBPkXMKA7LBaVRHM1v5o+Xb40X 7DwSKglkbt6HgX7/nDMX4qk9Kxb2ZEVz+XOozPta2NRoZmaz8y7Xyqc/+4+UTzHH XsNJ2H5yVEo= =GPtw -----END PGP SIGNATURE----- From solipsis at pitrou.net Fri Sep 20 15:42:17 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 20 Sep 2013 15:42:17 +0200 Subject: [Python-Dev] sys.intern should work on bytes References: <523C2F3C.7020605@jcea.es> <CAMpsgwammTPe_2UAM+AdqxmXP8c6hnqPiOXwMUKLvyTjFnQL6g@mail.gmail.com> <523C4A3D.90901@jcea.es> <20130920153112.5af58729@pitrou.net> <523C4F6D.8060602@jcea.es> Message-ID: <20130920154217.7f564312@pitrou.net> Le Fri, 20 Sep 2013 15:36:45 +0200, Jesus Cea <jcea at jcea.es> a ?crit : > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > On 20/09/13 15:31, Antoine Pitrou wrote: > > sys.intern is an internal interpreter optimization and should be > > orthogonal to pickling. If pickle can't detect already-seen bytes > > object, then you may file an improvement request on the bug > > tracker. > > Understood. Thanks for the clarification. > > Pickle manage correctly "same object" references, but not "different > objects but equivalent". That is the issue. Ah, well, in that case the issue is not in pickle, it's in your code. pickle doesn't try to guess if "equal" is really functionally equivalent to "identical". Regards Antoine. From solipsis at pitrou.net Fri Sep 20 15:44:15 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 20 Sep 2013 15:44:15 +0200 Subject: [Python-Dev] sys.intern should work on bytes References: <523C2F3C.7020605@jcea.es> <20130920141551.0ac2c769@pitrou.net> <523C4E91.4050905@jcea.es> Message-ID: <20130920154415.5e6cfa09@pitrou.net> Le Fri, 20 Sep 2013 15:33:05 +0200, Jesus Cea <jcea at jcea.es> a ?crit : > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > On 20/09/13 14:15, Antoine Pitrou wrote: > > From http://docs.python.org/3.3/library/sys.html#sys.intern > > > > """sys.intern(string) > > > > Enter string in the table of ?interned? strings and return the > > interned string [...]""" > > > > > > In Python 3 context, "string" means "str". > > I read that, Antoine. In fact I read the manual, I thought it was a > mistake carried over from 2.x documentation, I tried it just in case > before reporting the "documentation mistake", and I was surprised it > was actually true :-). > > I know that intern is used for performance reasons internally to the > interpreter. But I am thinking about memory usage optimizations. For > instance, I have a pickle that is 14MB in size, when "interning" the > strings on it (there are a lot of redundancy), the new size is only > 3MB and it loads faster. I can do it because most data in the pickle > are strings, I could NOT do it if I used bytes. > > I could do a manual "intern" for hashable objects by hand using an > "object:object" dictionary (that would work for integers too), but I > wonder if extending builtin "sys.intern" would be something to > consider. > > Anyway, this pattern is easy enough: > > Instead of > > object = sys.intern(object) > > I could do > > interned = dict() > ... > object = interned.setdefault(object, object) Yes. The main difference is that sys.intern() will remove the interned strings when every external reference vanishes. It requires either weakref'ability (which both str and bytes lack) or special cooperation from the object destructor (which is why sys.intern() is restricted to str instead of working with arbitrary objects). Regards Antoine. From jcea at jcea.es Fri Sep 20 15:46:41 2013 From: jcea at jcea.es (Jesus Cea) Date: Fri, 20 Sep 2013 15:46:41 +0200 Subject: [Python-Dev] dict.setdefault(object, object) instead of "sys.intern()" (was Re: sys.intern should work on bytes) In-Reply-To: <CAPZV6o_z6gyiGSEwP2i7uDagjxdn8i1p-cUWEHN-JAUXR=+zFw@mail.gmail.com> References: <523C2F3C.7020605@jcea.es> <CAMpsgwammTPe_2UAM+AdqxmXP8c6hnqPiOXwMUKLvyTjFnQL6g@mail.gmail.com> <523C4A3D.90901@jcea.es> <CAPZV6o_z6gyiGSEwP2i7uDagjxdn8i1p-cUWEHN-JAUXR=+zFw@mail.gmail.com> Message-ID: <523C51C1.9090901@jcea.es> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 20/09/13 15:33, Benjamin Peterson wrote: > Well, the pickler should memoize bytes objects if you have lots of > the same one in a pickle... Only if they are the very same object. Not diferent bytes objects with the same value. Pickle doesn't do "a==b" but "id(a)==id(b)". Yes, I know that "a==b" would break mutable objects. It is just an example. I don't want to pursue that path. Performance of pickle is already appallingly slow. In my project, I will do the redundancy removal on my own way, as explained in ither message on this thread. Example: * Original pickle: 14416284 bytes * Pickle with "interned" strings: 3004880 bytes (quite an improvement, but this is particular to my case, I have a lot of string duplications here. The pickle also loads a bit faster) * Pickle including an extra dictionary of "interned" strings, created using the "interned.setdefault(object,object)" pattern: 5126587 bytes. Sniff. Could I do this more compactly?. - -- Jes?s Cea Avi?n _/_/ _/_/_/ _/_/_/ jcea at jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ Twitter: @jcea _/_/ _/_/ _/_/_/_/_/ jabber / xmpp:jcea at jabber.org _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQCVAwUBUjxRwZlgi5GaxT1NAQKW8wP/dhVa/v3RZbOKvOtogpHGs5nZyjhtChwn lFK1Lr1wl/+6IgCjgu9axkrRM0LLRaBN91HW+e9AkAM9XSFBQp6qAAqjJpI/jLDp xRLW9fMRHpD21m1tG9zxziz4ACCLNNDnlsyY9l7oHHbMzaAX6Gbigyml3hEbj0uK G5hk4VhyKEY= =m/3T -----END PGP SIGNATURE----- From jcea at jcea.es Fri Sep 20 15:54:48 2013 From: jcea at jcea.es (Jesus Cea) Date: Fri, 20 Sep 2013 15:54:48 +0200 Subject: [Python-Dev] sys.intern should work on bytes In-Reply-To: <20130920154415.5e6cfa09@pitrou.net> References: <523C2F3C.7020605@jcea.es> <20130920141551.0ac2c769@pitrou.net> <523C4E91.4050905@jcea.es> <20130920154415.5e6cfa09@pitrou.net> Message-ID: <523C53A8.4070106@jcea.es> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 20/09/13 15:44, Antoine Pitrou wrote: > Yes. The main difference is that sys.intern() will remove the > interned strings when every external reference vanishes. It > requires either weakref'ability (which both str and bytes lack) or > special cooperation from the object destructor (which is why > sys.intern() is restricted to str instead of working with arbitrary > objects). Great comment. Thanks. Why str/bytes doesn't support weakrefs, beside memory use? - -- Jes?s Cea Avi?n _/_/ _/_/_/ _/_/_/ jcea at jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ Twitter: @jcea _/_/ _/_/ _/_/_/_/_/ jabber / xmpp:jcea at jabber.org _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQCVAwUBUjxTqJlgi5GaxT1NAQIbvwP/fs7e5MJwF0pCa0NObTx0xN8CFQIX9/jt VkQ1Q7lPLSuRlZMC2B+0xfp9QoAsD6N3xqSXwG+T9uf7w6nZ9y3keI06kAdSn/Cz D8EyxoeuNk2aGq0VIzMO260mgs9Gr+3DtWAROcWCMG+8sr5uekJ/LAhI04/xMkqZ zr7aOy1xDgk= =hzGP -----END PGP SIGNATURE----- From benjamin at python.org Fri Sep 20 15:57:26 2013 From: benjamin at python.org (Benjamin Peterson) Date: Fri, 20 Sep 2013 09:57:26 -0400 Subject: [Python-Dev] sys.intern should work on bytes In-Reply-To: <523C53A8.4070106@jcea.es> References: <523C2F3C.7020605@jcea.es> <20130920141551.0ac2c769@pitrou.net> <523C4E91.4050905@jcea.es> <20130920154415.5e6cfa09@pitrou.net> <523C53A8.4070106@jcea.es> Message-ID: <CAPZV6o_z3svzeznqB0KK8sk+pzXh8o121cKBPjTEA04DSVJ0MQ@mail.gmail.com> 2013/9/20 Jesus Cea <jcea at jcea.es>: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > On 20/09/13 15:44, Antoine Pitrou wrote: > >> Yes. The main difference is that sys.intern() will remove the >> interned strings when every external reference vanishes. It >> requires either weakref'ability (which both str and bytes lack) or >> special cooperation from the object destructor (which is why >> sys.intern() is restricted to str instead of working with arbitrary >> objects). > > Great comment. Thanks. > > Why str/bytes doesn't support weakrefs, beside memory use? Is increased memory use for every str/bytes object not a good enough reason? -- Regards, Benjamin From status at bugs.python.org Fri Sep 20 18:07:41 2013 From: status at bugs.python.org (Python tracker) Date: Fri, 20 Sep 2013 18:07:41 +0200 (CEST) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20130920160741.ECE98568E3@psf.upfronthosting.co.za> ACTIVITY SUMMARY (2013-09-13 - 2013-09-20) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 4228 (+16) closed 26602 (+31) total 30830 (+47) Open issues with patches: 1949 Issues opened (37) ================== #12085: subprocess.Popen.__del__ raises AttributeError if __init__ was http://bugs.python.org/issue12085 reopened by serhiy.storchaka #14927: add "Do not supply 'int' argument" to random.shuffle docstring http://bugs.python.org/issue14927 reopened by orsenthil #19011: Enum should have a __getattr__ that makes all the instances av http://bugs.python.org/issue19011 opened by lambacck #19012: urllib2: bad proxy configuration throws "getaddrinfo" error http://bugs.python.org/issue19012 opened by SjoerdOptLand #19014: Allow memoryview.cast() for empty views http://bugs.python.org/issue19014 opened by serhiy.storchaka #19016: autospecced namedtuples should be truthy by default http://bugs.python.org/issue19016 opened by skrisman #19017: selectors: towards uniform EBADF handling http://bugs.python.org/issue19017 opened by neologix #19019: Investigate using Apple clang for building OS X installers http://bugs.python.org/issue19019 opened by ned.deily #19020: Regression: Windows-tkinter-idle, unicode, and 0xxx filename http://bugs.python.org/issue19020 opened by terry.reedy #19021: AttributeError in Popen.__del__ http://bugs.python.org/issue19021 opened by serhiy.storchaka #19022: Improve handling of type.__abstractmethods__ descriptor http://bugs.python.org/issue19022 opened by ncoghlan #19023: ctypes docs: Unimplemented and undocumented features http://bugs.python.org/issue19023 opened by syeberman #19024: Document asterisk (*), splat or star operator http://bugs.python.org/issue19024 opened by techtonik #19025: Deleting attribute of Enum gives misleading error message http://bugs.python.org/issue19025 opened by vajrasky #19027: undefined symbol: _PyParser_Grammar http://bugs.python.org/issue19027 opened by arigo #19028: tkinter.tkapp.merge() fails on non-strings http://bugs.python.org/issue19028 opened by serhiy.storchaka #19030: inspect.getmembers and inspect.classify_class_attrs mishandle http://bugs.python.org/issue19030 opened by ethan.furman #19031: Make help() enum aware http://bugs.python.org/issue19031 opened by ethan.furman #19034: More useful repr for Tcl_Obj http://bugs.python.org/issue19034 opened by serhiy.storchaka #19035: tokenize.generate_tokens treat '\f' symbol as the end of file http://bugs.python.org/issue19035 opened by Alexey.Umnov #19036: setlocale fails due to locale.h being wrapped up in LANGINFO c http://bugs.python.org/issue19036 opened by alanh #19040: Problems with overriding Enum.__new__ http://bugs.python.org/issue19040 opened by Drekin #19042: Idle: add option to autosave 'Untitled' edit window http://bugs.python.org/issue19042 opened by terry.reedy #19043: Remove detailed listing of all versions from LICENSE, Doc/lice http://bugs.python.org/issue19043 opened by georg.brandl #19044: getaddrinfo raises near-useless exception http://bugs.python.org/issue19044 opened by Nikratio #19045: Make on Solaris 11 x64 with OracleStudio12.3 failed http://bugs.python.org/issue19045 opened by podlipnik #19046: SystemError: ..\Objects\weakrefobject.c:903: bad argument to i http://bugs.python.org/issue19046 opened by mherrmann.at #19047: Assorted weakref docs improvements http://bugs.python.org/issue19047 opened by ncoghlan #19048: itertools.tee doesn't have a __sizeof__ method http://bugs.python.org/issue19048 opened by pitrou #19049: itertools.tee uses int for indices http://bugs.python.org/issue19049 opened by pitrou #19050: crash while writing to a closed file descriptor http://bugs.python.org/issue19050 opened by damiro #19051: Unify buffered readers http://bugs.python.org/issue19051 opened by serhiy.storchaka #19053: read1() from zipfile returns empty data http://bugs.python.org/issue19053 opened by serhiy.storchaka #19054: Descriptors howto http://bugs.python.org/issue19054 opened by marco.buttu #19055: Regular expressions: * does not match as many repetitions as p http://bugs.python.org/issue19055 opened by Jason.Stumpf #19056: Windows 7, script exec not working without explicit cal of pyt http://bugs.python.org/issue19056 opened by tsunwell #19057: Sometimes urllib2 raises URLError when trying POST with httpS http://bugs.python.org/issue19057 opened by mrDoctorWho0.. Most recent 15 issues with no replies (15) ========================================== #19056: Windows 7, script exec not working without explicit cal of pyt http://bugs.python.org/issue19056 #19054: Descriptors howto http://bugs.python.org/issue19054 #19040: Problems with overriding Enum.__new__ http://bugs.python.org/issue19040 #19034: More useful repr for Tcl_Obj http://bugs.python.org/issue19034 #19031: Make help() enum aware http://bugs.python.org/issue19031 #19028: tkinter.tkapp.merge() fails on non-strings http://bugs.python.org/issue19028 #19027: undefined symbol: _PyParser_Grammar http://bugs.python.org/issue19027 #19022: Improve handling of type.__abstractmethods__ descriptor http://bugs.python.org/issue19022 #19010: Make XMLPullParser in ElementTree inherit from XMLParser http://bugs.python.org/issue19010 #19005: PyIter_Next crashes if passed a non-iterator http://bugs.python.org/issue19005 #19003: email.generator.BytesGenerator corrupts data by changing line http://bugs.python.org/issue19003 #18982: Add tests for CLI of the calendar module http://bugs.python.org/issue18982 #18979: Use argparse in the uu module http://bugs.python.org/issue18979 #18977: The -t option has no effect in for uu command-line http://bugs.python.org/issue18977 #18976: distutils/command/build_ext passes wrong linker flags http://bugs.python.org/issue18976 Most recent 15 issues waiting for review (15) ============================================= #19054: Descriptors howto http://bugs.python.org/issue19054 #19053: read1() from zipfile returns empty data http://bugs.python.org/issue19053 #19051: Unify buffered readers http://bugs.python.org/issue19051 #19049: itertools.tee uses int for indices http://bugs.python.org/issue19049 #19048: itertools.tee doesn't have a __sizeof__ method http://bugs.python.org/issue19048 #19043: Remove detailed listing of all versions from LICENSE, Doc/lice http://bugs.python.org/issue19043 #19036: setlocale fails due to locale.h being wrapped up in LANGINFO c http://bugs.python.org/issue19036 #19034: More useful repr for Tcl_Obj http://bugs.python.org/issue19034 #19030: inspect.getmembers and inspect.classify_class_attrs mishandle http://bugs.python.org/issue19030 #19028: tkinter.tkapp.merge() fails on non-strings http://bugs.python.org/issue19028 #19025: Deleting attribute of Enum gives misleading error message http://bugs.python.org/issue19025 #19023: ctypes docs: Unimplemented and undocumented features http://bugs.python.org/issue19023 #19021: AttributeError in Popen.__del__ http://bugs.python.org/issue19021 #19020: Regression: Windows-tkinter-idle, unicode, and 0xxx filename http://bugs.python.org/issue19020 #19016: autospecced namedtuples should be truthy by default http://bugs.python.org/issue19016 Top 10 most discussed issues (10) ================================= #19048: itertools.tee doesn't have a __sizeof__ method http://bugs.python.org/issue19048 28 msgs #18986: Add a case-insensitive case-preserving dict http://bugs.python.org/issue18986 20 msgs #18967: Find a less conflict prone approach to Misc/NEWS http://bugs.python.org/issue18967 18 msgs #11798: Test cases not garbage collected after run http://bugs.python.org/issue11798 15 msgs #19030: inspect.getmembers and inspect.classify_class_attrs mishandle http://bugs.python.org/issue19030 14 msgs #16042: smtplib: unlimited readline() from connection http://bugs.python.org/issue16042 10 msgs #19036: setlocale fails due to locale.h being wrapped up in LANGINFO c http://bugs.python.org/issue19036 10 msgs #19049: itertools.tee uses int for indices http://bugs.python.org/issue19049 8 msgs #14927: add "Do not supply 'int' argument" to random.shuffle docstring http://bugs.python.org/issue14927 7 msgs #18990: Return root element from ElementTree.XMLPullParser.close() to http://bugs.python.org/issue18990 7 msgs Issues closed (30) ================== #13758: compile() should not encode 'filename' (at least on Windows) http://bugs.python.org/issue13758 closed by haypo #14984: netrc module allows read of non-secured .netrc file http://bugs.python.org/issue14984 closed by r.david.murray #16201: socket.gethostbyname incorrectly parses ip http://bugs.python.org/issue16201 closed by neologix #17734: Failure when running test_builtin after test_genexps http://bugs.python.org/issue17734 closed by serhiy.storchaka #17764: Support http.server passing bind address via commend line argu http://bugs.python.org/issue17764 closed by orsenthil #18230: test_builtin fails/hangs when run after test_getopt http://bugs.python.org/issue18230 closed by serhiy.storchaka #18526: Add resource management/guarding to unittest http://bugs.python.org/issue18526 closed by zach.ware #18824: Adding LogRecord attribute "traceback" http://bugs.python.org/issue18824 closed by vinay.sajip #18856: Added test coverage for calendar print functions http://bugs.python.org/issue18856 closed by ezio.melotti #18929: inspect.classify_class_attrs ignores metaclass http://bugs.python.org/issue18929 closed by python-dev #18937: add unittest assertion for logging http://bugs.python.org/issue18937 closed by pitrou #18945: Name collision handling in tempfile is not covered by tests http://bugs.python.org/issue18945 closed by python-dev #18951: In unittest.TestCase.assertRegex change "re" and "regex" to "r http://bugs.python.org/issue18951 closed by ezio.melotti #18955: Confusing documentation in Lib/importlib/util.py http://bugs.python.org/issue18955 closed by brett.cannon #18981: Typo in the ctypes tests http://bugs.python.org/issue18981 closed by ezio.melotti #18989: reuse of enum names in class creation inconsistent http://bugs.python.org/issue18989 closed by python-dev #18995: Enum does not work with reversed http://bugs.python.org/issue18995 closed by ethan.furman #18998: iter() not working in ElementTree http://bugs.python.org/issue18998 closed by eli.bendersky #19013: unittest's own test suite is not CLI-friendly http://bugs.python.org/issue19013 closed by pitrou #19015: Too long command returns 32512 http://bugs.python.org/issue19015 closed by Sworddragon #19018: Heapq.merge suppreses IndexError from user generator http://bugs.python.org/issue19018 closed by rhettinger #19026: OrderedDict should not accept dict as parameter http://bugs.python.org/issue19026 closed by rhettinger #19029: tix.py uses StringType http://bugs.python.org/issue19029 closed by serhiy.storchaka #19032: __reduce_ex__ on lock object http://bugs.python.org/issue19032 closed by cool-RR #19033: Python 3 won't go on PC-BSD 9.1 http://bugs.python.org/issue19033 closed by ned.deily #19037: mailbox module modifies maildir file times after moving from t http://bugs.python.org/issue19037 closed by r.david.murray #19038: Fix sort order in Misc/ACKS. http://bugs.python.org/issue19038 closed by georg.brandl #19039: sphinx search, result sorting http://bugs.python.org/issue19039 closed by georg.brandl #19041: requested (DLL) module could not be found http://bugs.python.org/issue19041 closed by r.david.murray #19052: Python's CGIHTTPServer does not handle Expect: 100-continue gr http://bugs.python.org/issue19052 closed by Joseph.Warren From pje at telecommunity.com Fri Sep 20 18:50:44 2013 From: pje at telecommunity.com (PJ Eby) Date: Fri, 20 Sep 2013 12:50:44 -0400 Subject: [Python-Dev] sys.intern should work on bytes In-Reply-To: <523C53A8.4070106@jcea.es> References: <523C2F3C.7020605@jcea.es> <20130920141551.0ac2c769@pitrou.net> <523C4E91.4050905@jcea.es> <20130920154415.5e6cfa09@pitrou.net> <523C53A8.4070106@jcea.es> Message-ID: <CALeMXf7Qo94xYsP_8cufhp=OR0x4QyWu-gnzn=fq1k7NF2QEuw@mail.gmail.com> On Fri, Sep 20, 2013 at 9:54 AM, Jesus Cea <jcea at jcea.es> wrote: > Why str/bytes doesn't support weakrefs, beside memory use? The typical use case for weakrefs is to break reference cycles, but str and bytes can't *be* part of a reference cycle, so outside of interning-like use cases, there's no need for weakref support there. From anikom15 at gmail.com Sat Sep 21 03:27:16 2013 From: anikom15 at gmail.com (=?iso-8859-1?Q?Westley_Mart=EDnez?=) Date: Fri, 20 Sep 2013 18:27:16 -0700 Subject: [Python-Dev] Use an empty def as a lambda In-Reply-To: <CAA1uNA=tPhX2y0wP8tdNGfxZ1iV8OJthdd9YpXQsbRHCZDzb_g@mail.gmail.com> References: <CAA1uNA=tPhX2y0wP8tdNGfxZ1iV8OJthdd9YpXQsbRHCZDzb_g@mail.gmail.com> Message-ID: <001e01ceb669$b6a1c740$23e555c0$@gmail.com> 'def' is no more ambiguous than 'lambda', and is in fact more ambiguous, for 'def' doesn't lend itself to anything other than the word define, whilst 'lambda' can only mean lambda function. Calling def explicit is silly. It only makes sense because def arbitrarily means a function in Python (I'm proposing def become func or proc in Python 4000). To call lambda too 'computer sciencey' is equally ridiculous, for pro- gramming is a key spawn of computer science. A programmer needs to have some knowledge of computer science to program, just like a physicist needs knowledge of calculus to understand mechanics. > -----Original Message----- > From: Python-Dev [mailto:python-dev-bounces+anikom15=gmail.com at python.org] On > Behalf Of Ben Gift > Sent: Thursday, September 19, 2013 1:54 PM > To: python-dev at python.org > Subject: [Python-Dev] Use an empty def as a lambda > > I think the lambda keyword is difficult to understand for many people. It > would be more pythonic to use an empty def call instead. > > > For instance this: > > > words.sort(key = lambda x: x[2]) > > > could look like this: > > words.sort(key = def (x): x[2]) > > > It's obvious and explicit that we're creating an unnamed, anonymous function > this way. From regebro at gmail.com Sat Sep 21 07:42:19 2013 From: regebro at gmail.com (Lennart Regebro) Date: Sat, 21 Sep 2013 07:42:19 +0200 Subject: [Python-Dev] Use an empty def as a lambda In-Reply-To: <CAA1uNA=tPhX2y0wP8tdNGfxZ1iV8OJthdd9YpXQsbRHCZDzb_g@mail.gmail.com> References: <CAA1uNA=tPhX2y0wP8tdNGfxZ1iV8OJthdd9YpXQsbRHCZDzb_g@mail.gmail.com> Message-ID: <CAL0kPAV8DYq5LwoBtuK8ELZyN6CxLymD_o9UK=fU7uLGpF92Zg@mail.gmail.com> On Thu, Sep 19, 2013 at 10:54 PM, Ben Gift <benhgift at gmail.com> wrote: > I think the lambda keyword is difficult to understand for many people. It > would be more pythonic to use an empty def call instead. I agree, but that ship has sailed, at least until the time when Python 2 is dead. I don't want these kinds of syntax changes for a *looooong* time now, personally. //Lennart From stefan_ml at behnel.de Sat Sep 21 11:15:28 2013 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 21 Sep 2013 11:15:28 +0200 Subject: [Python-Dev] dict.setdefault(object, object) instead of "sys.intern()" (was Re: sys.intern should work on bytes) In-Reply-To: <523C51C1.9090901@jcea.es> References: <523C2F3C.7020605@jcea.es> <CAMpsgwammTPe_2UAM+AdqxmXP8c6hnqPiOXwMUKLvyTjFnQL6g@mail.gmail.com> <523C4A3D.90901@jcea.es> <CAPZV6o_z6gyiGSEwP2i7uDagjxdn8i1p-cUWEHN-JAUXR=+zFw@mail.gmail.com> <523C51C1.9090901@jcea.es> Message-ID: <l1jo38$a9q$1@ger.gmane.org> Jesus Cea, 20.09.2013 15:46: > On 20/09/13 15:33, Benjamin Peterson wrote: >> Well, the pickler should memoize bytes objects if you have lots of >> the same one in a pickle... > > Only if they are the very same object. Not diferent bytes objects with > the same value. Pickle doesn't do "a==b" but "id(a)==id(b)". > > Yes, I know that "a==b" would break mutable objects. It is just an > example. > > I don't want to pursue that path. Performance of pickle is already > appallingly slow. > > In my project, I will do the redundancy removal on my own way, as > explained in ither message on this thread. > > Example: > > * Original pickle: 14416284 bytes > > * Pickle with "interned" strings: 3004880 bytes > (quite an improvement, but this is particular to my case, I have a lot > of string duplications here. The pickle also loads a bit faster) > > * Pickle including an extra dictionary of "interned" strings, created > using the "interned.setdefault(object,object)" pattern: 5126587 bytes. > Sniff. > > Could I do this more compactly?. ISTM that what you are looking for is a compression-like pattern that efficiently encodes repeated literals (i.e. constants of safe types) in the pickle. That could be achieved by extending the pickle protocol to include backreferences to earlier objects, I guess (I'm not all that familiar with the internals of the pickle format). Any of the well known compression algorithms that are capable of handling streaming data would apply here. Assuming you don't want to simply send the pickle output through gzip & friends, that is... It also seems to me that python-dev isn't the right place to discuss this. python-ideas seems more appropriate for now. Stefan From rymg19 at gmail.com Sat Sep 21 19:55:20 2013 From: rymg19 at gmail.com (Ryan) Date: Sat, 21 Sep 2013 12:55:20 -0500 Subject: [Python-Dev] Use an empty def as a lambda In-Reply-To: <001e01ceb669$b6a1c740$23e555c0$@gmail.com> References: <CAA1uNA=tPhX2y0wP8tdNGfxZ1iV8OJthdd9YpXQsbRHCZDzb_g@mail.gmail.com> <001e01ceb669$b6a1c740$23e555c0$@gmail.com> Message-ID: <5e5e1efe-5560-41ea-ba84-e474baa25985@email.android.com> Change def to func? That's the worst idea I've heard yet. Def is already there; why break all existing code just for a word? "Westley Mart?nez" <anikom15 at gmail.com> wrote: >'def' is no more ambiguous than 'lambda', and is in fact more >ambiguous, >for 'def' doesn't lend itself to anything other than the word define, >whilst 'lambda' can only mean lambda function. Calling def explicit is >silly. It only makes sense because def arbitrarily means a function in >Python (I'm proposing def become func or proc in Python 4000). > >To call lambda too 'computer sciencey' is equally ridiculous, for pro- >gramming is a key spawn of computer science. A programmer needs to >have >some knowledge of computer science to program, just like a physicist >needs knowledge of calculus to understand mechanics. > >> -----Original Message----- >> From: Python-Dev >[mailto:python-dev-bounces+anikom15=gmail.com at python.org] On >> Behalf Of Ben Gift >> Sent: Thursday, September 19, 2013 1:54 PM >> To: python-dev at python.org >> Subject: [Python-Dev] Use an empty def as a lambda >> >> I think the lambda keyword is difficult to understand for many >people. It >> would be more pythonic to use an empty def call instead. >> >> >> For instance this: >> >> >> words.sort(key = lambda x: x[2]) >> >> >> could look like this: >> >> words.sort(key = def (x): x[2]) >> >> >> It's obvious and explicit that we're creating an unnamed, anonymous >function >> this way. > > >_______________________________________________ >Python-Dev mailing list >Python-Dev at python.org >https://mail.python.org/mailman/listinfo/python-dev >Unsubscribe: >https://mail.python.org/mailman/options/python-dev/rymg19%40gmail.com -- Sent from my Android phone with K-9 Mail. Please excuse my brevity. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130921/e254e254/attachment.html> From rdmurray at bitdance.com Sat Sep 21 21:00:39 2013 From: rdmurray at bitdance.com (R. David Murray) Date: Sat, 21 Sep 2013 15:00:39 -0400 Subject: [Python-Dev] Use an empty def as a lambda In-Reply-To: <5e5e1efe-5560-41ea-ba84-e474baa25985@email.android.com> References: <CAA1uNA=tPhX2y0wP8tdNGfxZ1iV8OJthdd9YpXQsbRHCZDzb_g@mail.gmail.com> <001e01ceb669$b6a1c740$23e555c0$@gmail.com> <5e5e1efe-5560-41ea-ba84-e474baa25985@email.android.com> Message-ID: <20130921190040.6B5BB250955@webabinitio.net> On Sat, 21 Sep 2013 12:55:20 -0500, Ryan <rymg19 at gmail.com> wrote: > "Westley Mart??nez" <anikom15 at gmail.com> wrote: > >'def' is no more ambiguous than 'lambda', and is in fact more > >ambiguous, > >for 'def' doesn't lend itself to anything other than the word define, > >whilst 'lambda' can only mean lambda function. Calling def explicit is > >silly. It only makes sense because def arbitrarily means a function in > >Python (I'm proposing def become func or proc in Python 4000). > > Change def to func? That's the worst idea I've heard yet. Def is > already there; why break all existing code just for a word? Note that whatever happens with Python4, breaking of backward compatibility (by anything other than the deletion of cruft) is very unlikely to be part of it. --David From anikom15 at gmail.com Sat Sep 21 21:22:14 2013 From: anikom15 at gmail.com (=?UTF-8?Q?Westley_Mart=C3=ADnez?=) Date: Sat, 21 Sep 2013 12:22:14 -0700 Subject: [Python-Dev] Use an empty def as a lambda In-Reply-To: <5e5e1efe-5560-41ea-ba84-e474baa25985@email.android.com> References: <CAA1uNA=tPhX2y0wP8tdNGfxZ1iV8OJthdd9YpXQsbRHCZDzb_g@mail.gmail.com> <001e01ceb669$b6a1c740$23e555c0$@gmail.com> <5e5e1efe-5560-41ea-ba84-e474baa25985@email.android.com> Message-ID: <000f01ceb6ff$e1e9e180$a5bda480$@gmail.com> When I say Python 4000, I don't actually mean Python 4.x, I mean the next backwards-incompatible revision to Python (if it comes to pass). My reasoning is that we use class to make classes, lambda to make lambda functions, and def to make--well not defs--functions, which doesn't really make sense to me. Changing lambda to def would break compatibility as well and restrict the grammar further. In addition, it provides little foreseeable benefit since the term def has no real meaning outside of Python, whilst lambda has a clearly defined meaning. Had the original C pro- grammers chosen def in lieu of typedef or #def in lieu of #define, I doubt that def would have been chosen as the keyword for functions in Python. > -----Original Message----- > From: Ryan [mailto:rymg19 at gmail.com] > Sent: Saturday, September 21, 2013 10:55 AM > To: Westley Mart?nez; 'Ben Gift'; python-dev at python.org > Subject: Re: [Python-Dev] Use an empty def as a lambda > > Change def to func? That's the worst idea I've heard yet. Def is already > there; why break all existing code just for a word? > > > "Westley Mart?nez" <anikom15 at gmail.com> wrote: > > 'def' is no more ambiguous than 'lambda', and is in fact more ambiguous, > for 'def' doesn't lend itself to anything other than the word define, > whilst 'lambda' can only mean lambda function. Calling def explicit is > silly. It only makes sense because def arbitrarily means a function in > Python (I'm proposing def become func or proc in Python 4000). > > To call lambda too 'computer sciencey' is equally ridiculous, for pro- > gramming is a key spawn of computer science. A programmer needs to have > some knowledge of computer science to program, just like a physicist > needs knowledge of calculus to understand mechanics. > > > -----Original Message----- > From: Python-Dev [mailto:python-dev- > bounces+anikom15=gmail.com at python.org] On > Behalf Of Ben Gift > Sent: Thursday, September 19, 2013 1:54 PM > To: python-dev at python.org > Subject: [Python-Dev] Use an empty def as a lambda > > I think the lambda keyword is difficult to understand for many > people. It > would be more pythonic to use an empty def call instead. > > > For instance this: > > > words.sort(key = lambda x: x[2]) > > > could look like this: > > words.sort(key = def (x): x[2]) > > > It's obvious and explicit that we're creating an unnamed, > anonymous function > this way. > > > > ________________________________ > > > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python- > dev/rymg19%40gmail.com > > > -- > Sent from my Android phone with K-9 Mail. Please excuse my brevity. From ben+python at benfinney.id.au Sat Sep 21 21:56:02 2013 From: ben+python at benfinney.id.au (Ben Finney) Date: Sun, 22 Sep 2013 05:56:02 +1000 Subject: [Python-Dev] Use an empty def as a lambda References: <CAA1uNA=tPhX2y0wP8tdNGfxZ1iV8OJthdd9YpXQsbRHCZDzb_g@mail.gmail.com> <001e01ceb669$b6a1c740$23e555c0$@gmail.com> <5e5e1efe-5560-41ea-ba84-e474baa25985@email.android.com> <000f01ceb6ff$e1e9e180$a5bda480$@gmail.com> Message-ID: <7wk3iauey5.fsf@benfinney.id.au> Westley Mart?nez <anikom15 at gmail.com> writes: > My reasoning is that we use class to make classes, lambda to make > lambda functions, and def to make--well not defs--functions, which > doesn't really make sense to me. Your reasoning is flawed. There is no such thing in Python as a ?lambda function?. Python has functions. It doesn't matter whether you use a ?lambda? or ?def? statement to create it, there's no resulting difference in the type of the object. It is a function. So: you make a class with a ?class? statement; you make a function using either a ?def? statement or a ?lambda? expression. There is no third type of object being discussed here. -- \ ?Faith may be defined briefly as an illogical belief in the | `\ occurrence of the improbable.? ?Henry L. Mencken | _o__) | Ben Finney From tjreedy at udel.edu Sat Sep 21 23:16:41 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 21 Sep 2013 17:16:41 -0400 Subject: [Python-Dev] Revert #12085 fix for __del__ attribute error message Message-ID: <l1l2bm$1bp$1@ger.gmane.org> When an AttributeError is raised in a __del__ method, it is caught and ignored, except that it is not completely ignored but is replaced by a warning message sent to stderr. Example: >>> class C(): def __del__(self): raise AttributeError >>> c=C() >>> del c Exception AttributeError: AttributeError() in <bound method C.__del__ of <__main__.C object at 0x000000000351A198>> ignored The AttributeError is ignored, I presume, because one may be expected if __new__ runs but __init__ does not, because of an exception. AttributeErrors can also be expected during exit cleanup. The message seems not to be a Warning. So it seems that is cannot be suppressed. The OP of http://bugs.python.org/issue12085 encountered this message with an unnecessary use of subprocesses.Popen. (There is better code that avoids the problem.) He considered the message a problem as it interfered with his output. He suggested the fix of changing 'self.attr' to "getattr(self. 'attr', False)". Viktor Stinner applied this fix. I believe it should be reverted for three reasons: 1. It results in TypeError on exit when gettattr is set to None before __delete__ is called. http://bugs.python.org/issue19021. Even is that is fixed with an alternate patch (creating a 'backup' class attribute), I think it would still be wrong. 2. If we do not agree with the OP that putting output on stderr is bad, then no fix is needed and there should be none. 3. If we do agree that the unconditional output is a problem, then there should be a general fix. It is, in a sense, just happenstance that the OP encountered this with Popen. Some possibilities: a. Decide the the message is not needed; just silently ignore the __del__ AttributeError. I am not sure what its intended use is. b. Make the message a Warning that can be blocked, perhaps a RuntimeWarning. Does this warning message predate the warnings module, new in 2.1? c. If the __del__ AttributeError occurs while processing the __init__ error, just chain it normally. But this would, I presume interfere with catching the __init__ exception. d. Add a missing attribute in cleanup message to end of the __init__ exception message. -- Terry Jan Reedy From rdmurray at bitdance.com Sun Sep 22 00:15:24 2013 From: rdmurray at bitdance.com (R. David Murray) Date: Sat, 21 Sep 2013 18:15:24 -0400 Subject: [Python-Dev] Revert #12085 fix for __del__ attribute error message In-Reply-To: <l1l2bm$1bp$1@ger.gmane.org> References: <l1l2bm$1bp$1@ger.gmane.org> Message-ID: <20130921221524.55D4225007F@webabinitio.net> On Sat, 21 Sep 2013 17:16:41 -0400, Terry Reedy <tjreedy at udel.edu> wrote: > When an AttributeError is raised in a __del__ method, it is caught and > ignored, except that it is not completely ignored but is replaced by a > warning message sent to stderr. Example: > >>> class C(): > def __del__(self): raise AttributeError > > >>> c=C() > >>> del c > Exception AttributeError: AttributeError() in <bound method C.__del__ of > <__main__.C object at 0x000000000351A198>> ignored This is a replacement for a traceback. In later Python versions, the full traceback is printed. In the general case it represents a bug in the code that should be fixed. Most such errors arise from the vagaries of module finalization (such as your issue 19021), but not all of them do: the rest represent real bugs in __del__ methods (which are executed asynchronously in the general case). So the question is, is the bug in the user code, or the stdlib code? >From the issue, it sounds like it could be considered either (or most likely, both). --David From tjreedy at udel.edu Sun Sep 22 01:55:05 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 21 Sep 2013 19:55:05 -0400 Subject: [Python-Dev] Revert #12085 fix for __del__ attribute error message In-Reply-To: <20130921221524.55D4225007F@webabinitio.net> References: <l1l2bm$1bp$1@ger.gmane.org> <20130921221524.55D4225007F@webabinitio.net> Message-ID: <l1lbkl$k0g$1@ger.gmane.org> On 9/21/2013 6:15 PM, R. David Murray wrote: > On Sat, 21 Sep 2013 17:16:41 -0400, Terry Reedy <tjreedy at udel.edu> wrote: >> When an AttributeError is raised in a __del__ method, it is caught and >> ignored, except that it is not completely ignored but is replaced by a >> warning message sent to stderr. Example: >> >>> class C(): >> def __del__(self): raise AttributeError >> >> >>> c=C() >> >>> del c >> Exception AttributeError: AttributeError() in <bound method C.__del__ of >> <__main__.C object at 0x000000000351A198>> ignored > > This is a replacement for a traceback. In later Python versions, the > full traceback is printed. The above is 3.3.2. In 3.4.0a2, the traceback of the ignored exception is indeed printed. Exception ignored in: <bound method C.__del__ of <__main__.C object at 0x00000000039946D8>> Traceback (most recent call last): File "<pyshell#0>", line 2, in __del__ AttributeError: > In the general case it represents a bug in > the code that should be fixed. Most such errors arise from the vagaries > of module finalization (such as your issue 19021), Lets call that a buglet ;-). Not really harmful, but annoying. Accepting that even such buglets 'should' be fixed in the stdllib, so that the message does not appear, is there any reason *not* to make it a RuntimeWarning so that users who care about clean output can filter it out while waiting for us to fix it? This would be a separate issue from #12085. > but not all of them > do: the rest represent real bugs in __del__ methods (which are executed > asynchronously in the general case). Which is why the message should be printed, so the developer can decide. -- Terry Jan Reedy From anikom15 at gmail.com Sun Sep 22 03:49:39 2013 From: anikom15 at gmail.com (=?UTF-8?Q?Westley_Mart=C3=ADnez?=) Date: Sat, 21 Sep 2013 18:49:39 -0700 Subject: [Python-Dev] Use an empty def as a lambda In-Reply-To: <7wk3iauey5.fsf@benfinney.id.au> References: <CAA1uNA=tPhX2y0wP8tdNGfxZ1iV8OJthdd9YpXQsbRHCZDzb_g@mail.gmail.com> <001e01ceb669$b6a1c740$23e555c0$@gmail.com> <5e5e1efe-5560-41ea-ba84-e474baa25985@email.android.com> <000f01ceb6ff$e1e9e180$a5bda480$@gmail.com> <7wk3iauey5.fsf@benfinney.id.au> Message-ID: <004701ceb736$014a0bf0$03de23d0$@gmail.com> > -----Original Message----- > From: Python-Dev [mailto:python-dev-bounces+anikom15=gmail.com at python.org] On > Behalf Of Ben Finney > Sent: Saturday, September 21, 2013 12:56 PM > To: python-dev at python.org > Subject: Re: [Python-Dev] Use an empty def as a lambda > > Westley Mart?nez <anikom15 at gmail.com> writes: > > > My reasoning is that we use class to make classes, lambda to make > > lambda functions, and def to make--well not defs--functions, which > > doesn't really make sense to me. > > Your reasoning is flawed. There is no such thing in Python as a ?lambda > function?. > > Python has functions. It doesn't matter whether you use a ?lambda? or > ?def? statement to create it, there's no resulting difference in the > type of the object. It is a function. > > So: you make a class with a ?class? statement; you make a function using > either a ?def? statement or a ?lambda? expression. There is no third > type of object being discussed here. This is true. The final object the Python creates is, to my knowledge, the same regardless of whether it uses define or lambda. But it's irrelevant if it's the same or not, since one could argue that every- thing is going to end up as a series of 0s and 1s. In that sense a function is nothing more than a glorified integer. Lambda function is just a term, like method or procedure instead function, although it is certainly more specific in the sense that it is anonymous. In my list I was citing general concepts, not specific Python objects. class is obvious, lambda is obvious, def is not as obvious, though certainly more obvious than C-style declarations. If lambda should change I think anon or expr would be more suitable, or even func (so long as def not become func). From guido at python.org Sun Sep 22 04:30:19 2013 From: guido at python.org (Guido van Rossum) Date: Sat, 21 Sep 2013 19:30:19 -0700 Subject: [Python-Dev] Revert #12085 fix for __del__ attribute error message In-Reply-To: <l1lbkl$k0g$1@ger.gmane.org> References: <l1l2bm$1bp$1@ger.gmane.org> <20130921221524.55D4225007F@webabinitio.net> <l1lbkl$k0g$1@ger.gmane.org> Message-ID: <CAP7+vJLHO12ZNwX4UZ0c_BF9pr+xxxLZfy2dGjBcMVenGow9vQ@mail.gmail.com> Exceptions in __del__ point to bugs (sometimes in the stdlib) that should be fixed, period. The only reason they do not result in exceptions that are properly bubbled up and catchable is because __del__ is called from a DECREF macro which has no return value. Also, IMO writing to stderr is fair game -- reporting errors is what it is for. As to making them warnings, I don't know that the warnings machinery is easily adaptable for this purpose. Warnings can be suppressed but they can also be turned into full exceptions; the latter doesn't really apply here (see previous paragraph). But I would not object if someone found a way to do this, though I'd prefer the default behavior to remain what it is in 3.4 (print a full traceback). On Sat, Sep 21, 2013 at 4:55 PM, Terry Reedy <tjreedy at udel.edu> wrote: > On 9/21/2013 6:15 PM, R. David Murray wrote: > >> On Sat, 21 Sep 2013 17:16:41 -0400, Terry Reedy <tjreedy at udel.edu> wrote: >> >>> When an AttributeError is raised in a __del__ method, it is caught and >>> ignored, except that it is not completely ignored but is replaced by a >>> warning message sent to stderr. Example: >>> >>> class C(): >>> def __del__(self): raise AttributeError >>> >>> >>> c=C() >>> >>> del c >>> Exception AttributeError: AttributeError() in <bound method C.__del__ of >>> <__main__.C object at 0x000000000351A198>> ignored >>> >> >> This is a replacement for a traceback. In later Python versions, the >> full traceback is printed. >> > > The above is 3.3.2. In 3.4.0a2, the traceback of the ignored exception is > indeed printed. > > Exception ignored in: <bound method C.__del__ of <__main__.C object at > 0x00000000039946D8>> > Traceback (most recent call last): > File "<pyshell#0>", line 2, in __del__ > AttributeError: > > > In the general case it represents a bug in >> the code that should be fixed. Most such errors arise from the vagaries >> of module finalization (such as your issue 19021), >> > > Lets call that a buglet ;-). Not really harmful, but annoying. Accepting > that even such buglets 'should' be fixed in the stdllib, so that the > message does not appear, is there any reason *not* to make it a > RuntimeWarning so that users who care about clean output can filter it out > while waiting for us to fix it? > > This would be a separate issue from #12085. > > > > but not all of them > >> do: the rest represent real bugs in __del__ methods (which are executed >> asynchronously in the general case). >> > > Which is why the message should be printed, so the developer can decide. > > -- > Terry Jan Reedy > > > ______________________________**_________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/**mailman/listinfo/python-dev<https://mail.python.org/mailman/listinfo/python-dev> > Unsubscribe: https://mail.python.org/**mailman/options/python-dev/** > guido%40python.org<https://mail.python.org/mailman/options/python-dev/guido%40python.org> > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130921/7e693193/attachment-0001.html> From steve at pearwood.info Sun Sep 22 04:49:16 2013 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 22 Sep 2013 12:49:16 +1000 Subject: [Python-Dev] Use an empty def as a lambda In-Reply-To: <004701ceb736$014a0bf0$03de23d0$@gmail.com> References: <CAA1uNA=tPhX2y0wP8tdNGfxZ1iV8OJthdd9YpXQsbRHCZDzb_g@mail.gmail.com> <001e01ceb669$b6a1c740$23e555c0$@gmail.com> <5e5e1efe-5560-41ea-ba84-e474baa25985@email.android.com> <000f01ceb6ff$e1e9e180$a5bda480$@gmail.com> <7wk3iauey5.fsf@benfinney.id.au> <004701ceb736$014a0bf0$03de23d0$@gmail.com> Message-ID: <20130922024916.GC19939@ando> Guys, this thread is not Python-Dev territory. It should have gone to Python-Ideas. I repeat what I posted two days ago: Proposals for changes to syntax and functionality are normally expected to gather feedback on python-ideas before coming to python-dev for final approval or rejection. https://mail.python.org/mailman/listinfo/python-ideas http://docs.python.org/devguide/communication.html Please drop the discussion here. -- Steven From steve at pearwood.info Sun Sep 22 05:13:04 2013 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 22 Sep 2013 13:13:04 +1000 Subject: [Python-Dev] Best practice for documentation for std lib Message-ID: <20130922031304.GE19939@ando> Hi all, I have a question about how I should manage documentation for the statistics module for Python 3.4. At the moment, I have extensive docstrings in the module itself. I don't believe anyone has flagged that as "too much information" in a code review, so I'm going to assume that large docstrings will be acceptable. However, I have been asked to ensure that there is a separate statistics.rst file for documentation. I don't want to have to maintain the documentation in two places, both in the .py module and in .rst file. Can anyone give me some pointers as to best practice in this situation? Is there a "How To Write Docs For The Standard Library" document somewhere? Perhaps I missed it, but my searches found nothing useful. I have read this: http://docs.python.org/devguide/documenting.html but it didn't shed any light on my situation. Thanks in advance, -- Steve From benjamin at python.org Sun Sep 22 05:25:10 2013 From: benjamin at python.org (Benjamin Peterson) Date: Sat, 21 Sep 2013 23:25:10 -0400 Subject: [Python-Dev] Best practice for documentation for std lib In-Reply-To: <20130922031304.GE19939@ando> References: <20130922031304.GE19939@ando> Message-ID: <CAPZV6o8HnVfc-wFdPjHZtbh_s4X7MU-Kqi0QewUR2ar_n3Gogg@mail.gmail.com> There's not really much to do but maintain them separately. Truncate the docstrings if it makes life easier. 2013/9/21 Steven D'Aprano <steve at pearwood.info>: > Hi all, > > I have a question about how I should manage documentation for the > statistics module for Python 3.4. At the moment, I have extensive > docstrings in the module itself. I don't believe anyone has flagged that > as "too much information" in a code review, so I'm going to assume that > large docstrings will be acceptable. > > However, I have been asked to ensure that there is a separate > statistics.rst file for documentation. > > I don't want to have to maintain the documentation in two places, both > in the .py module and in .rst file. Can anyone give me some pointers as > to best practice in this situation? Is there a "How To Write Docs For > The Standard Library" document somewhere? Perhaps I missed it, but my > searches found nothing useful. I have read this: > > http://docs.python.org/devguide/documenting.html > > but it didn't shed any light on my situation. > > > Thanks in advance, > > > > -- > Steve > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/benjamin%40python.org -- Regards, Benjamin From victor.stinner at gmail.com Sun Sep 22 09:58:42 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Sun, 22 Sep 2013 09:58:42 +0200 Subject: [Python-Dev] Best practice for documentation for std lib In-Reply-To: <20130922031304.GE19939@ando> References: <20130922031304.GE19939@ando> Message-ID: <CAMpsgwYaGPf=rw=istdUnmMUEDcvANzsON3i5G3z9vpqScopeA@mail.gmail.com> I don't like having to browse the source code to read the documentation. I prefer short docstring and longer reST documentation. In ReST, you get a table of content and nice links to functions, modules, etc. When I read doc, I like having two levels: tutorial listing most important functions and parameters with examples, and a full API doc. Using reST, you can also use tables, notes, footnotes, etc. Victor -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130922/3017cf02/attachment.html> From solipsis at pitrou.net Sun Sep 22 10:19:36 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 22 Sep 2013 10:19:36 +0200 Subject: [Python-Dev] Revert #12085 fix for __del__ attribute error message References: <l1l2bm$1bp$1@ger.gmane.org> <20130921221524.55D4225007F@webabinitio.net> <l1lbkl$k0g$1@ger.gmane.org> Message-ID: <20130922101936.53b9e68b@fsol> On Sat, 21 Sep 2013 19:55:05 -0400 Terry Reedy <tjreedy at udel.edu> wrote: > > > In the general case it represents a bug in > > the code that should be fixed. Most such errors arise from the vagaries > > of module finalization (such as your issue 19021), > > Lets call that a buglet ;-). Not really harmful, but annoying. It's not a buglet. A buggy __del__ method is as much a bug as any other buggy method. __del__ issues can lead to resources not being properly finalized and released to the system. > Accepting > that even such buglets 'should' be fixed in the stdllib, so that the > message does not appear, is there any reason *not* to make it a > RuntimeWarning so that users who care about clean output can filter it > out while waiting for us to fix it? Yes, the reason is that these are real exceptions, not warnings. The reason the exceptions are not propagated is that they can occur at any point (finalization is pretty much asynchronous, it can occur from unrelated places), so propagating them to whatever code happens to execute at the time would be a huge instability factor. Making them warnings means they could be silenced depending on the current warning settings. Regards Antoine. From solipsis at pitrou.net Sun Sep 22 10:20:46 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 22 Sep 2013 10:20:46 +0200 Subject: [Python-Dev] Best practice for documentation for std lib References: <20130922031304.GE19939@ando> Message-ID: <20130922102046.764abea5@fsol> On Sun, 22 Sep 2013 13:13:04 +1000 Steven D'Aprano <steve at pearwood.info> wrote: > Hi all, > > I have a question about how I should manage documentation for the > statistics module for Python 3.4. At the moment, I have extensive > docstrings in the module itself. I don't believe anyone has flagged that > as "too much information" in a code review, so I'm going to assume that > large docstrings will be acceptable. Related question: do the extensive docstrings make "help(stats)" painful to browse through? Regards Antoine. From g.brandl at gmx.net Sun Sep 22 10:48:33 2013 From: g.brandl at gmx.net (Georg Brandl) Date: Sun, 22 Sep 2013 10:48:33 +0200 Subject: [Python-Dev] Revert #12085 fix for __del__ attribute error message In-Reply-To: <20130922101936.53b9e68b@fsol> References: <l1l2bm$1bp$1@ger.gmane.org> <20130921221524.55D4225007F@webabinitio.net> <l1lbkl$k0g$1@ger.gmane.org> <20130922101936.53b9e68b@fsol> Message-ID: <l1marj$uta$1@ger.gmane.org> On 09/22/2013 10:19 AM, Antoine Pitrou wrote: > On Sat, 21 Sep 2013 19:55:05 -0400 > Terry Reedy <tjreedy at udel.edu> wrote: >> >> > In the general case it represents a bug in >> > the code that should be fixed. Most such errors arise from the vagaries >> > of module finalization (such as your issue 19021), >> >> Lets call that a buglet ;-). Not really harmful, but annoying. > > It's not a buglet. A buggy __del__ method is as much a bug as any other > buggy method. __del__ issues can lead to resources not being properly > finalized and released to the system. > >> Accepting >> that even such buglets 'should' be fixed in the stdllib, so that the >> message does not appear, is there any reason *not* to make it a >> RuntimeWarning so that users who care about clean output can filter it >> out while waiting for us to fix it? > > Yes, the reason is that these are real exceptions, not warnings. The > reason the exceptions are not propagated is that they can occur at any > point (finalization is pretty much asynchronous, it can occur from > unrelated places), so propagating them to whatever code happens to > execute at the time would be a huge instability factor. > > Making them warnings means they could be silenced depending on the > current warning settings. Which is not too bad, right? (Assuming it's not silent by default.) However, I'm not sure invoking the whole complicated warning filtering and emitting code is what we want to do in a destructor... Georg From g.brandl at gmx.net Sun Sep 22 10:54:42 2013 From: g.brandl at gmx.net (Georg Brandl) Date: Sun, 22 Sep 2013 10:54:42 +0200 Subject: [Python-Dev] Best practice for documentation for std lib In-Reply-To: <20130922031304.GE19939@ando> References: <20130922031304.GE19939@ando> Message-ID: <l1mb75$3vk$1@ger.gmane.org> On 09/22/2013 05:13 AM, Steven D'Aprano wrote: > Hi all, > > I have a question about how I should manage documentation for the > statistics module for Python 3.4. At the moment, I have extensive > docstrings in the module itself. I don't believe anyone has flagged that > as "too much information" in a code review, so I'm going to assume that > large docstrings will be acceptable. > > However, I have been asked to ensure that there is a separate > statistics.rst file for documentation. > > I don't want to have to maintain the documentation in two places, both > in the .py module and in .rst file. Can anyone give me some pointers as > to best practice in this situation? Is there a "How To Write Docs For > The Standard Library" document somewhere? Perhaps I missed it, but my > searches found nothing useful. I have read this: > > http://docs.python.org/devguide/documenting.html > > but it didn't shed any light on my situation. This is the "How To Write etc." document. If your docstrings really are complete, and marked up correctly, the new module *could* be the first to use Sphinx autodoc for the stdlib documentation. However, some core devs (including me) have stated opposition in the past. The reason I myself have never really wanted to do this for the stdlib, apart from most modules needing complete rewrites of their docstrings, is that when documenting a standard module, you have to go through a few hoops to make sure the build process actually imports the correct module to document, and not the one of the Python version used to build the docs (which can be different). I don't really buy the argument "but then there is no complete documentation set in the checkout" -- who reads that in preference to docs.python.org? And if anybody does want plain-text docs, they are already available for build and for download anyway (reST processed by Sphinx to remove non-plain markup). cheers, Georg From solipsis at pitrou.net Sun Sep 22 10:55:37 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 22 Sep 2013 10:55:37 +0200 Subject: [Python-Dev] Revert #12085 fix for __del__ attribute error message References: <l1l2bm$1bp$1@ger.gmane.org> <20130921221524.55D4225007F@webabinitio.net> <l1lbkl$k0g$1@ger.gmane.org> <20130922101936.53b9e68b@fsol> <l1marj$uta$1@ger.gmane.org> Message-ID: <20130922105537.1ec46f3f@fsol> On Sun, 22 Sep 2013 10:48:33 +0200 Georg Brandl <g.brandl at gmx.net> wrote: > On 09/22/2013 10:19 AM, Antoine Pitrou wrote: > > On Sat, 21 Sep 2013 19:55:05 -0400 > > Terry Reedy <tjreedy at udel.edu> wrote: > >> > >> > In the general case it represents a bug in > >> > the code that should be fixed. Most such errors arise from the vagaries > >> > of module finalization (such as your issue 19021), > >> > >> Lets call that a buglet ;-). Not really harmful, but annoying. > > > > It's not a buglet. A buggy __del__ method is as much a bug as any other > > buggy method. __del__ issues can lead to resources not being properly > > finalized and released to the system. > > > >> Accepting > >> that even such buglets 'should' be fixed in the stdllib, so that the > >> message does not appear, is there any reason *not* to make it a > >> RuntimeWarning so that users who care about clean output can filter it > >> out while waiting for us to fix it? > > > > Yes, the reason is that these are real exceptions, not warnings. The > > reason the exceptions are not propagated is that they can occur at any > > point (finalization is pretty much asynchronous, it can occur from > > unrelated places), so propagating them to whatever code happens to > > execute at the time would be a huge instability factor. > > > > Making them warnings means they could be silenced depending on the > > current warning settings. > > Which is not too bad, right? (Assuming it's not silent by default.) The default is to print a given warning message only once, which isn't the normal semantics of exceptions. > However, I'm not sure invoking the whole complicated warning filtering > and emitting code is what we want to do in a destructor... Good point, too :-) Regards Antoine. From ncoghlan at gmail.com Sun Sep 22 12:16:00 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 22 Sep 2013 20:16:00 +1000 Subject: [Python-Dev] Best practice for documentation for std lib In-Reply-To: <l1mb75$3vk$1@ger.gmane.org> References: <20130922031304.GE19939@ando> <l1mb75$3vk$1@ger.gmane.org> Message-ID: <CADiSq7cs5bTkzRREL5FWiqXA7Z+a6MKm9oyk=vu9ku9ivsmVDw@mail.gmail.com> On 22 September 2013 18:54, Georg Brandl <g.brandl at gmx.net> wrote: > I don't really buy the argument "but then there is no complete documentation > set in the checkout" -- who reads that in preference to docs.python.org? > And if anybody does want plain-text docs, they are already available for build > and for download anyway (reST processed by Sphinx to remove non-plain markup). This argument only applies to projects which have source and docs in separate checkouts, which doesn't apply to CPython :) As others have noted, the preferred approach is indeed to maintain the prose docs independently of the docstrings. The latter are often trimmed to just be a quick reminder of the details of how the function works for those that already know, while the prose docs go into more depth and have more examples. It's a bit of a pain, and we do occasionally get bug reports where the docstrings get out of date, but it's the least bad of the currently available options. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From python-dev at masklinn.net Sun Sep 22 13:05:14 2013 From: python-dev at masklinn.net (Xavier Morel) Date: Sun, 22 Sep 2013 13:05:14 +0200 Subject: [Python-Dev] Best practice for documentation for std lib In-Reply-To: <CADiSq7cs5bTkzRREL5FWiqXA7Z+a6MKm9oyk=vu9ku9ivsmVDw@mail.gmail.com> References: <20130922031304.GE19939@ando> <l1mb75$3vk$1@ger.gmane.org> <CADiSq7cs5bTkzRREL5FWiqXA7Z+a6MKm9oyk=vu9ku9ivsmVDw@mail.gmail.com> Message-ID: <4DF8E630-28E5-4635-BFBD-1F03481FD9E3@masklinn.net> On 2013-09-22, at 12:16 , Nick Coghlan wrote: > > It's a bit of a pain, and we do occasionally get bug reports where the > docstrings get out of date, but it's the least bad of the currently > available options. Is it really less bad than allowing limited fine-grained use of autodoc? Not necessarily class-level and definitely not module-level, but function- and method-level autodoc could allow avoiding duplication and make it clearer that the prose and docstrings are getting out of sync. From g.brandl at gmx.net Sun Sep 22 13:18:00 2013 From: g.brandl at gmx.net (Georg Brandl) Date: Sun, 22 Sep 2013 13:18:00 +0200 Subject: [Python-Dev] Best practice for documentation for std lib In-Reply-To: <CADiSq7cs5bTkzRREL5FWiqXA7Z+a6MKm9oyk=vu9ku9ivsmVDw@mail.gmail.com> References: <20130922031304.GE19939@ando> <l1mb75$3vk$1@ger.gmane.org> <CADiSq7cs5bTkzRREL5FWiqXA7Z+a6MKm9oyk=vu9ku9ivsmVDw@mail.gmail.com> Message-ID: <l1mjjq$e7e$1@ger.gmane.org> On 09/22/2013 12:16 PM, Nick Coghlan wrote: > On 22 September 2013 18:54, Georg Brandl <g.brandl at gmx.net> wrote: >> I don't really buy the argument "but then there is no complete documentation >> set in the checkout" -- who reads that in preference to docs.python.org? >> And if anybody does want plain-text docs, they are already available for build >> and for download anyway (reST processed by Sphinx to remove non-plain markup). > > This argument only applies to projects which have source and docs in > separate checkouts, which doesn't apply to CPython :) > > As others have noted, the preferred approach is indeed to maintain the > prose docs independently of the docstrings. The latter are often > trimmed to just be a quick reminder of the details of how the function > works for those that already know, while the prose docs go into more > depth and have more examples. That is the convincing argument, yes. It could be argued that autodoc allows adding content in the rst file itself, such as .. autofunction:: mean <docstring would go here in the output> Examples:: >>> mean(life) "Try and be nice to people, avoid eating fat, read a good book ..." But that means that when touching the docstring, one always has to look in the rst file as well to check if the added content still makes sense. cheers, Georg From steve at pearwood.info Sun Sep 22 14:53:23 2013 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 22 Sep 2013 22:53:23 +1000 Subject: [Python-Dev] Best practice for documentation for std lib In-Reply-To: <20130922102046.764abea5@fsol> References: <20130922031304.GE19939@ando> <20130922102046.764abea5@fsol> Message-ID: <20130922125323.GI19939@ando> On Sun, Sep 22, 2013 at 10:20:46AM +0200, Antoine Pitrou wrote: > On Sun, 22 Sep 2013 13:13:04 +1000 > Steven D'Aprano <steve at pearwood.info> wrote: > > Hi all, > > > > I have a question about how I should manage documentation for the > > statistics module for Python 3.4. At the moment, I have extensive > > docstrings in the module itself. I don't believe anyone has flagged that > > as "too much information" in a code review, so I'm going to assume that > > large docstrings will be acceptable. > > Related question: do the extensive docstrings make "help(stats)" > painful to browse through? Not to me. I can page through help(statistics) with 18 presses of the space bar, versus 20 for random or 45 for unittest. (29 lines per page.) Admittedly statistics has fewer functions/classes than random, but I find that fewer, larger pieces of documentation are easier to read than lots of tiny one-line mentions that don't actually tell you anything. E.g. from unittest: | Methods defined here: | | __init__(self, stream, descriptions, verbosity) | | addError(self, test, err) | | addExpectedFailure(self, test, err) | | addFailure(self, test, err) | | addSkip(self, test, reason) and so on for nearly a page. unittest is also packed with many, many one-line methods listed as deprecated. I admit I'm a bit of a stats and maths junkie, I read stats text books for fun. So perhaps I'm not the best person to judge how much information is too much information. Comments to the tracker please: http://bugs.python.org/issue18606 -- Steven From eliben at gmail.com Sun Sep 22 14:54:57 2013 From: eliben at gmail.com (Eli Bendersky) Date: Sun, 22 Sep 2013 05:54:57 -0700 Subject: [Python-Dev] Best practice for documentation for std lib In-Reply-To: <20130922031304.GE19939@ando> References: <20130922031304.GE19939@ando> Message-ID: <CAF-Rda_sTOiKi6c9ihFVL-N90PC-xHxOAMm1im0s-KVyy=L_Ng@mail.gmail.com> On Sat, Sep 21, 2013 at 8:13 PM, Steven D'Aprano <steve at pearwood.info>wrote: > Hi all, > > I have a question about how I should manage documentation for the > statistics module for Python 3.4. At the moment, I have extensive > docstrings in the module itself. I don't believe anyone has flagged that > as "too much information" in a code review, so I'm going to assume that > large docstrings will be acceptable. > > However, I have been asked to ensure that there is a separate > statistics.rst file for documentation. > > I don't want to have to maintain the documentation in two places, both > in the .py module and in .rst file. Can anyone give me some pointers as > to best practice in this situation? Is there a "How To Write Docs For > The Standard Library" document somewhere? Perhaps I missed it, but my > searches found nothing useful. I have read this: > > http://docs.python.org/devguide/documenting.html > > but it didn't shed any light on my situation. > IMHO the right way to think about it is that the .rst files are by far the more important documentation. Sometimes we forget that most Python programmers are people who won't go into the source to read docstrings. Moreover, the nice web layout, table of contents, index, and link-ability of .rst is very important - I also prefer to read it as opposed to going through docstrings. I only go to docstrings/code when I didn't find something in the .rst docs, at this point also usually opening a bug to fix that. So whatever you do for statistics, full .rst docs must be there. It's the docstrings that are "optional" here. The best solution may be auto generating, for sure. But a module without .rst documented is unacceptable. Eli -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130922/8d9cbea9/attachment.html> From g.brandl at gmx.net Sun Sep 22 15:06:40 2013 From: g.brandl at gmx.net (Georg Brandl) Date: Sun, 22 Sep 2013 15:06:40 +0200 Subject: [Python-Dev] Best practice for documentation for std lib In-Reply-To: <CAF-Rda_sTOiKi6c9ihFVL-N90PC-xHxOAMm1im0s-KVyy=L_Ng@mail.gmail.com> References: <20130922031304.GE19939@ando> <CAF-Rda_sTOiKi6c9ihFVL-N90PC-xHxOAMm1im0s-KVyy=L_Ng@mail.gmail.com> Message-ID: <l1mpvi$b1r$1@ger.gmane.org> On 09/22/2013 02:54 PM, Eli Bendersky wrote: > > > > On Sat, Sep 21, 2013 at 8:13 PM, Steven D'Aprano <steve at pearwood.info > <mailto:steve at pearwood.info>> wrote: > > Hi all, > > I have a question about how I should manage documentation for the > statistics module for Python 3.4. At the moment, I have extensive > docstrings in the module itself. I don't believe anyone has flagged that > as "too much information" in a code review, so I'm going to assume that > large docstrings will be acceptable. > > However, I have been asked to ensure that there is a separate > statistics.rst file for documentation. > > I don't want to have to maintain the documentation in two places, both > in the .py module and in .rst file. Can anyone give me some pointers as > to best practice in this situation? Is there a "How To Write Docs For > The Standard Library" document somewhere? Perhaps I missed it, but my > searches found nothing useful. I have read this: > > http://docs.python.org/devguide/documenting.html > > but it didn't shed any light on my situation. > > > IMHO the right way to think about it is that the .rst files are by far the more > important documentation. Sometimes we forget that most Python programmers are > people who won't go into the source to read docstrings. Moreover, the nice web > layout, table of contents, index, and link-ability of .rst is very important - I > also prefer to read it as opposed to going through docstrings. Note -- using autodoc gives you this. > I only go to docstrings/code when I didn't find something in the .rst docs, at > this point also usually opening a bug to fix that. > So whatever you do for statistics, full .rst docs must be there. I guess you don't mean .rst here; you mean .html (or .pdf, etc), whatever the toolchain outputs. cheers, Georg From p.f.moore at gmail.com Sun Sep 22 15:46:02 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 22 Sep 2013 14:46:02 +0100 Subject: [Python-Dev] Best practice for documentation for std lib In-Reply-To: <CAF-Rda_sTOiKi6c9ihFVL-N90PC-xHxOAMm1im0s-KVyy=L_Ng@mail.gmail.com> References: <20130922031304.GE19939@ando> <CAF-Rda_sTOiKi6c9ihFVL-N90PC-xHxOAMm1im0s-KVyy=L_Ng@mail.gmail.com> Message-ID: <CACac1F-xcmXNGuBzd9wqFteitqhhJG+RNi+rh4sM6DE4PbyjpQ@mail.gmail.com> On 22 September 2013 13:54, Eli Bendersky <eliben at gmail.com> wrote: > IMHO the right way to think about it is that the .rst files are by far the > more important documentation. Sometimes we forget that most Python > programmers are people who won't go into the source to read docstrings. While I agree entirely with this, it's worth noting that with the increasing popularity of IPython (even outside of the scientific community), more and more people are becoming used to using docstrings as a first means of finding out about a function's behaviour. Paul From eliben at gmail.com Sun Sep 22 16:16:57 2013 From: eliben at gmail.com (Eli Bendersky) Date: Sun, 22 Sep 2013 07:16:57 -0700 Subject: [Python-Dev] Best practice for documentation for std lib In-Reply-To: <l1mpvi$b1r$1@ger.gmane.org> References: <20130922031304.GE19939@ando> <CAF-Rda_sTOiKi6c9ihFVL-N90PC-xHxOAMm1im0s-KVyy=L_Ng@mail.gmail.com> <l1mpvi$b1r$1@ger.gmane.org> Message-ID: <CAF-Rda_W_M3nJBbzMZxXv8oTYVmOTW5FuJxO996ESZBozS_53Q@mail.gmail.com> On Sun, Sep 22, 2013 at 6:06 AM, Georg Brandl <g.brandl at gmx.net> wrote: > On 09/22/2013 02:54 PM, Eli Bendersky wrote: > > > > > > > > On Sat, Sep 21, 2013 at 8:13 PM, Steven D'Aprano <steve at pearwood.info > > <mailto:steve at pearwood.info>> wrote: > > > > Hi all, > > > > I have a question about how I should manage documentation for the > > statistics module for Python 3.4. At the moment, I have extensive > > docstrings in the module itself. I don't believe anyone has flagged > that > > as "too much information" in a code review, so I'm going to assume > that > > large docstrings will be acceptable. > > > > However, I have been asked to ensure that there is a separate > > statistics.rst file for documentation. > > > > I don't want to have to maintain the documentation in two places, > both > > in the .py module and in .rst file. Can anyone give me some pointers > as > > to best practice in this situation? Is there a "How To Write Docs For > > The Standard Library" document somewhere? Perhaps I missed it, but my > > searches found nothing useful. I have read this: > > > > http://docs.python.org/devguide/documenting.html > > > > but it didn't shed any light on my situation. > > > > > > IMHO the right way to think about it is that the .rst files are by far > the more > > important documentation. Sometimes we forget that most Python > programmers are > > people who won't go into the source to read docstrings. Moreover, the > nice web > > layout, table of contents, index, and link-ability of .rst is very > important - I > > also prefer to read it as opposed to going through docstrings. > > Note -- using autodoc gives you this. > > > I only go to docstrings/code when I didn't find something in the .rst > docs, at > > this point also usually opening a bug to fix that. > > So whatever you do for statistics, full .rst docs must be there. > > I guess you don't mean .rst here; you mean .html (or .pdf, etc), whatever > the > toolchain outputs. > Yes, by .rst docs I mean the final HTML output, not the .rst files themselves :-) Paul Moore: >While I agree entirely with this, it's worth noting that with the increasing popularity of IPython (even outside of the scientific community), more and more people are becoming used to using docstrings as a first means of finding out about a function's behaviour. -- That's a good point. I would still posit that HTML documentation gets by far the most use, but docstrings are definitely important too. One more point in favor of either: 1. Maintaining both, as tiresome as it is (we try to do this, not always successfully, for all stdlib modules). 2. autodoc Eli -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130922/2791ac89/attachment.html> From ncoghlan at gmail.com Sun Sep 22 16:25:52 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 23 Sep 2013 00:25:52 +1000 Subject: [Python-Dev] Best practice for documentation for std lib In-Reply-To: <CAF-Rda_W_M3nJBbzMZxXv8oTYVmOTW5FuJxO996ESZBozS_53Q@mail.gmail.com> References: <20130922031304.GE19939@ando> <CAF-Rda_sTOiKi6c9ihFVL-N90PC-xHxOAMm1im0s-KVyy=L_Ng@mail.gmail.com> <l1mpvi$b1r$1@ger.gmane.org> <CAF-Rda_W_M3nJBbzMZxXv8oTYVmOTW5FuJxO996ESZBozS_53Q@mail.gmail.com> Message-ID: <CADiSq7eG_pg6WUHSUr_zjw4aUuv5fXE=4xXKhzsNjjFcdL7oEw@mail.gmail.com> On 23 September 2013 00:16, Eli Bendersky <eliben at gmail.com> wrote: > That's a good point. I would still posit that HTML documentation gets by far > the most use, but docstrings are definitely important too. One more point in > favor of either: > > 1. Maintaining both, as tiresome as it is (we try to do this, not always > successfully, for all stdlib modules). > 2. autodoc FWIW, I've generally found *tactical* use of autodoc (i.e. function and method level usage for cases where the docstrings and prose docs *were* the same) to be quite effective. Then if there later proved to value in splitting them for a given case, that's what I would do. It isn't an all-or-nothing decision. As Georg noted, we'd have to do some fancy footwork to make sure autodoc didn't pick up the wrong module versions for the standard library docs, though. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From brett at python.org Sun Sep 22 16:34:12 2013 From: brett at python.org (Brett Cannon) Date: Sun, 22 Sep 2013 10:34:12 -0400 Subject: [Python-Dev] Best practice for documentation for std lib In-Reply-To: <20130922125323.GI19939@ando> References: <20130922031304.GE19939@ando> <20130922102046.764abea5@fsol> <20130922125323.GI19939@ando> Message-ID: <CAP1=2W7ymWcx_Yj6Xsb0Ty1Wj5pUf+_1XtqhPAVyRKWgiQ37mg@mail.gmail.com> On Sun, Sep 22, 2013 at 8:53 AM, Steven D'Aprano <steve at pearwood.info>wrote: > On Sun, Sep 22, 2013 at 10:20:46AM +0200, Antoine Pitrou wrote: > > On Sun, 22 Sep 2013 13:13:04 +1000 > > Steven D'Aprano <steve at pearwood.info> wrote: > > > Hi all, > > > > > > I have a question about how I should manage documentation for the > > > statistics module for Python 3.4. At the moment, I have extensive > > > docstrings in the module itself. I don't believe anyone has flagged > that > > > as "too much information" in a code review, so I'm going to assume that > > > large docstrings will be acceptable. > > > > Related question: do the extensive docstrings make "help(stats)" > > painful to browse through? > > Not to me. I can page through help(statistics) with 18 presses of the > space bar, versus 20 for random or 45 for unittest. (29 lines per page.) > > Admittedly statistics has fewer functions/classes than random, but I > find that fewer, larger pieces of documentation are easier to read than > lots of tiny one-line mentions that don't actually tell you anything. > E.g. from unittest: > > | Methods defined here: > | > | __init__(self, stream, descriptions, verbosity) > | > | addError(self, test, err) > | > | addExpectedFailure(self, test, err) > | > | addFailure(self, test, err) > | > | addSkip(self, test, reason) > > and so on for nearly a page. unittest is also packed with many, many > one-line methods listed as deprecated. > > I admit I'm a bit of a stats and maths junkie, I read stats text books > for fun. So perhaps I'm not the best person to judge how much > information is too much information. Comments to the tracker please: > > http://bugs.python.org/issue18606 The rule of thumb I go by is the docstring should be enough to answer the question "what args does this thing take and what does it do in general to know it's the function I want and another one in the same module?" quickly and succinctly; i.e. just enough so that help() reminds you about details for a module you are already familiar with that might come up while at the interpreter prompt. Everything else -- in-depth discussion of the algorithms, extra examples, why you want to use this function, etc. -- all go in the .rst docs. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130922/669ee320/attachment-0001.html> From brewdog.org at masklinn.net Sun Sep 22 09:05:14 2013 From: brewdog.org at masklinn.net (Xavier Morel) Date: Sun, 22 Sep 2013 09:05:14 +0200 Subject: [Python-Dev] Best practice for documentation for std lib In-Reply-To: <CAPZV6o8HnVfc-wFdPjHZtbh_s4X7MU-Kqi0QewUR2ar_n3Gogg@mail.gmail.com> References: <20130922031304.GE19939@ando> <CAPZV6o8HnVfc-wFdPjHZtbh_s4X7MU-Kqi0QewUR2ar_n3Gogg@mail.gmail.com> Message-ID: <E6A3A834-3AEF-4A7F-9A03-E3E5CA1A988F@masklinn.net> > On 22 Sep 2013, at 05:25, Benjamin Peterson <benjamin at python.org> wrote: > > There's not really much to do but maintain them separately. Truncate > the docstrings if it makes life easier. Autodoc could be enabled and allowed in a limited manner. From barry at python.org Sun Sep 22 17:49:33 2013 From: barry at python.org (Barry Warsaw) Date: Sun, 22 Sep 2013 11:49:33 -0400 Subject: [Python-Dev] Best practice for documentation for std lib In-Reply-To: <CAP1=2W7ymWcx_Yj6Xsb0Ty1Wj5pUf+_1XtqhPAVyRKWgiQ37mg@mail.gmail.com> References: <20130922031304.GE19939@ando> <20130922102046.764abea5@fsol> <20130922125323.GI19939@ando> <CAP1=2W7ymWcx_Yj6Xsb0Ty1Wj5pUf+_1XtqhPAVyRKWgiQ37mg@mail.gmail.com> Message-ID: <20130922114933.7c8ecde7@anarchist> On Sep 22, 2013, at 10:34 AM, Brett Cannon wrote: >The rule of thumb I go by is the docstring should be enough to answer the >question "what args does this thing take and what does it do in general to >know it's the function I want and another one in the same module?" quickly >and succinctly; i.e. just enough so that help() reminds you about details >for a module you are already familiar with that might come up while at the >interpreter prompt. Everything else -- in-depth discussion of the >algorithms, extra examples, why you want to use this function, etc. -- all >go in the .rst docs. That's exactly my own rule of thumb too, so +1. -Barry From eliben at gmail.com Sun Sep 22 18:37:15 2013 From: eliben at gmail.com (Eli Bendersky) Date: Sun, 22 Sep 2013 09:37:15 -0700 Subject: [Python-Dev] Best practice for documentation for std lib In-Reply-To: <20130922114933.7c8ecde7@anarchist> References: <20130922031304.GE19939@ando> <20130922102046.764abea5@fsol> <20130922125323.GI19939@ando> <CAP1=2W7ymWcx_Yj6Xsb0Ty1Wj5pUf+_1XtqhPAVyRKWgiQ37mg@mail.gmail.com> <20130922114933.7c8ecde7@anarchist> Message-ID: <CAF-Rda_NnM-9Nt8uhc5b3k9MQwY0qVthki=fZdnPWvufoiXm=Q@mail.gmail.com> On Sun, Sep 22, 2013 at 8:49 AM, Barry Warsaw <barry at python.org> wrote: > On Sep 22, 2013, at 10:34 AM, Brett Cannon wrote: > > >The rule of thumb I go by is the docstring should be enough to answer the > >question "what args does this thing take and what does it do in general to > >know it's the function I want and another one in the same module?" quickly > >and succinctly; i.e. just enough so that help() reminds you about details > >for a module you are already familiar with that might come up while at the > >interpreter prompt. Everything else -- in-depth discussion of the > >algorithms, extra examples, why you want to use this function, etc. -- all > >go in the .rst docs. > > That's exactly my own rule of thumb too, so +1. > It makes a lot of sense to me too. That said, I don't see a reason why we can't auto-generate the referenc-y stuff from docstrings into the .rst automatically, i.e.: module.rst: Algorithms, bla bla bla, examples bla bla bla <ref:module.py Foo.bar> # << Auto generator pastes function signature and docstring here More algorithms, more bla bla, examples. Eli -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130922/05930883/attachment.html> From stephen at xemacs.org Sun Sep 22 18:53:10 2013 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Mon, 23 Sep 2013 01:53:10 +0900 Subject: [Python-Dev] Best practice for documentation for std lib In-Reply-To: <CAF-Rda_sTOiKi6c9ihFVL-N90PC-xHxOAMm1im0s-KVyy=L_Ng@mail.gmail.com> References: <20130922031304.GE19939@ando> <CAF-Rda_sTOiKi6c9ihFVL-N90PC-xHxOAMm1im0s-KVyy=L_Ng@mail.gmail.com> Message-ID: <87ioxs3iix.fsf@uwakimon.sk.tsukuba.ac.jp> Eli Bendersky writes: > IMHO the right way to think about it is that the .rst files are by > far the more important documentation. Sometimes we forget that > most Python programmers are people who won't go into the source Why "source"? The whole point of docstrings is that they are *not* comments found only in the source, but available at run time. In fact, programmers who also use environments like Lisp or R (not to forget Idle) will reach for "help(mean)", and that works fine for Steven, because he provides such nice docstrings. Some people prefer to write separate manuals, and some modules *should* be documented that way because their internal complexity or whatever. That's true, but I would hope authors who prefer "literate programming" (or the poor man's lit prog that is writing only docstrings) are encouraged to do so when appropriate. Of course, like any other contribution, since that style is *not* currently supported by python-dev, they'd be asked to step up and support it themselves -- if a user reports the docs won't build, they need to address that like they would a build bug in the code. From guido at python.org Sun Sep 22 19:07:28 2013 From: guido at python.org (Guido van Rossum) Date: Sun, 22 Sep 2013 10:07:28 -0700 Subject: [Python-Dev] Best practice for documentation for std lib In-Reply-To: <87ioxs3iix.fsf@uwakimon.sk.tsukuba.ac.jp> References: <20130922031304.GE19939@ando> <CAF-Rda_sTOiKi6c9ihFVL-N90PC-xHxOAMm1im0s-KVyy=L_Ng@mail.gmail.com> <87ioxs3iix.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <CAP7+vJJucubZtXHhnNbAcE0GWfQTnOozHK9W=uv3CWkMX15XGA@mail.gmail.com> On Sun, Sep 22, 2013 at 9:53 AM, Stephen J. Turnbull <stephen at xemacs.org>wrote: > Eli Bendersky writes: > > > IMHO the right way to think about it is that the .rst files are by > > far the more important documentation. Sometimes we forget that > > most Python programmers are people who won't go into the source > > Why "source"? The whole point of docstrings is that they are *not* > comments found only in the source, but available at run time. In fact, > programmers who also use environments like Lisp or R (not to forget > Idle) will reach for "help(mean)", and that works fine for Steven, > because he provides such nice docstrings. > > Some people prefer to write separate manuals, and some modules > *should* be documented that way because their internal complexity or > whatever. That's true, but I would hope authors who prefer "literate > programming" (or the poor man's lit prog that is writing only > docstrings) are encouraged to do so when appropriate. > > Of course, like any other contribution, since that style is *not* > currently supported by python-dev, they'd be asked to step up and > support it themselves -- if a user reports the docs won't build, they > need to address that like they would a build bug in the code. Authors writing 3rd party packages can do what they want. But for the stdlib it's been settled for ages: docstrings should be concise (but not cryptic(*)), longer documentation go into the separately written text for docs.python.org. (*) This is too concise to my taste: $ ls -? ls: illegal option -- ? usage: ls [-ABCFGHLOPRSTUWabcdefghiklmnopqrstuwx1] [file ...] $ -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130922/a257de78/attachment.html> From eliben at gmail.com Sun Sep 22 19:25:08 2013 From: eliben at gmail.com (Eli Bendersky) Date: Sun, 22 Sep 2013 10:25:08 -0700 Subject: [Python-Dev] Best practice for documentation for std lib In-Reply-To: <CAP7+vJJucubZtXHhnNbAcE0GWfQTnOozHK9W=uv3CWkMX15XGA@mail.gmail.com> References: <20130922031304.GE19939@ando> <CAF-Rda_sTOiKi6c9ihFVL-N90PC-xHxOAMm1im0s-KVyy=L_Ng@mail.gmail.com> <87ioxs3iix.fsf@uwakimon.sk.tsukuba.ac.jp> <CAP7+vJJucubZtXHhnNbAcE0GWfQTnOozHK9W=uv3CWkMX15XGA@mail.gmail.com> Message-ID: <CAF-Rda-qJ3fZQTpVO_x31WhWOaf7Q4+0n7w-sFZgK8tXWLL0zw@mail.gmail.com> On Sun, Sep 22, 2013 at 10:07 AM, Guido van Rossum <guido at python.org> wrote: > On Sun, Sep 22, 2013 at 9:53 AM, Stephen J. Turnbull <stephen at xemacs.org>wrote: > >> Eli Bendersky writes: >> >> > IMHO the right way to think about it is that the .rst files are by >> > far the more important documentation. Sometimes we forget that >> > most Python programmers are people who won't go into the source >> >> Why "source"? The whole point of docstrings is that they are *not* >> comments found only in the source, but available at run time. In fact, >> programmers who also use environments like Lisp or R (not to forget >> Idle) will reach for "help(mean)", and that works fine for Steven, >> because he provides such nice docstrings. >> >> Some people prefer to write separate manuals, and some modules >> *should* be documented that way because their internal complexity or >> whatever. That's true, but I would hope authors who prefer "literate >> programming" (or the poor man's lit prog that is writing only >> docstrings) are encouraged to do so when appropriate. >> >> Of course, like any other contribution, since that style is *not* >> currently supported by python-dev, they'd be asked to step up and >> support it themselves -- if a user reports the docs won't build, they >> need to address that like they would a build bug in the code. > > > Authors writing 3rd party packages can do what they want. > > But for the stdlib it's been settled for ages: docstrings should be > concise (but not cryptic(*)), longer documentation go into the separately > written text for docs.python.org. > I think there's a general agreement in this thread that we don't intend to change the status quo. Both .rst docs and docstrings are important. The remaining question is - can we use some tool to generates parts of the former from the latter and thus avoid duplication and rot? Eli -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130922/047b7764/attachment.html> From ethan at stoneleaf.us Sun Sep 22 18:32:31 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Sun, 22 Sep 2013 09:32:31 -0700 Subject: [Python-Dev] Best practice for documentation for std lib In-Reply-To: <20130922114933.7c8ecde7@anarchist> References: <20130922031304.GE19939@ando> <20130922102046.764abea5@fsol> <20130922125323.GI19939@ando> <CAP1=2W7ymWcx_Yj6Xsb0Ty1Wj5pUf+_1XtqhPAVyRKWgiQ37mg@mail.gmail.com> <20130922114933.7c8ecde7@anarchist> Message-ID: <523F1B9F.90101@stoneleaf.us> On 09/22/2013 08:49 AM, Barry Warsaw wrote: > On Sep 22, 2013, at 10:34 AM, Brett Cannon wrote: > >> The rule of thumb I go by is the docstring should be enough to answer the >> question "what args does this thing take and what does it do in general to >> know it's the function I want and another one in the same module?" quickly >> and succinctly; i.e. just enough so that help() reminds you about details >> for a module you are already familiar with that might come up while at the >> interpreter prompt. Everything else -- in-depth discussion of the >> algorithms, extra examples, why you want to use this function, etc. -- all >> go in the .rst docs. > > That's exactly my own rule of thumb too, so +1. Another +1. So it that three rules or three thumbs? ;) -- ~Ethan~ From tjreedy at udel.edu Sun Sep 22 19:35:05 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 22 Sep 2013 13:35:05 -0400 Subject: [Python-Dev] Revert #12085 fix for __del__ attribute error message In-Reply-To: <CAP7+vJLHO12ZNwX4UZ0c_BF9pr+xxxLZfy2dGjBcMVenGow9vQ@mail.gmail.com> References: <l1l2bm$1bp$1@ger.gmane.org> <20130921221524.55D4225007F@webabinitio.net> <l1lbkl$k0g$1@ger.gmane.org> <CAP7+vJLHO12ZNwX4UZ0c_BF9pr+xxxLZfy2dGjBcMVenGow9vQ@mail.gmail.com> Message-ID: <l1n9o6$ap3$1@ger.gmane.org> On 9/21/2013 10:30 PM, Guido van Rossum wrote: > Exceptions in __del__ point to bugs (sometimes in the stdlib) that > should be fixed, period. The only reason they do not result in > exceptions that are properly bubbled up and catchable is because __del__ > is called from a DECREF macro which has no return value. That is clear enough. What fooled me is the word 'ignored', in both the doc and message. How about 'skipped' (for technical reasons)? > Also, IMO > writing to stderr is fair game -- reporting errors is what it is for. So developers who really want to control all screen output should redirect or capture it somehow. > As to making them warnings, I don't know that the warnings machinery is > easily adaptable for this purpose. Warnings can be suppressed but they > can also be turned into full exceptions; the latter doesn't really apply > here (see previous paragraph). But I would not object if someone found a > way to do this, though I'd prefer the default behavior to remain what it > is in 3.4 (print a full traceback). Antoine and Georg think it a dubious idea, so I will not pursue it. Developers who encounter messages from the stdlib can report and wait for a fix. -- Terry Jan Reedy From tjreedy at udel.edu Sun Sep 22 19:45:25 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 22 Sep 2013 13:45:25 -0400 Subject: [Python-Dev] Best practice for documentation for std lib In-Reply-To: <CADiSq7eG_pg6WUHSUr_zjw4aUuv5fXE=4xXKhzsNjjFcdL7oEw@mail.gmail.com> References: <20130922031304.GE19939@ando> <CAF-Rda_sTOiKi6c9ihFVL-N90PC-xHxOAMm1im0s-KVyy=L_Ng@mail.gmail.com> <l1mpvi$b1r$1@ger.gmane.org> <CAF-Rda_W_M3nJBbzMZxXv8oTYVmOTW5FuJxO996ESZBozS_53Q@mail.gmail.com> <CADiSq7eG_pg6WUHSUr_zjw4aUuv5fXE=4xXKhzsNjjFcdL7oEw@mail.gmail.com> Message-ID: <l1nabi$h23$1@ger.gmane.org> On 9/22/2013 10:25 AM, Nick Coghlan wrote: > As Georg noted, we'd have to do some fancy footwork to make sure > autodoc didn't pick up the wrong module versions for the standard > library docs, though. Is that a one-time nuisance, a per-module nuisance, a per-autodoc-use nuisance, or a per-build nuisance? -- Terry Jan Reedy From anikom15 at gmail.com Sun Sep 22 20:02:17 2013 From: anikom15 at gmail.com (=?UTF-8?Q?Westley_Mart=C3=ADnez?=) Date: Sun, 22 Sep 2013 11:02:17 -0700 Subject: [Python-Dev] Best practice for documentation for std lib In-Reply-To: <CAP1=2W7ymWcx_Yj6Xsb0Ty1Wj5pUf+_1XtqhPAVyRKWgiQ37mg@mail.gmail.com> References: <20130922031304.GE19939@ando> <20130922102046.764abea5@fsol> <20130922125323.GI19939@ando> <CAP1=2W7ymWcx_Yj6Xsb0Ty1Wj5pUf+_1XtqhPAVyRKWgiQ37mg@mail.gmail.com> Message-ID: <003301ceb7bd$e1a59430$a4f0bc90$@gmail.com> Since help() is usually the first thing I use to remind myself of an API, it's imperative that the doc strings are up to date and make sense to end users. So, autogeneration of doc strings would be good for someone like me who hardly uses the html docs. > -----Original Message----- > From: Python-Dev [mailto:python-dev-bounces+anikom15=gmail.com at python.org] On > Behalf Of Brett Cannon > Sent: Sunday, September 22, 2013 7:34 AM > To: Steven D'Aprano > Cc: python-dev > Subject: Re: [Python-Dev] Best practice for documentation for std lib > > > > > On Sun, Sep 22, 2013 at 8:53 AM, Steven D'Aprano <steve at pearwood.info> wrote: > > > On Sun, Sep 22, 2013 at 10:20:46AM +0200, Antoine Pitrou wrote: > > On Sun, 22 Sep 2013 13:13:04 +1000 > > Steven D'Aprano <steve at pearwood.info> wrote: > > > Hi all, > > > > > > I have a question about how I should manage documentation for the > > > statistics module for Python 3.4. At the moment, I have extensive > > > docstrings in the module itself. I don't believe anyone has flagged > that > > > as "too much information" in a code review, so I'm going to assume > that > > > large docstrings will be acceptable. > > > > Related question: do the extensive docstrings make "help(stats)" > > painful to browse through? > > > Not to me. I can page through help(statistics) with 18 presses of the > space bar, versus 20 for random or 45 for unittest. (29 lines per page.) > > Admittedly statistics has fewer functions/classes than random, but I > find that fewer, larger pieces of documentation are easier to read than > lots of tiny one-line mentions that don't actually tell you anything. > E.g. from unittest: > > | Methods defined here: > | > | __init__(self, stream, descriptions, verbosity) > | > | addError(self, test, err) > | > | addExpectedFailure(self, test, err) > | > | addFailure(self, test, err) > | > | addSkip(self, test, reason) > > and so on for nearly a page. unittest is also packed with many, many > one-line methods listed as deprecated. > > I admit I'm a bit of a stats and maths junkie, I read stats text books > for fun. So perhaps I'm not the best person to judge how much > information is too much information. Comments to the tracker please: > > http://bugs.python.org/issue18606 > > > The rule of thumb I go by is the docstring should be enough to answer the > question "what args does this thing take and what does it do in general to > know it's the function I want and another one in the same module?" quickly and > succinctly; i.e. just enough so that help() reminds you about details for a > module you are already familiar with that might come up while at the > interpreter prompt. Everything else -- in-depth discussion of the algorithms, > extra examples, why you want to use this function, etc. -- all go in the .rst > docs. From guido at python.org Sun Sep 22 20:34:56 2013 From: guido at python.org (Guido van Rossum) Date: Sun, 22 Sep 2013 11:34:56 -0700 Subject: [Python-Dev] Best practice for documentation for std lib In-Reply-To: <003301ceb7bd$e1a59430$a4f0bc90$@gmail.com> References: <20130922031304.GE19939@ando> <20130922102046.764abea5@fsol> <20130922125323.GI19939@ando> <CAP1=2W7ymWcx_Yj6Xsb0Ty1Wj5pUf+_1XtqhPAVyRKWgiQ37mg@mail.gmail.com> <003301ceb7bd$e1a59430$a4f0bc90$@gmail.com> Message-ID: <CAP7+vJLKVNWEDT62U4pJ7FGARthhVqb74hnLWznBag353s-Obw@mail.gmail.com> On Sun, Sep 22, 2013 at 11:02 AM, Westley Mart?nez <anikom15 at gmail.com>wrote: > Since help() is usually the first thing I use to remind myself of an > API, it's imperative that the doc strings are up to date and make sense > to end users. > > So, autogeneration of doc strings would be good for someone like me who > hardly uses the html docs. > You seem to misunderstand the use of "autogeneration". It refers to generating the .rst docs from the docstrings in the source. And FWIW, I'm against that practice. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130922/1425e7e9/attachment-0001.html> From guido at python.org Sun Sep 22 20:40:41 2013 From: guido at python.org (Guido van Rossum) Date: Sun, 22 Sep 2013 11:40:41 -0700 Subject: [Python-Dev] Best practice for documentation for std lib In-Reply-To: <CAF-Rda-qJ3fZQTpVO_x31WhWOaf7Q4+0n7w-sFZgK8tXWLL0zw@mail.gmail.com> References: <20130922031304.GE19939@ando> <CAF-Rda_sTOiKi6c9ihFVL-N90PC-xHxOAMm1im0s-KVyy=L_Ng@mail.gmail.com> <87ioxs3iix.fsf@uwakimon.sk.tsukuba.ac.jp> <CAP7+vJJucubZtXHhnNbAcE0GWfQTnOozHK9W=uv3CWkMX15XGA@mail.gmail.com> <CAF-Rda-qJ3fZQTpVO_x31WhWOaf7Q4+0n7w-sFZgK8tXWLL0zw@mail.gmail.com> Message-ID: <CAP7+vJLmLZOYmMeoKqoqrAGqVF=Z41iYBRiis5buAMyaON+_+A@mail.gmail.com> On Sun, Sep 22, 2013 at 10:25 AM, Eli Bendersky <eliben at gmail.com> wrote: > > I think there's a general agreement in this thread that we don't intend to > change the status quo. Both .rst docs and docstrings are important. The > remaining question is - can we use some tool to generates parts of the > former from the latter and thus avoid duplication and rot? > I don't think that duplication is much of an issue. Natural language understanding is not at the level yet where you can generate a meaningful summary from a longer text fully automatically (let alone vice versa :-) so I think having to write both a concise docstring and a longer more detailed description for the Doc tree is not a waste of effort at all. As for rot, it's just as likely that rot occurs as a *result* of autogeneration. Having to edit/patch the source code in order to improve the documentation most likely adds an extra barrier towards improving the docs. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130922/85f5f82b/attachment.html> From guido at python.org Sun Sep 22 20:41:29 2013 From: guido at python.org (Guido van Rossum) Date: Sun, 22 Sep 2013 11:41:29 -0700 Subject: [Python-Dev] Revert #12085 fix for __del__ attribute error message In-Reply-To: <l1n9o6$ap3$1@ger.gmane.org> References: <l1l2bm$1bp$1@ger.gmane.org> <20130921221524.55D4225007F@webabinitio.net> <l1lbkl$k0g$1@ger.gmane.org> <CAP7+vJLHO12ZNwX4UZ0c_BF9pr+xxxLZfy2dGjBcMVenGow9vQ@mail.gmail.com> <l1n9o6$ap3$1@ger.gmane.org> Message-ID: <CAP7+vJL+0XEBpOnVbTpZJNLzahgm8EkyuXdGdhE27TVwHu4GAg@mail.gmail.com> On Sun, Sep 22, 2013 at 10:35 AM, Terry Reedy <tjreedy at udel.edu> wrote: > On 9/21/2013 10:30 PM, Guido van Rossum wrote: > >> Exceptions in __del__ point to bugs (sometimes in the stdlib) that >> should be fixed, period. The only reason they do not result in >> exceptions that are properly bubbled up and catchable is because __del__ >> is called from a DECREF macro which has no return value. >> > > That is clear enough. What fooled me is the word 'ignored', in both the > doc and message. How about 'skipped' (for technical reasons)? That's a good point, although I'm not sure 'skipped' is better. Maybe use a more neutral verb like 'occurred'? -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130922/5efd7650/attachment.html> From eliben at gmail.com Sun Sep 22 20:44:43 2013 From: eliben at gmail.com (Eli Bendersky) Date: Sun, 22 Sep 2013 11:44:43 -0700 Subject: [Python-Dev] Best practice for documentation for std lib In-Reply-To: <CAP7+vJLmLZOYmMeoKqoqrAGqVF=Z41iYBRiis5buAMyaON+_+A@mail.gmail.com> References: <20130922031304.GE19939@ando> <CAF-Rda_sTOiKi6c9ihFVL-N90PC-xHxOAMm1im0s-KVyy=L_Ng@mail.gmail.com> <87ioxs3iix.fsf@uwakimon.sk.tsukuba.ac.jp> <CAP7+vJJucubZtXHhnNbAcE0GWfQTnOozHK9W=uv3CWkMX15XGA@mail.gmail.com> <CAF-Rda-qJ3fZQTpVO_x31WhWOaf7Q4+0n7w-sFZgK8tXWLL0zw@mail.gmail.com> <CAP7+vJLmLZOYmMeoKqoqrAGqVF=Z41iYBRiis5buAMyaON+_+A@mail.gmail.com> Message-ID: <CAF-Rda8H2hi5V=HakbGsD2WzH0HQ6hLuxkQYZ3ZC+MzDfb1hVw@mail.gmail.com> On Sun, Sep 22, 2013 at 11:40 AM, Guido van Rossum <guido at python.org> wrote: > On Sun, Sep 22, 2013 at 10:25 AM, Eli Bendersky <eliben at gmail.com> wrote: > >> >> I think there's a general agreement in this thread that we don't intend >> to change the status quo. Both .rst docs and docstrings are important. The >> remaining question is - can we use some tool to generates parts of the >> former from the latter and thus avoid duplication and rot? >> > > I don't think that duplication is much of an issue. Natural language > understanding is not at the level yet where you can generate a meaningful > summary from a longer text fully automatically (let alone vice versa :-) so > I think having to write both a concise docstring and a longer more detailed > description for the Doc tree is not a waste of effort at all. > I don't think the proposal is to generate summaries from a longer text. AFAIU, the proposal is to embed parts of the concise docstring into the more verbose .rst documentation. I write .rst docs quite a lot, and as such I do notice the annoying amount of duplication between docstrings and .rst docs. Without doubt, all the free-flow text and examples in .rst have to be written manually and are very important. But for dry method references, it's certainly interesting to explore the idea of writing them once in the code and then having Sphinx automatically insert the relevant parts into the .rst before generating HTML from it. For end users (those who read the docs online) the result is indistinguishable from what we have now. For us devs it means writing the same text only once and maintaining it in a single place. Think about the new statistics module as a guinea pig. Steven will have a whole lot of copy-pasting to do :-) > > As for rot, it's just as likely that rot occurs as a *result* of > autogeneration. Having to edit/patch the source code in order to improve > the documentation most likely adds an extra barrier towards improving the > docs. > This is a valid concern, but perhaps one that can be addressed separately? (i.e. lowering that barrier of entry). Eli -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130922/98d4d2df/attachment.html> From anikom15 at gmail.com Sun Sep 22 21:24:04 2013 From: anikom15 at gmail.com (=?iso-8859-1?Q?Westley_Mart=EDnez?=) Date: Sun, 22 Sep 2013 12:24:04 -0700 Subject: [Python-Dev] Best practice for documentation for std lib In-Reply-To: <CAP7+vJLKVNWEDT62U4pJ7FGARthhVqb74hnLWznBag353s-Obw@mail.gmail.com> References: <20130922031304.GE19939@ando> <20130922102046.764abea5@fsol> <20130922125323.GI19939@ando> <CAP1=2W7ymWcx_Yj6Xsb0Ty1Wj5pUf+_1XtqhPAVyRKWgiQ37mg@mail.gmail.com> <003301ceb7bd$e1a59430$a4f0bc90$@gmail.com> <CAP7+vJLKVNWEDT62U4pJ7FGARthhVqb74hnLWznBag353s-Obw@mail.gmail.com> Message-ID: <005001ceb7c9$4dc308e0$e9491aa0$@gmail.com> > From: gvanrossum at gmail.com [mailto:gvanrossum at gmail.com] On Behalf Of Guido > van Rossum > Sent: Sunday, September 22, 2013 11:35 AM > > You seem to misunderstand the use of "autogeneration". It refers to generating > the .rst docs from the docstrings in the source. And FWIW, I'm against that > practice. Oh I see. Well in that case, the docstrings can still become outdated, and so then the .rst docs will be outdated, too. It doesn't seem to offer much benefit since you still have to keep both updated, plus you have an extra tool that must be maintained. From tjreedy at udel.edu Sun Sep 22 22:37:08 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 22 Sep 2013 16:37:08 -0400 Subject: [Python-Dev] Revert #12085 fix for __del__ attribute error message In-Reply-To: <CAP7+vJL+0XEBpOnVbTpZJNLzahgm8EkyuXdGdhE27TVwHu4GAg@mail.gmail.com> References: <l1l2bm$1bp$1@ger.gmane.org> <20130921221524.55D4225007F@webabinitio.net> <l1lbkl$k0g$1@ger.gmane.org> <CAP7+vJLHO12ZNwX4UZ0c_BF9pr+xxxLZfy2dGjBcMVenGow9vQ@mail.gmail.com> <l1n9o6$ap3$1@ger.gmane.org> <CAP7+vJL+0XEBpOnVbTpZJNLzahgm8EkyuXdGdhE27TVwHu4GAg@mail.gmail.com> Message-ID: <l1nkdh$lh7$1@ger.gmane.org> On 9/22/2013 2:41 PM, Guido van Rossum wrote: > On Sun, Sep 22, 2013 at 10:35 AM, Terry Reedy <tjreedy at udel.edu > <mailto:tjreedy at udel.edu>> wrote: > > On 9/21/2013 10:30 PM, Guido van Rossum wrote: > > Exceptions in __del__ point to bugs (sometimes in the stdlib) that > should be fixed, period. The only reason they do not result in > exceptions that are properly bubbled up and catchable is because > __del__ > is called from a DECREF macro which has no return value. > > > That is clear enough. What fooled me is the word 'ignored', in both > the doc and message. How about 'skipped' (for technical reasons)? > > > That's a good point, although I'm not sure 'skipped' is better. Only slightly ;-). The problem with both words is that they try to say two things. What happened, and what Python did about it. > Maybe use a more neutral verb like 'occurred'? "Exception occurred in ..." is even better at say what happened. I think we should then add an explict statement as to what Python did, and hint at what the user should do, something like "Although caught internally, it still indicates a problem." Otherwise, when no other output follows, as in ... > del c Exception ocurred in: <bound method C.__del__ ... Traceback (most recent call last): File "<pyshell#0>", line 2, in __del__ AttributeError: >>> It may not be completely obvious to a non-expert that the traceback is not a 'real' traceback from an exception that was allowed to propagate, and that it did not stop execution and cannot be caught. -- Terry Jan Reedy From python-dev at masklinn.net Sun Sep 22 23:41:39 2013 From: python-dev at masklinn.net (Xavier Morel) Date: Sun, 22 Sep 2013 23:41:39 +0200 Subject: [Python-Dev] Best practice for documentation for std lib In-Reply-To: <005001ceb7c9$4dc308e0$e9491aa0$@gmail.com> References: <20130922031304.GE19939@ando> <20130922102046.764abea5@fsol> <20130922125323.GI19939@ando> <CAP1=2W7ymWcx_Yj6Xsb0Ty1Wj5pUf+_1XtqhPAVyRKWgiQ37mg@mail.gmail.com> <003301ceb7bd$e1a59430$a4f0bc90$@gmail.com> <CAP7+vJLKVNWEDT62U4pJ7FGARthhVqb74hnLWznBag353s-Obw@mail.gmail.com> <005001ceb7c9$4dc308e0$e9491aa0$@gmail.com> Message-ID: <06E35F19-F893-415B-B127-77D1588A565A@masklinn.net> On 2013-09-22, at 21:24 , Westley Mart?nez wrote: >> From: gvanrossum at gmail.com [mailto:gvanrossum at gmail.com] On Behalf Of Guido >> van Rossum >> Sent: Sunday, September 22, 2013 11:35 AM >> >> You seem to misunderstand the use of "autogeneration". It refers to generating >> the .rst docs from the docstrings in the source. And FWIW, I'm against that >> practice. > > Oh I see. Well in that case, the docstrings can still become outdated, > and so then the .rst docs will be outdated, too. The points here are that there's a single source of truth (so we can't have conflicting docstring and rst documentation), and documentation becoming outdated can be noticed from both docstring and published documentation. > It doesn't seem to > offer much benefit since you still have to keep both updated, plus you > have an extra tool that must be maintained. There is no extra tool, autodoc is part of the standard Sphinx distribution: http://sphinx-doc.org/ext/autodoc.html From anikom15 at gmail.com Sun Sep 22 23:49:56 2013 From: anikom15 at gmail.com (=?iso-8859-1?Q?Westley_Mart=EDnez?=) Date: Sun, 22 Sep 2013 14:49:56 -0700 Subject: [Python-Dev] Best practice for documentation for std lib In-Reply-To: <06E35F19-F893-415B-B127-77D1588A565A@masklinn.net> References: <20130922031304.GE19939@ando> <20130922102046.764abea5@fsol> <20130922125323.GI19939@ando> <CAP1=2W7ymWcx_Yj6Xsb0Ty1Wj5pUf+_1XtqhPAVyRKWgiQ37mg@mail.gmail.com> <003301ceb7bd$e1a59430$a4f0bc90$@gmail.com> <CAP7+vJLKVNWEDT62U4pJ7FGARthhVqb74hnLWznBag353s-Obw@mail.gmail.com> <005001ceb7c9$4dc308e0$e9491aa0$@gmail.com> <06E35F19-F893-415B-B127-77D1588A565A@masklinn.net> Message-ID: <000001ceb7dd$aedfa1b0$0c9ee510$@gmail.com> Well, I'm wholly confused now, so I'll leave this discussion to the devs. > -----Original Message----- > From: Python-Dev [mailto:python-dev-bounces+anikom15=gmail.com at python.org] On > Behalf Of Xavier Morel > Sent: Sunday, September 22, 2013 2:42 PM > To: python-dev > Subject: Re: [Python-Dev] Best practice for documentation for std lib > > > On 2013-09-22, at 21:24 , Westley Mart?nez wrote: > > >> From: gvanrossum at gmail.com [mailto:gvanrossum at gmail.com] On Behalf Of Guido > >> van Rossum > >> Sent: Sunday, September 22, 2013 11:35 AM > >> > >> You seem to misunderstand the use of "autogeneration". It refers to > generating > >> the .rst docs from the docstrings in the source. And FWIW, I'm against that > >> practice. > > > > Oh I see. Well in that case, the docstrings can still become outdated, > > and so then the .rst docs will be outdated, too. > > The points here are that there's a single source of truth (so we can't > have conflicting docstring and rst documentation), and documentation > becoming outdated can be noticed from both docstring and published > documentation. > > > It doesn't seem to > > offer much benefit since you still have to keep both updated, plus you > > have an extra tool that must be maintained. > > There is no extra tool, autodoc is part of the standard Sphinx > distribution: http://sphinx-doc.org/ext/autodoc.html > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python- > dev/anikom15%40gmail.com From guido at python.org Mon Sep 23 00:03:10 2013 From: guido at python.org (Guido van Rossum) Date: Sun, 22 Sep 2013 15:03:10 -0700 Subject: [Python-Dev] Best practice for documentation for std lib In-Reply-To: <CAF-Rda8H2hi5V=HakbGsD2WzH0HQ6hLuxkQYZ3ZC+MzDfb1hVw@mail.gmail.com> References: <20130922031304.GE19939@ando> <CAF-Rda_sTOiKi6c9ihFVL-N90PC-xHxOAMm1im0s-KVyy=L_Ng@mail.gmail.com> <87ioxs3iix.fsf@uwakimon.sk.tsukuba.ac.jp> <CAP7+vJJucubZtXHhnNbAcE0GWfQTnOozHK9W=uv3CWkMX15XGA@mail.gmail.com> <CAF-Rda-qJ3fZQTpVO_x31WhWOaf7Q4+0n7w-sFZgK8tXWLL0zw@mail.gmail.com> <CAP7+vJLmLZOYmMeoKqoqrAGqVF=Z41iYBRiis5buAMyaON+_+A@mail.gmail.com> <CAF-Rda8H2hi5V=HakbGsD2WzH0HQ6hLuxkQYZ3ZC+MzDfb1hVw@mail.gmail.com> Message-ID: <CAP7+vJ+oVnq07Lfh_R1nqaqtK-EnxDaqPeZVTQMQTzfcEdqagg@mail.gmail.com> On Sun, Sep 22, 2013 at 11:44 AM, Eli Bendersky <eliben at gmail.com> wrote: > > On Sun, Sep 22, 2013 at 11:40 AM, Guido van Rossum <guido at python.org>wrote: > >> On Sun, Sep 22, 2013 at 10:25 AM, Eli Bendersky <eliben at gmail.com> wrote: >> >>> >>> I think there's a general agreement in this thread that we don't intend >>> to change the status quo. Both .rst docs and docstrings are important. The >>> remaining question is - can we use some tool to generates parts of the >>> former from the latter and thus avoid duplication and rot? >>> >> >> I don't think that duplication is much of an issue. Natural language >> understanding is not at the level yet where you can generate a meaningful >> summary from a longer text fully automatically (let alone vice versa :-) so >> I think having to write both a concise docstring and a longer more detailed >> description for the Doc tree is not a waste of effort at all. >> > > I don't think the proposal is to generate summaries from a longer text. > I know, I was being facetious. :-) > AFAIU, the proposal is to embed parts of the concise docstring into the > more verbose .rst documentation. > I still think that's a bad idea. Someone editing the docstring may introduce a terminology change or some other style/grammar/flow change that makes perfectly sense by itself but doesn't when taken in the context of the larger .rst doc (e.g. it could introducing duplication or cause dissonance between the two parts). Also, someone who wants to improve the docs would have to figure out how to edit the source code if they wanted to change some part of the docs. > I write .rst docs quite a lot, and as such I do notice the annoying amount > of duplication between docstrings and .rst docs. Without doubt, all the > free-flow text and examples in .rst have to be written manually and are > very important. But for dry method references, it's certainly interesting > to explore the idea of writing them once in the code and then having Sphinx > automatically insert the relevant parts into the .rst before generating > HTML from it. For end users (those who read the docs online) the result is > indistinguishable from what we have now. For us devs it means writing the > same text only once and maintaining it in a single place. > You seem to have caught the DRY bug. But it doesn't always make sense to factor things so that everything is said in exactly one place. (For an example of abstraction gone wild, ironically, see docutils and sphinx. I challenge you to find out the actual code that translates e.g. :meth:`foobar` into <a href="(what goes here?)">foobar</a>. :-) > Think about the new statistics module as a guinea pig. Steven will have a > whole lot of copy-pasting to do :-) > The docstrings should probably be limited to the amount of text that's currently devoted to each function in the PEP. What's currently in the docstrings should go into the .rst docs. > > As for rot, it's just as likely that rot occurs as a *result* of > autogeneration. > > Having to edit/patch the source code in order to improve the > > documentation most likely adds an extra barrier towards improving the > docs. > > This is a valid concern, but perhaps one that can be addressed separately? > (i.e. lowering that barrier of entry). > I can't see that there's any way to interpret the change you propose (changing things so that in order to improve the HTML docs you might need to edit both .rst and .py files) as lowering the barrier to entry. Please give it up. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130922/5f9f495d/attachment-0001.html> From guido at python.org Mon Sep 23 01:19:21 2013 From: guido at python.org (Guido van Rossum) Date: Sun, 22 Sep 2013 16:19:21 -0700 Subject: [Python-Dev] Best practice for documentation for std lib In-Reply-To: <06E35F19-F893-415B-B127-77D1588A565A@masklinn.net> References: <20130922031304.GE19939@ando> <20130922102046.764abea5@fsol> <20130922125323.GI19939@ando> <CAP1=2W7ymWcx_Yj6Xsb0Ty1Wj5pUf+_1XtqhPAVyRKWgiQ37mg@mail.gmail.com> <003301ceb7bd$e1a59430$a4f0bc90$@gmail.com> <CAP7+vJLKVNWEDT62U4pJ7FGARthhVqb74hnLWznBag353s-Obw@mail.gmail.com> <005001ceb7c9$4dc308e0$e9491aa0$@gmail.com> <06E35F19-F893-415B-B127-77D1588A565A@masklinn.net> Message-ID: <CAP7+vJLRn-HAhNwBGLOmdPZA7YBq5JCs=H19FVeard6Y8yqYNQ@mail.gmail.com> On Sun, Sep 22, 2013 at 2:41 PM, Xavier Morel <python-dev at masklinn.net>wrote: > The points here are that there's a single source of truth (so we can't > have conflicting docstring and rst documentation), and documentation > becoming outdated can be noticed from both docstring and published > documentation. > Another case of DRY madness. It seems too many programmers see documentation as unpleasant red tape they want to cut through as quickly as possible, instead of an opportunity to communicate with their *users*. To the contrary: users should be the most important people in the world if you're writing code that's worth documenting at all. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130922/e5b2ccc3/attachment.html> From steve at pearwood.info Mon Sep 23 02:07:44 2013 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 23 Sep 2013 10:07:44 +1000 Subject: [Python-Dev] Best practice for documentation for std lib In-Reply-To: <CAF-Rda_sTOiKi6c9ihFVL-N90PC-xHxOAMm1im0s-KVyy=L_Ng@mail.gmail.com> References: <20130922031304.GE19939@ando> <CAF-Rda_sTOiKi6c9ihFVL-N90PC-xHxOAMm1im0s-KVyy=L_Ng@mail.gmail.com> Message-ID: <20130923000744.GM19939@ando> On Sun, Sep 22, 2013 at 05:54:57AM -0700, Eli Bendersky wrote: > IMHO the right way to think about it is that the .rst files are by far the > more important documentation. Sometimes we forget that most Python > programmers are people who won't go into the source to read docstrings. Docstrings are never more than one command away in the interactive interpreter: help(some.function) If you're going to the source to read docstrings, you're doing it wrong :-) I always go for interactive help first, and the web docs second. The only reason I go to the source is to read the source *code*, not the docstrings. -- Steven From guido at python.org Mon Sep 23 02:30:43 2013 From: guido at python.org (Guido van Rossum) Date: Sun, 22 Sep 2013 17:30:43 -0700 Subject: [Python-Dev] Best practice for documentation for std lib In-Reply-To: <20130923000744.GM19939@ando> References: <20130922031304.GE19939@ando> <CAF-Rda_sTOiKi6c9ihFVL-N90PC-xHxOAMm1im0s-KVyy=L_Ng@mail.gmail.com> <20130923000744.GM19939@ando> Message-ID: <CAP7+vJJAfVCT5TPnaPTZTGTFKWqpay0_41H1UpB8v3F8=G6W7w@mail.gmail.com> On Sun, Sep 22, 2013 at 5:07 PM, Steven D'Aprano <steve at pearwood.info>wrote: > If you're going to the source to read docstrings, you're doing it wrong > :-) > Be that as it may, when I'm reading the source I'm grateful for docstrings. *And* I'm grateful when they are short. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130922/cce93e8f/attachment.html> From steve at pearwood.info Mon Sep 23 02:31:42 2013 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 23 Sep 2013 10:31:42 +1000 Subject: [Python-Dev] Best practice for documentation for std lib In-Reply-To: <CAP7+vJJucubZtXHhnNbAcE0GWfQTnOozHK9W=uv3CWkMX15XGA@mail.gmail.com> References: <20130922031304.GE19939@ando> <CAF-Rda_sTOiKi6c9ihFVL-N90PC-xHxOAMm1im0s-KVyy=L_Ng@mail.gmail.com> <87ioxs3iix.fsf@uwakimon.sk.tsukuba.ac.jp> <CAP7+vJJucubZtXHhnNbAcE0GWfQTnOozHK9W=uv3CWkMX15XGA@mail.gmail.com> Message-ID: <20130923003142.GO19939@ando> On Sun, Sep 22, 2013 at 10:07:28AM -0700, Guido van Rossum wrote: > Authors writing 3rd party packages can do what they want. > > But for the stdlib it's been settled for ages: docstrings should be concise > (but not cryptic(*)), longer documentation go into the separately written > text for docs.python.org. It is the second half that I'm not sure about. How long is *too long* for a doc string? My own preference is to err on the side of too much rather than too little, since I live by help() and only fall back on the web documentation when I really must. Rather than keep talking in generalities, I'll come up with a first draft rst document over the next week or so, put it on the tracker, and wait for some concrete feedback on that. What's the policy about linking to external content from the web docs? -- Steven From ethan at stoneleaf.us Mon Sep 23 01:52:36 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Sun, 22 Sep 2013 16:52:36 -0700 Subject: [Python-Dev] Enum Eccentricities Message-ID: <523F82C4.4060202@stoneleaf.us> http://bugs.python.org/issue19011 So, as anyone who has worked with Enum is probably aware by now, Enum's are a strange duck -- you might even call them platypuses. For example, Enum members are instances of the class, but they are defined inside the class structure, and new ones cannot be created afterwards; Enum classes also support iteration and containment checks. What I'm looking for feedback on is the question is #19011: should an Enum member be considered a class level attribute? On the one hand, they are defined inside the class, and they are accessed via dot notation (EnumClass.member). On the other hand, inside the class they look like 3 and 5 and 'up' and 'east', and they don't live in the class dictionary. Also, if we change Enum so that members do act more like class attributes, then things like Color.red.blue.green.blue will result in Color.blue, and that seems stranger to me than having class instances be available on the class without be full-fledged class-attributes. Thoughts? Opinions? Pearls of wisdom? -- ~Ethan~ From ethan at stoneleaf.us Mon Sep 23 03:16:58 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Sun, 22 Sep 2013 18:16:58 -0700 Subject: [Python-Dev] PEP 455: TransformDict In-Reply-To: <CAP7+vJJzzCOLz0WCX23=aZFDP4-_OWum1weV_-KjZs3=2CMC5A@mail.gmail.com> References: <20130913204058.518f43d3@fsol> <CAP7+vJJzzCOLz0WCX23=aZFDP4-_OWum1weV_-KjZs3=2CMC5A@mail.gmail.com> Message-ID: <523F968A.6090907@stoneleaf.us> On 09/14/2013 12:31 PM, Guido van Rossum wrote: > On Fri, Sep 13, 2013 at 11:40 AM, Antoine Pitrou wrote: >> >> Following the python-dev discussion, I've written a PEP to recap the >> proposal and the various arguments. It's inlined below, and it will >> probably appear soon at http://www.python.org/dev/peps/pep-0455/, too. > > Thanks, Antoine! > > Raymond Hettinger has volunteered to be the PEP dictator (is that the word we use?) for this PEP. Are we close to asking for pronouncement? I haven't seen any chatter for a few days. -- ~Ethan~ From guido at python.org Mon Sep 23 04:04:04 2013 From: guido at python.org (Guido van Rossum) Date: Sun, 22 Sep 2013 19:04:04 -0700 Subject: [Python-Dev] Best practice for documentation for std lib In-Reply-To: <20130923003142.GO19939@ando> References: <20130922031304.GE19939@ando> <CAF-Rda_sTOiKi6c9ihFVL-N90PC-xHxOAMm1im0s-KVyy=L_Ng@mail.gmail.com> <87ioxs3iix.fsf@uwakimon.sk.tsukuba.ac.jp> <CAP7+vJJucubZtXHhnNbAcE0GWfQTnOozHK9W=uv3CWkMX15XGA@mail.gmail.com> <20130923003142.GO19939@ando> Message-ID: <CAP7+vJLThCjrCGGaJLMNd2Uza6wanODnipjmZ3LBZpTkub=uCg@mail.gmail.com> On Sun, Sep 22, 2013 at 5:31 PM, Steven D'Aprano <steve at pearwood.info>wrote: > On Sun, Sep 22, 2013 at 10:07:28AM -0700, Guido van Rossum wrote: > > > Authors writing 3rd party packages can do what they want. > > > > But for the stdlib it's been settled for ages: docstrings should be > concise > > (but not cryptic(*)), longer documentation go into the separately written > > text for docs.python.org. > > It is the second half that I'm not sure about. How long is *too long* > for a doc string? My own preference is to err on the side of too much > rather than too little, since I live by help() and only fall back on the > web documentation when I really must. > I guess preferences differ. I like reading the source, and often I find overly long docstrings distracting. If I had the final word, I'd recommend using the current docstrings as the .rst contents and replacing the docstrings with the 1-2 line function descriptions from the PEP, e.g.: * median(data) -> median (middle value) of data, taking the average of the two middle values when there are an even number of values. But omitting the signature proper, and replacing "->" with "Returns" or "Returns the". E.g. def median(data): """Returns the median (middle value) of data, taking the average of the two middle values when there are an even number of values. """ ... I'd personally also rewrite them all so that the first line of the docstring is a phrase that can stand by itself, as I describe in PEP 8, but this is used only sporadically in the stdlib. > Rather than keep talking in generalities, I'll come up with a first > draft rst document over the next week or so, put it on the tracker, and > wait for some concrete feedback on that. > > What's the policy about linking to external content from the web docs? > If it's for background information or an introduction to the theory that's fine. If it's as a substitute for proper documentation of an API I'd frown upon it. Someone in a spaceship on its way to Mars with a copy of docs.python.org should have no problems using the library, as long as they are familiar with the basic theory such as might be explained on Wikipedia (of which the spaceship would of course also have a copy :-). -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130922/79218312/attachment.html> From tjreedy at udel.edu Mon Sep 23 04:25:57 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 22 Sep 2013 22:25:57 -0400 Subject: [Python-Dev] Best practice for documentation for std lib In-Reply-To: <CAP7+vJLThCjrCGGaJLMNd2Uza6wanODnipjmZ3LBZpTkub=uCg@mail.gmail.com> References: <20130922031304.GE19939@ando> <CAF-Rda_sTOiKi6c9ihFVL-N90PC-xHxOAMm1im0s-KVyy=L_Ng@mail.gmail.com> <87ioxs3iix.fsf@uwakimon.sk.tsukuba.ac.jp> <CAP7+vJJucubZtXHhnNbAcE0GWfQTnOozHK9W=uv3CWkMX15XGA@mail.gmail.com> <20130923003142.GO19939@ando> <CAP7+vJLThCjrCGGaJLMNd2Uza6wanODnipjmZ3LBZpTkub=uCg@mail.gmail.com> Message-ID: <l1o8ri$mjv$1@ger.gmane.org> On 9/22/2013 10:04 PM, Guido van Rossum wrote: > If I had the final word, I'd recommend using the current docstrings as > the .rst contents and replacing the docstrings with the 1-2 line > function descriptions from the PEP, e.g.: > > * median(data) -> median (middle value) of data, taking the > average of the two middle values when there are an even > number of values. > > But omitting the signature proper, and replacing "->" with "Returns" or > "Returns the". E.g. > > def median(data): > """Returns the median (middle value) of data, taking the > average of the two middle values when there are an even > number of values. > """ > ... > > I'd personally also rewrite them all so that the first line of the > docstring is a phrase that can stand by itself, as I describe in PEP 8, > but this is used only sporadically in the stdlib. I am gradually changing Idle docstrings, though only Idle developers will ever see them. Writing a 60 char summary forces a clear understanding of the function. Applied to the above. def median(data): """Return the median (middle value) of data. Use the average of the two middle values when there are an even number of values. """ ('Return' rather than 'Returns' is the current convention.) -- Terry Jan Reedy From guido at python.org Mon Sep 23 04:37:49 2013 From: guido at python.org (Guido van Rossum) Date: Sun, 22 Sep 2013 19:37:49 -0700 Subject: [Python-Dev] Enum Eccentricities In-Reply-To: <523F82C4.4060202@stoneleaf.us> References: <523F82C4.4060202@stoneleaf.us> Message-ID: <CAP7+vJLRtx3nQyvxN68M8dAPoe4evKOVXyQ19WEgp5Cd8EHFTw@mail.gmail.com> On Sun, Sep 22, 2013 at 4:52 PM, Ethan Furman <ethan at stoneleaf.us> wrote: > http://bugs.python.org/**issue19011 <http://bugs.python.org/issue19011> > > So, as anyone who has worked with Enum is probably aware by now, Enum's > are a strange duck -- you might even call them platypuses. > > For example, Enum members are instances of the class, but they are defined > inside the class structure, and new ones cannot be created afterwards; Enum > classes also support iteration and containment checks. > > What I'm looking for feedback on is the question is #19011: should an Enum > member be considered a class level attribute? > > On the one hand, they are defined inside the class, and they are accessed > via dot notation (EnumClass.member). > > On the other hand, inside the class they look like 3 and 5 and 'up' and > 'east', and they don't live in the class dictionary. > > Also, if we change Enum so that members do act more like class attributes, > then things like Color.red.blue.green.blue will result in Color.blue, and > that seems stranger to me than having class instances be available on the > class without be full-fledged class-attributes. > > Thoughts? Opinions? Pearls of wisdom? > I wouldn't lose much sleep over this. Classes can override __getattribute__ so that instance variables appear to exist even though they are not in the instance's __dict__. It's the same for metaclasses. As for attributes appearing different inside the class than when accessed as an attribute, that's not unusual -- descriptors can legitimately do that. It was more common in Python 2, where this applied to all unbound methods, but even in Python 3 it applies to static and class methods. I would draw the line at being able to access members as attributes of other members. Making Color.red.blue be a spelling for Color.blue feels like an abomination. If you can make dir(Color) return the strings 'red', 'blue' etc. in addition to other class attributes without making Color.red.blue work, go for it. If you can't, that's fine too. Users should (obviously) steer clear from relying on either behavior. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130922/80bf98a6/attachment.html> From guido at python.org Mon Sep 23 04:44:09 2013 From: guido at python.org (Guido van Rossum) Date: Sun, 22 Sep 2013 19:44:09 -0700 Subject: [Python-Dev] Best practice for documentation for std lib In-Reply-To: <l1o8ri$mjv$1@ger.gmane.org> References: <20130922031304.GE19939@ando> <CAF-Rda_sTOiKi6c9ihFVL-N90PC-xHxOAMm1im0s-KVyy=L_Ng@mail.gmail.com> <87ioxs3iix.fsf@uwakimon.sk.tsukuba.ac.jp> <CAP7+vJJucubZtXHhnNbAcE0GWfQTnOozHK9W=uv3CWkMX15XGA@mail.gmail.com> <20130923003142.GO19939@ando> <CAP7+vJLThCjrCGGaJLMNd2Uza6wanODnipjmZ3LBZpTkub=uCg@mail.gmail.com> <l1o8ri$mjv$1@ger.gmane.org> Message-ID: <CAP7+vJJSvX78f2+i7_C5mNMdOQpfSD_oKqC01Z8CEHaU2krqTw@mail.gmail.com> On Sun, Sep 22, 2013 at 7:25 PM, Terry Reedy <tjreedy at udel.edu> wrote: > On 9/22/2013 10:04 PM, Guido van Rossum wrote: > > If I had the final word, I'd recommend using the current docstrings as >> the .rst contents and replacing the docstrings with the 1-2 line >> function descriptions from the PEP, e.g.: >> >> * median(data) -> median (middle value) of data, taking the >> average of the two middle values when there are an even >> number of values. >> >> But omitting the signature proper, and replacing "->" with "Returns" or >> "Returns the". E.g. >> >> def median(data): >> """Returns the median (middle value) of data, taking the >> average of the two middle values when there are an even >> number of values. >> """ >> ... >> >> I'd personally also rewrite them all so that the first line of the >> docstring is a phrase that can stand by itself, as I describe in PEP 8, >> but this is used only sporadically in the stdlib. >> > > I am gradually changing Idle docstrings, though only Idle developers will > ever see them. Writing a 60 char summary forces a clear understanding of > the function. Applied to the above. > > def median(data): > """Return the median (middle value) of data. > > Use the average of the two middle values when > > there are an even number of values. > """ > Glad you like it. I still do, too, but I've given up hope to convince all core developers to stick to this style. :-( > ('Return' rather than 'Returns' is the current convention.) > That's actually a religious argument which in the stdlib takes no strict position -- a quick grep shows that both are used, although 'Return' is more frequent by a 5-to-1 margin. IIRC in the Java world you *have* to use 'Returns', but I don't buy the argument from nit-picky grammarians that leads to this rule. (It's something about the documentation not being a command. But English is more flexible than that.) -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130922/8eb6438d/attachment-0001.html> From greg.ewing at canterbury.ac.nz Mon Sep 23 02:53:38 2013 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 23 Sep 2013 12:53:38 +1200 Subject: [Python-Dev] Enum Eccentricities In-Reply-To: <523F82C4.4060202@stoneleaf.us> References: <523F82C4.4060202@stoneleaf.us> Message-ID: <523F9112.8050506@canterbury.ac.nz> Ethan Furman wrote: > Also, if we change Enum so that members do act more like class > attributes, then things like Color.red.blue.green.blue will result in > Color.blue, I thought we already decided it was worth making that not happen? -- Greg From ncoghlan at gmail.com Mon Sep 23 05:24:56 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 23 Sep 2013 13:24:56 +1000 Subject: [Python-Dev] Revert #12085 fix for __del__ attribute error message In-Reply-To: <l1nkdh$lh7$1@ger.gmane.org> References: <l1l2bm$1bp$1@ger.gmane.org> <20130921221524.55D4225007F@webabinitio.net> <l1lbkl$k0g$1@ger.gmane.org> <CAP7+vJLHO12ZNwX4UZ0c_BF9pr+xxxLZfy2dGjBcMVenGow9vQ@mail.gmail.com> <l1n9o6$ap3$1@ger.gmane.org> <CAP7+vJL+0XEBpOnVbTpZJNLzahgm8EkyuXdGdhE27TVwHu4GAg@mail.gmail.com> <l1nkdh$lh7$1@ger.gmane.org> Message-ID: <CADiSq7eGpFtPbRc8BmiP2H9V2PVMma3ycX3Ky_MJ-F+TpEiU6w@mail.gmail.com> On 23 Sep 2013 06:38, "Terry Reedy" <tjreedy at udel.edu> wrote: > > On 9/22/2013 2:41 PM, Guido van Rossum wrote: >> >> On Sun, Sep 22, 2013 at 10:35 AM, Terry Reedy <tjreedy at udel.edu >> <mailto:tjreedy at udel.edu>> wrote: >> >> On 9/21/2013 10:30 PM, Guido van Rossum wrote: >> >> Exceptions in __del__ point to bugs (sometimes in the stdlib) that >> should be fixed, period. The only reason they do not result in >> exceptions that are properly bubbled up and catchable is because >> __del__ >> is called from a DECREF macro which has no return value. >> >> >> That is clear enough. What fooled me is the word 'ignored', in both >> the doc and message. How about 'skipped' (for technical reasons)? >> >> >> That's a good point, although I'm not sure 'skipped' is better. > > > Only slightly ;-). The problem with both words is that they try to say two things. What happened, and what Python did about it. > > >> Maybe use a more neutral verb like 'occurred'? > > > "Exception occurred in ..." is even better at say what happened. > > I think we should then add an explict statement as to what Python did, and hint at what the user should do, something like > "Although caught internally, it still indicates a problem." Brevity is still a virtue. The relevant C API function is called "PyErr_WriteUnraisable", so just starting the message as something like "Unraisable exception suppressed in..." might work. Cheers, Nick. > > Otherwise, when no other output follows, as in > ... > > del c > Exception ocurred in: <bound method C.__del__ ... > > Traceback (most recent call last): > File "<pyshell#0>", line 2, in __del__ > AttributeError: > >>> > It may not be completely obvious to a non-expert that the traceback is not a 'real' traceback from an exception that was allowed to propagate, and that it did not stop execution and cannot be caught. > > > -- > Terry Jan Reedy > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130923/651f5019/attachment.html> From z at etiol.net Mon Sep 23 04:41:39 2013 From: z at etiol.net (Zero Piraeus) Date: Sun, 22 Sep 2013 23:41:39 -0300 Subject: [Python-Dev] Enum Eccentricities In-Reply-To: <523F82C4.4060202@stoneleaf.us> References: <523F82C4.4060202@stoneleaf.us> Message-ID: <20130923024139.GA26561@piedra> : On Sun, Sep 22, 2013 at 04:52:36PM -0700, Ethan Furman wrote: > So, as anyone who has worked with Enum is probably aware by now, > Enum's are a strange duck -- you might even call them platypuses. Yes :-) > What I'm looking for feedback on is the question is #19011: should an > Enum member be considered a class level attribute? I may be misunderstanding the use case given in the issue, but it seems to me that having to use Color.red.__class__.blue (what is being complained about in the issue), while not exactly pretty, makes a lot more semantic sense than Color.red.blue ... which is just bizarre. Enum members aren't class attributes, even if the way they're defined makes them look as though they are. Allowing this is just asking for more confusion on the part of anyone using them IMHO. -[]z. -- Zero Piraeus: omnia omnibus http://etiol.net/pubkey.asc From guido at python.org Mon Sep 23 06:29:13 2013 From: guido at python.org (Guido van Rossum) Date: Sun, 22 Sep 2013 21:29:13 -0700 Subject: [Python-Dev] Revert #12085 fix for __del__ attribute error message In-Reply-To: <CADiSq7eGpFtPbRc8BmiP2H9V2PVMma3ycX3Ky_MJ-F+TpEiU6w@mail.gmail.com> References: <l1l2bm$1bp$1@ger.gmane.org> <20130921221524.55D4225007F@webabinitio.net> <l1lbkl$k0g$1@ger.gmane.org> <CAP7+vJLHO12ZNwX4UZ0c_BF9pr+xxxLZfy2dGjBcMVenGow9vQ@mail.gmail.com> <l1n9o6$ap3$1@ger.gmane.org> <CAP7+vJL+0XEBpOnVbTpZJNLzahgm8EkyuXdGdhE27TVwHu4GAg@mail.gmail.com> <l1nkdh$lh7$1@ger.gmane.org> <CADiSq7eGpFtPbRc8BmiP2H9V2PVMma3ycX3Ky_MJ-F+TpEiU6w@mail.gmail.com> Message-ID: <CAP7+vJKAP5R+2R++z8+DQjSW6EVycoczB2gWeo759-NG4G6Veg@mail.gmail.com> On Sunday, September 22, 2013, Nick Coghlan wrote: > > Brevity is still a virtue. The relevant C API function is called > "PyErr_WriteUnraisable", so just starting the message as something like > "Unraisable exception suppressed in..." might work. > Somehow "unraisable" sounds too technical, and "suppressed" is hardly right given that it is printed... Let's leave well enough alone. --Guido -- --Guido van Rossum (on iPad) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130922/270f4e67/attachment.html> From ethan at stoneleaf.us Mon Sep 23 06:06:40 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Sun, 22 Sep 2013 21:06:40 -0700 Subject: [Python-Dev] Revert #12085 fix for __del__ attribute error message In-Reply-To: <CADiSq7eGpFtPbRc8BmiP2H9V2PVMma3ycX3Ky_MJ-F+TpEiU6w@mail.gmail.com> References: <l1l2bm$1bp$1@ger.gmane.org> <20130921221524.55D4225007F@webabinitio.net> <l1lbkl$k0g$1@ger.gmane.org> <CAP7+vJLHO12ZNwX4UZ0c_BF9pr+xxxLZfy2dGjBcMVenGow9vQ@mail.gmail.com> <l1n9o6$ap3$1@ger.gmane.org> <CAP7+vJL+0XEBpOnVbTpZJNLzahgm8EkyuXdGdhE27TVwHu4GAg@mail.gmail.com> <l1nkdh$lh7$1@ger.gmane.org> <CADiSq7eGpFtPbRc8BmiP2H9V2PVMma3ycX3Ky_MJ-F+TpEiU6w@mail.gmail.com> Message-ID: <523FBE50.4060404@stoneleaf.us> On 09/22/2013 08:24 PM, Nick Coghlan wrote: > > On 23 Sep 2013 06:38, "Terry Reedy" <tjreedy at udel.edu <mailto:tjreedy at udel.edu>> wrote: >> >> On 9/22/2013 2:41 PM, Guido van Rossum wrote: >>> >>> On Sun, Sep 22, 2013 at 10:35 AM, Terry Reedy <tjreedy at udel.edu <mailto:tjreedy at udel.edu> >>> <mailto:tjreedy at udel.edu <mailto:tjreedy at udel.edu>>> wrote: >>> >>> On 9/21/2013 10:30 PM, Guido van Rossum wrote: >>> >>> Exceptions in __del__ point to bugs (sometimes in the stdlib) that >>> should be fixed, period. The only reason they do not result in >>> exceptions that are properly bubbled up and catchable is because >>> __del__ >>> is called from a DECREF macro which has no return value. >>> >>> >>> That is clear enough. What fooled me is the word 'ignored', in both >>> the doc and message. How about 'skipped' (for technical reasons)? >>> >>> >>> That's a good point, although I'm not sure 'skipped' is better. >> >> >> Only slightly ;-). The problem with both words is that they try to say two things. What happened, and what Python did about it. >> >> >>> Maybe use a more neutral verb like 'occurred'? >> >> >> "Exception occurred in ..." is even better at say what happened. >> >> I think we should then add an explict statement as to what Python did, and hint at what the user should do, something like >> "Although caught internally, it still indicates a problem." > > Brevity is still a virtue. The relevant C API function is called "PyErr_WriteUnraisable", so just starting the message > as something like "Unraisable exception suppressed in..." might work. I like that! +1 -- ~Ethan~ From guido at python.org Mon Sep 23 06:25:23 2013 From: guido at python.org (Guido van Rossum) Date: Sun, 22 Sep 2013 21:25:23 -0700 Subject: [Python-Dev] Enum Eccentricities In-Reply-To: <20130923024139.GA26561@piedra> References: <523F82C4.4060202@stoneleaf.us> <20130923024139.GA26561@piedra> Message-ID: <CAP7+vJJgbzS6jG-yzFKEUNiEHnTzH_7yotv9sCXh-M=hZ85vDg@mail.gmail.com> On Sunday, September 22, 2013, Zero Piraeus wrote: > I may be misunderstanding the use case given in the issue, but it seems > to me that having to use > > Color.red.__class__.blue > > (what is being complained about in the issue), while not exactly pretty, > makes a lot more semantic sense than > > Color.red.blue > > ... which is just bizarre. Right. > Enum members aren't class attributes, even if the way they're defined > makes them look as though they are. Allowing this is just asking for > more confusion on the part of anyone using them IMHO. > Depends on how you define "class sttribute". --Guido -- --Guido van Rossum (on iPad) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130922/2f06f791/attachment.html> From v+python at g.nevcal.com Mon Sep 23 07:28:05 2013 From: v+python at g.nevcal.com (Glenn Linderman) Date: Sun, 22 Sep 2013 22:28:05 -0700 Subject: [Python-Dev] Revert #12085 fix for __del__ attribute error message In-Reply-To: <CAP7+vJKAP5R+2R++z8+DQjSW6EVycoczB2gWeo759-NG4G6Veg@mail.gmail.com> References: <l1l2bm$1bp$1@ger.gmane.org> <20130921221524.55D4225007F@webabinitio.net> <l1lbkl$k0g$1@ger.gmane.org> <CAP7+vJLHO12ZNwX4UZ0c_BF9pr+xxxLZfy2dGjBcMVenGow9vQ@mail.gmail.com> <l1n9o6$ap3$1@ger.gmane.org> <CAP7+vJL+0XEBpOnVbTpZJNLzahgm8EkyuXdGdhE27TVwHu4GAg@mail.gmail.com> <l1nkdh$lh7$1@ger.gmane.org> <CADiSq7eGpFtPbRc8BmiP2H9V2PVMma3ycX3Ky_MJ-F+TpEiU6w@mail.gmail.com> <CAP7+vJKAP5R+2R++z8+DQjSW6EVycoczB2gWeo759-NG4G6Veg@mail.gmail.com> Message-ID: <523FD165.1000300@g.nevcal.com> On 9/22/2013 9:29 PM, Guido van Rossum wrote: > On Sunday, September 22, 2013, Nick Coghlan wrote: > > Brevity is still a virtue. The relevant C API function is called > "PyErr_WriteUnraisable", so just starting the message as something > like "Unraisable exception suppressed in..." might work. > > > Somehow "unraisable" sounds too technical, and "suppressed" is hardly > right given that it is printed... > > Let's leave well enough alone. "printable-only exception in..." -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130922/04ed4af6/attachment.html> From turnbull at sk.tsukuba.ac.jp Mon Sep 23 07:34:54 2013 From: turnbull at sk.tsukuba.ac.jp (Stephen J. Turnbull) Date: Mon, 23 Sep 2013 14:34:54 +0900 Subject: [Python-Dev] Best practice for documentation for std lib In-Reply-To: <CAP7+vJLThCjrCGGaJLMNd2Uza6wanODnipjmZ3LBZpTkub=uCg@mail.gmail.com> References: <20130922031304.GE19939@ando> <CAF-Rda_sTOiKi6c9ihFVL-N90PC-xHxOAMm1im0s-KVyy=L_Ng@mail.gmail.com> <87ioxs3iix.fsf@uwakimon.sk.tsukuba.ac.jp> <CAP7+vJJucubZtXHhnNbAcE0GWfQTnOozHK9W=uv3CWkMX15XGA@mail.gmail.com> <20130923003142.GO19939@ando> <CAP7+vJLThCjrCGGaJLMNd2Uza6wanODnipjmZ3LBZpTkub=uCg@mail.gmail.com> Message-ID: <87d2o02j9d.fsf@uwakimon.sk.tsukuba.ac.jp> Guido van Rossum writes: > On Sun, Sep 22, 2013 at 5:31 PM, Steven D'Aprano <steve at pearwood.info> wrote: >> My own preference is to err on the side of too much rather than >> too little, since I live by help() and only fall back on the web >> documentation when I really must. > I guess preferences differ. Indeed. Abstractly I agree with Steven: there are some modules such as statistics where it would be nice for users to have long docstrings that might even be the full source for the web docs if the module author so prefers. But given that Python has an existing policy of concise docstrings (which indeed is useful when browsing code) and a separate manual, I withdraw my support for changing practice. (Sorry, Steven.) From greg.ewing at canterbury.ac.nz Mon Sep 23 08:11:27 2013 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 23 Sep 2013 18:11:27 +1200 Subject: [Python-Dev] Revert #12085 fix for __del__ attribute error message In-Reply-To: <CAP7+vJKAP5R+2R++z8+DQjSW6EVycoczB2gWeo759-NG4G6Veg@mail.gmail.com> References: <l1l2bm$1bp$1@ger.gmane.org> <20130921221524.55D4225007F@webabinitio.net> <l1lbkl$k0g$1@ger.gmane.org> <CAP7+vJLHO12ZNwX4UZ0c_BF9pr+xxxLZfy2dGjBcMVenGow9vQ@mail.gmail.com> <l1n9o6$ap3$1@ger.gmane.org> <CAP7+vJL+0XEBpOnVbTpZJNLzahgm8EkyuXdGdhE27TVwHu4GAg@mail.gmail.com> <l1nkdh$lh7$1@ger.gmane.org> <CADiSq7eGpFtPbRc8BmiP2H9V2PVMma3ycX3Ky_MJ-F+TpEiU6w@mail.gmail.com> <CAP7+vJKAP5R+2R++z8+DQjSW6EVycoczB2gWeo759-NG4G6Veg@mail.gmail.com> Message-ID: <523FDB8F.1080207@canterbury.ac.nz> Guido van Rossum wrote: > Somehow "unraisable" sounds too technical, It's not even really accurate. It's been raised, it just can't be propagated any further. But "unpropagatable exception" would be a bit of a mouthful. -- Greg From larry at hastings.org Mon Sep 23 09:57:01 2013 From: larry at hastings.org (Larry Hastings) Date: Mon, 23 Sep 2013 08:57:01 +0100 Subject: [Python-Dev] Best practice for documentation for std lib In-Reply-To: <CAP7+vJJSvX78f2+i7_C5mNMdOQpfSD_oKqC01Z8CEHaU2krqTw@mail.gmail.com> References: <20130922031304.GE19939@ando> <CAF-Rda_sTOiKi6c9ihFVL-N90PC-xHxOAMm1im0s-KVyy=L_Ng@mail.gmail.com> <87ioxs3iix.fsf@uwakimon.sk.tsukuba.ac.jp> <CAP7+vJJucubZtXHhnNbAcE0GWfQTnOozHK9W=uv3CWkMX15XGA@mail.gmail.com> <20130923003142.GO19939@ando> <CAP7+vJLThCjrCGGaJLMNd2Uza6wanODnipjmZ3LBZpTkub=uCg@mail.gmail.com> <l1o8ri$mjv$1@ger.gmane.org> <CAP7+vJJSvX78f2+i7_C5mNMdOQpfSD_oKqC01Z8CEHaU2krqTw@mail.gmail.com> Message-ID: <523FF44D.1000303@hastings.org> On 09/23/2013 03:44 AM, Guido van Rossum wrote: > On Sun, Sep 22, 2013 at 7:25 PM, Terry Reedy <tjreedy at udel.edu > <mailto:tjreedy at udel.edu>> wrote: > > I am gradually changing Idle docstrings, though only Idle > developers will ever see them. Writing a 60 char summary forces a > clear understanding of the function. > > > Glad you like it. I still do, too, but I've given up hope to convince > all core developers to stick to this style. :-( ... Argument Clinic to the rescue? Since the last time this subject came up, Clinic has started enforcing a summary line in docstrings. I didn't realize it had to be 60 columns though, should I add that? Is the entire docstring supposed to be 60 columns max? p.s. status update: AC is done enough to be worth considering checking in. Now I have to finish the PEP and write the documentation. //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130923/772a7caa/attachment.html> From hrvoje.niksic at avl.com Mon Sep 23 10:03:12 2013 From: hrvoje.niksic at avl.com (Hrvoje Niksic) Date: Mon, 23 Sep 2013 10:03:12 +0200 Subject: [Python-Dev] sys.intern should work on bytes In-Reply-To: <CALeMXf7Qo94xYsP_8cufhp=OR0x4QyWu-gnzn=fq1k7NF2QEuw@mail.gmail.com> References: <523C2F3C.7020605@jcea.es> <20130920141551.0ac2c769@pitrou.net> <523C4E91.4050905@jcea.es> <20130920154415.5e6cfa09@pitrou.net> <523C53A8.4070106@jcea.es> <CALeMXf7Qo94xYsP_8cufhp=OR0x4QyWu-gnzn=fq1k7NF2QEuw@mail.gmail.com> Message-ID: <523FF5C0.7080701@avl.com> On 09/20/2013 06:50 PM, PJ Eby wrote: > On Fri, Sep 20, 2013 at 9:54 AM, Jesus Cea <jcea at jcea.es> wrote: >> Why str/bytes doesn't support weakrefs, beside memory use? > > The typical use case for weakrefs is to break reference cycles, Another typical use case, and the prime reason why languages without reference counting tend to introduce weak references, is managing object caches with automatic disposal of otherwise unused items. Such a cache is rarely necessary for primitive objects, so Python's choice to spare memory for weakrefs is quite justified. However, if one wanted to implement their own sys.intern(), inability to refer to strings would become a problem. This is one reason why sys.intern() directly fiddles with reference counts instead of reusing the weakref machinery. (The other of course being that intern predates weakrefs by many years.) From ncoghlan at gmail.com Mon Sep 23 10:17:51 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 23 Sep 2013 18:17:51 +1000 Subject: [Python-Dev] Revert #12085 fix for __del__ attribute error message In-Reply-To: <523FDB8F.1080207@canterbury.ac.nz> References: <l1l2bm$1bp$1@ger.gmane.org> <20130921221524.55D4225007F@webabinitio.net> <l1lbkl$k0g$1@ger.gmane.org> <CAP7+vJLHO12ZNwX4UZ0c_BF9pr+xxxLZfy2dGjBcMVenGow9vQ@mail.gmail.com> <l1n9o6$ap3$1@ger.gmane.org> <CAP7+vJL+0XEBpOnVbTpZJNLzahgm8EkyuXdGdhE27TVwHu4GAg@mail.gmail.com> <l1nkdh$lh7$1@ger.gmane.org> <CADiSq7eGpFtPbRc8BmiP2H9V2PVMma3ycX3Ky_MJ-F+TpEiU6w@mail.gmail.com> <CAP7+vJKAP5R+2R++z8+DQjSW6EVycoczB2gWeo759-NG4G6Veg@mail.gmail.com> <523FDB8F.1080207@canterbury.ac.nz> Message-ID: <CADiSq7caBSr4ujFo0HBYGF91SJp2NcP-_SreBqad-=KpcZdeCw@mail.gmail.com> On 23 September 2013 16:11, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote: > Guido van Rossum wrote: >> >> Somehow "unraisable" sounds too technical, > > > It's not even really accurate. It's been raised, it just > can't be propagated any further. But "unpropagatable > exception" would be a bit of a mouthful. I felt I needed a reminder of the context where this string gets printed, so I created this toy example: >>> class C: ... def __del__(self): ... raise RuntimeError("Going away now") ... >>> x = C() >>> del x Exception ignored in: <bound method C.__del__ of <__main__.C object at 0x7f98b8b61538>> Traceback (most recent call last): File "<stdin>", line 3, in __del__ RuntimeError: Going away now Here's what I suggest changing that error to: >>> del x Unraisable exception suppressed when calling <bound method C.__del__ of <__main__.C object at 0x7f98b8b61538>> Traceback (most recent call last): File "<stdin>", line 3, in __del__ RuntimeError: Going away now First and foremost, "Unraisable exception suppressed" should be quite search engine friendly. The only current hits for that are on archived versions of this thread, whereas the existing "Exception ignored" sends you down a rabbit hole of Stack Overflow answers related to broken pipe handling. If we create a new FAQ entry for "What does 'Exception ignored' or 'Unraisable exception suppressed' mean?" when making the change for 3.4, then that will definitely help 3.4 users and may eventually help users of other versions as well :) Secondly, it concisely explains exactly what is going on, so developers that look it up once should be able to remember the meaning in the future: - the given exception was suppressed (*what* happened) - it was suppressed because Python couldn't raise it any further (*why* it happened) I think it's OK for the terms to be a little technical and arcane because problems relating to broken finalizers and failures during interpreter startup and shutdown *are* a little arcane. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From solipsis at pitrou.net Mon Sep 23 10:45:00 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 23 Sep 2013 10:45:00 +0200 Subject: [Python-Dev] Revert #12085 fix for __del__ attribute error message References: <l1l2bm$1bp$1@ger.gmane.org> <20130921221524.55D4225007F@webabinitio.net> <l1lbkl$k0g$1@ger.gmane.org> <CAP7+vJLHO12ZNwX4UZ0c_BF9pr+xxxLZfy2dGjBcMVenGow9vQ@mail.gmail.com> <l1n9o6$ap3$1@ger.gmane.org> <CAP7+vJL+0XEBpOnVbTpZJNLzahgm8EkyuXdGdhE27TVwHu4GAg@mail.gmail.com> <l1nkdh$lh7$1@ger.gmane.org> <CADiSq7eGpFtPbRc8BmiP2H9V2PVMma3ycX3Ky_MJ-F+TpEiU6w@mail.gmail.com> <CAP7+vJKAP5R+2R++z8+DQjSW6EVycoczB2gWeo759-NG4G6Veg@mail.gmail.com> <523FDB8F.1080207@canterbury.ac.nz> <CADiSq7caBSr4ujFo0HBYGF91SJp2NcP-_SreBqad-=KpcZdeCw@mail.gmail.com> Message-ID: <20130923104500.5645fdbb@pitrou.net> Le Mon, 23 Sep 2013 18:17:51 +1000, Nick Coghlan <ncoghlan at gmail.com> a ?crit : > > Here's what I suggest changing that error to: > > >>> del x > Unraisable exception suppressed when calling <bound method C.__del__ > of <__main__.C object at 0x7f98b8b61538>> > Traceback (most recent call last): > File "<stdin>", line 3, in __del__ > RuntimeError: Going away now Why not simply "Exception automatically caught in <bound method C.__del__> [...]" ? Regards Antoine. From ncoghlan at gmail.com Mon Sep 23 10:51:04 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 23 Sep 2013 18:51:04 +1000 Subject: [Python-Dev] Revert #12085 fix for __del__ attribute error message In-Reply-To: <20130923104500.5645fdbb@pitrou.net> References: <l1l2bm$1bp$1@ger.gmane.org> <20130921221524.55D4225007F@webabinitio.net> <l1lbkl$k0g$1@ger.gmane.org> <CAP7+vJLHO12ZNwX4UZ0c_BF9pr+xxxLZfy2dGjBcMVenGow9vQ@mail.gmail.com> <l1n9o6$ap3$1@ger.gmane.org> <CAP7+vJL+0XEBpOnVbTpZJNLzahgm8EkyuXdGdhE27TVwHu4GAg@mail.gmail.com> <l1nkdh$lh7$1@ger.gmane.org> <CADiSq7eGpFtPbRc8BmiP2H9V2PVMma3ycX3Ky_MJ-F+TpEiU6w@mail.gmail.com> <CAP7+vJKAP5R+2R++z8+DQjSW6EVycoczB2gWeo759-NG4G6Veg@mail.gmail.com> <523FDB8F.1080207@canterbury.ac.nz> <CADiSq7caBSr4ujFo0HBYGF91SJp2NcP-_SreBqad-=KpcZdeCw@mail.gmail.com> <20130923104500.5645fdbb@pitrou.net> Message-ID: <CADiSq7dJAdStAzPYijQTC_fC6_01Jkt4kidSh+YDxWNNv_0tOA@mail.gmail.com> On 23 September 2013 18:45, Antoine Pitrou <solipsis at pitrou.net> wrote: > Le Mon, 23 Sep 2013 18:17:51 +1000, > Nick Coghlan <ncoghlan at gmail.com> a ?crit : >> >> Here's what I suggest changing that error to: >> >> >>> del x >> Unraisable exception suppressed when calling <bound method C.__del__ >> of <__main__.C object at 0x7f98b8b61538>> >> Traceback (most recent call last): >> File "<stdin>", line 3, in __del__ >> RuntimeError: Going away now > > Why not simply "Exception automatically caught in <bound method > C.__del__> [...]" ? It only answers the "what" (i.e. the exception was automatically caught), without addressing the "why" (i.e. because there wasn't anything else useful the interpreter could do with it) "Suppressed" also says more than "caught" does, since "caught and reraised the exception" makes sense, while "suppressed and reraised the exception" contradicts itself. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Mon Sep 23 13:15:09 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 23 Sep 2013 21:15:09 +1000 Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for pronouncement? Message-ID: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com> With the last round of updates, I believe PEP 453 is ready for Martin's pronouncement. HTML: http://www.python.org/dev/peps/pep-0453/ Major diffs: http://hg.python.org/peps/rev/b2993450b32a Many of the updates are just clarifications in response to questions, but the actual significant changes are: - the proposed module name is now "ensurepip" (thanks Antoine for that suggestion) - distros are granted considerable latitude to ensure the distro pip package is used with the system Python installation, so long as they ensure virtual environments continue to work as expected - the two open questions regarding uninstallation and tweaking the directory layout on Windows are decided in favour of the approaches that were already documented in the PEP (since we hadn't received any objections to them in earlier discussions) Regards, Nick. ===================== PEP: 453 Title: Explicit bootstrapping of pip in Python installations Version: $Revision$ Last-Modified: $Date$ Author: Donald Stufft <donald at stufft.io>, Nick Coghlan <ncoghlan at gmail.com> BDFL-Delegate: Martin von L?wis Status: Draft Type: Process Content-Type: text/x-rst Created: 10-Aug-2013 Post-History: 30-Aug-2013, 15-Sep-2013, 18-Sep-2013, 19-Sep-2013, 23-Sep-2013 Abstract ======== This PEP proposes that the `pip`_ package manager be made available by default when installing CPython and when creating virtual environments using the standard library's ``venv`` module via the ``pyvenv`` command line utility). To clearly demarcate development responsibilities, and to avoid inadvertently downgrading ``pip`` when updating CPython, the proposed mechanism to achieve this is to include an explicit `pip`_ bootstrapping mechanism in the standard library that is invoked automatically by the CPython installers provided on python.org. The PEP also strongly recommends that CPython redistributors and other Python implementations ensure that ``pip`` is available by default, or at the very least, explicitly document the fact that it is not included. Proposal ======== This PEP proposes the inclusion of an ``ensurepip`` bootstrapping module in Python 3.4, as well as in the next maintenance releases of Python 3.3 and 2.7. This PEP does *not* propose making pip (or any dependencies) directly available as part of the standard library. Instead, pip will be a bundled application provided along with CPython for the convenience of Python users, but subject to its own development life cycle and able to be upgraded independently of the core interpreter and standard library. Rationale ========= Currently, on systems without a platform package manager and repository, installing a third-party Python package into a freshly installed Python requires first identifying an appropriate package manager and then installing it. Even on systems that *do* have a platform package manager, it is unlikely to include every package that is available on the Python Package Index, and even when a desired third-party package is available, the correct name in the platform package manager may not be clear. This means that, to work effectively with the Python Package Index ecosystem, users must know which package manager to install, where to get it, and how to install it. The effect of this is that third-party Python projects are currently required to choose from a variety of undesirable alternatives: * Assume the user already has a suitable cross-platform package manager installed. * Duplicate the instructions and tell their users how to install the package manager. * Completely forgo the use of dependencies to ease installation concerns for their users. All of these available options have significant drawbacks. If a project simply assumes a user already has the tooling then beginning users may get a confusing error message when the installation command doesn't work. Some operating systems may ease this pain by providing a global hook that looks for commands that don't exist and suggest an OS package they can install to make the command work, but that only works on systems with platform package managers (such as major Linux distributions). No such assistance is available for Windows and Mac OS X users. The challenges of dealing with this problem are a regular feature of feedback the core Python developers receive from professional educators and others introducing new users to Python. If a project chooses to duplicate the installation instructions and tell their users how to install the package manager before telling them how to install their own project then whenever these instructions need updates they need updating by every project that has duplicated them. This is particular problematic when there are multiple competing installation tools available, and different projects recommend different tools. This specific problem can be partially alleviated by strongly promoting ``pip`` as the default installer and recommending that other projects reference `pip's own bootstrapping instructions <http://www.pip-installer.org/en/latest/installing.html>`__ rather than duplicating them. However the user experience created by this approach still isn't good (especially on Windows, where downloading and running the ``get-pip.py`` bootstrap script with the default OS configuration is significantly more painful than downloading and running a binary executable or installer). The situation becomes even more complicated when multiple Python versions are involved (for example, parallel installations of Python 2 and Python 3), since that makes it harder to create and maintain good platform specific ``pip`` installers independently of the CPython installers. The projects that have decided to forgo dependencies altogether are forced to either duplicate the efforts of other projects by inventing their own solutions to problems or are required to simply include the other projects in their own source trees. Both of these options present their own problems either in duplicating maintenance work across the ecosystem or potentially leaving users vulnerable to security issues because the included code or duplicated efforts are not automatically updated when upstream releases a new version. By providing a cross-platform package manager by default it will be easier for users trying to install these third-party packages as well as easier for the people distributing them as they should now be able to safely assume that most users will have the appropriate installation tools available. This is expected to become more important in the future as the Wheel_ package format (deliberately) does not have a built in "installer" in the form of ``setup.py`` so users wishing to install from a wheel file will want an installer even in the simplest cases. Reducing the burden of actually installing a third-party package should also decrease the pressure to add every useful module to the standard library. This will allow additions to the standard library to focus more on why Python should have a particular tool out of the box, and why it is reasonable for that package to adopt the standard library's 18-24 month feature release cycle, instead of using the general difficulty of installing third-party packages as justification for inclusion. Providing a standard installation system also helps with bootstrapping alternate build and installer systems, such as ``setuptools``, ``zc.buildout`` and the ``hashdist``/``conda`` combination that is aimed specifically at the scientific community. So long as ``pip install <tool>`` works, then a standard Python-specific installer provides a reasonably secure, cross platform mechanism to get access to these utilities. Why pip? -------- ``pip`` has been chosen as the preferred default installer, as it addresses several design and user experience issues with its predecessor ``easy_install`` (these issues can't readily be fixed in ``easy_install`` itself due to backwards compatibility concerns). ``pip`` is also well suited to working within the bounds of a single Python runtime installation (including associated virtual environments), which is a desirable feature for a tool bundled with CPython. Other tools like ``zc.buildout`` and ``conda`` are more ambitious in their aims (and hence substantially better than ``pip`` at handling external binary dependencies), so it makes sense for the Python ecosystem to treat them more like platform package managers to inter operate with rather than as the default cross-platform installation tool. This relationship is similar to that between ``pip`` and platform package management systems like ``apt`` and ``yum`` (which are also designed to handle arbitrary binary dependencies). Explicit bootstrapping mechanism ================================ An additional module called ``ensurepip`` will be added to the standard library whose purpose is to install pip and any of its dependencies into the appropriate location (most commonly site-packages). It will expose a callable named ``bootstrap()`` as well as offer direct execution via ``python -m ensurepip``. The bootstrap will *not* contact PyPI, but instead rely on a private copy of pip stored inside the standard library. Accordingly, only options related to the installation location will be supported (``--user``, ``--root``, etc). It is considered desirable that users be strongly encouraged to use the latest available version of ``pip``, in order to take advantage of the ongoing efforts to improve the security of the PyPI based ecosystem, as well as benefiting from the efforts to improve the speed, reliability and flexibility of that ecosystem. In order to satisfy this goal of providing the most recent version of ``pip`` by default, the private copy of ``pip`` will be updated in CPython maintenance releases, which should align well with the 6-month cycle used for new ``pip`` releases. Security considerations ----------------------- The design in this PEP has been deliberately chosen to avoid making any significant changes to the trust model of the CPython installers for end users that do not subsequently make use of ``pip``. The installers will contain all the components of a fully functioning version of Python, including the ``pip`` installer. The installation process will *not* require network access, and will *not* rely on trusting the security of the network connection established between ``pip`` and the Python package index. Only users that choose to use ``pip`` directly will need to pay attention to any PyPI related security considerations. Implementation strategy ----------------------- To ensure there is no need for network access when installing Python or creating virtual environments, the ``ensurepip`` module will, as an implementation detail, include a complete private copy of pip and its dependencies which will be used to extract pip and install it into the target environment. It is important to stress that this private copy of pip is *only* an implementation detail and it should *not* be relied on or assumed to exist beyond the public capabilities exposed through the ``ensurepip`` module (and indirectly through ``venv``). There is not yet a reference ``ensurepip`` implementation. The existing ``get-pip.py`` bootstrap script demonstrates an earlier variation of the general concept, but the standard library version would take advantage of the improved distribution capabilities offered by the CPython installers to include private copies of ``pip`` and ``setuptools`` as wheel files (rather than as embedded base64 encoded data), and would not try to contact PyPI (instead installing directly from the private wheel files. Rather than including separate code to handle the bootstrapping, the ``ensurepip`` module will manipulate sys.path appropriately to allow the wheel files to be used to install themselves, either into the current Python installation or into a virtual environment (as determined by the options passed to the bootstrap command). It is proposed that the implementation be carried out in five separate steps (all steps after the first are independent of each other and can be carried out in any order): * the first step would add the ``ensurepip`` module and the private copies of the most recently released versions of pip and setuptools, and update the "Installing Python Modules" documentation. This change would be applied to Python 2.7, 3.3 and 3.4. * the Windows installer would be updated to offer the new ``pip`` installation option for Python 2.7.6, 3.3.3 and 3.4.0. * the Mac OS X installer would be updated to offer the new ``pip`` installation option for Python 2.7.6, 3.3.3 and 3.4.0. * the ``venv`` module and ``pyvenv`` command would be updated to make use of ``ensurepip`` in Python 3.4+ * the PATH handling and ``sysconfig`` directory layout on Windows would be updated for Python 3.4+ Proposed CLI ------------ The proposed CLI is based on a subset of the existing ``pip install`` options:: Usage: python -m ensurepip [options] General Options: -h, --help Show help. -v, --verbose Give more output. Option is additive, and can be used up to 3 times. -V, --version Show the pip version that would be extracted and exit. -q, --quiet Give less output. Installation Options: -U, --upgrade Upgrade pip and dependencies, even if already installed --user Install using the user scheme. --root <dir> Install everything relative to this alternate root directory. In most cases, end users won't need to use this CLI directly, as ``pip`` should have been installed automatically when installing Python or when creating a virtual environment. Users that want to retrieve the latest version from PyPI, or otherwise need more flexibility, should invoke the extracted ``pip`` appropriately. Proposed module API ------------------- The proposed ``ensurepip`` module API consists of the following two functions:: def version(): """ Returns a string specifying the bundled version of pip. """ def bootstrap(root=None, upgrade=False, user=False, verbosity=0): """ Bootstrap pip into the current Python installation (or the given root directory). """ Invocation from the CPython installers -------------------------------------- The CPython Windows and Mac OS X installers will each gain a new option: * Install pip (the default Python package management utility)? This option will be checked by default. If the option is checked, then the installer will invoke the following command with the just installed Python:: python -m ensurepip --upgrade This ensures that, by default, installing or updating CPython will ensure that the installed version of pip is at least as recent as the one included with that version of CPython. If a newer version of pip has already been installed then ``python -m ensurepip --upgrade`` will simply return without doing anything. Installing from source ---------------------- While the prebuilt binary installers will be updated to run ``python -m ensurepip`` by default, no such change will be made to the ``make install`` and ``make altinstall`` commands of the source distribution. ``ensurepip`` itself (including the private copy of ``pip`` and its dependencies) will still be installed normally (as it is a regular part of the standard library), only the implicit installation of pip and its dependencies will be skipped. Keeping the pip bootstrapping as a separate step for ``make``-based installations should minimize the changes CPython redistributors need to make to their build processes. Avoiding the layer of indirection through ``make`` for the ``ensurepip`` invocation avoids any challenges associated with determining where to install the extracted ``pip``. Changes to virtual environments ------------------------------- Python 3.3 included a standard library approach to virtual Python environments through the ``venv`` module. Since its release it has become clear that very few users have been willing to use this feature directly, in part due to the lack of an installer present by default inside of the virtual environment. They have instead opted to continue using the ``virtualenv`` package which *does* include pip installed by default. To make the ``venv`` more useful to users it will be modified to issue the pip bootstrap by default inside of the new environment while creating it. This will allow people the same convenience inside of the virtual environment as this PEP provides outside of it as well as bringing the ``venv`` module closer to feature parity with the external ``virtualenv`` package, making it a more suitable replacement. To handle cases where a user does not wish to have pip bootstrapped into their virtual environment a ``--without-pip`` option will be added. The ``venv.EnvBuilder`` and ``venv.create`` APIs will be updated to accept one new parameter: ``with_pip`` (defaulting to ``False``). The new default for the module API is chosen for backwards compatibility with the current behaviour (as it is assumed that most invocation of the ``venv`` module happens through third part tools that likely will not want ``pip`` installed without explicitly requesting it), while the default for the command line interface is chosen to try to ensure ``pip`` is available in most virtual environments without additional action on the part of the end user. This particular change will be made only for Python 3.4 and later versions. The third-party ``virtualenv`` project will still be needed to obtain a consistent cross-version experience in Python 3.3 and 2.7. Documentation ------------- The "Installing Python Modules" section of the standard library documentation will be updated to recommend the use of the bootstrapped `pip` installer. It will give a brief description of the most common commands and options, but delegate to the externally maintained ``pip`` documentation for the full details. The existing content of the module installation guide will be retained, but under a new "Invoking distutils directly" subsection. Bundling CA certificates with CPython ------------------------------------- The ``ensurepip`` implementation will include the ``pip`` CA bundle along with the rest of ``pip``. This means CPython effectively includes a CA bundle that is used solely by ``pip`` after it has been extracted. This is considered preferable to relying solely on the system certificate stores, as it ensures that ``pip`` will behave the same across all supported versions of Python, even those prior to Python 3.4 that cannot access the system certificate store on Windows. Automatic installation of setuptools ------------------------------------ ``pip`` currently depends on ``setuptools`` to handle metadata generation during the build process, along with some other features. While work is ongoing to reduce or eliminate this dependency, it is not clear if that work will be complete for pip 1.5 (which is the version likely to be current when Python 3.4.0 is released). This PEP proposes that, if pip still requires it as a dependency, ``ensurepip`` will include a private copy of ``setuptools`` (in addition to the private copy of ``ensurepip``). ``python -m ensurepip`` will then install the private copy in addition to installing ``pip`` itself. However, this behavior is officially considered an implementation detail. Other projects which explicitly require ``setuptools`` must still provide an appropriate dependency declaration, rather than assuming ``setuptools`` will always be installed alongside ``pip``. Once pip is able to run ``pip install --upgrade pip`` without needing ``setuptools`` installed first, then the private copy of ``setuptools`` will be removed from ``ensurepip`` in subsequent CPython releases. Updating the private copy of pip -------------------------------- In order to keep up with evolutions in packaging as well as providing users with as recent version a possible the ``ensurepip`` module will be regularly updated to the latest versions of everything it bootstraps. After each new ``pip`` release, and again during the preparation for any release of Python (including feature releases), a script, provided as part of this PEP, will be run to ensure the private copies stored in the CPython source repository have been updated to the latest versions. Updating the ensurepip module API and CLI ----------------------------------------- Like ``venv`` and ``pyvenv``, the ``ensurepip`` module API and CLI will be governed by the normal rules for the standard library: no new features are permitted in maintenance releases. However, the embedded components may be updated as noted above, so the extracted ``pip`` may offer additional functionality in maintenance releases. Feature addition in maintenance releases ======================================== Adding a new module to the standard library in Python 2.7 and 3.3 maintenance releases breaks the usual policy of "no new features in maintenance releases". It is being proposed in this case as the current bootstrapping issues for the third-party Python package ecosystem greatly affects the experience of new users, especially on Python 2 where many Python 3 standard library improvements are available as backports on PyPI, but are not included in the Python 2 standard library. By updating Python 2.7, 3.3 and 3.4 to easily bootstrap the PyPI ecosystem, this change should aid the vast majority of current Python users, rather than only those with the freedom to adopt Python 3.4 as soon as it is released. Uninstallation ============== No changes are proposed to the uninstallation process by this PEP. The bootstrapped pip will be installed the same way as any other pip installed packages, and will be handled in the same way as any other post-install additions to the Python environment. At least on Windows, that means the bootstrapped files will be left behind after uninstallation, since those files won't be associated with the Python MSI installer. While the case can be made for the CPython installers clearing out these directories automatically, changing that behaviour is considered outside the scope of this PEP. Script Execution on Windows =========================== While the Windows installer was updated in Python 3.3 to optionally make ``python`` available on the PATH, no such change was made to include the Scripts directory. Independently of this PEP, a proposal has also been made to rename the ``Tools\Scripts`` subdirectory to ``bin`` in order to improve consistency with the typical script installation directory names on \*nix systems. Accordingly, in addition to adding the option to extract and install ``pip`` during installation, this PEP proposes that the Windows installer (and ``sysconfig``) in Python 3.4 and later be updated to: - install scripts to PythonXY\bin rather than PythonXY\Tools\Scripts - add PythonXY\bin to the Windows PATH (in addition to PythonXY) when the PATH modification option is enabled during installation For Python 2.7 and 3.3, it is proposed that the only change be the one to bootstrap ``pip`` by default. This means that, for Python 3.3, the most reliable way to invoke pip on Windows (without tinkering manually with PATH) will actually be ``py -m pip`` (or ``py -3 -m pip`` to select the Python 3 version if both Python 2 and 3 are installed) rather than simply calling ``pip``. For Python 2.7 and 3.2, the most reliable mechanism will be to install the standalone Python launcher for Windows and then use ``py -m pip`` as noted above. Adding the scripts directory to the system PATH would mean that ``pip`` works reliably in the "only one Python installation on the system PATH" case, with ``py -m pip``, ``pipX``, or ``pipX.Y`` needed only to select a non-default version in the parallel installation case (and outside a virtual environment). This change should also make the ``pyvenv`` command substantially easier to invoke on Windows, along with all scripts installed by ``pip``, ``easy_install`` and similar tools. While the script invocations on recent versions of Python will run through the Python launcher for Windows, this shouldn't cause any issues, as long as the Python files in the Scripts directory correctly specify a Python version in their shebang line or have an adjacent Windows executable (as ``easy_install`` and ``pip`` do). Recommendations for Downstream Distributors =========================================== A common source of Python installations are through downstream distributors such as the various Linux Distributions [#ubuntu]_ [#debian]_ [#fedora]_, OSX package managers [#homebrew]_, or Python-specific tools [#conda]_. In order to provide a consistent, user-friendly experience to all users of Python regardless of how they attained Python this PEP recommends and asks that downstream distributors: * Ensure that whenever Python is installed pip is also installed. * This may take the form of separate packages with dependencies on each other so that installing the Python package installs the pip package and installing the pip package installs the Python package. * Another reasonable way to implement this is to package pip separately but ensure that there is some sort of global hook that will recommend installing the separate pip package when a user executes ``pip`` without it being installed. Systems that choose this option should ensure that the ``pyvenv`` command still installs pip into the virtual environment by default, but may modify the ``ensurepip`` module in the system Python installation to redirect to the platform provided mechanism when installing ``pip`` globally. * Do not remove the bundled copy of pip. * This is required for installation of pip into a virtual environment by the ``venv`` module. * This is similar to the existing ``virtualenv`` package for which many downstream distributors have already made exception to the common "debundling" policy. * This does mean that if ``pip`` needs to be updated due to a security issue, so does the private copy in the ``ensurepip`` bootstrap module * However, altering the private copy of pip to remove the embedded CA certificate bundle and rely on the system CA bundle instead is a reasonable change. * Migrate build systems to utilize `pip`_ and `Wheel`_ instead of directly using ``setup.py``. * This will ensure that downstream packages can more easily utilize the new metadata formats which may not have a ``setup.py``. * Ensure that all features of this PEP continue to work with any modifications made to the redistributed version of Python. * Checking the version of pip that will be bootstrapped using ``python -m ensurepip --version`` or ``ensurepip.version()``. * Installation of pip into a global or virtual python environment using ``python -m ensurepip`` or ``ensurepip.bootstrap()``. * ``pip install --upgrade pip`` in a global installation should not affect any already created virtual environments (but is permitted to affect future virtual environments, even though it will not do so when using the upstream version of ``ensurepip``). * ``pip install --upgrade pip`` in a virtual environment should not affect the global installation. In the event that a Python redistributor chooses *not* to follow these recommendations, we request that they explicitly document this fact and provide their users with suitable guidance on translating upstream ``pip`` based installation instructions into something appropriate for the platform. Other Python implementations are also encouraged to follow these guidelines where applicable. Policies & Governance ===================== The maintainers of the bootstrapped software and the CPython core team will work together in order to address the needs of both. The bootstrapped software will still remain external to CPython and this PEP does not include CPython subsuming the development responsibilities or design decisions of the bootstrapped software. This PEP aims to decrease the burden on end users wanting to use third-party packages and the decisions inside it are pragmatic ones that represent the trust that the Python community has already placed in the Python Packaging Authority as the authors and maintainers of ``pip``, ``setuptools``, PyPI, ``virtualenv`` and other related projects. Backwards Compatibility ----------------------- The public API and CLI of the ``ensurepip`` module itself will fall under the typical backwards compatibility policy of Python for its standard library. The externally developed software that this PEP bundles does not. Most importantly, this means that the bootstrapped version of pip may gain new features in CPython maintenance releases, and pip continues to operate on its own 6 month release cycle rather than CPython's 18-24 month cycle. Security Releases ----------------- Any security update that affects the ``ensurepip`` module will be shared prior to release with the Python Security Response Team (security at python.org). The PSRT will then decide if the reported issue warrants a security release of CPython with an updated private copy of ``pip``. Appendix: Rejected Proposals ============================ Automatically contacting PyPI when bootstrapping pip ---------------------------------------------------- Earlier versions of this PEP called the bootstrapping module ``getpip`` and defaulted to downloading and installing ``pip`` from PyPI, with the private copy used only as a fallback option or when explicitly requested. This resulted in several complex edge cases, along with difficulties in defining a clean API and CLI for the bootstrap module. It also significantly altered the default trust model for the binary installers published on python.org, as end users would need to explicitly *opt-out* of trusting the security of the PyPI ecosystem (rather than opting in to it by explicitly invoking ``pip`` following installation). As a result, the PEP was simplified to the current design, where the bootstrapping *always* uses the private copy of ``pip``. Contacting PyPI is now always an explicit separate step, with direct access to the full pip interface. Implicit bootstrap ------------------ `PEP439`_, the predecessor for this PEP, proposes its own solution. Its solution involves shipping a fake ``pip`` command that when executed would implicitly bootstrap and install pip if it does not already exist. This has been rejected because it is too "magical". It hides from the end user when exactly the pip command will be installed or that it is being installed at all. It also does not provide any recommendations or considerations towards downstream packagers who wish to manage the globally installed pip through the mechanisms typical for their system. The implicit bootstrap mechanism also ran into possible permissions issues, if a user inadvertently attempted to bootstrap pip without write access to the appropriate installation directories. Including pip directly in the standard library ---------------------------------------------- Similar to this PEP is the proposal of just including pip in the standard library. This would ensure that Python always includes pip and fixes all of the end user facing problems with not having pip present by default. This has been rejected because we've learned, through the inclusion and history of ``distutils`` in the standard library, that losing the ability to update the packaging tools independently can leave the tooling in a state of constant limbo. Making it unable to ever reasonably evolve in a time frame that actually affects users as any new features will not be available to the general population for *years*. Allowing the packaging tools to progress separately from the Python release and adoption schedules allows the improvements to be used by *all* members of the Python community and not just those able to live on the bleeding edge of Python releases. There have also been issues in the past with the "dual maintenance" problem if a project continues to be maintained externally while *also* having a fork maintained in the standard library. Since external maintenance of ``pip`` will always be needed to support earlier Python versions, the proposed bootstrapping mechanism will becoming the explicit responsibility of the CPython core developers (assisted by the pip developers), while pip issues reported to the CPython tracker will be migrated to the pip issue tracker. There will no doubt still be some user confusion over which tracker to use, but hopefully less than has been seen historically when including complete public copies of third-party projects in the standard library. The approach described in this PEP also avoids some technical issues related to handling CPython maintenance updates when pip has been independently updated to a more recent version. The proposed pip-based bootstrapping mechanism handles that automatically, since pip and the system installer never get into a fight about who owns the pip installation (it is always managed through pip, either directly, or indirectly via the ``ensurepip`` bootstrap module). Finally, the separate bootstrapping step means it also easy to avoid installing ``pip`` at all if end users so desire. This is often the case if integrators are using system packages to handle installation of components written in multiple languages using a common set of tools. Defaulting to --user installation --------------------------------- Some consideration was given to bootstrapping pip into the per-user site-packages directory by default. However, this behavior would be surprising (as it differs from the default behavior of pip itself) and is also not currently considered reliable (there are some edge cases which are not handled correctly when pip is installed into the user site-packages directory rather than the system site-packages). .. _Wheel: http://www.python.org/dev/peps/pep-0427/ .. _pip: http://www.pip-installer.org .. _setuptools: https://pypi.python.org/pypi/setuptools .. _PEP439: http://www.python.org/dev/peps/pep-0439/ References ========== .. [1] Discussion thread 1 (distutils-sig) (https://mail.python.org/pipermail/distutils-sig/2013-August/022529.html) .. [2] Discussion thread 2 (distutils-sig) (https://mail.python.org/pipermail/distutils-sig/2013-September/022702.html) .. [3] Discussion thread 3 (python-dev) (https://mail.python.org/pipermail/python-dev/2013-September/128723.html) .. [4] Discussion thread 4 (python-dev) (https://mail.python.org/pipermail/python-dev/2013-September/128780.html) .. [#ubuntu] `Ubuntu <http://www.ubuntu.com/>` .. [#debian] `Debian <http://www.debian.org>` .. [#fedora] `Fedora <https://fedoraproject.org/>` .. [#homebrew] `Homebrew <http://brew.sh/>` .. [#conda] `Conda <http://www.continuum.io/blog/conda>` Copyright ========= This document has been placed in the public domain. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From walter at livinglogic.de Mon Sep 23 13:27:47 2013 From: walter at livinglogic.de (=?UTF-8?B?V2FsdGVyIETDtnJ3YWxk?=) Date: Mon, 23 Sep 2013 13:27:47 +0200 Subject: [Python-Dev] Best practice for documentation for std lib In-Reply-To: <CAP1=2W7ymWcx_Yj6Xsb0Ty1Wj5pUf+_1XtqhPAVyRKWgiQ37mg@mail.gmail.com> References: <20130922031304.GE19939@ando> <20130922102046.764abea5@fsol> <20130922125323.GI19939@ando> <CAP1=2W7ymWcx_Yj6Xsb0Ty1Wj5pUf+_1XtqhPAVyRKWgiQ37mg@mail.gmail.com> Message-ID: <524025B3.7020305@livinglogic.de> On 22.09.13 16:34, Brett Cannon wrote: > The rule of thumb I go by is the docstring should be enough to answer > the question "what args does this thing take and what does it do in > general to know it's the function I want and another one in the same > module?" quickly and succinctly; i.e. just enough so that help() reminds > you about details for a module you are already familiar with that might > come up while at the interpreter prompt. Everything else -- in-depth > discussion of the algorithms, extra examples, why you want to use this > function, etc. -- all go in the .rst docs. It would be great if the docstring contained a link to the online documentation. Servus, Walter From fred at fdrake.net Mon Sep 23 15:38:06 2013 From: fred at fdrake.net (Fred Drake) Date: Mon, 23 Sep 2013 09:38:06 -0400 Subject: [Python-Dev] Best practice for documentation for std lib In-Reply-To: <524025B3.7020305@livinglogic.de> References: <20130922031304.GE19939@ando> <20130922102046.764abea5@fsol> <20130922125323.GI19939@ando> <CAP1=2W7ymWcx_Yj6Xsb0Ty1Wj5pUf+_1XtqhPAVyRKWgiQ37mg@mail.gmail.com> <524025B3.7020305@livinglogic.de> Message-ID: <CAFT4OTGV=xf24Sufd=C_Ej5Mqg7KcOHv0ABWRSD=+dg8HkARFg@mail.gmail.com> On Mon, Sep 23, 2013 at 7:27 AM, Walter D?rwald <walter at livinglogic.de> wrote: > It would be great if the docstring contained a link to the online > documentation. The docstring itself, or the presentation generated by help() ? -Fred -- Fred L. Drake, Jr. <fred at fdrake.net> "A storm broke loose in my mind." --Albert Einstein From chris at kateandchris.net Mon Sep 23 15:45:46 2013 From: chris at kateandchris.net (Chris Lambacher) Date: Mon, 23 Sep 2013 09:45:46 -0400 Subject: [Python-Dev] Enum Eccentricities In-Reply-To: <20130923024139.GA26561@piedra> References: <523F82C4.4060202@stoneleaf.us> <20130923024139.GA26561@piedra> Message-ID: <CAAXXHgKPNsXH1dgv2GeqhVC9+s1YaziCYXq0Aah-e=a3mSLAmg@mail.gmail.com> On Sun, Sep 22, 2013 at 10:41 PM, Zero Piraeus <z at etiol.net> wrote: > I may be misunderstanding the use case given in the issue, > To clarify the use case, since it is my bug, this is so that the enum values are always available for comparison. The exact use case is in Django templates where a value comes from the database. If you want to compare you either have to use __class__ which I would say is a code smell, or you have to provide the Enum class. The code we are using for this is open source http://github.com/tindie/django-tidyenum. An example of how this will be used in practice is: {% if object.state == object.state.completed %} some html {% endif %} > but it seems > to me that having to use > > Color.red.__class__.blue > > (what is being complained about in the issue), while not exactly pretty, > makes a lot more semantic sense than > > Color.red.blue > > ... which is just bizarre. > Any more bizarre than any other class that has properties of it's own type? a = 0 a.real.numerator.real.numerator It is only weird because we are not used to seeing classes where the properties are instances of the class. I am not a core developer, but I have been programming in Python for more than 10 years, and I (and the other similarly experienced developers I work with) found that not having access to the class level properties was weird (yes I am aware that they are not actually class level properties but everywhere else Enum works hard to make it look like they are. See for instance the __dir__ method: http://hg.python.org/cpython/file/ed011b0d7daf/Lib/enum.py#l242). -Chris -- Christopher Lambacher chris at kateandchris.net -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130923/91305d45/attachment.html> From walter at livinglogic.de Mon Sep 23 15:51:27 2013 From: walter at livinglogic.de (=?UTF-8?B?V2FsdGVyIETDtnJ3YWxk?=) Date: Mon, 23 Sep 2013 15:51:27 +0200 Subject: [Python-Dev] Best practice for documentation for std lib In-Reply-To: <CAFT4OTGV=xf24Sufd=C_Ej5Mqg7KcOHv0ABWRSD=+dg8HkARFg@mail.gmail.com> References: <20130922031304.GE19939@ando> <20130922102046.764abea5@fsol> <20130922125323.GI19939@ando> <CAP1=2W7ymWcx_Yj6Xsb0Ty1Wj5pUf+_1XtqhPAVyRKWgiQ37mg@mail.gmail.com> <524025B3.7020305@livinglogic.de> <CAFT4OTGV=xf24Sufd=C_Ej5Mqg7KcOHv0ABWRSD=+dg8HkARFg@mail.gmail.com> Message-ID: <5240475F.4030407@livinglogic.de> On 23.09.13 15:38, Fred Drake wrote: > On Mon, Sep 23, 2013 at 7:27 AM, Walter D?rwald <walter at livinglogic.de> wrote: >> It would be great if the docstring contained a link to the online >> documentation. > > The docstring itself, or the presentation generated by help() ? The presentation generated by help(), or the output of IPython's foo? or foo?? syntax. Servus, Walter From solipsis at pitrou.net Mon Sep 23 16:07:45 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 23 Sep 2013 16:07:45 +0200 Subject: [Python-Dev] Best practice for documentation for std lib References: <20130922031304.GE19939@ando> <20130922102046.764abea5@fsol> <20130922125323.GI19939@ando> <CAP1=2W7ymWcx_Yj6Xsb0Ty1Wj5pUf+_1XtqhPAVyRKWgiQ37mg@mail.gmail.com> <524025B3.7020305@livinglogic.de> <CAFT4OTGV=xf24Sufd=C_Ej5Mqg7KcOHv0ABWRSD=+dg8HkARFg@mail.gmail.com> <5240475F.4030407@livinglogic.de> Message-ID: <20130923160745.12e73f1f@pitrou.net> Le Mon, 23 Sep 2013 15:51:27 +0200, Walter D?rwald <walter at livinglogic.de> a ?crit : > On 23.09.13 15:38, Fred Drake wrote: > > > On Mon, Sep 23, 2013 at 7:27 AM, Walter D?rwald > > <walter at livinglogic.de> wrote: > >> It would be great if the docstring contained a link to the online > >> documentation. > > > > The docstring itself, or the presentation generated by help() ? > > The presentation generated by help(), or the output of IPython's foo? > or foo?? syntax. Perhaps objects could gain some kind of optional __docurl__ attribute, to avoid cluttering __doc__ with external links. Not sure how that could be auto-generated, though (we certainly don't want to maintain doc links manually). Regards Antoine. From rdmurray at bitdance.com Mon Sep 23 16:43:10 2013 From: rdmurray at bitdance.com (R. David Murray) Date: Mon, 23 Sep 2013 10:43:10 -0400 Subject: [Python-Dev] Best practice for documentation for std lib In-Reply-To: <CAP7+vJLRn-HAhNwBGLOmdPZA7YBq5JCs=H19FVeard6Y8yqYNQ@mail.gmail.com> References: <20130922031304.GE19939@ando> <20130922102046.764abea5@fsol> <20130922125323.GI19939@ando> <CAP1=2W7ymWcx_Yj6Xsb0Ty1Wj5pUf+_1XtqhPAVyRKWgiQ37mg@mail.gmail.com> <003301ceb7bd$e1a59430$a4f0bc90$@gmail.com> <CAP7+vJLKVNWEDT62U4pJ7FGARthhVqb74hnLWznBag353s-Obw@mail.gmail.com> <005001ceb7c9$4dc308e0$e9491aa0$@gmail.com> <06E35F19-F893-415B-B127-77D1588A565A@masklinn.net> <CAP7+vJLRn-HAhNwBGLOmdPZA7YBq5JCs=H19FVeard6Y8yqYNQ@mail.gmail.com> Message-ID: <20130923144311.23751250959@webabinitio.net> On Sun, 22 Sep 2013 16:19:21 -0700, Guido van Rossum <guido at python.org> wrote: > On Sun, Sep 22, 2013 at 2:41 PM, Xavier Morel <python-dev at masklinn.net>wrote: > > > The points here are that there's a single source of truth (so we can't > > have conflicting docstring and rst documentation), and documentation > > becoming outdated can be noticed from both docstring and published > > documentation. Another thing that hasn't been mentioned about docstrings vs rst docs, is that even when the text is identical, it generally isn't. By that I mean the rst docs have ReST markup, but the docstrings don't. So using autodoc doesn't just mean adding autodoc to our doc building toolchain, it *also* means adding support in pydoc for turning ReST into plain text. And that still leaves the markup in the docstring, where it will be very distracting while actually reading the source code. Which, unlike other commenters, I spend more time doing that than I do using help (and that doesn't apply to just the stdlib for me). > > > Another case of DRY madness. > > It seems too many programmers see documentation as unpleasant red tape they > want to cut through as quickly as possible, instead of an opportunity to > communicate with their *users*. To the contrary: users should be the most > important people in the world if you're writing code that's worth > documenting at all. I won't pretend that I find having to edit two places when updating fundamental documentation pleasant, but... I find that as often as not I want to word things *differently* in the docstring vs the rst docs for the same function. Sometimes the difference is subtle; most often it is an omission of detail along the lines Guido suggests. But when writing the rst version, it generally isn't that I'm simply continuing on after the first paragraph; instead, the organization of even that starting text is different, because I'm aware that the reader has a different mindset (quick help vs reference documentation) when reading the two texts.[*] As with the question of NEWS items versus checkin messages, the intended audience, or in this case the mindset of the intended audience, is *different*, and so the text should very often be different as well[**]. --David [*] I posit that this is even *more* true for those who say they only use help and only fall back to the full docs when the docstrings aren't enough. [**] I also wonder if long function docstrings have a negative impact on usability in IDEs that pop up windows with docstring information in them. From guido at python.org Mon Sep 23 16:53:00 2013 From: guido at python.org (Guido van Rossum) Date: Mon, 23 Sep 2013 07:53:00 -0700 Subject: [Python-Dev] Enum Eccentricities In-Reply-To: <CAAXXHgKPNsXH1dgv2GeqhVC9+s1YaziCYXq0Aah-e=a3mSLAmg@mail.gmail.com> References: <523F82C4.4060202@stoneleaf.us> <20130923024139.GA26561@piedra> <CAAXXHgKPNsXH1dgv2GeqhVC9+s1YaziCYXq0Aah-e=a3mSLAmg@mail.gmail.com> Message-ID: <CAP7+vJKisYvdOug-fmR_SZEET7icHGjEx2pFYYHXyZHrhjWhzw@mail.gmail.com> On Mon, Sep 23, 2013 at 6:45 AM, Chris Lambacher <chris at kateandchris.net>wrote: > > On Sun, Sep 22, 2013 at 10:41 PM, Zero Piraeus <z at etiol.net> wrote: > >> I may be misunderstanding the use case given in the issue, >> > > To clarify the use case, since it is my bug, this is so that the enum > values are always available for comparison. The exact use case is in Django > templates where a value comes from the database. If you want to compare you > either have to use __class__ which I would say is a code smell, or you have > to provide the Enum class. The code we are using for this is open source > http://github.com/tindie/django-tidyenum. An example of how this will be > used in practice is: > > {% if object.state == object.state.completed %} > some html > {% endif %} > Now I see your use case. But I disagree that the best solution is to allow accessing the enum values as attributes on object.state -- I would recommend using __class__ to make it clear to the reader what's going on, or to add the right Enum subclass to your template parameters. Tbe expression you show in your example here will just baffle most readers who haven't thought deeply about the issue. (How would you compare a value that is a timedelta with a known timedelta? You'd have to import the datetime module or use __class__, right?) > but it seems >> to me that having to use >> >> Color.red.__class__.blue >> >> (what is being complained about in the issue), while not exactly pretty, >> makes a lot more semantic sense than >> >> Color.red.blue >> >> ... which is just bizarre. >> > > Any more bizarre than any other class that has properties of it's own type? > Yes, because you rarely if ever see them accessed that way. > a = 0 > a.real.numerator.real.numerator > But that's different! Each of the attributes here (.real, .numerator) is defined as an instance attribute. > > It is only weird because we are not used to seeing classes where the > properties are instances of the class. I am not a core developer, but I > have been programming in Python for more than 10 years, and I (and the > other similarly experienced developers I work with) found that not having > access to the class level properties was weird (yes I am aware that they > are not actually class level properties but everywhere else Enum works hard > to make it look like they are. See for instance the __dir__ method: > http://hg.python.org/cpython/file/ed011b0d7daf/Lib/enum.py#l242). > Sorry to burst your bubble, but there is no rule that because you can access something on the class you should be able to access it on the instance. Try asking an instance for its class's __mro__ or __bases__. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130923/f6dc14c3/attachment.html> From guido at python.org Mon Sep 23 16:56:00 2013 From: guido at python.org (Guido van Rossum) Date: Mon, 23 Sep 2013 07:56:00 -0700 Subject: [Python-Dev] Best practice for documentation for std lib In-Reply-To: <523FF44D.1000303@hastings.org> References: <20130922031304.GE19939@ando> <CAF-Rda_sTOiKi6c9ihFVL-N90PC-xHxOAMm1im0s-KVyy=L_Ng@mail.gmail.com> <87ioxs3iix.fsf@uwakimon.sk.tsukuba.ac.jp> <CAP7+vJJucubZtXHhnNbAcE0GWfQTnOozHK9W=uv3CWkMX15XGA@mail.gmail.com> <20130923003142.GO19939@ando> <CAP7+vJLThCjrCGGaJLMNd2Uza6wanODnipjmZ3LBZpTkub=uCg@mail.gmail.com> <l1o8ri$mjv$1@ger.gmane.org> <CAP7+vJJSvX78f2+i7_C5mNMdOQpfSD_oKqC01Z8CEHaU2krqTw@mail.gmail.com> <523FF44D.1000303@hastings.org> Message-ID: <CAP7+vJLcGYXfS5iSjmCqh4BKFmatjWoGdZi5DednagfcvcUf1w@mail.gmail.com> On Mon, Sep 23, 2013 at 12:57 AM, Larry Hastings <larry at hastings.org> wrote: > On 09/23/2013 03:44 AM, Guido van Rossum wrote: > > On Sun, Sep 22, 2013 at 7:25 PM, Terry Reedy <tjreedy at udel.edu> wrote: > >> I am gradually changing Idle docstrings, though only Idle developers will >> ever see them. Writing a 60 char summary forces a clear understanding of >> the function. > > > Glad you like it. I still do, too, but I've given up hope to convince > all core developers to stick to this style. :-( > > > ... Argument Clinic to the rescue? Since the last time this subject came > up, Clinic has started enforcing a summary line in docstrings. I didn't > realize it had to be 60 columns though, should I add that? Is the entire > docstring supposed to be 60 columns max? > I think 60 is just a guideline. In stdlib .py source code I want it not to extend beyond the 79th column (see recent PEP 8 argument). For a typical class, where the docstring is indented 4 spaces, that leaves 72 characters for the summary line (including a final period!). For a method it's 68 due to the extra indent. FWIW I also prefer having the summary line on the same line as the opening """: def foo(): """This is the summary. This is the rest. """ > p.s. status update: AC is done enough to be worth considering checking > in. Now I have to finish the PEP and write the documentation. > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130923/8acbe2fd/attachment.html> From guido at python.org Mon Sep 23 16:56:47 2013 From: guido at python.org (Guido van Rossum) Date: Mon, 23 Sep 2013 07:56:47 -0700 Subject: [Python-Dev] Best practice for documentation for std lib In-Reply-To: <524025B3.7020305@livinglogic.de> References: <20130922031304.GE19939@ando> <20130922102046.764abea5@fsol> <20130922125323.GI19939@ando> <CAP1=2W7ymWcx_Yj6Xsb0Ty1Wj5pUf+_1XtqhPAVyRKWgiQ37mg@mail.gmail.com> <524025B3.7020305@livinglogic.de> Message-ID: <CAP7+vJLDo9KkdzzQfnJe_bTzunAP6+gBznY1Dx5pvxBqva0YjQ@mail.gmail.com> On Mon, Sep 23, 2013 at 4:27 AM, Walter D?rwald <walter at livinglogic.de>wrote: > It would be great if the docstring contained a link to the online > documentation. > That would have to be a feature of help(), not hardcoded in each docstring. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130923/abcc8955/attachment.html> From steve at pearwood.info Mon Sep 23 17:16:23 2013 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 24 Sep 2013 01:16:23 +1000 Subject: [Python-Dev] Enum Eccentricities In-Reply-To: <CAP7+vJKisYvdOug-fmR_SZEET7icHGjEx2pFYYHXyZHrhjWhzw@mail.gmail.com> References: <523F82C4.4060202@stoneleaf.us> <20130923024139.GA26561@piedra> <CAAXXHgKPNsXH1dgv2GeqhVC9+s1YaziCYXq0Aah-e=a3mSLAmg@mail.gmail.com> <CAP7+vJKisYvdOug-fmR_SZEET7icHGjEx2pFYYHXyZHrhjWhzw@mail.gmail.com> Message-ID: <20130923151623.GB7989@ando> On Mon, Sep 23, 2013 at 07:53:00AM -0700, Guido van Rossum wrote: > there is no rule that because you can > access something on the class you should be able to access it on the > instance. Try asking an instance for its class's __mro__ or __bases__. It might not be a rule, but it's certainly the norm. I reckon that class attributes that aren't accessible from the instance are significantly more surprising than Color.red.blue. I know I'm in a minority here, but Color.red.blue seems obvious and straightforward to me. The fact that it doesn't work surprises me. Given that instance = Color.red assert isinstance(instance, Color) # well of course it is assert hasattr(Color, "blue") I would expect instance.blue to work, and I'm completely at a loss as to how Enum has managed to prevent it. -- Steven From solipsis at pitrou.net Mon Sep 23 17:22:45 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 23 Sep 2013 17:22:45 +0200 Subject: [Python-Dev] Revert #12085 fix for __del__ attribute error message In-Reply-To: <CADiSq7dJAdStAzPYijQTC_fC6_01Jkt4kidSh+YDxWNNv_0tOA@mail.gmail.com> References: <l1l2bm$1bp$1@ger.gmane.org> <20130921221524.55D4225007F@webabinitio.net> <l1lbkl$k0g$1@ger.gmane.org> <CAP7+vJLHO12ZNwX4UZ0c_BF9pr+xxxLZfy2dGjBcMVenGow9vQ@mail.gmail.com> <l1n9o6$ap3$1@ger.gmane.org> <CAP7+vJL+0XEBpOnVbTpZJNLzahgm8EkyuXdGdhE27TVwHu4GAg@mail.gmail.com> <l1nkdh$lh7$1@ger.gmane.org> <CADiSq7eGpFtPbRc8BmiP2H9V2PVMma3ycX3Ky_MJ-F+TpEiU6w@mail.gmail.com> <CAP7+vJKAP5R+2R++z8+DQjSW6EVycoczB2gWeo759-NG4G6Veg@mail.gmail.com> <523FDB8F.1080207@canterbury.ac.nz> <CADiSq7caBSr4ujFo0HBYGF91SJp2NcP-_SreBqad-=KpcZdeCw@mail.gmail.com> <20130923104500.5645fdbb@pitrou.net> <CADiSq7dJAdStAzPYijQTC_fC6_01Jkt4kidSh+YDxWNNv_0tOA@mail.gmail.com> Message-ID: <20130923172245.0cbaec81@fsol> On Mon, 23 Sep 2013 18:51:04 +1000 Nick Coghlan <ncoghlan at gmail.com> wrote: > On 23 September 2013 18:45, Antoine Pitrou <solipsis at pitrou.net> wrote: > > Le Mon, 23 Sep 2013 18:17:51 +1000, > > Nick Coghlan <ncoghlan at gmail.com> a ?crit : > >> > >> Here's what I suggest changing that error to: > >> > >> >>> del x > >> Unraisable exception suppressed when calling <bound method C.__del__ > >> of <__main__.C object at 0x7f98b8b61538>> > >> Traceback (most recent call last): > >> File "<stdin>", line 3, in __del__ > >> RuntimeError: Going away now > > > > Why not simply "Exception automatically caught in <bound method > > C.__del__> [...]" ? > > It only answers the "what" (i.e. the exception was automatically > caught), without addressing the "why" (i.e. because there wasn't > anything else useful the interpreter could do with it) Yes, but I agree with Greg that "unraisable" is wrong. After all, it was raised, and it can even be caught by the programmer (inside __del__). Regards Antoine. From solipsis at pitrou.net Mon Sep 23 17:23:20 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 23 Sep 2013 17:23:20 +0200 Subject: [Python-Dev] Best practice for documentation for std lib References: <20130922031304.GE19939@ando> <CAF-Rda_sTOiKi6c9ihFVL-N90PC-xHxOAMm1im0s-KVyy=L_Ng@mail.gmail.com> <87ioxs3iix.fsf@uwakimon.sk.tsukuba.ac.jp> <CAP7+vJJucubZtXHhnNbAcE0GWfQTnOozHK9W=uv3CWkMX15XGA@mail.gmail.com> <20130923003142.GO19939@ando> <CAP7+vJLThCjrCGGaJLMNd2Uza6wanODnipjmZ3LBZpTkub=uCg@mail.gmail.com> <l1o8ri$mjv$1@ger.gmane.org> <CAP7+vJJSvX78f2+i7_C5mNMdOQpfSD_oKqC01Z8CEHaU2krqTw@mail.gmail.com> <523FF44D.1000303@hastings.org> Message-ID: <20130923172320.5da170cb@fsol> On Mon, 23 Sep 2013 08:57:01 +0100 Larry Hastings <larry at hastings.org> wrote: > On 09/23/2013 03:44 AM, Guido van Rossum wrote: > > On Sun, Sep 22, 2013 at 7:25 PM, Terry Reedy <tjreedy at udel.edu > > <mailto:tjreedy at udel.edu>> wrote: > > > > I am gradually changing Idle docstrings, though only Idle > > developers will ever see them. Writing a 60 char summary forces a > > clear understanding of the function. > > > > > > Glad you like it. I still do, too, but I've given up hope to convince > > all core developers to stick to this style. :-( > > ... Argument Clinic to the rescue? Since the last time this subject > came up, Clinic has started enforcing a summary line in docstrings. I > didn't realize it had to be 60 columns though, should I add that? Is the > entire docstring supposed to be 60 columns max? > > p.s. status update: AC is done enough to be worth considering checking > in. Now I have to finish the PEP and write the documentation. I hope it can make it in for 3.4 :) cheers Antoine. From skip at pobox.com Mon Sep 23 17:18:22 2013 From: skip at pobox.com (Skip Montanaro) Date: Mon, 23 Sep 2013 10:18:22 -0500 Subject: [Python-Dev] Best practice for documentation for std lib In-Reply-To: <CAP7+vJLDo9KkdzzQfnJe_bTzunAP6+gBznY1Dx5pvxBqva0YjQ@mail.gmail.com> References: <20130922031304.GE19939@ando> <20130922102046.764abea5@fsol> <20130922125323.GI19939@ando> <CAP1=2W7ymWcx_Yj6Xsb0Ty1Wj5pUf+_1XtqhPAVyRKWgiQ37mg@mail.gmail.com> <524025B3.7020305@livinglogic.de> <CAP7+vJLDo9KkdzzQfnJe_bTzunAP6+gBznY1Dx5pvxBqva0YjQ@mail.gmail.com> Message-ID: <CANc-5UzYUtta=4Q3c7n-LXdcYBnWrh5VYdMwSNPJ=5a0HuVfig@mail.gmail.com> >> It would be great if the docstring contained a link to the online >> documentation. > > That would have to be a feature of help(), not hardcoded in each docstring. That *is* a feature of the help function: Help on built-in module sys: >>> help(sys) NAME sys FILE (built-in) MODULE DOCS http://docs.python.org/library/sys ... (pydoc too, though I'm 99.9% sure they use the same underlying facility Ping originally implemented.) Skip From d.s.seljebotn at astro.uio.no Mon Sep 23 17:17:31 2013 From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn) Date: Mon, 23 Sep 2013 17:17:31 +0200 Subject: [Python-Dev] Best practice for documentation for std lib In-Reply-To: <20130923144311.23751250959@webabinitio.net> References: <20130922031304.GE19939@ando> <20130922102046.764abea5@fsol> <20130922125323.GI19939@ando> <CAP1=2W7ymWcx_Yj6Xsb0Ty1Wj5pUf+_1XtqhPAVyRKWgiQ37mg@mail.gmail.com> <003301ceb7bd$e1a59430$a4f0bc90$@gmail.com> <CAP7+vJLKVNWEDT62U4pJ7FGARthhVqb74hnLWznBag353s-Obw@mail.gmail.com> <005001ceb7c9$4dc308e0$e9491aa0$@gmail.com> <06E35F19-F893-415B-B127-77D1588A565A@masklinn.net> <CAP7+vJLRn-HAhNwBGLOmdPZA7YBq5JCs=H19FVeard6Y8yqYNQ@mail.gmail.com> <20130923144311.23751250959@webabinitio.net> Message-ID: <52405B8B.4090704@astro.uio.no> On 09/23/2013 04:43 PM, R. David Murray wrote: > On Sun, 22 Sep 2013 16:19:21 -0700, Guido van Rossum <guido at python.org> wrote: >> On Sun, Sep 22, 2013 at 2:41 PM, Xavier Morel <python-dev at masklinn.net>wrote: >> >>> The points here are that there's a single source of truth (so we can't >>> have conflicting docstring and rst documentation), and documentation >>> becoming outdated can be noticed from both docstring and published >>> documentation. > > Another thing that hasn't been mentioned about docstrings vs rst docs, > is that even when the text is identical, it generally isn't. By that I > mean the rst docs have ReST markup, but the docstrings don't. So using FYI, the scientific Python community have their own standard (the NumPy docstring standard), with light RST markup in docstrings: https://github.com/numpy/numpy/blob/master/doc/HOWTO_DOCUMENT.rst.txt It's a fairly popular standard in the scientific sub-culture. I think there's something about scientific codes which tips the scales away from Guido's position of short docstrings being better; when working with computation, you pretty much need the documentation, formulas, references to paper it is based on and so on in order to read and understand the code in the first place, so interspersing makes more sense than it may in the stdlib. Seeing code with references "x", "alpha", "beta", "gamma" without their definition is pretty useless :-) Dag Sverre From guido at python.org Mon Sep 23 17:49:28 2013 From: guido at python.org (Guido van Rossum) Date: Mon, 23 Sep 2013 08:49:28 -0700 Subject: [Python-Dev] Enum Eccentricities In-Reply-To: <20130923151623.GB7989@ando> References: <523F82C4.4060202@stoneleaf.us> <20130923024139.GA26561@piedra> <CAAXXHgKPNsXH1dgv2GeqhVC9+s1YaziCYXq0Aah-e=a3mSLAmg@mail.gmail.com> <CAP7+vJKisYvdOug-fmR_SZEET7icHGjEx2pFYYHXyZHrhjWhzw@mail.gmail.com> <20130923151623.GB7989@ando> Message-ID: <CAP7+vJK3chF=TUvjJBD=gcTtAQ4eEcJ5jgkOSpADJUuLGnKqkQ@mail.gmail.com> On Mon, Sep 23, 2013 at 8:16 AM, Steven D'Aprano <steve at pearwood.info>wrote: > On Mon, Sep 23, 2013 at 07:53:00AM -0700, Guido van Rossum wrote: > > > there is no rule that because you can > > access something on the class you should be able to access it on the > > instance. Try asking an instance for its class's __mro__ or __bases__. > > It might not be a rule, but it's certainly the norm. I reckon that class > attributes that aren't accessible from the instance are significantly > more surprising than Color.red.blue. > Whether this feels right (to me) depends on whether the class attribute was *meant* to be used as an instance attribute as well, or not. For methods, that's an obvious yes. For class attributes used as defaults for instance variables, yes again. For class attributes that are meant to be per-class state rather than per-instance state, I would use the class name or __class__ to make it clear to the (human) reader that we're using a class variable. This also avoids accidentally overriding it with an instance variable (although that's not an issue for Enum). > I know I'm in a minority here, but Color.red.blue seems obvious and > straightforward to me. The fact that it doesn't work surprises me. Given > that > > instance = Color.red > assert isinstance(instance, Color) # well of course it is > assert hasattr(Color, "blue") > > I would expect instance.blue to work, and I'm completely at a loss as to > how Enum has managed to prevent it. > Through the magic of metaclasses. :-) The only attributes that an enum instance should have are things like .name and .value, and the methods implementing various standard protocols like __repr__ and __eq__. The whole point of Enum is that it involves a fair bit of magic to offer a robust way to define Enums that "feel right" as Enums. You shouldn't think of it as a slightly odd class. You should think of it as a different data type altogether that happens to be implemented using metaclass tecnology. But many things you "know" about classes don't apply to Enums; this is just one of many. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130923/734eabf0/attachment.html> From ethan at stoneleaf.us Mon Sep 23 18:12:27 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 23 Sep 2013 09:12:27 -0700 Subject: [Python-Dev] Enum Eccentricities In-Reply-To: <20130923151623.GB7989@ando> References: <523F82C4.4060202@stoneleaf.us> <20130923024139.GA26561@piedra> <CAAXXHgKPNsXH1dgv2GeqhVC9+s1YaziCYXq0Aah-e=a3mSLAmg@mail.gmail.com> <CAP7+vJKisYvdOug-fmR_SZEET7icHGjEx2pFYYHXyZHrhjWhzw@mail.gmail.com> <20130923151623.GB7989@ando> Message-ID: <5240686B.3040504@stoneleaf.us> On 09/23/2013 08:16 AM, Steven D'Aprano wrote: > > I would expect instance.blue to work, and I'm completely at a loss as to > how Enum has managed to prevent it. [A peek behind the curtains...] Currently, Enum members do not live in the class dict. So when, for example, blue is searched for in red, it is not found in the instance dict, and it is not found in the class dict. As you know, __getattr__ will then be invoked -- but the Enum class does not have its own __getattr__, nor its own __getattribute__, and so we get an AttributeError. Well, you may ask, if blue does not live in the class dict, how does Color.blue work? I'm glad you asked. ;) Color is of type EnumMeta, and EnumMeta /does/ have __getattr__, so a failed /class/ lookup will invoke the metaclass __getattr__, which will search in the right place, find, and return, Color.blue. -- ~Ethan~ From rdmurray at bitdance.com Mon Sep 23 18:23:55 2013 From: rdmurray at bitdance.com (R. David Murray) Date: Mon, 23 Sep 2013 12:23:55 -0400 Subject: [Python-Dev] Revert #12085 fix for __del__ attribute error message In-Reply-To: <20130923172245.0cbaec81@fsol> References: <l1l2bm$1bp$1@ger.gmane.org> <20130921221524.55D4225007F@webabinitio.net> <l1lbkl$k0g$1@ger.gmane.org> <CAP7+vJLHO12ZNwX4UZ0c_BF9pr+xxxLZfy2dGjBcMVenGow9vQ@mail.gmail.com> <l1n9o6$ap3$1@ger.gmane.org> <CAP7+vJL+0XEBpOnVbTpZJNLzahgm8EkyuXdGdhE27TVwHu4GAg@mail.gmail.com> <l1nkdh$lh7$1@ger.gmane.org> <CADiSq7eGpFtPbRc8BmiP2H9V2PVMma3ycX3Ky_MJ-F+TpEiU6w@mail.gmail.com> <CAP7+vJKAP5R+2R++z8+DQjSW6EVycoczB2gWeo759-NG4G6Veg@mail.gmail.com> <523FDB8F.1080207@canterbury.ac.nz> <CADiSq7caBSr4ujFo0HBYGF91SJp2NcP-_SreBqad-=KpcZdeCw@mail.gmail.com> <20130923104500.5645fdbb@pitrou.net> <CADiSq7dJAdStAzPYijQTC_fC6_01Jkt4kidSh+YDxWNNv_0tOA@mail.gmail.com> <20130923172245.0cbaec81@fsol> Message-ID: <20130923162356.2135F25095A@webabinitio.net> On Mon, 23 Sep 2013 17:22:45 +0200, Antoine Pitrou <solipsis at pitrou.net> wrote: > On Mon, 23 Sep 2013 18:51:04 +1000 > Nick Coghlan <ncoghlan at gmail.com> wrote: > > On 23 September 2013 18:45, Antoine Pitrou <solipsis at pitrou.net> wrote: > > > Le Mon, 23 Sep 2013 18:17:51 +1000, > > > Nick Coghlan <ncoghlan at gmail.com> a ??crit : > > >> > > >> Here's what I suggest changing that error to: > > >> > > >> >>> del x > > >> Unraisable exception suppressed when calling <bound method C.__del__ > > >> of <__main__.C object at 0x7f98b8b61538>> > > >> Traceback (most recent call last): > > >> File "<stdin>", line 3, in __del__ > > >> RuntimeError: Going away now > > > > > > Why not simply "Exception automatically caught in <bound method > > > C.__del__> [...]" ? > > > > It only answers the "what" (i.e. the exception was automatically > > caught), without addressing the "why" (i.e. because there wasn't > > anything else useful the interpreter could do with it) > > Yes, but I agree with Greg that "unraisable" is wrong. After all, it > was raised, and it can even be caught by the programmer (inside > __del__). Would it work to say "Asynchronous exception suppressed..."? It's not-entirely-precise, but it's less imprecise than "unraisable". --David From z at etiol.net Mon Sep 23 17:17:35 2013 From: z at etiol.net (Zero Piraeus) Date: Mon, 23 Sep 2013 12:17:35 -0300 Subject: [Python-Dev] Enum Eccentricities In-Reply-To: <CAAXXHgKPNsXH1dgv2GeqhVC9+s1YaziCYXq0Aah-e=a3mSLAmg@mail.gmail.com> References: <523F82C4.4060202@stoneleaf.us> <20130923024139.GA26561@piedra> <CAAXXHgKPNsXH1dgv2GeqhVC9+s1YaziCYXq0Aah-e=a3mSLAmg@mail.gmail.com> Message-ID: <20130923151735.GA3364@piedra> On Mon, Sep 23, 2013 at 09:45:46AM -0400, Chris Lambacher wrote: > [...] The exact use case is in Django templates where a value comes > from the database. If you want to compare you either have to use > __class__ which I would say is a code smell, or you have > to provide the Enum class. I'm having a hard time seeing why the latter is problematic, I must admit (certainly no more so than the "Alice in Wonderland" side effects previously described). > [...] An example of how this will be used in practice is: > > {% if object.state == object.state.completed %} > some html > {% endif %} The names used slightly obscure the weirdness of it, but what you're really saying there is: if my_state == my_state.another_state ... which feels more like a code smell to me than {% if object.state == State.completed %} some html {% endif %} That's quite intelligible, and doesn't require anyone to know that an Enum member's siblings can, in your proposal, be accessed directly via dot notation (an unintuitive state of affairs, to me at least). -[]z. -- Zero Piraeus: omnia omnibus http://etiol.net/pubkey.asc From guido at python.org Mon Sep 23 18:50:49 2013 From: guido at python.org (Guido van Rossum) Date: Mon, 23 Sep 2013 09:50:49 -0700 Subject: [Python-Dev] Enum Eccentricities In-Reply-To: <20130923151735.GA3364@piedra> References: <523F82C4.4060202@stoneleaf.us> <20130923024139.GA26561@piedra> <CAAXXHgKPNsXH1dgv2GeqhVC9+s1YaziCYXq0Aah-e=a3mSLAmg@mail.gmail.com> <20130923151735.GA3364@piedra> Message-ID: <CAP7+vJJbci1FND=AEqjxu6HK0hNjtJ3Ud_Dp_Sks_tKpn5XRaA@mail.gmail.com> On Mon, Sep 23, 2013 at 8:17 AM, Zero Piraeus <z at etiol.net> wrote: > On Mon, Sep 23, 2013 at 09:45:46AM -0400, Chris Lambacher wrote: > > [...] The exact use case is in Django templates where a value comes > > from the database. If you want to compare you either have to use > > __class__ which I would say is a code smell, or you have > > to provide the Enum class. > > I'm having a hard time seeing why the latter is problematic, I must > admit (certainly no more so than the "Alice in Wonderland" side effects > previously described). > > > [...] An example of how this will be used in practice is: > > > > {% if object.state == object.state.completed %} > > some html > > {% endif %} > > The names used slightly obscure the weirdness of it, but what you're > really saying there is: > > if my_state == my_state.another_state > > ... which feels more like a code smell to me than > > {% if object.state == State.completed %} > some html > {% endif %} > > That's quite intelligible, and doesn't require anyone to know that an > Enum member's siblings can, in your proposal, be accessed directly via > dot notation (an unintuitive state of affairs, to me at least). > Right. The OP is just concerned that (because these are Django templates) he will have to pass in the 'State' class as a separate template parameter for this to work. But to me that's a problem with Django, and not something for which the Enum type should bend over backwards to cover up for. Given that it's a Djano weakness, IMO the __class__ solution is reasonable enough, although in theory it would allow having object.state be something of the wrong class that happens to have a 'completed' attribute -- that would be a bug of a different color, though. :-) -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130923/5695d625/attachment.html> From stephen at xemacs.org Mon Sep 23 19:46:30 2013 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Tue, 24 Sep 2013 02:46:30 +0900 Subject: [Python-Dev] Enum Eccentricities In-Reply-To: <CAP7+vJJbci1FND=AEqjxu6HK0hNjtJ3Ud_Dp_Sks_tKpn5XRaA@mail.gmail.com> References: <523F82C4.4060202@stoneleaf.us> <20130923024139.GA26561@piedra> <CAAXXHgKPNsXH1dgv2GeqhVC9+s1YaziCYXq0Aah-e=a3mSLAmg@mail.gmail.com> <20130923151735.GA3364@piedra> <CAP7+vJJbci1FND=AEqjxu6HK0hNjtJ3Ud_Dp_Sks_tKpn5XRaA@mail.gmail.com> Message-ID: <8761tr2zyh.fsf@uwakimon.sk.tsukuba.ac.jp> Guido van Rossum writes: > On Mon, Sep 23, 2013 at 8:17 AM, Zero Piraeus <z at etiol.net> wrote: >> On Mon, Sep 23, 2013 at 09:45:46AM -0400, Chris Lambacher wrote: >>> [...] An example of how this will be used in practice is:> >>> ? ? {% if object.state == object.state.completed %} >>> ? ? ? some html >>> ? ? {% endif %} [Zero Piraeus suggests instead:] >> ? ??{% if object.state == State.completed %} >>? ? ? ?some html >>? ? ?{% endif %} >> >> That's quite intelligible, and doesn't require anyone to know that >> an Enum member's siblings can, in your proposal, be accessed directly >> via dot notation (an unintuitive state of affairs, to me at least). > Right. The OP is just concerned that (because these are Django > templates) he will have to pass in the 'State' class as a separate > template parameter for this to work. Given your earlier description of what makes sense for class attributes, an alternative solution might be to put State-valued class attributes (constants) on DjangoObject (the object's class), like DjangoObject.completed_state = State.completed, and so on. Then you write "{% if object.state == object.completed_state %}". IIUC, you wouldn't have a problem with that? It still doesn't feel quite right, but given the limitations of a template language, it might grow on me. Another alternative would be to have attributes like 'completed' be *boolean* properties computed from a State-valued attribute, and write just "{% if object.completed %}". This actually feels good to me given it's a templating language. But I don't know if either of those is reasonable in the context. From tjreedy at udel.edu Mon Sep 23 19:52:28 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 23 Sep 2013 13:52:28 -0400 Subject: [Python-Dev] Best practice for documentation for std lib In-Reply-To: <CAP7+vJLcGYXfS5iSjmCqh4BKFmatjWoGdZi5DednagfcvcUf1w@mail.gmail.com> References: <20130922031304.GE19939@ando> <CAF-Rda_sTOiKi6c9ihFVL-N90PC-xHxOAMm1im0s-KVyy=L_Ng@mail.gmail.com> <87ioxs3iix.fsf@uwakimon.sk.tsukuba.ac.jp> <CAP7+vJJucubZtXHhnNbAcE0GWfQTnOozHK9W=uv3CWkMX15XGA@mail.gmail.com> <20130923003142.GO19939@ando> <CAP7+vJLThCjrCGGaJLMNd2Uza6wanODnipjmZ3LBZpTkub=uCg@mail.gmail.com> <l1o8ri$mjv$1@ger.gmane.org> <CAP7+vJJSvX78f2+i7_C5mNMdOQpfSD_oKqC01Z8CEHaU2krqTw@mail.gmail.com> <523FF44D.1000303@hastings.org> <CAP7+vJLcGYXfS5iSjmCqh4BKFmatjWoGdZi5DednagfcvcUf1w@mail.gmail.com> Message-ID: <l1pv4k$ls7$1@ger.gmane.org> On 9/23/2013 10:56 AM, Guido van Rossum wrote: > I think 60 is just a guideline. In stdlib .py source code I want it not > to extend beyond the 79th column (see recent PEP 8 argument). For a PEP 8 says "Limit all lines to a maximum of 79 characters. For flowing long blocks of text with fewer structural restrictions (docstrings or comments), the line length should be limited to 72 characters." I do not understand 72 instead of 79, but to me, that means 72 chars total, including indents, triple quote and period, which means the cursor not past 73. If that is not what you mean, please clarify. > typical class, where the docstring is indented 4 spaces, that leaves 72 > characters for the summary line (including a final period!). For a > method it's 68 due to the extra indent. 72 - 2 indents - """ - . = 60, which is a bit skimpy ;-). Why not the full 79 at least? -- Terry Jan Reedy From guido at python.org Mon Sep 23 19:53:34 2013 From: guido at python.org (Guido van Rossum) Date: Mon, 23 Sep 2013 10:53:34 -0700 Subject: [Python-Dev] Enum Eccentricities In-Reply-To: <8761tr2zyh.fsf@uwakimon.sk.tsukuba.ac.jp> References: <523F82C4.4060202@stoneleaf.us> <20130923024139.GA26561@piedra> <CAAXXHgKPNsXH1dgv2GeqhVC9+s1YaziCYXq0Aah-e=a3mSLAmg@mail.gmail.com> <20130923151735.GA3364@piedra> <CAP7+vJJbci1FND=AEqjxu6HK0hNjtJ3Ud_Dp_Sks_tKpn5XRaA@mail.gmail.com> <8761tr2zyh.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <CAP7+vJLbU0=_zXtimTMbTUpKSrCHk=SboAHXspvws8v1f-Xssw@mail.gmail.com> I am quickly losing interest in this -- I was only jumping in because I feared there was support for making Color.red.blue "work". I don't think I hve to worry about that any more -- what's best for Django templates is up to the template author. On Mon, Sep 23, 2013 at 10:46 AM, Stephen J. Turnbull <stephen at xemacs.org>wrote: > Guido van Rossum writes: > > On Mon, Sep 23, 2013 at 8:17 AM, Zero Piraeus <z at etiol.net> wrote: > >> On Mon, Sep 23, 2013 at 09:45:46AM -0400, Chris Lambacher wrote: > > >>> [...] An example of how this will be used in practice is:> > >>> {% if object.state == object.state.completed %} > >>> some html > >>> {% endif %} > > [Zero Piraeus suggests instead:] > > >> {% if object.state == State.completed %} > >> some html > >> {% endif %} > >> > >> That's quite intelligible, and doesn't require anyone to know that > >> an Enum member's siblings can, in your proposal, be accessed directly > >> via dot notation (an unintuitive state of affairs, to me at least). > > > Right. The OP is just concerned that (because these are Django > > templates) he will have to pass in the 'State' class as a separate > > template parameter for this to work. > > Given your earlier description of what makes sense for class > attributes, an alternative solution might be to put State-valued class > attributes (constants) on DjangoObject (the object's class), like > DjangoObject.completed_state = State.completed, and so on. Then you > write "{% if object.state == object.completed_state %}". > > IIUC, you wouldn't have a problem with that? It still doesn't feel > quite right, but given the limitations of a template language, it > might grow on me. > > Another alternative would be to have attributes like 'completed' be > *boolean* properties computed from a State-valued attribute, and write > just "{% if object.completed %}". This actually feels good to me > given it's a templating language. > > But I don't know if either of those is reasonable in the context. > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130923/d855f562/attachment.html> From guido at python.org Mon Sep 23 19:59:06 2013 From: guido at python.org (Guido van Rossum) Date: Mon, 23 Sep 2013 10:59:06 -0700 Subject: [Python-Dev] Best practice for documentation for std lib In-Reply-To: <l1pv4k$ls7$1@ger.gmane.org> References: <20130922031304.GE19939@ando> <CAF-Rda_sTOiKi6c9ihFVL-N90PC-xHxOAMm1im0s-KVyy=L_Ng@mail.gmail.com> <87ioxs3iix.fsf@uwakimon.sk.tsukuba.ac.jp> <CAP7+vJJucubZtXHhnNbAcE0GWfQTnOozHK9W=uv3CWkMX15XGA@mail.gmail.com> <20130923003142.GO19939@ando> <CAP7+vJLThCjrCGGaJLMNd2Uza6wanODnipjmZ3LBZpTkub=uCg@mail.gmail.com> <l1o8ri$mjv$1@ger.gmane.org> <CAP7+vJJSvX78f2+i7_C5mNMdOQpfSD_oKqC01Z8CEHaU2krqTw@mail.gmail.com> <523FF44D.1000303@hastings.org> <CAP7+vJLcGYXfS5iSjmCqh4BKFmatjWoGdZi5DednagfcvcUf1w@mail.gmail.com> <l1pv4k$ls7$1@ger.gmane.org> Message-ID: <CAP7+vJ+wJX78hA7OzD48k-uk=DKiTks8vs5WHF-E-frQ6eMRFw@mail.gmail.com> On Mon, Sep 23, 2013 at 10:52 AM, Terry Reedy <tjreedy at udel.edu> wrote: > On 9/23/2013 10:56 AM, Guido van Rossum wrote: > > I think 60 is just a guideline. In stdlib .py source code I want it not >> to extend beyond the 79th column (see recent PEP 8 argument). For a >> > > PEP 8 says "Limit all lines to a maximum of 79 characters. > > For flowing long blocks of text with fewer structural restrictions > (docstrings or comments), the line length should be limited to 72 > characters." > > I do not understand 72 instead of 79, but to me, that means 72 chars > total, including indents, triple quote and period, which means the cursor > not past 73. If that is not what you mean, please clarify. This is partly based on Emacs defaults (when I hit ESC-q, it reflows a paragraph using the 72 char limit), partly on old punched card conventions (columns 73-80 were reserved for sequence numbers), partly just to limit the line length to of *columns* of text to something a bit narrower than the full line width (i.e. 80 chars) to improve readability. But the PEP specifically qualifies this with "for flowing long blocks of text" which is meant to apply only to blocks of reflowable text that don't fit on a line. If it fits on a line, you can go to 79. > typical class, where the docstring is indented 4 spaces, that leaves 72 > characters for the summary line (including a final period!). For a > method it's 68 due to the extra indent. > 72 - 2 indents - """ - . = 60, which is a bit skimpy ;-). Why not the full > 79 at least? > For the summary line, that's fine (and actually how I interpret the PEP). -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130923/22e2c0be/attachment-0001.html> From tjreedy at udel.edu Mon Sep 23 20:08:26 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 23 Sep 2013 14:08:26 -0400 Subject: [Python-Dev] Best practice for documentation for std lib In-Reply-To: <CANc-5UzYUtta=4Q3c7n-LXdcYBnWrh5VYdMwSNPJ=5a0HuVfig@mail.gmail.com> References: <20130922031304.GE19939@ando> <20130922102046.764abea5@fsol> <20130922125323.GI19939@ando> <CAP1=2W7ymWcx_Yj6Xsb0Ty1Wj5pUf+_1XtqhPAVyRKWgiQ37mg@mail.gmail.com> <524025B3.7020305@livinglogic.de> <CAP7+vJLDo9KkdzzQfnJe_bTzunAP6+gBznY1Dx5pvxBqva0YjQ@mail.gmail.com> <CANc-5UzYUtta=4Q3c7n-LXdcYBnWrh5VYdMwSNPJ=5a0HuVfig@mail.gmail.com> Message-ID: <l1q02i$16h$1@ger.gmane.org> On 9/23/2013 11:18 AM, Skip Montanaro wrote: >>> It would be great if the docstring contained a link to the online >>> documentation. >> >> That would have to be a feature of help(), not hardcoded in each docstring. > > That *is* a feature of the help function: > > Help on built-in module sys: > >>>> help(sys) > NAME > sys > > FILE > (built-in) > > MODULE DOCS > http://docs.python.org/library/sys On 3.3.2 and 3.4.0: Help on built-in module sys: NAME sys MODULE REFERENCE http://docs.python.org/3.3/library/sys ... FILE (built-in) On Windows, one will not see the top of the help(sys) output unless one either uses Idle or increases the number of lines in the console window from 300. -- Terry Jan Reedy From tjreedy at udel.edu Mon Sep 23 20:27:25 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 23 Sep 2013 14:27:25 -0400 Subject: [Python-Dev] Revert #12085 fix for __del__ attribute error message In-Reply-To: <20130923162356.2135F25095A@webabinitio.net> References: <l1l2bm$1bp$1@ger.gmane.org> <20130921221524.55D4225007F@webabinitio.net> <l1lbkl$k0g$1@ger.gmane.org> <CAP7+vJLHO12ZNwX4UZ0c_BF9pr+xxxLZfy2dGjBcMVenGow9vQ@mail.gmail.com> <l1n9o6$ap3$1@ger.gmane.org> <CAP7+vJL+0XEBpOnVbTpZJNLzahgm8EkyuXdGdhE27TVwHu4GAg@mail.gmail.com> <l1nkdh$lh7$1@ger.gmane.org> <CADiSq7eGpFtPbRc8BmiP2H9V2PVMma3ycX3Ky_MJ-F+TpEiU6w@mail.gmail.com> <CAP7+vJKAP5R+2R++z8+DQjSW6EVycoczB2gWeo759-NG4G6Veg@mail.gmail.com> <523FDB8F.1080207@canterbury.ac.nz> <CADiSq7caBSr4ujFo0HBYGF91SJp2NcP-_SreBqad-=KpcZdeCw@mail.gmail.com> <20130923104500.5645fdbb@pitrou.net> <CADiSq7dJAdStAzPYijQTC_fC6_01Jkt4kidSh+YDxWNNv_0tOA@mail.gmail.com> <20130923172245.0cbaec81@fsol> <20130923162356.2135F25095A@webabinitio.net> Message-ID: <l1q164$fgi$1@ger.gmane.org> On 9/23/2013 12:23 PM, R. David Murray wrote: > On Mon, 23 Sep 2013 17:22:45 +0200, Antoine Pitrou <solipsis at pitrou.net> wrote: >> On Mon, 23 Sep 2013 18:51:04 +1000 >> Nick Coghlan <ncoghlan at gmail.com> wrote: >>> On 23 September 2013 18:45, Antoine Pitrou <solipsis at pitrou.net> wrote: >>>> Le Mon, 23 Sep 2013 18:17:51 +1000, >>>> Nick Coghlan <ncoghlan at gmail.com> a ?crit : >>>>> >>>>> Here's what I suggest changing that error to: >>>>> >>>>>>>> del x >>>>> Unraisable exception suppressed when calling <bound method C.__del__ >>>>> of <__main__.C object at 0x7f98b8b61538>> >>>>> Traceback (most recent call last): >>>>> File "<stdin>", line 3, in __del__ >>>>> RuntimeError: Going away now >>>> >>>> Why not simply "Exception automatically caught in <bound method >>>> C.__del__> [...]" ? >>> >>> It only answers the "what" (i.e. the exception was automatically >>> caught), without addressing the "why" (i.e. because there wasn't >>> anything else useful the interpreter could do with it) >> >> Yes, but I agree with Greg that "unraisable" is wrong. After all, it >> was raised, and it can even be caught by the programmer (inside >> __del__). > > Would it work to say "Asynchronous exception suppressed..."? It's > not-entirely-precise, As in the example above ('del x'). > but it's less imprecise than "unraisable". How 'troublesome'? That is always accurate, as proven by this thread. We really need an "Understanding Exceptions" HOWTO, and I expect we will get one. So I agree with Nick that something easily searched would be good. -- Terry Jan Reedy From tjreedy at udel.edu Mon Sep 23 21:01:56 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 23 Sep 2013 15:01:56 -0400 Subject: [Python-Dev] Best practice for documentation for std lib In-Reply-To: <CAP7+vJJSvX78f2+i7_C5mNMdOQpfSD_oKqC01Z8CEHaU2krqTw@mail.gmail.com> References: <20130922031304.GE19939@ando> <CAF-Rda_sTOiKi6c9ihFVL-N90PC-xHxOAMm1im0s-KVyy=L_Ng@mail.gmail.com> <87ioxs3iix.fsf@uwakimon.sk.tsukuba.ac.jp> <CAP7+vJJucubZtXHhnNbAcE0GWfQTnOozHK9W=uv3CWkMX15XGA@mail.gmail.com> <20130923003142.GO19939@ando> <CAP7+vJLThCjrCGGaJLMNd2Uza6wanODnipjmZ3LBZpTkub=uCg@mail.gmail.com> <l1o8ri$mjv$1@ger.gmane.org> <CAP7+vJJSvX78f2+i7_C5mNMdOQpfSD_oKqC01Z8CEHaU2krqTw@mail.gmail.com> Message-ID: <l1q36s$99l$1@ger.gmane.org> On 9/22/2013 10:44 PM, Guido van Rossum wrote: > Glad you like it. I still do, too, but I've given up hope to convince > all core developers to stick to this style. :-( >[me] ('Return' rather than 'Returns' is the current convention.) > That's actually a religious argument which in the stdlib takes no strict > position -- a quick grep shows that both are used, although 'Return' is > more frequent by a 5-to-1 margin. In the .rst docs, 'Return' versus 'Returns', exact uppercase word match, is a little over 3 to 1. I am sure I have seen 'Return' and similiar directive forms ('Print', 'Store', 'Compare', etc) recommended as current doc style, as prefered by the current doc crew. > IIRC in the Java world you *have* to > use 'Returns', but I don't buy the argument from nit-picky grammarians > that leads to this rule. (It's something about the documentation not > being a command. But English is more flexible than that.) My take is that 'Returns' describes to the programmer what the function (interpreter) does, while 'Return' says what the programmer says to the interpreter when using the function. I strongly prefer the directive form. Why? For one thing, *because* it is different from normal descriptive text, such as the first sentence of this paragraph. For another, the descriptive form seems addressed to me as code reader while the directive form seems addressed to me as code writer. For me, the latter seems more energizing. -- Terry Jan Reedy From python at mrabarnett.plus.com Mon Sep 23 21:11:35 2013 From: python at mrabarnett.plus.com (MRAB) Date: Mon, 23 Sep 2013 20:11:35 +0100 Subject: [Python-Dev] Best practice for documentation for std lib In-Reply-To: <l1q36s$99l$1@ger.gmane.org> References: <20130922031304.GE19939@ando> <CAF-Rda_sTOiKi6c9ihFVL-N90PC-xHxOAMm1im0s-KVyy=L_Ng@mail.gmail.com> <87ioxs3iix.fsf@uwakimon.sk.tsukuba.ac.jp> <CAP7+vJJucubZtXHhnNbAcE0GWfQTnOozHK9W=uv3CWkMX15XGA@mail.gmail.com> <20130923003142.GO19939@ando> <CAP7+vJLThCjrCGGaJLMNd2Uza6wanODnipjmZ3LBZpTkub=uCg@mail.gmail.com> <l1o8ri$mjv$1@ger.gmane.org> <CAP7+vJJSvX78f2+i7_C5mNMdOQpfSD_oKqC01Z8CEHaU2krqTw@mail.gmail.com> <l1q36s$99l$1@ger.gmane.org> Message-ID: <52409267.20206@mrabarnett.plus.com> On 23/09/2013 20:01, Terry Reedy wrote: > On 9/22/2013 10:44 PM, Guido van Rossum wrote: > >> Glad you like it. I still do, too, but I've given up hope to convince >> all core developers to stick to this style. :-( > > >[me] ('Return' rather than 'Returns' is the current convention.) > >> That's actually a religious argument which in the stdlib takes no strict >> position -- a quick grep shows that both are used, although 'Return' is >> more frequent by a 5-to-1 margin. > > In the .rst docs, 'Return' versus 'Returns', exact uppercase word match, > is a little over 3 to 1. I am sure I have seen 'Return' and similiar > directive forms ('Print', 'Store', 'Compare', etc) recommended as > current doc style, as prefered by the current doc crew. > > > IIRC in the Java world you *have* to >> use 'Returns', but I don't buy the argument from nit-picky grammarians >> that leads to this rule. (It's something about the documentation not >> being a command. But English is more flexible than that.) > > My take is that 'Returns' describes to the programmer what the function > (interpreter) does, while 'Return' says what the programmer says to the > interpreter when using the function. I strongly prefer the directive > form. Why? For one thing, *because* it is different from normal > descriptive text, such as the first sentence of this paragraph. For > another, the descriptive form seems addressed to me as code reader while > the directive form seems addressed to me as code writer. For me, the > latter seems more energizing. > <pedantic>I think you mean "imperative" vs "indicative".</pedantic> From alexander.belopolsky at gmail.com Mon Sep 23 21:18:01 2013 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Mon, 23 Sep 2013 15:18:01 -0400 Subject: [Python-Dev] Best practice for documentation for std lib In-Reply-To: <l1q36s$99l$1@ger.gmane.org> References: <20130922031304.GE19939@ando> <CAF-Rda_sTOiKi6c9ihFVL-N90PC-xHxOAMm1im0s-KVyy=L_Ng@mail.gmail.com> <87ioxs3iix.fsf@uwakimon.sk.tsukuba.ac.jp> <CAP7+vJJucubZtXHhnNbAcE0GWfQTnOozHK9W=uv3CWkMX15XGA@mail.gmail.com> <20130923003142.GO19939@ando> <CAP7+vJLThCjrCGGaJLMNd2Uza6wanODnipjmZ3LBZpTkub=uCg@mail.gmail.com> <l1o8ri$mjv$1@ger.gmane.org> <CAP7+vJJSvX78f2+i7_C5mNMdOQpfSD_oKqC01Z8CEHaU2krqTw@mail.gmail.com> <l1q36s$99l$1@ger.gmane.org> Message-ID: <CAP7h-xZ-E_3--JrQGE+htXZStagmBjHmozW6v6jEbiXUicQ8fQ@mail.gmail.com> On Mon, Sep 23, 2013 at 3:01 PM, Terry Reedy <tjreedy at udel.edu> wrote: > > IIRC in the Java world you *have* to > >> use 'Returns', but I don't buy the argument from nit-picky grammarians >> that leads to this rule. (It's something about the documentation not >> being a command. But English is more flexible than that.) >> > > My take is that 'Returns' describes to the programmer what the function > (interpreter) does, while 'Return' says what the programmer says to the > interpreter when using the function. I strongly prefer the directive form. +1 I don't think "Returns bar." is a valid English sentence because it lacks subject. I would not mind def foo(): """returns bar""" which I would read as "Function foo() returns bar," but in this case "returns" should be in lower case. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130923/3264fc0a/attachment.html> From fred at fdrake.net Mon Sep 23 21:37:54 2013 From: fred at fdrake.net (Fred Drake) Date: Mon, 23 Sep 2013 15:37:54 -0400 Subject: [Python-Dev] Best practice for documentation for std lib In-Reply-To: <l1q36s$99l$1@ger.gmane.org> References: <20130922031304.GE19939@ando> <CAF-Rda_sTOiKi6c9ihFVL-N90PC-xHxOAMm1im0s-KVyy=L_Ng@mail.gmail.com> <87ioxs3iix.fsf@uwakimon.sk.tsukuba.ac.jp> <CAP7+vJJucubZtXHhnNbAcE0GWfQTnOozHK9W=uv3CWkMX15XGA@mail.gmail.com> <20130923003142.GO19939@ando> <CAP7+vJLThCjrCGGaJLMNd2Uza6wanODnipjmZ3LBZpTkub=uCg@mail.gmail.com> <l1o8ri$mjv$1@ger.gmane.org> <CAP7+vJJSvX78f2+i7_C5mNMdOQpfSD_oKqC01Z8CEHaU2krqTw@mail.gmail.com> <l1q36s$99l$1@ger.gmane.org> Message-ID: <CAFT4OTFEpNcYAT0Z9ZkSbYodESNpoQeQVorAQWo83E9Rv2kihA@mail.gmail.com> On Mon, Sep 23, 2013 at 3:01 PM, Terry Reedy <tjreedy at udel.edu> wrote: > 'Return' versus 'Returns', exact uppercase word match, is a little over 3 to > 1. I am sure I have seen 'Return' and similiar directive forms ('Print', > 'Store', 'Compare', etc) recommended as current doc style, as prefered by > the current doc crew. I don't know about the current crew, but this dates way back to Guido's initial LaTeX documentation, and Guido's strong preference on this. -Fred -- Fred L. Drake, Jr. <fred at fdrake.net> "A storm broke loose in my mind." --Albert Einstein From ncoghlan at gmail.com Mon Sep 23 23:19:14 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 24 Sep 2013 07:19:14 +1000 Subject: [Python-Dev] Revert #12085 fix for __del__ attribute error message In-Reply-To: <20130923172245.0cbaec81@fsol> References: <l1l2bm$1bp$1@ger.gmane.org> <20130921221524.55D4225007F@webabinitio.net> <l1lbkl$k0g$1@ger.gmane.org> <CAP7+vJLHO12ZNwX4UZ0c_BF9pr+xxxLZfy2dGjBcMVenGow9vQ@mail.gmail.com> <l1n9o6$ap3$1@ger.gmane.org> <CAP7+vJL+0XEBpOnVbTpZJNLzahgm8EkyuXdGdhE27TVwHu4GAg@mail.gmail.com> <l1nkdh$lh7$1@ger.gmane.org> <CADiSq7eGpFtPbRc8BmiP2H9V2PVMma3ycX3Ky_MJ-F+TpEiU6w@mail.gmail.com> <CAP7+vJKAP5R+2R++z8+DQjSW6EVycoczB2gWeo759-NG4G6Veg@mail.gmail.com> <523FDB8F.1080207@canterbury.ac.nz> <CADiSq7caBSr4ujFo0HBYGF91SJp2NcP-_SreBqad-=KpcZdeCw@mail.gmail.com> <20130923104500.5645fdbb@pitrou.net> <CADiSq7dJAdStAzPYijQTC_fC6_01Jkt4kidSh+YDxWNNv_0tOA@mail.gmail.com> <20130923172245.0cbaec81@fsol> Message-ID: <CADiSq7deVSSxwSVwq91DfNaiJwLdN=WyVzFfE7uASkeSpeMUsw@mail.gmail.com> On 24 Sep 2013 01:24, "Antoine Pitrou" <solipsis at pitrou.net> wrote: > > On Mon, 23 Sep 2013 18:51:04 +1000 > Nick Coghlan <ncoghlan at gmail.com> wrote: > > On 23 September 2013 18:45, Antoine Pitrou <solipsis at pitrou.net> wrote: > > > Le Mon, 23 Sep 2013 18:17:51 +1000, > > > Nick Coghlan <ncoghlan at gmail.com> a ?crit : > > >> > > >> Here's what I suggest changing that error to: > > >> > > >> >>> del x > > >> Unraisable exception suppressed when calling <bound method C.__del__ > > >> of <__main__.C object at 0x7f98b8b61538>> > > >> Traceback (most recent call last): > > >> File "<stdin>", line 3, in __del__ > > >> RuntimeError: Going away now > > > > > > Why not simply "Exception automatically caught in <bound method > > > C.__del__> [...]" ? > > > > It only answers the "what" (i.e. the exception was automatically > > caught), without addressing the "why" (i.e. because there wasn't > > anything else useful the interpreter could do with it) > > Yes, but I agree with Greg that "unraisable" is wrong. After all, it > was raised, and it can even be caught by the programmer (inside > __del__). The word doesn't literally mean the exception itself was unraisable. It means it was raised, we caught it and we're writing it to stderr because we *can't raise it again*. The relevant C API function is just called "PyErr_WriteUnraisable", not "PyErr_WriteUnraisableButThatIsTechnicallyWrongSinceItWasAlreadyRaisedAndWeJustCaughtItAndAreNowReportingItToStdErr". Cheers, Nick. > > Regards > > Antoine. > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130924/3d93cb57/attachment.html> From solipsis at pitrou.net Mon Sep 23 23:35:07 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 23 Sep 2013 23:35:07 +0200 Subject: [Python-Dev] Revert #12085 fix for __del__ attribute error message In-Reply-To: <CADiSq7deVSSxwSVwq91DfNaiJwLdN=WyVzFfE7uASkeSpeMUsw@mail.gmail.com> References: <l1l2bm$1bp$1@ger.gmane.org> <20130921221524.55D4225007F@webabinitio.net> <l1lbkl$k0g$1@ger.gmane.org> <CAP7+vJLHO12ZNwX4UZ0c_BF9pr+xxxLZfy2dGjBcMVenGow9vQ@mail.gmail.com> <l1n9o6$ap3$1@ger.gmane.org> <CAP7+vJL+0XEBpOnVbTpZJNLzahgm8EkyuXdGdhE27TVwHu4GAg@mail.gmail.com> <l1nkdh$lh7$1@ger.gmane.org> <CADiSq7eGpFtPbRc8BmiP2H9V2PVMma3ycX3Ky_MJ-F+TpEiU6w@mail.gmail.com> <CAP7+vJKAP5R+2R++z8+DQjSW6EVycoczB2gWeo759-NG4G6Veg@mail.gmail.com> <523FDB8F.1080207@canterbury.ac.nz> <CADiSq7caBSr4ujFo0HBYGF91SJp2NcP-_SreBqad-=KpcZdeCw@mail.gmail.com> <20130923104500.5645fdbb@pitrou.net> <CADiSq7dJAdStAzPYijQTC_fC6_01Jkt4kidSh+YDxWNNv_0tOA@mail.gmail.com> <20130923172245.0cbaec81@fsol> <CADiSq7deVSSxwSVwq91DfNaiJwLdN=WyVzFfE7uASkeSpeMUsw@mail.gmail.com> Message-ID: <20130923233507.4e5ebb9f@fsol> On Tue, 24 Sep 2013 07:19:14 +1000 Nick Coghlan <ncoghlan at gmail.com> wrote: > On 24 Sep 2013 01:24, "Antoine Pitrou" <solipsis at pitrou.net> wrote: > > > > On Mon, 23 Sep 2013 18:51:04 +1000 > > Nick Coghlan <ncoghlan at gmail.com> wrote: > > > On 23 September 2013 18:45, Antoine Pitrou <solipsis at pitrou.net> wrote: > > > > Le Mon, 23 Sep 2013 18:17:51 +1000, > > > > Nick Coghlan <ncoghlan at gmail.com> a ?crit : > > > >> > > > >> Here's what I suggest changing that error to: > > > >> > > > >> >>> del x > > > >> Unraisable exception suppressed when calling <bound method C.__del__ > > > >> of <__main__.C object at 0x7f98b8b61538>> > > > >> Traceback (most recent call last): > > > >> File "<stdin>", line 3, in __del__ > > > >> RuntimeError: Going away now > > > > > > > > Why not simply "Exception automatically caught in <bound method > > > > C.__del__> [...]" ? > > > > > > It only answers the "what" (i.e. the exception was automatically > > > caught), without addressing the "why" (i.e. because there wasn't > > > anything else useful the interpreter could do with it) > > > > Yes, but I agree with Greg that "unraisable" is wrong. After all, it > > was raised, and it can even be caught by the programmer (inside > > __del__). > > The word doesn't literally mean the exception itself was unraisable. It > means it was raised, we caught it and we're writing it to stderr because we > *can't raise it again*. But that's because you already know what it's supposed to convey. The average user doesn't, and only sees "unraisable". > The relevant C API function is just called "PyErr_WriteUnraisable", not > "PyErr_WriteUnraisableButThatIsTechnicallyWrongSinceItWasAlreadyRaisedAndWeJustCaughtItAndAreNowReportingItToStdErr". "PyErr_WriteUnraisable" is right at the point where it used: I can't raise that exception at this point, therefore I call PyErr_WriteUnraisable. However, from the point of view of the user reading the traceback, the exception *was* raised. Regards Antoine. From ethan at stoneleaf.us Mon Sep 23 23:26:17 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 23 Sep 2013 14:26:17 -0700 Subject: [Python-Dev] Revert #12085 fix for __del__ attribute error message In-Reply-To: <CADiSq7deVSSxwSVwq91DfNaiJwLdN=WyVzFfE7uASkeSpeMUsw@mail.gmail.com> References: <l1l2bm$1bp$1@ger.gmane.org> <20130921221524.55D4225007F@webabinitio.net> <l1lbkl$k0g$1@ger.gmane.org> <CAP7+vJLHO12ZNwX4UZ0c_BF9pr+xxxLZfy2dGjBcMVenGow9vQ@mail.gmail.com> <l1n9o6$ap3$1@ger.gmane.org> <CAP7+vJL+0XEBpOnVbTpZJNLzahgm8EkyuXdGdhE27TVwHu4GAg@mail.gmail.com> <l1nkdh$lh7$1@ger.gmane.org> <CADiSq7eGpFtPbRc8BmiP2H9V2PVMma3ycX3Ky_MJ-F+TpEiU6w@mail.gmail.com> <CAP7+vJKAP5R+2R++z8+DQjSW6EVycoczB2gWeo759-NG4G6Veg@mail.gmail.com> <523FDB8F.1080207@canterbury.ac.nz> <CADiSq7caBSr4ujFo0HBYGF91SJp2NcP-_SreBqad-=KpcZdeCw@mail.gmail.com> <20130923104500.5645fdbb@pitrou.net> <CADiSq7dJAdStAzPYijQTC_fC6_01Jkt4kidSh+YDxWNNv_0tOA@mail.gmail.com> <20130923172245.0cbaec81@fsol> <CADiSq7deVSSxwSVwq91DfNaiJwLdN=WyVzFfE7uASkeSpeMUsw@mail.gmail.com> Message-ID: <5240B1F9.7020602@stoneleaf.us> On 09/23/2013 02:19 PM, Nick Coghlan wrote: > > The relevant C API function is just called "PyErr_WriteUnraisable", not > "PyErr_WriteUnraisableButThatIsTechnicallyWrongSinceItWasAlreadyRaisedAndWeJustCaughtItAndAreNowReportingItToStdErr". Wow. How many legs does that HumptyCamel have, anyway? ;) -- ~Ethan~ From greg.ewing at canterbury.ac.nz Tue Sep 24 00:07:45 2013 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 24 Sep 2013 10:07:45 +1200 Subject: [Python-Dev] Enum Eccentricities In-Reply-To: <20130923151623.GB7989@ando> References: <523F82C4.4060202@stoneleaf.us> <20130923024139.GA26561@piedra> <CAAXXHgKPNsXH1dgv2GeqhVC9+s1YaziCYXq0Aah-e=a3mSLAmg@mail.gmail.com> <CAP7+vJKisYvdOug-fmR_SZEET7icHGjEx2pFYYHXyZHrhjWhzw@mail.gmail.com> <20130923151623.GB7989@ando> Message-ID: <5240BBB1.5010900@canterbury.ac.nz> Steven D'Aprano wrote: > It might not be a rule, but it's certainly the norm. I reckon that class > attributes that aren't accessible from the instance are significantly > more surprising than Color.red.blue. There are really two different kinds of things that we refer to as "class attributes". One is things that really are attributes of the class itself, and the other is things that are meant to serve as default or shared instance attributes. The confusion comes in because we use the same terminology for both, and because Python doesn't provide any straightforward way of creating user-defined class-only attributes, so shared attributes tend to get abused for that purpose. Then when someone comes along and creates a true class-only attribute, people get all surprised and complain about it. They shouldn't, IMO. -- Greg From greg.ewing at canterbury.ac.nz Tue Sep 24 00:11:54 2013 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 24 Sep 2013 10:11:54 +1200 Subject: [Python-Dev] Revert #12085 fix for __del__ attribute error message In-Reply-To: <20130923172245.0cbaec81@fsol> References: <l1l2bm$1bp$1@ger.gmane.org> <20130921221524.55D4225007F@webabinitio.net> <l1lbkl$k0g$1@ger.gmane.org> <CAP7+vJLHO12ZNwX4UZ0c_BF9pr+xxxLZfy2dGjBcMVenGow9vQ@mail.gmail.com> <l1n9o6$ap3$1@ger.gmane.org> <CAP7+vJL+0XEBpOnVbTpZJNLzahgm8EkyuXdGdhE27TVwHu4GAg@mail.gmail.com> <l1nkdh$lh7$1@ger.gmane.org> <CADiSq7eGpFtPbRc8BmiP2H9V2PVMma3ycX3Ky_MJ-F+TpEiU6w@mail.gmail.com> <CAP7+vJKAP5R+2R++z8+DQjSW6EVycoczB2gWeo759-NG4G6Veg@mail.gmail.com> <523FDB8F.1080207@canterbury.ac.nz> <CADiSq7caBSr4ujFo0HBYGF91SJp2NcP-_SreBqad-=KpcZdeCw@mail.gmail.com> <20130923104500.5645fdbb@pitrou.net> <CADiSq7dJAdStAzPYijQTC_fC6_01Jkt4kidSh+YDxWNNv_0tOA@mail.gmail.com> <20130923172245.0cbaec81@fsol> Message-ID: <5240BCAA.5030604@canterbury.ac.nz> Antoine Pitrou wrote: > Yes, but I agree with Greg that "unraisable" is wrong. After all, it > was raised, and it can even be caught by the programmer (inside > __del__). How about something like "Uncaught exception in __del__ method ignored"? It explains fairly clearly what has happened, and also indicates what do do about it -- catch it in the __del__ method. -- Greg From ethan at stoneleaf.us Mon Sep 23 23:38:48 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 23 Sep 2013 14:38:48 -0700 Subject: [Python-Dev] Revert #12085 fix for __del__ attribute error message In-Reply-To: <20130923233507.4e5ebb9f@fsol> References: <l1l2bm$1bp$1@ger.gmane.org> <20130921221524.55D4225007F@webabinitio.net> <l1lbkl$k0g$1@ger.gmane.org> <CAP7+vJLHO12ZNwX4UZ0c_BF9pr+xxxLZfy2dGjBcMVenGow9vQ@mail.gmail.com> <l1n9o6$ap3$1@ger.gmane.org> <CAP7+vJL+0XEBpOnVbTpZJNLzahgm8EkyuXdGdhE27TVwHu4GAg@mail.gmail.com> <l1nkdh$lh7$1@ger.gmane.org> <CADiSq7eGpFtPbRc8BmiP2H9V2PVMma3ycX3Ky_MJ-F+TpEiU6w@mail.gmail.com> <CAP7+vJKAP5R+2R++z8+DQjSW6EVycoczB2gWeo759-NG4G6Veg@mail.gmail.com> <523FDB8F.1080207@canterbury.ac.nz> <CADiSq7caBSr4ujFo0HBYGF91SJp2NcP-_SreBqad-=KpcZdeCw@mail.gmail.com> <20130923104500.5645fdbb@pitrou.net> <CADiSq7dJAdStAzPYijQTC_fC6_01Jkt4kidSh+YDxWNNv_0tOA@mail.gmail.com> <20130923172245.0cbaec81@fsol> <CADiSq7deVSSxwSVwq91DfNaiJwLdN=WyVzFfE7uASkeSpeMUsw@mail.gmail.com> <20130923233507.4e5ebb9f@fsol> Message-ID: <5240B4E8.6020701@stoneleaf.us> On 09/23/2013 02:35 PM, Antoine Pitrou wrote: > On Tue, 24 Sep 2013 07:19:14 +1000 > Nick Coghlan <ncoghlan at gmail.com> wrote: >> On 24 Sep 2013 01:24, "Antoine Pitrou" <solipsis at pitrou.net> wrote: >>> >>> On Mon, 23 Sep 2013 18:51:04 +1000 >>> Nick Coghlan <ncoghlan at gmail.com> wrote: >>>> On 23 September 2013 18:45, Antoine Pitrou <solipsis at pitrou.net> wrote: >>>>> Le Mon, 23 Sep 2013 18:17:51 +1000, >>>>> Nick Coghlan <ncoghlan at gmail.com> a ?crit : >>>>>> >>>>>> Here's what I suggest changing that error to: >>>>>> >>>>>>>>> del x >>>>>> Unraisable exception suppressed when calling <bound method C.__del__ >>>>>> of <__main__.C object at 0x7f98b8b61538>> >>>>>> Traceback (most recent call last): >>>>>> File "<stdin>", line 3, in __del__ >>>>>> RuntimeError: Going away now >>>>> >>>>> Why not simply "Exception automatically caught in <bound method >>>>> C.__del__> [...]" ? >>>> >>>> It only answers the "what" (i.e. the exception was automatically >>>> caught), without addressing the "why" (i.e. because there wasn't >>>> anything else useful the interpreter could do with it) >>> >>> Yes, but I agree with Greg that "unraisable" is wrong. After all, it >>> was raised, and it can even be caught by the programmer (inside >>> __del__). >> >> The word doesn't literally mean the exception itself was unraisable. It >> means it was raised, we caught it and we're writing it to stderr because we >> *can't raise it again*. > > But that's because you already know what it's supposed to convey. The > average user doesn't, and only sees "unraisable". All the more reason to have text in the error message that is easily searchable. -- ~Ethan~ From solipsis at pitrou.net Tue Sep 24 00:29:55 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 24 Sep 2013 00:29:55 +0200 Subject: [Python-Dev] Revert #12085 fix for __del__ attribute error message References: <l1l2bm$1bp$1@ger.gmane.org> <20130921221524.55D4225007F@webabinitio.net> <l1lbkl$k0g$1@ger.gmane.org> <CAP7+vJLHO12ZNwX4UZ0c_BF9pr+xxxLZfy2dGjBcMVenGow9vQ@mail.gmail.com> <l1n9o6$ap3$1@ger.gmane.org> <CAP7+vJL+0XEBpOnVbTpZJNLzahgm8EkyuXdGdhE27TVwHu4GAg@mail.gmail.com> <l1nkdh$lh7$1@ger.gmane.org> <CADiSq7eGpFtPbRc8BmiP2H9V2PVMma3ycX3Ky_MJ-F+TpEiU6w@mail.gmail.com> <CAP7+vJKAP5R+2R++z8+DQjSW6EVycoczB2gWeo759-NG4G6Veg@mail.gmail.com> <523FDB8F.1080207@canterbury.ac.nz> <CADiSq7caBSr4ujFo0HBYGF91SJp2NcP-_SreBqad-=KpcZdeCw@mail.gmail.com> <20130923104500.5645fdbb@pitrou.net> <CADiSq7dJAdStAzPYijQTC_fC6_01Jkt4kidSh+YDxWNNv_0tOA@mail.gmail.com> <20130923172245.0cbaec81@fsol> <CADiSq7deVSSxwSVwq91DfNaiJwLdN=WyVzFfE7uASkeSpeMUsw@mail.gmail.com> <20130923233507.4e5ebb9f@fsol> <5240B4E8.6020701@stoneleaf.us> Message-ID: <20130924002955.49a99f9f@fsol> On Mon, 23 Sep 2013 14:38:48 -0700 Ethan Furman <ethan at stoneleaf.us> wrote: > > But that's because you already know what it's supposed to convey. The > > average user doesn't, and only sees "unraisable". > > All the more reason to have text in the error message that is easily searchable. Then I propose "CA6yuaYV6dygPfJRxUrhtg". It should be easily searchable ;-) Seriously, "easily searchable" is a rather weak criterion for an error message. It should be easily understandable and non-misleading first. Regards Antoine. From python at mrabarnett.plus.com Tue Sep 24 01:04:28 2013 From: python at mrabarnett.plus.com (MRAB) Date: Tue, 24 Sep 2013 00:04:28 +0100 Subject: [Python-Dev] Revert #12085 fix for __del__ attribute error message In-Reply-To: <CADiSq7deVSSxwSVwq91DfNaiJwLdN=WyVzFfE7uASkeSpeMUsw@mail.gmail.com> References: <l1l2bm$1bp$1@ger.gmane.org> <20130921221524.55D4225007F@webabinitio.net> <l1lbkl$k0g$1@ger.gmane.org> <CAP7+vJLHO12ZNwX4UZ0c_BF9pr+xxxLZfy2dGjBcMVenGow9vQ@mail.gmail.com> <l1n9o6$ap3$1@ger.gmane.org> <CAP7+vJL+0XEBpOnVbTpZJNLzahgm8EkyuXdGdhE27TVwHu4GAg@mail.gmail.com> <l1nkdh$lh7$1@ger.gmane.org> <CADiSq7eGpFtPbRc8BmiP2H9V2PVMma3ycX3Ky_MJ-F+TpEiU6w@mail.gmail.com> <CAP7+vJKAP5R+2R++z8+DQjSW6EVycoczB2gWeo759-NG4G6Veg@mail.gmail.com> <523FDB8F.1080207@canterbury.ac.nz> <CADiSq7caBSr4ujFo0HBYGF91SJp2NcP-_SreBqad-=KpcZdeCw@mail.gmail.com> <20130923104500.5645fdbb@pitrou.net> <CADiSq7dJAdStAzPYijQTC_fC6_01Jkt4kidSh+YDxWNNv_0tOA@mail.gmail.com> <20130923172245.0cbaec81@fsol> <CADiSq7deVSSxwSVwq91DfNaiJwLdN=WyVzFfE7uASkeSpeMUsw@mail.gmail.com> Message-ID: <5240C8FC.8070608@mrabarnett.plus.com> On 23/09/2013 22:19, Nick Coghlan wrote: > > On 24 Sep 2013 01:24, "Antoine Pitrou" <solipsis at pitrou.net > <mailto:solipsis at pitrou.net>> wrote: > > > > On Mon, 23 Sep 2013 18:51:04 +1000 > > Nick Coghlan <ncoghlan at gmail.com <mailto:ncoghlan at gmail.com>> wrote: > > > On 23 September 2013 18:45, Antoine Pitrou <solipsis at pitrou.net > <mailto:solipsis at pitrou.net>> wrote: > > > > Le Mon, 23 Sep 2013 18:17:51 +1000, > > > > Nick Coghlan <ncoghlan at gmail.com <mailto:ncoghlan at gmail.com>> a > ?crit : > > > >> > > > >> Here's what I suggest changing that error to: > > > >> > > > >> >>> del x > > > >> Unraisable exception suppressed when calling <bound method C.__del__ > > > >> of <__main__.C object at 0x7f98b8b61538>> > > > >> Traceback (most recent call last): > > > >> File "<stdin>", line 3, in __del__ > > > >> RuntimeError: Going away now > > > > > > > > Why not simply "Exception automatically caught in <bound method > > > > C.__del__> [...]" ? > > > > > > It only answers the "what" (i.e. the exception was automatically > > > caught), without addressing the "why" (i.e. because there wasn't > > > anything else useful the interpreter could do with it) > > > > Yes, but I agree with Greg that "unraisable" is wrong. After all, it > > was raised, and it can even be caught by the programmer (inside > > __del__). > > The word doesn't literally mean the exception itself was unraisable. It > means it was raised, we caught it and we're writing it to stderr because > we *can't raise it again*. > Ah, you mean "unreraisable". :-) > The relevant C API function is just called "PyErr_WriteUnraisable", not > "PyErr_WriteUnraisableButThatIsTechnicallyWrongSinceItWasAlreadyRaisedAndWeJustCaughtItAndAreNowReportingItToStdErr". > From ncoghlan at gmail.com Tue Sep 24 01:25:10 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 24 Sep 2013 09:25:10 +1000 Subject: [Python-Dev] Enum Eccentricities In-Reply-To: <5240BBB1.5010900@canterbury.ac.nz> References: <523F82C4.4060202@stoneleaf.us> <20130923024139.GA26561@piedra> <CAAXXHgKPNsXH1dgv2GeqhVC9+s1YaziCYXq0Aah-e=a3mSLAmg@mail.gmail.com> <CAP7+vJKisYvdOug-fmR_SZEET7icHGjEx2pFYYHXyZHrhjWhzw@mail.gmail.com> <20130923151623.GB7989@ando> <5240BBB1.5010900@canterbury.ac.nz> Message-ID: <CADiSq7evT-spFX+cHAQB_K2jH=5YM-uFOYGFGsPvOaEQYaYxzQ@mail.gmail.com> On 24 Sep 2013 08:09, "Greg Ewing" <greg.ewing at canterbury.ac.nz> wrote: > > Steven D'Aprano wrote: >> >> It might not be a rule, but it's certainly the norm. I reckon that class attributes that aren't accessible from the instance are significantly more surprising than Color.red.blue. > > > There are really two different kinds of things that we > refer to as "class attributes". One is things that really > are attributes of the class itself, and the other is > things that are meant to serve as default or shared > instance attributes. > > The confusion comes in because we use the same terminology > for both, and because Python doesn't provide any straightforward > way of creating user-defined class-only attributes, Using @property in a metaclass definition isn't *that* complicated :) (says the guy who helps maintain the type system) > so > shared attributes tend to get abused for that purpose. > Then when someone comes along and creates a true > class-only attribute, people get all surprised and > complain about it. One of the interesting aspects of adding Enum has been the subtle descriptor handling bugs it has uncovered in the inspect module :) > They shouldn't, IMO. There's a helper for Enum's descriptors that will probably be exposed through the types module in the next 3.4 alpha (tentative name is types.DynamicClassAttribute). It's the inverse, though - it throws AttributeError when looked up on the *class* in order to trigger __getattr__ on the metaclass. Cheers, Nick. > > -- > Greg > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130924/155de6aa/attachment.html> From nad at acm.org Tue Sep 24 01:34:54 2013 From: nad at acm.org (Ned Deily) Date: Mon, 23 Sep 2013 16:34:54 -0700 Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for pronouncement? References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com> Message-ID: <nad-F39A9C.16345423092013@news.gmane.org> In general, I think this is a very important usability feature and I am in favor of the general approach. Good work, all! I do have some comments, primarily about items that are not currently addressed. > ``ensurepip`` itself (including the private copy of ``pip`` and its > dependencies) will still be installed normally (as it is a regular > part of the standard library), only the implicit installation of pip and > its dependencies will be skipped. The PEP should address the license implications of this since the PEP will add the source of pip and setuptools to all new Python releases, both source and binary installers releases. For Python itself, we go to some trouble to ensure that all contributions are contributed under a suitable license. What steps are needed, initially and on-going, to ensure that these third-party items to be pulled in at each release are compatible with the license of Python itself? Who will do this: release managers? Are updates needed in the internal and external documentation for releases (README, LICENSE)? I took a quick look at the current source releases for both pip and setuptools. I noted a reference to the LGPL for pip's CA certs bundle; I didn't see a license file for setuptools. > This PEP proposes that, if pip still requires it as a dependency, > ``ensurepip`` will include a private copy of ``setuptools`` (in addition > to the private copy of ``ensurepip``). ``python -m ensurepip`` will then > install the private copy in addition to installing ``pip`` itself. > > However, this behavior is officially considered an implementation > detail. Other projects which explicitly require ``setuptools`` must still > provide an appropriate dependency declaration, rather than assuming > ``setuptools`` will always be installed alongside ``pip``. > > Once pip is able to run ``pip install --upgrade pip`` without needing > ``setuptools`` installed first, then the private copy of ``setuptools`` > will be removed from ``ensurepip`` in subsequent CPython releases. The PEP does not sufficiently address the issue of the setuptools-provided easy_install command. I think it is important to do that for several reasons: 1. The PEP should make clear whether the easy_install scripts are or are not installed, globally or in a venv, as a side effect of running ensurepip, at least while pip has the implicit private dependency on setuptools. By default, the scripts are installed so ensurepip would need to take action to prevent this if that is not desired. 2. Despite the current strong preference for using pip as a command line installer, the fact is that much third-party documentation still refers to and recommends using easy_install for their projects. That is not going to change overnight. The PEP should identify some steps needed to facilitate the migration, like perhaps adding something to the "Installing Python Modules" documentation explicitly deprecating easy_install use and/or giving pip equivalents for the most common easy_install scenarios (or linking to other documentation that does so). 3. A specific issue for OS X users is that, as part of OS X, Apple ships versions of Python 2 and various third-party packages, including setuptools (older, pre-reunification versions, at that) but not pip. That means that OS X ships with easy_install and easy_install-2.n commands in /usr/bin, including 2.7 in recent OS X releases. This has been an ongoing source of confusion for OS X users who install a newer version of Python. Depending on the distributor and/or source build configure options, the new version of Python will be installed to a different path, like /usr/local/bin. Which instance is used is generally managed by direct PATH environment manipulations (or by launcher programs similar to the Windows py launcher). If setuptools is not installed using the new instance, users invoking the easy_install command will by default end up with distributions being installed to a system Python rather than to the new Python. This leads to the unfortunately common scenario of: sudo easy_install blah # --> installs to system Python 2.7 python -c 'import blah' # --> fails because using newer Python 2.7 This may very well be the most common current usability problem today with python.org (and other third-party) Pythons on OS X. With this PEP, we have an opportunity to fix this problem in one of two ways. As long as pip requires setuptools and we do not inhibit the installation of the easy_install scripts is to the same directory as the python command itself, the new easy_install scripts will shadow the Apple-supplied ones just as the python commands do. If it is decided to not install the easy_install scripts now or in the future (e.g. when pip no longer depends on setuptools), ensurepip could install dummy easy_install scripts (if there are none already installed there) that merely emit a message to the user to encourage migration to pip and suggest, if needed, to use pip to install setuptools to replace the placeholder easy_install. Either approach would be a huge usability improvement over the situation today. (The actual implementation details for either approach are slightly more complicated for OS X framework builds like the python.org installers as the actual scripts directory is a ``bin`` directory within the framework itself with optional symlinks from ``/usr/local/bin`` to the fw bin directory for specific commands. The installer will now need to also install symlinks in ``/usr/local/bin`` for ``pip`` and, as proposed above, for ``easy_install``.) > A common source of Python installations are through downstream distributors > such as the various Linux Distributions [#ubuntu]_ [#debian]_ [#fedora]_, OSX > package managers [#homebrew]_, or Python-specific tools [#conda]_. In order If you are going to call out Homebrew, you should include the other major OS X package managers, MacPorts and Fink. > .. [#homebrew] `Homebrew <http://brew.sh/>` .. [#macports] `MacPorts <http://macports.org>` .. [#fink] `Fink <http://finkproject.org>` One final comment is that the PEP does not go into any detail on how it will be implemented. As it stands, there is a fair amount of one-time work, including implementing ensurepip, changes to the Windows installer, changes to the OS X installer, documentation changes, all with testing and backporting to 2.7.x and 3.3.x. Then there are the on-going process changes for all future releases, impacting all release team members, which will need to be documented in PEP 101. Do we have an understanding of who is do the big pieces and by when? For the record, I am willing to do the extra one-time and ongoing work for the OS X installers but it would be helpful to know how we are going to get it all done in time for upcoming releases. -- Ned Deily, nad at acm.org From donald at stufft.io Tue Sep 24 01:39:53 2013 From: donald at stufft.io (Donald Stufft) Date: Mon, 23 Sep 2013 19:39:53 -0400 Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for pronouncement? In-Reply-To: <nad-F39A9C.16345423092013@news.gmane.org> References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com> <nad-F39A9C.16345423092013@news.gmane.org> Message-ID: <C9FCE644-3132-4B99-B4CF-1927D7B565EE@stufft.io> On Sep 23, 2013, at 7:34 PM, Ned Deily <nad at acm.org> wrote: > One final comment is that the PEP does not go into any detail on how it > will be implemented. As it stands, there is a fair amount of one-time > work, including implementing ensurepip, changes to the Windows installer, > changes to the OS X installer, documentation changes, all with testing and > backporting to 2.7.x and 3.3.x. Then there are the on-going process changes > for all future releases, impacting all release team members, which will need > to be documented in PEP 101. Do we have an understanding of who is do the > big pieces and by when? For the record, I am willing to do the extra > one-time and ongoing work for the OS X installers but it would be helpful > to know how we are going to get it all done in time for upcoming releases. I'm in a meeting right now so I only skimmed this email, but I wanted to mention here that I've been assuming I would be writing the python portions of this. I don't know how to do the OSX/Window installer pieces and I was hoping that the relevant people would be at least willing to help me figure it out or would implement the parts that need to be done inside the installers (mostly running the command). ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: <http://mail.python.org/pipermail/python-dev/attachments/20130923/16ddab36/attachment.sig> From donald at stufft.io Tue Sep 24 02:12:01 2013 From: donald at stufft.io (Donald Stufft) Date: Mon, 23 Sep 2013 20:12:01 -0400 Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for pronouncement? In-Reply-To: <nad-F39A9C.16345423092013@news.gmane.org> References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com> <nad-F39A9C.16345423092013@news.gmane.org> Message-ID: <44F4E1F8-5533-45A7-810E-B9C13530E9DD@stufft.io> On Sep 23, 2013, at 7:34 PM, Ned Deily <nad at acm.org> wrote: > In general, I think this is a very important usability feature and I > am in favor of the general approach. Good work, all! I do have some > comments, primarily about items that are not currently addressed. > >> ``ensurepip`` itself (including the private copy of ``pip`` and its >> dependencies) will still be installed normally (as it is a regular >> part of the standard library), only the implicit installation of pip and >> its dependencies will be skipped. > > The PEP should address the license implications of this since the PEP will > add the source of pip and setuptools to all new Python releases, both > source and binary installers releases. For Python itself, we go to some > trouble to ensure that all contributions are contributed under a suitable > license. What steps are needed, initially and on-going, to ensure that > these third-party items to be pulled in at each release are compatible > with the license of Python itself? Who will do this: release managers? > Are updates needed in the internal and external documentation for releases > (README, LICENSE)? I took a quick look at the current source releases for > both pip and setuptools. I noted a reference to the LGPL for pip's CA > certs bundle; I didn't see a license file for setuptools. As far as I know the certificate bundle is licensed under either GPL, MPL, or LGPL. However there is debate about wether a CA bundle *can* be licensed at all (see https://github.com/pypa/pip/pull/971). Pip also includes some code taken from other libraries however everything is licensed as either 1, 2, or 3 clause BSD (besides the aforementioned certificates). I can't think of us being ok with adding a copyleft license such as GPL so code we bring in will likely be restricted to things like BSD. > >> This PEP proposes that, if pip still requires it as a dependency, >> ``ensurepip`` will include a private copy of ``setuptools`` (in addition >> to the private copy of ``ensurepip``). ``python -m ensurepip`` will then >> install the private copy in addition to installing ``pip`` itself. >> >> However, this behavior is officially considered an implementation >> detail. Other projects which explicitly require ``setuptools`` must still >> provide an appropriate dependency declaration, rather than assuming >> ``setuptools`` will always be installed alongside ``pip``. >> >> Once pip is able to run ``pip install --upgrade pip`` without needing >> ``setuptools`` installed first, then the private copy of ``setuptools`` >> will be removed from ``ensurepip`` in subsequent CPython releases. > > The PEP does not sufficiently address the issue of the setuptools-provided > easy_install command. I think it is important to do that for several > reasons: > > 1. The PEP should make clear whether the easy_install scripts are or > are not installed, globally or in a venv, as a side effect of running > ensurepip, at least while pip has the implicit private dependency > on setuptools. By default, the scripts are installed so ensurepip > would need to take action to prevent this if that is not desired. > > 2. Despite the current strong preference for using pip as a command > line installer, the fact is that much third-party documentation still refers > to and recommends using easy_install for their projects. That is not > going to change overnight. The PEP should identify some steps needed to > facilitate the migration, like perhaps adding something to the "Installing > Python Modules" documentation explicitly deprecating easy_install use > and/or giving pip equivalents for the most common easy_install > scenarios (or linking to other documentation that does so). > > 3. A specific issue for OS X users is that, as part of OS X, Apple ships > versions of Python 2 and various third-party packages, including > setuptools (older, pre-reunification versions, at that) but not pip. > That means that OS X ships with easy_install and easy_install-2.n > commands in /usr/bin, including 2.7 in recent OS X releases. This has > been an ongoing source of confusion for OS X users who install a newer > version of Python. Depending on the distributor and/or source build > configure options, the new version of Python will be installed to a > different path, like /usr/local/bin. Which instance is used > is generally managed by direct PATH environment manipulations (or by > launcher programs similar to the Windows py launcher). If setuptools > is not installed using the new instance, users invoking the easy_install > command will by default end up with distributions being installed to a > system Python rather than to the new Python. This leads to the > unfortunately common scenario of: > > sudo easy_install blah # --> installs to system Python 2.7 > python -c 'import blah' # --> fails because using newer Python 2.7 > > This may very well be the most common current usability problem today > with python.org (and other third-party) Pythons on OS X. > > With this PEP, we have an opportunity to fix this problem in one of > two ways. As long as pip requires setuptools and we do not inhibit > the installation of the easy_install scripts is to the same directory > as the python command itself, the new easy_install scripts will > shadow the Apple-supplied ones just as the python commands do. If it > is decided to not install the easy_install scripts now or in the > future (e.g. when pip no longer depends on setuptools), > ensurepip could install dummy easy_install scripts (if there > are none already installed there) that merely emit a message to the user > to encourage migration to pip and suggest, if needed, to use pip to install > setuptools to replace the placeholder easy_install. Either approach > would be a huge usability improvement over the situation today. > > (The actual implementation details for either approach are slightly more > complicated for OS X framework builds like the python.org installers > as the actual scripts directory is a ``bin`` directory within the > framework itself with optional symlinks from ``/usr/local/bin`` to > the fw bin directory for specific commands. The installer will now need > to also install symlinks in ``/usr/local/bin`` for ``pip`` and, as > proposed above, for ``easy_install``.) So this is an interesting question. On one hand I would say we shouldn't be installing easy_install as that feels very user facing to me and the fact that setuptools is being installed is an implementation detail. On the other hand if we put in stub commands that just simply direct the user to use pip then people who *want* to use easy_install (perhaps for Egg support) won't be able too (unless perhaps something is released on PyPI that restores the easy_install commands?) > >> A common source of Python installations are through downstream distributors >> such as the various Linux Distributions [#ubuntu]_ [#debian]_ [#fedora]_, OSX >> package managers [#homebrew]_, or Python-specific tools [#conda]_. In order > > If you are going to call out Homebrew, you should include the other major > OS X package managers, MacPorts and Fink. > >> .. [#homebrew] `Homebrew <http://brew.sh/>` > > .. [#macports] `MacPorts <http://macports.org>` > .. [#fink] `Fink <http://finkproject.org>` I can add them, I added homebrew because it's what I use :) Didn't really think to hard about being super inclusive. > > One final comment is that the PEP does not go into any detail on how it > will be implemented. As it stands, there is a fair amount of one-time > work, including implementing ensurepip, changes to the Windows installer, > changes to the OS X installer, documentation changes, all with testing and > backporting to 2.7.x and 3.3.x. Then there are the on-going process changes > for all future releases, impacting all release team members, which will need > to be documented in PEP 101. Do we have an understanding of who is do the > big pieces and by when? For the record, I am willing to do the extra > one-time and ongoing work for the OS X installers but it would be helpful > to know how we are going to get it all done in time for upcoming releases. To further expand upon my original answer, I'm volunteering to do the initial work on the ensurepip library, the scripts that will automated the ongoing maintenance work, and the back porting of both of the above. I can also attempt to do the OSX Installer and Windows installer but I have zero idea how the installers actually function so it's probably better for someone else to do that. Since it sounds like you're willing to do the work for the OSX installer that saves me from having to flail around trying to figure out how to do that part, so hopefully MvL or someone can do the same with the Windows installer. I'm not sure what needs done outside of the up front work, do I just propose changes to PEP 101? Do I make a whole new PEP? Is there more than just updating PEP 101? > > -- > Ned Deily, > nad at acm.org > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: <http://mail.python.org/pipermail/python-dev/attachments/20130923/2f69b4a1/attachment-0001.sig> From donald at stufft.io Tue Sep 24 02:35:11 2013 From: donald at stufft.io (Donald Stufft) Date: Mon, 23 Sep 2013 20:35:11 -0400 Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for pronouncement? In-Reply-To: <44F4E1F8-5533-45A7-810E-B9C13530E9DD@stufft.io> References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com> <nad-F39A9C.16345423092013@news.gmane.org> <44F4E1F8-5533-45A7-810E-B9C13530E9DD@stufft.io> Message-ID: <A897426D-A70B-4AD8-B775-E456207E24F0@stufft.io> On Sep 23, 2013, at 8:12 PM, Donald Stufft <donald at stufft.io> wrote: >> >> >>> A common source of Python installations are through downstream distributors >>> such as the various Linux Distributions [#ubuntu]_ [#debian]_ [#fedora]_, OSX >>> package managers [#homebrew]_, or Python-specific tools [#conda]_. In order >> >> If you are going to call out Homebrew, you should include the other major >> OS X package managers, MacPorts and Fink. >> >>> .. [#homebrew] `Homebrew <http://brew.sh/>` >> >> .. [#macports] `MacPorts <http://macports.org>` >> .. [#fink] `Fink <http://finkproject.org>` > > I can add them, I added homebrew because it's what I use :) Didn't really > think to hard about being super inclusive. Updated this in PEP453. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: <http://mail.python.org/pipermail/python-dev/attachments/20130923/0a9d3e5d/attachment.sig> From stephen at xemacs.org Tue Sep 24 02:50:15 2013 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Tue, 24 Sep 2013 09:50:15 +0900 Subject: [Python-Dev] Revert #12085 fix for __del__ attribute error message In-Reply-To: <5240C8FC.8070608@mrabarnett.plus.com> References: <l1l2bm$1bp$1@ger.gmane.org> <20130921221524.55D4225007F@webabinitio.net> <l1lbkl$k0g$1@ger.gmane.org> <CAP7+vJLHO12ZNwX4UZ0c_BF9pr+xxxLZfy2dGjBcMVenGow9vQ@mail.gmail.com> <l1n9o6$ap3$1@ger.gmane.org> <CAP7+vJL+0XEBpOnVbTpZJNLzahgm8EkyuXdGdhE27TVwHu4GAg@mail.gmail.com> <l1nkdh$lh7$1@ger.gmane.org> <CADiSq7eGpFtPbRc8BmiP2H9V2PVMma3ycX3Ky_MJ-F+TpEiU6w@mail.gmail.com> <CAP7+vJKAP5R+2R++z8+DQjSW6EVycoczB2gWeo759-NG4G6Veg@mail.gmail.com> <523FDB8F.1080207@canterbury.ac.nz> <CADiSq7caBSr4ujFo0HBYGF91SJp2NcP-_SreBqad-=KpcZdeCw@mail.gmail.com> <20130923104500.5645fdbb@pitrou.net> <CADiSq7dJAdStAzPYijQTC_fC6_01Jkt4kidSh+YDxWNNv_0tOA@mail.gmail.com> <20130923172245.0cbaec81@fsol> <CADiSq7deVSSxwSVwq91DfNaiJwLdN=WyVzFfE7uASkeSpeMUsw@mail.gmail.com> <5240C8FC.8070608@mrabarnett.plus.com> Message-ID: <87zjr311rs.fsf@uwakimon.sk.tsukuba.ac.jp> MRAB writes: > > The word doesn't literally mean the exception itself was unraisable. It > > means it was raised, we caught it and we're writing it to stderr because > > we *can't raise it again*. > Ah, you mean "unreraisable". :-) +1 Ugly as sin, but satisfies all other criteria (except for Antoine's "easily understandable" which simply can't be satisfied). From steve at pearwood.info Tue Sep 24 03:31:56 2013 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 24 Sep 2013 11:31:56 +1000 Subject: [Python-Dev] Best practice for documentation for std lib In-Reply-To: <CAP7h-xZ-E_3--JrQGE+htXZStagmBjHmozW6v6jEbiXUicQ8fQ@mail.gmail.com> References: <20130922031304.GE19939@ando> <CAF-Rda_sTOiKi6c9ihFVL-N90PC-xHxOAMm1im0s-KVyy=L_Ng@mail.gmail.com> <87ioxs3iix.fsf@uwakimon.sk.tsukuba.ac.jp> <CAP7+vJJucubZtXHhnNbAcE0GWfQTnOozHK9W=uv3CWkMX15XGA@mail.gmail.com> <20130923003142.GO19939@ando> <CAP7+vJLThCjrCGGaJLMNd2Uza6wanODnipjmZ3LBZpTkub=uCg@mail.gmail.com> <l1o8ri$mjv$1@ger.gmane.org> <CAP7+vJJSvX78f2+i7_C5mNMdOQpfSD_oKqC01Z8CEHaU2krqTw@mail.gmail.com> <l1q36s$99l$1@ger.gmane.org> <CAP7h-xZ-E_3--JrQGE+htXZStagmBjHmozW6v6jEbiXUicQ8fQ@mail.gmail.com> Message-ID: <20130924013155.GD7989@ando> This is getting off-topic, if you're not interested in English grammar you should stop reading. On Mon, Sep 23, 2013 at 03:18:01PM -0400, Alexander Belopolsky wrote: > I don't think "Returns bar." is a valid English sentence because it lacks > subject. Subjectless sentences are unusual in English, but you do see them. Sentences consisting of only an interjection are subjectless: "Ouch!" "Hear hear!" "Rubbish!" "Oh dear!" Imperative sentences often have no explicit subject: "Close the door." "Put that light out!" "Follow me." Conversational English (especially spoken English) often displays the phenomenon called "Conversational Deletion", where the beginning of sentences are eroded away, dropping (e.g.) possessives, articles, and subject nouns. "Hope this helps." "See you next week." "No need to get upset!" In the example given, "Returns bar", I would identify this as an example of conversational deletion. The full sentence would be "This function returns bar". Personally, I don't mind such a conversational style, although many people consider it too informal for written English, even docstrings :-) > I would not mind > > def foo(): > """returns bar""" > > which I would read as "Function foo() returns bar," but in this case > "returns" should be in lower case. I certainly don't like that. Sentences, even eroded sentences, start with capital letters in English. Unless you are the poet e.e. cummings, capital letters are non-negotiable. -- Steven From alexander.belopolsky at gmail.com Tue Sep 24 03:59:55 2013 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Mon, 23 Sep 2013 21:59:55 -0400 Subject: [Python-Dev] Best practice for documentation for std lib In-Reply-To: <20130924013155.GD7989@ando> References: <20130922031304.GE19939@ando> <CAF-Rda_sTOiKi6c9ihFVL-N90PC-xHxOAMm1im0s-KVyy=L_Ng@mail.gmail.com> <87ioxs3iix.fsf@uwakimon.sk.tsukuba.ac.jp> <CAP7+vJJucubZtXHhnNbAcE0GWfQTnOozHK9W=uv3CWkMX15XGA@mail.gmail.com> <20130923003142.GO19939@ando> <CAP7+vJLThCjrCGGaJLMNd2Uza6wanODnipjmZ3LBZpTkub=uCg@mail.gmail.com> <l1o8ri$mjv$1@ger.gmane.org> <CAP7+vJJSvX78f2+i7_C5mNMdOQpfSD_oKqC01Z8CEHaU2krqTw@mail.gmail.com> <l1q36s$99l$1@ger.gmane.org> <CAP7h-xZ-E_3--JrQGE+htXZStagmBjHmozW6v6jEbiXUicQ8fQ@mail.gmail.com> <20130924013155.GD7989@ando> Message-ID: <CAP7h-xaDEMaCj9e=nX5e2j+Qw8GGMEwVGPLMY7OBoqcKJShRVg@mail.gmail.com> Given near-consensus on the "Return ..." form, the following discussion is of academic interest only. On Mon, Sep 23, 2013 at 9:31 PM, Steven D'Aprano <steve at pearwood.info>wrote: > > I would not mind > > > > def foo(): > > """returns bar""" > > > > which I would read as "Function foo() returns bar," but in this case > > "returns" should be in lower case. > > I certainly don't like that. Sentences, even eroded sentences, start > with capital letters in English. Unless you are the poet e.e. cummings, > capital letters are non-negotiable. I still believe "Returns bar." is a non-sentence even if "Return bar!" is. While absence of subject noun in itself may not be an issue, but combined with a singular third-person verb form, it makes "Returns bar." grammatically incorrect. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130923/5de51848/attachment.html> From nad at acm.org Tue Sep 24 06:32:37 2013 From: nad at acm.org (Ned Deily) Date: Mon, 23 Sep 2013 21:32:37 -0700 Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for pronouncement? References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com> <nad-F39A9C.16345423092013@news.gmane.org> <44F4E1F8-5533-45A7-810E-B9C13530E9DD@stufft.io> Message-ID: <nad-AC1D3B.21323723092013@news.gmane.org> In article <44F4E1F8-5533-45A7-810E-B9C13530E9DD at stufft.io>, Donald Stufft <donald at stufft.io> wrote: > On Sep 23, 2013, at 7:34 PM, Ned Deily <nad at acm.org> wrote: > > [... license implications ...] > > As far as I know the certificate bundle is licensed under either GPL, MPL, > or LGPL. However there is debate about wether a CA bundle *can* be > licensed at all (see https://github.com/pypa/pip/pull/971). > > Pip also includes some code taken from other libraries however everything > is licensed as either 1, 2, or 3 clause BSD (besides the aforementioned > certificates). I can't think of us being ok with adding a copyleft license > such as GPL so code we bring in will likely be restricted to things like BSD. IANAL, but I think it would be good if, at least, the setuptools license were clearer and the LGPL reference for the cert bundle was changed. It *might* be a good idea to get an opinion from Van Lindberg. But I'm happy to defer to Martin's judgement on this. > > [... easy_install on OS X ...] > > So this is an interesting question. On one hand I would say we shouldn't be > installing easy_install as that feels very user facing to me and the fact > that > setuptools is being installed is an implementation detail. On the other hand > if we put in stub commands that just simply direct the user to use pip then > people who *want* to use easy_install (perhaps for Egg support) won't be > able too (unless perhaps something is released on PyPI that restores the > easy_install commands?) I was thinking that, in the latter case, those users who really want to use easy_install could be told to just use pip to install a PyPI version of setuptools which would replace the stub commands with the real things - assuming that works. > > [... implementation plan ...] > > To further expand upon my original answer, I'm volunteering to do the initial > work on the ensurepip library, the scripts that will automated the ongoing > maintenance work, and the back porting of both of the above. I can also > attempt > to do the OSX Installer and Windows installer but I have zero idea how the > installers actually function so it's probably better for someone else to do > that. > > Since it sounds like you're willing to do the work for the OSX installer that > saves me from having to flail around trying to figure out how to do that > part, > so hopefully MvL or someone can do the same with the Windows installer. > > I'm not sure what needs done outside of the up front work, do I just propose > changes to PEP 101? Do I make a whole new PEP? Is there more than > just updating PEP 101? I think the thing to do is get review buy-in from the release managers for the affected active releases (2.7.x = Benjamin, 3.3.x = Georg, 3.4.x = Larry) and let them worry about updating PEP 101. The key point is that this PEP is implicitly adding some new responsibilities for the release team; I think they just need to be explicit. -- Ned Deily, nad at acm.org From donald at stufft.io Tue Sep 24 06:35:05 2013 From: donald at stufft.io (Donald Stufft) Date: Tue, 24 Sep 2013 00:35:05 -0400 Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for pronouncement? In-Reply-To: <nad-AC1D3B.21323723092013@news.gmane.org> References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com> <nad-F39A9C.16345423092013@news.gmane.org> <44F4E1F8-5533-45A7-810E-B9C13530E9DD@stufft.io> <nad-AC1D3B.21323723092013@news.gmane.org> Message-ID: <F6BE0C28-86D2-4F25-9F11-93AB0FC3C067@stufft.io> On Sep 24, 2013, at 12:32 AM, Ned Deily <nad at acm.org> wrote: > IANAL, but I think it would be good if, at least, the setuptools license were > clearer and the LGPL reference for the cert bundle was changed. It *might* be > a good idea to get an opinion from Van Lindberg. But I'm happy to defer to > Martin's judgement on this. After your concern was raised I went ahead and emailed VanL. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: <http://mail.python.org/pipermail/python-dev/attachments/20130924/5ca2f486/attachment.sig> From ncoghlan at gmail.com Tue Sep 24 09:25:10 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 24 Sep 2013 17:25:10 +1000 Subject: [Python-Dev] Revert #12085 fix for __del__ attribute error message In-Reply-To: <20130924002955.49a99f9f@fsol> References: <l1l2bm$1bp$1@ger.gmane.org> <20130921221524.55D4225007F@webabinitio.net> <l1lbkl$k0g$1@ger.gmane.org> <CAP7+vJLHO12ZNwX4UZ0c_BF9pr+xxxLZfy2dGjBcMVenGow9vQ@mail.gmail.com> <l1n9o6$ap3$1@ger.gmane.org> <CAP7+vJL+0XEBpOnVbTpZJNLzahgm8EkyuXdGdhE27TVwHu4GAg@mail.gmail.com> <l1nkdh$lh7$1@ger.gmane.org> <CADiSq7eGpFtPbRc8BmiP2H9V2PVMma3ycX3Ky_MJ-F+TpEiU6w@mail.gmail.com> <CAP7+vJKAP5R+2R++z8+DQjSW6EVycoczB2gWeo759-NG4G6Veg@mail.gmail.com> <523FDB8F.1080207@canterbury.ac.nz> <CADiSq7caBSr4ujFo0HBYGF91SJp2NcP-_SreBqad-=KpcZdeCw@mail.gmail.com> <20130923104500.5645fdbb@pitrou.net> <CADiSq7dJAdStAzPYijQTC_fC6_01Jkt4kidSh+YDxWNNv_0tOA@mail.gmail.com> <20130923172245.0cbaec81@fsol> <CADiSq7deVSSxwSVwq91DfNaiJwLdN=WyVzFfE7uASkeSpeMUsw@mail.gmail.com> <20130923233507.4e5ebb9f@fsol> <5240B4E8.6020701@stoneleaf.us> <20130924002955.49a99f9f@fsol> Message-ID: <CADiSq7euer--7omtAdA=gWi_BOV2YWE-TgwEsbdRZA7xveQUdw@mail.gmail.com> On 24 September 2013 08:29, Antoine Pitrou <solipsis at pitrou.net> wrote: > On Mon, 23 Sep 2013 14:38:48 -0700 > Ethan Furman <ethan at stoneleaf.us> wrote: >> > But that's because you already know what it's supposed to convey. The >> > average user doesn't, and only sees "unraisable". >> >> All the more reason to have text in the error message that is easily searchable. > > Then I propose "CA6yuaYV6dygPfJRxUrhtg". It should be easily > searchable ;-) > > Seriously, "easily searchable" is a rather weak criterion for an > error message. It should be easily understandable and non-misleading > first. You are setting the bar unreasonably high for an error message that has to convey a complex concept in as few words as possible. There is *NO* wording that can concisely express the concepts involved without resorting to jargon, because the concepts behind it are *complex and unintuitive*. The current wording is flat out wrong, because the exception isn't being ignored, it's being printed to stderr. If it was genuinely being ignored, people wouldn't complain about it. Jargon that can be easily looked up with a search engine is greatly superior to a message that is simply wrong, as the former provides a gateway to understanding, just like coming across a word you don't understand when reading a novel. Preferring the status quo because you're holding out a forlorn hope for a concise wording that explains: - there are places where exceptions may occur but the interpreter can't reraise them - this is one of those cases, so we're printing it to stderr instead *without users needing to do any research*. A verbose wording is no good either, as that degenerates into a wall of text that people will *still* have to extract pieces from to plug into a search engine to figure out what they mean. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Tue Sep 24 09:29:59 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 24 Sep 2013 17:29:59 +1000 Subject: [Python-Dev] Revert #12085 fix for __del__ attribute error message In-Reply-To: <87zjr311rs.fsf@uwakimon.sk.tsukuba.ac.jp> References: <l1l2bm$1bp$1@ger.gmane.org> <20130921221524.55D4225007F@webabinitio.net> <l1lbkl$k0g$1@ger.gmane.org> <CAP7+vJLHO12ZNwX4UZ0c_BF9pr+xxxLZfy2dGjBcMVenGow9vQ@mail.gmail.com> <l1n9o6$ap3$1@ger.gmane.org> <CAP7+vJL+0XEBpOnVbTpZJNLzahgm8EkyuXdGdhE27TVwHu4GAg@mail.gmail.com> <l1nkdh$lh7$1@ger.gmane.org> <CADiSq7eGpFtPbRc8BmiP2H9V2PVMma3ycX3Ky_MJ-F+TpEiU6w@mail.gmail.com> <CAP7+vJKAP5R+2R++z8+DQjSW6EVycoczB2gWeo759-NG4G6Veg@mail.gmail.com> <523FDB8F.1080207@canterbury.ac.nz> <CADiSq7caBSr4ujFo0HBYGF91SJp2NcP-_SreBqad-=KpcZdeCw@mail.gmail.com> <20130923104500.5645fdbb@pitrou.net> <CADiSq7dJAdStAzPYijQTC_fC6_01Jkt4kidSh+YDxWNNv_0tOA@mail.gmail.com> <20130923172245.0cbaec81@fsol> <CADiSq7deVSSxwSVwq91DfNaiJwLdN=WyVzFfE7uASkeSpeMUsw@mail.gmail.com> <5240C8FC.8070608@mrabarnett.plus.com> <87zjr311rs.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <CADiSq7erOnT=PkjN3L__3teOZDjQtV3XU4sEOTTVs0dwsqcp5Q@mail.gmail.com> On 24 September 2013 10:50, Stephen J. Turnbull <stephen at xemacs.org> wrote: > MRAB writes: > > > > The word doesn't literally mean the exception itself was unraisable. It > > > means it was raised, we caught it and we're writing it to stderr because > > > we *can't raise it again*. > > > Ah, you mean "unreraisable". :-) > > +1 > > Ugly as sin, but satisfies all other criteria (except for Antoine's "easily > understandable" which simply can't be satisfied). If you're drawing a distinction between the first time an exception hits the eval loop and the second and subsequent times, then neither "Unraisable" nor "Unreraisable" is 100% correct. I just think it's a meaningless distinction, which is why I favour "Unraisable" - at the point in time where that message is displayed, that exception cannot be raised any further, whether it was created directly from C or was received from an underlying call back into Python code.. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Tue Sep 24 09:32:15 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 24 Sep 2013 17:32:15 +1000 Subject: [Python-Dev] Revert #12085 fix for __del__ attribute error message In-Reply-To: <CADiSq7euer--7omtAdA=gWi_BOV2YWE-TgwEsbdRZA7xveQUdw@mail.gmail.com> References: <l1l2bm$1bp$1@ger.gmane.org> <20130921221524.55D4225007F@webabinitio.net> <l1lbkl$k0g$1@ger.gmane.org> <CAP7+vJLHO12ZNwX4UZ0c_BF9pr+xxxLZfy2dGjBcMVenGow9vQ@mail.gmail.com> <l1n9o6$ap3$1@ger.gmane.org> <CAP7+vJL+0XEBpOnVbTpZJNLzahgm8EkyuXdGdhE27TVwHu4GAg@mail.gmail.com> <l1nkdh$lh7$1@ger.gmane.org> <CADiSq7eGpFtPbRc8BmiP2H9V2PVMma3ycX3Ky_MJ-F+TpEiU6w@mail.gmail.com> <CAP7+vJKAP5R+2R++z8+DQjSW6EVycoczB2gWeo759-NG4G6Veg@mail.gmail.com> <523FDB8F.1080207@canterbury.ac.nz> <CADiSq7caBSr4ujFo0HBYGF91SJp2NcP-_SreBqad-=KpcZdeCw@mail.gmail.com> <20130923104500.5645fdbb@pitrou.net> <CADiSq7dJAdStAzPYijQTC_fC6_01Jkt4kidSh+YDxWNNv_0tOA@mail.gmail.com> <20130923172245.0cbaec81@fsol> <CADiSq7deVSSxwSVwq91DfNaiJwLdN=WyVzFfE7uASkeSpeMUsw@mail.gmail.com> <20130923233507.4e5ebb9f@fsol> <5240B4E8.6020701@stoneleaf.us> <20130924002955.49a99f9f@fsol> <CADiSq7euer--7omtAdA=gWi_BOV2YWE-TgwEsbdRZA7xveQUdw@mail.gmail.com> Message-ID: <CADiSq7ehgCz7K=9BJbhNp5J4mimmZPoGOoU6_E+ogKipN3dQkw@mail.gmail.com> On 24 September 2013 17:25, Nick Coghlan <ncoghlan at gmail.com> wrote: > Preferring the status quo because > you're holding out a forlorn hope for a concise wording that explains: > > - there are places where exceptions may occur but the interpreter > can't reraise them > - this is one of those cases, so we're printing it to stderr instead > > *without users needing to do any research*. Oops, reworded the second part of that sentence without fixing the first part. It should have said: ============== It doesn't make sense to prefer the status quo because you're holding out a forlorn hope for a concise wording that explains: - there are places where exceptions may occur but the interpreter can't reraise them - this is one of those cases, so we're printing it to stderr instead *without users needing to do any research*. ============== Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From solipsis at pitrou.net Tue Sep 24 09:34:17 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 24 Sep 2013 09:34:17 +0200 Subject: [Python-Dev] Revert #12085 fix for __del__ attribute error message In-Reply-To: <CADiSq7euer--7omtAdA=gWi_BOV2YWE-TgwEsbdRZA7xveQUdw@mail.gmail.com> References: <l1l2bm$1bp$1@ger.gmane.org> <20130921221524.55D4225007F@webabinitio.net> <l1lbkl$k0g$1@ger.gmane.org> <CAP7+vJLHO12ZNwX4UZ0c_BF9pr+xxxLZfy2dGjBcMVenGow9vQ@mail.gmail.com> <l1n9o6$ap3$1@ger.gmane.org> <CAP7+vJL+0XEBpOnVbTpZJNLzahgm8EkyuXdGdhE27TVwHu4GAg@mail.gmail.com> <l1nkdh$lh7$1@ger.gmane.org> <CADiSq7eGpFtPbRc8BmiP2H9V2PVMma3ycX3Ky_MJ-F+TpEiU6w@mail.gmail.com> <CAP7+vJKAP5R+2R++z8+DQjSW6EVycoczB2gWeo759-NG4G6Veg@mail.gmail.com> <523FDB8F.1080207@canterbury.ac.nz> <CADiSq7caBSr4ujFo0HBYGF91SJp2NcP-_SreBqad-=KpcZdeCw@mail.gmail.com> <20130923104500.5645fdbb@pitrou.net> <CADiSq7dJAdStAzPYijQTC_fC6_01Jkt4kidSh+YDxWNNv_0tOA@mail.gmail.com> <20130923172245.0cbaec81@fsol> <CADiSq7deVSSxwSVwq91DfNaiJwLdN=WyVzFfE7uASkeSpeMUsw@mail.gmail.com> <20130923233507.4e5ebb9f@fsol> <5240B4E8.6020701@stoneleaf.us> <20130924002955.49a99f9f@fsol> <CADiSq7euer--7omtAdA=gWi_BOV2YWE-TgwEsbdRZA7xveQUdw@mail.gmail.com> Message-ID: <20130924093417.34abbc42@fsol> On Tue, 24 Sep 2013 17:25:10 +1000 Nick Coghlan <ncoghlan at gmail.com> wrote: > > You are setting the bar unreasonably high for an error message that > has to convey a complex concept in as few words as possible. There is > *NO* wording that can concisely express the concepts involved without > resorting to jargon, because the concepts behind it are *complex and > unintuitive*. The current wording is flat out wrong, because the > exception isn't being ignored, it's being printed to stderr. If it was > genuinely being ignored, people wouldn't complain about it. > > Jargon that can be easily looked up with a search engine is greatly > superior to a message that is simply wrong, as the former provides a > gateway to understanding, just like coming across a word you don't > understand when reading a novel. "Unraisable" is not a word I don't understand, it's a word that I understand and which conveys the wrong meaning. If you want something that people won't understand, you can use something like "asynchronous exception". > Preferring the status quo because > you're holding out a forlorn hope for a concise wording that explains: I've proposed other options. Regards Antoine. From ncoghlan at gmail.com Tue Sep 24 09:52:32 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 24 Sep 2013 17:52:32 +1000 Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for pronouncement? In-Reply-To: <nad-F39A9C.16345423092013@news.gmane.org> References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com> <nad-F39A9C.16345423092013@news.gmane.org> Message-ID: <CADiSq7cuAYtpwqh7Z2R7n9ButRt7-4PP9_oFrk69feVnWOCiSA@mail.gmail.com> On 24 September 2013 09:34, Ned Deily <nad at acm.org> wrote: > In general, I think this is a very important usability feature and I > am in favor of the general approach. Good work, all! I do have some > comments, primarily about items that are not currently addressed. Your reply and Barry's suggest that Betteridge's law [1] applies to email subject lines, too ;) As far as easy_install goes, my current plan was actually to tackle that on the upstream side. If pip still depends on setuptools by the time of the Python 3.4 release, then it will depend on the *real* setuptools, easy_install and all. From my perspective, one golden rule of this integration is that we do *not* mess with the contents of the wheel files for pip and its dependencies - they're pristine upstream releases. This is mostly for technical reasons, but it also draws a sharp line of demarcation for any "aggregation or derivation?" questions, too. If pip has been updated by the time of its inclusion to depend on a cut down setuptools derivative that omits easy_install (or pip has switched to its own internal replacements instead), so much the better, but I consider that to be essentially independent of the CPython bundling situation, since it isn't something we have direct control over, and I consider the slight downside of potentially installing easy_install alongside pip to be dwarfed by the benefits of installing pip. As far as I am aware, the licensing on setuptools is currently limited to the "ZPL or PSF" declaration in the distribution metadata. Cheers, Nick. [1] https://en.wikipedia.org/wiki/Betteridge%27s_law_of_headlines -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Tue Sep 24 10:06:15 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 24 Sep 2013 18:06:15 +1000 Subject: [Python-Dev] Revert #12085 fix for __del__ attribute error message In-Reply-To: <20130924093417.34abbc42@fsol> References: <l1l2bm$1bp$1@ger.gmane.org> <20130921221524.55D4225007F@webabinitio.net> <l1lbkl$k0g$1@ger.gmane.org> <CAP7+vJLHO12ZNwX4UZ0c_BF9pr+xxxLZfy2dGjBcMVenGow9vQ@mail.gmail.com> <l1n9o6$ap3$1@ger.gmane.org> <CAP7+vJL+0XEBpOnVbTpZJNLzahgm8EkyuXdGdhE27TVwHu4GAg@mail.gmail.com> <l1nkdh$lh7$1@ger.gmane.org> <CADiSq7eGpFtPbRc8BmiP2H9V2PVMma3ycX3Ky_MJ-F+TpEiU6w@mail.gmail.com> <CAP7+vJKAP5R+2R++z8+DQjSW6EVycoczB2gWeo759-NG4G6Veg@mail.gmail.com> <523FDB8F.1080207@canterbury.ac.nz> <CADiSq7caBSr4ujFo0HBYGF91SJp2NcP-_SreBqad-=KpcZdeCw@mail.gmail.com> <20130923104500.5645fdbb@pitrou.net> <CADiSq7dJAdStAzPYijQTC_fC6_01Jkt4kidSh+YDxWNNv_0tOA@mail.gmail.com> <20130923172245.0cbaec81@fsol> <CADiSq7deVSSxwSVwq91DfNaiJwLdN=WyVzFfE7uASkeSpeMUsw@mail.gmail.com> <20130923233507.4e5ebb9f@fsol> <5240B4E8.6020701@stoneleaf.us> <20130924002955.49a99f9f@fsol> <CADiSq7euer--7omtAdA=gWi_BOV2YWE-TgwEsbdRZA7xveQUdw@mail.gmail.com> <20130924093417.34abbc42@fsol> Message-ID: <CADiSq7dQ=YcEiTea3GrrQJ64ypBDCH4sN+fHJirPL4hjkep5FA@mail.gmail.com> On 24 September 2013 17:34, Antoine Pitrou <solipsis at pitrou.net> wrote: > On Tue, 24 Sep 2013 17:25:10 +1000 > Nick Coghlan <ncoghlan at gmail.com> wrote: >> >> You are setting the bar unreasonably high for an error message that >> has to convey a complex concept in as few words as possible. There is >> *NO* wording that can concisely express the concepts involved without >> resorting to jargon, because the concepts behind it are *complex and >> unintuitive*. The current wording is flat out wrong, because the >> exception isn't being ignored, it's being printed to stderr. If it was >> genuinely being ignored, people wouldn't complain about it. >> >> Jargon that can be easily looked up with a search engine is greatly >> superior to a message that is simply wrong, as the former provides a >> gateway to understanding, just like coming across a word you don't >> understand when reading a novel. > > "Unraisable" is not a word I don't understand, it's a word that I > understand and which conveys the wrong meaning. How is it wrong? At the point where the interpreter says "This exception is now unraisable", what, precisely, is it saying that is wrong? It isn't saying "this has never been raised". It is saying, "where it is currently being processed, this exception cannot be raised". > If you want something that people won't understand, you can use > something like "asynchronous exception". Asynchronous exception is *even more* wrong, because that's the terminology used for an exception injected into the current thread by a different thread. >> Preferring the status quo because >> you're holding out a forlorn hope for a concise wording that explains: > > I've proposed other options. "Automatically caught" says nothing about why the exception is being printed to stderr instead of propagating normally. Exceptions are automatically caught by any matching except clause all the time, but most of those don't result in errors printed to stderr. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From solipsis at pitrou.net Tue Sep 24 10:16:11 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 24 Sep 2013 10:16:11 +0200 Subject: [Python-Dev] Revert #12085 fix for __del__ attribute error message In-Reply-To: <CADiSq7dQ=YcEiTea3GrrQJ64ypBDCH4sN+fHJirPL4hjkep5FA@mail.gmail.com> References: <l1l2bm$1bp$1@ger.gmane.org> <CAP7+vJLHO12ZNwX4UZ0c_BF9pr+xxxLZfy2dGjBcMVenGow9vQ@mail.gmail.com> <l1n9o6$ap3$1@ger.gmane.org> <CAP7+vJL+0XEBpOnVbTpZJNLzahgm8EkyuXdGdhE27TVwHu4GAg@mail.gmail.com> <l1nkdh$lh7$1@ger.gmane.org> <CADiSq7eGpFtPbRc8BmiP2H9V2PVMma3ycX3Ky_MJ-F+TpEiU6w@mail.gmail.com> <CAP7+vJKAP5R+2R++z8+DQjSW6EVycoczB2gWeo759-NG4G6Veg@mail.gmail.com> <523FDB8F.1080207@canterbury.ac.nz> <CADiSq7caBSr4ujFo0HBYGF91SJp2NcP-_SreBqad-=KpcZdeCw@mail.gmail.com> <20130923104500.5645fdbb@pitrou.net> <CADiSq7dJAdStAzPYijQTC_fC6_01Jkt4kidSh+YDxWNNv_0tOA@mail.gmail.com> <20130923172245.0cbaec81@fsol> <CADiSq7deVSSxwSVwq91DfNaiJwLdN=WyVzFfE7uASkeSpeMUsw@mail.gmail.com> <20130923233507.4e5ebb9f@fsol> <5240B4E8.6020701@stoneleaf.us> <20130924002955.49a99f9f@fsol> <CADiSq7euer--7omtAdA=gWi_BOV2YWE-TgwEsbdRZA7xveQUdw@mail.gmail.com> <20130924093417.34abbc42@fsol> <CADiSq7dQ=YcEiTea3GrrQJ64ypBDCH4sN+fHJirPL4hjkep5FA@mail.gmail.com> Message-ID: <20130924101611.5526f552@fsol> On Tue, 24 Sep 2013 18:06:15 +1000 Nick Coghlan <ncoghlan at gmail.com> wrote: > > How is it wrong? At the point where the interpreter says "This > exception is now unraisable", what, precisely, is it saying that is > wrong? > It isn't saying "this has never been raised". It is saying, "where it > is currently being processed, this exception cannot be raised". Well, it is saying it. If it's conceptually unraisable, it can't be raised. I know your point is that it is only unraisable *now*, but that's not the intuitive interpretation. > >> Preferring the status quo because > >> you're holding out a forlorn hope for a concise wording that explains: > > > > I've proposed other options. > > "Automatically caught" says nothing about why the exception is being > printed to stderr instead of propagating normally. Exceptions are > automatically caught by any matching except clause all the time, but > most of those don't result in errors printed to stderr. I would say they are manually caught by except clauses (user code == manual intervention), and automatically caught when "silenced" or "unraised" by the interpreter. Regards Antoine. From python at mrabarnett.plus.com Tue Sep 24 11:58:54 2013 From: python at mrabarnett.plus.com (MRAB) Date: Tue, 24 Sep 2013 10:58:54 +0100 Subject: [Python-Dev] Revert #12085 fix for __del__ attribute error message In-Reply-To: <CADiSq7dQ=YcEiTea3GrrQJ64ypBDCH4sN+fHJirPL4hjkep5FA@mail.gmail.com> References: <l1l2bm$1bp$1@ger.gmane.org> <l1lbkl$k0g$1@ger.gmane.org> <CAP7+vJLHO12ZNwX4UZ0c_BF9pr+xxxLZfy2dGjBcMVenGow9vQ@mail.gmail.com> <l1n9o6$ap3$1@ger.gmane.org> <CAP7+vJL+0XEBpOnVbTpZJNLzahgm8EkyuXdGdhE27TVwHu4GAg@mail.gmail.com> <l1nkdh$lh7$1@ger.gmane.org> <CADiSq7eGpFtPbRc8BmiP2H9V2PVMma3ycX3Ky_MJ-F+TpEiU6w@mail.gmail.com> <CAP7+vJKAP5R+2R++z8+DQjSW6EVycoczB2gWeo759-NG4G6Veg@mail.gmail.com> <523FDB8F.1080207@canterbury.ac.nz> <CADiSq7caBSr4ujFo0HBYGF91SJp2NcP-_SreBqad-=KpcZdeCw@mail.gmail.com> <20130923104500.5645fdbb@pitrou.net> <CADiSq7dJAdStAzPYijQTC_fC6_01Jkt4kidSh+YDxWNNv_0tOA@mail.gmail.com> <20130923172245.0cbaec81@fsol> <CADiSq7deVSSxwSVwq91DfNaiJwLdN=WyVzFfE7uASkeSpeMUsw@mail.gmail.com> <20130923233507.4e5ebb9f@fsol> <5240B4E8.6020701@stoneleaf.us> <20130924002955.49a99f9f@fsol> <CADiSq7euer--7omtAdA=gWi_BOV2YWE-TgwEsbdRZA7xveQUdw@mail.gmail.com> <20130924093417.34abbc42@fsol> <CADiSq7dQ=YcEiTea3GrrQJ64ypBDCH4sN+fHJirPL4hjkep5FA@mail.gmail.com> Message-ID: <5241625E.7030701@mrabarnett.plus.com> On 24/09/2013 09:06, Nick Coghlan wrote: > On 24 September 2013 17:34, Antoine Pitrou <solipsis at pitrou.net> wrote: >> On Tue, 24 Sep 2013 17:25:10 +1000 >> Nick Coghlan <ncoghlan at gmail.com> wrote: >>> >>> You are setting the bar unreasonably high for an error message that >>> has to convey a complex concept in as few words as possible. There is >>> *NO* wording that can concisely express the concepts involved without >>> resorting to jargon, because the concepts behind it are *complex and >>> unintuitive*. The current wording is flat out wrong, because the >>> exception isn't being ignored, it's being printed to stderr. If it was >>> genuinely being ignored, people wouldn't complain about it. >>> >>> Jargon that can be easily looked up with a search engine is greatly >>> superior to a message that is simply wrong, as the former provides a >>> gateway to understanding, just like coming across a word you don't >>> understand when reading a novel. >> >> "Unraisable" is not a word I don't understand, it's a word that I >> understand and which conveys the wrong meaning. > > How is it wrong? At the point where the interpreter says "This > exception is now unraisable", what, precisely, is it saying that is > wrong? > > It isn't saying "this has never been raised". It is saying, "where it > is currently being processed, this exception cannot be raised". > >> If you want something that people won't understand, you can use >> something like "asynchronous exception". > > Asynchronous exception is *even more* wrong, because that's the > terminology used for an exception injected into the current thread by > a different thread. > >>> Preferring the status quo because >>> you're holding out a forlorn hope for a concise wording that explains: >> >> I've proposed other options. > > "Automatically caught" says nothing about why the exception is being > printed to stderr instead of propagating normally. Exceptions are > automatically caught by any matching except clause all the time, but > most of those don't result in errors printed to stderr. > Why not just say something like "Cannot propagate exception..."; it's simpler than "Unpropagatable exception...". From walter at livinglogic.de Tue Sep 24 12:21:45 2013 From: walter at livinglogic.de (=?UTF-8?B?V2FsdGVyIETDtnJ3YWxk?=) Date: Tue, 24 Sep 2013 12:21:45 +0200 Subject: [Python-Dev] Best practice for documentation for std lib In-Reply-To: <CANc-5UzYUtta=4Q3c7n-LXdcYBnWrh5VYdMwSNPJ=5a0HuVfig@mail.gmail.com> References: <20130922031304.GE19939@ando> <20130922102046.764abea5@fsol> <20130922125323.GI19939@ando> <CAP1=2W7ymWcx_Yj6Xsb0Ty1Wj5pUf+_1XtqhPAVyRKWgiQ37mg@mail.gmail.com> <524025B3.7020305@livinglogic.de> <CAP7+vJLDo9KkdzzQfnJe_bTzunAP6+gBznY1Dx5pvxBqva0YjQ@mail.gmail.com> <CANc-5UzYUtta=4Q3c7n-LXdcYBnWrh5VYdMwSNPJ=5a0HuVfig@mail.gmail.com> Message-ID: <524167B9.1090004@livinglogic.de> On 23.09.13 17:18, Skip Montanaro wrote: >>> It would be great if the docstring contained a link to the online >>> documentation. >> >> That would have to be a feature of help(), not hardcoded in each docstring. > > That *is* a feature of the help function: > > Help on built-in module sys: > >>>> help(sys) > NAME > sys > > FILE > (built-in) > > MODULE DOCS > http://docs.python.org/library/sys > ... > > (pydoc too, though I'm 99.9% sure they use the same underlying > facility Ping originally implemented.) Hmm, but it doesn't work for functions: >>> import sys >>> help(sys.settracee) Help on built-in function settrace in module sys: settrace(...) settrace(function) Set the global debug tracing function. It will be called on each function call. See the debugger chapter in the library manual. Servus, Walter From ncoghlan at gmail.com Tue Sep 24 14:51:28 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 24 Sep 2013 22:51:28 +1000 Subject: [Python-Dev] Revert #12085 fix for __del__ attribute error message In-Reply-To: <5241625E.7030701@mrabarnett.plus.com> References: <l1l2bm$1bp$1@ger.gmane.org> <l1lbkl$k0g$1@ger.gmane.org> <CAP7+vJLHO12ZNwX4UZ0c_BF9pr+xxxLZfy2dGjBcMVenGow9vQ@mail.gmail.com> <l1n9o6$ap3$1@ger.gmane.org> <CAP7+vJL+0XEBpOnVbTpZJNLzahgm8EkyuXdGdhE27TVwHu4GAg@mail.gmail.com> <l1nkdh$lh7$1@ger.gmane.org> <CADiSq7eGpFtPbRc8BmiP2H9V2PVMma3ycX3Ky_MJ-F+TpEiU6w@mail.gmail.com> <CAP7+vJKAP5R+2R++z8+DQjSW6EVycoczB2gWeo759-NG4G6Veg@mail.gmail.com> <523FDB8F.1080207@canterbury.ac.nz> <CADiSq7caBSr4ujFo0HBYGF91SJp2NcP-_SreBqad-=KpcZdeCw@mail.gmail.com> <20130923104500.5645fdbb@pitrou.net> <CADiSq7dJAdStAzPYijQTC_fC6_01Jkt4kidSh+YDxWNNv_0tOA@mail.gmail.com> <20130923172245.0cbaec81@fsol> <CADiSq7deVSSxwSVwq91DfNaiJwLdN=WyVzFfE7uASkeSpeMUsw@mail.gmail.com> <20130923233507.4e5ebb9f@fsol> <5240B4E8.6020701@stoneleaf.us> <20130924002955.49a99f9f@fsol> <CADiSq7euer--7omtAdA=gWi_BOV2YWE-TgwEsbdRZA7xveQUdw@mail.gmail.com> <20130924093417.34abbc42@fsol> <CADiSq7dQ=YcEiTea3GrrQJ64ypBDCH4sN+fHJirPL4hjkep5FA@mail.gmail.com> <5241625E.7030701@mrabarnett.plus.com> Message-ID: <CADiSq7eiAkDBe9MwodRD3NDuGn935goYo7OKPu7M_aNe1buvHw@mail.gmail.com> On 24 Sep 2013 20:06, "MRAB" <python at mrabarnett.plus.com> wrote: > > On 24/09/2013 09:06, Nick Coghlan wrote: >> >> On 24 September 2013 17:34, Antoine Pitrou <solipsis at pitrou.net> wrote: >>> >>> On Tue, 24 Sep 2013 17:25:10 +1000 >>> Nick Coghlan <ncoghlan at gmail.com> wrote: >>>> >>>> >>>> You are setting the bar unreasonably high for an error message that >>>> has to convey a complex concept in as few words as possible. There is >>>> *NO* wording that can concisely express the concepts involved without >>>> resorting to jargon, because the concepts behind it are *complex and >>>> unintuitive*. The current wording is flat out wrong, because the >>>> exception isn't being ignored, it's being printed to stderr. If it was >>>> genuinely being ignored, people wouldn't complain about it. >>>> >>>> Jargon that can be easily looked up with a search engine is greatly >>>> superior to a message that is simply wrong, as the former provides a >>>> gateway to understanding, just like coming across a word you don't >>>> understand when reading a novel. >>> >>> >>> "Unraisable" is not a word I don't understand, it's a word that I >>> understand and which conveys the wrong meaning. >> >> >> How is it wrong? At the point where the interpreter says "This >> exception is now unraisable", what, precisely, is it saying that is >> wrong? >> >> It isn't saying "this has never been raised". It is saying, "where it >> is currently being processed, this exception cannot be raised". >> >>> If you want something that people won't understand, you can use >>> something like "asynchronous exception". >> >> >> Asynchronous exception is *even more* wrong, because that's the >> terminology used for an exception injected into the current thread by >> a different thread. >> >>>> Preferring the status quo because >>>> you're holding out a forlorn hope for a concise wording that explains: >>> >>> >>> I've proposed other options. >> >> >> "Automatically caught" says nothing about why the exception is being >> printed to stderr instead of propagating normally. Exceptions are >> automatically caught by any matching except clause all the time, but >> most of those don't result in errors printed to stderr. >> > Why not just say something like "Cannot propagate exception..."; it's > simpler than "Unpropagatable exception...". That would definitely be an improvement on the status quo and avoids Antoine's concern about an adjective being interpreted as an inherent property of the exception rather than the circumstances where the exception was encountered. Cheers, Nick. > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130924/124db00f/attachment.html> From zuo at chopin.edu.pl Tue Sep 24 15:53:06 2013 From: zuo at chopin.edu.pl (Jan Kaliszewski) Date: Tue, 24 Sep 2013 15:53:06 +0200 Subject: [Python-Dev] =?utf-8?q?Revert_=2312085_fix_for_=5F=5Fdel=5F=5F_at?= =?utf-8?q?tribute_error_message?= In-Reply-To: <20130924101611.5526f552@fsol> References: <l1l2bm$1bp$1@ger.gmane.org> <CAP7+vJLHO12ZNwX4UZ0c_BF9pr+xxxLZfy2dGjBcMVenGow9vQ@mail.gmail.com> <l1n9o6$ap3$1@ger.gmane.org> <CAP7+vJL+0XEBpOnVbTpZJNLzahgm8EkyuXdGdhE27TVwHu4GAg@mail.gmail.com> <l1nkdh$lh7$1@ger.gmane.org> <CADiSq7eGpFtPbRc8BmiP2H9V2PVMma3ycX3Ky_MJ-F+TpEiU6w@mail.gmail.com> <CAP7+vJKAP5R+2R++z8+DQjSW6EVycoczB2gWeo759-NG4G6Veg@mail.gmail.com> <523FDB8F.1080207@canterbury.ac.nz> <CADiSq7caBSr4ujFo0HBYGF91SJp2NcP-_SreBqad-=KpcZdeCw@mail.gmail.com> <20130923104500.5645fdbb@pitrou.net> <CADiSq7dJAdStAzPYijQTC_fC6_01Jkt4kidSh+YDxWNNv_0tOA@mail.gmail.com> <20130923172245.0cbaec81@fsol> <CADiSq7deVSSxwSVwq91DfNaiJwLdN=WyVzFfE7uASkeSpeMUsw@mail.gmail.com> <20130923233507.4e5ebb9f@fsol> <5240B4E8.6020701@stoneleaf.us> <20130924002955.49a99f9f@fsol> <CADiSq7euer--7omtAdA=gWi_BOV2YWE-TgwEsbdRZA7xveQUdw@mail.gmail.com> <20130924093417.34abbc42@fsol> <CADiSq7dQ=YcEiTea3GrrQJ64ypBDCH4sN+fHJirPL4hjkep5FA@mail.gmail.com> <20130924101611.5526f552@fsol> Message-ID: <3e055c69fc6126dc0e4d0a27b9ada354@chopin.edu.pl> 24.09.2013 10:16, Antoine Pitrou wrote: > On Tue, 24 Sep 2013 18:06:15 +1000 > Nick Coghlan <ncoghlan at gmail.com> wrote: >> >> How is it wrong? At the point where the interpreter says "This >> exception is now unraisable", what, precisely, is it saying that is >> wrong? >> It isn't saying "this has never been raised". It is saying, "where >> it >> is currently being processed, this exception cannot be raised". > > Well, it is saying it. If it's conceptually unraisable, it can't be > raised. I know your point is that it is only unraisable *now*, but > that's not the intuitive interpretation. And what about: Exception not propagated from <bound method C.__del__ of <__main__.C object at 0x7f98b8b61538>> ... Or: Exception that cannot be propagated from <bound method C.__del__ of <__main__.C object at 0x7f98b8b61538>> ... Cheers. *j From v+python at g.nevcal.com Tue Sep 24 18:33:21 2013 From: v+python at g.nevcal.com (Glenn Linderman) Date: Tue, 24 Sep 2013 09:33:21 -0700 Subject: [Python-Dev] Revert #12085 fix for __del__ attribute error message In-Reply-To: <CADiSq7eiAkDBe9MwodRD3NDuGn935goYo7OKPu7M_aNe1buvHw@mail.gmail.com> References: <l1l2bm$1bp$1@ger.gmane.org> <l1n9o6$ap3$1@ger.gmane.org> <CAP7+vJL+0XEBpOnVbTpZJNLzahgm8EkyuXdGdhE27TVwHu4GAg@mail.gmail.com> <l1nkdh$lh7$1@ger.gmane.org> <CADiSq7eGpFtPbRc8BmiP2H9V2PVMma3ycX3Ky_MJ-F+TpEiU6w@mail.gmail.com> <CAP7+vJKAP5R+2R++z8+DQjSW6EVycoczB2gWeo759-NG4G6Veg@mail.gmail.com> <523FDB8F.1080207@canterbury.ac.nz> <CADiSq7caBSr4ujFo0HBYGF91SJp2NcP-_SreBqad-=KpcZdeCw@mail.gmail.com> <20130923104500.5645fdbb@pitrou.net> <CADiSq7dJAdStAzPYijQTC_fC6_01Jkt4kidSh+YDxWNNv_0tOA@mail.gmail.com> <20130923172245.0cbaec81@fsol> <CADiSq7deVSSxwSVwq91DfNaiJwLdN=WyVzFfE7uASkeSpeMUsw@mail.gmail.com> <20130923233507.4e5ebb9f@fsol> <5240B4E8.6020701@stoneleaf.us> <20130924002955.49a99f9f@fsol> <CADiSq7euer--7omtAdA=gWi_BOV2YWE-TgwEsbdRZA7xveQUdw@mail.gmail.com> <20130924093417.34abbc42@fsol> <CADiSq7dQ=YcEiTea3GrrQJ64ypBDCH4sN+fHJirPL4hjkep5FA@mail.gmail.com> <5241625E.7030701@mrabarnett.plus.com> <CADiSq7eiAkDBe9MwodRD3NDuGn935goYo7OKPu7M_aNe1buvHw@mail.gmail.com> Message-ID: <5241BED1.7010905@g.nevcal.com> On 9/24/2013 5:51 AM, Nick Coghlan wrote: > > > Why not just say something like "Cannot propagate exception..."; it's > > simpler than "Unpropagatable exception...". > > That would definitely be an improvement on the status quo and avoids > Antoine's concern about an adjective being interpreted as an inherent > property of the exception rather than the circumstances where the > exception was encountered. > First one I've heard that accurately and unambiguously and briefly describes the issue. +1 -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130924/9e724b78/attachment.html> From g.brandl at gmx.net Wed Sep 25 08:22:27 2013 From: g.brandl at gmx.net (Georg Brandl) Date: Wed, 25 Sep 2013 08:22:27 +0200 Subject: [Python-Dev] Revert #12085 fix for __del__ attribute error message In-Reply-To: <5240BCAA.5030604@canterbury.ac.nz> References: <l1l2bm$1bp$1@ger.gmane.org> <20130921221524.55D4225007F@webabinitio.net> <l1lbkl$k0g$1@ger.gmane.org> <CAP7+vJLHO12ZNwX4UZ0c_BF9pr+xxxLZfy2dGjBcMVenGow9vQ@mail.gmail.com> <l1n9o6$ap3$1@ger.gmane.org> <CAP7+vJL+0XEBpOnVbTpZJNLzahgm8EkyuXdGdhE27TVwHu4GAg@mail.gmail.com> <l1nkdh$lh7$1@ger.gmane.org> <CADiSq7eGpFtPbRc8BmiP2H9V2PVMma3ycX3Ky_MJ-F+TpEiU6w@mail.gmail.com> <CAP7+vJKAP5R+2R++z8+DQjSW6EVycoczB2gWeo759-NG4G6Veg@mail.gmail.com> <523FDB8F.1080207@canterbury.ac.nz> <CADiSq7caBSr4ujFo0HBYGF91SJp2NcP-_SreBqad-=KpcZdeCw@mail.gmail.com> <20130923104500.5645fdbb@pitrou.net> <CADiSq7dJAdStAzPYijQTC_fC6_01Jkt4kidSh+YDxWNNv_0tOA@mail.gmail.com> <20130923172245.0cbaec81@fsol> <5240BCAA.5030604@canterbury.ac.nz> Message-ID: <l1tvd9$6rg$1@ger.gmane.org> Am 24.09.2013 00:11, schrieb Greg Ewing: > Antoine Pitrou wrote: >> Yes, but I agree with Greg that "unraisable" is wrong. After all, it >> was raised, and it can even be caught by the programmer (inside >> __del__). > > How about something like "Uncaught exception in __del__ > method ignored"? It explains fairly clearly what has > happened, and also indicates what do do about it -- > catch it in the __del__ method. "Exception in __del__ caught and not propagated:" Georg From g.brandl at gmx.net Wed Sep 25 08:30:37 2013 From: g.brandl at gmx.net (Georg Brandl) Date: Wed, 25 Sep 2013 08:30:37 +0200 Subject: [Python-Dev] Best practice for documentation for std lib In-Reply-To: <CAP7+vJ+oVnq07Lfh_R1nqaqtK-EnxDaqPeZVTQMQTzfcEdqagg@mail.gmail.com> References: <20130922031304.GE19939@ando> <CAF-Rda_sTOiKi6c9ihFVL-N90PC-xHxOAMm1im0s-KVyy=L_Ng@mail.gmail.com> <87ioxs3iix.fsf@uwakimon.sk.tsukuba.ac.jp> <CAP7+vJJucubZtXHhnNbAcE0GWfQTnOozHK9W=uv3CWkMX15XGA@mail.gmail.com> <CAF-Rda-qJ3fZQTpVO_x31WhWOaf7Q4+0n7w-sFZgK8tXWLL0zw@mail.gmail.com> <CAP7+vJLmLZOYmMeoKqoqrAGqVF=Z41iYBRiis5buAMyaON+_+A@mail.gmail.com> <CAF-Rda8H2hi5V=HakbGsD2WzH0HQ6hLuxkQYZ3ZC+MzDfb1hVw@mail.gmail.com> <CAP7+vJ+oVnq07Lfh_R1nqaqtK-EnxDaqPeZVTQMQTzfcEdqagg@mail.gmail.com> Message-ID: <l1tvsj$jmv$1@ger.gmane.org> Am 23.09.2013 00:03, schrieb Guido van Rossum: > AFAIU, the proposal is to embed parts of the concise docstring into the more > verbose .rst documentation. > > > I still think that's a bad idea. Someone editing the docstring may introduce a > terminology change or some other style/grammar/flow change that makes perfectly > sense by itself but doesn't when taken in the context of the larger .rst doc > (e.g. it could introducing duplication or cause dissonance between the two > parts). Yes, this style is better suited for smaller documentations where there isn't that much more info in the .rst than in the docstring -- the .rst giving maybe an introduction and then just a logical order in which docstrings are pulled. Since we have the policy (now reconfirmed) of small docstrings and more verbose prose on docs.python.org, this would not be feasible there. > Also, someone who wants to improve the docs would have to figure out how > to edit the source code if they wanted to change some part of the docs. I agree that is another good point. And should the "suggest a change on docs.python.org" feature ever be shipped, it will be much harder, if not impossible, to generate a patch and let developers submit it automatically. > I write .rst docs quite a lot, and as such I do notice the annoying amount > of duplication between docstrings and .rst docs. Without doubt, all the > free-flow text and examples in .rst have to be written manually and are very > important. But for dry method references, it's certainly interesting to > explore the idea of writing them once in the code and then having Sphinx > automatically insert the relevant parts into the .rst before generating HTML > from it. For end users (those who read the docs online) the result is > indistinguishable from what we have now. For us devs it means writing the > same text only once and maintaining it in a single place. > > > You seem to have caught the DRY bug. But it doesn't always make sense to factor > things so that everything is said in exactly one place. (For an example of > abstraction gone wild, ironically, see docutils and sphinx. I challenge you to > find out the actual code that translates e.g. :meth:`foobar` into <a href="(what > goes here?)">foobar</a>. :-) PSA: I can only recommend to everyone not to take up Guido's dare... (and I'm not terribly proud of that). cheers, Georg From eliben at gmail.com Wed Sep 25 15:15:19 2013 From: eliben at gmail.com (Eli Bendersky) Date: Wed, 25 Sep 2013 06:15:19 -0700 Subject: [Python-Dev] astgen.py vs. ASDL for ASTs Message-ID: <CAF-Rda_bRpq_aGtiUSyODiykB=C0eA4HWGCBmnHK0bZuTxtU3w@mail.gmail.com> Hello, Some history (as best as I could collect it) followed by a question: Before Python 2.5, the ast stdlib module was auto generated by a script named astgen.py from a textual AST definition in Tools/compiler/ast.txt. Since 2.5 (http://www.python.org/dev/peps/pep-0339/) ASTs are part of the normal compilation flow by the Python compiler itself, and the ast module uses the same ASTs generated from Parser/Python.asdl My question is, when the switch was made in 2.5 - why didn't the existing AST-generating code was used and the path moved to ASDL instead? What advantages does ASDL have over the previous approach? One reason I could think of is that ASDL nodes are typed and that's maybe better for the generated C code to handle. [My interest here is personal. One of my projects (pycparser) uses a astgen.py-like approach, and in a new project I'm considering the options again and remembered ASDL. ADSL's documentation it extremely scarce online - seems like CPython is one of its only somewhat-visible users these days] Thanks in advance, Eli -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130925/c16b75c0/attachment.html> From benjamin at python.org Wed Sep 25 15:28:08 2013 From: benjamin at python.org (Benjamin Peterson) Date: Wed, 25 Sep 2013 09:28:08 -0400 Subject: [Python-Dev] astgen.py vs. ASDL for ASTs In-Reply-To: <CAF-Rda_bRpq_aGtiUSyODiykB=C0eA4HWGCBmnHK0bZuTxtU3w@mail.gmail.com> References: <CAF-Rda_bRpq_aGtiUSyODiykB=C0eA4HWGCBmnHK0bZuTxtU3w@mail.gmail.com> Message-ID: <CAPZV6o-CUGTB0cK4cXPamQACWcWZRQ57eP=HVOD1_dNKXm7qRQ@mail.gmail.com> 2013/9/25 Eli Bendersky <eliben at gmail.com>: > Hello, > > Some history (as best as I could collect it) followed by a question: > > Before Python 2.5, the ast stdlib module was auto generated by a script > named astgen.py from a textual AST definition in Tools/compiler/ast.txt. Sorry, what Python 2.4 ast stdlib module? Any module named "ast" (or "_ast") didn't exist until 2.5. -- Regards, Benjamin From brett at python.org Wed Sep 25 15:35:02 2013 From: brett at python.org (Brett Cannon) Date: Wed, 25 Sep 2013 09:35:02 -0400 Subject: [Python-Dev] astgen.py vs. ASDL for ASTs In-Reply-To: <CAF-Rda_bRpq_aGtiUSyODiykB=C0eA4HWGCBmnHK0bZuTxtU3w@mail.gmail.com> References: <CAF-Rda_bRpq_aGtiUSyODiykB=C0eA4HWGCBmnHK0bZuTxtU3w@mail.gmail.com> Message-ID: <CAP1=2W6E+9a-R+CHP2O_ev42SCQnRrBngR7i5PV9jrPNUYT4hw@mail.gmail.com> cc'ing Jeremy Hylton who made the decision to use Zephyr. On Wed, Sep 25, 2013 at 9:15 AM, Eli Bendersky <eliben at gmail.com> wrote: > Hello, > > Some history (as best as I could collect it) followed by a question: > > Before Python 2.5, the ast stdlib module was auto generated by a script > named astgen.py from a textual AST definition in Tools/compiler/ast.txt. > > Since 2.5 (http://www.python.org/dev/peps/pep-0339/) ASTs are part of the > normal compilation flow by the Python compiler itself, and the ast module > uses the same ASTs generated from Parser/Python.asdl > > My question is, when the switch was made in 2.5 - why didn't the existing > AST-generating code was used and the path moved to ASDL instead? What > advantages does ASDL have over the previous approach? One reason I could > think of is that ASDL nodes are typed and that's maybe better for the > generated C code to handle. > > [My interest here is personal. One of my projects (pycparser) uses a > astgen.py-like approach, and in a new project I'm considering the options > again and remembered ASDL. ADSL's documentation it extremely scarce online > - seems like CPython is one of its only somewhat-visible users these days] > > Thanks in advance, > Eli > > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130925/17b719c5/attachment.html> From brett at python.org Wed Sep 25 15:35:22 2013 From: brett at python.org (Brett Cannon) Date: Wed, 25 Sep 2013 09:35:22 -0400 Subject: [Python-Dev] astgen.py vs. ASDL for ASTs In-Reply-To: <CAPZV6o-CUGTB0cK4cXPamQACWcWZRQ57eP=HVOD1_dNKXm7qRQ@mail.gmail.com> References: <CAF-Rda_bRpq_aGtiUSyODiykB=C0eA4HWGCBmnHK0bZuTxtU3w@mail.gmail.com> <CAPZV6o-CUGTB0cK4cXPamQACWcWZRQ57eP=HVOD1_dNKXm7qRQ@mail.gmail.com> Message-ID: <CAP1=2W42p+D-bdV8H3m+KGKjDLbe5_Fc=7r-cpceg6-YuyRrmQ@mail.gmail.com> On Wed, Sep 25, 2013 at 9:28 AM, Benjamin Peterson <benjamin at python.org>wrote: > 2013/9/25 Eli Bendersky <eliben at gmail.com>: > > Hello, > > > > Some history (as best as I could collect it) followed by a question: > > > > Before Python 2.5, the ast stdlib module was auto generated by a script > > named astgen.py from a textual AST definition in Tools/compiler/ast.txt. > > Sorry, what Python 2.4 ast stdlib module? Any module named "ast" (or > "_ast") didn't exist until 2.5. > I think Eli is talking about the compiler package's own AST. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130925/da14ed8d/attachment.html> From eliben at gmail.com Wed Sep 25 15:35:53 2013 From: eliben at gmail.com (Eli Bendersky) Date: Wed, 25 Sep 2013 06:35:53 -0700 Subject: [Python-Dev] astgen.py vs. ASDL for ASTs In-Reply-To: <CAPZV6o-CUGTB0cK4cXPamQACWcWZRQ57eP=HVOD1_dNKXm7qRQ@mail.gmail.com> References: <CAF-Rda_bRpq_aGtiUSyODiykB=C0eA4HWGCBmnHK0bZuTxtU3w@mail.gmail.com> <CAPZV6o-CUGTB0cK4cXPamQACWcWZRQ57eP=HVOD1_dNKXm7qRQ@mail.gmail.com> Message-ID: <CAF-Rda-Ro0RqPc1Jwx-85iYFuJuu3cZFbfr=9U4JHfjS3-d-8Q@mail.gmail.com> On Wed, Sep 25, 2013 at 6:28 AM, Benjamin Peterson <benjamin at python.org>wrote: > 2013/9/25 Eli Bendersky <eliben at gmail.com>: > > Hello, > > > > Some history (as best as I could collect it) followed by a question: > > > > Before Python 2.5, the ast stdlib module was auto generated by a script > > named astgen.py from a textual AST definition in Tools/compiler/ast.txt. > > Sorry, what Python 2.4 ast stdlib module? Any module named "ast" (or > "_ast") didn't exist until 2.5. > Lib/compiler/ast.py (that's in 2.4) Sorry, I should have been more precise -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130925/2a547895/attachment.html> From eliben at gmail.com Wed Sep 25 15:36:51 2013 From: eliben at gmail.com (Eli Bendersky) Date: Wed, 25 Sep 2013 06:36:51 -0700 Subject: [Python-Dev] astgen.py vs. ASDL for ASTs In-Reply-To: <CAF-Rda-Ro0RqPc1Jwx-85iYFuJuu3cZFbfr=9U4JHfjS3-d-8Q@mail.gmail.com> References: <CAF-Rda_bRpq_aGtiUSyODiykB=C0eA4HWGCBmnHK0bZuTxtU3w@mail.gmail.com> <CAPZV6o-CUGTB0cK4cXPamQACWcWZRQ57eP=HVOD1_dNKXm7qRQ@mail.gmail.com> <CAF-Rda-Ro0RqPc1Jwx-85iYFuJuu3cZFbfr=9U4JHfjS3-d-8Q@mail.gmail.com> Message-ID: <CAF-Rda9jxiS90PcRj9uW66okRSNb+gO7QFxra462HZtFVydU5A@mail.gmail.com> On Wed, Sep 25, 2013 at 6:35 AM, Eli Bendersky <eliben at gmail.com> wrote: > > > > On Wed, Sep 25, 2013 at 6:28 AM, Benjamin Peterson <benjamin at python.org>wrote: > >> 2013/9/25 Eli Bendersky <eliben at gmail.com>: >> > Hello, >> > >> > Some history (as best as I could collect it) followed by a question: >> > >> > Before Python 2.5, the ast stdlib module was auto generated by a script >> > named astgen.py from a textual AST definition in Tools/compiler/ast.txt. >> >> Sorry, what Python 2.4 ast stdlib module? Any module named "ast" (or >> "_ast") didn't exist until 2.5. >> > > Lib/compiler/ast.py (that's in 2.4) > > Sorry, I should have been more precise > http://docs.python.org/release/2.4/lib/module-compiler.ast.html -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130925/7d5d27a9/attachment.html> From rajalokan at gmail.com Wed Sep 25 10:07:28 2013 From: rajalokan at gmail.com (alok bhan) Date: Wed, 25 Sep 2013 13:37:28 +0530 Subject: [Python-Dev] Installing python27-devel on Centos5.8 running python2.4 Message-ID: <CAL9ehhpSNL3RfCgka-F=gMo4bMen7pgrYJcujO+gFOLT4VK6jw@mail.gmail.com> Hi All I need help in installing python development packages (python27-devel) on Centos5.8. Upgrading hardware is not an option as of now. I've installed python2.7 in the same machine by $ wget --no-check-certificate http://www.python.org/ftp/python/2.7.3/Python-2.7.3.tgz $ ./configure --prefix=/usr/local/Python2.7 $ make altinstall Now I need help in installing python-devel package for 2.7. $ sudo yum info python2.7-devel gives no info. Whereas the package is there for python26-devel. Tried many suggestions after googling but still no luck. Any help or pointer for exact steps will be great. Thanks Alok Bhan -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130925/b75d965f/attachment.html> From brett at python.org Wed Sep 25 18:17:12 2013 From: brett at python.org (Brett Cannon) Date: Wed, 25 Sep 2013 12:17:12 -0400 Subject: [Python-Dev] Installing python27-devel on Centos5.8 running python2.4 In-Reply-To: <CAL9ehhpSNL3RfCgka-F=gMo4bMen7pgrYJcujO+gFOLT4VK6jw@mail.gmail.com> References: <CAL9ehhpSNL3RfCgka-F=gMo4bMen7pgrYJcujO+gFOLT4VK6jw@mail.gmail.com> Message-ID: <CAP1=2W6ja8FnoMnCm5U1CjJOnLUx-waJx=Ezz4jFw3U6VphA=A@mail.gmail.com> This list is for the developing *of *Python, not it's use. You can try asking on python-list for some help. On Wed, Sep 25, 2013 at 4:07 AM, alok bhan <rajalokan at gmail.com> wrote: > Hi All > > I need help in installing python development packages (python27-devel) on > Centos5.8. Upgrading hardware is not an option as of now. > > I've installed python2.7 in the same machine by > $ wget --no-check-certificate > http://www.python.org/ftp/python/2.7.3/Python-2.7.3.tgz > $ ./configure --prefix=/usr/local/Python2.7 > $ make altinstall > > Now I need help in installing python-devel package for 2.7. > > $ sudo yum info python2.7-devel gives no info. Whereas the package is > there for python26-devel. > > Tried many suggestions after googling but still no luck. Any help or > pointer for exact steps will be great. > > Thanks > Alok Bhan > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/brett%40python.org > > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130925/196a0926/attachment.html> From barry at python.org Wed Sep 25 22:50:22 2013 From: barry at python.org (Barry Warsaw) Date: Wed, 25 Sep 2013 16:50:22 -0400 Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for pronouncement? References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com> Message-ID: <20130925165022.3d67e290@anarchist> On Sep 23, 2013, at 09:15 PM, Nick Coghlan wrote: >With the last round of updates, I believe PEP 453 is ready for >Martin's pronouncement. I want to raise an objection to PEP's proposal to add this as a new feature to Python 2.7 and 3.3. I understand the rationale as stated here: http://www.python.org/dev/peps/pep-0453/#id38 but I still object. ;) I think we've learned it's generally pretty risky to add new features in point releases, and this is a fairly major (and would be the first?) violation of the principal of Python 2.7's conservative maintenance. At the very least, I'd like to know why there's no other alternative. For example, you could provide a PyPI package with this functionality, and let the binary packagers adopt it into their binary packages, for platforms that care. So for example, Linux systems which start from the source tarball could opt out (by doing nothing special) since there are other ways to achieve a similar benefit. For OS X and Windows users getting binary downloads from www.python.org, I think it would be acceptable if the .dmg or .msi included this external package. Why does it have to be added to the source tree for stable releases? Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: <http://mail.python.org/pipermail/python-dev/attachments/20130925/369258fa/attachment.sig> From solipsis at pitrou.net Wed Sep 25 23:04:20 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 25 Sep 2013 23:04:20 +0200 Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for pronouncement? References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com> <20130925165022.3d67e290@anarchist> Message-ID: <20130925230420.4dac278e@fsol> On Wed, 25 Sep 2013 16:50:22 -0400 Barry Warsaw <barry at python.org> wrote: > On Sep 23, 2013, at 09:15 PM, Nick Coghlan wrote: > > >With the last round of updates, I believe PEP 453 is ready for > >Martin's pronouncement. > > I want to raise an objection to PEP's proposal to add this as a new feature to > Python 2.7 and 3.3. I understand the rationale as stated here: > > http://www.python.org/dev/peps/pep-0453/#id38 > > but I still object. ;) > > I think we've learned it's generally pretty risky to add new features in point > releases, and this is a fairly major (and would be the first?) violation of > the principal of Python 2.7's conservative maintenance. > > At the very least, I'd like to know why there's no other alternative. For > example, you could provide a PyPI package with this functionality, and let the > binary packagers adopt it into their binary packages, for platforms that care. > > So for example, Linux systems which start from the source tarball could opt > out (by doing nothing special) since there are other ways to achieve a similar > benefit. For OS X and Windows users getting binary downloads from > www.python.org, I think it would be acceptable if the .dmg or .msi included > this external package. If it's bundled in the official binary installers, it makes little sense not to add to the source tree, IMHO. First because most people get one of the installers rather than the source tree (on http://www.python.org/webstats/, MSI builds win by ten to one). Second because the few people who get the source tree are experienced users who will get bitten less by hypothetical quirks or oddities. Third because otherwise we're probably making things harder for our beloved packagers ;-) Regards Antoine. From ncoghlan at gmail.com Wed Sep 25 23:14:18 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 26 Sep 2013 07:14:18 +1000 Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for pronouncement? In-Reply-To: <20130925165022.3d67e290@anarchist> References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com> <20130925165022.3d67e290@anarchist> Message-ID: <CADiSq7dxDZE_bRSdJnVfd_t3bVnO=irw2MoV8SOofXE-gta8KA@mail.gmail.com> On 26 Sep 2013 06:53, "Barry Warsaw" <barry at python.org> wrote: > > On Sep 23, 2013, at 09:15 PM, Nick Coghlan wrote: > > >With the last round of updates, I believe PEP 453 is ready for > >Martin's pronouncement. > > I want to raise an objection to PEP's proposal to add this as a new feature to > Python 2.7 and 3.3. I understand the rationale as stated here: > > http://www.python.org/dev/peps/pep-0453/#id38 > > but I still object. ;) > > I think we've learned it's generally pretty risky to add new features in point > releases, and this is a fairly major (and would be the first?) violation of > the principal of Python 2.7's conservative maintenance. > > At the very least, I'd like to know why there's no other alternative. For > example, you could provide a PyPI package with this functionality, and let the > binary packagers adopt it into their binary packages, for platforms that care. > > So for example, Linux systems which start from the source tarball could opt > out (by doing nothing special) since there are other ways to achieve a similar > benefit. For OS X and Windows users getting binary downloads from > www.python.org, I think it would be acceptable if the .dmg or .msi included > this external package. > > Why does it have to be added to the source tree for stable releases? If it can get into the installers another way, it doesn't, really. It only needs to be in the source tree for 3.4+. Alternatively, we could be explicit that the "don't remove it" guideline only applies in 3.4+, where it will be needed by pyvenv. That way, people doing custom builds from source can easily bootstrap pip and we don't need to come up with an alternate solution for getting them into the installers. Cheers, Nick. > > Cheers, > -Barry > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130926/7bd25867/attachment.html> From donald at stufft.io Wed Sep 25 23:33:24 2013 From: donald at stufft.io (Donald Stufft) Date: Wed, 25 Sep 2013 17:33:24 -0400 Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for pronouncement? In-Reply-To: <20130925165022.3d67e290@anarchist> References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com> <20130925165022.3d67e290@anarchist> Message-ID: <109B26FC-32A9-4DDE-9639-E48A0389455D@stufft.io> On Sep 25, 2013, at 4:50 PM, Barry Warsaw <barry at python.org> wrote: > Why does it have to be added to the source tree for stable releases? I think it should be placed in the source tree for the stable releases. The reasoning is that 2.7 is going to stick around for a long time. Immediately this won't be ubiquitous but as time goes on you'll be able to be ensured that a ``python -m ensure pip`` exists so that in situations where you don't have pip you'll be able to install it. While not directly relevant to the change I do think this is something users support. I've received a fair but of feedback as I was writing the original draft of the PEP and then throughout the process when me and Nick were working on it. Almost all of it was positive (some of it extremely so) a fair bit of them pointed out the backport to 2.7 as something they were *really* wanting. An early draft of this did not have the backport to 2.7 and when I showed *that* version around to get feedback people were less enthusiastic about it and generally viewed it as "nice but worthless to me for N years". What users want isn't rationale in and of itself but I think it's an important data point, especially given how long 2.7.LASTEVER is going to be relevant to end users. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: <http://mail.python.org/pipermail/python-dev/attachments/20130925/54b62fb6/attachment.sig> From barry at python.org Wed Sep 25 23:51:29 2013 From: barry at python.org (Barry Warsaw) Date: Wed, 25 Sep 2013 17:51:29 -0400 Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for pronouncement? In-Reply-To: <109B26FC-32A9-4DDE-9639-E48A0389455D@stufft.io> References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com> <20130925165022.3d67e290@anarchist> <109B26FC-32A9-4DDE-9639-E48A0389455D@stufft.io> Message-ID: <20130925175129.1edc4348@anarchist> On Sep 25, 2013, at 05:33 PM, Donald Stufft wrote: >I think it should be placed in the source tree for the stable releases. The >reasoning is that 2.7 is going to stick around for a long time. Immediately >this won't be ubiquitous but as time goes on you'll be able to be ensured >that a ``python -m ensure pip`` exists so that in situations where you don't >have pip you'll be able to install it. > >While not directly relevant to the change I do think this is something users >support. I've received a fair but of feedback as I was writing the original >draft of the PEP and then throughout the process when me and Nick >were working on it. Almost all of it was positive (some of it extremely so) >a fair bit of them pointed out the backport to 2.7 as something they were >*really* wanting. > >An early draft of this did not have the backport to 2.7 and when I showed >*that* version around to get feedback people were less enthusiastic >about it and generally viewed it as "nice but worthless to me for N years". Yeah, I get all this, but it's essentially the same reasoning that lead to Python 2.2.1's addition of True and False. The same flaw occurs though because you cannot guarantee that invocation of `python -m ensurepip` will work unless you micro version check (LBYL) or prepare to catch resulting errors. This essentially means that if you want to write portable Python 2.7 code and scripts, you have to be very cautious, or what works on one box won't necessarily work on another, even though they both have "Python 2.7". It also means that anybody who's documenting about this great new feature has to warn people that, well, maybe it won't work on your machine even though you have Python 2.7 because you don't have a new enough Python 2.7. It means that not all Python 2.7's are alike in a rather fundamental and highly visible way. I personally think that's a recipe for more problems than it solves, and if I was the RM for 2.7 I would not allow it. But I'm not. >What users want isn't rationale in and of itself but I think it's an important >data point, especially given how long 2.7.LASTEVER is going to be >relevant to end users. Users want what users want. It's our job to make the best technical decisions based on all the facts. I understand that Python 2.7 will be around for a long time, and that it would be very convenient to do this. Why is this not opening up the door to more new features being added in future Python 2.7 point releases (or any other stable release)? At least I think the burden should be very high, and the PEP should do a better job of explaining why *no* other option will accomplish the goal. Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: <http://mail.python.org/pipermail/python-dev/attachments/20130925/91c80684/attachment.sig> From barry at python.org Wed Sep 25 23:53:48 2013 From: barry at python.org (Barry Warsaw) Date: Wed, 25 Sep 2013 17:53:48 -0400 Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for pronouncement? In-Reply-To: <CADiSq7dxDZE_bRSdJnVfd_t3bVnO=irw2MoV8SOofXE-gta8KA@mail.gmail.com> References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com> <20130925165022.3d67e290@anarchist> <CADiSq7dxDZE_bRSdJnVfd_t3bVnO=irw2MoV8SOofXE-gta8KA@mail.gmail.com> Message-ID: <20130925175348.20f2e263@anarchist> On Sep 26, 2013, at 07:14 AM, Nick Coghlan wrote: >On 26 Sep 2013 06:53, "Barry Warsaw" <barry at python.org> wrote: >> >> Why does it have to be added to the source tree for stable releases? > >If it can get into the installers another way, it doesn't, really. It only >needs to be in the source tree for 3.4+. Sure, new features can of course go in the development release. :) -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: <http://mail.python.org/pipermail/python-dev/attachments/20130925/7ca753d7/attachment.sig> From donald at stufft.io Thu Sep 26 00:15:18 2013 From: donald at stufft.io (Donald Stufft) Date: Wed, 25 Sep 2013 18:15:18 -0400 Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for pronouncement? In-Reply-To: <20130925175129.1edc4348@anarchist> References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com> <20130925165022.3d67e290@anarchist> <109B26FC-32A9-4DDE-9639-E48A0389455D@stufft.io> <20130925175129.1edc4348@anarchist> Message-ID: <55AAF039-CCAB-467F-8FA3-74D31CB80A66@stufft.io> On Sep 25, 2013, at 5:51 PM, Barry Warsaw <barry at python.org> wrote: > On Sep 25, 2013, at 05:33 PM, Donald Stufft wrote: > >> I think it should be placed in the source tree for the stable releases. The >> reasoning is that 2.7 is going to stick around for a long time. Immediately >> this won't be ubiquitous but as time goes on you'll be able to be ensured >> that a ``python -m ensure pip`` exists so that in situations where you don't >> have pip you'll be able to install it. >> >> While not directly relevant to the change I do think this is something users >> support. I've received a fair but of feedback as I was writing the original >> draft of the PEP and then throughout the process when me and Nick >> were working on it. Almost all of it was positive (some of it extremely so) >> a fair bit of them pointed out the backport to 2.7 as something they were >> *really* wanting. >> >> An early draft of this did not have the backport to 2.7 and when I showed >> *that* version around to get feedback people were less enthusiastic >> about it and generally viewed it as "nice but worthless to me for N years". > > Yeah, I get all this, but it's essentially the same reasoning that lead to > Python 2.2.1's addition of True and False. The same flaw occurs though > because you cannot guarantee that invocation of `python -m ensurepip` will > work unless you micro version check (LBYL) or prepare to catch resulting > errors. This essentially means that if you want to write portable Python 2.7 > code and scripts, you have to be very cautious, or what works on one box won't > necessarily work on another, even though they both have "Python 2.7". > > It also means that anybody who's documenting about this great new feature has > to warn people that, well, maybe it won't work on your machine even though you > have Python 2.7 because you don't have a new enough Python 2.7. > > It means that not all Python 2.7's are alike in a rather fundamental and > highly visible way. > > I personally think that's a recipe for more problems than it solves, and if I > was the RM for 2.7 I would not allow it. But I'm not. Well I don't think many scripts will be executing ensurepip, maybe i'm wrong, but yes it does mean that not all Python 2.7's are alike. Most likely though at some point 2.7.XXX is going to be near standard as the 2.7 hold outs stay on it and time progresses. It more or less shifts the "time until projects can assume people have pip available or easily installable" from "until 3.4+ is ubiquitous and 2.7 is gone (Years?) to "until Python 2.7.6 is ubiquitous" which will be a much shorter time. > >> What users want isn't rationale in and of itself but I think it's an important >> data point, especially given how long 2.7.LASTEVER is going to be >> relevant to end users. > > Users want what users want. It's our job to make the best technical decisions > based on all the facts. I understand that Python 2.7 will be around for a > long time, and that it would be very convenient to do this. Why is this not > opening up the door to more new features being added in future Python 2.7 > point releases (or any other stable release)? > Other than the fact that it in itself is an exception to the rule this actually strengthens that because it makes it easier for people to install new things into Python 2.7 outside of the release cycle. So things like the new enum module can more easily be backported and used as an installable than as part of the language. The obvious exemption to this is language level features, but as this itself isn't a language level feature i'm not sure how relevant that is. > At least I think the burden should be very high, and the PEP should do a > better job of explaining why *no* other option will accomplish the goal. This is totally fair and we can expand on it more for sure. > > Cheers, > -Barry > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: <http://mail.python.org/pipermail/python-dev/attachments/20130925/8cc17b8c/attachment.sig> From ncoghlan at gmail.com Thu Sep 26 00:51:31 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 26 Sep 2013 08:51:31 +1000 Subject: [Python-Dev] [Python-checkins] cpython: Close #19030: improvements to inspect and Enum. In-Reply-To: <3clLrQ2f4wz7LjP@mail.python.org> References: <3clLrQ2f4wz7LjP@mail.python.org> Message-ID: <CADiSq7eBoK_i0qtQqt7NQu7RNM5FHYdfz__mJq+Kb+Y3iH3e5w@mail.gmail.com> Hmm, I think I just noticed a subtle bug in the new version that I missed previously - I believe it may now misclassify a function in an instance dict as a method. Would need to try it out to be sure. Cheers, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130926/e56e5739/attachment.html> From ncoghlan at gmail.com Thu Sep 26 01:01:12 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 26 Sep 2013 09:01:12 +1000 Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for pronouncement? In-Reply-To: <20130925175129.1edc4348@anarchist> References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com> <20130925165022.3d67e290@anarchist> <109B26FC-32A9-4DDE-9639-E48A0389455D@stufft.io> <20130925175129.1edc4348@anarchist> Message-ID: <CADiSq7fVrruk1zcE-ZwHUOyUqOPhTBnn2jcp9rKQ8dS6QeL+Zw@mail.gmail.com> On 26 Sep 2013 07:54, "Barry Warsaw" <barry at python.org> wrote: > > On Sep 25, 2013, at 05:33 PM, Donald Stufft wrote: > > >I think it should be placed in the source tree for the stable releases. The > >reasoning is that 2.7 is going to stick around for a long time. Immediately > >this won't be ubiquitous but as time goes on you'll be able to be ensured > >that a ``python -m ensure pip`` exists so that in situations where you don't > >have pip you'll be able to install it. > > > >While not directly relevant to the change I do think this is something users > >support. I've received a fair but of feedback as I was writing the original > >draft of the PEP and then throughout the process when me and Nick > >were working on it. Almost all of it was positive (some of it extremely so) > >a fair bit of them pointed out the backport to 2.7 as something they were > >*really* wanting. > > > >An early draft of this did not have the backport to 2.7 and when I showed > >*that* version around to get feedback people were less enthusiastic > >about it and generally viewed it as "nice but worthless to me for N years". > > Yeah, I get all this, but it's essentially the same reasoning that lead to > Python 2.2.1's addition of True and False. The same flaw occurs though > because you cannot guarantee that invocation of `python -m ensurepip` will > work unless you micro version check (LBYL) or prepare to catch resulting > errors. This essentially means that if you want to write portable Python 2.7 > code and scripts, you have to be very cautious, or what works on one box won't > necessarily work on another, even though they both have "Python 2.7". There's a bit of a difference between adding new builtins and adding a new utility module, though. The only regular invocations of ensurepip in 2.7 and 3.3 should be from our installers and people doing a custom build from source. Would a leading underscore in the module name make you more comfortable with the idea? It's really intended mostly as a hidden implementation detail of the installers and pyvenv anyway, so calling it "_ensurepip" would help make that explicit while still letting people invoke it directly if absolutely necessary. Cheers, Nick. > > It also means that anybody who's documenting about this great new feature has > to warn people that, well, maybe it won't work on your machine even though you > have Python 2.7 because you don't have a new enough Python 2.7. > > It means that not all Python 2.7's are alike in a rather fundamental and > highly visible way. > > I personally think that's a recipe for more problems than it solves, and if I > was the RM for 2.7 I would not allow it. But I'm not. > > >What users want isn't rationale in and of itself but I think it's an important > >data point, especially given how long 2.7.LASTEVER is going to be > >relevant to end users. > > Users want what users want. It's our job to make the best technical decisions > based on all the facts. I understand that Python 2.7 will be around for a > long time, and that it would be very convenient to do this. Why is this not > opening up the door to more new features being added in future Python 2.7 > point releases (or any other stable release)? > > At least I think the burden should be very high, and the PEP should do a > better job of explaining why *no* other option will accomplish the goal. > > Cheers, > -Barry > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130926/b22663ba/attachment-0001.html> From barry at python.org Thu Sep 26 01:04:10 2013 From: barry at python.org (Barry Warsaw) Date: Wed, 25 Sep 2013 19:04:10 -0400 Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for pronouncement? In-Reply-To: <55AAF039-CCAB-467F-8FA3-74D31CB80A66@stufft.io> References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com> <20130925165022.3d67e290@anarchist> <109B26FC-32A9-4DDE-9639-E48A0389455D@stufft.io> <20130925175129.1edc4348@anarchist> <55AAF039-CCAB-467F-8FA3-74D31CB80A66@stufft.io> Message-ID: <20130925190410.0a5729f9@anarchist> On Sep 25, 2013, at 06:15 PM, Donald Stufft wrote: >Well I don't think many scripts will be executing ensurepip, maybe i'm wrong, >but yes it does mean that not all Python 2.7's are alike. Most likely though >at some point 2.7.XXX is going to be near standard as the 2.7 hold outs stay >on it and time progresses. I think it's too optimistic to hope that we'll ever see significant convergence across the Python 2.7 ecosystem. For example, I'd be really surprised if you'd see this show up in a stable release of Debian or Ubuntu (the only ones I can speak somewhat authoritatively about), even if we wanted to cherrypick updates in newer Python 2.7 point releases. Linux distros tend to be even more anal about avoiding new features in stable releases, for the same really good reasons, IMHO :). I think for the platforms you're targeting (OS X, Windows), you're going to see a wide variety for a very long time. It's depressing how infrequently people upgrade *anything*. Another reason to oppose this is what I've heard quite often from people regarding Python 2.7. I've been told that many folks are actually really happy with using 2.7 precisely because it extremely stable. They don't have to worry about their stuff breaking or incompatibilities cropping up, because it *doesn't* change. Python 2.7 is like a long-term maintenance release, with guaranteed multi-year stability, and I think while that was unintended, it's a *good* thing. This PEP proposes to break that, and I'm loathe to give up that good reputation for this particular feature. >It more or less shifts the "time until projects can assume people have pip >available or easily installable" from "until 3.4+ is ubiquitous and 2.7 is >gone (Years?) to "until Python 2.7.6 is ubiquitous" which will be a much >shorter time. Yeah, maybe, but I'm pretty skeptical about the enthusiasm for upgrades. ;) >> Users want what users want. It's our job to make the best technical >> decisions based on all the facts. I understand that Python 2.7 will be >> around for a long time, and that it would be very convenient to do this. >> Why is this not opening up the door to more new features being added in >> future Python 2.7 point releases (or any other stable release)? >> > >Other than the fact that it in itself is an exception to the rule this >actually strengthens that because it makes it easier for people to install >new things into Python 2.7 outside of the release cycle. So things like the >new enum module can more easily be backported and used as an installable than >as part of the language. The obvious exemption to this is language level >features, but as this itself isn't a language level feature i'm not sure how >relevant that is. So, why shouldn't we add enum to the Python 2.7 stdlib? Or any other new feature? Just be aware that if this PEP is accepted for Python 2.7, the bar will be set much lower for other Really Useful Features That Make Users Lives Better. >> At least I think the burden should be very high, and the PEP should do a >> better job of explaining why *no* other option will accomplish the goal. > >This is totally fair and we can expand on it more for sure. Cool. Maybe in the course of that discussion, a better alternative that doesn't add a new feature to Python 2.7 will present itself. I really hope so. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: <http://mail.python.org/pipermail/python-dev/attachments/20130925/05772c7d/attachment.sig> From donald at stufft.io Thu Sep 26 01:08:32 2013 From: donald at stufft.io (Donald Stufft) Date: Wed, 25 Sep 2013 19:08:32 -0400 Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for pronouncement? In-Reply-To: <20130925190410.0a5729f9@anarchist> References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com> <20130925165022.3d67e290@anarchist> <109B26FC-32A9-4DDE-9639-E48A0389455D@stufft.io> <20130925175129.1edc4348@anarchist> <55AAF039-CCAB-467F-8FA3-74D31CB80A66@stufft.io> <20130925190410.0a5729f9@anarchist> Message-ID: <7FBFFC39-A0AF-47FD-9715-944D13212D10@stufft.io> On Sep 25, 2013, at 7:04 PM, Barry Warsaw <barry at python.org> wrote: > So, why shouldn't we add enum to the Python 2.7 stdlib? Or any other new > feature? Just be aware that if this PEP is accepted for Python 2.7, the bar > will be set much lower for other Really Useful Features That Make Users Lives > Better. Because with PEP453 you can just ``pip install enum34`` it :) ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: <http://mail.python.org/pipermail/python-dev/attachments/20130925/53fd5d4a/attachment.sig> From barry at python.org Thu Sep 26 01:08:54 2013 From: barry at python.org (Barry Warsaw) Date: Wed, 25 Sep 2013 19:08:54 -0400 Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for pronouncement? In-Reply-To: <CADiSq7fVrruk1zcE-ZwHUOyUqOPhTBnn2jcp9rKQ8dS6QeL+Zw@mail.gmail.com> References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com> <20130925165022.3d67e290@anarchist> <109B26FC-32A9-4DDE-9639-E48A0389455D@stufft.io> <20130925175129.1edc4348@anarchist> <CADiSq7fVrruk1zcE-ZwHUOyUqOPhTBnn2jcp9rKQ8dS6QeL+Zw@mail.gmail.com> Message-ID: <20130925190854.6ac76200@anarchist> On Sep 26, 2013, at 09:01 AM, Nick Coghlan wrote: >Would a leading underscore in the module name make you more comfortable >with the idea? It's really intended mostly as a hidden implementation >detail of the installers and pyvenv anyway, so calling it "_ensurepip" >would help make that explicit while still letting people invoke it directly >if absolutely necessary. If it was also undocumented, and we can guarantee that it will have no impact outside of the module itself, then maybe. I'm not sure it would be enough for me to accept it if I were the RM, but I'm not (thankfully. :). I still think it's optimistic to pin any hopes on its widespread availability in future 2.7 point releases on people's actual systems. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: <http://mail.python.org/pipermail/python-dev/attachments/20130925/6c283445/attachment.sig> From barry at python.org Thu Sep 26 01:15:05 2013 From: barry at python.org (Barry Warsaw) Date: Wed, 25 Sep 2013 19:15:05 -0400 Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for pronouncement? In-Reply-To: <7FBFFC39-A0AF-47FD-9715-944D13212D10@stufft.io> References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com> <20130925165022.3d67e290@anarchist> <109B26FC-32A9-4DDE-9639-E48A0389455D@stufft.io> <20130925175129.1edc4348@anarchist> <55AAF039-CCAB-467F-8FA3-74D31CB80A66@stufft.io> <20130925190410.0a5729f9@anarchist> <7FBFFC39-A0AF-47FD-9715-944D13212D10@stufft.io> Message-ID: <20130925191505.59fb9c0f@anarchist> On Sep 25, 2013, at 07:08 PM, Donald Stufft wrote: >On Sep 25, 2013, at 7:04 PM, Barry Warsaw <barry at python.org> wrote: > >> So, why shouldn't we add enum to the Python 2.7 stdlib? >Because with PEP453 you can just ``pip install enum34`` it :) Of course, pip messing with global Python state on a package managed system like most Linux distros, is a whole 'nuther happy fun ball of killer beeswax. :) mixing-metaphors-ly y'rs, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: <http://mail.python.org/pipermail/python-dev/attachments/20130925/cd35739a/attachment.sig> From donald at stufft.io Thu Sep 26 01:15:41 2013 From: donald at stufft.io (Donald Stufft) Date: Wed, 25 Sep 2013 19:15:41 -0400 Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for pronouncement? In-Reply-To: <20130925190410.0a5729f9@anarchist> References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com> <20130925165022.3d67e290@anarchist> <109B26FC-32A9-4DDE-9639-E48A0389455D@stufft.io> <20130925175129.1edc4348@anarchist> <55AAF039-CCAB-467F-8FA3-74D31CB80A66@stufft.io> <20130925190410.0a5729f9@anarchist> Message-ID: <D995BB4B-F2AA-40B7-BBF3-3DDF36CCB5B9@stufft.io> On Sep 25, 2013, at 7:04 PM, Barry Warsaw <barry at python.org> wrote: > Another reason to oppose this is what I've heard quite often from people > regarding Python 2.7. I've been told that many folks are actually really > happy with using 2.7 precisely because it extremely stable. They don't have > to worry about their stuff breaking or incompatibilities cropping up, because > it *doesn't* change. Python 2.7 is like a long-term maintenance release, with > guaranteed multi-year stability, and I think while that was unintended, it's a > *good* thing. This PEP proposes to break that, and I'm loathe to give up that > good reputation for this particular feature. Maybe I'm just naive but can you expand on how adding a module that nothing else in the system imports (besides the installers; but you weren't arguing against adding this to the installers) would break someones use any other module? If they don't import it (which the vast bulk of people won't directly, nor at all during the operation of their applications) how does it's existence on the file system risk a breakage to their system? ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: <http://mail.python.org/pipermail/python-dev/attachments/20130925/bb9e739e/attachment.sig> From donald at stufft.io Thu Sep 26 01:18:56 2013 From: donald at stufft.io (Donald Stufft) Date: Wed, 25 Sep 2013 19:18:56 -0400 Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for pronouncement? In-Reply-To: <20130925191505.59fb9c0f@anarchist> References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com> <20130925165022.3d67e290@anarchist> <109B26FC-32A9-4DDE-9639-E48A0389455D@stufft.io> <20130925175129.1edc4348@anarchist> <55AAF039-CCAB-467F-8FA3-74D31CB80A66@stufft.io> <20130925190410.0a5729f9@anarchist> <7FBFFC39-A0AF-47FD-9715-944D13212D10@stufft.io> <20130925191505.59fb9c0f@anarchist> Message-ID: <DA69848B-8532-4DB8-B489-CCFF5D7E6373@stufft.io> On Sep 25, 2013, at 7:15 PM, Barry Warsaw <barry at python.org> wrote: > Of course, pip messing with global Python state on a package managed system > like most Linux distros, is a whole 'nuther happy fun ball of killer > beeswax. :) > > mixing-metaphors-ly y'rs, > -Barry <virtualenv invocation not shown> (For reals a pip and apt-get playing nicely is on my stack of PEPs to do) ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: <http://mail.python.org/pipermail/python-dev/attachments/20130925/331eef01/attachment.sig> From cs at zip.com.au Thu Sep 26 01:05:20 2013 From: cs at zip.com.au (Cameron Simpson) Date: Thu, 26 Sep 2013 09:05:20 +1000 Subject: [Python-Dev] Revert #12085 fix for __del__ attribute error message In-Reply-To: <5241BED1.7010905@g.nevcal.com> <l1tvd9$6rg$1@ger.gmane.org> References: <5241BED1.7010905@g.nevcal.com> <l1tvd9$6rg$1@ger.gmane.org> Message-ID: <20130925230520.GA94785@cskk.homeip.net> [ I've replied to two messages here: Georg's and Glenn's (supporting MRAB's). - Cameron ] On 25Sep2013 08:22, Georg Brandl <g.brandl at gmx.net> wrote: | Am 24.09.2013 00:11, schrieb Greg Ewing: | > How about something like "Uncaught exception in __del__ | > method ignored"? It explains fairly clearly what has | > happened, and also indicates what do do about it -- | > catch it in the __del__ method. | | "Exception in __del__ caught and not propagated:" | Georg On 24Sep2013 09:33, Glenn Linderman <v+python at g.nevcal.com> wrote: | [MRAB]: | >> Why not just say something like "Cannot propagate exception..."; it's | >> simpler than "Unpropagatable exception...". [...] | | First one I've heard that accurately and unambiguously and briefly | describes the issue. | +1 I'm strongly in favour of Georg's one ("Exception in __del__ caught and not propagated"). Why? It says simply and clearly what has happened. It denotes the relevant context (__del__) in which it happened. The reader can then decide to find out why that decision may have been made. Why not MRAB's? ("Cannot propagate exception...") While better than "Unpropagatable exception" and "unraisable" and "unreraisable", it has the same flaw that I think underlies Antoine's concerns: it suggests the reason there's an error printed instead of further propagation is a _property of the exception_. It doesn't say it outright, but as an outsider that is definitely what I would at first infer. So: a small +0.1 for "Cannot propagate exception..." And: a big +2 for "Exception in __del__ caught and not propagated:". Cheers, -- Cameron Simpson <cs at zip.com.au> Let the bullet of your thought achieve the true and final path of its trajectory before it reaches the ear of the listener, lest it plow out the other side of his head. - Henry David Thoreau From python at mrabarnett.plus.com Thu Sep 26 02:17:14 2013 From: python at mrabarnett.plus.com (MRAB) Date: Thu, 26 Sep 2013 01:17:14 +0100 Subject: [Python-Dev] Revert #12085 fix for __del__ attribute error message In-Reply-To: <20130925230520.GA94785@cskk.homeip.net> References: <5241BED1.7010905@g.nevcal.com> <l1tvd9$6rg$1@ger.gmane.org> <20130925230520.GA94785@cskk.homeip.net> Message-ID: <52437D0A.7080501@mrabarnett.plus.com> On 26/09/2013 00:05, Cameron Simpson wrote: > [ I've replied to two messages here: Georg's and Glenn's (supporting MRAB's). > - Cameron > ] > > On 25Sep2013 08:22, Georg Brandl <g.brandl at gmx.net> wrote: > | Am 24.09.2013 00:11, schrieb Greg Ewing: > | > How about something like "Uncaught exception in __del__ > | > method ignored"? It explains fairly clearly what has > | > happened, and also indicates what do do about it -- > | > catch it in the __del__ method. > | > | "Exception in __del__ caught and not propagated:" > | Georg > > > On 24Sep2013 09:33, Glenn Linderman <v+python at g.nevcal.com> wrote: > | [MRAB]: > | >> Why not just say something like "Cannot propagate exception..."; it's > | >> simpler than "Unpropagatable exception...". [...] > | > | First one I've heard that accurately and unambiguously and briefly > | describes the issue. > | +1 > > I'm strongly in favour of Georg's one ("Exception in __del__ caught and not propagated"). > > Why? > > It says simply and clearly what has happened. > It denotes the relevant context (__del__) in which it happened. > The reader can then decide to find out why that decision may have been made. > > Why not MRAB's? ("Cannot propagate exception...") > > While better than "Unpropagatable exception" and "unraisable" and > "unreraisable", it has the same flaw that I think underlies Antoine's > concerns: it suggests the reason there's an error printed instead > of further propagation is a _property of the exception_. > It doesn't say it outright, but as an outsider that is definitely > what I would at first infer. > > So: a small +0.1 for "Cannot propagate exception..." > > And: a big +2 for "Exception in __del__ caught and not propagated:". > Well, my suggestion was more about eliminating the adjective+noun form; it seemed it just wasn't possible to cram the desired meaning into a single word, so it was better not to try! From v+python at g.nevcal.com Thu Sep 26 03:46:39 2013 From: v+python at g.nevcal.com (Glenn Linderman) Date: Wed, 25 Sep 2013 18:46:39 -0700 Subject: [Python-Dev] Revert #12085 fix for __del__ attribute error message In-Reply-To: <52437D0A.7080501@mrabarnett.plus.com> References: <5241BED1.7010905@g.nevcal.com> <l1tvd9$6rg$1@ger.gmane.org> <20130925230520.GA94785@cskk.homeip.net> <52437D0A.7080501@mrabarnett.plus.com> Message-ID: <524391FF.4010209@g.nevcal.com> On 9/25/2013 5:17 PM, MRAB wrote: > On 26/09/2013 00:05, Cameron Simpson wrote: >> [ I've replied to two messages here: Georg's and Glenn's (supporting >> MRAB's). >> - Cameron >> ] >> >> On 25Sep2013 08:22, Georg Brandl <g.brandl at gmx.net> wrote: >> | Am 24.09.2013 00:11, schrieb Greg Ewing: >> | > How about something like "Uncaught exception in __del__ >> | > method ignored"? It explains fairly clearly what has >> | > happened, and also indicates what do do about it -- >> | > catch it in the __del__ method. >> | >> | "Exception in __del__ caught and not propagated:" >> | Georg >> >> >> On 24Sep2013 09:33, Glenn Linderman <v+python at g.nevcal.com> wrote: >> | [MRAB]: >> | >> Why not just say something like "Cannot propagate exception..."; >> it's >> | >> simpler than "Unpropagatable exception...". [...] >> | >> | First one I've heard that accurately and unambiguously and briefly >> | describes the issue. >> | +1 >> >> I'm strongly in favour of Georg's one ("Exception in __del__ caught >> and not propagated"). >> >> Why? >> >> It says simply and clearly what has happened. >> It denotes the relevant context (__del__) in which it happened. >> The reader can then decide to find out why that decision may have >> been made. >> >> Why not MRAB's? ("Cannot propagate exception...") >> >> While better than "Unpropagatable exception" and "unraisable" and >> "unreraisable", it has the same flaw that I think underlies Antoine's >> concerns: it suggests the reason there's an error printed instead >> of further propagation is a _property of the exception_. >> It doesn't say it outright, but as an outsider that is definitely >> what I would at first infer. >> >> So: a small +0.1 for "Cannot propagate exception..." >> >> And: a big +2 for "Exception in __del__ caught and not propagated:". >> > Well, my suggestion was more about eliminating the adjective+noun > form; it seemed it just wasn't > possible to cram the desired meaning into a single word, so it was > better not to try! Georg hadn't suggested his yet when I approved MRAB's, but your reasoning is sound, Cameron, unless there is some reason that the message can't or shouldn't be as long as what Georg suggested. In between in length and clarity might be: __del__ cannot propagate exception .... All of these are less confusing that the adj+noun form, that MRAB was the first to get away from. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130925/8c6dff61/attachment-0001.html> From stephen at xemacs.org Thu Sep 26 03:52:25 2013 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Thu, 26 Sep 2013 10:52:25 +0900 Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for pronouncement? In-Reply-To: <7FBFFC39-A0AF-47FD-9715-944D13212D10@stufft.io> References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com> <20130925165022.3d67e290@anarchist> <109B26FC-32A9-4DDE-9639-E48A0389455D@stufft.io> <20130925175129.1edc4348@anarchist> <55AAF039-CCAB-467F-8FA3-74D31CB80A66@stufft.io> <20130925190410.0a5729f9@anarchist> <7FBFFC39-A0AF-47FD-9715-944D13212D10@stufft.io> Message-ID: <87bo3g1h9i.fsf@uwakimon.sk.tsukuba.ac.jp> Donald Stufft writes: > On Sep 25, 2013, at 7:04 PM, Barry Warsaw <barry at python.org> wrote: > > > So, why shouldn't we add enum to the Python 2.7 stdlib? Or any > > other new feature? Just be aware that if this PEP is accepted > > for Python 2.7, the bar will be set much lower for other Really > > Useful Features That Make Users Lives Better. > > Because with PEP453 you can just ``pip install enum34`` it :) Uh, I've been doing that for years (well, not with ``enum34``, to be sure). And not just with 2.7, 2.6 and 2.5 too. Who needs PEP 453? Not me (ironically, I'll probably have a PEP 453 Python within 48 hours of its public release). OTOH, as Barry points out, the target population doesn't upgrade very often. What gets them to upgrade (or install in parallel) is the installation of a package that requires an upgraded Python. But that typically means a bump from x.y to x.z, and that will never happen to 2.7. So they'll see a failure of some kind anyway if the ensurepip feature is used! That is going to blowback on distributors, which is going to make PEP 453 real popular with them. Not. From ncoghlan at gmail.com Thu Sep 26 06:30:18 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 26 Sep 2013 14:30:18 +1000 Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for pronouncement? In-Reply-To: <20130925190854.6ac76200@anarchist> References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com> <20130925165022.3d67e290@anarchist> <109B26FC-32A9-4DDE-9639-E48A0389455D@stufft.io> <20130925175129.1edc4348@anarchist> <CADiSq7fVrruk1zcE-ZwHUOyUqOPhTBnn2jcp9rKQ8dS6QeL+Zw@mail.gmail.com> <20130925190854.6ac76200@anarchist> Message-ID: <CADiSq7cMMkTjArg=0=rmDypHMo5aACYNQX=ct18N9jzru2=JQQ@mail.gmail.com> On 26 September 2013 09:08, Barry Warsaw <barry at python.org> wrote: > On Sep 26, 2013, at 09:01 AM, Nick Coghlan wrote: > >>Would a leading underscore in the module name make you more comfortable >>with the idea? It's really intended mostly as a hidden implementation >>detail of the installers and pyvenv anyway, so calling it "_ensurepip" >>would help make that explicit while still letting people invoke it directly >>if absolutely necessary. > > If it was also undocumented, and we can guarantee that it will have no impact > outside of the module itself, then maybe. I'm not sure it would be enough for > me to accept it if I were the RM, but I'm not (thankfully. :). > > I still think it's optimistic to pin any hopes on its widespread availability > in future 2.7 point releases on people's actual systems. The recommended user experience for 2.7 users on Windows and Mac OS X will be "Install the latest Python 2.7 from python.org". This feature is *not* aimed at existing users *at all*. For Python 2.7, the benefits are aimed almost entirely at beginners installing Python for the first time, regardless of platform. There's a *side effect* for existing users, where for Windows and Mac OS X users, the pip bootstrapping instructions become "install the latest maintenance release" rather than downloading and running the bootstrap script from pip-installer.org. The presence or absence of _ensurepip should be completely irrelevant to end users, as it will be there in 2.7 solely to make the new feature of the installer work. Your alternative proposal is to instead come up with a *new* mechanism to support the feature, specifically for 2.7. That's increasing the chance of breakage, not reducing it, *because the _ensurepip tests would no longer be executed by the Python 2.7 buildbots*. That said, there are changes that I think are definitely worth making due to the concerns you raise: - the module name should be "_ensurepip" in all versions - the PEP should explicitly state that the "don't remove _ensurepip and it's wheel files" caveat for redistributors applies only in 3.4+ (where removing it will break pyvenv) The benefits are most obvious in the case of 2.7, so both Donald and I are also fine with skipping making any changes to Python 3.3. The kind of environment that could make it difficult to upgrade from Python 3.3. to 3.4 is unlikely to condone the installation and use of pip anyway. There is one other change which is potentially reasonable, and that's changing the default state of the "Install pip (the default Python package management utility)?" checkbox. I believe it should be checked by default in all versions, but if Benjamin decides that it should be unchecked by default in 2.7, I'd be willing to go along with that. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Thu Sep 26 06:39:40 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 26 Sep 2013 14:39:40 +1000 Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for pronouncement? In-Reply-To: <87bo3g1h9i.fsf@uwakimon.sk.tsukuba.ac.jp> References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com> <20130925165022.3d67e290@anarchist> <109B26FC-32A9-4DDE-9639-E48A0389455D@stufft.io> <20130925175129.1edc4348@anarchist> <55AAF039-CCAB-467F-8FA3-74D31CB80A66@stufft.io> <20130925190410.0a5729f9@anarchist> <7FBFFC39-A0AF-47FD-9715-944D13212D10@stufft.io> <87bo3g1h9i.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <CADiSq7dJYDJwok1iccJ2mW9Axi_1JOYZ8rzDk50DxGpWDk4kwA@mail.gmail.com> On 26 September 2013 11:52, Stephen J. Turnbull <stephen at xemacs.org> wrote: > Donald Stufft writes: > > > On Sep 25, 2013, at 7:04 PM, Barry Warsaw <barry at python.org> wrote: > > > > > So, why shouldn't we add enum to the Python 2.7 stdlib? Or any > > > other new feature? Just be aware that if this PEP is accepted > > > for Python 2.7, the bar will be set much lower for other Really > > > Useful Features That Make Users Lives Better. > > > > Because with PEP453 you can just ``pip install enum34`` it :) > > Uh, I've been doing that for years (well, not with ``enum34``, to be > sure). And not just with 2.7, 2.6 and 2.5 too. Who needs PEP 453? > Not me (ironically, I'll probably have a PEP 453 Python within 48 > hours of its public release). New users on Windows and Mac OS X. I've heard many more complaints from folks running tutorials about the pip bootstrapping process than I ever have from the community at large about the GIL :P Linux and other *nix platforms are affected primarily due to the fact that we also want pyvenv to start making pip available by default in 3.4+, so we need the PEP to account for making that possible while still playing nice with the Linux distro vendors. > OTOH, as Barry points out, the target population doesn't upgrade very > often. If somebody already has Python installed, they're not really part of the target population, except in the sense that the pip bootstrap instruction become "upgrade Python to the latest maintenance release" rather than "got to this website, download and run this bootstrap script". (Keeping those instructions consistent for 3.3 users as well would be the advantage of updating 3.3 in addition to 2.7 and 3.4) > What gets them to upgrade (or install in parallel) is the > installation of a package that requires an upgraded Python. But that > typically means a bump from x.y to x.z, and that will never happen to > 2.7. So they'll see a failure of some kind anyway if the ensurepip > feature is used! That is going to blowback on distributors, which is > going to make PEP 453 real popular with them. Not. I'm not sure what usage model you're assuming for _ensurepip, but it appears to be wrong. End users should be able to just run pip, and either have it work, or else get a message from the OS vendor telling them to install the appropriate system package. If it doesn't work, then the advice will be to upgrade to the latest Python maintenance release. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Thu Sep 26 06:59:17 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 26 Sep 2013 14:59:17 +1000 Subject: [Python-Dev] Revert #12085 fix for __del__ attribute error message In-Reply-To: <20130925230520.GA94785@cskk.homeip.net> References: <5241BED1.7010905@g.nevcal.com> <l1tvd9$6rg$1@ger.gmane.org> <20130925230520.GA94785@cskk.homeip.net> Message-ID: <CADiSq7fywLacQywujMma+HTov948FXgmtJ51gRon_iU4Fc2nxg@mail.gmail.com> On 26 September 2013 09:05, Cameron Simpson <cs at zip.com.au> wrote: > On 24Sep2013 09:33, Glenn Linderman <v+python at g.nevcal.com> wrote: > | [MRAB]: > | >> Why not just say something like "Cannot propagate exception..."; it's > | >> simpler than "Unpropagatable exception...". [...] > | > | First one I've heard that accurately and unambiguously and briefly > | describes the issue. > | +1 > > I'm strongly in favour of Georg's one ("Exception in __del__ caught and not propagated"). > > Why? > > It says simply and clearly what has happened. > It denotes the relevant context (__del__) in which it happened. > The reader can then decide to find out why that decision may have been made. Such a change is highly unlikely to happen, as it would require changing every location where we call PyErr_WriteUnraisable. > Why not MRAB's? ("Cannot propagate exception...") > > While better than "Unpropagatable exception" and "unraisable" and > "unreraisable", it has the same flaw that I think underlies Antoine's > concerns: it suggests the reason there's an error printed instead > of further propagation is a _property of the exception_. > It doesn't say it outright, but as an outsider that is definitely > what I would at first infer. It's substantially better than the status quo, though, and it shouldn't require any changes outside PyErr_WriteUnraisable. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Thu Sep 26 06:54:49 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 26 Sep 2013 14:54:49 +1000 Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for pronouncement? In-Reply-To: <CADiSq7cMMkTjArg=0=rmDypHMo5aACYNQX=ct18N9jzru2=JQQ@mail.gmail.com> References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com> <20130925165022.3d67e290@anarchist> <109B26FC-32A9-4DDE-9639-E48A0389455D@stufft.io> <20130925175129.1edc4348@anarchist> <CADiSq7fVrruk1zcE-ZwHUOyUqOPhTBnn2jcp9rKQ8dS6QeL+Zw@mail.gmail.com> <20130925190854.6ac76200@anarchist> <CADiSq7cMMkTjArg=0=rmDypHMo5aACYNQX=ct18N9jzru2=JQQ@mail.gmail.com> Message-ID: <CADiSq7cmSqnWCL0-s-g5dc1AgRRkfhd0VjYVSDj47bWQh537NA@mail.gmail.com> On 26 September 2013 14:30, Nick Coghlan <ncoghlan at gmail.com> wrote: > That said, there are changes that I think are definitely worth making > due to the concerns you raise: > > - the module name should be "_ensurepip" in all versions > - the PEP should explicitly state that the "don't remove _ensurepip > and it's wheel files" caveat for redistributors applies only in 3.4+ > (where removing it will break pyvenv) Donald pointed out it makes more sense to continue with the idea of a properly documented public "ensurepip" module in 3.4+, and have the "_ensurepip" version as an implementation detail of the 2.7 and 3.3 installers that is included in the stdlib primarily so it can be covered by the existing buildbot fleet. Redistributors will be free to remove "_ensurepip" from 2.7 and 3.3, but the PEP will continue to advise strongly against removing "ensurepip" from 3.4+ (since it will be a documented feature and a dependency of pyvenv) > The benefits are most obvious in the case of 2.7, so both Donald and I > are also fine with skipping making any changes to Python 3.3. The kind > of environment that could make it difficult to upgrade from Python > 3.3. to 3.4 is unlikely to condone the installation and use of pip > anyway. In replying to Stephen I realised the benefit of making a similar change in the 3.3 installer is that it means the preferred pip bootstrapping instructions for both 2.7 and 3.3 can be to update to the latest maintenance release of CPython, rather than needing to tell 3.3 users to consider upgrading to 3.4 when that release contains potentially breaking changes to the way file descriptors are handled. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From arigo at tunes.org Thu Sep 26 07:42:54 2013 From: arigo at tunes.org (Armin Rigo) Date: Thu, 26 Sep 2013 07:42:54 +0200 Subject: [Python-Dev] Revert #12085 fix for __del__ attribute error message In-Reply-To: <CADiSq7fywLacQywujMma+HTov948FXgmtJ51gRon_iU4Fc2nxg@mail.gmail.com> References: <5241BED1.7010905@g.nevcal.com> <l1tvd9$6rg$1@ger.gmane.org> <20130925230520.GA94785@cskk.homeip.net> <CADiSq7fywLacQywujMma+HTov948FXgmtJ51gRon_iU4Fc2nxg@mail.gmail.com> Message-ID: <CAMSv6X1tQGPhLs0rnvNwV12z9czPkLWZx8dk9kmtYQ_RNwnY_Q@mail.gmail.com> Hi Nick, On Thu, Sep 26, 2013 at 6:59 AM, Nick Coghlan <ncoghlan at gmail.com> wrote: >> I'm strongly in favour of Georg's one ("Exception in __del__ caught and not propagated"). > > Such a change is highly unlikely to happen, as it would require > changing every location where we call PyErr_WriteUnraisable. Er, why? It seems to me it's a matter of changing these three lines in PyErr_WriteUnraisable(): - PyFile_WriteString("Exception ignored in: ", f); + PyFile_WriteString("Exception in ", f); PyFile_WriteObject(obj, f, 0); - PyFile_WriteString("\n", f); + PyFile_WriteString(" caught and not propagated:\n", f); I don't see what makes this technically different from the other solution, "Cannot propagate exception..." A bient?t, Armin. From ncoghlan at gmail.com Thu Sep 26 08:22:34 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 26 Sep 2013 16:22:34 +1000 Subject: [Python-Dev] Revert #12085 fix for __del__ attribute error message In-Reply-To: <CAMSv6X1tQGPhLs0rnvNwV12z9czPkLWZx8dk9kmtYQ_RNwnY_Q@mail.gmail.com> References: <5241BED1.7010905@g.nevcal.com> <l1tvd9$6rg$1@ger.gmane.org> <20130925230520.GA94785@cskk.homeip.net> <CADiSq7fywLacQywujMma+HTov948FXgmtJ51gRon_iU4Fc2nxg@mail.gmail.com> <CAMSv6X1tQGPhLs0rnvNwV12z9czPkLWZx8dk9kmtYQ_RNwnY_Q@mail.gmail.com> Message-ID: <CADiSq7eiZku6BXEd3DtOGg4BRPNP3-QvtNNQbq0=SRf7eCJKYA@mail.gmail.com> On 26 September 2013 15:42, Armin Rigo <arigo at tunes.org> wrote: > Hi Nick, > > On Thu, Sep 26, 2013 at 6:59 AM, Nick Coghlan <ncoghlan at gmail.com> wrote: >>> I'm strongly in favour of Georg's one ("Exception in __del__ caught and not propagated"). >> >> Such a change is highly unlikely to happen, as it would require >> changing every location where we call PyErr_WriteUnraisable. > > Er, why? It seems to me it's a matter of changing these three lines > in PyErr_WriteUnraisable(): > > - PyFile_WriteString("Exception ignored in: ", f); > + PyFile_WriteString("Exception in ", f); > PyFile_WriteObject(obj, f, 0); > - PyFile_WriteString("\n", f); > + PyFile_WriteString(" caught and not propagated:\n", f); > > I don't see what makes this technically different from the other > solution, "Cannot propagate exception..." Sure, that's doable, but it dumps the full repr of "obj" in the middle of the sentence. The thing that's not practical is the neat and tidy wording Georg proposed, because the thing passed as "obj" is actually an arbitrary Python object that may have a messy repr (like a bound method, which is what gets passed in the __del__ case), so there's definite merit in keeping that repr at the *end* of the header line. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From solipsis at pitrou.net Thu Sep 26 08:35:36 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 26 Sep 2013 08:35:36 +0200 Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for pronouncement? References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com> <20130925165022.3d67e290@anarchist> <109B26FC-32A9-4DDE-9639-E48A0389455D@stufft.io> <20130925175129.1edc4348@anarchist> <55AAF039-CCAB-467F-8FA3-74D31CB80A66@stufft.io> <20130925190410.0a5729f9@anarchist> <7FBFFC39-A0AF-47FD-9715-944D13212D10@stufft.io> <20130925191505.59fb9c0f@anarchist> Message-ID: <20130926083536.04344210@fsol> On Wed, 25 Sep 2013 19:15:05 -0400 Barry Warsaw <barry at python.org> wrote: > On Sep 25, 2013, at 07:08 PM, Donald Stufft wrote: > > >On Sep 25, 2013, at 7:04 PM, Barry Warsaw <barry at python.org> wrote: > > > >> So, why shouldn't we add enum to the Python 2.7 stdlib? > > >Because with PEP453 you can just ``pip install enum34`` it :) > > Of course, pip messing with global Python state on a package managed system > like most Linux distros, is a whole 'nuther happy fun ball of killer > beeswax. :) pip install --user enum34 :-) From g.brandl at gmx.net Thu Sep 26 08:53:14 2013 From: g.brandl at gmx.net (Georg Brandl) Date: Thu, 26 Sep 2013 08:53:14 +0200 Subject: [Python-Dev] Revert #12085 fix for __del__ attribute error message In-Reply-To: <CADiSq7eiZku6BXEd3DtOGg4BRPNP3-QvtNNQbq0=SRf7eCJKYA@mail.gmail.com> References: <5241BED1.7010905@g.nevcal.com> <l1tvd9$6rg$1@ger.gmane.org> <20130925230520.GA94785@cskk.homeip.net> <CADiSq7fywLacQywujMma+HTov948FXgmtJ51gRon_iU4Fc2nxg@mail.gmail.com> <CAMSv6X1tQGPhLs0rnvNwV12z9czPkLWZx8dk9kmtYQ_RNwnY_Q@mail.gmail.com> <CADiSq7eiZku6BXEd3DtOGg4BRPNP3-QvtNNQbq0=SRf7eCJKYA@mail.gmail.com> Message-ID: <l20liu$gla$1@ger.gmane.org> Am 26.09.2013 08:22, schrieb Nick Coghlan: > On 26 September 2013 15:42, Armin Rigo <arigo at tunes.org> wrote: >> Hi Nick, >> >> On Thu, Sep 26, 2013 at 6:59 AM, Nick Coghlan <ncoghlan at gmail.com> wrote: >>>> I'm strongly in favour of Georg's one ("Exception in __del__ caught and not propagated"). >>> >>> Such a change is highly unlikely to happen, as it would require >>> changing every location where we call PyErr_WriteUnraisable. >> >> Er, why? It seems to me it's a matter of changing these three lines >> in PyErr_WriteUnraisable(): >> >> - PyFile_WriteString("Exception ignored in: ", f); >> + PyFile_WriteString("Exception in ", f); >> PyFile_WriteObject(obj, f, 0); >> - PyFile_WriteString("\n", f); >> + PyFile_WriteString(" caught and not propagated:\n", f); >> >> I don't see what makes this technically different from the other >> solution, "Cannot propagate exception..." > > Sure, that's doable, but it dumps the full repr of "obj" in the middle > of the sentence. The thing that's not practical is the neat and tidy > wording Georg proposed, because the thing passed as "obj" is actually > an arbitrary Python object that may have a messy repr (like a bound > method, which is what gets passed in the __del__ case), so there's > definite merit in keeping that repr at the *end* of the header line. Then this should be fine, I guess? Exception caught and not propagated in: <....> Georg From ncoghlan at gmail.com Thu Sep 26 09:17:02 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 26 Sep 2013 17:17:02 +1000 Subject: [Python-Dev] Revert #12085 fix for __del__ attribute error message In-Reply-To: <l20liu$gla$1@ger.gmane.org> References: <5241BED1.7010905@g.nevcal.com> <l1tvd9$6rg$1@ger.gmane.org> <20130925230520.GA94785@cskk.homeip.net> <CADiSq7fywLacQywujMma+HTov948FXgmtJ51gRon_iU4Fc2nxg@mail.gmail.com> <CAMSv6X1tQGPhLs0rnvNwV12z9czPkLWZx8dk9kmtYQ_RNwnY_Q@mail.gmail.com> <CADiSq7eiZku6BXEd3DtOGg4BRPNP3-QvtNNQbq0=SRf7eCJKYA@mail.gmail.com> <l20liu$gla$1@ger.gmane.org> Message-ID: <CADiSq7fBvyQjNJp4u6AVaQRWkJfYFTBMX=Nb8fjSv=8JBG0H5w@mail.gmail.com> On 26 September 2013 16:53, Georg Brandl <g.brandl at gmx.net> wrote: >> Sure, that's doable, but it dumps the full repr of "obj" in the middle >> of the sentence. The thing that's not practical is the neat and tidy >> wording Georg proposed, because the thing passed as "obj" is actually >> an arbitrary Python object that may have a messy repr (like a bound >> method, which is what gets passed in the __del__ case), so there's >> definite merit in keeping that repr at the *end* of the header line. > > Then this should be fine, I guess? > > Exception caught and not propagated in: <....> Sure. I still prefer something like "Could not propagate exception from:" or "Caller could not propagate exception from <repr>" that better indicates we're suppressing it because it's infeasible to raise it rather than just because we feel like it, but any of them would offer a decent improvement over the status quo. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From eliben at gmail.com Thu Sep 26 15:52:20 2013 From: eliben at gmail.com (Eli Bendersky) Date: Thu, 26 Sep 2013 06:52:20 -0700 Subject: [Python-Dev] asdl.py and Python-ast.[hc] Message-ID: <CAF-Rda-dChgtDxtxcnXtUSrDDgNDCR2o5F665vk3P2gvN9UiNQ@mail.gmail.com> Hi All, Earlier this morning I had a slight tackle with a couple of the 3.4 bots (sorry everyone!). I fixed some problems in asdl.py - http://hg.python.org/cpython/rev/21d46e3ae60c - and used the 'with' statement. Some bots don't have Python 2.6+ and couldn't bootstrap Python-ast.h/c Two questions: * Should I always check-in Python-ast.h and Python-ast.c when I touch asdl* ? The generated files are unchanged, it's only the timestamp that changed. * Can we, in theory, use new Pythons for asdl* code, because Python-ast.* are, in fact, checked in so they don't have to be rebuilt by the bots or users? Thanks in advance, Eli -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130926/407f3bb2/attachment.html> From brett at python.org Thu Sep 26 16:00:15 2013 From: brett at python.org (Brett Cannon) Date: Thu, 26 Sep 2013 10:00:15 -0400 Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for pronouncement? In-Reply-To: <D995BB4B-F2AA-40B7-BBF3-3DDF36CCB5B9@stufft.io> References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com> <20130925165022.3d67e290@anarchist> <109B26FC-32A9-4DDE-9639-E48A0389455D@stufft.io> <20130925175129.1edc4348@anarchist> <55AAF039-CCAB-467F-8FA3-74D31CB80A66@stufft.io> <20130925190410.0a5729f9@anarchist> <D995BB4B-F2AA-40B7-BBF3-3DDF36CCB5B9@stufft.io> Message-ID: <CAP1=2W7bukJGw8BWgk389MkKbo-DHjRsbv+eZAS2FFH2+nguFg@mail.gmail.com> On Wed, Sep 25, 2013 at 7:15 PM, Donald Stufft <donald at stufft.io> wrote: > > On Sep 25, 2013, at 7:04 PM, Barry Warsaw <barry at python.org> wrote: > > > Another reason to oppose this is what I've heard quite often from people > > regarding Python 2.7. I've been told that many folks are actually really > > happy with using 2.7 precisely because it extremely stable. They don't > have > > to worry about their stuff breaking or incompatibilities cropping up, > because > > it *doesn't* change. Python 2.7 is like a long-term maintenance > release, with > > guaranteed multi-year stability, and I think while that was unintended, > it's a > > *good* thing. This PEP proposes to break that, and I'm loathe to give > up that > > good reputation for this particular feature. > > Maybe I'm just naive but can you expand on how adding a module that nothing > else in the system imports (besides the installers; but you weren't > arguing against > adding this to the installers) would break someones use any other module? > If > they don't import it (which the vast bulk of people won't directly, nor at > all during > the operation of their applications) how does it's existence on the file > system > risk a breakage to their system? I think Barry's worry is precisely the fact that users do silly things, so having a new module suddenly show up in the stdlib can be a problem if people start using the module. We could conceivably try and not expose the module on sys.path somehow so that it can't be directly imported by anyone who doesn't have any business using it unless they jump through some hoops (e.g. stick it in Tools for Python 2.7 so the installers can get access but it won't end up in the stdlib install). Otherwise if Barry is worrying about a command-line tool being installed for some users of Python 2.7 vs. not then I don't know what the worry is because that happens on OS upgrades as well, let alone people who just happen to have done the install themselves vs. not. People are already told to install pip anyway, so if instructions point to ensurepip on PyPI as the canonical way to install pip then whether it's installed or not is fine as it will be mostly a no-op for 2.7.6 users and an actual install for others. But by including it then it does simplify teaching scenarios and long tail helps get pip out faster and more easily. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130926/589b77d4/attachment.html> From solipsis at pitrou.net Thu Sep 26 16:10:46 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 26 Sep 2013 16:10:46 +0200 Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for pronouncement? References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com> <20130925165022.3d67e290@anarchist> <109B26FC-32A9-4DDE-9639-E48A0389455D@stufft.io> <20130925175129.1edc4348@anarchist> <CADiSq7fVrruk1zcE-ZwHUOyUqOPhTBnn2jcp9rKQ8dS6QeL+Zw@mail.gmail.com> <20130925190854.6ac76200@anarchist> <CADiSq7cMMkTjArg=0=rmDypHMo5aACYNQX=ct18N9jzru2=JQQ@mail.gmail.com> <CADiSq7cmSqnWCL0-s-g5dc1AgRRkfhd0VjYVSDj47bWQh537NA@mail.gmail.com> Message-ID: <20130926161046.17a6d4f6@pitrou.net> Le Thu, 26 Sep 2013 14:54:49 +1000, Nick Coghlan <ncoghlan at gmail.com> a ?crit : > On 26 September 2013 14:30, Nick Coghlan <ncoghlan at gmail.com> wrote: > > That said, there are changes that I think are definitely worth > > making due to the concerns you raise: > > > > - the module name should be "_ensurepip" in all versions > > - the PEP should explicitly state that the "don't remove _ensurepip > > and it's wheel files" caveat for redistributors applies only in 3.4+ > > (where removing it will break pyvenv) > > Donald pointed out it makes more sense to continue with the idea of a > properly documented public "ensurepip" module in 3.4+, and have the > "_ensurepip" version as an implementation detail of the 2.7 and 3.3 > installers that is included in the stdlib primarily so it can be > covered by the existing buildbot fleet. Hmm, but what is the point of "_ensurepip" exactly? Are people supposed to type "python -m _ensurepip"? With all due respect, Barry's argument looks rather paranoid to me. I would suggest a clear choice: - either having "ensurepip" in 2.7 is useful and we endorse it as a public module (not something hidden somewhere) - which I personally think is reasonable - or it's not useful and we don't introduce it at all A middleground doesn't make sense here, except in a broken "design by committee" sense. Regards Antoine. From brett at python.org Thu Sep 26 16:18:30 2013 From: brett at python.org (Brett Cannon) Date: Thu, 26 Sep 2013 10:18:30 -0400 Subject: [Python-Dev] asdl.py and Python-ast.[hc] In-Reply-To: <CAF-Rda-dChgtDxtxcnXtUSrDDgNDCR2o5F665vk3P2gvN9UiNQ@mail.gmail.com> References: <CAF-Rda-dChgtDxtxcnXtUSrDDgNDCR2o5F665vk3P2gvN9UiNQ@mail.gmail.com> Message-ID: <CAP1=2W5XB+iWi0Dv4Tzb1CMmn=25GBoaLZLsXMY1UfQG_ifF0w@mail.gmail.com> On Thu, Sep 26, 2013 at 9:52 AM, Eli Bendersky <eliben at gmail.com> wrote: > Hi All, > > Earlier this morning I had a slight tackle with a couple of the 3.4 bots > (sorry everyone!). I fixed some problems in asdl.py - > http://hg.python.org/cpython/rev/21d46e3ae60c - and used the 'with' > statement. Some bots don't have Python 2.6+ and couldn't bootstrap > Python-ast.h/c > > Two questions: > > * Should I always check-in Python-ast.h and Python-ast.c when I touch > asdl* ? The generated files are unchanged, it's only the timestamp that > changed. > If Python-ast.* are checked in then yes. > * Can we, in theory, use new Pythons for asdl* code, because Python-ast.* > are, in fact, checked in so they don't have to be rebuilt by the bots or > users? > I don't see why not. the touch extension for hg is there specifically for these files to prevent having to regenerate them. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130926/6ed35b8c/attachment.html> From donald at stufft.io Thu Sep 26 16:22:55 2013 From: donald at stufft.io (Donald Stufft) Date: Thu, 26 Sep 2013 10:22:55 -0400 Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for pronouncement? In-Reply-To: <20130926161046.17a6d4f6@pitrou.net> References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com> <20130925165022.3d67e290@anarchist> <109B26FC-32A9-4DDE-9639-E48A0389455D@stufft.io> <20130925175129.1edc4348@anarchist> <CADiSq7fVrruk1zcE-ZwHUOyUqOPhTBnn2jcp9rKQ8dS6QeL+Zw@mail.gmail.com> <20130925190854.6ac76200@anarchist> <CADiSq7cMMkTjArg=0=rmDypHMo5aACYNQX=ct18N9jzru2=JQQ@mail.gmail.com> <CADiSq7cmSqnWCL0-s-g5dc1AgRRkfhd0VjYVSDj47bWQh537NA@mail.gmail.com> <20130926161046.17a6d4f6@pitrou.net> Message-ID: <97A4ED42-8961-4B01-BE20-DF684FC0930E@stufft.io> Ideally people won't be typing either of them because it'll be installed automatically. They might in some cases (accidentally uninstalled pip?) I agree that it seems there is paranoia going on here and that the risk is low and making it just be a special cased new feature is ok. However the point of the underscore prefix on 2.7 and 3.3 is to more effectively communicate that on these pythons you shouldn't rely on that module existing. I think that's worse situation then just making it ensurepip everywhere but better than 2.7 not getting it at all or Barry's suggestion. > On Sep 26, 2013, at 10:10 AM, Antoine Pitrou <solipsis at pitrou.net> wrote: > > Le Thu, 26 Sep 2013 14:54:49 +1000, > Nick Coghlan <ncoghlan at gmail.com> a ?crit : >> On 26 September 2013 14:30, Nick Coghlan <ncoghlan at gmail.com> wrote: >>> That said, there are changes that I think are definitely worth >>> making due to the concerns you raise: >>> >>> - the module name should be "_ensurepip" in all versions >>> - the PEP should explicitly state that the "don't remove _ensurepip >>> and it's wheel files" caveat for redistributors applies only in 3.4+ >>> (where removing it will break pyvenv) >> >> Donald pointed out it makes more sense to continue with the idea of a >> properly documented public "ensurepip" module in 3.4+, and have the >> "_ensurepip" version as an implementation detail of the 2.7 and 3.3 >> installers that is included in the stdlib primarily so it can be >> covered by the existing buildbot fleet. > > Hmm, but what is the point of "_ensurepip" exactly? Are people supposed > to type "python -m _ensurepip"? > > With all due respect, Barry's argument looks rather paranoid to me. > I would suggest a clear choice: > - either having "ensurepip" in 2.7 is useful and we endorse it as a > public module (not something hidden somewhere) - which I personally > think is reasonable > - or it's not useful and we don't introduce it at all > > A middleground doesn't make sense here, except in a broken "design by > committee" sense. > > Regards > > Antoine. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io From eliben at gmail.com Thu Sep 26 16:27:05 2013 From: eliben at gmail.com (Eli Bendersky) Date: Thu, 26 Sep 2013 07:27:05 -0700 Subject: [Python-Dev] asdl.py and Python-ast.[hc] In-Reply-To: <CAP1=2W5XB+iWi0Dv4Tzb1CMmn=25GBoaLZLsXMY1UfQG_ifF0w@mail.gmail.com> References: <CAF-Rda-dChgtDxtxcnXtUSrDDgNDCR2o5F665vk3P2gvN9UiNQ@mail.gmail.com> <CAP1=2W5XB+iWi0Dv4Tzb1CMmn=25GBoaLZLsXMY1UfQG_ifF0w@mail.gmail.com> Message-ID: <CAF-Rda_Wq1Be1K01Afow3sLM-cw6Dx56Y9uxwogtb7O-vLN9fQ@mail.gmail.com> On Thu, Sep 26, 2013 at 7:18 AM, Brett Cannon <brett at python.org> wrote: > > > > On Thu, Sep 26, 2013 at 9:52 AM, Eli Bendersky <eliben at gmail.com> wrote: > >> Hi All, >> >> Earlier this morning I had a slight tackle with a couple of the 3.4 bots >> (sorry everyone!). I fixed some problems in asdl.py - >> http://hg.python.org/cpython/rev/21d46e3ae60c - and used the 'with' >> statement. Some bots don't have Python 2.6+ and couldn't bootstrap >> Python-ast.h/c >> >> Two questions: >> >> * Should I always check-in Python-ast.h and Python-ast.c when I touch >> asdl* ? The generated files are unchanged, it's only the timestamp that >> changed. >> > > If Python-ast.* are checked in then yes. > Hmm, how do I do that? hg doesn't accept them for commit since their contents are not changed. > > >> * Can we, in theory, use new Pythons for asdl* code, because Python-ast.* >> are, in fact, checked in so they don't have to be rebuilt by the bots or >> users? >> > > I don't see why not. the touch extension for hg is there specifically for > these files to prevent having to regenerate them. > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130926/2d229fc4/attachment.html> From benjamin at python.org Thu Sep 26 16:28:46 2013 From: benjamin at python.org (Benjamin Peterson) Date: Thu, 26 Sep 2013 10:28:46 -0400 Subject: [Python-Dev] asdl.py and Python-ast.[hc] In-Reply-To: <CAF-Rda-dChgtDxtxcnXtUSrDDgNDCR2o5F665vk3P2gvN9UiNQ@mail.gmail.com> References: <CAF-Rda-dChgtDxtxcnXtUSrDDgNDCR2o5F665vk3P2gvN9UiNQ@mail.gmail.com> Message-ID: <CAPZV6o_y-8f0TJgEC=YKg+ZkV6vEzhet0U3rqV9DXo-PBaw=3g@mail.gmail.com> 2013/9/26 Eli Bendersky <eliben at gmail.com>: > Hi All, > > Earlier this morning I had a slight tackle with a couple of the 3.4 bots > (sorry everyone!). I fixed some problems in asdl.py - > http://hg.python.org/cpython/rev/21d46e3ae60c - and used the 'with' > statement. Some bots don't have Python 2.6+ and couldn't bootstrap > Python-ast.h/c > > Two questions: > > * Should I always check-in Python-ast.h and Python-ast.c when I touch asdl* > ? The generated files are unchanged, it's only the timestamp that changed. > * Can we, in theory, use new Pythons for asdl* code, because Python-ast.* > are, in fact, checked in so they don't have to be rebuilt by the bots or > users? We should have the buildbots run "make touch", so they don't need to run asdl_c.py. -- Regards, Benjamin From solipsis at pitrou.net Thu Sep 26 16:28:42 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 26 Sep 2013 16:28:42 +0200 Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for pronouncement? References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com> <20130925165022.3d67e290@anarchist> <109B26FC-32A9-4DDE-9639-E48A0389455D@stufft.io> <20130925175129.1edc4348@anarchist> <CADiSq7fVrruk1zcE-ZwHUOyUqOPhTBnn2jcp9rKQ8dS6QeL+Zw@mail.gmail.com> <20130925190854.6ac76200@anarchist> <CADiSq7cMMkTjArg=0=rmDypHMo5aACYNQX=ct18N9jzru2=JQQ@mail.gmail.com> <CADiSq7cmSqnWCL0-s-g5dc1AgRRkfhd0VjYVSDj47bWQh537NA@mail.gmail.com> <20130926161046.17a6d4f6@pitrou.net> <97A4ED42-8961-4B01-BE20-DF684FC0930E@stufft.io> Message-ID: <20130926162842.58b74a5c@pitrou.net> Le Thu, 26 Sep 2013 10:22:55 -0400, Donald Stufft <donald at stufft.io> a ?crit : > Ideally people won't be typing either of them because it'll be > installed automatically. They might in some cases (accidentally > uninstalled pip?) Installing from source perhaps. > I agree that it seems there is paranoia going on here and that the > risk is low and making it just be a special cased new feature is ok. > However the point of the underscore prefix on 2.7 and 3.3 is to more > effectively communicate that on these pythons you shouldn't rely on > that module existing. That's a fair argument, although it also makes the thing a bit ugly. Regards Antoine. From g.brandl at gmx.net Thu Sep 26 18:07:39 2013 From: g.brandl at gmx.net (Georg Brandl) Date: Thu, 26 Sep 2013 18:07:39 +0200 Subject: [Python-Dev] cpython: Don't use fancy new Python features like 'with' - some bots don't have them In-Reply-To: <3cly4S54LQz7Lk4@mail.python.org> References: <3cly4S54LQz7Lk4@mail.python.org> Message-ID: <52445BCB.10501@gmx.net> Am 26.09.2013 15:42, schrieb eli.bendersky: > http://hg.python.org/cpython/rev/931d95e9067f > changeset: 85801:931d95e9067f > user: Eli Bendersky <eliben at gmail.com> > date: Thu Sep 26 06:41:36 2013 -0700 > summary: > Don't use fancy new Python features like 'with' - some bots don't have them > and can't bootstrap the parser. > > files: > Parser/asdl.py | 5 ++++- > 1 files changed, 4 insertions(+), 1 deletions(-) > > > diff --git a/Parser/asdl.py b/Parser/asdl.py > --- a/Parser/asdl.py > +++ b/Parser/asdl.py > @@ -398,8 +398,11 @@ > scanner = ASDLScanner() > parser = ASDLParser() > > - with open(file) as f: > + try: > + f = open(file) > buf = f.read() > + finally: > + f.close() > tokens = scanner.tokenize(buf) > try: > return parser.parse(tokens) The open call needs to go outside the try-finally. cheers, Georg From eliben at gmail.com Thu Sep 26 18:36:07 2013 From: eliben at gmail.com (Eli Bendersky) Date: Thu, 26 Sep 2013 09:36:07 -0700 Subject: [Python-Dev] cpython: Don't use fancy new Python features like 'with' - some bots don't have them In-Reply-To: <52445BCB.10501@gmx.net> References: <3cly4S54LQz7Lk4@mail.python.org> <52445BCB.10501@gmx.net> Message-ID: <CAF-Rda8WQmQqsRe_MaeBzzVb9mEAkvhue7dAtbOujPsuML2Z5w@mail.gmail.com> On Thu, Sep 26, 2013 at 9:07 AM, Georg Brandl <g.brandl at gmx.net> wrote: > Am 26.09.2013 15:42, schrieb eli.bendersky: > > http://hg.python.org/cpython/rev/931d95e9067f > > changeset: 85801:931d95e9067f > > user: Eli Bendersky <eliben at gmail.com> > > date: Thu Sep 26 06:41:36 2013 -0700 > > summary: > > Don't use fancy new Python features like 'with' - some bots don't have > them > > and can't bootstrap the parser. > > > > files: > > Parser/asdl.py | 5 ++++- > > 1 files changed, 4 insertions(+), 1 deletions(-) > > > > > > diff --git a/Parser/asdl.py b/Parser/asdl.py > > --- a/Parser/asdl.py > > +++ b/Parser/asdl.py > > @@ -398,8 +398,11 @@ > > scanner = ASDLScanner() > > parser = ASDLParser() > > > > - with open(file) as f: > > + try: > > + f = open(file) > > buf = f.read() > > + finally: > > + f.close() > > tokens = scanner.tokenize(buf) > > try: > > return parser.parse(tokens) > > The open call needs to go outside the try-finally.\ > Done, thanks. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130926/721d6715/attachment-0001.html> From eliben at gmail.com Thu Sep 26 18:37:03 2013 From: eliben at gmail.com (Eli Bendersky) Date: Thu, 26 Sep 2013 09:37:03 -0700 Subject: [Python-Dev] asdl.py and Python-ast.[hc] In-Reply-To: <CAPZV6o_y-8f0TJgEC=YKg+ZkV6vEzhet0U3rqV9DXo-PBaw=3g@mail.gmail.com> References: <CAF-Rda-dChgtDxtxcnXtUSrDDgNDCR2o5F665vk3P2gvN9UiNQ@mail.gmail.com> <CAPZV6o_y-8f0TJgEC=YKg+ZkV6vEzhet0U3rqV9DXo-PBaw=3g@mail.gmail.com> Message-ID: <CAF-Rda_vthC4GdvJARvs75TYZTUNPLWbF2a4Uvm=7_V67bZvXg@mail.gmail.com> On Thu, Sep 26, 2013 at 7:28 AM, Benjamin Peterson <benjamin at python.org>wrote: > 2013/9/26 Eli Bendersky <eliben at gmail.com>: > > Hi All, > > > > Earlier this morning I had a slight tackle with a couple of the 3.4 bots > > (sorry everyone!). I fixed some problems in asdl.py - > > http://hg.python.org/cpython/rev/21d46e3ae60c - and used the 'with' > > statement. Some bots don't have Python 2.6+ and couldn't bootstrap > > Python-ast.h/c > > > > Two questions: > > > > * Should I always check-in Python-ast.h and Python-ast.c when I touch > asdl* > > ? The generated files are unchanged, it's only the timestamp that > changed. > > * Can we, in theory, use new Pythons for asdl* code, because Python-ast.* > > are, in fact, checked in so they don't have to be rebuilt by the bots or > > users? > > We should have the buildbots run "make touch", so they don't need to > run asdl_c.py. > How should we go about doing this? I don't think we have it documented in the devguide how to tweak the buildbot scripts? Eli -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130926/2c4cee6e/attachment.html> From donald at stufft.io Thu Sep 26 18:40:39 2013 From: donald at stufft.io (Donald Stufft) Date: Thu, 26 Sep 2013 12:40:39 -0400 Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for pronouncement? In-Reply-To: <20130926162842.58b74a5c@pitrou.net> References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com> <20130925165022.3d67e290@anarchist> <109B26FC-32A9-4DDE-9639-E48A0389455D@stufft.io> <20130925175129.1edc4348@anarchist> <CADiSq7fVrruk1zcE-ZwHUOyUqOPhTBnn2jcp9rKQ8dS6QeL+Zw@mail.gmail.com> <20130925190854.6ac76200@anarchist> <CADiSq7cMMkTjArg=0=rmDypHMo5aACYNQX=ct18N9jzru2=JQQ@mail.gmail.com> <CADiSq7cmSqnWCL0-s-g5dc1AgRRkfhd0VjYVSDj47bWQh537NA@mail.gmail.com> <20130926161046.17a6d4f6@pitrou.net> <97A4ED42-8961-4B01-BE20-DF684FC0930E@stufft.io> <20130926162842.58b74a5c@pitrou.net> Message-ID: <1201A938-D911-4541-8B92-1B9B7000FE6A@stufft.io> On Sep 26, 2013, at 10:28 AM, Antoine Pitrou <solipsis at pitrou.net> wrote: > Le Thu, 26 Sep 2013 10:22:55 -0400, > Donald Stufft <donald at stufft.io> a ?crit : >> Ideally people won't be typing either of them because it'll be >> installed automatically. They might in some cases (accidentally >> uninstalled pip?) > > Installing from source perhaps. Ah yea, installing from source is the time when they'd execute it, that slipped my mind. > >> I agree that it seems there is paranoia going on here and that the >> risk is low and making it just be a special cased new feature is ok. >> However the point of the underscore prefix on 2.7 and 3.3 is to more >> effectively communicate that on these pythons you shouldn't rely on >> that module existing. > > That's a fair argument, although it also makes the thing a bit ugly. Yea It's not my preferred thing to do and I think that calling it just ensurepip is fine, however I think if folks are worried about it, it could offer a compromise. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: <http://mail.python.org/pipermail/python-dev/attachments/20130926/8728ea81/attachment.sig> From benjamin at python.org Thu Sep 26 20:43:48 2013 From: benjamin at python.org (Benjamin Peterson) Date: Thu, 26 Sep 2013 14:43:48 -0400 Subject: [Python-Dev] asdl.py and Python-ast.[hc] In-Reply-To: <CAF-Rda_vthC4GdvJARvs75TYZTUNPLWbF2a4Uvm=7_V67bZvXg@mail.gmail.com> References: <CAF-Rda-dChgtDxtxcnXtUSrDDgNDCR2o5F665vk3P2gvN9UiNQ@mail.gmail.com> <CAPZV6o_y-8f0TJgEC=YKg+ZkV6vEzhet0U3rqV9DXo-PBaw=3g@mail.gmail.com> <CAF-Rda_vthC4GdvJARvs75TYZTUNPLWbF2a4Uvm=7_V67bZvXg@mail.gmail.com> Message-ID: <CAPZV6o9kt3qONZK=_y2pJCou0iBPrqpQeQ9vhVYkyof+3caxrA@mail.gmail.com> 2013/9/26 Eli Bendersky <eliben at gmail.com>: > > > > On Thu, Sep 26, 2013 at 7:28 AM, Benjamin Peterson <benjamin at python.org> > wrote: >> >> 2013/9/26 Eli Bendersky <eliben at gmail.com>: >> > Hi All, >> > >> > Earlier this morning I had a slight tackle with a couple of the 3.4 bots >> > (sorry everyone!). I fixed some problems in asdl.py - >> > http://hg.python.org/cpython/rev/21d46e3ae60c - and used the 'with' >> > statement. Some bots don't have Python 2.6+ and couldn't bootstrap >> > Python-ast.h/c >> > >> > Two questions: >> > >> > * Should I always check-in Python-ast.h and Python-ast.c when I touch >> > asdl* >> > ? The generated files are unchanged, it's only the timestamp that >> > changed. >> > * Can we, in theory, use new Pythons for asdl* code, because >> > Python-ast.* >> > are, in fact, checked in so they don't have to be rebuilt by the bots or >> > users? >> >> We should have the buildbots run "make touch", so they don't need to >> run asdl_c.py. > > > How should we go about doing this? Complain to Antoine I suppose. :) -- Regards, Benjamin From eliben at gmail.com Thu Sep 26 20:46:07 2013 From: eliben at gmail.com (Eli Bendersky) Date: Thu, 26 Sep 2013 11:46:07 -0700 Subject: [Python-Dev] asdl.py and Python-ast.[hc] In-Reply-To: <CAPZV6o9kt3qONZK=_y2pJCou0iBPrqpQeQ9vhVYkyof+3caxrA@mail.gmail.com> References: <CAF-Rda-dChgtDxtxcnXtUSrDDgNDCR2o5F665vk3P2gvN9UiNQ@mail.gmail.com> <CAPZV6o_y-8f0TJgEC=YKg+ZkV6vEzhet0U3rqV9DXo-PBaw=3g@mail.gmail.com> <CAF-Rda_vthC4GdvJARvs75TYZTUNPLWbF2a4Uvm=7_V67bZvXg@mail.gmail.com> <CAPZV6o9kt3qONZK=_y2pJCou0iBPrqpQeQ9vhVYkyof+3caxrA@mail.gmail.com> Message-ID: <CAF-Rda-d1_0So4JYCw9Zo6EuShRVoEkx1=x2J_ZRru2XpGMBkQ@mail.gmail.com> > >> 2013/9/26 Eli Bendersky <eliben at gmail.com>: > >> > Hi All, > >> > > >> > Earlier this morning I had a slight tackle with a couple of the 3.4 > bots > >> > (sorry everyone!). I fixed some problems in asdl.py - > >> > http://hg.python.org/cpython/rev/21d46e3ae60c - and used the 'with' > >> > statement. Some bots don't have Python 2.6+ and couldn't bootstrap > >> > Python-ast.h/c > >> > > >> > Two questions: > >> > > >> > * Should I always check-in Python-ast.h and Python-ast.c when I touch > >> > asdl* > >> > ? The generated files are unchanged, it's only the timestamp that > >> > changed. > >> > * Can we, in theory, use new Pythons for asdl* code, because > >> > Python-ast.* > >> > are, in fact, checked in so they don't have to be rebuilt by the bots > or > >> > users? > >> > >> We should have the buildbots run "make touch", so they don't need to > >> run asdl_c.py. > > > > > > How should we go about doing this? > > Complain to Antoine I suppose. :) > [+ Antoine] Done :) While we're at it, it seems that .hgtouch is wrong: Include/ast.h: Parser/Python.asdl Parser/asdl.py Parser/asdl_c.py Python/Python-ast.c: Include/ast.h The file Include/ast.h is not, in fact, auto-generated. But Include/Python-ast.h *is*, and it does not appear in this file. Typo? [+Ezio] Eli -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130926/6c4cb3a2/attachment.html> From tjreedy at udel.edu Thu Sep 26 21:51:25 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 26 Sep 2013 15:51:25 -0400 Subject: [Python-Dev] Revert #12085 fix for __del__ attribute error message In-Reply-To: <CADiSq7fBvyQjNJp4u6AVaQRWkJfYFTBMX=Nb8fjSv=8JBG0H5w@mail.gmail.com> References: <5241BED1.7010905@g.nevcal.com> <l1tvd9$6rg$1@ger.gmane.org> <20130925230520.GA94785@cskk.homeip.net> <CADiSq7fywLacQywujMma+HTov948FXgmtJ51gRon_iU4Fc2nxg@mail.gmail.com> <CAMSv6X1tQGPhLs0rnvNwV12z9czPkLWZx8dk9kmtYQ_RNwnY_Q@mail.gmail.com> <CADiSq7eiZku6BXEd3DtOGg4BRPNP3-QvtNNQbq0=SRf7eCJKYA@mail.gmail.com> <l20liu$gla$1@ger.gmane.org> <CADiSq7fBvyQjNJp4u6AVaQRWkJfYFTBMX=Nb8fjSv=8JBG0H5w@mail.gmail.com> Message-ID: <l2237l$u8h$1@ger.gmane.org> On 9/26/2013 3:17 AM, Nick Coghlan wrote: > On 26 September 2013 16:53, Georg Brandl <g.brandl at gmx.net> wrote: >>> Sure, that's doable, but it dumps the full repr of "obj" in the middle >>> of the sentence. The thing that's not practical is the neat and tidy >>> wording Georg proposed, because the thing passed as "obj" is actually >>> an arbitrary Python object that may have a messy repr (like a bound >>> method, which is what gets passed in the __del__ case), so there's >>> definite merit in keeping that repr at the *end* of the header line. >> >> Then this should be fine, I guess? >> >> Exception caught and not propagated in: <....> > > Sure. I still prefer something like "Could not propagate exception > from:" or "Caller could not propagate exception from <repr>" that > better indicates we're suppressing it because it's infeasible to raise > it rather than just because we feel like it, but any of them would > offer a decent improvement over the status quo. With the full traceback printed, with the line where the exception originated, I do not think that the representation of the object is needed. It was a substitute for the traceback. -- Terry Jan Reedy From solipsis at pitrou.net Thu Sep 26 22:45:22 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 26 Sep 2013 22:45:22 +0200 Subject: [Python-Dev] asdl.py and Python-ast.[hc] In-Reply-To: <CAF-Rda-d1_0So4JYCw9Zo6EuShRVoEkx1=x2J_ZRru2XpGMBkQ@mail.gmail.com> References: <CAF-Rda-dChgtDxtxcnXtUSrDDgNDCR2o5F665vk3P2gvN9UiNQ@mail.gmail.com> <CAPZV6o_y-8f0TJgEC=YKg+ZkV6vEzhet0U3rqV9DXo-PBaw=3g@mail.gmail.com> <CAF-Rda_vthC4GdvJARvs75TYZTUNPLWbF2a4Uvm=7_V67bZvXg@mail.gmail.com> <CAPZV6o9kt3qONZK=_y2pJCou0iBPrqpQeQ9vhVYkyof+3caxrA@mail.gmail.com> <CAF-Rda-d1_0So4JYCw9Zo6EuShRVoEkx1=x2J_ZRru2XpGMBkQ@mail.gmail.com> Message-ID: <20130926224522.58719924@fsol> On Thu, 26 Sep 2013 11:46:07 -0700 Eli Bendersky <eliben at gmail.com> wrote: > > >> 2013/9/26 Eli Bendersky <eliben at gmail.com>: > > >> > Hi All, > > >> > > > >> > Earlier this morning I had a slight tackle with a couple of the 3.4 > > bots > > >> > (sorry everyone!). I fixed some problems in asdl.py - > > >> > http://hg.python.org/cpython/rev/21d46e3ae60c - and used the 'with' > > >> > statement. Some bots don't have Python 2.6+ and couldn't bootstrap > > >> > Python-ast.h/c > > >> > > > >> > Two questions: > > >> > > > >> > * Should I always check-in Python-ast.h and Python-ast.c when I touch > > >> > asdl* > > >> > ? The generated files are unchanged, it's only the timestamp that > > >> > changed. > > >> > * Can we, in theory, use new Pythons for asdl* code, because > > >> > Python-ast.* > > >> > are, in fact, checked in so they don't have to be rebuilt by the bots > > or > > >> > users? > > >> > > >> We should have the buildbots run "make touch", so they don't need to > > >> run asdl_c.py. > > > > > > > > > How should we go about doing this? > > > > Complain to Antoine I suppose. :) > > > [+ Antoine] > Done :) "make touch"? Wow, I'm learning new stuff everyday :-) (what point is there in adding things to Makefile if nobody knows about them?) cheers Antoine. From eliben at gmail.com Thu Sep 26 22:49:58 2013 From: eliben at gmail.com (Eli Bendersky) Date: Thu, 26 Sep 2013 13:49:58 -0700 Subject: [Python-Dev] asdl.py and Python-ast.[hc] In-Reply-To: <20130926224522.58719924@fsol> References: <CAF-Rda-dChgtDxtxcnXtUSrDDgNDCR2o5F665vk3P2gvN9UiNQ@mail.gmail.com> <CAPZV6o_y-8f0TJgEC=YKg+ZkV6vEzhet0U3rqV9DXo-PBaw=3g@mail.gmail.com> <CAF-Rda_vthC4GdvJARvs75TYZTUNPLWbF2a4Uvm=7_V67bZvXg@mail.gmail.com> <CAPZV6o9kt3qONZK=_y2pJCou0iBPrqpQeQ9vhVYkyof+3caxrA@mail.gmail.com> <CAF-Rda-d1_0So4JYCw9Zo6EuShRVoEkx1=x2J_ZRru2XpGMBkQ@mail.gmail.com> <20130926224522.58719924@fsol> Message-ID: <CAF-Rda-mfUSkWwyZYaqW2_2xJSemaP8vUDytSDY98NpwDXsAMw@mail.gmail.com> On Thu, Sep 26, 2013 at 1:45 PM, Antoine Pitrou <solipsis at pitrou.net> wrote: > On Thu, 26 Sep 2013 11:46:07 -0700 > Eli Bendersky <eliben at gmail.com> wrote: > > > > >> 2013/9/26 Eli Bendersky <eliben at gmail.com>: > > > >> > Hi All, > > > >> > > > > >> > Earlier this morning I had a slight tackle with a couple of the > 3.4 > > > bots > > > >> > (sorry everyone!). I fixed some problems in asdl.py - > > > >> > http://hg.python.org/cpython/rev/21d46e3ae60c - and used the > 'with' > > > >> > statement. Some bots don't have Python 2.6+ and couldn't bootstrap > > > >> > Python-ast.h/c > > > >> > > > > >> > Two questions: > > > >> > > > > >> > * Should I always check-in Python-ast.h and Python-ast.c when I > touch > > > >> > asdl* > > > >> > ? The generated files are unchanged, it's only the timestamp that > > > >> > changed. > > > >> > * Can we, in theory, use new Pythons for asdl* code, because > > > >> > Python-ast.* > > > >> > are, in fact, checked in so they don't have to be rebuilt by the > bots > > > or > > > >> > users? > > > >> > > > >> We should have the buildbots run "make touch", so they don't need to > > > >> run asdl_c.py. > > > > > > > > > > > > How should we go about doing this? > > > > > > Complain to Antoine I suppose. :) > > > > > [+ Antoine] > > Done :) > > "make touch"? Wow, I'm learning new stuff everyday :-) > > (what point is there in adding things to Makefile if nobody knows about > them?) > The devguide would be a good place to mention this. http://bugs.python.org/issue15964 tracked the task for a while, but I'm not sure what the status is. Eli -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130926/cfe9adcb/attachment.html> From solipsis at pitrou.net Thu Sep 26 23:00:33 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 26 Sep 2013 23:00:33 +0200 Subject: [Python-Dev] asdl.py and Python-ast.[hc] References: <CAF-Rda-dChgtDxtxcnXtUSrDDgNDCR2o5F665vk3P2gvN9UiNQ@mail.gmail.com> <CAPZV6o_y-8f0TJgEC=YKg+ZkV6vEzhet0U3rqV9DXo-PBaw=3g@mail.gmail.com> <CAF-Rda_vthC4GdvJARvs75TYZTUNPLWbF2a4Uvm=7_V67bZvXg@mail.gmail.com> <CAPZV6o9kt3qONZK=_y2pJCou0iBPrqpQeQ9vhVYkyof+3caxrA@mail.gmail.com> Message-ID: <20130926230033.7bd7c584@fsol> On Thu, 26 Sep 2013 14:43:48 -0400 Benjamin Peterson <benjamin at python.org> wrote: > 2013/9/26 Eli Bendersky <eliben at gmail.com>: > > > > > > > > On Thu, Sep 26, 2013 at 7:28 AM, Benjamin Peterson <benjamin at python.org> > > wrote: > >> > >> 2013/9/26 Eli Bendersky <eliben at gmail.com>: > >> > Hi All, > >> > > >> > Earlier this morning I had a slight tackle with a couple of the 3.4 bots > >> > (sorry everyone!). I fixed some problems in asdl.py - > >> > http://hg.python.org/cpython/rev/21d46e3ae60c - and used the 'with' > >> > statement. Some bots don't have Python 2.6+ and couldn't bootstrap > >> > Python-ast.h/c > >> > > >> > Two questions: > >> > > >> > * Should I always check-in Python-ast.h and Python-ast.c when I touch > >> > asdl* > >> > ? The generated files are unchanged, it's only the timestamp that > >> > changed. > >> > * Can we, in theory, use new Pythons for asdl* code, because > >> > Python-ast.* > >> > are, in fact, checked in so they don't have to be rebuilt by the bots or > >> > users? > >> > >> We should have the buildbots run "make touch", so they don't need to > >> run asdl_c.py. > > > > > > How should we go about doing this? > > Complain to Antoine I suppose. :) Here you are: http://buildbot.python.org/all/builders/AMD64%20Snow%20Leop%203.x/builds/28/steps/compile/logs/stdio Of course, when it's using "-jN" there may be a race condition :-) Regards Antoine. From eliben at gmail.com Thu Sep 26 23:19:59 2013 From: eliben at gmail.com (Eli Bendersky) Date: Thu, 26 Sep 2013 14:19:59 -0700 Subject: [Python-Dev] asdl.py and Python-ast.[hc] In-Reply-To: <20130926230033.7bd7c584@fsol> References: <CAF-Rda-dChgtDxtxcnXtUSrDDgNDCR2o5F665vk3P2gvN9UiNQ@mail.gmail.com> <CAPZV6o_y-8f0TJgEC=YKg+ZkV6vEzhet0U3rqV9DXo-PBaw=3g@mail.gmail.com> <CAF-Rda_vthC4GdvJARvs75TYZTUNPLWbF2a4Uvm=7_V67bZvXg@mail.gmail.com> <CAPZV6o9kt3qONZK=_y2pJCou0iBPrqpQeQ9vhVYkyof+3caxrA@mail.gmail.com> <20130926230033.7bd7c584@fsol> Message-ID: <CAF-Rda-x35WUuXns-WEuKvKkr=D0=KajyZr2KrhMFn-JNwMVpQ@mail.gmail.com> On Thu, Sep 26, 2013 at 2:00 PM, Antoine Pitrou <solipsis at pitrou.net> wrote: > On Thu, 26 Sep 2013 14:43:48 -0400 > Benjamin Peterson <benjamin at python.org> wrote: > > 2013/9/26 Eli Bendersky <eliben at gmail.com>: > > > > > > > > > > > > On Thu, Sep 26, 2013 at 7:28 AM, Benjamin Peterson < > benjamin at python.org> > > > wrote: > > >> > > >> 2013/9/26 Eli Bendersky <eliben at gmail.com>: > > >> > Hi All, > > >> > > > >> > Earlier this morning I had a slight tackle with a couple of the 3.4 > bots > > >> > (sorry everyone!). I fixed some problems in asdl.py - > > >> > http://hg.python.org/cpython/rev/21d46e3ae60c - and used the 'with' > > >> > statement. Some bots don't have Python 2.6+ and couldn't bootstrap > > >> > Python-ast.h/c > > >> > > > >> > Two questions: > > >> > > > >> > * Should I always check-in Python-ast.h and Python-ast.c when I > touch > > >> > asdl* > > >> > ? The generated files are unchanged, it's only the timestamp that > > >> > changed. > > >> > * Can we, in theory, use new Pythons for asdl* code, because > > >> > Python-ast.* > > >> > are, in fact, checked in so they don't have to be rebuilt by the > bots or > > >> > users? > > >> > > >> We should have the buildbots run "make touch", so they don't need to > > >> run asdl_c.py. > > > > > > > > > How should we go about doing this? > > > > Complain to Antoine I suppose. :) > > Here you are: > > http://buildbot.python.org/all/builders/AMD64%20Snow%20Leop%203.x/builds/28/steps/compile/logs/stdio > > Of course, when it's using "-jN" there may be a race condition :-) > Right, which is probably even worse than before because now it's non-deterministic (first build will fail, subsequent will succeed). Is there a way to split it to: $ make touch $ make -jN all ? -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130926/412467a4/attachment-0001.html> From ncoghlan at gmail.com Thu Sep 26 23:20:49 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 27 Sep 2013 07:20:49 +1000 Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for pronouncement? In-Reply-To: <20130926161046.17a6d4f6@pitrou.net> References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com> <20130925165022.3d67e290@anarchist> <109B26FC-32A9-4DDE-9639-E48A0389455D@stufft.io> <20130925175129.1edc4348@anarchist> <CADiSq7fVrruk1zcE-ZwHUOyUqOPhTBnn2jcp9rKQ8dS6QeL+Zw@mail.gmail.com> <20130925190854.6ac76200@anarchist> <CADiSq7cMMkTjArg=0=rmDypHMo5aACYNQX=ct18N9jzru2=JQQ@mail.gmail.com> <CADiSq7cmSqnWCL0-s-g5dc1AgRRkfhd0VjYVSDj47bWQh537NA@mail.gmail.com> <20130926161046.17a6d4f6@pitrou.net> Message-ID: <CADiSq7c2BF1+p5GoVQ_Ftnqtk0dfPrzXXurg2uyuKW7gTQLtYg@mail.gmail.com> On 27 Sep 2013 00:12, "Antoine Pitrou" <solipsis at pitrou.net> wrote: > > Le Thu, 26 Sep 2013 14:54:49 +1000, > Nick Coghlan <ncoghlan at gmail.com> a ?crit : > > On 26 September 2013 14:30, Nick Coghlan <ncoghlan at gmail.com> wrote: > > > That said, there are changes that I think are definitely worth > > > making due to the concerns you raise: > > > > > > - the module name should be "_ensurepip" in all versions > > > - the PEP should explicitly state that the "don't remove _ensurepip > > > and it's wheel files" caveat for redistributors applies only in 3.4+ > > > (where removing it will break pyvenv) > > > > Donald pointed out it makes more sense to continue with the idea of a > > properly documented public "ensurepip" module in 3.4+, and have the > > "_ensurepip" version as an implementation detail of the 2.7 and 3.3 > > installers that is included in the stdlib primarily so it can be > > covered by the existing buildbot fleet. > > Hmm, but what is the point of "_ensurepip" exactly? Are people supposed > to type "python -m _ensurepip"? > > With all due respect, Barry's argument looks rather paranoid to me. > I would suggest a clear choice: > - either having "ensurepip" in 2.7 is useful and we endorse it as a > public module (not something hidden somewhere) - which I personally > think is reasonable > - or it's not useful and we don't introduce it at all > > A middleground doesn't make sense here, except in a broken "design by > committee" sense. The *only* reason to have _ensurepip in the older versions is to make the "install pip" option in the installers work. By contrast, in 3.4+ it will be integrated into pyvenv and a fully supported API. The two levels make sense to me, especially since pyvenv in 3.3 *won't* be updated to invoke it automatically. 2.7 and 3.3 users will still need to install and use virtualenv to get pip automatically installed in virtual environments. Cheers, Nick. > > Regards > > Antoine. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130927/85418481/attachment.html> From solipsis at pitrou.net Thu Sep 26 23:23:42 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 26 Sep 2013 23:23:42 +0200 Subject: [Python-Dev] asdl.py and Python-ast.[hc] In-Reply-To: <CAF-Rda-x35WUuXns-WEuKvKkr=D0=KajyZr2KrhMFn-JNwMVpQ@mail.gmail.com> References: <CAF-Rda-dChgtDxtxcnXtUSrDDgNDCR2o5F665vk3P2gvN9UiNQ@mail.gmail.com> <CAPZV6o_y-8f0TJgEC=YKg+ZkV6vEzhet0U3rqV9DXo-PBaw=3g@mail.gmail.com> <CAF-Rda_vthC4GdvJARvs75TYZTUNPLWbF2a4Uvm=7_V67bZvXg@mail.gmail.com> <CAPZV6o9kt3qONZK=_y2pJCou0iBPrqpQeQ9vhVYkyof+3caxrA@mail.gmail.com> <20130926230033.7bd7c584@fsol> <CAF-Rda-x35WUuXns-WEuKvKkr=D0=KajyZr2KrhMFn-JNwMVpQ@mail.gmail.com> Message-ID: <20130926232342.2da80669@fsol> On Thu, 26 Sep 2013 14:19:59 -0700 Eli Bendersky <eliben at gmail.com> wrote: > On Thu, Sep 26, 2013 at 2:00 PM, Antoine Pitrou <solipsis at pitrou.net> wrote: > > > On Thu, 26 Sep 2013 14:43:48 -0400 > > Benjamin Peterson <benjamin at python.org> wrote: > > > 2013/9/26 Eli Bendersky <eliben at gmail.com>: > > > > > > > > > > > > > > > > On Thu, Sep 26, 2013 at 7:28 AM, Benjamin Peterson < > > benjamin at python.org> > > > > wrote: > > > >> > > > >> 2013/9/26 Eli Bendersky <eliben at gmail.com>: > > > >> > Hi All, > > > >> > > > > >> > Earlier this morning I had a slight tackle with a couple of the 3.4 > > bots > > > >> > (sorry everyone!). I fixed some problems in asdl.py - > > > >> > http://hg.python.org/cpython/rev/21d46e3ae60c - and used the 'with' > > > >> > statement. Some bots don't have Python 2.6+ and couldn't bootstrap > > > >> > Python-ast.h/c > > > >> > > > > >> > Two questions: > > > >> > > > > >> > * Should I always check-in Python-ast.h and Python-ast.c when I > > touch > > > >> > asdl* > > > >> > ? The generated files are unchanged, it's only the timestamp that > > > >> > changed. > > > >> > * Can we, in theory, use new Pythons for asdl* code, because > > > >> > Python-ast.* > > > >> > are, in fact, checked in so they don't have to be rebuilt by the > > bots or > > > >> > users? > > > >> > > > >> We should have the buildbots run "make touch", so they don't need to > > > >> run asdl_c.py. > > > > > > > > > > > > How should we go about doing this? > > > > > > Complain to Antoine I suppose. :) > > > > Here you are: > > > > http://buildbot.python.org/all/builders/AMD64%20Snow%20Leop%203.x/builds/28/steps/compile/logs/stdio > > > > Of course, when it's using "-jN" there may be a race condition :-) > > > > Right, which is probably even worse than before because now it's > non-deterministic (first build will fail, subsequent will succeed). Only if "N" in "-jN" is big enough, I suspect. > Is there a way to split it to: > > $ make touch > $ make -jN all Probably but... I'm really too lazy to do this myself. Buildbot isn't very pleasant. Perhaps someone else wants to do it? Regards Antoine. From solipsis at pitrou.net Fri Sep 27 14:09:50 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 27 Sep 2013 14:09:50 +0200 Subject: [Python-Dev] asdl.py and Python-ast.[hc] References: <CAF-Rda-dChgtDxtxcnXtUSrDDgNDCR2o5F665vk3P2gvN9UiNQ@mail.gmail.com> <CAPZV6o_y-8f0TJgEC=YKg+ZkV6vEzhet0U3rqV9DXo-PBaw=3g@mail.gmail.com> Message-ID: <20130927140950.25c11567@pitrou.net> Le Thu, 26 Sep 2013 10:28:46 -0400, Benjamin Peterson <benjamin at python.org> a ?crit : > 2013/9/26 Eli Bendersky <eliben at gmail.com>: > > Hi All, > > > > Earlier this morning I had a slight tackle with a couple of the 3.4 > > bots (sorry everyone!). I fixed some problems in asdl.py - > > http://hg.python.org/cpython/rev/21d46e3ae60c - and used the 'with' > > statement. Some bots don't have Python 2.6+ and couldn't bootstrap > > Python-ast.h/c > > > > Two questions: > > > > * Should I always check-in Python-ast.h and Python-ast.c when I > > touch asdl* ? The generated files are unchanged, it's only the > > timestamp that changed. > > * Can we, in theory, use new Pythons for asdl* code, because > > Python-ast.* are, in fact, checked in so they don't have to be > > rebuilt by the bots or users? > > We should have the buildbots run "make touch", so they don't need to > run asdl_c.py. So, you're gonna like this: hg --config extensions.touch=Tools/hg/hgtouch.py touch -v Tools/hg/hgtouch.py:21: Warning: 'with' will become a reserved keyword in Python 2.6 *** failed to import extension touch from Tools/hg/hgtouch.py: invalid syntax (hgtouch.py, line 21) hg: unknown command 'touch' http://buildbot.python.org/all/buildslaves/bolen-ubuntu (that buildbot runs Ubuntu 8.04 LTS, for the record) Regards Antoine. From eliben at gmail.com Fri Sep 27 15:46:44 2013 From: eliben at gmail.com (Eli Bendersky) Date: Fri, 27 Sep 2013 06:46:44 -0700 Subject: [Python-Dev] asdl.py and Python-ast.[hc] In-Reply-To: <20130927140950.25c11567@pitrou.net> References: <CAF-Rda-dChgtDxtxcnXtUSrDDgNDCR2o5F665vk3P2gvN9UiNQ@mail.gmail.com> <CAPZV6o_y-8f0TJgEC=YKg+ZkV6vEzhet0U3rqV9DXo-PBaw=3g@mail.gmail.com> <20130927140950.25c11567@pitrou.net> Message-ID: <CAF-Rda-dcwFA__pa2qA-EMMih5f2Xmr5EvXL0p=WWvtJi2YGyg@mail.gmail.com> On Fri, Sep 27, 2013 at 5:09 AM, Antoine Pitrou <solipsis at pitrou.net> wrote: > Le Thu, 26 Sep 2013 10:28:46 -0400, > Benjamin Peterson <benjamin at python.org> a ?crit : > > 2013/9/26 Eli Bendersky <eliben at gmail.com>: > > > Hi All, > > > > > > Earlier this morning I had a slight tackle with a couple of the 3.4 > > > bots (sorry everyone!). I fixed some problems in asdl.py - > > > http://hg.python.org/cpython/rev/21d46e3ae60c - and used the 'with' > > > statement. Some bots don't have Python 2.6+ and couldn't bootstrap > > > Python-ast.h/c > > > > > > Two questions: > > > > > > * Should I always check-in Python-ast.h and Python-ast.c when I > > > touch asdl* ? The generated files are unchanged, it's only the > > > timestamp that changed. > > > * Can we, in theory, use new Pythons for asdl* code, because > > > Python-ast.* are, in fact, checked in so they don't have to be > > > rebuilt by the bots or users? > > > > We should have the buildbots run "make touch", so they don't need to > > run asdl_c.py. > > So, you're gonna like this: > > hg --config extensions.touch=Tools/hg/hgtouch.py touch -v > Tools/hg/hgtouch.py:21: Warning: 'with' will become a reserved keyword in > Python 2.6 > *** failed to import extension touch from Tools/hg/hgtouch.py: invalid > syntax (hgtouch.py, line 21) > hg: unknown command 'touch' > > http://buildbot.python.org/all/buildslaves/bolen-ubuntu > > (that buildbot runs Ubuntu 8.04 LTS, for the record) > We have to define a clear boundary between what needs to be compatible to old Pythons and what doesn't. As far as auto-generated files are concerned, once we set up the bots properly and fix the .hgtouch bugs, there's no reason for anyone to need an old Python for anything except the hgtouch extension itself. But the latter has to be back-ported to work on older Pythons (I suppose from 2.4, or is there reason to go earlier?) FWIW Antoine, I'll be happy to help with the buildbot setup, but you'll have to tell me how to get access to our buildbot scripts :-) Eli -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130927/48959312/attachment.html> From stephen at xemacs.org Fri Sep 27 16:08:33 2013 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Fri, 27 Sep 2013 23:08:33 +0900 Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for pronouncement? In-Reply-To: <CADiSq7dJYDJwok1iccJ2mW9Axi_1JOYZ8rzDk50DxGpWDk4kwA@mail.gmail.com> References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com> <20130925165022.3d67e290@anarchist> <109B26FC-32A9-4DDE-9639-E48A0389455D@stufft.io> <20130925175129.1edc4348@anarchist> <55AAF039-CCAB-467F-8FA3-74D31CB80A66@stufft.io> <20130925190410.0a5729f9@anarchist> <7FBFFC39-A0AF-47FD-9715-944D13212D10@stufft.io> <87bo3g1h9i.fsf@uwakimon.sk.tsukuba.ac.jp> <CADiSq7dJYDJwok1iccJ2mW9Axi_1JOYZ8rzDk50DxGpWDk4kwA@mail.gmail.com> Message-ID: <8761tm1hni.fsf@uwakimon.sk.tsukuba.ac.jp> Nick Coghlan writes: > I'm not sure what usage model you're assuming for _ensurepip, but it > appears to be wrong. End users should be able to just run pip, and > either have it work, or else get a message from the OS vendor telling > them to install the appropriate system package. I don't understand how you arrange for that message on existing installs. Wouldn't it be easier to just lobby the distros to make Python dependent on pip? And speaking of vendors, do you expect Apple and Microsoft to provide such a message? And such a system package? If you already are running a Linux distro or MacPorts, you do "apt-get python-pip" and "port install py-pip" respectively. I bet Cygwin is the same with yet another spelling. Where's the problem? You say: > New users on Windows and Mac OS X. I've heard many more complaints > from folks running tutorials about the pip bootstrapping process than > I ever have from the community at large about the GIL :P I bet those users are *not* running third-party distros, but rather are sitting in front of pretty close to plain vanilla factory installs of the OS, no? And "new users" on Mac OS X already have "old installs" of Python, no? That's my model. In that model I don't see backporting PEP 453 to Python 2.7 as being a sufficiently reliable way to provide a smooth user experience to justify breaking the "no new features" rule (which is at the "read my lips" level after the True/False fiasco). Get a commitment from Apple to put 2.7.6 in their next upgrades for their OS, and then maybe you'd have enough leverage to tip the balance. I certainly would concede the point. But without that, you're telling Mac users "you have to upgrade Python from a 3rd party site." Is that really the way to make new users participating in a tutorial session happy? (You tell me, I'm just introspecting here.) Steve From p.f.moore at gmail.com Fri Sep 27 16:26:41 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 27 Sep 2013 15:26:41 +0100 Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for pronouncement? In-Reply-To: <8761tm1hni.fsf@uwakimon.sk.tsukuba.ac.jp> References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com> <20130925165022.3d67e290@anarchist> <109B26FC-32A9-4DDE-9639-E48A0389455D@stufft.io> <20130925175129.1edc4348@anarchist> <55AAF039-CCAB-467F-8FA3-74D31CB80A66@stufft.io> <20130925190410.0a5729f9@anarchist> <7FBFFC39-A0AF-47FD-9715-944D13212D10@stufft.io> <87bo3g1h9i.fsf@uwakimon.sk.tsukuba.ac.jp> <CADiSq7dJYDJwok1iccJ2mW9Axi_1JOYZ8rzDk50DxGpWDk4kwA@mail.gmail.com> <8761tm1hni.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <CACac1F-soYfp5Bg0sq+zgL6he301Vn7j2nRyTSodXoHBi+3K8Q@mail.gmail.com> On 27 September 2013 15:08, Stephen J. Turnbull <stephen at xemacs.org> wrote: >> New users on Windows and Mac OS X. I've heard many more complaints > > from folks running tutorials about the pip bootstrapping process than > > I ever have from the community at large about the GIL :P > > I bet those users are *not* running third-party distros, but rather > are sitting in front of pretty close to plain vanilla factory installs > of the OS, no? And "new users" on Mac OS X already have "old installs" > of Python, no? Windows users who don't use a third-party distro like Enthought, generally download the python.org installer. At the moment, that doesn't give them a "pip" command. So if they want to install any third party package, they have to start by installing pip. The instructions for that are reasonably clear, but non-trivial, largely because tools like curl are not commonly available on Windows, and by default running a Python script may not do what you expect. Rather than try to fix these problems (which are *hard*) the intent is to have the pip command installed by the python.org installer. So, for Windows users, installing a package becomes "pip install XXX". The problem is that this doesn't work for existing releases (2.7 and 3.3). Rather than have a convoluted set of instructions that goes "if you're not on Python 3.4+, (basically do what we have at the moment)" the PEP makes it possible to say "If you have a Python older than 3.4, upgrade to the latest maintenance release of your version of Python, then use pip". People who have reasons not to use the latest *maintenance* release are assumed to be special cases who can either deal with the more complex current process, or can work something out for themselves. (They may not be, but there's not much we can do in practice to help them). Nobody in the basic target group should be running ensurepip (under any name) manually. I can't speak for Linux distros or OSX users, but for Windows I do believe that this is a significant improvement, and worth the (IMO, negligible) risk involved in adding this to a maintenance release. There *may* be some mileage in a debate about why we don't just bundle pip in the installer, rather than having helpers in core Python itself, but the long & short of it is that ensurepip is needed for 3.4+ to make installing pip in venvs work by default, and why have a completely different and so less well tested mechanism for older versions? Paul. From martin at v.loewis.de Fri Sep 27 16:26:07 2013 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri, 27 Sep 2013 16:26:07 +0200 Subject: [Python-Dev] asdl.py and Python-ast.[hc] In-Reply-To: <CAF-Rda-dChgtDxtxcnXtUSrDDgNDCR2o5F665vk3P2gvN9UiNQ@mail.gmail.com> References: <CAF-Rda-dChgtDxtxcnXtUSrDDgNDCR2o5F665vk3P2gvN9UiNQ@mail.gmail.com> Message-ID: <5245957F.2040002@v.loewis.de> Am 26.09.13 15:52, schrieb Eli Bendersky: > * Should I always check-in Python-ast.h and Python-ast.c when I touch > asdl* ? The generated files are unchanged, it's only the timestamp that > changed. If they really didn't change, I don't think it matters much. I believe there is a fundamental problem in Mercurial which fails to get it "right" - with the defense that one shouldn't check in generated files in the first place. "hg touch" is supposed to work around this limitation. So when you only check in the generator, anybody updating should do "make touch" after the update, which should touch the generated files even though they didn't change. > * Can we, in theory, use new Pythons for asdl* code, because > Python-ast.* are, in fact, checked in so they don't have to be rebuilt > by the bots or users? I'd say yes. That's the point of checking in generated files, so that users don't need to run the generator. > While we're at it, it seems that .hgtouch is wrong: > > Include/ast.h: Parser/Python.asdl Parser/asdl.py Parser/asdl_c.py > Python/Python-ast.c: Include/ast.h > > The file Include/ast.h is not, It may well be incorrect - feel free to fix it. Regards, Martin From martin at v.loewis.de Fri Sep 27 16:34:41 2013 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri, 27 Sep 2013 16:34:41 +0200 Subject: [Python-Dev] asdl.py and Python-ast.[hc] In-Reply-To: <20130926230033.7bd7c584@fsol> References: <CAF-Rda-dChgtDxtxcnXtUSrDDgNDCR2o5F665vk3P2gvN9UiNQ@mail.gmail.com> <CAPZV6o_y-8f0TJgEC=YKg+ZkV6vEzhet0U3rqV9DXo-PBaw=3g@mail.gmail.com> <CAF-Rda_vthC4GdvJARvs75TYZTUNPLWbF2a4Uvm=7_V67bZvXg@mail.gmail.com> <CAPZV6o9kt3qONZK=_y2pJCou0iBPrqpQeQ9vhVYkyof+3caxrA@mail.gmail.com> <20130926230033.7bd7c584@fsol> Message-ID: <52459781.5020300@v.loewis.de> Am 26.09.13 23:00, schrieb Antoine Pitrou: > Here you are: > http://buildbot.python.org/all/builders/AMD64%20Snow%20Leop%203.x/builds/28/steps/compile/logs/stdio > > Of course, when it's using "-jN" there may be a race condition :-) Could you make "make touch" a separate build step? Regards, Martin From breamoreboy at yahoo.co.uk Fri Sep 27 16:39:50 2013 From: breamoreboy at yahoo.co.uk (Mark Lawrence) Date: Fri, 27 Sep 2013 15:39:50 +0100 Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for pronouncement? In-Reply-To: <CACac1F-soYfp5Bg0sq+zgL6he301Vn7j2nRyTSodXoHBi+3K8Q@mail.gmail.com> References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com> <20130925165022.3d67e290@anarchist> <109B26FC-32A9-4DDE-9639-E48A0389455D@stufft.io> <20130925175129.1edc4348@anarchist> <55AAF039-CCAB-467F-8FA3-74D31CB80A66@stufft.io> <20130925190410.0a5729f9@anarchist> <7FBFFC39-A0AF-47FD-9715-944D13212D10@stufft.io> <87bo3g1h9i.fsf@uwakimon.sk.tsukuba.ac.jp> <CADiSq7dJYDJwok1iccJ2mW9Axi_1JOYZ8rzDk50DxGpWDk4kwA@mail.gmail.com> <8761tm1hni.fsf@uwakimon.sk.tsukuba.ac.jp> <CACac1F-soYfp5Bg0sq+zgL6he301Vn7j2nRyTSodXoHBi+3K8Q@mail.gmail.com> Message-ID: <l245bj$gn3$1@ger.gmane.org> On 27/09/2013 15:26, Paul Moore wrote: > So, for Windows users, installing a package becomes "pip install XXX". > Apologies if this has already been said but it only works if you've already installed an appropriate compiler. I've lost count of the number of times I've tried this, got the (rather cyptic) "error: Unable to find vcvarsall.bat" message and then gone off to find a suitable binary download. -- Cheers. Mark Lawrence From donald at stufft.io Fri Sep 27 16:42:29 2013 From: donald at stufft.io (Donald Stufft) Date: Fri, 27 Sep 2013 10:42:29 -0400 Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for pronouncement? In-Reply-To: <l245bj$gn3$1@ger.gmane.org> References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com> <20130925165022.3d67e290@anarchist> <109B26FC-32A9-4DDE-9639-E48A0389455D@stufft.io> <20130925175129.1edc4348@anarchist> <55AAF039-CCAB-467F-8FA3-74D31CB80A66@stufft.io> <20130925190410.0a5729f9@anarchist> <7FBFFC39-A0AF-47FD-9715-944D13212D10@stufft.io> <87bo3g1h9i.fsf@uwakimon.sk.tsukuba.ac.jp> <CADiSq7dJYDJwok1iccJ2mW9Axi_1JOYZ8rzDk50DxGpWDk4kwA@mail.gmail.com> <8761tm1hni.fsf@uwakimon.sk.tsukuba.ac.jp> <CACac1F-soYfp5Bg0sq+zgL6he301Vn7j2nRyTSodXoHBi+3K8Q@mail.gmail.com> <l245bj$gn3$1@ger.gmane.org> Message-ID: <5CA1E89E-DCB6-4483-97DC-251841D22A15@stufft.io> On Sep 27, 2013, at 10:39 AM, Mark Lawrence <breamoreboy at yahoo.co.uk> wrote: > Apologies if this has already been said but it only works if you've already installed an appropriate compiler. I've lost count of the number of times I've tried this, got the (rather cyptic) "error: Unable to find vcvarsall.bat" message and then gone off to find a suitable binary download. Going forward Wheels are binary packages that pip can install. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: <http://mail.python.org/pipermail/python-dev/attachments/20130927/d33b015b/attachment.sig> From solipsis at pitrou.net Fri Sep 27 16:48:33 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 27 Sep 2013 16:48:33 +0200 Subject: [Python-Dev] asdl.py and Python-ast.[hc] References: <CAF-Rda-dChgtDxtxcnXtUSrDDgNDCR2o5F665vk3P2gvN9UiNQ@mail.gmail.com> <CAPZV6o_y-8f0TJgEC=YKg+ZkV6vEzhet0U3rqV9DXo-PBaw=3g@mail.gmail.com> <CAF-Rda_vthC4GdvJARvs75TYZTUNPLWbF2a4Uvm=7_V67bZvXg@mail.gmail.com> <CAPZV6o9kt3qONZK=_y2pJCou0iBPrqpQeQ9vhVYkyof+3caxrA@mail.gmail.com> <20130926230033.7bd7c584@fsol> <52459781.5020300@v.loewis.de> Message-ID: <20130927164833.3f0ccf1f@pitrou.net> Le Fri, 27 Sep 2013 16:34:41 +0200, "Martin v. L?wis" <martin at v.loewis.de> a ?crit : > Am 26.09.13 23:00, schrieb Antoine Pitrou: > > > Here you are: > > http://buildbot.python.org/all/builders/AMD64%20Snow%20Leop%203.x/builds/28/steps/compile/logs/stdio > > > > Of course, when it's using "-jN" there may be a race condition :-) > > Could you make "make touch" a separate build step? I'm not sure it fits in buildbot's standard "GNUAutoconf" build factory: http://docs.buildbot.net/current/manual/cfg-buildfactories.html#index-1 Note that "compile" is a "shell command" so perhaps it is actually possible to pass "make touch && make -jN". Otherwise, another build factory will have to be used. I'll give an access to Eli so that he can try to figure it out :-) Regards Antoine. From p.f.moore at gmail.com Fri Sep 27 17:06:33 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 27 Sep 2013 16:06:33 +0100 Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for pronouncement? In-Reply-To: <l245bj$gn3$1@ger.gmane.org> References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com> <20130925165022.3d67e290@anarchist> <109B26FC-32A9-4DDE-9639-E48A0389455D@stufft.io> <20130925175129.1edc4348@anarchist> <55AAF039-CCAB-467F-8FA3-74D31CB80A66@stufft.io> <20130925190410.0a5729f9@anarchist> <7FBFFC39-A0AF-47FD-9715-944D13212D10@stufft.io> <87bo3g1h9i.fsf@uwakimon.sk.tsukuba.ac.jp> <CADiSq7dJYDJwok1iccJ2mW9Axi_1JOYZ8rzDk50DxGpWDk4kwA@mail.gmail.com> <8761tm1hni.fsf@uwakimon.sk.tsukuba.ac.jp> <CACac1F-soYfp5Bg0sq+zgL6he301Vn7j2nRyTSodXoHBi+3K8Q@mail.gmail.com> <l245bj$gn3$1@ger.gmane.org> Message-ID: <CACac1F_fifT=+2XQ34xGFksPre_tai7rzhxfu3nC68-Q=KB-cg@mail.gmail.com> On 27 September 2013 15:39, Mark Lawrence <breamoreboy at yahoo.co.uk> wrote: > On 27/09/2013 15:26, Paul Moore wrote: > >> So, for Windows users, installing a package becomes "pip install XXX". >> > > Apologies if this has already been said but it only works if you've already > installed an appropriate compiler. I've lost count of the number of times > I've tried this, got the (rather cyptic) "error: Unable to find > vcvarsall.bat" message and then gone off to find a suitable binary download. I was assuming that you had whatever's necessary to install - a compiler, any necessary libraries, whatever. Or more plausibly, as Donald pointed out, a wheel. But yes, you're right, this doesn't solve the problem of *building* distributions. It's not intended to (except by enabling communities to start distributing wheels as a pain-free binary solution). Whether that's an issue depends on the community (AIUI, it's a major hassle for scientific users, not so much for web developers, for example) Paul From eliben at gmail.com Fri Sep 27 17:26:33 2013 From: eliben at gmail.com (Eli Bendersky) Date: Fri, 27 Sep 2013 08:26:33 -0700 Subject: [Python-Dev] asdl.py and Python-ast.[hc] In-Reply-To: <5245957F.2040002@v.loewis.de> References: <CAF-Rda-dChgtDxtxcnXtUSrDDgNDCR2o5F665vk3P2gvN9UiNQ@mail.gmail.com> <5245957F.2040002@v.loewis.de> Message-ID: <CAF-Rda9wcfBw+4e1sNXCNiLhqeVL-aV=5MnH0NqAvSnWrk61YA@mail.gmail.com> On Fri, Sep 27, 2013 at 7:26 AM, "Martin v. L?wis" <martin at v.loewis.de>wrote: > Am 26.09.13 15:52, schrieb Eli Bendersky: > > * Should I always check-in Python-ast.h and Python-ast.c when I touch > > asdl* ? The generated files are unchanged, it's only the timestamp that > > changed. > > If they really didn't change, I don't think it matters much. I believe > there is a fundamental problem in Mercurial which fails to get it > "right" - with the defense that one shouldn't check in generated files > in the first place. > > "hg touch" is supposed to work around this limitation. So when you only > check in the generator, anybody updating should do "make touch" after > the update, which should touch the generated files even though they > didn't change. > > > * Can we, in theory, use new Pythons for asdl* code, because > > Python-ast.* are, in fact, checked in so they don't have to be rebuilt > > by the bots or users? > > I'd say yes. That's the point of checking in generated files, so that > users don't need to run the generator. > Thank you for the explanation, Martin. > > > While we're at it, it seems that .hgtouch is wrong: > > > > Include/ast.h: Parser/Python.asdl Parser/asdl.py Parser/asdl_c.py > > Python/Python-ast.c: Include/ast.h > > > > The file Include/ast.h is not, > > It may well be incorrect - feel free to fix it. > Thanks. There are a couple of things to address before we can try to actually run 'make touch' on all bots. I opened http://bugs.python.org/issue19106 to track this. Eli -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130927/72272b0c/attachment-0001.html> From eliben at gmail.com Fri Sep 27 18:02:53 2013 From: eliben at gmail.com (Eli Bendersky) Date: Fri, 27 Sep 2013 09:02:53 -0700 Subject: [Python-Dev] Official github mirror for CPython? In-Reply-To: <CAD+XWwrCD_tLM0b1OFea5ZZq7oH+1TYR_yNyp=mNDKJe7ZBALQ@mail.gmail.com> References: <CAF-Rda-=wFp_pWSa2gqFxet1M7pf68si=LKZ7FyGU7UrfRS3FA@mail.gmail.com> <51F1380E.9070608@python.org> <CAD+XWwrCD_tLM0b1OFea5ZZq7oH+1TYR_yNyp=mNDKJe7ZBALQ@mail.gmail.com> Message-ID: <CAF-Rda8oajAZUf-AaVnHi4q8h9JSSyTRRsNMar8iUf5-opXF_Q@mail.gmail.com> On Thu, Jul 25, 2013 at 7:48 AM, Brian Curtin <brian at python.org> wrote: > On Thu, Jul 25, 2013 at 9:37 AM, Christian Heimes <christian at python.org> > wrote: > > Am 25.07.2013 16:29, schrieb Eli Bendersky: > >> Hi all, > >> > >> I've been looking for a Github mirror for Python, and found two: > >> > >> * https://github.com/python-git/python has a lot of > forks/watches/starts > >> but seems to be very out of date (last updated 4 years ago) > >> * https://github.com/python-mirror/python doesn't appear to be very > >> popular but is updated daily > >> > >> Are some of you the owners of these repositories? Should we consolidate > >> to a single "semi-official" mirror? > > > > +1 > > > > Does the PSF have an official account on github? We have one on > bitbucket... > > I don't remember who runs this, and I thought I was in it (maybe just > on BB), but: https://github.com/python > The list of members for https://github.com/python is https://github.com/python?tab=members How can I get added to that list? When that happens, I can try to set-up a regularly updated CPython mirror. Eli -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130927/ebbc1e68/attachment.html> From status at bugs.python.org Fri Sep 27 18:07:42 2013 From: status at bugs.python.org (Python tracker) Date: Fri, 27 Sep 2013 18:07:42 +0200 (CEST) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20130927160742.ACD2956A31@psf.upfronthosting.co.za> ACTIVITY SUMMARY (2013-09-20 - 2013-09-27) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 4256 (+28) closed 26624 (+22) total 30880 (+50) Open issues with patches: 1963 Issues opened (44) ================== #19058: test_sys.test_ioencoding_nonascii() fails with ASCII locale en http://bugs.python.org/issue19058 opened by pitrou #19060: docs: note that subprocess doesn't replace os.exec* http://bugs.python.org/issue19060 opened by techtonik #19062: problems with pyshell to get userdir http://bugs.python.org/issue19062 opened by weizhao #19063: Python 3.3.3 encodes emails containing non-ascii data as 7bit http://bugs.python.org/issue19063 opened by apollo13 #19065: sqlite3 timestamp adapter chokes on timezones http://bugs.python.org/issue19065 opened by acdha #19066: os.execv fails with spaced names on Windows http://bugs.python.org/issue19066 opened by techtonik #19067: Built-in range docstrings are not PEP-8 compatible http://bugs.python.org/issue19067 opened by marco.buttu #19068: Some built-in complex docstrings are not PEP-8 compatible http://bugs.python.org/issue19068 opened by marco.buttu #19069: Built-in float docstrings are not PEP-8 compatible http://bugs.python.org/issue19069 opened by marco.buttu #19070: In place operators of weakref.proxy() not returning self. http://bugs.python.org/issue19070 opened by grahamd #19071: Documentation on what self is for module-level functions is mi http://bugs.python.org/issue19071 opened by grahamd #19072: classmethod doesn't honour descriptor protocol of wrapped call http://bugs.python.org/issue19072 opened by grahamd #19073: Inability to specific __qualname__ as a property on a class in http://bugs.python.org/issue19073 opened by grahamd #19074: Add PySide to GUI FAQ http://bugs.python.org/issue19074 opened by techtonik #19075: Add sorting algorithm visualization to turtledemo http://bugs.python.org/issue19075 opened by Jason.Yeo #19076: Pdb.do_break calls error with obsolete file kwarg http://bugs.python.org/issue19076 opened by Michael.Smith #19077: More robust TemporaryDirectory cleanup http://bugs.python.org/issue19077 opened by serhiy.storchaka #19078: Allow reversed(memoryview), like memoryview http://bugs.python.org/issue19078 opened by Claudiu.Popa #19079: chameleon benchmark fails on 3.4 http://bugs.python.org/issue19079 opened by pitrou #19080: Enrich SyntaxError with additional information http://bugs.python.org/issue19080 opened by alonho #19081: zipimport behaves badly when the zip file changes while the pr http://bugs.python.org/issue19081 opened by gregory.p.smith #19082: Lib/xmlrpc/client.py demo code points to the dead server http://bugs.python.org/issue19082 opened by vajrasky #19083: IDNA prefix should be case insensitive http://bugs.python.org/issue19083 opened by Pepijn.de.Vos #19084: No way to use TLS-PSK from python ssl http://bugs.python.org/issue19084 opened by karlp #19085: Add tkinter basic options tests http://bugs.python.org/issue19085 opened by serhiy.storchaka #19086: Make fsum usable incrementally. http://bugs.python.org/issue19086 opened by oscarbenjamin #19087: bytearray front-slicing not optimized http://bugs.python.org/issue19087 opened by pitrou #19088: TypeError with pickle in embedded python3.3 when starting mult http://bugs.python.org/issue19088 opened by Larry.Pete #19089: Windows: Make Ctrl-D exit key combination cross-platform http://bugs.python.org/issue19089 opened by techtonik #19091: ast.parse gives wrong position for some Names when non-ascii c http://bugs.python.org/issue19091 opened by Aivar.Annamaa #19092: ExitStack.__exit__ incorrectly suppresses exceptions in __exit http://bugs.python.org/issue19092 opened by hniksic #19094: urljoin should raise a TypeError if URL is not a string http://bugs.python.org/issue19094 opened by jason.coombs #19095: Document SSLSocket.getpeercert always returns None without do_ http://bugs.python.org/issue19095 opened by dsuch #19096: multiprocessing.Pool._terminate_pool restarts workers during s http://bugs.python.org/issue19096 opened by ecatmur #19097: bool(cgi.FieldStorage(...)) may be False unexpectedly http://bugs.python.org/issue19097 opened by gvanrossum #19099: struct.pack fails first time with unicode fmt http://bugs.python.org/issue19099 opened by miwa #19100: Use backslashreplace in pprint http://bugs.python.org/issue19100 opened by serhiy.storchaka #19102: Add tests for CLI of the tabnanny module http://bugs.python.org/issue19102 opened by berker.peksag #19103: Use pprint in displayhook http://bugs.python.org/issue19103 opened by serhiy.storchaka #19104: pprint produces invalid output for long strings http://bugs.python.org/issue19104 opened by serhiy.storchaka #19105: pprint doesn't use all width http://bugs.python.org/issue19105 opened by serhiy.storchaka #19106: Prepare .hgtouch and Tools/hg/hgtouch.py to run on the bots http://bugs.python.org/issue19106 opened by eli.bendersky #19107: CSV output of benchmark runner tries to write to binary file http://bugs.python.org/issue19107 opened by scoder #19108: Benchmark runner tries to execute external Python command and http://bugs.python.org/issue19108 opened by scoder Most recent 15 issues with no replies (15) ========================================== #19108: Benchmark runner tries to execute external Python command and http://bugs.python.org/issue19108 #19107: CSV output of benchmark runner tries to write to binary file http://bugs.python.org/issue19107 #19105: pprint doesn't use all width http://bugs.python.org/issue19105 #19102: Add tests for CLI of the tabnanny module http://bugs.python.org/issue19102 #19100: Use backslashreplace in pprint http://bugs.python.org/issue19100 #19099: struct.pack fails first time with unicode fmt http://bugs.python.org/issue19099 #19086: Make fsum usable incrementally. http://bugs.python.org/issue19086 #19085: Add tkinter basic options tests http://bugs.python.org/issue19085 #19080: Enrich SyntaxError with additional information http://bugs.python.org/issue19080 #19076: Pdb.do_break calls error with obsolete file kwarg http://bugs.python.org/issue19076 #19075: Add sorting algorithm visualization to turtledemo http://bugs.python.org/issue19075 #19072: classmethod doesn't honour descriptor protocol of wrapped call http://bugs.python.org/issue19072 #19071: Documentation on what self is for module-level functions is mi http://bugs.python.org/issue19071 #19069: Built-in float docstrings are not PEP-8 compatible http://bugs.python.org/issue19069 #19068: Some built-in complex docstrings are not PEP-8 compatible http://bugs.python.org/issue19068 Most recent 15 issues waiting for review (15) ============================================= #19104: pprint produces invalid output for long strings http://bugs.python.org/issue19104 #19103: Use pprint in displayhook http://bugs.python.org/issue19103 #19102: Add tests for CLI of the tabnanny module http://bugs.python.org/issue19102 #19100: Use backslashreplace in pprint http://bugs.python.org/issue19100 #19094: urljoin should raise a TypeError if URL is not a string http://bugs.python.org/issue19094 #19092: ExitStack.__exit__ incorrectly suppresses exceptions in __exit http://bugs.python.org/issue19092 #19087: bytearray front-slicing not optimized http://bugs.python.org/issue19087 #19085: Add tkinter basic options tests http://bugs.python.org/issue19085 #19083: IDNA prefix should be case insensitive http://bugs.python.org/issue19083 #19082: Lib/xmlrpc/client.py demo code points to the dead server http://bugs.python.org/issue19082 #19081: zipimport behaves badly when the zip file changes while the pr http://bugs.python.org/issue19081 #19078: Allow reversed(memoryview), like memoryview http://bugs.python.org/issue19078 #19077: More robust TemporaryDirectory cleanup http://bugs.python.org/issue19077 #19075: Add sorting algorithm visualization to turtledemo http://bugs.python.org/issue19075 #19072: classmethod doesn't honour descriptor protocol of wrapped call http://bugs.python.org/issue19072 Top 10 most discussed issues (10) ================================= #12085: subprocess.Popen.__del__ raises AttributeError if __init__ was http://bugs.python.org/issue12085 11 msgs #16038: ftplib: unlimited readline() from connection http://bugs.python.org/issue16038 11 msgs #18990: Return root element from ElementTree.XMLPullParser.close() to http://bugs.python.org/issue18990 11 msgs #19009: Enhance HTTPResponse.readline() performance http://bugs.python.org/issue19009 10 msgs #19089: Windows: Make Ctrl-D exit key combination cross-platform http://bugs.python.org/issue19089 10 msgs #19087: bytearray front-slicing not optimized http://bugs.python.org/issue19087 9 msgs #11380: Improve reporting of broken stdout pipe during interpreter shu http://bugs.python.org/issue11380 7 msgs #19021: AttributeError in Popen.__del__ http://bugs.python.org/issue19021 7 msgs #14983: email.generator should always add newlines after closing bound http://bugs.python.org/issue14983 6 msgs #19024: Document asterisk (*), splat or star operator http://bugs.python.org/issue19024 6 msgs Issues closed (22) ================== #17473: -m is not universally applicable http://bugs.python.org/issue17473 closed by ncoghlan #17777: Unrecognized string literal escape sequences give SyntaxErrors http://bugs.python.org/issue17777 closed by tim.golden #18050: embedded interpreter or virtualenv fails with "ImportError: ca http://bugs.python.org/issue18050 closed by serhiy.storchaka #18626: Make "python -m inspect <name>" meaningful http://bugs.python.org/issue18626 closed by python-dev #18978: Allow urllib.request.Request subclasses to override method http://bugs.python.org/issue18978 closed by python-dev #18996: unittest: more helpful truncating long strings http://bugs.python.org/issue18996 closed by serhiy.storchaka #19025: Deleting attribute of Enum gives misleading error message http://bugs.python.org/issue19025 closed by python-dev #19028: tkinter.tkapp.merge() fails on non-strings http://bugs.python.org/issue19028 closed by serhiy.storchaka #19030: inspect.getmembers and inspect.classify_class_attrs mishandle http://bugs.python.org/issue19030 closed by python-dev #19034: More useful repr for Tcl_Obj http://bugs.python.org/issue19034 closed by serhiy.storchaka #19043: Remove detailed listing of all versions from LICENSE, Doc/lice http://bugs.python.org/issue19043 closed by python-dev #19044: getaddrinfo raises near-useless exception http://bugs.python.org/issue19044 closed by neologix #19047: Assorted weakref docs improvements http://bugs.python.org/issue19047 closed by python-dev #19049: itertools.tee uses int for indices http://bugs.python.org/issue19049 closed by pitrou #19054: Descriptors howto http://bugs.python.org/issue19054 closed by rhettinger #19056: Windows 7, script exec not working without explicit cal of pyt http://bugs.python.org/issue19056 closed by eric.smith #19059: test_posix failure on OS X snow leopord buildbot http://bugs.python.org/issue19059 closed by ned.deily #19061: Shelve documentation security warning is not visible http://bugs.python.org/issue19061 closed by python-dev #19064: can't run py3 benchmarks http://bugs.python.org/issue19064 closed by pitrou #19093: Spam http://bugs.python.org/issue19093 closed by berker.peksag #19098: sys.setrecursionlimit(2**30) breaks interactive shell http://bugs.python.org/issue19098 closed by python-dev #19101: "make touch" broke "x86 Ubuntu Shared 3.x" buildbot http://bugs.python.org/issue19101 closed by benjamin.peterson From rdmurray at bitdance.com Fri Sep 27 18:15:31 2013 From: rdmurray at bitdance.com (R. David Murray) Date: Fri, 27 Sep 2013 12:15:31 -0400 Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for pronouncement? In-Reply-To: <CACac1F-soYfp5Bg0sq+zgL6he301Vn7j2nRyTSodXoHBi+3K8Q@mail.gmail.com> References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com> <20130925165022.3d67e290@anarchist> <109B26FC-32A9-4DDE-9639-E48A0389455D@stufft.io> <20130925175129.1edc4348@anarchist> <55AAF039-CCAB-467F-8FA3-74D31CB80A66@stufft.io> <20130925190410.0a5729f9@anarchist> <7FBFFC39-A0AF-47FD-9715-944D13212D10@stufft.io> <87bo3g1h9i.fsf@uwakimon.sk.tsukuba.ac.jp> <CADiSq7dJYDJwok1iccJ2mW9Axi_1JOYZ8rzDk50DxGpWDk4kwA@mail.gmail.com> <8761tm1hni.fsf@uwakimon.sk.tsukuba.ac.jp> <CACac1F-soYfp5Bg0sq+zgL6he301Vn7j2nRyTSodXoHBi+3K8Q@mail.gmail.com> Message-ID: <20130927161531.87256250948@webabinitio.net> On Fri, 27 Sep 2013 15:26:41 +0100, Paul Moore <p.f.moore at gmail.com> wrote: > On 27 September 2013 15:08, Stephen J. Turnbull <stephen at xemacs.org> wrote: > >> New users on Windows and Mac OS X. I've heard many more complaints > > > from folks running tutorials about the pip bootstrapping process than > > > I ever have from the community at large about the GIL :P > > > > I bet those users are *not* running third-party distros, but rather > > are sitting in front of pretty close to plain vanilla factory installs > > of the OS, no? And "new users" on Mac OS X already have "old installs" > > of Python, no? > > Windows users who don't use a third-party distro like Enthought, > generally download the python.org installer. At the moment, that > doesn't give them a "pip" command. So if they want to install any > third party package, they have to start by installing pip. The > instructions for that are reasonably clear, but non-trivial, largely > because tools like curl are not commonly available on Windows, and by > default running a Python script may not do what you expect. > > Rather than try to fix these problems (which are *hard*) the intent is > to have the pip command installed by the python.org installer. [...] > I can't speak for Linux distros or OSX users, but for Windows I do > believe that this is a significant improvement, and worth the (IMO, > negligible) risk involved in adding this to a maintenance release. I'm not an OS X user, and probably most people on this list use macports or something similar, which essentially puts them in the same boat as the linux users...and there's a section in the PEP about that (that's where the message about installing pip if you run pip and the distro didn't include it with python is supposed to come from). For OS X users *not* using something like macports, I'm pretty sure they are going to be in a similar boat to the Windows users, with just a touch of added confusion coming from the fact that an older version of Python is already installed. But the instructions they will find on the web for installing package X (once this change hits the field) will be to install the newest version of 2.7 (or 3) using the python.org installer, and then they will have the pip command and can go from there. --David From dholth at gmail.com Fri Sep 27 19:25:40 2013 From: dholth at gmail.com (Daniel Holth) Date: Fri, 27 Sep 2013 13:25:40 -0400 Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for pronouncement? In-Reply-To: <20130927161531.87256250948@webabinitio.net> References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com> <20130925165022.3d67e290@anarchist> <109B26FC-32A9-4DDE-9639-E48A0389455D@stufft.io> <20130925175129.1edc4348@anarchist> <55AAF039-CCAB-467F-8FA3-74D31CB80A66@stufft.io> <20130925190410.0a5729f9@anarchist> <7FBFFC39-A0AF-47FD-9715-944D13212D10@stufft.io> <87bo3g1h9i.fsf@uwakimon.sk.tsukuba.ac.jp> <CADiSq7dJYDJwok1iccJ2mW9Axi_1JOYZ8rzDk50DxGpWDk4kwA@mail.gmail.com> <8761tm1hni.fsf@uwakimon.sk.tsukuba.ac.jp> <CACac1F-soYfp5Bg0sq+zgL6he301Vn7j2nRyTSodXoHBi+3K8Q@mail.gmail.com> <20130927161531.87256250948@webabinitio.net> Message-ID: <CAG8k2+56OHiCHaZFLKn2naMcpkR6rEjcRZcokXmb288rW9JWXw@mail.gmail.com> On Fri, Sep 27, 2013 at 12:15 PM, R. David Murray <rdmurray at bitdance.com> wrote: > On Fri, 27 Sep 2013 15:26:41 +0100, Paul Moore <p.f.moore at gmail.com> wrote: >> On 27 September 2013 15:08, Stephen J. Turnbull <stephen at xemacs.org> wrote: >> >> New users on Windows and Mac OS X. I've heard many more complaints >> > > from folks running tutorials about the pip bootstrapping process than >> > > I ever have from the community at large about the GIL :P >> > >> > I bet those users are *not* running third-party distros, but rather >> > are sitting in front of pretty close to plain vanilla factory installs >> > of the OS, no? And "new users" on Mac OS X already have "old installs" >> > of Python, no? >> >> Windows users who don't use a third-party distro like Enthought, >> generally download the python.org installer. At the moment, that >> doesn't give them a "pip" command. So if they want to install any >> third party package, they have to start by installing pip. The >> instructions for that are reasonably clear, but non-trivial, largely >> because tools like curl are not commonly available on Windows, and by >> default running a Python script may not do what you expect. >> >> Rather than try to fix these problems (which are *hard*) the intent is >> to have the pip command installed by the python.org installer. > [...] >> I can't speak for Linux distros or OSX users, but for Windows I do >> believe that this is a significant improvement, and worth the (IMO, >> negligible) risk involved in adding this to a maintenance release. > > I'm not an OS X user, and probably most people on this list use macports > or something similar, which essentially puts them in the same boat as > the linux users...and there's a section in the PEP about that (that's > where the message about installing pip if you run pip and the distro > didn't include it with python is supposed to come from). > > For OS X users *not* using something like macports, I'm pretty sure they > are going to be in a similar boat to the Windows users, with just a touch > of added confusion coming from the fact that an older version of Python > is already installed. But the instructions they will find on the web > for installing package X (once this change hits the field) will be to > install the newest version of 2.7 (or 3) using the python.org installer, > and then they will have the pip command and can go from there. > > --David OS X and Linux change Python in ways that can be confusing to new and experienced users, like not installing the stdlib source code by default, or not installing the profiler, or simply by being out of date. On these platforms attempting to use the system Python for development can be a costly mistake instead of a convenience. Users get a more consistent experience by starting with the installers from python.org. Hopefully the tutorials will reflect the consistency added by this PEP. This PEP only gets pip, an operation which is a recurring inconvenience for me even though I use Linux. It does not solve any of the problems you may have after pip has been installed. From stephen at xemacs.org Fri Sep 27 20:14:10 2013 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sat, 28 Sep 2013 03:14:10 +0900 Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for pronouncement? In-Reply-To: <CACac1F-soYfp5Bg0sq+zgL6he301Vn7j2nRyTSodXoHBi+3K8Q@mail.gmail.com> References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com> <20130925165022.3d67e290@anarchist> <109B26FC-32A9-4DDE-9639-E48A0389455D@stufft.io> <20130925175129.1edc4348@anarchist> <55AAF039-CCAB-467F-8FA3-74D31CB80A66@stufft.io> <20130925190410.0a5729f9@anarchist> <7FBFFC39-A0AF-47FD-9715-944D13212D10@stufft.io> <87bo3g1h9i.fsf@uwakimon.sk.tsukuba.ac.jp> <CADiSq7dJYDJwok1iccJ2mW9Axi_1JOYZ8rzDk50DxGpWDk4kwA@mail.gmail.com> <8761tm1hni.fsf@uwakimon.sk.tsukuba.ac.jp> <CACac1F-soYfp5Bg0sq+zgL6he301Vn7j2nRyTSodXoHBi+3K8Q@mail.gmail.com> Message-ID: <871u4a16a5.fsf@uwakimon.sk.tsukuba.ac.jp> Paul Moore writes: > I can't speak for Linux distros or OSX users, but for Windows I do > believe that this is a significant improvement, Nobody doubts this. > and worth the (IMO, negligible) risk involved in adding this to a > maintenance release. Doesn't that argument apply equally well to any new feature? The proponents of such changes always characterize them as "worth the (IMO, negligible) risk involved". I am sure that the proponents of any such feature are equally sincere about that! Also, I think that proponents of backporting this PEP are missing something important. Specifically, why are we encouraging the use of Python 2.7 for "new users"? Shouldn't we use this as an opportunity to say, "Move to Python 3.4 and forget the 'agony of installing pip' issue forever?" From raymond.hettinger at gmail.com Fri Sep 27 20:15:10 2013 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Fri, 27 Sep 2013 14:15:10 -0400 Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for pronouncement? In-Reply-To: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com> References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com> Message-ID: <3DCB2AFA-0971-47E2-9930-B749B8D2BDCD@gmail.com> On Sep 23, 2013, at 7:15 AM, Nick Coghlan <ncoghlan at gmail.com> wrote: > With the last round of updates, I believe PEP 453 is ready for > Martin's pronouncement. Personally, I'm very excited and happy that this or something like it is coming close to fruition. My experiences in userland suggest that this will be a big step forward for Python usability. Raymond -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130927/1ca8aa25/attachment.html> From donald at stufft.io Fri Sep 27 20:22:50 2013 From: donald at stufft.io (Donald Stufft) Date: Fri, 27 Sep 2013 14:22:50 -0400 Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for pronouncement? In-Reply-To: <871u4a16a5.fsf@uwakimon.sk.tsukuba.ac.jp> References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com> <20130925165022.3d67e290@anarchist> <109B26FC-32A9-4DDE-9639-E48A0389455D@stufft.io> <20130925175129.1edc4348@anarchist> <55AAF039-CCAB-467F-8FA3-74D31CB80A66@stufft.io> <20130925190410.0a5729f9@anarchist> <7FBFFC39-A0AF-47FD-9715-944D13212D10@stufft.io> <87bo3g1h9i.fsf@uwakimon.sk.tsukuba.ac.jp> <CADiSq7dJYDJwok1iccJ2mW9Axi_1JOYZ8rzDk50DxGpWDk4kwA@mail.gmail.com> <8761tm1hni.fsf@uwakimon.sk.tsukuba.ac.jp> <CACac1F-soYfp5Bg0sq+zgL6he301Vn7j2nRyTSodXoHBi+3K8Q@mail.gmail.com> <871u4a16a5.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <14A88ACC-9868-4A80-ABB7-99AEC9EE96F7@stufft.io> On Sep 27, 2013, at 2:14 PM, "Stephen J. Turnbull" <stephen at xemacs.org> wrote: > Also, I think that proponents of backporting this PEP are missing > something important. Specifically, why are we encouraging the use of > Python 2.7 for "new users"? Shouldn't we use this as an opportunity > to say, "Move to Python 3.4 and forget the 'agony of installing pip' > issue forever?" Because reality is that new users are still likely to be using Python 2.7. Python 3 is just now starting to be really usable, however there's a huge corpus of existing tutorials, course work, books etc for Python 2.7. As Python 3 becomes more usable that existing corpus of material will be ported over to Python 3 but in the interim there is still a pretty large hurdle for new users to get over. That's assuming that they are even able to use Python 3 and whatever they are trying to do with all of their libraries are ported to Python3. I still think Python 2.7 is a better target for new users because if you're using Python 3.x theirs a high chance you'll need to port a library or two still. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: <http://mail.python.org/pipermail/python-dev/attachments/20130927/00c035b6/attachment.sig> From guido at python.org Fri Sep 27 20:31:32 2013 From: guido at python.org (Guido van Rossum) Date: Fri, 27 Sep 2013 11:31:32 -0700 Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for pronouncement? In-Reply-To: <14A88ACC-9868-4A80-ABB7-99AEC9EE96F7@stufft.io> References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com> <20130925165022.3d67e290@anarchist> <109B26FC-32A9-4DDE-9639-E48A0389455D@stufft.io> <20130925175129.1edc4348@anarchist> <55AAF039-CCAB-467F-8FA3-74D31CB80A66@stufft.io> <20130925190410.0a5729f9@anarchist> <7FBFFC39-A0AF-47FD-9715-944D13212D10@stufft.io> <87bo3g1h9i.fsf@uwakimon.sk.tsukuba.ac.jp> <CADiSq7dJYDJwok1iccJ2mW9Axi_1JOYZ8rzDk50DxGpWDk4kwA@mail.gmail.com> <8761tm1hni.fsf@uwakimon.sk.tsukuba.ac.jp> <CACac1F-soYfp5Bg0sq+zgL6he301Vn7j2nRyTSodXoHBi+3K8Q@mail.gmail.com> <871u4a16a5.fsf@uwakimon.sk.tsukuba.ac.jp> <14A88ACC-9868-4A80-ABB7-99AEC9EE96F7@stufft.io> Message-ID: <CAP7+vJJetVNPuNZfR+VEPUAbsUYQZNiT0wZ3W8V9MHY2xp6+ug@mail.gmail.com> On Fri, Sep 27, 2013 at 11:22 AM, Donald Stufft <donald at stufft.io> wrote: > > On Sep 27, 2013, at 2:14 PM, "Stephen J. Turnbull" <stephen at xemacs.org> > wrote: > > > Also, I think that proponents of backporting this PEP are missing > > something important. Specifically, why are we encouraging the use of > > Python 2.7 for "new users"? Shouldn't we use this as an opportunity > > to say, "Move to Python 3.4 and forget the 'agony of installing pip' > > issue forever?" > > Because reality is that new users are still likely to be using Python 2.7. > Python 3 > is just now starting to be really usable, however there's a huge corpus of > existing > tutorials, course work, books etc for Python 2.7. As Python 3 becomes more > usable > that existing corpus of material will be ported over to Python 3 but in > the interim > there is still a pretty large hurdle for new users to get over. > > That's assuming that they are even able to use Python 3 and whatever they > are trying > to do with all of their libraries are ported to Python3. I still think > Python 2.7 is a better > target for new users because if you're using Python 3.x theirs a high > chance you'll > need to port a library or two still. > Based on my day-to-day experience this is still very true. (And yes, I'm slowly turning the tide. But it will take a long time and I am committed to giving users the choice.) -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130927/b2fafc1b/attachment-0001.html> From tjreedy at udel.edu Fri Sep 27 20:50:22 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 27 Sep 2013 14:50:22 -0400 Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for pronouncement? In-Reply-To: <CACac1F-soYfp5Bg0sq+zgL6he301Vn7j2nRyTSodXoHBi+3K8Q@mail.gmail.com> References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com> <20130925165022.3d67e290@anarchist> <109B26FC-32A9-4DDE-9639-E48A0389455D@stufft.io> <20130925175129.1edc4348@anarchist> <55AAF039-CCAB-467F-8FA3-74D31CB80A66@stufft.io> <20130925190410.0a5729f9@anarchist> <7FBFFC39-A0AF-47FD-9715-944D13212D10@stufft.io> <87bo3g1h9i.fsf@uwakimon.sk.tsukuba.ac.jp> <CADiSq7dJYDJwok1iccJ2mW9Axi_1JOYZ8rzDk50DxGpWDk4kwA@mail.gmail.com> <8761tm1hni.fsf@uwakimon.sk.tsukuba.ac.jp> <CACac1F-soYfp5Bg0sq+zgL6he301Vn7j2nRyTSodXoHBi+3K8Q@mail.gmail.com> Message-ID: <l24k17$efa$1@ger.gmane.org> On 9/27/2013 10:26 AM, Paul Moore wrote: [snip longish post] I want to summarize the main points of what I believe Paul said and strongly agree with them. * For this issue, and especially for backporting to 2.7/3.3, we should consider Windows, Mac, and *nix distributions as separate cases. * For Windows users, it would be really good if the PSF installer for the next releases of all versions at least optionally resulted in the installation of the most recent pip possible. I add: just how that is accomplished is not the users concern, except possibly for using the net to check for a later version of pip. Also, we can encourage but not command Enthought and ActiveState to do the same suite in their plus distributions. * For 2.7/3.3, Windows users have no need to run _ensurepip after installation. I add: for 2.7/3.3, there is consequently no need for _ensurepip to be in /Lib after installation, even if temporarily added*. If it is not there, there is no change the the stdlib, and hence no violation of the 'no new features' policy. The optional installation of pip is not a change to Python itself. * Where the 2.7/3.3 installers merely bundle pip or use the 3.4 procedure with an extra delete step strikes me as an implementation detail best decided by the installer maker (Martin). -- Terry Jan Reedy From donald at stufft.io Fri Sep 27 21:10:43 2013 From: donald at stufft.io (Donald Stufft) Date: Fri, 27 Sep 2013 15:10:43 -0400 Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for pronouncement? In-Reply-To: <l24k17$efa$1@ger.gmane.org> References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com> <20130925165022.3d67e290@anarchist> <109B26FC-32A9-4DDE-9639-E48A0389455D@stufft.io> <20130925175129.1edc4348@anarchist> <55AAF039-CCAB-467F-8FA3-74D31CB80A66@stufft.io> <20130925190410.0a5729f9@anarchist> <7FBFFC39-A0AF-47FD-9715-944D13212D10@stufft.io> <87bo3g1h9i.fsf@uwakimon.sk.tsukuba.ac.jp> <CADiSq7dJYDJwok1iccJ2mW9Axi_1JOYZ8rzDk50DxGpWDk4kwA@mail.gmail.com> <8761tm1hni.fsf@uwakimon.sk.tsukuba.ac.jp> <CACac1F-soYfp5Bg0sq+zgL6he301Vn7j2nRyTSodXoHBi+3K8Q@mail.gmail.com> <l24k17$efa$1@ger.gmane.org> Message-ID: <2E8C3B8B-808C-40CA-9914-D70E6627AA3F@stufft.io> On Sep 27, 2013, at 2:50 PM, Terry Reedy <tjreedy at udel.edu> wrote: > I add: for 2.7/3.3, there is consequently no need for _ensurepip to be in /Lib after installation, even if temporarily added*. If it is not there, there is no change the the stdlib, and hence no violation of the 'no new features' policy. The optional installation of pip is not a change to Python itself. This sounds like a really bad idea to me. You're going to end up with a different stdlib not only by minor release, but by if they installed through an installer or not. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: <http://mail.python.org/pipermail/python-dev/attachments/20130927/1c7a9ce8/attachment.sig> From tjreedy at udel.edu Fri Sep 27 22:09:01 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 27 Sep 2013 16:09:01 -0400 Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for pronouncement? In-Reply-To: <2E8C3B8B-808C-40CA-9914-D70E6627AA3F@stufft.io> References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com> <20130925165022.3d67e290@anarchist> <109B26FC-32A9-4DDE-9639-E48A0389455D@stufft.io> <20130925175129.1edc4348@anarchist> <55AAF039-CCAB-467F-8FA3-74D31CB80A66@stufft.io> <20130925190410.0a5729f9@anarchist> <7FBFFC39-A0AF-47FD-9715-944D13212D10@stufft.io> <87bo3g1h9i.fsf@uwakimon.sk.tsukuba.ac.jp> <CADiSq7dJYDJwok1iccJ2mW9Axi_1JOYZ8rzDk50DxGpWDk4kwA@mail.gmail.com> <8761tm1hni.fsf@uwakimon.sk.tsukuba.ac.jp> <CACac1F-soYfp5Bg0sq+zgL6he301Vn7j2nRyTSodXoHBi+3K8Q@mail.gmail.com> <l24k17$efa$1@ger.gmane.org> <2E8C3B8B-808C-40CA-9914-D70E6627AA3F@stufft.io> Message-ID: <l24okm$5lf$1@ger.gmane.org> On 9/27/2013 3:10 PM, Donald Stufft wrote: > > On Sep 27, 2013, at 2:50 PM, Terry Reedy <tjreedy at udel.edu> wrote: > >> I add: for 2.7/3.3, there is consequently no need for _ensurepip to be in /Lib after installation, even if temporarily added*. If it is not there, there is no change the the stdlib, and hence no violation of the 'no new features' policy. The optional installation of pip is not a change to Python itself. > > This sounds like a really bad idea to me. Why would you think that an idea aimed at ending an argument blocking your proposal is bad. This seems like a really bad response to me. > You're going to end up with a different stdlib not only by minor release, but by if they installed through an installer or not. The current proposal is to add a new module to the stdlib in a bugfix release, which looks like a violation of current policy. We agree that that end result of pip installed would be good. We are arguing over whether adding '_' to the name makes it not a violation or whether the good outweighs the bad of a violation. I claim that the arguement is not necessary and can be ended by not making the addition or by hiding it. I presume your objection refers to the fact that one can clone the repository and compile Python on Windows, albeit with some difficulty. My three responses: 1. I do not consider the the result to be 'installed Python', at least not as I have used the project file. 2. The ratio of people building Python on Windows to those downloading and running an installer is so close to 0 that it can be ignored. People who build Python on Windows are not typical Python beginners. 3. If you do not agree with 1 and 2 and object to _ensurepip being in /Lib in such limited circumstances, then either put it in /Tools/scripts or do not use it at all. I already said that the 2.7/3.3 Windows installer maker (Martin) should decide whether to even use it. 4. The argument for including _ensurepip somewhere in the repository it that people who *do* build python.exe could then use it to install pip the first time. /Tools/scripts would be sufficient for this. 5. The result of not having /Lib/_ensurepip in installed Python would, in any case, be a lessor violation of the policy. -- Terry Jan Reedy From donald at stufft.io Fri Sep 27 22:29:34 2013 From: donald at stufft.io (Donald Stufft) Date: Fri, 27 Sep 2013 16:29:34 -0400 Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for pronouncement? In-Reply-To: <l24okm$5lf$1@ger.gmane.org> References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com> <20130925165022.3d67e290@anarchist> <109B26FC-32A9-4DDE-9639-E48A0389455D@stufft.io> <20130925175129.1edc4348@anarchist> <55AAF039-CCAB-467F-8FA3-74D31CB80A66@stufft.io> <20130925190410.0a5729f9@anarchist> <7FBFFC39-A0AF-47FD-9715-944D13212D10@stufft.io> <87bo3g1h9i.fsf@uwakimon.sk.tsukuba.ac.jp> <CADiSq7dJYDJwok1iccJ2mW9Axi_1JOYZ8rzDk50DxGpWDk4kwA@mail.gmail.com> <8761tm1hni.fsf@uwakimon.sk.tsukuba.ac.jp> <CACac1F-soYfp5Bg0sq+zgL6he301Vn7j2nRyTSodXoHBi+3K8Q@mail.gmail.com> <l24k17$efa$1@ger.gmane.org> <2E8C3B8B-808C-40CA-9914-D70E6627AA3F@stufft.io> <l24okm$5lf$1@ger.gmane.org> Message-ID: <7A7A9106-0668-4E26-A132-23F59296E5B3@stufft.io> On Sep 27, 2013, at 4:09 PM, Terry Reedy <tjreedy at udel.edu> wrote: > On 9/27/2013 3:10 PM, Donald Stufft wrote: >> >> On Sep 27, 2013, at 2:50 PM, Terry Reedy <tjreedy at udel.edu> wrote: >> >>> I add: for 2.7/3.3, there is consequently no need for _ensurepip to be in /Lib after installation, even if temporarily added*. If it is not there, there is no change the the stdlib, and hence no violation of the 'no new features' policy. The optional installation of pip is not a change to Python itself. >> >> This sounds like a really bad idea to me. > > Why would you think that an idea aimed at ending an argument blocking your proposal is bad. This seems like a really bad response to me. Because I think it's a fundamentally bad idea. > >> You're going to end up with a different stdlib not only by minor release, but by if they installed through an installer or not. > > The current proposal is to add a new module to the stdlib in a bugfix release, which looks like a violation of current policy. We agree that that end result of pip installed would be good. We are arguing over whether adding '_' to the name makes it not a violation or whether the good outweighs the bad of a violation. I claim that the arguement is not necessary and can be ended by not making the addition or by hiding it. > > I presume your objection refers to the fact that one can clone the repository and compile Python on Windows, albeit with some difficulty. My three responses: > > 1. I do not consider the the result to be 'installed Python', at least not as I have used the project file. > > 2. The ratio of people building Python on Windows to those downloading and running an installer is so close to 0 that it can be ignored. People who build Python on Windows are not typical Python beginners. > > 3. If you do not agree with 1 and 2 and object to _ensurepip being in /Lib in such limited circumstances, then either put it in /Tools/scripts or do not use it at all. I already said that the 2.7/3.3 Windows installer maker (Martin) should decide whether to even use it. > > 4. The argument for including _ensurepip somewhere in the repository it that people who *do* build python.exe could then use it to install pip the first time. /Tools/scripts would be sufficient for this. > > 5. The result of not having /Lib/_ensurepip in installed Python would, in any case, be a lessor violation of the policy. If it lives in the source tree how are you going to provent it from existing when someone installs on Linux? OSX? One of the BSDs? It seems to me your proposal is to add the _ensurepip module? except when they install it via Windows installer. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: <http://mail.python.org/pipermail/python-dev/attachments/20130927/a0c00446/attachment.sig> From zachary.ware+pydev at gmail.com Fri Sep 27 23:16:50 2013 From: zachary.ware+pydev at gmail.com (Zachary Ware) Date: Fri, 27 Sep 2013 16:16:50 -0500 Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for pronouncement? In-Reply-To: <7A7A9106-0668-4E26-A132-23F59296E5B3@stufft.io> References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com> <20130925165022.3d67e290@anarchist> <109B26FC-32A9-4DDE-9639-E48A0389455D@stufft.io> <20130925175129.1edc4348@anarchist> <55AAF039-CCAB-467F-8FA3-74D31CB80A66@stufft.io> <20130925190410.0a5729f9@anarchist> <7FBFFC39-A0AF-47FD-9715-944D13212D10@stufft.io> <87bo3g1h9i.fsf@uwakimon.sk.tsukuba.ac.jp> <CADiSq7dJYDJwok1iccJ2mW9Axi_1JOYZ8rzDk50DxGpWDk4kwA@mail.gmail.com> <8761tm1hni.fsf@uwakimon.sk.tsukuba.ac.jp> <CACac1F-soYfp5Bg0sq+zgL6he301Vn7j2nRyTSodXoHBi+3K8Q@mail.gmail.com> <l24k17$efa$1@ger.gmane.org> <2E8C3B8B-808C-40CA-9914-D70E6627AA3F@stufft.io> <l24okm$5lf$1@ger.gmane.org> <7A7A9106-0668-4E26-A132-23F59296E5B3@stufft.io> Message-ID: <CAKJDb-M6LqJyoWaPoquy0apoLJ+Ffr9irNBudrX0F10KwoqmbQ@mail.gmail.com> On Fri, Sep 27, 2013 at 3:29 PM, Donald Stufft <donald at stufft.io> wrote: > <snip> > > If it lives in the source tree how are you going to provent it from existing when someone installs on Linux? OSX? One of the BSDs? If someone is building their own Python from source--regardless of platform--they're obviously going to have _ensurepip available one way or another, and such users will need to have it available to match the capabilities of the installer (or create their own). But if you're building your own Python, you should already know enough about what's going on to know when and whether _ensurepip should be used and how. On the other hand, when you use an installer (be it Windows, OSX, or otherwise), you really only need _ensurepip one time, ever: at install time. Even then, you don't actually need to know anything about _ensurepip at all, just whether to check the box or not. If you decide you need it later, you can always re-install. > It seems to me your [Terry's] proposal is to add the _ensurepip module? except when they install it via Windows installer. The way I read Terry's proposal, it is to never add the _ensurepip *module*, but to use (or make available, whichever makes sense in a given case) the _ensurepip *script* when it is requested at install-time: specifically, put _ensurepip.py in Tools/scripts, PC/, Mac/, or really anywhere but Lib/ so that builders can find it, but Python can't. In other words, don't make "import _ensurepip" possible out of the box, and a lot of the "don't add new features in maintenance releases" blockade disappears, because you aren't actually adding any new capability to Python itself or its standard library. Of course, this proposal only applies to 2.7/3.3. Since _ensurepip will be used for venv as well as initial install in 3.4, it should be a full-blown stdlib module in that version, probably even without the leading underscore. -- Zach From jnoller at gmail.com Fri Sep 27 23:41:30 2013 From: jnoller at gmail.com (Jesse Noller) Date: Fri, 27 Sep 2013 16:41:30 -0500 Subject: [Python-Dev] Official github mirror for CPython? In-Reply-To: <CAF-Rda8oajAZUf-AaVnHi4q8h9JSSyTRRsNMar8iUf5-opXF_Q@mail.gmail.com> References: <CAF-Rda-=wFp_pWSa2gqFxet1M7pf68si=LKZ7FyGU7UrfRS3FA@mail.gmail.com> <51F1380E.9070608@python.org> <CAD+XWwrCD_tLM0b1OFea5ZZq7oH+1TYR_yNyp=mNDKJe7ZBALQ@mail.gmail.com> <CAF-Rda8oajAZUf-AaVnHi4q8h9JSSyTRRsNMar8iUf5-opXF_Q@mail.gmail.com> Message-ID: <CACQrdOnmuN6UupMuptKqL7HHmrMCBRoQLLDVK18=_NKadq-gPA@mail.gmail.com> Email me the name of the repo you want, and your github username (preferably off list so I don't miss it!) and I will set you up Eli On Fri, Sep 27, 2013 at 11:02 AM, Eli Bendersky <eliben at gmail.com> wrote: > > > > On Thu, Jul 25, 2013 at 7:48 AM, Brian Curtin <brian at python.org> wrote: >> >> On Thu, Jul 25, 2013 at 9:37 AM, Christian Heimes <christian at python.org> >> wrote: >> > Am 25.07.2013 16:29, schrieb Eli Bendersky: >> >> Hi all, >> >> >> >> I've been looking for a Github mirror for Python, and found two: >> >> >> >> * https://github.com/python-git/python has a lot of >> >> forks/watches/starts >> >> but seems to be very out of date (last updated 4 years ago) >> >> * https://github.com/python-mirror/python doesn't appear to be very >> >> popular but is updated daily >> >> >> >> Are some of you the owners of these repositories? Should we consolidate >> >> to a single "semi-official" mirror? >> > >> > +1 >> > >> > Does the PSF have an official account on github? We have one on >> > bitbucket... >> >> I don't remember who runs this, and I thought I was in it (maybe just >> on BB), but: https://github.com/python > > > The list of members for https://github.com/python is > https://github.com/python?tab=members > > How can I get added to that list? When that happens, I can try to set-up a > regularly updated CPython mirror. > > Eli > > > > From guido at python.org Sat Sep 28 00:33:27 2013 From: guido at python.org (Guido van Rossum) Date: Fri, 27 Sep 2013 15:33:27 -0700 Subject: [Python-Dev] Getting Tulip (PEP 3156) into the 3.4 stdlib, marked provisional, named asyncio Message-ID: <CAP7+vJKSOHYeCqhsJVF_7epTDeOw4UP3crsK_mY7HaFYnwG1aA@mail.gmail.com> I've been looking at my progress with Tulip and the 3.4 release schedule (PEP 429) and it looks like I will have to do some kind of sprint to get it into the release in time for beta 1, which is planned for Nov 24. Ideally I'd get it into alpha 4, which is scheduled for Oct 20 -- that's in about three weeks, and probably too tight. Even Nov 24 is aggressive, because the PEP (PEP 3156) hasn't even been discussed formally, let alone accepted! Fortunately I'm pretty happy with most of the APIs defined in the PEP -- there are some holes but I expect no big changes at this point. It's clear that the only way we can get this into the stdlib is to mark it as provisional, per PEP 411. That's fine with me. I expect that the kind of changes the code will see between the 3.4 and 3.5 releases are well within the bounds of the expectations set by that PEP -- while it is clear that everything is possible (even withdrawal), the most likely outcome is that the API will be extended or tweaked at most, and not significantly changed in a backward compatible way -- without completely ruling out that some part of the API may deserve a serious overhaul based on experience. That's exactly how I feel about Tulip. I propose that the new library will be named asyncio. It would be odd if we kept the name tulip (stdlib modules rarely have "creative" names), and "asynchronous I/O" seems to cover the contents quite well. I propose that we do the bikeshedding about the API on the python-tulip at googlegroups.com mailing list; I'm not sure what we would gain by holding the discussion on python-dev (although clearly we should report back here with important milestones in the review). I will post a message with the kick-off there. I hope people will respect my choice of venue, and limit themselves to procedural issued on python-dev. E.g. it's fine to debate on python-dev whether I'm overstepping my BDFL authority here, or whether the proposed API is of sufficiently wide appeal. But I'd prefer to discuss e.g. the return type of start_serving() or the choice of yield-from over yield on python-tulip. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130927/cd769af7/attachment.html> From barry at python.org Sat Sep 28 02:44:17 2013 From: barry at python.org (Barry Warsaw) Date: Fri, 27 Sep 2013 20:44:17 -0400 Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for pronouncement? In-Reply-To: <CADiSq7cMMkTjArg=0=rmDypHMo5aACYNQX=ct18N9jzru2=JQQ@mail.gmail.com> References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com> <20130925165022.3d67e290@anarchist> <109B26FC-32A9-4DDE-9639-E48A0389455D@stufft.io> <20130925175129.1edc4348@anarchist> <CADiSq7fVrruk1zcE-ZwHUOyUqOPhTBnn2jcp9rKQ8dS6QeL+Zw@mail.gmail.com> <20130925190854.6ac76200@anarchist> <CADiSq7cMMkTjArg=0=rmDypHMo5aACYNQX=ct18N9jzru2=JQQ@mail.gmail.com> Message-ID: <20130927204417.73058d41@anarchist> On Sep 26, 2013, at 02:30 PM, Nick Coghlan wrote: >- the module name should be "_ensurepip" in all versions >- the PEP should explicitly state that the "don't remove _ensurepip >and it's wheel files" caveat for redistributors applies only in 3.4+ >(where removing it will break pyvenv) I'm sorry that I probably won't be able to engage more on this thread, but I think my points should be fairly well understood by the PEP czar and RM. Should the PEP be accepted, I think both of these are good changes. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: <http://mail.python.org/pipermail/python-dev/attachments/20130927/97c19dfc/attachment.sig> From ncoghlan at gmail.com Sat Sep 28 02:40:30 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 28 Sep 2013 10:40:30 +1000 Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for pronouncement? In-Reply-To: <8761tm1hni.fsf@uwakimon.sk.tsukuba.ac.jp> References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com> <20130925165022.3d67e290@anarchist> <109B26FC-32A9-4DDE-9639-E48A0389455D@stufft.io> <20130925175129.1edc4348@anarchist> <55AAF039-CCAB-467F-8FA3-74D31CB80A66@stufft.io> <20130925190410.0a5729f9@anarchist> <7FBFFC39-A0AF-47FD-9715-944D13212D10@stufft.io> <87bo3g1h9i.fsf@uwakimon.sk.tsukuba.ac.jp> <CADiSq7dJYDJwok1iccJ2mW9Axi_1JOYZ8rzDk50DxGpWDk4kwA@mail.gmail.com> <8761tm1hni.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <CADiSq7d9ipdAiP8yLCsO0OGoTRLvcL1_0gEHdig=UdH_iLWFgQ@mail.gmail.com> On 28 Sep 2013 00:08, "Stephen J. Turnbull" <stephen at xemacs.org> wrote: > > Nick Coghlan writes: > > > I'm not sure what usage model you're assuming for _ensurepip, but it > > appears to be wrong. End users should be able to just run pip, and > > either have it work, or else get a message from the OS vendor telling > > them to install the appropriate system package. > > I don't understand how you arrange for that message on existing > installs. Wouldn't it be easier to just lobby the distros to make > Python dependent on pip? Most distros of interest already emit that message - it's part of a standard mechanism to check for system packages if a command isn't found (and I believe even RHEL and derivatives may support this for pip if the EPEL repo is enabled). > And speaking of vendors, do you expect Apple > and Microsoft to provide such a message? And such a system package? No, that's why the proposal is to modify the CPython installers for those platforms. > > If you already are running a Linux distro or MacPorts, you do "apt-get > python-pip" and "port install py-pip" respectively. I bet Cygwin is > the same with yet another spelling. Where's the problem? You say: > > > New users on Windows and Mac OS X. I've heard many more complaints > > from folks running tutorials about the pip bootstrapping process than > > I ever have from the community at large about the GIL :P > > I bet those users are *not* running third-party distros, but rather > are sitting in front of pretty close to plain vanilla factory installs > of the OS, no? And "new users" on Mac OS X already have "old installs" > of Python, no? No. Instuctors tell users on Mac OS X to install from python.org or use one of the third party package managers. > > That's my model. In that model I don't see backporting PEP 453 to > Python 2.7 as being a sufficiently reliable way to provide a smooth > user experience to justify breaking the "no new features" rule (which > is at the "read my lips" level after the True/False fiasco). You have confirmed my belief that your model is incorrect. The feedback I have received is that the majority of beginners are introduced to Python by downloading the binary installers from python.org for Windows or Mac OS X. > Get a commitment from Apple to put 2.7.6 in their next upgrades for > their OS, and then maybe you'd have enough leverage to tip the > balance. I certainly would concede the point. But without that, > you're telling Mac users "you have to upgrade Python from a 3rd party > site." Is that really the way to make new users participating in a > tutorial session happy? (You tell me, I'm just introspecting here.) Yes, that's exactly what happens. People don't use the system Python on Mac OS X the way they do on *nix systems. Cheers, Nick. > > Steve -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130928/8896b222/attachment-0001.html> From benjamin at python.org Sat Sep 28 03:19:57 2013 From: benjamin at python.org (Benjamin Peterson) Date: Fri, 27 Sep 2013 21:19:57 -0400 Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for pronouncement? In-Reply-To: <20130927204417.73058d41@anarchist> References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com> <20130925165022.3d67e290@anarchist> <109B26FC-32A9-4DDE-9639-E48A0389455D@stufft.io> <20130925175129.1edc4348@anarchist> <CADiSq7fVrruk1zcE-ZwHUOyUqOPhTBnn2jcp9rKQ8dS6QeL+Zw@mail.gmail.com> <20130925190854.6ac76200@anarchist> <CADiSq7cMMkTjArg=0=rmDypHMo5aACYNQX=ct18N9jzru2=JQQ@mail.gmail.com> <20130927204417.73058d41@anarchist> Message-ID: <CAPZV6o8wZ_ii=mfSJft_FeB4SaqK-L_Or6-6dsmX2B9-3XK+nA@mail.gmail.com> 2013/9/27 Barry Warsaw <barry at python.org>: > On Sep 26, 2013, at 02:30 PM, Nick Coghlan wrote: > >>- the module name should be "_ensurepip" in all versions >>- the PEP should explicitly state that the "don't remove _ensurepip >>and it's wheel files" caveat for redistributors applies only in 3.4+ >>(where removing it will break pyvenv) > > I'm sorry that I probably won't be able to engage more on this thread, but I > think my points should be fairly well understood by the PEP czar and RM. > Should the PEP be accepted, I think both of these are good changes. And FWIW, I generally agree with Barry about adding things to 2.7. -- Regards, Benjamin From brett at python.org Sat Sep 28 03:20:10 2013 From: brett at python.org (Brett Cannon) Date: Fri, 27 Sep 2013 21:20:10 -0400 Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for pronouncement? In-Reply-To: <CAKJDb-M6LqJyoWaPoquy0apoLJ+Ffr9irNBudrX0F10KwoqmbQ@mail.gmail.com> References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com> <20130925165022.3d67e290@anarchist> <109B26FC-32A9-4DDE-9639-E48A0389455D@stufft.io> <20130925175129.1edc4348@anarchist> <55AAF039-CCAB-467F-8FA3-74D31CB80A66@stufft.io> <20130925190410.0a5729f9@anarchist> <7FBFFC39-A0AF-47FD-9715-944D13212D10@stufft.io> <87bo3g1h9i.fsf@uwakimon.sk.tsukuba.ac.jp> <CADiSq7dJYDJwok1iccJ2mW9Axi_1JOYZ8rzDk50DxGpWDk4kwA@mail.gmail.com> <8761tm1hni.fsf@uwakimon.sk.tsukuba.ac.jp> <CACac1F-soYfp5Bg0sq+zgL6he301Vn7j2nRyTSodXoHBi+3K8Q@mail.gmail.com> <l24k17$efa$1@ger.gmane.org> <2E8C3B8B-808C-40CA-9914-D70E6627AA3F@stufft.io> <l24okm$5lf$1@ger.gmane.org> <7A7A9106-0668-4E26-A132-23F59296E5B3@stufft.io> <CAKJDb-M6LqJyoWaPoquy0apoLJ+Ffr9irNBudrX0F10KwoqmbQ@mail.gmail.com> Message-ID: <CAP1=2W44X6+zaQ9=-J4SdekUJ0CxWy7b8YYk=eWM=88CODbn5w@mail.gmail.com> On Fri, Sep 27, 2013 at 5:16 PM, Zachary Ware <zachary.ware+pydev at gmail.com>wrote: > On Fri, Sep 27, 2013 at 3:29 PM, Donald Stufft <donald at stufft.io> wrote: > > > <snip> > > > > If it lives in the source tree how are you going to provent it from > existing when someone installs on Linux? OSX? One of the BSDs? > > If someone is building their own Python from source--regardless of > platform--they're obviously going to have _ensurepip available one way > or another, and such users will need to have it available to match the > capabilities of the installer (or create their own). But if you're > building your own Python, you should already know enough about what's > going on to know when and whether _ensurepip should be used and how. > On the other hand, when you use an installer (be it Windows, OSX, or > otherwise), you really only need _ensurepip one time, ever: at install > time. Even then, you don't actually need to know anything about > _ensurepip at all, just whether to check the box or not. If you > decide you need it later, you can always re-install. > > > It seems to me your [Terry's] proposal is to add the _ensurepip module? > except when they install it via Windows installer. > > The way I read Terry's proposal, it is to never add the _ensurepip > *module*, but to use (or make available, whichever makes sense in a > given case) the _ensurepip *script* when it is requested at > install-time: specifically, put _ensurepip.py in Tools/scripts, PC/, > Mac/, or really anywhere but Lib/ so that builders can find it, but > Python can't. In other words, don't make "import _ensurepip" possible > out of the box, and a lot of the "don't add new features in > maintenance releases" blockade disappears, because you aren't actually > adding any new capability to Python itself or its standard library. > > Of course, this proposal only applies to 2.7/3.3. Since _ensurepip > will be used for venv as well as initial install in 3.4, it should be > a full-blown stdlib module in that version, probably even without the > leading underscore. > That's how I read the proposal as well. If that works then great, otherwise naming it _ensurepip in 2.7/3.3 should be enough to warn anyone that they are playing with fire. I think Martin, Ned, etc. would need to weigh in on whether it's at all an issue having the ensurepip script somewhere that doesn't get installed but executable from the installer is at all an issue. For source it can just be in Tools for 2.7/3.3 if that's the route chosen. Regardless, one of these two options is enough and I suspect we can stop debating and let Martin make his BDFAP decision. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130927/be2d734f/attachment.html> From brett at python.org Sat Sep 28 03:14:28 2013 From: brett at python.org (Brett Cannon) Date: Fri, 27 Sep 2013 21:14:28 -0400 Subject: [Python-Dev] Getting Tulip (PEP 3156) into the 3.4 stdlib, marked provisional, named asyncio In-Reply-To: <CAP7+vJKSOHYeCqhsJVF_7epTDeOw4UP3crsK_mY7HaFYnwG1aA@mail.gmail.com> References: <CAP7+vJKSOHYeCqhsJVF_7epTDeOw4UP3crsK_mY7HaFYnwG1aA@mail.gmail.com> Message-ID: <CAP1=2W7A2Q3t7Bk7iDZ9ZPprFdi+yr152EgVV=ACaTZkaBbUmw@mail.gmail.com> On Fri, Sep 27, 2013 at 6:33 PM, Guido van Rossum <guido at python.org> wrote: > I've been looking at my progress with Tulip and the 3.4 release schedule > (PEP 429) and it looks like I will have to do some kind of sprint to get it > into the release in time for beta 1, which is planned for Nov 24. Ideally > I'd get it into alpha 4, which is scheduled for Oct 20 -- that's in about > three weeks, and probably too tight. > > Even Nov 24 is aggressive, because the PEP (PEP 3156) hasn't even been > discussed formally, let alone accepted! Fortunately I'm pretty happy with > most of the APIs defined in the PEP -- there are some holes but I expect no > big changes at this point. > > It's clear that the only way we can get this into the stdlib is to mark it > as provisional, per PEP 411. That's fine with me. I expect that the kind of > changes the code will see between the 3.4 and 3.5 releases are well within > the bounds of the expectations set by that PEP -- while it is clear that > everything is possible (even withdrawal), the most likely outcome is that > the API will be extended or tweaked at most, and not significantly changed > in a backward compatible way -- without completely ruling out that some > part of the API may deserve a serious overhaul based on experience. That's > exactly how I feel about Tulip. > > I propose that the new library will be named asyncio. It would be odd if > we kept the name tulip (stdlib modules rarely have "creative" names), and > "asynchronous I/O" seems to cover the contents quite well. > > I propose that we do the bikeshedding about the API on the > python-tulip at googlegroups.com mailing list; I'm not sure what we would > gain by holding the discussion on python-dev (although clearly we should > report back here with important milestones in the review). I will post a > message with the kick-off there. > > I hope people will respect my choice of venue, and limit themselves to > procedural issued on python-dev. E.g. it's fine to debate on python-dev > whether I'm overstepping my BDFL authority here, or whether the proposed > API is of sufficiently wide appeal. But I'd prefer to discuss e.g. the > return type of start_serving() or the choice of yield-from over yield on > python-tulip. > I don't see any issue with redirecting the discussion. python-tulip@ is acting like a SIG for the module, so no real precedent beyond it not being hosted as a mail.python.org list. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130927/10f6e022/attachment.html> From donald at stufft.io Sat Sep 28 04:50:19 2013 From: donald at stufft.io (Donald Stufft) Date: Fri, 27 Sep 2013 22:50:19 -0400 Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for pronouncement? In-Reply-To: <CAP1=2W44X6+zaQ9=-J4SdekUJ0CxWy7b8YYk=eWM=88CODbn5w@mail.gmail.com> References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com> <20130925165022.3d67e290@anarchist> <109B26FC-32A9-4DDE-9639-E48A0389455D@stufft.io> <20130925175129.1edc4348@anarchist> <55AAF039-CCAB-467F-8FA3-74D31CB80A66@stufft.io> <20130925190410.0a5729f9@anarchist> <7FBFFC39-A0AF-47FD-9715-944D13212D10@stufft.io> <87bo3g1h9i.fsf@uwakimon.sk.tsukuba.ac.jp> <CADiSq7dJYDJwok1iccJ2mW9Axi_1JOYZ8rzDk50DxGpWDk4kwA@mail.gmail.com> <8761tm1hni.fsf@uwakimon.sk.tsukuba.ac.jp> <CACac1F-soYfp5Bg0sq+zgL6he301Vn7j2nRyTSodXoHBi+3K8Q@mail.gmail.com> <l24k17$efa$1@ger.gmane.org> <2E8C3B8B-808C-40CA-9914-D70E6627AA3F@stufft.io> <l24okm$5lf$1@ger.gmane.org> <7A7A9106-0668-4E26-A132-23F59296E5B3@stufft.io> <CAKJDb-M6LqJyoWaPoquy0apoLJ+Ffr9irNBudrX0F10KwoqmbQ@mail.gmail.com> <CAP1=2W44X6+zaQ9=-J4SdekUJ0CxWy7b8YYk=eWM=88CODbn5w@mail.gmail.com> Message-ID: <B601223B-DA26-4B50-B9B6-1AA8E394FAC3@stufft.io> On Sep 27, 2013, at 9:20 PM, Brett Cannon <brett at python.org> wrote: > > > > On Fri, Sep 27, 2013 at 5:16 PM, Zachary Ware <zachary.ware+pydev at gmail.com> wrote: > On Fri, Sep 27, 2013 at 3:29 PM, Donald Stufft <donald at stufft.io> wrote: > > > <snip> > > > > If it lives in the source tree how are you going to provent it from existing when someone installs on Linux? OSX? One of the BSDs? > > If someone is building their own Python from source--regardless of > platform--they're obviously going to have _ensurepip available one way > or another, and such users will need to have it available to match the > capabilities of the installer (or create their own). But if you're > building your own Python, you should already know enough about what's > going on to know when and whether _ensurepip should be used and how. > On the other hand, when you use an installer (be it Windows, OSX, or > otherwise), you really only need _ensurepip one time, ever: at install > time. Even then, you don't actually need to know anything about > _ensurepip at all, just whether to check the box or not. If you > decide you need it later, you can always re-install. > > > It seems to me your [Terry's] proposal is to add the _ensurepip module? except when they install it via Windows installer. > > The way I read Terry's proposal, it is to never add the _ensurepip > *module*, but to use (or make available, whichever makes sense in a > given case) the _ensurepip *script* when it is requested at > install-time: specifically, put _ensurepip.py in Tools/scripts, PC/, > Mac/, or really anywhere but Lib/ so that builders can find it, but > Python can't. In other words, don't make "import _ensurepip" possible > out of the box, and a lot of the "don't add new features in > maintenance releases" blockade disappears, because you aren't actually > adding any new capability to Python itself or its standard library. > > Of course, this proposal only applies to 2.7/3.3. Since _ensurepip > will be used for venv as well as initial install in 3.4, it should be > a full-blown stdlib module in that version, probably even without the > leading underscore. > > That's how I read the proposal as well. If that works then great, otherwise naming it _ensurepip in 2.7/3.3 should be enough to warn anyone that they are playing with fire. I think Martin, Ned, etc. would need to weigh in on whether it's at all an issue having the ensurepip script somewhere that doesn't get installed but executable from the installer is at all an issue. For source it can just be in Tools for 2.7/3.3 if that's the route chosen. > > Regardless, one of these two options is enough and I suspect we can stop debating and let Martin make his BDFAP decision. Yes, no matter which way the decision for 2.7 and 3.3 goes the proposal is for "ensurepip" in 3.4+ ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130927/ce587a9b/attachment.html> -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: <http://mail.python.org/pipermail/python-dev/attachments/20130927/ce587a9b/attachment.sig> From mordred at inaugust.com Sat Sep 28 04:55:52 2013 From: mordred at inaugust.com (Monty Taylor) Date: Fri, 27 Sep 2013 22:55:52 -0400 Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for pronouncement? In-Reply-To: <B601223B-DA26-4B50-B9B6-1AA8E394FAC3@stufft.io> References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com> <20130925165022.3d67e290@anarchist> <109B26FC-32A9-4DDE-9639-E48A0389455D@stufft.io> <20130925175129.1edc4348@anarchist> <55AAF039-CCAB-467F-8FA3-74D31CB80A66@stufft.io> <20130925190410.0a5729f9@anarchist> <7FBFFC39-A0AF-47FD-9715-944D13212D10@stufft.io> <87bo3g1h9i.fsf@uwakimon.sk.tsukuba.ac.jp> <CADiSq7dJYDJwok1iccJ2mW9Axi_1JOYZ8rzDk50DxGpWDk4kwA@mail.gmail.com> <8761tm1hni.fsf@uwakimon.sk.tsukuba.ac.jp> <CACac1F-soYfp5Bg0sq+zgL6he301Vn7j2nRyTSodXoHBi+3K8Q@mail.gmail.com> <l24k17$efa$1@ger.gmane.org> <2E8C3B8B-808C-40CA-9914-D70E6627AA3F@stufft.io> <l24okm$5lf$1@ger.gmane.org> <7A7A9106-0668-4E26-A132-23F59296E5B3@stufft.io> <CAKJDb-M6LqJyoWaPoquy0apoLJ+Ffr9irNBudrX0F10KwoqmbQ@mail.gmail.com> <CAP1=2W44X6+zaQ9=-J4SdekUJ0CxWy7b8YYk=eWM=88CODbn5w@mail.gmail.com> <B601223B-DA26-4B50-B9B6-1AA8E394FAC3@stufft.io> Message-ID: <52464538.9040306@inaugust.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 09/27/2013 10:50 PM, Donald Stufft wrote: > > On Sep 27, 2013, at 9:20 PM, Brett Cannon <brett at python.org > <mailto:brett at python.org>> wrote: > >> >> >> >> On Fri, Sep 27, 2013 at 5:16 PM, Zachary Ware >> <zachary.ware+pydev at gmail.com >> <mailto:zachary.ware+pydev at gmail.com>> wrote: >> >> On Fri, Sep 27, 2013 at 3:29 PM, Donald Stufft <donald at stufft.io >> <mailto:donald at stufft.io>> wrote: >>> >> <snip> >>> >>> If it lives in the source tree how are you going to provent it >> from existing when someone installs on Linux? OSX? One of the >> BSDs? >> >> If someone is building their own Python from source--regardless >> of platform--they're obviously going to have _ensurepip available >> one way or another, and such users will need to have it available >> to match the capabilities of the installer (or create their own). >> But if you're building your own Python, you should already know >> enough about what's going on to know when and whether _ensurepip >> should be used and how. On the other hand, when you use an >> installer (be it Windows, OSX, or otherwise), you really only >> need _ensurepip one time, ever: at install time. Even then, you >> don't actually need to know anything about _ensurepip at all, >> just whether to check the box or not. If you decide you need it >> later, you can always re-install. >> >>> It seems to me your [Terry's] proposal is to add the >>> _ensurepip >> module? except when they install it via Windows installer. >> >> The way I read Terry's proposal, it is to never add the >> _ensurepip *module*, but to use (or make available, whichever >> makes sense in a given case) the _ensurepip *script* when it is >> requested at install-time: specifically, put _ensurepip.py in >> Tools/scripts, PC/, Mac/, or really anywhere but Lib/ so that >> builders can find it, but Python can't. In other words, don't >> make "import _ensurepip" possible out of the box, and a lot of >> the "don't add new features in maintenance releases" blockade >> disappears, because you aren't actually adding any new capability >> to Python itself or its standard library. >> >> Of course, this proposal only applies to 2.7/3.3. Since >> _ensurepip will be used for venv as well as initial install in >> 3.4, it should be a full-blown stdlib module in that version, >> probably even without the leading underscore. >> >> >> That's how I read the proposal as well. If that works then >> great, otherwise naming it _ensurepip in 2.7/3.3 should be enough >> to warn anyone that they are playing with fire. I think Martin, >> Ned, etc. would need to weigh in on whether it's at all an issue >> having the ensurepip script somewhere that doesn't get installed >> but executable from the installer is at all an issue. For source >> it can just be in Tools for 2.7/3.3 if that's the route chosen. >> >> Regardless, one of these two options is enough and I suspect we >> can stop debating and let Martin make his BDFAP decision. > > Yes, no matter which way the decision for 2.7 and 3.3 goes the > proposal is for "ensurepip" in 3.4+ I'm in favor of whatever causes it to get available in 2.7/3.3 in a way that's predictable. OpenStack is both very highly invested in pip installation of software and dependencies and also seemingly maniacally obsessed with continuing to support the older pythons people have. (eek, we have to support 2.6 until a new RHEL comes out) To that end, we wind up bootstrapping into pip via distro packages, but then it's too old, so then people pip install -U pip - systemwide, so then pip gets broken on redhat and weird on ubuntu because of fail, and then they get angry and start yelling and I have to go hide in a hole. If we could start, in tooling, depending on some predictable manner other than the distros to bootstrap into a sensible and modern pip, I believe my life would be so much better that I might grow a herd of unicorns. Monty -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.12 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlJGRTgACgkQ2Jv7/VK1RgF3PQCg18X5gBomDNvI0zTUcFJyN5As misAoOcnFdJYpAM8ur/lfDhKFGxh4VmQ =BuJf -----END PGP SIGNATURE----- From stephen at xemacs.org Sat Sep 28 05:48:05 2013 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sat, 28 Sep 2013 12:48:05 +0900 Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for pronouncement? In-Reply-To: <CADiSq7d9ipdAiP8yLCsO0OGoTRLvcL1_0gEHdig=UdH_iLWFgQ@mail.gmail.com> References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com> <20130925165022.3d67e290@anarchist> <109B26FC-32A9-4DDE-9639-E48A0389455D@stufft.io> <20130925175129.1edc4348@anarchist> <55AAF039-CCAB-467F-8FA3-74D31CB80A66@stufft.io> <20130925190410.0a5729f9@anarchist> <7FBFFC39-A0AF-47FD-9715-944D13212D10@stufft.io> <87bo3g1h9i.fsf@uwakimon.sk.tsukuba.ac.jp> <CADiSq7dJYDJwok1iccJ2mW9Axi_1JOYZ8rzDk50DxGpWDk4kwA@mail.gmail.com> <8761tm1hni.fsf@uwakimon.sk.tsukuba.ac.jp> <CADiSq7d9ipdAiP8yLCsO0OGoTRLvcL1_0gEHdig=UdH_iLWFgQ@mail.gmail.com> Message-ID: <87zjqxzjwq.fsf@uwakimon.sk.tsukuba.ac.jp> Nick Coghlan writes: > You have confirmed my belief that your model is incorrect. *shrug* I just think the risks are higher than acknowledged (just because you have so far failed to imagine a problem doesn't mean it won't appear), and that the meta effect that "Even Guido admits that Python 3 isn't ready for prime time" is perverse. We know, even those who have written blanket statements to that effect in this thread, that that is false unless conditioned on specific applications. I understand that the real motivation is that it's churlish to not relieve the pain of users who have decided for their own good reasons to use Python 2.7, and perverse to ignore the needs of the teachers who are going to educate the users about Python 3 at the time they consider appropriate. But the meta-message *received* by the public is not going to accurately reflect that motivation, and is not going to be helpful in encouraging those who already *can* move to Python 3 to do so. Anyway, clearly this exception is heading for approval, and the PEP with it. I recommend that the "Feature addition in maintenance releases" section be amended to read in its entirety: The additions of the new module to the standard library in the maintenance releases of 2.7 and 3.3 were granted explicit exceptions to the rule "no new features in maintenance releases." These exceptions were explicitly discussed, and approved in consultation with the affected release managers, separately from the rest of the PEP. They do not represent a change in policy, and must not be considered a precedent for other such exceptions. Just the facts, ma'am. It's a bad idea to include bullshit about the benefit-cost ratio, because it will be cited in future requests for similar exceptions. To the extent that people interpret this as a forecast and support for a long life for Python 2.7, there is substantial risk of such requests. Footnotes: [1] I do it that way myself, always with the most recent Python 3, but haven't yet gotten to the point of needing "pip" (use cases that happen to be met by the stdlib). From donald at stufft.io Sat Sep 28 06:12:50 2013 From: donald at stufft.io (Donald Stufft) Date: Sat, 28 Sep 2013 00:12:50 -0400 Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for pronouncement? In-Reply-To: <87zjqxzjwq.fsf@uwakimon.sk.tsukuba.ac.jp> References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com> <20130925165022.3d67e290@anarchist> <109B26FC-32A9-4DDE-9639-E48A0389455D@stufft.io> <20130925175129.1edc4348@anarchist> <55AAF039-CCAB-467F-8FA3-74D31CB80A66@stufft.io> <20130925190410.0a5729f9@anarchist> <7FBFFC39-A0AF-47FD-9715-944D13212D10@stufft.io> <87bo3g1h9i.fsf@uwakimon.sk.tsukuba.ac.jp> <CADiSq7dJYDJwok1iccJ2mW9Axi_1JOYZ8rzDk50DxGpWDk4kwA@mail.gmail.com> <8761tm1hni.fsf@uwakimon.sk.tsukuba.ac.jp> <CADiSq7d9ipdAiP8yLCsO0OGoTRLvcL1_0gEHdig=UdH_iLWFgQ@mail.gmail.com> <87zjqxzjwq.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <F8BBC650-8D26-4937-A70A-729FFBB197B7@stufft.io> On Sep 27, 2013, at 11:48 PM, "Stephen J. Turnbull" <stephen at xemacs.org> wrote: > Nick Coghlan writes: > >> You have confirmed my belief that your model is incorrect. > > *shrug* I just think the risks are higher than acknowledged (just > because you have so far failed to imagine a problem doesn't mean it > won't appear), and that the meta effect that "Even Guido admits that > Python 3 isn't ready for prime time" is perverse. We know, even those > who have written blanket statements to that effect in this thread, > that that is false unless conditioned on specific applications. I haven't seen anyone say Python 3 isn't ready to be used, just that it makes more sense for beginners to use 2.7, and I think it does still. Porting libraries from 2.x to 3.x and translaing the existing corpus of tutorials, tips, posts, etc isn't a trivial task and one that beginners are unlikely to grok. > > I understand that the real motivation is that it's churlish to not > relieve the pain of users who have decided for their own good reasons > to use Python 2.7, and perverse to ignore the needs of the teachers > who are going to educate the users about Python 3 at the time they > consider appropriate. But the meta-message *received* by the public > is not going to accurately reflect that motivation, and is not going > to be helpful in encouraging those who already *can* move to Python 3 > to do so. > > Anyway, clearly this exception is heading for approval, and the PEP > with it. I recommend that the "Feature addition in maintenance > releases" section be amended to read in its entirety: > > The additions of the new module to the standard library in the > maintenance releases of 2.7 and 3.3 were granted explicit > exceptions to the rule "no new features in maintenance releases." > These exceptions were explicitly discussed, and approved in > consultation with the affected release managers, separately from > the rest of the PEP. They do not represent a change in policy, > and must not be considered a precedent for other such exceptions. > > Just the facts, ma'am. It's a bad idea to include bullshit about the > benefit-cost ratio, because it will be cited in future requests for > similar exceptions. To the extent that people interpret this as a > forecast and support for a long life for Python 2.7, there is > substantial risk of such requests. Maybe my understanding of the PEP process is flawed, but isn't part of the point of it to codify the *reasons* for the decisions that were made? That's why they include information about dissenting opinions and such? I don't think it's dangerous to cite the reasons the decision was came to. Perhaps it can be toned down but I think it's useful to document the discussion. I've got a partially done update that tries to capture the dissenting opinions as well for completeness sake. > > > Footnotes: > [1] I do it that way myself, always with the most recent Python 3, > but haven't yet gotten to the point of needing "pip" (use cases that > happen to be met by the stdlib). > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: <http://mail.python.org/pipermail/python-dev/attachments/20130928/58a78cf8/attachment-0001.sig> From ncoghlan at gmail.com Sat Sep 28 07:59:05 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 28 Sep 2013 15:59:05 +1000 Subject: [Python-Dev] Getting Tulip (PEP 3156) into the 3.4 stdlib, marked provisional, named asyncio In-Reply-To: <CAP7+vJKSOHYeCqhsJVF_7epTDeOw4UP3crsK_mY7HaFYnwG1aA@mail.gmail.com> References: <CAP7+vJKSOHYeCqhsJVF_7epTDeOw4UP3crsK_mY7HaFYnwG1aA@mail.gmail.com> Message-ID: <CADiSq7c22aKMqW_u1GfLjA=uFuGR3e83R-ynVzpbG_nH9Am+2w@mail.gmail.com> It sounds like a reasonable approach to me. In terms of naming, would you consider "concurrent.asyncio"? When we created that parent namespace for futures, one of the other suggested submodules discussed was the standard event loop API. Cheers, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130928/aa4ee948/attachment.html> From ncoghlan at gmail.com Sat Sep 28 08:51:27 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 28 Sep 2013 16:51:27 +1000 Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for pronouncement? In-Reply-To: <F8BBC650-8D26-4937-A70A-729FFBB197B7@stufft.io> References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com> <20130925165022.3d67e290@anarchist> <109B26FC-32A9-4DDE-9639-E48A0389455D@stufft.io> <20130925175129.1edc4348@anarchist> <55AAF039-CCAB-467F-8FA3-74D31CB80A66@stufft.io> <20130925190410.0a5729f9@anarchist> <7FBFFC39-A0AF-47FD-9715-944D13212D10@stufft.io> <87bo3g1h9i.fsf@uwakimon.sk.tsukuba.ac.jp> <CADiSq7dJYDJwok1iccJ2mW9Axi_1JOYZ8rzDk50DxGpWDk4kwA@mail.gmail.com> <8761tm1hni.fsf@uwakimon.sk.tsukuba.ac.jp> <CADiSq7d9ipdAiP8yLCsO0OGoTRLvcL1_0gEHdig=UdH_iLWFgQ@mail.gmail.com> <87zjqxzjwq.fsf@uwakimon.sk.tsukuba.ac.jp> <F8BBC650-8D26-4937-A70A-729FFBB197B7@stufft.io> Message-ID: <CADiSq7eDBzCeP3mw6Mc2j7voiQvxTLw35ysxfO4OJWqYKwQzvw@mail.gmail.com> On 28 Sep 2013 14:12, "Donald Stufft" <donald at stufft.io> wrote: > > > On Sep 27, 2013, at 11:48 PM, "Stephen J. Turnbull" <stephen at xemacs.org> wrote: > > > Nick Coghlan writes: > > > >> You have confirmed my belief that your model is incorrect. > > > > *shrug* I just think the risks are higher than acknowledged (just > > because you have so far failed to imagine a problem doesn't mean it > > won't appear), and that the meta effect that "Even Guido admits that > > Python 3 isn't ready for prime time" is perverse. We know, even those > > who have written blanket statements to that effect in this thread, > > that that is false unless conditioned on specific applications. > > I haven't seen anyone say Python 3 isn't ready to be used, just that it > makes more sense for beginners to use 2.7, and I think it does still. > Porting libraries from 2.x to 3.x and translaing the existing corpus of > tutorials, tips, posts, etc isn't a trivial task and one that beginners > are unlikely to grok. > > > > > I understand that the real motivation is that it's churlish to not > > relieve the pain of users who have decided for their own good reasons > > to use Python 2.7, and perverse to ignore the needs of the teachers > > who are going to educate the users about Python 3 at the time they > > consider appropriate. But the meta-message *received* by the public > > is not going to accurately reflect that motivation, and is not going > > to be helpful in encouraging those who already *can* move to Python 3 > > to do so. > > > > Anyway, clearly this exception is heading for approval, and the PEP > > with it. I recommend that the "Feature addition in maintenance > > releases" section be amended to read in its entirety: > > > > The additions of the new module to the standard library in the > > maintenance releases of 2.7 and 3.3 were granted explicit > > exceptions to the rule "no new features in maintenance releases." > > These exceptions were explicitly discussed, and approved in > > consultation with the affected release managers, separately from > > the rest of the PEP. They do not represent a change in policy, > > and must not be considered a precedent for other such exceptions. > > > > Just the facts, ma'am. It's a bad idea to include bullshit about the > > benefit-cost ratio, because it will be cited in future requests for > > similar exceptions. To the extent that people interpret this as a > > forecast and support for a long life for Python 2.7, there is > > substantial risk of such requests. > > Maybe my understanding of the PEP process is flawed, but isn't > part of the point of it to codify the *reasons* for the decisions that > were made? That's why they include information about dissenting > opinions and such? > > I don't think it's dangerous to cite the reasons the decision was came > to. Perhaps it can be toned down but I think it's useful to document > the discussion. I've got a partially done update that tries to capture > the dissenting opinions as well for completeness sake. We'll put something in pointing out that accepting this change actually makes it even *harder* to advocate for further feature additions in Python 2.7 maintenance releases as there is *zero* chance of backporting language changes, and this PEP means library and builtin updates can easily be a pip install away if someone is willing to create the backport and put it on PyPI (Brandon Rhodes already created the "backports" namespace package as a common home for such efforts, or there's the "2" suffix that has been used in a couple of cases). Cheers, Nick. > > > > > > > Footnotes: > > [1] I do it that way myself, always with the most recent Python 3, > > but haven't yet gotten to the point of needing "pip" (use cases that > > happen to be met by the stdlib). > > _______________________________________________ > > Python-Dev mailing list > > Python-Dev at python.org > > https://mail.python.org/mailman/listinfo/python-dev > > Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io > > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130928/152955fa/attachment.html> From nad at acm.org Sat Sep 28 09:20:40 2013 From: nad at acm.org (Ned Deily) Date: Sat, 28 Sep 2013 00:20:40 -0700 Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for pronouncement? References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com> <20130925165022.3d67e290@anarchist> <109B26FC-32A9-4DDE-9639-E48A0389455D@stufft.io> <20130925175129.1edc4348@anarchist> <55AAF039-CCAB-467F-8FA3-74D31CB80A66@stufft.io> <20130925190410.0a5729f9@anarchist> <7FBFFC39-A0AF-47FD-9715-944D13212D10@stufft.io> <87bo3g1h9i.fsf@uwakimon.sk.tsukuba.ac.jp> <CADiSq7dJYDJwok1iccJ2mW9Axi_1JOYZ8rzDk50DxGpWDk4kwA@mail.gmail.com> <8761tm1hni.fsf@uwakimon.sk.tsukuba.ac.jp> <CACac1F-soYfp5Bg0sq+zgL6he301Vn7j2nRyTSodXoHBi+3K8Q@mail.gmail.com> <l24k17$efa$1@ger.gmane.org> <2E8C3B8B-808C-40CA-9914-D70E6627AA3F@stufft.io> <l24okm$5lf$1@ger.gmane.org> <7A7A9106-0668-4E26-A132-23F59296E5B3@stufft.io> <CAKJDb-M6LqJyoWaPoquy0apoLJ+Ffr9irNBudrX0F10KwoqmbQ@mail.gmail.com> <CAP1=2W44X6+zaQ9=-J4SdekUJ0CxWy7b8YYk=eWM=88CODbn5w@mail.gmail.com> Message-ID: <nad-5BEB10.00204028092013@news.gmane.org> In article <CAP1=2W44X6+zaQ9=-J4SdekUJ0CxWy7b8YYk=eWM=88CODbn5w at mail.gmail.com>, Brett Cannon <brett at python.org> wrote: > On Fri, Sep 27, 2013 at 5:16 PM, Zachary Ware > <zachary.ware+pydev at gmail.com>wrote: > > The way I read Terry's proposal, it is to never add the _ensurepip > > *module*, but to use (or make available, whichever makes sense in a > > given case) the _ensurepip *script* when it is requested at > > install-time: specifically, put _ensurepip.py in Tools/scripts, PC/, > > Mac/, or really anywhere but Lib/ so that builders can find it, but > > Python can't. In other words, don't make "import _ensurepip" possible > > out of the box, and a lot of the "don't add new features in > > maintenance releases" blockade disappears, because you aren't actually > > adding any new capability to Python itself or its standard library. > > > > Of course, this proposal only applies to 2.7/3.3. Since _ensurepip > > will be used for venv as well as initial install in 3.4, it should be > > a full-blown stdlib module in that version, probably even without the > > leading underscore. > That's how I read the proposal as well. If that works then great, otherwise > naming it _ensurepip in 2.7/3.3 should be enough to warn anyone that they > are playing with fire. I think Martin, Ned, etc. would need to weigh in on > whether it's at all an issue having the ensurepip script somewhere that > doesn't get installed but executable from the installer is at all an issue. > For source it can just be in Tools for 2.7/3.3 if that's the route chosen. Putting an "ensurepip.py" script (with or without the "_") in Tools/ should be fine for the needs of 2.7 and/or 3.3 OS X installers. Without venv in the picture, it doesn't need to do much; the installer build script itself could handle everything without an ensurepip. There are still other changes in the installer that would need to be made, particularly if it is necessary to make the installation of pip (and its dependency, setuptools) optional but all of those would be isolated to the Mac subdirectory. -- Ned Deily, nad at acm.org From techtonik at gmail.com Sat Sep 28 09:41:12 2013 From: techtonik at gmail.com (anatoly techtonik) Date: Sat, 28 Sep 2013 10:41:12 +0300 Subject: [Python-Dev] PEP 453 (pip bootstrap) TL;DR and Usage Scenario Message-ID: <CAPkN8xLpE=3ftJuhu4_UuxBmYXrm8Up9m7dQMVQk9XZ5eOYR6A@mail.gmail.com> On Mon, Sep 23, 2013 at 2:15 PM, Nick Coghlan <ncoghlan at gmail.com> wrote: > With the last round of updates, I believe PEP 453 is ready for > Martin's pronouncement. > > HTML: http://www.python.org/dev/peps/pep-0453/ > Major diffs: http://hg.python.org/peps/rev/b2993450b32a I'd enjoy concise PEP texts, but it is extremely hard to write one, so is it possible to at least add a TL;DR chapter that just shows the concept in 15 seconds without all the gory details? What is the entrypoint in the bootstrap process and knowledge requirements for the user? For the same sake, is it possible to include real world usage scenario as a code paste example of user session. For instance: # --- 001: user reads install instruction on <package> site # 'pip install <package>' # user executes 'pip install <package>' on Windows > pip install <package> 'pip' is not recognized as an internal or external command, operable program or batch file. # user ...<does what>... ? [ ] # user executes 'pip install <package>' on Ubuntu $ pip install <package> The program 'pip' is currently not installed. You can install it by typing: sudo apt-get install pip $ sudo apt-get install pip # (user has sudo - done) # user doesn't have sudo, command fails # user ...<does what>... ? [ ] # --- 002: user reads install instruction on <package> site # 'python -m pip install <package>' $ python -m pip install <package> /usr/bin/python: No module named pip # user ...<does what>... ? [ ] -- anatoly t. From solipsis at pitrou.net Sat Sep 28 12:55:21 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 28 Sep 2013 12:55:21 +0200 Subject: [Python-Dev] Getting Tulip (PEP 3156) into the 3.4 stdlib, marked provisional, named asyncio References: <CAP7+vJKSOHYeCqhsJVF_7epTDeOw4UP3crsK_mY7HaFYnwG1aA@mail.gmail.com> <CADiSq7c22aKMqW_u1GfLjA=uFuGR3e83R-ynVzpbG_nH9Am+2w@mail.gmail.com> Message-ID: <20130928125521.1317b48a@fsol> On Sat, 28 Sep 2013 15:59:05 +1000 Nick Coghlan <ncoghlan at gmail.com> wrote: > It sounds like a reasonable approach to me. > > In terms of naming, would you consider "concurrent.asyncio"? When we > created that parent namespace for futures, one of the other suggested > submodules discussed was the standard event loop API. -10. Please, let's stop it. Regards Antoine. From eliben at gmail.com Sat Sep 28 14:56:46 2013 From: eliben at gmail.com (Eli Bendersky) Date: Sat, 28 Sep 2013 05:56:46 -0700 Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for pronouncement? In-Reply-To: <CAPZV6o8wZ_ii=mfSJft_FeB4SaqK-L_Or6-6dsmX2B9-3XK+nA@mail.gmail.com> References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com> <20130925165022.3d67e290@anarchist> <109B26FC-32A9-4DDE-9639-E48A0389455D@stufft.io> <20130925175129.1edc4348@anarchist> <CADiSq7fVrruk1zcE-ZwHUOyUqOPhTBnn2jcp9rKQ8dS6QeL+Zw@mail.gmail.com> <20130925190854.6ac76200@anarchist> <CADiSq7cMMkTjArg=0=rmDypHMo5aACYNQX=ct18N9jzru2=JQQ@mail.gmail.com> <20130927204417.73058d41@anarchist> <CAPZV6o8wZ_ii=mfSJft_FeB4SaqK-L_Or6-6dsmX2B9-3XK+nA@mail.gmail.com> Message-ID: <CAF-Rda_2FoPEOt7GEOLN-XgeAiOh8Hw6NjSV3nbo5U6YS9=TSA@mail.gmail.com> On Fri, Sep 27, 2013 at 6:19 PM, Benjamin Peterson <benjamin at python.org>wrote: > 2013/9/27 Barry Warsaw <barry at python.org>: > > On Sep 26, 2013, at 02:30 PM, Nick Coghlan wrote: > > > >>- the module name should be "_ensurepip" in all versions > >>- the PEP should explicitly state that the "don't remove _ensurepip > >>and it's wheel files" caveat for redistributors applies only in 3.4+ > >>(where removing it will break pyvenv) > > > > I'm sorry that I probably won't be able to engage more on this thread, > but I > > think my points should be fairly well understood by the PEP czar and RM. > > Should the PEP be accepted, I think both of these are good changes. > > And FWIW, I generally agree with Barry about adding things to 2.7. > +1. Not to mention the general problem of adding new features in maintenance releases (what? it doesn't work for you? Check it's 2.7.6 and not 2.7.5!) , at this point in the trajectory of Python 3.x and 2.7 it sends a very wrong message. Eli -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130928/14c9fb59/attachment.html> From barry at python.org Sat Sep 28 16:35:41 2013 From: barry at python.org (Barry Warsaw) Date: Sat, 28 Sep 2013 10:35:41 -0400 Subject: [Python-Dev] Getting Tulip (PEP 3156) into the 3.4 stdlib, marked provisional, named asyncio In-Reply-To: <CAP1=2W7A2Q3t7Bk7iDZ9ZPprFdi+yr152EgVV=ACaTZkaBbUmw@mail.gmail.com> References: <CAP7+vJKSOHYeCqhsJVF_7epTDeOw4UP3crsK_mY7HaFYnwG1aA@mail.gmail.com> <CAP1=2W7A2Q3t7Bk7iDZ9ZPprFdi+yr152EgVV=ACaTZkaBbUmw@mail.gmail.com> Message-ID: <20130928103541.67ccf3bd@limelight.wooz.org> On Sep 27, 2013, at 09:14 PM, Brett Cannon wrote: >I don't see any issue with redirecting the discussion. python-tulip@ is >acting like a SIG for the module, so no real precedent beyond it not being >hosted as a mail.python.org list. The PEP process even allows for this formally. Please add a Discussions-To header to PEP 3156. See PEP 1 for details. -Barry From barry at python.org Sat Sep 28 16:38:48 2013 From: barry at python.org (Barry Warsaw) Date: Sat, 28 Sep 2013 10:38:48 -0400 Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for pronouncement? In-Reply-To: <87zjqxzjwq.fsf@uwakimon.sk.tsukuba.ac.jp> References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com> <20130925165022.3d67e290@anarchist> <109B26FC-32A9-4DDE-9639-E48A0389455D@stufft.io> <20130925175129.1edc4348@anarchist> <55AAF039-CCAB-467F-8FA3-74D31CB80A66@stufft.io> <20130925190410.0a5729f9@anarchist> <7FBFFC39-A0AF-47FD-9715-944D13212D10@stufft.io> <87bo3g1h9i.fsf@uwakimon.sk.tsukuba.ac.jp> <CADiSq7dJYDJwok1iccJ2mW9Axi_1JOYZ8rzDk50DxGpWDk4kwA@mail.gmail.com> <8761tm1hni.fsf@uwakimon.sk.tsukuba.ac.jp> <CADiSq7d9ipdAiP8yLCsO0OGoTRLvcL1_0gEHdig=UdH_iLWFgQ@mail.gmail.com> <87zjqxzjwq.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <20130928103848.0c1ed468@limelight.wooz.org> On Sep 28, 2013, at 12:48 PM, Stephen J. Turnbull wrote: >*shrug* I just think the risks are higher than acknowledged (just >because you have so far failed to imagine a problem doesn't mean it >won't appear), and that the meta effect that "Even Guido admits that >Python 3 isn't ready for prime time" is perverse. We know, even those >who have written blanket statements to that effect in this thread, >that that is false unless conditioned on specific applications. > >I understand that the real motivation is that it's churlish to not >relieve the pain of users who have decided for their own good reasons >to use Python 2.7, and perverse to ignore the needs of the teachers >who are going to educate the users about Python 3 at the time they >consider appropriate. But the meta-message *received* by the public >is not going to accurately reflect that motivation, and is not going >to be helpful in encouraging those who already *can* move to Python 3 >to do so. FWIW, +1 > >Anyway, clearly this exception is heading for approval, and the PEP >with it. I recommend that the "Feature addition in maintenance >releases" section be amended to read in its entirety: > > The additions of the new module to the standard library in the > maintenance releases of 2.7 and 3.3 were granted explicit > exceptions to the rule "no new features in maintenance releases." > These exceptions were explicitly discussed, and approved in > consultation with the affected release managers, separately from > the rest of the PEP. They do not represent a change in policy, > and must not be considered a precedent for other such exceptions. Call this the "Bush v. Gore" amendment. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: <http://mail.python.org/pipermail/python-dev/attachments/20130928/8f2ccb29/attachment.sig> From rawthemetal at gmail.com Sat Sep 28 13:28:04 2013 From: rawthemetal at gmail.com (Kevin Ngugi) Date: Sat, 28 Sep 2013 14:28:04 +0300 Subject: [Python-Dev] toolbar Message-ID: <CANS-bzPK2Krd9F39DxxFKZhgPv5vYwUUbHZSwL=BcAeGt1YmxA@mail.gmail.com> Hi, I just downloaded Python 3.3 top teach myself how to program, I am new to programming, but the guide I am using requires me to access the toolbar, which I cannot seem to find. How do I find it and have it displayed on the interface? I tried v 3.1 but its still absent. Thank you. From eliben at gmail.com Sat Sep 28 18:07:00 2013 From: eliben at gmail.com (Eli Bendersky) Date: Sat, 28 Sep 2013 09:07:00 -0700 Subject: [Python-Dev] Getting Tulip (PEP 3156) into the 3.4 stdlib, marked provisional, named asyncio In-Reply-To: <CAP7+vJKSOHYeCqhsJVF_7epTDeOw4UP3crsK_mY7HaFYnwG1aA@mail.gmail.com> References: <CAP7+vJKSOHYeCqhsJVF_7epTDeOw4UP3crsK_mY7HaFYnwG1aA@mail.gmail.com> Message-ID: <CAF-Rda-_qGpkQEg2Zu9FtFo_LRMnki7owgoKLsa5g3nwjOe+ng@mail.gmail.com> On Fri, Sep 27, 2013 at 3:33 PM, Guido van Rossum <guido at python.org> wrote: > I've been looking at my progress with Tulip and the 3.4 release schedule > (PEP 429) and it looks like I will have to do some kind of sprint to get it > into the release in time for beta 1, which is planned for Nov 24. Ideally > I'd get it into alpha 4, which is scheduled for Oct 20 -- that's in about > three weeks, and probably too tight. > > Even Nov 24 is aggressive, because the PEP (PEP 3156) hasn't even been > discussed formally, let alone accepted! Fortunately I'm pretty happy with > most of the APIs defined in the PEP -- there are some holes but I expect no > big changes at this point. > > It's clear that the only way we can get this into the stdlib is to mark it > as provisional, per PEP 411. That's fine with me. I expect that the kind of > changes the code will see between the 3.4 and 3.5 releases are well within > the bounds of the expectations set by that PEP -- while it is clear that > everything is possible (even withdrawal), the most likely outcome is that > the API will be extended or tweaked at most, and not significantly changed > in a backward compatible way -- without completely ruling out that some > part of the API may deserve a serious overhaul based on experience. That's > exactly how I feel about Tulip. > > I propose that the new library will be named asyncio. It would be odd if > we kept the name tulip (stdlib modules rarely have "creative" names), and > "asynchronous I/O" seems to cover the contents quite well. > > I propose that we do the bikeshedding about the API on the > python-tulip at googlegroups.com mailing list; I'm not sure what we would > gain by holding the discussion on python-dev (although clearly we should > report back here with important milestones in the review). I will post a > message with the kick-off there. > Sounds good. But once it's in stdlib, I think it would be proper to shift the discussion into the normal pydev channels (python-dev, issue tracker, etc.). Is this the plan? Eli -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130928/11ccc2d5/attachment.html> From tjreedy at udel.edu Sat Sep 28 18:14:34 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 28 Sep 2013 12:14:34 -0400 Subject: [Python-Dev] toolbar In-Reply-To: <CANS-bzPK2Krd9F39DxxFKZhgPv5vYwUUbHZSwL=BcAeGt1YmxA@mail.gmail.com> References: <CANS-bzPK2Krd9F39DxxFKZhgPv5vYwUUbHZSwL=BcAeGt1YmxA@mail.gmail.com> Message-ID: <5247006A.2070307@udel.edu> On 9/28/2013 7:28 AM, Kevin Ngugi wrote: > Hi, I just downloaded Python 3.3 top teach myself how to program, I am > new to programming, but the guide I am using requires me to access the > toolbar, which I cannot seem to find. How do I find it and have it > displayed on the interface? I tried v 3.1 but its still absent. Thank > you. python-dev is for development of Python. Please ask questions about using Python elsewhere, such as python-list. When you do, say what 'guide' you are using and what 'toolbar' it refers to. I don't know. And please do not answer this email. -- Terry Jan Reedy From python at mrabarnett.plus.com Sat Sep 28 18:36:35 2013 From: python at mrabarnett.plus.com (MRAB) Date: Sat, 28 Sep 2013 17:36:35 +0100 Subject: [Python-Dev] toolbar In-Reply-To: <CANS-bzPK2Krd9F39DxxFKZhgPv5vYwUUbHZSwL=BcAeGt1YmxA@mail.gmail.com> References: <CANS-bzPK2Krd9F39DxxFKZhgPv5vYwUUbHZSwL=BcAeGt1YmxA@mail.gmail.com> Message-ID: <52470593.1010109@mrabarnett.plus.com> On 28/09/2013 12:28, Kevin Ngugi wrote: > Hi, I just downloaded Python 3.3 top teach myself how to program, I am > new to programming, but the guide I am using requires me to access the > toolbar, which I cannot seem to find. How do I find it and have it > displayed on the interface? I tried v 3.1 but its still absent. Thank > you. > This list ("python-dev") is for the development _of_ Python, not _with_ Python, which is "python-list". Also, you'll need to say what you mean by "the toolbar". From guido at python.org Sat Sep 28 18:52:28 2013 From: guido at python.org (Guido van Rossum) Date: Sat, 28 Sep 2013 09:52:28 -0700 Subject: [Python-Dev] Getting Tulip (PEP 3156) into the 3.4 stdlib, marked provisional, named asyncio In-Reply-To: <CADiSq7c22aKMqW_u1GfLjA=uFuGR3e83R-ynVzpbG_nH9Am+2w@mail.gmail.com> References: <CAP7+vJKSOHYeCqhsJVF_7epTDeOw4UP3crsK_mY7HaFYnwG1aA@mail.gmail.com> <CADiSq7c22aKMqW_u1GfLjA=uFuGR3e83R-ynVzpbG_nH9Am+2w@mail.gmail.com> Message-ID: <CAP7+vJJfMP_1DFDc8dSGJo=JiE2hPGK_2AzrWUZUpxdiJ=qRPg@mail.gmail.com> On Fri, Sep 27, 2013 at 10:59 PM, Nick Coghlan <ncoghlan at gmail.com> wrote: > It sounds like a reasonable approach to me. > > In terms of naming, would you consider "concurrent.asyncio"? When we > created that parent namespace for futures, one of the other suggested > submodules discussed was the standard event loop API. > Hm. I want the threading and event world to be very clearly separate and different, since accidentally combining them is disastrous. So the concurrent package is the *last* place where I want asyncio to live. (And I realize there is also some multiprocessing support in that package -- but it still uses threads to wait for things.) -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130928/1d8e1e25/attachment.html> From jnoller at gmail.com Sat Sep 28 18:53:47 2013 From: jnoller at gmail.com (Jesse Noller) Date: Sat, 28 Sep 2013 11:53:47 -0500 Subject: [Python-Dev] Getting Tulip (PEP 3156) into the 3.4 stdlib, marked provisional, named asyncio In-Reply-To: <CADiSq7c22aKMqW_u1GfLjA=uFuGR3e83R-ynVzpbG_nH9Am+2w@mail.gmail.com> References: <CAP7+vJKSOHYeCqhsJVF_7epTDeOw4UP3crsK_mY7HaFYnwG1aA@mail.gmail.com> <CADiSq7c22aKMqW_u1GfLjA=uFuGR3e83R-ynVzpbG_nH9Am+2w@mail.gmail.com> Message-ID: <E41A420B-C78E-415B-950D-E9DC54D9D06D@gmail.com> > On Sep 28, 2013, at 12:59 AM, Nick Coghlan <ncoghlan at gmail.com> wrote: > > It sounds like a reasonable approach to me. > > In terms of naming, would you consider "concurrent.asyncio"? When we created that parent namespace for futures, one of the other suggested submodules discussed was the standard event loop API. > +1 but up to guido > Cheers, > Nick. > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/jnoller%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130928/8977b54b/attachment.html> From guido at python.org Sat Sep 28 19:02:19 2013 From: guido at python.org (Guido van Rossum) Date: Sat, 28 Sep 2013 10:02:19 -0700 Subject: [Python-Dev] Getting Tulip (PEP 3156) into the 3.4 stdlib, marked provisional, named asyncio In-Reply-To: <CAF-Rda-_qGpkQEg2Zu9FtFo_LRMnki7owgoKLsa5g3nwjOe+ng@mail.gmail.com> References: <CAP7+vJKSOHYeCqhsJVF_7epTDeOw4UP3crsK_mY7HaFYnwG1aA@mail.gmail.com> <CAF-Rda-_qGpkQEg2Zu9FtFo_LRMnki7owgoKLsa5g3nwjOe+ng@mail.gmail.com> Message-ID: <CAP7+vJJ2_Hu-krRae81jO0ZMpN7oC+7bSmv_YkJS2hcw2dJNcQ@mail.gmail.com> On Sat, Sep 28, 2013 at 9:07 AM, Eli Bendersky <eliben at gmail.com> wrote: > > > Sounds good. But once it's in stdlib, I think it would be proper to shift > the discussion into the normal pydev channels (python-dev, issue tracker, > etc.). Is this the plan? > Hadn't really thought about that. I think there's a precedent for aspects of Python that have their own mailing list (e.g. import-sig). Also, python-dev is usually pretty hostile about people asking usage questions -- there's no such taboo on the Tulip list. I can certainly see how you wouldn't want to have to look at multiple trackers. Possibly the Tulip tracker could be limited to discussion of the Python 3.3 backport that I hope to maintain in that project. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130928/75087793/attachment-0001.html> From breamoreboy at yahoo.co.uk Sat Sep 28 19:42:55 2013 From: breamoreboy at yahoo.co.uk (Mark Lawrence) Date: Sat, 28 Sep 2013 18:42:55 +0100 Subject: [Python-Dev] toolbar In-Reply-To: <CANS-bzPK2Krd9F39DxxFKZhgPv5vYwUUbHZSwL=BcAeGt1YmxA@mail.gmail.com> References: <CANS-bzPK2Krd9F39DxxFKZhgPv5vYwUUbHZSwL=BcAeGt1YmxA@mail.gmail.com> Message-ID: <l274en$m3g$1@ger.gmane.org> On 28/09/2013 12:28, Kevin Ngugi wrote: > Hi, I just downloaded Python 3.3 top teach myself how to program, I am > new to programming, but the guide I am using requires me to access the > toolbar, which I cannot seem to find. How do I find it and have it > displayed on the interface? I tried v 3.1 but its still absent. Thank > you. > Further to other replies you might be better off on the tutor mailing list see https://mail.python.org/mailman/listinfo/tutor -- Cheers. Mark Lawrence From tjreedy at udel.edu Sat Sep 28 22:19:44 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 28 Sep 2013 16:19:44 -0400 Subject: [Python-Dev] Best practice for documentation for std lib In-Reply-To: <CAP7+vJJSvX78f2+i7_C5mNMdOQpfSD_oKqC01Z8CEHaU2krqTw@mail.gmail.com> References: <20130922031304.GE19939@ando> <CAF-Rda_sTOiKi6c9ihFVL-N90PC-xHxOAMm1im0s-KVyy=L_Ng@mail.gmail.com> <87ioxs3iix.fsf@uwakimon.sk.tsukuba.ac.jp> <CAP7+vJJucubZtXHhnNbAcE0GWfQTnOozHK9W=uv3CWkMX15XGA@mail.gmail.com> <20130923003142.GO19939@ando> <CAP7+vJLThCjrCGGaJLMNd2Uza6wanODnipjmZ3LBZpTkub=uCg@mail.gmail.com> <l1o8ri$mjv$1@ger.gmane.org> <CAP7+vJJSvX78f2+i7_C5mNMdOQpfSD_oKqC01Z8CEHaU2krqTw@mail.gmail.com> Message-ID: <l27dkq$vgc$1@ger.gmane.org> On 9/22/2013 10:44 PM, Guido van Rossum wrote: > On Sun, Sep 22, 2013 at 7:25 PM, Terry Reedy <tjreedy at udel.edu > ('Return' rather than 'Returns' is the current convention.) > That's actually a religious argument which in the stdlib takes no strict > position -- a quick grep shows that both are used, although 'Return' is > more frequent by a 5-to-1 margin. I wrote 'current convention' not just as a statistical statement, but as a prescription I had read, even though I did not remember just where. PEP8 says "PEP 257 describes good docstring conventions." I followed that clue found the statement I had read. PEP 257 says ''' It [the one line docstring] prescribes the function or method's effect as a command ("Do this", "Return that"), not as a description; e.g. don't write "Returns the pathname ...". ''' Whether the reference in PEP 8 makes it a PEP 8 rule? or a 'strict stdlib position'? http://bugs.python.org/issue19067 proposes to change rangeobject docstrings. Is that OK or should the issue be rejected and closed? -- Terry Jan Reedy From ncoghlan at gmail.com Sun Sep 29 02:15:14 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 29 Sep 2013 10:15:14 +1000 Subject: [Python-Dev] Getting Tulip (PEP 3156) into the 3.4 stdlib, marked provisional, named asyncio In-Reply-To: <CAP7+vJJfMP_1DFDc8dSGJo=JiE2hPGK_2AzrWUZUpxdiJ=qRPg@mail.gmail.com> References: <CAP7+vJKSOHYeCqhsJVF_7epTDeOw4UP3crsK_mY7HaFYnwG1aA@mail.gmail.com> <CADiSq7c22aKMqW_u1GfLjA=uFuGR3e83R-ynVzpbG_nH9Am+2w@mail.gmail.com> <CAP7+vJJfMP_1DFDc8dSGJo=JiE2hPGK_2AzrWUZUpxdiJ=qRPg@mail.gmail.com> Message-ID: <CADiSq7ea-4YWBwRS_6v1m6fC0u1yTCu081ofeCuCUQQ4iTMX=w@mail.gmail.com> On 29 Sep 2013 02:52, "Guido van Rossum" <guido at python.org> wrote: > > On Fri, Sep 27, 2013 at 10:59 PM, Nick Coghlan <ncoghlan at gmail.com> wrote: >> >> It sounds like a reasonable approach to me. >> >> In terms of naming, would you consider "concurrent.asyncio"? When we created that parent namespace for futures, one of the other suggested submodules discussed was the standard event loop API. > > > Hm. I want the threading and event world to be very clearly separate and different, since accidentally combining them is disastrous. So the concurrent package is the *last* place where I want asyncio to live. (And I realize there is also some multiprocessing support in that package -- but it still uses threads to wait for things.) Makes sense to me! Cheers, Nick. > > -- > --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130929/4d5d483d/attachment.html> From ncoghlan at gmail.com Sun Sep 29 08:45:24 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 29 Sep 2013 16:45:24 +1000 Subject: [Python-Dev] PEP 453 updated with improved rationale Message-ID: <CADiSq7ffZq1y+3BnwzmM2FODixzmqmeXqZP=2dXxoBCuWnzfuA@mail.gmail.com> PEP 453 has been updated in response to the previous discussion thread. Updated version is online at http://www.python.org/dev/peps/pep-0453/ Relevant diffs can be found at: http://hg.python.org/peps/log/bdd50f92ee73/pep-0453.txt The bulk of the updates can be seen in the following two diffs: http://hg.python.org/peps/rev/58f33b8eb68d http://hg.python.org/peps/rev/438e3fa95c88 The underlying technical proposal itself hasn't actually changed from the previous iteration: we still want to make "ensurepip" a public module in all of Python 2.7, 3.3 and 3.4, and invoke it from the binary installers in those versions. However, the rationale sections of the PEP, both the general rationale for making the proposal at all and the specific rationale for the backporting exemption, have been updated significantly, and the PEP now includes more detail on various parts of the proposal (especially proposed changes to the documentation. The two backporting compromises discussed in the previous thread (making this a binary-installer-only feature and using a different module name for the backported version) are covered explicitly as rejected proposals. The two major additions to the rationale are a longer discourse on the differences between python-dev and distutils-sig in terms of what it means to "support" a version of Python and how this PEP is seen as a major enabling change for the future evolution of the Python packaging ecosystem, as well as much deeper coverage of the origins of the "no new features in maintenance releases" rule and how this case differs from the incidents that caused problems in the 2.2 series, and how it serves to *weaken* the case for any future exemptions rather than strengthen it. Regards, Nick. ================================ PEP: 453 Title: Explicit bootstrapping of pip in Python installations Version: $Revision$ Last-Modified: $Date$ Author: Donald Stufft <donald at stufft.io>, Nick Coghlan <ncoghlan at gmail.com> BDFL-Delegate: Martin von L?wis Status: Draft Type: Process Content-Type: text/x-rst Created: 10-Aug-2013 Post-History: 30-Aug-2013, 15-Sep-2013, 18-Sep-2013, 19-Sep-2013, 23-Sep-2013, 29-Sep-2013 Abstract ======== This PEP proposes that the `Installing Python Modules <http://docs.python.org/3/install>`__ guide be updated to officially recommend the use of ``pip`` as the default installer for Python packages, and that appropriate technical changes be made in Python 2.7, 3.3 and 3.4 to provide ``pip`` by default in support of that recommendation. Rationale ========= There are two related, but distinct rationales for the proposal in this PEP. The first relates to the experience of new users, while the second relates to better enabling the evolution of the broader Python packaging ecosystem. Improving the new user experience --------------------------------- Currently, on systems without a platform package manager and repository, installing a third-party Python package into a freshly installed Python requires first identifying an appropriate package manager and then installing it. Even on systems that *do* have a platform package manager, it is unlikely to include every package that is available on the Python Package Index, and even when a desired third-party package is available, the correct name in the platform package manager may not be clear. This means that, to work effectively with the Python Package Index ecosystem, users must know which package manager to install, where to get it, and how to install it. The effect of this is that third-party Python projects are currently required to choose from a variety of undesirable alternatives: * Assume the user already has a suitable cross-platform package manager installed. * Duplicate the instructions and tell their users how to install the package manager. * Completely forgo the use of dependencies to ease installation concerns for their users. All of these available options have significant drawbacks. If a project simply assumes a user already has the tooling then beginning users may get a confusing error message when the installation command doesn't work. Some operating systems may ease this pain by providing a global hook that looks for commands that don't exist and suggest an OS package they can install to make the command work, but that only works on systems with platform package managers that include a package that provides the relevant cross-platform installer command (such as many major Linux distributions). No such assistance is available for Windows and Mac OS X users, or more conservative Linux distributions. The challenges of dealing with this problem for beginners (who are often also completely new to programming, the use of command line tools and editing system environment variables) are a regular feature of feedback the core Python developers receive from professional educators and others introducing new users to Python. If a project chooses to duplicate the installation instructions and tell their users how to install the package manager before telling them how to install their own project then whenever these instructions need updates they need updating by every project that has duplicated them. This is particular problematic when there are multiple competing installation tools available, and different projects recommend different tools. This specific problem can be partially alleviated by strongly promoting ``pip`` as the default installer and recommending that other projects reference `pip's own bootstrapping instructions <http://www.pip-installer.org/en/latest/installing.html>`__ rather than duplicating them. However the user experience created by this approach still isn't good (especially on Windows, where downloading and running the ``get-pip.py`` bootstrap script with the default OS configuration is significantly more painful than downloading and running a binary executable or installer). The situation becomes even more complicated when multiple Python versions are involved (for example, parallel installations of Python 2 and Python 3), since that makes it harder to create and maintain good platform specific ``pip`` installers independently of the CPython installers. The projects that have decided to forgo dependencies altogether are forced to either duplicate the efforts of other projects by inventing their own solutions to problems or are required to simply include the other projects in their own source trees. Both of these options present their own problems either in duplicating maintenance work across the ecosystem or potentially leaving users vulnerable to security issues because the included code or duplicated efforts are not automatically updated when upstream releases a new version. By officially recommending and providing by default a specific cross-platform package manager it will be easier for users trying to install these third-party packages as well as easier for the people distributing them as they should now be able to safely assume that most users will have the appropriate installation tools available (or access to clear instructions on how to obtain them). This is expected to become more important in the future as the Wheel_ package format (deliberately) does not have a built in "installer" in the form of ``setup.py`` so users wishing to install from a wheel file will want an installer even in the simplest cases. Reducing the burden of actually installing a third-party package should also decrease the pressure to add every useful module to the standard library. This will allow additions to the standard library to focus more on why Python should have a particular tool out of the box, and why it is reasonable for that package to adopt the standard library's 18-24 month feature release cycle, instead of using the general difficulty of installing third-party packages as justification for inclusion. Providing a standard installation system also helps with bootstrapping alternate build and installer systems, such as ``setuptools``, ``zc.buildout`` and the ``hashdist``/``conda`` combination that is aimed specifically at the scientific community. So long as ``pip install <tool>`` works, then a standard Python-specific installer provides a reasonably secure, cross platform mechanism to get access to these utilities. Enabling the evolution of the broader Python packaging ecosystem ---------------------------------------------------------------- As no new packaging standard can achieve widespread adoption without a transition strategy that covers the versions of Python that are in widespread *current* use (rather than merely future versions, like most language features), the change proposed in this PEP is considered a necessary step in the evolution of the Python packaging ecosystem The broader community has embraced the Python Package Index as a mechanism for distributing and installing Python software, but the different concerns of language evolution and secure software distribution mean that a faster feature release cycle that encompasses older versions is needed to properly support the latter. In addition, the core CPython development team have the luxury of dropping support for earlier Python versions well before the rest of the community, as downstream commercial redistributors pick up the task of providing support for those versions to users that still need it, while many third party libraries maintain compatibility with those versions as long as they remain in widespread use. This means that the current ``setup.py install`` based model for package installation poses serious difficulties for the development and adoption of new packaging standards, as, depending on how a project writes their ``setup.py`` file, the installation command (along with other operations) may end up invoking the standard library's ``distutils`` package. As an indicator of how this may cause problems for the broader ecosystem, consider that the feature set of ``distutils`` in Python 2.6 was frozen in June 2008 (with the release of Python 2.6b1), while the feature set of ``distutils`` in Python 2.7 was frozen in April 2010 (with the release of Python 2.7b1). By contrast, using a separate installer application like ``pip`` (which ensures that even ``setup.py`` files that invoke ``distutils`` directly still support the new packaging standards) makes it possible to support new packaging standards in older versions of Python, just by upgrading ``pip`` (which receives new feature releases roughly every 6 months). The situation on older versions of Python is further improved by making it easier for end users to install and upgrade newer build systems like ``setuptools`` or improved PyPI upload utilities like ``twine``. It is not coincidental that this proposed model of using a separate installer program with more metadata heavy and less active distribution formats matches that used by most operating systems (including Windows since the introduction of the installer service and the MSI file format), as well as many other language specific installers. For Python 2.6, this compatibility issue is largely limited to various enterprise Linux distributions (and their downstream derivatives). These distributions often have even slower update cycles than CPython, so they offer full support for versions of Python that are considered "security fix only" versions upstream (and sometimes may even be to the point where the core development team no longer support them at all - you can still get commercial support for Python 2.3 if you really need it!). In practice, the fact that tools like ``wget`` and ``curl`` are readily available on Linux systems, that most users of Python on Linux are already familiar with the command line, and that most Linux distributions ship with a default configuration that makes running Python scripts easy, means that the existing ``pip`` bootstrapping instructions for any \*nix system are already quite straightforward. Even if ``pip`` isn't provided by the system package manager, then using ``wget`` or ``curl`` to retrieve the bootstrap script from www.pip-installer.org and then running it is just a couple of shell commands that can easily be copied and pasted as necessary. Accordingly, for any version of Python on any \*nix system, the need to bootstrap ``pip`` in older versions isn't considered a major barrier to adoption of new packaging standards, since it's just one more small speedbump encountered by users of these long term stable releases. For \*nix systems, this PEP's formal endorsement of ``pip`` as the preferred default packaging tool is seen as more important than the underlying technical details involved in making ``pip`` available by default, since it shifts the nature of the conversation between the developers of ``pip`` and downstream repackagers of both ``pip`` and CPython. For Python 2.7, on the other hand, the compatibility issue for adopting new metadata standards is far more widespread, as it affects the python.org binary installers for Windows and Mac OS X, as well as even relatively fast moving \*nix platforms. Firstly, and unlike Python 2.6, Python 2.7 is still a fully supported upstream version, and will remain so until the release of Python 2.7.9 (currently scheduled for May 2015), at which time it is expected to enter the usual "security fix only" mode. That means there are at least another 19 months where Python 2.7 is a deployment target for Python applications that enjoys full upstream support. Even after the core development team switches 2.7 to security release only mode in 2015, Python 2.7 will likely remain a commercially supported legacy target out beyond 2020. While Python 3 already presents a compelling alternative over Python 2 for *new* Python applications and deployments without an existing investment in Python 2 and without a dependency on specific Python 2 only third party modules (a set which is getting ever smaller over time), it is going to take longer to create compelling business cases to update existing Python 2.7 based infrastructure to Python 3, especially in situations where the culture of automated testing is weak (or nonexistent), making it difficult to effectively use the available migration utilities. It is quite likely that it is this difference in perspective regarding what it means for a version of Python to be "supported" which lies at the heart of the long history of conflicts between the developers of Python packaging tools and the core development team for CPython. A key goal of this PEP is thus to better enable the two groups to collaborate more effectively, by using the ``ensurepip`` module as the technical bridge between the two distinct software lifecycles and deployment models. Why pip? -------- ``pip`` has been chosen as the preferred default installer, as it is an already popular tool that addresses several design and user experience issues with its predecessor ``easy_install`` (these issues can't readily be fixed in ``easy_install`` itself due to backwards compatibility concerns). ``pip`` is also well suited to working within the bounds of a single Python runtime installation (including associated virtual environments), which is a desirable feature for a tool bundled with CPython. Other tools like ``zc.buildout`` and ``conda`` are more ambitious in their aims (and hence substantially better than ``pip`` at handling external binary dependencies), so it makes sense for the Python ecosystem to treat them more like platform package managers to interoperate with rather than as the default cross-platform installation tool. This relationship is similar to that between ``pip`` and platform package management systems like ``apt`` and ``yum`` (which are also designed to handle arbitrary binary dependencies). Proposal Overview ================= This PEP proposes that the `Installing Python Modules <http://docs.python.org/3/install>`__ guide be updated to officially recommend the use of ``pip`` as the default installer for Python packages, rather than the current approach of recommending the direct invocation of the ``setup.py install`` ``distutils`` command. However, to avoid recommending a tool that CPython does not provide, it is further proposed that the `pip`_ package manager be made available by default when installing CPython and when creating virtual environments using the standard library's ``venv`` module via the ``pyvenv`` command line utility). To support that end, this PEP proposes the inclusion of an ``ensurepip`` bootstrapping module in Python 3.4 (along with the upcoming maintenance releases of Python 2.7 and 3.3), as well as changes to the way Python installed scripts are handled on Windows (for Python 3.4 only). To clearly demarcate development responsibilities, and to avoid inadvertently downgrading ``pip`` when updating CPython, the proposed mechanism to achieve this is to include an explicit `pip`_ bootstrapping mechanism in the standard library that is invoked automatically by the CPython installers provided on python.org. To ensure the smoothest possible experience for new users of Python, this PEP also proposes that the ``ensurepip`` module and the option to install ``pip`` when installing CPython be backported to Python 2.7 and 3.3. It does *not* propose backporting any changes to ``pyvenv`` (in Python 3.3) or to Windows script handling (in either version). Finally, the PEP also strongly recommends that CPython redistributors and other Python implementations ensure that ``pip`` is available by default, or at the very least, explicitly document the fact that it is not included. This PEP does *not* propose making pip (or any dependencies) directly available as part of the standard library. Instead, pip will be a bundled application provided along with CPython for the convenience of Python users, but subject to its own development life cycle and able to be upgraded independently of the core interpreter and standard library. Explicit bootstrapping mechanism ================================ An additional module called ``ensurepip`` will be added to the standard library whose purpose is to install pip and any of its dependencies into the appropriate location (most commonly site-packages). It will expose a callable named ``bootstrap()`` as well as offer direct execution via ``python -m ensurepip``. The bootstrap will *not* contact PyPI, but instead rely on a private copy of pip stored inside the standard library. Accordingly, only options related to the installation location will be supported (``--user``, ``--root``, etc). It is considered desirable that users be strongly encouraged to use the latest available version of ``pip``, in order to take advantage of the ongoing efforts to improve the security of the PyPI based ecosystem, as well as benefiting from the efforts to improve the speed, reliability and flexibility of that ecosystem. In order to satisfy this goal of providing the most recent version of ``pip`` by default, the private copy of ``pip`` will be updated in CPython maintenance releases, which should align well with the 6-month cycle used for new ``pip`` releases. Security considerations ----------------------- The design in this PEP has been deliberately chosen to avoid making any significant changes to the trust model of the CPython installers for end users that do not subsequently make use of ``pip``. The installers will contain all the components of a fully functioning version of Python, including the ``pip`` installer. The installation process will *not* require network access, and will *not* rely on trusting the security of the network connection established between ``pip`` and the Python package index. Only users that choose to use ``pip`` directly will need to pay attention to any PyPI related security considerations. Reliability considerations -------------------------- By including the bootstrap as part of the standard library (rather than solely as a feature of the binary installers), the correct operation of the bootstrap command can be easily tested using the existing CPython buildbot infrastructure rather than adding significantly to the testing burden for the installers themselves. Implementation strategy ----------------------- To ensure there is no need for network access when installing Python or creating virtual environments, the ``ensurepip`` module will, as an implementation detail, include a complete private copy of pip and its dependencies which will be used to extract pip and install it into the target environment. It is important to stress that this private copy of pip is *only* an implementation detail and it should *not* be relied on or assumed to exist beyond the public capabilities exposed through the ``ensurepip`` module (and indirectly through ``venv``). There is not yet a reference ``ensurepip`` implementation. The existing ``get-pip.py`` bootstrap script demonstrates an earlier variation of the general concept, but the standard library version would take advantage of the improved distribution capabilities offered by the CPython installers to include private copies of ``pip`` and ``setuptools`` as wheel files (rather than as embedded base64 encoded data), and would not try to contact PyPI (instead installing directly from the private wheel files. Rather than including separate code to handle the bootstrapping, the ``ensurepip`` module will manipulate ``sys.path`` appropriately to allow the wheel files to be used to install themselves, either into the current Python installation or into a virtual environment (as determined by the options passed to the bootstrap command). It is proposed that the implementation be carried out in five separate steps (all steps after the first are independent of each other and can be carried out in any order): * the first step would add the ``ensurepip`` module and the private copies of the most recently released versions of pip and setuptools, and update the "Installing Python Modules" documentation. This change would be applied to Python 2.7, 3.3, and 3.4. * the Windows installer would be updated to offer the new ``pip`` installation option for Python 2.7.6, 3.3.3 and 3.4.0. * the Mac OS X installer would be updated to offer the new ``pip`` installation option for Python 2.7.6, 3.3.3 and 3.4.0. * the ``venv`` module and ``pyvenv`` command would be updated to make use of ``ensurepip`` in Python 3.4+ * the PATH handling and ``sysconfig`` directory layout on Windows would be updated for Python 3.4+ Proposed CLI ------------ The proposed CLI is based on a subset of the existing ``pip install`` options:: Usage: python -m ensurepip [options] General Options: -h, --help Show help. -v, --verbose Give more output. Option is additive, and can be used up to 3 times. -V, --version Show the pip version that would be extracted and exit. -q, --quiet Give less output. Installation Options: -U, --upgrade Upgrade pip and dependencies, even if already installed --user Install using the user scheme. --root <dir> Install everything relative to this alternate root directory. In most cases, end users won't need to use this CLI directly, as ``pip`` should have been installed automatically when installing Python or when creating a virtual environment. However, it is formally documented as a public interface to support at least these known use cases: * Windows and Mac OS X installations where the "Install pip" option was *not* chosen during installation * any installation where the user previously ran "pip uninstall pip" * virtual environments created with ``pyvenv`` in Python 3.3 Users that want to retrieve the latest version from PyPI, or otherwise need more flexibility, can then invoke the extracted ``pip`` appropriately. Proposed module API ------------------- The proposed ``ensurepip`` module API consists of the following two functions:: def version(): """ Returns a string specifying the bundled version of pip. """ def bootstrap(root=None, upgrade=False, user=False, verbosity=0): """ Bootstrap pip into the current Python installation (or the given root directory). """ Invocation from the CPython installers -------------------------------------- The CPython Windows and Mac OS X installers will each gain a new option: * Install pip (the default Python package management utility)? This option will be checked by default. If the option is checked, then the installer will invoke the following command with the just installed Python:: python -m ensurepip --upgrade This ensures that, by default, installing or updating CPython will ensure that the installed version of pip is at least as recent as the one included with that version of CPython. If a newer version of pip has already been installed then ``python -m ensurepip --upgrade`` will simply return without doing anything. Installing from source ---------------------- While the prebuilt binary installers will be updated to run ``python -m ensurepip`` by default, no such change will be made to the ``make install`` and ``make altinstall`` commands of the source distribution. ``ensurepip`` itself (including the private copy of ``pip`` and its dependencies) will still be installed normally (as it is a regular part of the standard library), only the implicit installation of pip and its dependencies will be skipped. Keeping the pip bootstrapping as a separate step for ``make``-based installations should minimize the changes CPython redistributors need to make to their build processes. Avoiding the layer of indirection through ``make`` for the ``ensurepip`` invocation avoids any challenges associated with determining where to install the extracted ``pip``. Changes to virtual environments ------------------------------- Python 3.3 included a standard library approach to virtual Python environments through the ``venv`` module. Since its release it has become clear that very few users have been willing to use this feature directly, in part due to the lack of an installer present by default inside of the virtual environment. They have instead opted to continue using the ``virtualenv`` package which *does* include pip installed by default. To make the ``venv`` more useful to users it will be modified to issue the pip bootstrap by default inside of the new environment while creating it. This will allow people the same convenience inside of the virtual environment as this PEP provides outside of it as well as bringing the ``venv`` module closer to feature parity with the external ``virtualenv`` package, making it a more suitable replacement. To handle cases where a user does not wish to have pip bootstrapped into their virtual environment a ``--without-pip`` option will be added. The ``venv.EnvBuilder`` and ``venv.create`` APIs will be updated to accept one new parameter: ``with_pip`` (defaulting to ``False``). The new default for the module API is chosen for backwards compatibility with the current behaviour (as it is assumed that most invocation of the ``venv`` module happens through third part tools that likely will not want ``pip`` installed without explicitly requesting it), while the default for the command line interface is chosen to try to ensure ``pip`` is available in most virtual environments without additional action on the part of the end user. This particular change will be made only for Python 3.4 and later versions. The third-party ``virtualenv`` project will still be needed to obtain a consistent cross-version experience in Python 3.3 and 2.7. Documentation ------------- The "Installing Python Modules" section of the standard library documentation in Python 2.7, 3.3 and 3.4 will be updated to recommend the use of the bootstrapped ``pip`` installer. It will give a brief description of the most common commands and options, but delegate to the externally maintained ``pip`` documentation for the full details. In Python 3.4, the ``pyvenv`` and ``venv`` documentation will also be updated to reference the revised module installation guide. In Python 2.7 and 3.3, the documentation will make clear that the feature was added in a maintenance release and users may need to upgrade in order to take advantage of it. Specifically, it is proposed to include the following warning as a note in the documentation for the ``ensurepip`` module in these versions (adjust version numbers for 3.3 as appropriate): This is an optional module, which may not be available in all installations of Python 2.7. It is provided solely to simplify the process of bootstrapping ``pip`` onto end user's systems. If it is not available, please investigate the following alternatives: * This module was first added in Python 2.7.6. If using an earlier maintenance release, it will not be available. If upgrading to a more recent maintenance release is not an option, consider the alternative bootstrapping mechanisms below. * Some platforms provide alternative mechanisms to obtain ``pip``. In such cases, the platform documentation should provide appropriate details. * If upgrading to the latest maintenance release is not feasible, and no platform specific instructions are provided, then refer to the upstream `pip bootstrapping instructions <http://www.pip-installer.org/en/latest/installing.html>`__. The existing content of the module installation guide will be retained in all versions, but under a new "Invoking distutils directly" subsection. Bundling CA certificates with CPython ------------------------------------- The ``ensurepip`` implementation will include the ``pip`` CA bundle along with the rest of ``pip``. This means CPython effectively includes a CA bundle that is used solely by ``pip`` after it has been extracted. This is considered preferable to relying solely on the system certificate stores, as it ensures that ``pip`` will behave the same across all supported versions of Python, even those prior to Python 3.4 that cannot access the system certificate store on Windows. Automatic installation of setuptools ------------------------------------ ``pip`` currently depends on ``setuptools`` to handle metadata generation during the build process, along with some other features. While work is ongoing to reduce or eliminate this dependency, it is not clear if that work will be complete for pip 1.5 (which is the version likely to be current when Python 3.4.0 is released). This PEP proposes that, if pip still requires it as a dependency, ``ensurepip`` will include a private copy of ``setuptools`` (in addition to the private copy of ``ensurepip``). ``python -m ensurepip`` will then install the private copy in addition to installing ``pip`` itself. However, this behavior is officially considered an implementation detail. Other projects which explicitly require ``setuptools`` must still provide an appropriate dependency declaration, rather than assuming ``setuptools`` will always be installed alongside ``pip``. Once pip is able to run ``pip install --upgrade pip`` without needing ``setuptools`` installed first, then the private copy of ``setuptools`` will be removed from ``ensurepip`` in subsequent CPython releases. As long as setuptools is needed, it will be a completely unmodified copy of the latest upstream setuptools release, including the ``easy_install`` script if the upstream setuptools continues to include it. The installation of ``easy_install`` along with ``pip`` isn't considered desirable, but installing a broken setuptools would be worse. This problem will naturally resolve itself once the ``pip`` developers have managed to eliminate their dependency on ``setuptools`` and the private copy of ``setuptools`` can be removed entirely from CPython. Updating the private copy of pip -------------------------------- In order to keep up with evolutions in packaging as well as providing users with as recent version a possible the ``ensurepip`` module will be regularly updated to the latest versions of everything it bootstraps. After each new ``pip`` release, and again during the preparation for any release of Python (including feature releases), a script, provided as part of the implementation for this PEP, will be run to ensure the private copies stored in the CPython source repository have been updated to the latest versions. Updating the ensurepip module API and CLI ----------------------------------------- Like ``venv`` and ``pyvenv``, the ``ensurepip`` module API and CLI will be governed by the normal rules for the standard library: no new features are permitted in maintenance releases. However, the embedded components may be updated as noted above, so the extracted ``pip`` may offer additional functionality in maintenance releases. Feature addition in maintenance releases ======================================== Adding a new module to the standard library in Python 2.7, and 3.3 maintenance releases breaks the usual policy of "no new features in maintenance releases". The rationale for doing so in this case is slightly different for each of the two versions. Rationale for this policy on maintenance releases ------------------------------------------------- Python's strict "no new features in maintenance releases" was instituted following the introduction of a number of new features over the course of the Python 2.2.x series. Firstly, the ``True`` and ``False`` builtins were added in Python 2.2.1 (at the time, they were merely aliases for the values ``1`` and ``0``, in Python 2.3 they became instances of the new ``bool`` type and in Python 3.0 they became true constants recognised by the compiler). Python 2.2.2 then made the situation worse by adding a new ``chars`` parameter to the ``lstrip`` and ``rstrip`` string methods, along with an entirely new ``zfill`` method. The corresponding changes in the ``string`` module were not incorporated until Python 2.2.3. The reason introducing new features in maintenance releases like this is problematic is that, except in the cases where code fails due to a bug in CPython, developers expect to be able to identify the supported Python versions for a library or application solely through the first two components of the version number. The introduction of new builtins and string methods in Python 2.2.1 and 2.2.2 resulted in many developers claiming Python 2.2 compatibility for code that did not in fact run on the original Python 2.2. In effect, Python 2.2.2 became the minimum usable version, since there was a relatively high chance of code breaking when run on 2.2 (or even 2.2.1). Scope of this proposal ---------------------- By contrast with the changes that caused such problems during the 2.2.x series, this PEP is merely proposing the addition of a new standard library module, rather than adding new builtins or changing the interface of a builtin type. The categorical difference between these kinds of changes has already been recognised in the Python 3.3 Language Moratorium (PEP 3003), where the addition of new builtins was disallowed outright and changes to builtin types required an explicit exemption. By contrast, adding new modules was explicitly permitted, even while the moratorium was in place. Furthermore, the proposed ``ensurepip`` module is only a means to the end of getting ``pip`` installed on the system. While "upgrade to the latest CPython maintenance release" will become the *recommended* approach to obtaining ``pip`` for users of Python 2.7 and 3.3 on Windows and Mac OS X systems, all of the existing ``pip`` bootstrapping mechanisms will still work in cases where upgrading Python isn't a practical alternative. As described in the documentation update proposal, the ``ensurepip`` documentation in older releases will include the text explaining how to obtain ``pip`` if updating to the latest maintenance release isn't an option, or if the module has been removed by a redistributor. This contrasts significantly with the changes made during the Python 2.2 series, where they were normal additions with no alternatives except to update to a sufficiently recent version of Python if a library or application depended on them. Potential consequences of permitting this exemption --------------------------------------------------- The concern has been expressed that approving an exemption to the "no new features in maintenance releases" policy in this case will open the flood gates to requests for more such exemptions in the future. It is the perspective of the PEP authors that the specific nature of this proposal should itself serve to allay those fears. Firstly, as a proposal to add a new module to the standard library, granting an exemption in this case sets no precedent for the more restricted categories identified in the PEP 3003 language moratorium. Secondly, this exemption is requested for a new module that *makes it easy to download other modules from PyPI*. If this PEP is accepted, then it can be reasonably assumed that modules on PyPI are only a ``pip install`` away for most users, with only those users that depend on standard library inclusion to make it through corporate compliance reviews still affected (and, for many such reviews, inclusion in a future version of the standard library will be enough for a backported version to be considered acceptable for use). Making ``pip`` readily available in all versions still under normal maintenance thus means that accepting this PEP should have the effect of *weakening* the case for any further exemptions to the policy, rather than strengthening it. Rationale for permitting the exemption in Python 2.7 ---------------------------------------------------- While Python 3 adoption is proceeding nicely, it remains the case that many new users of Python are introduced to Python 2.7 first. This may be because their instructors have yet to migrate their courses to Python 3, or because they work in an environment where Python 2 is still the preferred version, or simply because the frequently adopted approach of writing libraries in the common Python 2/3 subset means there are (as of September 2013) still more Python 2 only libraries than there are Python 3 only libraries. Since one of the primary aims of this PEP is to aid new Python users, it is contrary to its spirit to target *only* Python 3.4, when so many users in at least the next 12-18 months (where Python 2.7 is still fully supported by the core development team) are still going to be introduced to Python 2 before being introduced to Python 3. Users first installing Python 2.7 on Windows and Mac OS X following acceptance and release of this PEP won't even need to look up how to bootstrap ``pip``, since it will already be provided with the CPython installer. For those that already have Python installed, but are just beginning to explore the PyPI ecosystem, the bootstrapping instructions can be simplified to "just install the latest maintenance release of CPython". Making ``pip`` readily available also serves to ease the migration path from Python 2 to Python 3, as a number of the standard library additions in Python 3 are also available from PyPI for Python 2. Lowering the barrier to adoption for these backports makes it easier for current Python 2 users to selectively adopt backporting Python 3 versions, reducing the number of updates needed in any eventual Python 3 migration. Finally, this PEP solves a serious problem for the ``distutils-sig`` community, as it means we will finally have a standard mechanism decoupled from the standard library's development lifecycle that we can reasonably assume to be present on end user's systems (or at least readily available) that allows us to support new packaging standards in older versions of Python. A tentative, half-hearted endorsement from the CPython core development team that tries to hide the existence of the pip boostrapping support from end users is unlikely to provide quite the same benefits. Rationale for permitting the exemption in Python 3.3 ---------------------------------------------------- The rationale for permitting the exemption in Python 3.3 is admittedly not as strong as it is for Python 2.7, as instructors currently using Python 3.3 are quite likely to upgrade to Python 3.4 shortly after it is released. In the case of Python 3.3, the rationale is primarily a consistency argument, as it permits the recommended ``pip`` bootstrapping instructions for both 2.7 and 3.3 to be to upgrade to the latest maintenance version of CPython. While existing bootstrapping mechanisms will still be supported, the cases where they are needed should be reduced significantly. Adding the ``ensurepip`` module in Python 3.3 also makes the Python 3.3 version of the ``pyvenv`` utility far more useful (even without the integration proposed for Python 3.4), as it allows users to execute ``python -m ensurepip`` to bootstrap ``pip`` after activating an existing or newly created virtual environment. Uninstallation ============== No changes are proposed to the CPython uninstallation process by this PEP. The bootstrapped pip will be installed the same way as any other pip installed packages, and will be handled in the same way as any other post-install additions to the Python environment. At least on Windows, that means the bootstrapped files will be left behind after uninstallation, since those files won't be associated with the Python MSI installer. While the case can be made for the CPython installers clearing out these directories automatically, changing that behaviour is considered outside the scope of this PEP. Script Execution on Windows =========================== While the Windows installer was updated in Python 3.3 to optionally make ``python`` available on the PATH, no such change was made to include the Scripts directory. Independently of this PEP, a proposal has also been made to rename the ``Tools\Scripts`` subdirectory to ``bin`` in order to improve consistency with the typical script installation directory names on \*nix systems. Accordingly, in addition to adding the option to extract and install ``pip`` during installation, this PEP proposes that the Windows installer (and ``sysconfig``) in Python 3.4 and later be updated to: - install scripts to PythonXY\bin rather than PythonXY\Tools\Scripts - add PythonXY\bin to the Windows PATH (in addition to PythonXY) when the PATH modification option is enabled during installation For Python 2.7 and 3.3, it is proposed that the only change be the one to bootstrap ``pip`` by default. This means that, for Python 3.3, the most reliable way to invoke pip globally on Windows (without tinkering manually with PATH) will actually be ``py -m pip`` (or ``py -3 -m pip`` to select the Python 3 version if both Python 2 and 3 are installed) rather than simply calling ``pip``. For Python 2.7 and 3.2, the most reliable mechanism will be to install the standalone Python launcher for Windows and then use ``py -m pip`` as noted above. Adding the scripts directory to the system PATH would mean that ``pip`` works reliably in the "only one Python installation on the system PATH" case, with ``py -m pip``, ``pipX``, or ``pipX.Y`` needed only to select a non-default version in the parallel installation case (and outside a virtual environment). This change should also make the ``pyvenv`` command substantially easier to invoke on Windows, along with all scripts installed by ``pip``, ``easy_install`` and similar tools. While the script invocations on recent versions of Python will run through the Python launcher for Windows, this shouldn't cause any issues, as long as the Python files in the Scripts directory correctly specify a Python version in their shebang line or have an adjacent Windows executable (as ``easy_install`` and ``pip`` do). Recommendations for Downstream Distributors =========================================== A common source of Python installations are through downstream distributors such as the various Linux Distributions [#ubuntu]_ [#debian]_ [#fedora]_, OSX package managers [#homebrew]_ [#macports]_ [#fink]_, and commercial Python redistributors [#ContinuumIO]_ [#ActiveState]_ [#Enthought]_. In order to provide a consistent, user-friendly experience to all users of Python regardless of how they obtained Python this PEP recommends and asks that downstream distributors: * Ensure that whenever Python is installed ``pip`` is either installed or is otherwise made readily available to end users. * For redistributors using binary installers, this may take the form of optionally executing the ``ensurepip`` bootstrap during installation, similar to the CPython installers. * For redistributors using package management systems, it may take the form of separate packages with dependencies on each other so that installing the Python package installs the pip package and installing the pip package installs the Python package. * Another reasonable way to implement this is to package pip separately but ensure that there is some sort of global hook that will recommend installing the separate pip package when a user executes ``pip`` without it being installed. Systems that choose this option should ensure that the ``ensurepip`` module still installs pip directly when invoked inside a virtual environment, but may modify the module in the system Python installation to redirect to the platform provided mechanism when installing ``pip`` globally. * Even if pip is made available globally by other means, do not remove the ``ensurepip`` module in Python 3.3 or later. * In Python 3.3, ``ensurepip`` will be the recommended way of bootstrapping pip in virtual environments created through the ``venv`` module and the associated ``pyvenv`` command line tool. * Starting with Python 3.4, ``ensurepip`` will be required for automatic installation of pip into virtual environments by the ``venv`` module. * This is similar to the existing ``virtualenv`` package for which many downstream distributors have already made exception to the common "debundling" policy. * This does mean that if ``pip`` needs to be updated due to a security issue, so does the private copy in the ``ensurepip`` bootstrap module * However, altering the private copy of pip to remove the embedded CA certificate bundle and rely on the system CA bundle instead is a reasonable change. * If ``pip`` is made available globally by other means in Python 2.7, then it is acceptable (although not desirable) to disable the ``ensurepip`` module (as the third party ``virtualenv`` distribution is needed to create virtual environments in Python 2.7 and ``virtualenv`` already ensures ``pip`` is installed into the virtual environments it creates). Redistributors that take this course should ensure an appropriate error message is displayed if users attempt to import ``ensurepip``, rather than simply removing it entirely. * Ensure that all features of this PEP continue to work with any modifications made to the redistributed version of Python. * Checking the version of pip that will be bootstrapped using ``python -m ensurepip --version`` or ``ensurepip.version()``. * Installation of pip into a global or virtual python environment using ``python -m ensurepip`` or ``ensurepip.bootstrap()``. * ``pip install --upgrade pip`` in a global installation should not affect any already created virtual environments (but is permitted to affect future virtual environments, even though it will not do so when using the standard implementation of ``ensurepip``). * ``pip install --upgrade pip`` in a virtual environment should not affect the global installation. * Migrate build systems to utilize `pip`_ and `Wheel`_ wherever feasible and avoid directly invoking ``setup.py``. * This will help ensure a smoother and more timely migration to improved metadata formats as the Python packaging ecosystem continues to evolve. In the event that a Python redistributor chooses *not* to follow these recommendations, we request that they explicitly document this fact and provide their users with suitable guidance on translating upstream ``pip`` based installation instructions into something appropriate for the platform. Other Python implementations are also encouraged to follow these guidelines where applicable. Policies & Governance ===================== The maintainers of the bootstrapped software and the CPython core team will work together in order to address the needs of both. The bootstrapped software will still remain external to CPython and this PEP does not include CPython subsuming the development responsibilities or design decisions of the bootstrapped software. This PEP aims to decrease the burden on end users wanting to use third-party packages and the decisions inside it are pragmatic ones that represent the trust that the Python community has already placed in the Python Packaging Authority as the authors and maintainers of ``pip``, ``setuptools``, PyPI, ``virtualenv`` and other related projects. Backwards Compatibility ----------------------- The public API and CLI of the ``ensurepip`` module itself will fall under the typical backwards compatibility policy of Python for its standard library. The externally developed software that this PEP bundles does not. Most importantly, this means that the bootstrapped version of pip may gain new features in CPython maintenance releases, and pip continues to operate on its own 6 month release cycle rather than CPython's 18-24 month cycle. Security Releases ----------------- Any security update that affects the ``ensurepip`` module will be shared prior to release with the Python Security Response Team (security at python.org). The PSRT will then decide if the reported issue warrants a security release of CPython with an updated private copy of ``pip``. Licensing --------- ``pip`` is currently licensed as 1 Clause BSD, and it contains code taken from other projects. Additionally this PEP will include setuptools until such time as pip no longer requires it. The licenses for these appear in the table below. ================= ============ Project License ================= ============ requests Apache 2.0 six 1 Clause BSD html5lib 1 Clause BSD distlib PSF colorama 3 Clause BSD Mozilla CA Bundle LGPL setuptools PSF ================= ============ All of these licenses should be compatible with the PSF license. Additionally it is unclear if a CA Bundle is copyrightable material and thus if it needs or can be licensed at all. Appendix: Rejected Proposals ============================ Include pip *only* inside the installers in Python 2.7, and 3.3 --------------------------------------------------------------- An alternative to making an exception to the "no new features" policy in Python 2.7 and 3.3 would be to simply bundle pip with the installer and not modify the source tree at all. The motivation behind this modification is that adding a new feature in a maintenance release is a risky proposition and that doing it in this way doesn't violate that policy. This has been rejected because: * It's dubious to declare the binary installers beyond the scope of the "no new features in maintenance releases" policy. If the rationale for adding this feature to the standard library in a maintenance release isn't considered adequate, then it isn't clear why moving that complexity to the binary installers should change the verdict. * Attempting to hide the existence of the bootstrap module from end users makes it more difficult to write updated package installation documentation for Python 2.7 and 3.3 * For 3.3 users that choose to use ``pyvenv`` rather than ``virtualenv``, an explicit ``python -m ensurepip`` will be needed to bootstrap ``pip`` into virtual environments. This can only be documented clearly if the module is public * Making the bootstrap an installer only feature in Python 2.7 and 3.3 guarantees the introduction of cross-platform inconsistencies, whereas the proposal in this PEP more strongly encourages redistributors to offer a more consistent user experience. * Making the bootstrap an installer only feature in Python 2.7 and 3.3 would make it difficult to re-use the bootstrap implementation from 3.4. * Making the bootstrap an installer only feature prevents the buildbots from being able to run automatic tests against it, which would make ensuring that this feature remains working a much more difficult task. Use a different module name in Python 2.7, and 3.3 -------------------------------------------------- Naming the module ``_ensurepip`` in Python 2.7 and 3.3 was considered as another means of skirting the "no new features in maintenance releases" policy. However, similar to the proposal to only include the new feature in the installers rather than the standard library, this feels like relying on a technicality to nominally "comply" with the policy, while still breaking it in spirit. It is the considered opinion of the PEP authors that attempting to hide the addition of the ``ensurepip`` module in earlier versions will only serve to increase confusion rather than to reduce it, so the proposal remains to be up front about the fact that the policy is being broken in this case, and clearly documenting the rationale for doing so in this PEP. As noted in the section describing the proposed documentation updates, having ``ensurepip`` as a public module in these earlier versions also provides a convenient home for the fallback bootstrapping instructions in those cases where it *isn't* available. Automatically contacting PyPI when bootstrapping pip ---------------------------------------------------- Earlier versions of this PEP called the bootstrapping module ``getpip`` and defaulted to downloading and installing ``pip`` from PyPI, with the private copy used only as a fallback option or when explicitly requested. This resulted in several complex edge cases, along with difficulties in defining a clean API and CLI for the bootstrap module. It also significantly altered the default trust model for the binary installers published on python.org, as end users would need to explicitly *opt-out* of trusting the security of the PyPI ecosystem (rather than opting in to it by explicitly invoking ``pip`` following installation). As a result, the PEP was simplified to the current design, where the bootstrapping *always* uses the private copy of ``pip``. Contacting PyPI is now always an explicit separate step, with direct access to the full pip interface. Implicit bootstrap ------------------ `PEP439`_, the predecessor for this PEP, proposes its own solution. Its solution involves shipping a fake ``pip`` command that when executed would implicitly bootstrap and install pip if it does not already exist. This has been rejected because it is too "magical". It hides from the end user when exactly the pip command will be installed or that it is being installed at all. It also does not provide any recommendations or considerations towards downstream packagers who wish to manage the globally installed pip through the mechanisms typical for their system. The implicit bootstrap mechanism also ran into possible permissions issues, if a user inadvertently attempted to bootstrap pip without write access to the appropriate installation directories. Including pip directly in the standard library ---------------------------------------------- Similar to this PEP is the proposal of just including pip in the standard library. This would ensure that Python always includes pip and fixes all of the end user facing problems with not having pip present by default. This has been rejected because we've learned, through the inclusion and history of ``distutils`` in the standard library, that losing the ability to update the packaging tools independently can leave the tooling in a state of constant limbo. Making it unable to ever reasonably evolve in a time frame that actually affects users as any new features will not be available to the general population for *years*. Allowing the packaging tools to progress separately from the Python release and adoption schedules allows the improvements to be used by *all* members of the Python community and not just those able to live on the bleeding edge of Python releases. There have also been issues in the past with the "dual maintenance" problem if a project continues to be maintained externally while *also* having a fork maintained in the standard library. Since external maintenance of ``pip`` will always be needed to support earlier Python versions, the proposed bootstrapping mechanism will becoming the explicit responsibility of the CPython core developers (assisted by the pip developers), while pip issues reported to the CPython tracker will be migrated to the pip issue tracker. There will no doubt still be some user confusion over which tracker to use, but hopefully less than has been seen historically when including complete public copies of third-party projects in the standard library. The approach described in this PEP also avoids some technical issues related to handling CPython maintenance updates when pip has been independently updated to a more recent version. The proposed pip-based bootstrapping mechanism handles that automatically, since pip and the system installer never get into a fight about who owns the pip installation (it is always managed through pip, either directly, or indirectly via the ``ensurepip`` bootstrap module). Finally, the separate bootstrapping step means it is also easy to avoid installing ``pip`` at all if end users so desire. This is often the case if integrators are using system packages to handle installation of components written in multiple languages using a common set of tools. Defaulting to --user installation --------------------------------- Some consideration was given to bootstrapping pip into the per-user site-packages directory by default. However, this behavior would be surprising (as it differs from the default behavior of pip itself) and is also not currently considered reliable (there are some edge cases which are not handled correctly when pip is installed into the user site-packages directory rather than the system site-packages). .. _Wheel: http://www.python.org/dev/peps/pep-0427/ .. _pip: http://www.pip-installer.org .. _setuptools: https://pypi.python.org/pypi/setuptools .. _PEP439: http://www.python.org/dev/peps/pep-0439/ References ========== .. [1] Discussion thread 1 (distutils-sig) (https://mail.python.org/pipermail/distutils-sig/2013-August/022529.html) .. [2] Discussion thread 2 (distutils-sig) (https://mail.python.org/pipermail/distutils-sig/2013-September/022702.html) .. [3] Discussion thread 3 (python-dev) (https://mail.python.org/pipermail/python-dev/2013-September/128723.html) .. [4] Discussion thread 4 (python-dev) (https://mail.python.org/pipermail/python-dev/2013-September/128780.html) .. [5] Discussion thread 5 (python-dev) (https://mail.python.org/pipermail/python-dev/2013-September/128894.html) .. [#ubuntu] `Ubuntu <http://www.ubuntu.com/>` .. [#debian] `Debian <http://www.debian.org>` .. [#fedora] `Fedora <https://fedoraproject.org/>` .. [#homebrew] `Homebrew <http://brew.sh/>` .. [#macports] `MacPorts <http://macports.org>` .. [#fink] `Fink <http://finkproject.org>` .. [#ContinuumIO] `Anaconda <https://store.continuum.io/cshop/anaconda/>` .. [#ActiveState] `ActivePython <http://www.activestate.com/activepython>` .. [#Enthought] `Enthought Canopy <https://www.enthought.com/products/canopy/>` Copyright ========= This document has been placed in the public domain. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From zhangpeipei812 at outlook.com Sun Sep 29 14:51:37 2013 From: zhangpeipei812 at outlook.com (=?UTF-8?B?5byg5L2p5L2p?=) Date: Sun, 29 Sep 2013 20:51:37 +0800 Subject: [Python-Dev] Why not support user defined operator overloading ? Message-ID: <BLU403-EAS393D1EF893F5F5DC86003229D2B0@phx.gbl> Hello: As far as I know, there is not a language support user defined operator overloading. Python3 can overloading belowed operators. - negated + unchanged - minus + add * multiplication / division // true division % remainder ** power (Do I miss something ?) If we can overloading these operators, why we can't overloading other operators? (like .* often used in matrix, U in set operation) Regards! Peipei From python-dev at masklinn.net Sun Sep 29 17:06:32 2013 From: python-dev at masklinn.net (Xavier Morel) Date: Sun, 29 Sep 2013 17:06:32 +0200 Subject: [Python-Dev] Why not support user defined operator overloading ? In-Reply-To: <BLU403-EAS393D1EF893F5F5DC86003229D2B0@phx.gbl> References: <BLU403-EAS393D1EF893F5F5DC86003229D2B0@phx.gbl> Message-ID: <F53492D8-A90B-4449-9E83-25340AA63AEA@masklinn.net> On 2013-09-29, at 14:51 , ??? wrote: > Hello: > As far as I know, there is not a language support user defined operator overloading. > Python3 can overloading belowed operators. > - negated > + unchanged > > - minus > + add > * multiplication > / division > // true division > % remainder > ** power > (Do I miss something ?) ~ invert (unary) () call . get attribute [] get item << left shift >> right shift & binary and ^ xor | binary or And the inplace versions of most of these can be implemented separately, which can probably be counted as supplementary operators. > > If we can overloading these operators, why we can't overloading other operators? > (like .* often used in matrix, U in set operation) This is more of a python-ideas subject. And one of the reasons likely is that it would require significantly reworking the grammar to handle a kind of user-defined opname (similar to name, but for operator tokens), with user-defined priority and associativity, and the ability to import operators (or define how and when operators become available compared to their definition) That's a huge amount of complexity with little to gain. From steve at pearwood.info Sun Sep 29 17:33:38 2013 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 30 Sep 2013 01:33:38 +1000 Subject: [Python-Dev] Why not support user defined operator overloading ? In-Reply-To: <BLU403-EAS393D1EF893F5F5DC86003229D2B0@phx.gbl> References: <BLU403-EAS393D1EF893F5F5DC86003229D2B0@phx.gbl> Message-ID: <20130929153338.GB7989@ando> On Sun, Sep 29, 2013 at 08:51:37PM +0800, ??? wrote: > Hello: > As far as I know, there is not a language support user defined operator overloading. > Python3 can overloading belowed operators. [...] > (Do I miss something ?) Yes, many. http://docs.python.org/3/reference/datamodel.html#special-method-names http://docs.python.org/3/reference/datamodel.html#emulating-numeric-types > If we can overloading these operators, why we can't overloading other operators? > (like .* often used in matrix, U in set operation) http://www.python.org/dev/peps/pep-0211/ http://www.python.org/dev/peps/pep-0225/ There is a good argument for being able to overload element-wise and object-wise operators. However, I don't think it is good to be able to create arbitrary new operators, e.g. a X b for an X operator. We already have syntax for arbitrary functions, and they aren't limited to binary operators: X(a, b, c, d, ...) works better than a X b for most uses. -- Steven From guido at python.org Sun Sep 29 18:54:39 2013 From: guido at python.org (Guido van Rossum) Date: Sun, 29 Sep 2013 09:54:39 -0700 Subject: [Python-Dev] Getting Tulip (PEP 3156) into the 3.4 stdlib, marked provisional, named asyncio In-Reply-To: <CADiSq7ea-4YWBwRS_6v1m6fC0u1yTCu081ofeCuCUQQ4iTMX=w@mail.gmail.com> References: <CAP7+vJKSOHYeCqhsJVF_7epTDeOw4UP3crsK_mY7HaFYnwG1aA@mail.gmail.com> <CADiSq7c22aKMqW_u1GfLjA=uFuGR3e83R-ynVzpbG_nH9Am+2w@mail.gmail.com> <CAP7+vJJfMP_1DFDc8dSGJo=JiE2hPGK_2AzrWUZUpxdiJ=qRPg@mail.gmail.com> <CADiSq7ea-4YWBwRS_6v1m6fC0u1yTCu081ofeCuCUQQ4iTMX=w@mail.gmail.com> Message-ID: <CAP7+vJKZrFj9BoYQRPCiNJwtTPiFWS-tzmcBHGyEd8vneL0hrw@mail.gmail.com> So, with the naming settled (asyncio it is), and lots of other things still to do, I need a BDFL for PEP 3156. Any volunteers? If no-one volunteered I'll have to accept my own PEP at some point, but I don't really *want* to do that. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130929/d6414ece/attachment.html> From stephen at xemacs.org Sun Sep 29 19:03:57 2013 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Mon, 30 Sep 2013 02:03:57 +0900 Subject: [Python-Dev] Why not support user defined operator overloading ? In-Reply-To: <BLU403-EAS393D1EF893F5F5DC86003229D2B0@phx.gbl> References: <BLU403-EAS393D1EF893F5F5DC86003229D2B0@phx.gbl> Message-ID: <87vc1jzhj6.fsf@uwakimon.sk.tsukuba.ac.jp> ??? writes: > If we can overloading these operators, why we can't overloading > other operators? (like .* often used in matrix, U in set > operation) AIUI, it's considered unpythonic. Operators are considered to be part of the *syntax* of Python, unlike Haskell, where infix syntax can be used for any function, and operators can be called via function syntax. (The latter is true in Python as well, but it is very bad form to do that, sufficiently verbose that it is hardly tempting, and it can easily fail because Python operators are polymorphic, but the implementations via class methods are not.) I'm sure it also simplifies parsing. From solipsis at pitrou.net Sun Sep 29 19:16:53 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 29 Sep 2013 19:16:53 +0200 Subject: [Python-Dev] Getting Tulip (PEP 3156) into the 3.4 stdlib, marked provisional, named asyncio References: <CAP7+vJKSOHYeCqhsJVF_7epTDeOw4UP3crsK_mY7HaFYnwG1aA@mail.gmail.com> <CADiSq7c22aKMqW_u1GfLjA=uFuGR3e83R-ynVzpbG_nH9Am+2w@mail.gmail.com> <CAP7+vJJfMP_1DFDc8dSGJo=JiE2hPGK_2AzrWUZUpxdiJ=qRPg@mail.gmail.com> <CADiSq7ea-4YWBwRS_6v1m6fC0u1yTCu081ofeCuCUQQ4iTMX=w@mail.gmail.com> <CAP7+vJKZrFj9BoYQRPCiNJwtTPiFWS-tzmcBHGyEd8vneL0hrw@mail.gmail.com> Message-ID: <20130929191653.773d3d55@fsol> On Sun, 29 Sep 2013 09:54:39 -0700 Guido van Rossum <guido at python.org> wrote: > So, with the naming settled (asyncio it is), and lots of other things still > to do, I need a BDFL for PEP 3156. Any volunteers? If no-one volunteered > I'll have to accept my own PEP at some point, but I don't really *want* to > do that. Well, if you don't mind my slight pro-callback tropism, I'm definitely volunteering :-) Regards Antoine. From barry at python.org Sun Sep 29 20:11:01 2013 From: barry at python.org (Barry Warsaw) Date: Sun, 29 Sep 2013 14:11:01 -0400 Subject: [Python-Dev] 2.6.9rc1 Message-ID: <20130929141101.6d48bdae@anarchist> Just a quick reminder that I intend to cut 2.6.9rc1 tomorrow. Here are the open issues still on my list: - 16040 - nntplib: unlimited readline() from connection This one is waiting for a patch, but TBH if it doesn't make it into 2.6.9 I won't care too much. I also don't mind reviewing and applying a patch between 2.6.9rc1 and 2.6.9 final. - 16041 - poplib: unlimited readline() from connection This one has a patch for 2.7 which does not apply cleanly to 2.6. I'm of a similar mind as with #16040 - if we can get a clean patch after 2.6.9rc1, then fine, but otherwise I'm okay with this one not making it into 2.6.9 final. All other related issues have been fixed in the 2.6 branch. If anyone knows of any other issues that should be on the 2.6.9 radar, please let me know asap. Remember, 2.6.9 final is currently scheduled for October 28, and it will be the last official 2.6 release of the series. After this, we are retiring the 2.6 branch. Speak now or forever hold your peace. Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: <http://mail.python.org/pipermail/python-dev/attachments/20130929/83d998e1/attachment.sig> From ned at nedbatchelder.com Sun Sep 29 21:35:27 2013 From: ned at nedbatchelder.com (Ned Batchelder) Date: Sun, 29 Sep 2013 15:35:27 -0400 Subject: [Python-Dev] Why not support user defined operator overloading ? In-Reply-To: <F53492D8-A90B-4449-9E83-25340AA63AEA@masklinn.net> References: <BLU403-EAS393D1EF893F5F5DC86003229D2B0@phx.gbl> <F53492D8-A90B-4449-9E83-25340AA63AEA@masklinn.net> Message-ID: <524880FF.9010500@nedbatchelder.com> On 9/29/13 11:06 AM, Xavier Morel wrote: > This is more of a python-ideas subject. > > And one of the reasons likely is that it would require significantly > reworking the grammar to handle a kind of user-defined opname (similar > to name, but for operator tokens), with user-defined priority and > associativity, and the ability to import operators (or define how and > when operators become available compared to their definition) > > That's a huge amount of complexity with little to gain. It isn't just a matter of a more complex parser: where would the parser get the information about these new operators? The obvious first answer is that they would be defined as part of classes, but that means the operator definition is in an imported module some place. The importing module couldn't even be parsed until the class was imported. This is a completely different compilation and execution model than Python has now. Now you can compile a Python module without any access to the modules it will eventually import. Importing is a run-time operation. If you want user-defined classes to be able to define new syntax, then importing has to happen at compile time. That's a completely different language. --Ned. From jyrki at dywypi.org Sun Sep 29 21:59:12 2013 From: jyrki at dywypi.org (Jyrki Pulliainen) Date: Sun, 29 Sep 2013 21:59:12 +0200 Subject: [Python-Dev] 2.6.9rc1 In-Reply-To: <20130929141101.6d48bdae@anarchist> References: <20130929141101.6d48bdae@anarchist> Message-ID: <CAM0Q7WAVWe83B2T-9iUci7827R4pa+2hdLUVcOYOYgaqJR10aQ@mail.gmail.com> On Sun, Sep 29, 2013 at 8:11 PM, Barry Warsaw <barry at python.org> wrote: > Just a quick reminder that I intend to cut 2.6.9rc1 tomorrow. > > Here are the open issues still on my list: > > - 16040 - nntplib: unlimited readline() from connection > > This one is waiting for a patch, but TBH if it doesn't make it into > 2.6.9 I > won't care too much. I also don't mind reviewing and applying a patch > between 2.6.9rc1 and 2.6.9 final. > > - 16041 - poplib: unlimited readline() from connection > > This one has a patch for 2.7 which does not apply cleanly to 2.6. I'm > of a > similar mind as with #16040 - if we can get a clean patch after 2.6.9rc1, > then fine, but otherwise I'm okay with this one not making it into 2.6.9 > final. > > I added a patch for issue 16041 in 2.6. It's not too pretty, as I added another test case class with just a slightly different server implementation. This is mainly due to how the tests have been reworked for 2.7 and I didn't feel too comfortable of copying those tests back to 2.6. - Jyrki > All other related issues have been fixed in the 2.6 branch. > > If anyone knows of any other issues that should be on the 2.6.9 radar, > please > let me know asap. Remember, 2.6.9 final is currently scheduled for October > 28, and it will be the last official 2.6 release of the series. After > this, > we are retiring the 2.6 branch. Speak now or forever hold your peace. > > Cheers, > -Barry > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/jyrki%40dywypi.org > > -- - Jyrki -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130929/cc130736/attachment.html> From steve at pearwood.info Mon Sep 30 01:31:32 2013 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 30 Sep 2013 09:31:32 +1000 Subject: [Python-Dev] Why not support user defined operator overloading ? In-Reply-To: <524880FF.9010500@nedbatchelder.com> References: <BLU403-EAS393D1EF893F5F5DC86003229D2B0@phx.gbl> <F53492D8-A90B-4449-9E83-25340AA63AEA@masklinn.net> <524880FF.9010500@nedbatchelder.com> Message-ID: <20130929233132.GC7989@ando> On Sun, Sep 29, 2013 at 03:35:27PM -0400, Ned Batchelder wrote: > It isn't just a matter of a more complex parser: where would the parser > get the information about these new operators? The obvious first answer > is that they would be defined as part of classes, but that means the > operator definition is in an imported module some place. The importing > module couldn't even be parsed until the class was imported. This is a > completely different compilation and execution model than Python has now. The obvious place to define the operator's action is a class, but there is no need to define new syntax at runtime. The parser doesn't need to know the operator's action until runtime, no different from existing operators. For avoidance of doubt, I'm not suggesting this as a concrete proposal, just as a hypothetical. But if somebody wants to take it seriously and write up a PEP, here are two possibilities to get you started: * Unicode characters. It's 2013, and anyone still using an editor which only handles ASCII shouldn't be :-) There are dozens of non-alphanumeric characters in Unicode that are suitable as operators. Unless I've missed one, all the existing operators have category 'Sm', 'Sk', 'Po', or 'Pd', so Python might parse any character in those categories as a potential operator. E.g. all of these are in category Sm: ? ? ? ? ? ? ? ? ? * ASCII digraphs. A more conservative approach would be to define custom operators as two characters, not one. Say, @X might be parsed as operator X. In either case, having parsed "spam @X eggs" or "spam ? eggs", the interpreter could do this: call spam.__op__('X', eggs) # or '?' if that doesn't succeed, call eggs.__rop__('X', spam) raise OperatorError none of which need happen until runtime, same as existing operators. If you want to also support unary operators (prefix or postfix) then the parsing rules presumably will become more complicated, but the basic principle remains: the parser needs a pre-defined "template" to recognise what looks like an operator, and that can be baked into the language definition. The actual meaning of the operator can be determined at run time. Make no mistake, this is still a big change to Python's semantics. But it doesn't require Python to redefine syntax on the fly. -- Steven From larry at hastings.org Mon Sep 30 01:51:58 2013 From: larry at hastings.org (Larry Hastings) Date: Mon, 30 Sep 2013 00:51:58 +0100 Subject: [Python-Dev] Getting Tulip (PEP 3156) into the 3.4 stdlib, marked provisional, named asyncio In-Reply-To: <CAP7+vJKSOHYeCqhsJVF_7epTDeOw4UP3crsK_mY7HaFYnwG1aA@mail.gmail.com> References: <CAP7+vJKSOHYeCqhsJVF_7epTDeOw4UP3crsK_mY7HaFYnwG1aA@mail.gmail.com> Message-ID: <5248BD1E.1080003@hastings.org> On 09/27/2013 11:33 PM, Guido van Rossum wrote: > I've been looking at my progress with Tulip and the 3.4 release > schedule (PEP 429) and it looks like I will have to do some kind of > sprint to get it into the release in time for beta 1, which is planned > for Nov 24. Ideally I'd get it into alpha 4, which is scheduled for > Oct 20 -- that's in about three weeks, and probably too tight. > > Even Nov 24 is aggressive, because the PEP (PEP 3156) hasn't even been > discussed formally, let alone accepted! Fortunately I'm pretty happy > with most of the APIs defined in the PEP -- there are some holes but I > expect no big changes at this point. My guess is, a lot of people would be disappointed if Tulip missed 3.4. I suspect the community would rather we slip the beta a little if it meant it the difference between Tulip and no Tulip. I just hope you make it easy on me and beat the deadline ;-) Cheers, //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130930/45defbb1/attachment.html> From larry at hastings.org Mon Sep 30 02:04:45 2013 From: larry at hastings.org (Larry Hastings) Date: Mon, 30 Sep 2013 01:04:45 +0100 Subject: [Python-Dev] [RELEASED] Python 3.4.0a3 Message-ID: <5248C01D.7060701@hastings.org> On behalf of the Python development team, I'm pleased to announce the third alpha release of Python 3.4. This is a preview release, and its use is not recommended for production settings. Python 3.4 includes a range of improvements of the 3.x series, including hundreds of small improvements and bug fixes. Major new features and changes in the 3.4 release series so far include: * PEP 435, a standardized "enum" module * PEP 442, improved semantics for object finalization * PEP 443, adding single-dispatch generic functions to the standard library * PEP 445, a new C API for implementing custom memory allocators * PEP 446, changing file descriptors to not be inherited by default in subprocesses To download Python 3.4.0a3 visit: http://www.python.org/download/releases/3.4.0/ Please consider trying Python 3.4.0a3 with your code and reporting any issues you notice to: http://bugs.python.org/ Enjoy! -- Larry Hastings, Release Manager larry at hastings.org (on behalf of the entire python-dev team and 3.4's contributors) From zhangpeipei812 at outlook.com Mon Sep 30 02:23:12 2013 From: zhangpeipei812 at outlook.com (=?UTF-8?B?5byg5L2p5L2p?=) Date: Mon, 30 Sep 2013 08:23:12 +0800 Subject: [Python-Dev] Why not support user defined operator overloading ? Message-ID: <BLU406-EAS140395B430135783E5714B69D140@phx.gbl> Hello: I agree with Steven D'Aprano. Here is an example: class A(object): def __init__(self, value): self.value = value def add(self, other): return self.value + other.value __magic_method__ = {'+':add} a1 = A(1) a2 = A(2) We only need a macro expand a1 + a2 to a1.__magic__method__['+'](a, b) the later can be execute on Python. Regards peipei From eric at trueblade.com Mon Sep 30 02:40:19 2013 From: eric at trueblade.com (Eric V. Smith) Date: Sun, 29 Sep 2013 20:40:19 -0400 Subject: [Python-Dev] Getting Tulip (PEP 3156) into the 3.4 stdlib, marked provisional, named asyncio In-Reply-To: <CAP1=2W7A2Q3t7Bk7iDZ9ZPprFdi+yr152EgVV=ACaTZkaBbUmw@mail.gmail.com> References: <CAP7+vJKSOHYeCqhsJVF_7epTDeOw4UP3crsK_mY7HaFYnwG1aA@mail.gmail.com> <CAP1=2W7A2Q3t7Bk7iDZ9ZPprFdi+yr152EgVV=ACaTZkaBbUmw@mail.gmail.com> Message-ID: <5248C873.6060701@trueblade.com> On 9/27/2013 9:14 PM, Brett Cannon wrote: > I don't see any issue with redirecting the discussion. python-tulip@ is > acting like a SIG for the module, so no real precedent beyond it not > being hosted as a mail.python.org <http://mail.python.org> list. I'm sure I'm in the minority, but I'd like the discussion to take place on a python.org mailing list. I don't want to log in to a Google property, and I don't trust them with the mailing list archives. I know my voice counts less than active Tulip discussion participants, but now at least I feel better for having said something. -- Eric. From greg.ewing at canterbury.ac.nz Mon Sep 30 02:53:08 2013 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 30 Sep 2013 13:53:08 +1300 Subject: [Python-Dev] Why not support user defined operator overloading ? In-Reply-To: <20130929233132.GC7989@ando> References: <BLU403-EAS393D1EF893F5F5DC86003229D2B0@phx.gbl> <F53492D8-A90B-4449-9E83-25340AA63AEA@masklinn.net> <524880FF.9010500@nedbatchelder.com> <20130929233132.GC7989@ando> Message-ID: <5248CB74.6000002@canterbury.ac.nz> Steven D'Aprano wrote: > there > is no need to define new syntax at runtime. The parser doesn't need to > know the operator's action until runtime It does need to know the operator's precedence and associativity, though, which means either declaring it somewhere, or having some kind of fixed rule. -- Greg From guido at python.org Mon Sep 30 02:55:10 2013 From: guido at python.org (Guido van Rossum) Date: Sun, 29 Sep 2013 17:55:10 -0700 Subject: [Python-Dev] Getting Tulip (PEP 3156) into the 3.4 stdlib, marked provisional, named asyncio In-Reply-To: <5248C873.6060701@trueblade.com> References: <CAP7+vJKSOHYeCqhsJVF_7epTDeOw4UP3crsK_mY7HaFYnwG1aA@mail.gmail.com> <CAP1=2W7A2Q3t7Bk7iDZ9ZPprFdi+yr152EgVV=ACaTZkaBbUmw@mail.gmail.com> <5248C873.6060701@trueblade.com> Message-ID: <CAP7+vJ+5ij-w0KAcF+fkFnwgrWcRTKUgkSMoamT-GWHt7o9b1w@mail.gmail.com> On Sun, Sep 29, 2013 at 5:40 PM, Eric V. Smith <eric at trueblade.com> wrote: > On 9/27/2013 9:14 PM, Brett Cannon wrote: > > > I don't see any issue with redirecting the discussion. python-tulip@ is > > acting like a SIG for the module, so no real precedent beyond it not > > being hosted as a mail.python.org <http://mail.python.org> list. > > I'm sure I'm in the minority, but I'd like the discussion to take place > on a python.org mailing list. I don't want to log in to a Google > property, and I don't trust them with the mailing list archives. > > I know my voice counts less than active Tulip discussion participants, > but now at least I feel better for having said something. > I wish you'd said something a looong time ago when it would have been easy to move the list. Even if we moved it now we'd have split archives. Also, I'm not sure where the paranoia comes from. FWIW I'm less worried about Google reading my personal email than about the python.org webmasters reading it. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130929/1439158c/attachment.html> From guido at python.org Mon Sep 30 02:57:25 2013 From: guido at python.org (Guido van Rossum) Date: Sun, 29 Sep 2013 17:57:25 -0700 Subject: [Python-Dev] Getting Tulip (PEP 3156) into the 3.4 stdlib, marked provisional, named asyncio In-Reply-To: <20130929191653.773d3d55@fsol> References: <CAP7+vJKSOHYeCqhsJVF_7epTDeOw4UP3crsK_mY7HaFYnwG1aA@mail.gmail.com> <CADiSq7c22aKMqW_u1GfLjA=uFuGR3e83R-ynVzpbG_nH9Am+2w@mail.gmail.com> <CAP7+vJJfMP_1DFDc8dSGJo=JiE2hPGK_2AzrWUZUpxdiJ=qRPg@mail.gmail.com> <CADiSq7ea-4YWBwRS_6v1m6fC0u1yTCu081ofeCuCUQQ4iTMX=w@mail.gmail.com> <CAP7+vJKZrFj9BoYQRPCiNJwtTPiFWS-tzmcBHGyEd8vneL0hrw@mail.gmail.com> <20130929191653.773d3d55@fsol> Message-ID: <CAP7+vJJEJV+6iNBz36B3Ufg4X8cerJ5ZXV=q_ufu5YuWwCk42Q@mail.gmail.com> On Sun, Sep 29, 2013 at 10:16 AM, Antoine Pitrou <solipsis at pitrou.net>wrote: > On Sun, 29 Sep 2013 09:54:39 -0700 Guido van Rossum <guido at python.org> > wrote: > > So, with the naming settled (asyncio it is), and lots of other things > still > > to do, I need a BDFL for PEP 3156. Any volunteers? If no-one volunteered > > I'll have to accept my own PEP at some point, but I don't really *want* > to > > do that. > > Well, if you don't mind my slight pro-callback tropism, I'm definitely > volunteering :-) > I don't mind it, and if there's not too much objection from the community you've got the job! -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130929/c8f54425/attachment.html> From rovitotv at gmail.com Mon Sep 30 02:39:52 2013 From: rovitotv at gmail.com (Todd V. Rovito) Date: Sun, 29 Sep 2013 20:39:52 -0400 Subject: [Python-Dev] Completed: Google Summer of Code 2013 IDLE Unit Testing and Improvements Message-ID: <E51BE7FC-5555-4895-9472-4903B66F944B@gmail.com> In April 2013 we challenged students to not spend their summer idle around the swimming pool but rather spend their time on IDLE and make a difference. IDLE is Python?s Integrated Development Environment (IDE) that is shipped with each Python release (http://en.wikipedia.org/wiki/IDLE_(Python)). Since IDLE ships with Python it is often the first IDE a new Python programmer uses. The Python development community wants to make IDLE an awesome experience especially for people that are learning Python. Despite the fact that IDLE originally shipped in 1998 it had no unit test cases, making it difficult to confidently update the code and ship new versions. For this Google Summer of Code project our students Jayakrishnan Rajagopalasarma (http://123works.blogspot.com/search/label/GSoC%202013) and Phil Webster (http://weblog.philwebster.net/tag/gsoc/) had to not only create the unit tests but assist with design of the unit test framework (http://bugs.python.org/issue15392). By the end of the summer of 2013 Phil and Jayakrishnan have worked on five patches that have been committed to the CPython open source project. In addition the team has started work on an additional five issues (at bugs.python.org) but have not completed them yet. Jayakrishnan explored an idea to add automatic PEP8 checking to IDLE http://bugs.python.org/issue18704. Our hope is that both JayKrish and Phil continue to contribute to Python even now that GSoC is complete. The team wishes to thank Google and the Python Software Foundation for this opportunity to contribute to Python. Thanks to all the Python Core Developers that helped review and commit issues for the students. Special thanks goes to Terry Reedy who provided outstanding feedback on issues and performed all of the student?s commits. Finally thanks to Terri Oda for being a great GSoC administrator for the Python Software Foundation and helping by assigning an awesome co-mentor J.H. Hawley. The complete list of closed issues is below: http://bugs.python.org/issue18279 http://bugs.python.org/issue18189 http://bugs.python.org/issue18425 http://bugs.python.org/issue18226 http://bugs.python.org/issue18489 The complete list of started issues is below: http://bugs.python.org/issue18292 http://bugs.python.org/issue18910 http://bugs.python.org/issue18409 http://bugs.python.org/issue18410 http://bugs.python.org/issue18592 -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130929/4baf4560/attachment.html> From zhangpeipei812 at outlook.com Mon Sep 30 03:02:47 2013 From: zhangpeipei812 at outlook.com (=?UTF-8?B?5byg5L2p5L2p?=) Date: Mon, 30 Sep 2013 09:02:47 +0800 Subject: [Python-Dev] Why not support user defined operator overloading ? Message-ID: <BLU404-EAS117E3E2585DD92DEE3E0FBD9D140@phx.gbl> On 2013/9/30 8:53 Greg Ewing wrote: > It does need to know the operator's precedence and >associativity, though, which means either declaring >it somewhere, or having some kind of fixed rule I suggest all user defined operator are at lowest priority. Regards peipei From senthil at uthcode.com Mon Sep 30 04:00:19 2013 From: senthil at uthcode.com (Senthil Kumaran) Date: Sun, 29 Sep 2013 19:00:19 -0700 Subject: [Python-Dev] [Python-checkins] cpython (merge 3.3 -> default): Fix http.server's request handling case on trailing '/'. In-Reply-To: <523325E1.70608@trueblade.com> References: <3cbpGS1zZSz7Lk4@mail.python.org> <523325E1.70608@trueblade.com> Message-ID: <CAPOVWOSBa_OemAhOPtimi-j-h_dneshi+JpqMRvVdVr=RnoHLA@mail.gmail.com> On Fri, Sep 13, 2013 at 7:49 AM, Eric V. Smith <eric at trueblade.com> wrote: >> Patch contributed by Vajrasky Kok. Addresses Issue #17324 > >> + trailing_slash = True if path.rstrip().endswith('/') else False > > Wouldn't this be better just as: > trailing_slash = path.rstrip().endswith('/') I noticed this email late. Corrected it now. Thanks, Senthil From eliben at gmail.com Mon Sep 30 04:04:59 2013 From: eliben at gmail.com (Eli Bendersky) Date: Sun, 29 Sep 2013 19:04:59 -0700 Subject: [Python-Dev] Why not support user defined operator overloading ? In-Reply-To: <BLU404-EAS117E3E2585DD92DEE3E0FBD9D140@phx.gbl> References: <BLU404-EAS117E3E2585DD92DEE3E0FBD9D140@phx.gbl> Message-ID: <CAF-Rda9c8qz+doPvY6DL4hUu8=iW3XQjb71QFE-NOczGFH+1ag@mail.gmail.com> Can you guys please move this discussion to python-ideas? python-dev is the wrong mailing list. Eli On Sun, Sep 29, 2013 at 6:02 PM, ??? <zhangpeipei812 at outlook.com> wrote: > On 2013/9/30 8:53 Greg Ewing wrote: > > It does need to know the operator's precedence and > >associativity, though, which means either declaring > >it somewhere, or having some kind of fixed rule > > I suggest all user defined operator are at lowest priority. > > Regards > peipei > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/eliben%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130929/81694022/attachment.html> From senthil at uthcode.com Mon Sep 30 04:08:55 2013 From: senthil at uthcode.com (Senthil Kumaran) Date: Sun, 29 Sep 2013 19:08:55 -0700 Subject: [Python-Dev] cpython (merge 3.3 -> default): merge from 3.3 In-Reply-To: <20130912081423.4607ea82@fsol> References: <3cb8QF2vy7z7Ljh@mail.python.org> <20130912081423.4607ea82@fsol> Message-ID: <CAPOVWOTH_oWiyGiTJQr+KHJYi1GyH9Viux7d3hVXmjSgY5P6Nw@mail.gmail.com> On Wed, Sep 11, 2013 at 11:14 PM, Antoine Pitrou <solipsis at pitrou.net> wrote: > On Thu, 12 Sep 2013 07:57:25 +0200 (CEST) > senthil.kumaran <python-checkins at python.org> wrote: >> >> +<<<<<<< local >> Optional argument random is a 0-argument function returning a >> random float in [0.0, 1.0); if it is the default None, the >> standard random.random will be used. >> +======= >> + Optional arg random is a 0-argument function returning a random >> + float in [0.0, 1.0); by default, the standard random.random. >> + >> + Do not supply the 'int' argument. >> +>>>>>>> other > > Can you fix this? Had done it earlier. (http://hg.python.org/cpython/rev/1398dfb59fd9 ) I missed responding back. Sorry for the trouble. Thanks, Senthil From ned at nedbatchelder.com Mon Sep 30 04:09:34 2013 From: ned at nedbatchelder.com (Ned Batchelder) Date: Sun, 29 Sep 2013 22:09:34 -0400 Subject: [Python-Dev] Why not support user defined operator overloading ? In-Reply-To: <BLU404-EAS117E3E2585DD92DEE3E0FBD9D140@phx.gbl> References: <BLU404-EAS117E3E2585DD92DEE3E0FBD9D140@phx.gbl> Message-ID: <5248DD5E.8030109@nedbatchelder.com> On 9/29/13 9:02 PM, ??? wrote: > On 2013/9/30 8:53 Greg Ewing wrote: >> It does need to know the operator's precedence and >> associativity, though, which means either declaring >> it somewhere, or having some kind of fixed rule > I suggest all user defined operator are at lowest priority. Peipei, this is an interesting discussion, but doesn't belong on Python-Dev. The Python-Ideas mailing list is for exactly this kind of topic. --Ned. > Regards > peipei > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ned%40nedbatchelder.com From hodgestar+pythondev at gmail.com Mon Sep 30 09:05:11 2013 From: hodgestar+pythondev at gmail.com (Simon Cross) Date: Mon, 30 Sep 2013 09:05:11 +0200 Subject: [Python-Dev] Getting Tulip (PEP 3156) into the 3.4 stdlib, marked provisional, named asyncio In-Reply-To: <5248BD1E.1080003@hastings.org> References: <CAP7+vJKSOHYeCqhsJVF_7epTDeOw4UP3crsK_mY7HaFYnwG1aA@mail.gmail.com> <5248BD1E.1080003@hastings.org> Message-ID: <CAD5NRCEOsyucPzyA4u4whBiu8Zt+2HwkWY6V2Nmubp8r7AFwXg@mail.gmail.com> On Mon, Sep 30, 2013 at 1:51 AM, Larry Hastings <larry at hastings.org> wrote: > My guess is, a lot of people would be disappointed if Tulip missed 3.4. I > suspect the community would rather we slip the beta a little if it meant it > the difference between Tulip and no Tulip. +1 From martin at v.loewis.de Mon Sep 30 11:01:45 2013 From: martin at v.loewis.de (=?ISO-8859-15?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 30 Sep 2013 11:01:45 +0200 Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for pronouncement? In-Reply-To: <109B26FC-32A9-4DDE-9639-E48A0389455D@stufft.io> References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com> <20130925165022.3d67e290@anarchist> <109B26FC-32A9-4DDE-9639-E48A0389455D@stufft.io> Message-ID: <52493DF9.7010705@v.loewis.de> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Am 25.09.13 23:33, schrieb Donald Stufft: > An early draft of this did not have the backport to 2.7 and when I > showed *that* version around to get feedback people were less > enthusiastic about it and generally viewed it as "nice but > worthless to me for N years". I'm leaning towards the people that oppose addition of this feature to older Python releases. In particular, the objection that the new PEP is worthless for a long time to come, is in itself not really relevant: It is always the case that features proposed by a PEP reach users only years after they have been implemented. > What users want isn't rationale in and of itself but I think it's > an important data point, especially given how long 2.7.LASTEVER is > going to be relevant to end users. Well, I really really don't like this idea. 2.7 should not get new features; users who want new features need to switch to 3.x. Regards, Martin -----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.18 (Darwin) Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlJJPfkACgkQavBT8H2dyNL9SQCfceoTqIeCiwq0GMdaWeUSn/nV 6E4AmwRfmjFcSXjdmGLQbewVqxxYwRit =tLof -----END PGP SIGNATURE----- From donald at stufft.io Mon Sep 30 13:18:29 2013 From: donald at stufft.io (Donald Stufft) Date: Mon, 30 Sep 2013 07:18:29 -0400 Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for pronouncement? In-Reply-To: <52493DF9.7010705@v.loewis.de> References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com> <20130925165022.3d67e290@anarchist> <109B26FC-32A9-4DDE-9639-E48A0389455D@stufft.io> <52493DF9.7010705@v.loewis.de> Message-ID: <5B4A7F23-C9E7-464E-A7A2-05CE3084CC67@stufft.io> On Sep 30, 2013, at 5:01 AM, "Martin v. L?wis" <martin at v.loewis.de> wrote: > Signed PGP part > Am 25.09.13 23:33, schrieb Donald Stufft: > > An early draft of this did not have the backport to 2.7 and when I > > showed *that* version around to get feedback people were less > > enthusiastic about it and generally viewed it as "nice but > > worthless to me for N years". > > I'm leaning towards the people that oppose addition of this feature to > older Python releases. In particular, the objection that the new PEP is > worthless for a long time to come, is in itself not really relevant: > > It is always the case that features proposed by a PEP reach users only > years after they have been implemented. Well the point we tried to get across in the PEP is that a normal feature you can typically just install a backport from PyPI to gain it early. This isin't so much driven by "well it'd be nice for the stdlib to have X", but "well this is a real and valid pain point that causes pain for a lot of users". The 2.7 backport was driven by just how painful this particular pain point can be. I've personally had feedback that in tutorials at like PyCon or meet ups that easily 1/3 of the time can be spent in getting users setup with Python, setuptools, and pip. Obviously you're the delegate for this PEP and it's your final decision and either way If the PEP is accepted I'll do the implementation for 3.4. > > > What users want isn't rationale in and of itself but I think it's > > an important data point, especially given how long 2.7.LASTEVER is > > going to be relevant to end users. > > Well, I really really don't like this idea. 2.7 should not get new > features; users who want new features need to switch to 3.x. > > Regards, > Martin > > ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: <http://mail.python.org/pipermail/python-dev/attachments/20130930/e1243024/attachment.sig> From ncoghlan at gmail.com Mon Sep 30 13:32:13 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 30 Sep 2013 21:32:13 +1000 Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for pronouncement? In-Reply-To: <52493DF9.7010705@v.loewis.de> References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com> <20130925165022.3d67e290@anarchist> <109B26FC-32A9-4DDE-9639-E48A0389455D@stufft.io> <52493DF9.7010705@v.loewis.de> Message-ID: <CADiSq7czM+ib2GR+kSvkuy5_TUWvV75kNLoDE0GZsrjEOQUkfg@mail.gmail.com> On 30 Sep 2013 19:03, Martin v. L?wis <martin at v.loewis.de> wrote: > > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Am 25.09.13 23:33, schrieb Donald Stufft: > > An early draft of this did not have the backport to 2.7 and when I > > showed *that* version around to get feedback people were less > > enthusiastic about it and generally viewed it as "nice but > > worthless to me for N years". > > I'm leaning towards the people that oppose addition of this feature to > older Python releases. In particular, the objection that the new PEP is > worthless for a long time to come, is in itself not really relevant: > > It is always the case that features proposed by a PEP reach users only > years after they have been implemented. > > > What users want isn't rationale in and of itself but I think it's > > an important data point, especially given how long 2.7.LASTEVER is > > going to be relevant to end users. > > Well, I really really don't like this idea. 2.7 should not get new > features; users who want new features need to switch to 3.x. Before you make your final decision on this front, I'd like to record in the PEP my fallback plan for if the backporting portion of the PEP is rejected while the rest is accepted. After all, you should know what it is you're actually choosing between. I just need to run the details of the fallback proposal past Donald before publishing it under both our names. Regards, Nick. > > Regards, > Martin > > > -----BEGIN PGP SIGNATURE----- > Version: GnuPG/MacGPG2 v2.0.18 (Darwin) > Comment: GPGTools - http://gpgtools.org > Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ > > iEYEARECAAYFAlJJPfkACgkQavBT8H2dyNL9SQCfceoTqIeCiwq0GMdaWeUSn/nV > 6E4AmwRfmjFcSXjdmGLQbewVqxxYwRit > =tLof > -----END PGP SIGNATURE----- > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130930/188b3c6c/attachment.html> From martin at v.loewis.de Mon Sep 30 13:50:51 2013 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 30 Sep 2013 13:50:51 +0200 Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for pronouncement? In-Reply-To: <5B4A7F23-C9E7-464E-A7A2-05CE3084CC67@stufft.io> References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com> <20130925165022.3d67e290@anarchist> <109B26FC-32A9-4DDE-9639-E48A0389455D@stufft.io> <52493DF9.7010705@v.loewis.de> <5B4A7F23-C9E7-464E-A7A2-05CE3084CC67@stufft.io> Message-ID: <5249659B.1060305@v.loewis.de> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Am 30.09.13 13:18, schrieb Donald Stufft: > Well the point we tried to get across in the PEP is that a normal > feature you can typically just install a backport from PyPI to gain > it early. This isin't so much driven by "well it'd be nice for the > stdlib to have X", but "well this is a real and valid pain point > that causes pain for a lot of users". I see it exactly vice versa. A normal new language feature, you *cannot* get early from PyPI, e.g. if it's a syntax extension (e.g. the with statement). OTOH, *this* particular feature you can easily get from PyPI - just download and run the PIP installer. > The 2.7 backport was driven by just how painful this particular > pain point can be. I've personally had feedback that in tutorials > at like PyCon or meet ups that easily 1/3 of the time can be spent > in getting users setup with Python, setuptools, and pip. I'm sure there is something that can be done about it. What operating system are people using who have difficulties to set this all up? I see that pip doesn't have a Windows installer on PyPI. That looks like the real culprit to me. I'd be willing to work with you to provide one (that ideally bundles all dependencies). > Obviously you're the delegate for this PEP and it's your final > decision and either way If the PEP is accepted I'll do the > implementation for 3.4. The backporting is the main stumbling block. Otherwise, I'm in favor of the PEP. Regards, Martin -----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.18 (Darwin) Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlJJZZsACgkQavBT8H2dyNIAWQCeNi3OcIXaNGmFuDfPcTEfnBjV vTMAnAv7diLGOsJ2oNUqE7BHG1wIwMCP =r+wP -----END PGP SIGNATURE----- From ncoghlan at gmail.com Mon Sep 30 15:07:36 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 30 Sep 2013 23:07:36 +1000 Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for pronouncement? In-Reply-To: <5249659B.1060305@v.loewis.de> References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com> <20130925165022.3d67e290@anarchist> <109B26FC-32A9-4DDE-9639-E48A0389455D@stufft.io> <52493DF9.7010705@v.loewis.de> <5B4A7F23-C9E7-464E-A7A2-05CE3084CC67@stufft.io> <5249659B.1060305@v.loewis.de> Message-ID: <CADiSq7dVsXJpxHDx5OipH-x34g9KwR2zZh8LZ_6svYz_tW3zHw@mail.gmail.com> On 30 Sep 2013 21:52, Martin v. L?wis <martin at v.loewis.de> wrote: > > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Am 30.09.13 13:18, schrieb Donald Stufft: > > Well the point we tried to get across in the PEP is that a normal > > feature you can typically just install a backport from PyPI to gain > > it early. This isin't so much driven by "well it'd be nice for the > > stdlib to have X", but "well this is a real and valid pain point > > that causes pain for a lot of users". > > I see it exactly vice versa. A normal new language feature, you *cannot* > get early from PyPI, e.g. if it's a syntax extension (e.g. the with > statement). It's standard library improvements (like SSL hostname matching) that are relevant to this comparison, rather than the kinds of changes that were barred by the language moratorium. > OTOH, *this* particular feature you can easily get from > PyPI - just download and run the PIP installer. > > > The 2.7 backport was driven by just how painful this particular > > pain point can be. I've personally had feedback that in tutorials > > at like PyCon or meet ups that easily 1/3 of the time can be spent > > in getting users setup with Python, setuptools, and pip. > > I'm sure there is something that can be done about it. What operating > system are people using who have difficulties to set this all up? > > I see that pip doesn't have a Windows installer on PyPI. That looks > like the real culprit to me. I'd be willing to work with you to > provide one (that ideally bundles all dependencies). We'll take you up on that offer. Windows is the main problem, since other platforms offer wget/curl out of the box so bootstrapping is easy by comparison. > > > Obviously you're the delegate for this PEP and it's your final > > decision and either way If the PEP is accepted I'll do the > > implementation for 3.4. > > The backporting is the main stumbling block. Otherwise, I'm in favor > of the PEP. My plan now is to split the PEP in two, so the 3.4 changes can be accepted as non-controversial, including the offer of core dev assistance in creating and maintaining a Windows installer for pip to better support earlier versions. The backporting PEP will be deferred, for reconsideration some time in the future after the initial PEP has been implemented. Regards, Nick. > > Regards, > Martin > > -----BEGIN PGP SIGNATURE----- > Version: GnuPG/MacGPG2 v2.0.18 (Darwin) > Comment: GPGTools - http://gpgtools.org > Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ > > iEYEARECAAYFAlJJZZsACgkQavBT8H2dyNIAWQCeNi3OcIXaNGmFuDfPcTEfnBjV > vTMAnAv7diLGOsJ2oNUqE7BHG1wIwMCP > =r+wP > -----END PGP SIGNATURE----- > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130930/f58b06f0/attachment.html> From eliben at gmail.com Mon Sep 30 15:09:48 2013 From: eliben at gmail.com (Eli Bendersky) Date: Mon, 30 Sep 2013 06:09:48 -0700 Subject: [Python-Dev] Semi-official read-only Github mirror of the CPython Mercurial repository Message-ID: <CAF-Rda__36CpmDaVxLWo+iWhw5x9SqNFGjoQFXWM591PGNOzhg@mail.gmail.com> Hi all, https://github.com/python/cpython is now live as a semi-official, *read only* Github mirror of the CPython Mercurial repository. Let me know if you have any problems/concerns. I still haven't decided how often to update it (considering either just N times a day, or maybe use a Hg hook for batching). Suggestions are welcome. The methodology I used to create it is via hg-fast-export. I also tried to pack and gc the git repo as much as possible before the initial Github push - it went down from almost ~2GB to ~200MB (so this is the size of a fresh clone right now). Eli P.S. thanks Jesse for the keys to https://github.com/python -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130930/fd1dec8d/attachment.html> From donald at stufft.io Mon Sep 30 15:54:04 2013 From: donald at stufft.io (Donald Stufft) Date: Mon, 30 Sep 2013 09:54:04 -0400 Subject: [Python-Dev] Semi-official read-only Github mirror of the CPython Mercurial repository In-Reply-To: <CAF-Rda__36CpmDaVxLWo+iWhw5x9SqNFGjoQFXWM591PGNOzhg@mail.gmail.com> References: <CAF-Rda__36CpmDaVxLWo+iWhw5x9SqNFGjoQFXWM591PGNOzhg@mail.gmail.com> Message-ID: <145A4E43-27F1-46EF-91BE-383C352669FF@stufft.io> On Sep 30, 2013, at 9:09 AM, Eli Bendersky <eliben at gmail.com> wrote: > Hi all, > > https://github.com/python/cpython is now live as a semi-official, *read only* Github mirror of the CPython Mercurial repository. Let me know if you have any problems/concerns. > > I still haven't decided how often to update it (considering either just N times a day, or maybe use a Hg hook for batching). Suggestions are welcome. > > The methodology I used to create it is via hg-fast-export. I also tried to pack and gc the git repo as much as possible before the initial Github push - it went down from almost ~2GB to ~200MB (so this is the size of a fresh clone right now). > > Eli > > P.S. thanks Jesse for the keys to https://github.com/python > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io Awesome! I find Github way nicer for reading source than hg.python.org's web interface, any chance I could convince you to do this for the peps repo too? ;) ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130930/0d715fc5/attachment.html> -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: <http://mail.python.org/pipermail/python-dev/attachments/20130930/0d715fc5/attachment.sig> From eliben at gmail.com Mon Sep 30 15:58:49 2013 From: eliben at gmail.com (Eli Bendersky) Date: Mon, 30 Sep 2013 06:58:49 -0700 Subject: [Python-Dev] Semi-official read-only Github mirror of the CPython Mercurial repository In-Reply-To: <145A4E43-27F1-46EF-91BE-383C352669FF@stufft.io> References: <CAF-Rda__36CpmDaVxLWo+iWhw5x9SqNFGjoQFXWM591PGNOzhg@mail.gmail.com> <145A4E43-27F1-46EF-91BE-383C352669FF@stufft.io> Message-ID: <CAF-Rda-4hcFhHS_Wjmv4kBDzG2Pb+fU7SRH2RjX2WdhzE9_ubQ@mail.gmail.com> > > https://github.com/python/cpython is now live as a semi-official, *read > only* Github mirror of the CPython Mercurial repository. Let me know if you > have any problems/concerns. > > I still haven't decided how often to update it (considering either just N > times a day, or maybe use a Hg hook for batching). Suggestions are welcome. > > The methodology I used to create it is via hg-fast-export. I also tried to > pack and gc the git repo as much as possible before the initial Github push > - it went down from almost ~2GB to ~200MB (so this is the size of a fresh > clone right now). > > Eli > > P.S. thanks Jesse for the keys to https://github.com/python > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/donald%40stufft.io > > > Awesome! I find Github way nicer for reading source than hg.python.org's > web interface, any chance I could convince you to do this for the peps repo > too? ;) > Yes, that shouldn't pose a problem. I'll let it hum for a couple of days just to see everything is OK and then I'll add peps too. Eli -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130930/cf9fe737/attachment.html> From barry at python.org Mon Sep 30 16:02:56 2013 From: barry at python.org (Barry Warsaw) Date: Mon, 30 Sep 2013 10:02:56 -0400 Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for pronouncement? In-Reply-To: <CADiSq7dVsXJpxHDx5OipH-x34g9KwR2zZh8LZ_6svYz_tW3zHw@mail.gmail.com> References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com> <20130925165022.3d67e290@anarchist> <109B26FC-32A9-4DDE-9639-E48A0389455D@stufft.io> <52493DF9.7010705@v.loewis.de> <5B4A7F23-C9E7-464E-A7A2-05CE3084CC67@stufft.io> <5249659B.1060305@v.loewis.de> <CADiSq7dVsXJpxHDx5OipH-x34g9KwR2zZh8LZ_6svYz_tW3zHw@mail.gmail.com> Message-ID: <20130930100256.53949f86@limelight.wooz.org> On Sep 30, 2013, at 11:07 PM, Nick Coghlan wrote: >My plan now is to split the PEP in two, so the 3.4 changes can be accepted >as non-controversial, including the offer of core dev assistance in >creating and maintaining a Windows installer for pip to better support >earlier versions. The backporting PEP will be deferred, for reconsideration >some time in the future after the initial PEP has been implemented. +1, since I think there's little disagreement about adding this to 3.4. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: <http://mail.python.org/pipermail/python-dev/attachments/20130930/22f1995d/attachment-0001.sig> From skip at pobox.com Mon Sep 30 16:08:12 2013 From: skip at pobox.com (Skip Montanaro) Date: Mon, 30 Sep 2013 09:08:12 -0500 Subject: [Python-Dev] Semi-official read-only Github mirror of the CPython Mercurial repository In-Reply-To: <CAF-Rda__36CpmDaVxLWo+iWhw5x9SqNFGjoQFXWM591PGNOzhg@mail.gmail.com> References: <CAF-Rda__36CpmDaVxLWo+iWhw5x9SqNFGjoQFXWM591PGNOzhg@mail.gmail.com> Message-ID: <CANc-5UwZW3LLoYyQbcnZWWe8u=ZAJyHGuwARf=8mTLf3pqQz+Q@mail.gmail.com> > https://github.com/python/cpython is now live as a semi-official, *read > only* Github mirror of the CPython Mercurial repository. Let me know if you > have any problems/concerns. Thanks for this, Eli. I use git regularly at work, so I'm getting much more comfortable with it. Do you have a suggested workflow for people who might want to use Git in preference to Hg, but still have write access? Skip From skip at pobox.com Mon Sep 30 16:35:24 2013 From: skip at pobox.com (Skip Montanaro) Date: Mon, 30 Sep 2013 09:35:24 -0500 Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for pronouncement? In-Reply-To: <20130930100256.53949f86@limelight.wooz.org> References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com> <20130925165022.3d67e290@anarchist> <109B26FC-32A9-4DDE-9639-E48A0389455D@stufft.io> <52493DF9.7010705@v.loewis.de> <5B4A7F23-C9E7-464E-A7A2-05CE3084CC67@stufft.io> <5249659B.1060305@v.loewis.de> <CADiSq7dVsXJpxHDx5OipH-x34g9KwR2zZh8LZ_6svYz_tW3zHw@mail.gmail.com> <20130930100256.53949f86@limelight.wooz.org> Message-ID: <CANc-5UwkZ4BDNJ554Mnhuz6tawGYL+KdjgiG0C12di7pgncPAQ@mail.gmail.com> Splitting into two pieces also means you can implement it for 3.4 first and identify possible problems caused by preexisting pip installs before deciding whether to add it to 2.7 and 3.3. Skip From ncoghlan at gmail.com Mon Sep 30 16:47:11 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 1 Oct 2013 00:47:11 +1000 Subject: [Python-Dev] PEP 453 (pip bootstrapping) ready for pronouncement? In-Reply-To: <CANc-5UwkZ4BDNJ554Mnhuz6tawGYL+KdjgiG0C12di7pgncPAQ@mail.gmail.com> References: <CADiSq7fo+f6BfmUwYeCJ7cjRznGNKTJx2XnUjC7e7p8BEBuKMw@mail.gmail.com> <20130925165022.3d67e290@anarchist> <109B26FC-32A9-4DDE-9639-E48A0389455D@stufft.io> <52493DF9.7010705@v.loewis.de> <5B4A7F23-C9E7-464E-A7A2-05CE3084CC67@stufft.io> <5249659B.1060305@v.loewis.de> <CADiSq7dVsXJpxHDx5OipH-x34g9KwR2zZh8LZ_6svYz_tW3zHw@mail.gmail.com> <20130930100256.53949f86@limelight.wooz.org> <CANc-5UwkZ4BDNJ554Mnhuz6tawGYL+KdjgiG0C12di7pgncPAQ@mail.gmail.com> Message-ID: <CADiSq7csmzk6pvW2U4y6BMh1BEU5i7r_v_15nZ8W4O=-8xHDqw@mail.gmail.com> On 1 October 2013 00:35, Skip Montanaro <skip at pobox.com> wrote: > Splitting into two pieces also means you can implement it for 3.4 > first and identify possible problems caused by preexisting pip > installs before deciding whether to add it to 2.7 and 3.3. One of the key reasons for using the bootstrap mechanism rather than just bundling pip is so that the installers for maintenance releases won't have any more trouble with pre-existing installs of pip than pip itself does :) But yes, there are several advantages to splitting the PEP: - the Python 3.4 changes are non-controversial, so it makes sense to get them officially accepted - the Python 3.4 beta deadline is closer than the deadlines for the 3.3.3 and 2.7.6 maintenance releases - it allows the full extent of the changes proposed for backporting to be clear prior to the reconsideration of the separated PEP - a Windows installer for pip could be made available well in advance of the 3.3.3 and 2.7.6 maintenance releases - it allows a chance to see if a separate Windows installer for pip and documenting the bootstrapping instructions in the "Installing Python Modules" guide for older releases is sufficient to get new users over the "getting started with PyPI" barrier Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From eliben at gmail.com Mon Sep 30 17:33:21 2013 From: eliben at gmail.com (Eli Bendersky) Date: Mon, 30 Sep 2013 08:33:21 -0700 Subject: [Python-Dev] Semi-official read-only Github mirror of the CPython Mercurial repository In-Reply-To: <CANc-5UwZW3LLoYyQbcnZWWe8u=ZAJyHGuwARf=8mTLf3pqQz+Q@mail.gmail.com> References: <CAF-Rda__36CpmDaVxLWo+iWhw5x9SqNFGjoQFXWM591PGNOzhg@mail.gmail.com> <CANc-5UwZW3LLoYyQbcnZWWe8u=ZAJyHGuwARf=8mTLf3pqQz+Q@mail.gmail.com> Message-ID: <CAF-Rda8YjV8B2WeSg3Gk=Y459YNR40yzYU1ubyHo+uADc9jg7w@mail.gmail.com> On Mon, Sep 30, 2013 at 7:08 AM, Skip Montanaro <skip at pobox.com> wrote: > > https://github.com/python/cpython is now live as a semi-official, *read > > only* Github mirror of the CPython Mercurial repository. Let me know if > you > > have any problems/concerns. > > Thanks for this, Eli. I use git regularly at work, so I'm getting much > more comfortable with it. Do you have a suggested workflow for people > who might want to use Git in preference to Hg, but still have write > access? > Petri Lehtinen runs a clone at https://github.com/akheron/cpython which uses more advanced tricks like hg-git and his own git-hg wrapper to allow him to do this, AFAIK. If this path is important for you, contact Petri and he can provide guidance on how to set it up. For python/cpython I really wanted the simplest flow to reach a read-only stage, because I lack the necessary git/hg-fu to set up and maintain something more complex "semi-officially". I think the most common workflow is to do small changes vs. Mercurial anyway. When I'm working on longer-term patches I do them in my Git mirror (because I prefer Git branching) and then apply & test patches onto a Mercurial R/W clone before committing. Thus, for most users I think the read-only mirror is preferable because it's much easier to use and reason about. All writes go solely through Mercurial, so it's clear where the official source is :) Eli -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130930/e2c9204c/attachment.html> From N.D.Efford at leeds.ac.uk Mon Sep 30 12:39:40 2013 From: N.D.Efford at leeds.ac.uk (Nick Efford) Date: Mon, 30 Sep 2013 11:39:40 +0100 (BST) Subject: [Python-Dev] Python for new users In-Reply-To: <mailman.2808.1380306720.18129.python-dev@python.org> References: <mailman.2808.1380306720.18129.python-dev@python.org> Message-ID: <alpine.LRH.2.03.1309301053220.26214@leeds.ac.uk> In the 'PEP453 ready for pronouncement' thread, Donald said >> Because reality is that new users are still likely to be using Python 2.7. >> Python 3 is just now starting to be really usable, however there's a huge >> corpus of existing tutorials, course work, books etc for Python 2.7. As >> Python 3 becomes more usable that existing corpus of material will be >> ported over to Python 3 but in the interim there is still a pretty large >> hurdle for new users to get over. And Guido replied > Based on my day-to-day experience this is still very true. (And yes, I'm > slowly turning the tide. But it will take a long time and I am committed to > giving users the choice.) Widely-used and linked web resources tend to persist for a very long time, so we shouldn't use the prevalence of Python 2 resources as a reason for excessive caution. The key question is how much good material is available based on Python 3 - and this has improved significantly over the past couple of years. The classic "How to Think Like a Computer Scientist" has an excellent Python 3 version available at http://interactivepython.org, for example. Things are changing with print media, too. Pragmatic Programmers are about to publish the 2nd edition of "Practical Programming", based on Python 3. Most of the major academic publishers have released Python 3 books in the last 12 months. The tide is definitely turning, perhaps has already turned. Encouraging the continued use of 2.7 for existing programmers is entirely justifiable, but for *newcomers* to programming I think it is now much harder to justify. A stronger case could have been made a couple of years ago, when many important packages were not yet available for Python 3, but things have changed. Even big frameworks like Django are now usable with Python 3. If we aren't yet past the point where package availability shouldn't be regarded as an adoption barrier by beginners, we are surely very close. I've been teaching Python as a first language to university students for many years now, initially with Python 2 and for the last few years with Python 3. In my experience, they encounter fewer problems with Python 3 (just as Guido intended, no doubt :) The one stumbling block in the past has been package availability for project work, but I don't expect that to a problem this year. All of the web and GUI development work that they'll be doing with me, for example, will be done entirely in Python 3. Nick From eric at trueblade.com Mon Sep 30 19:38:07 2013 From: eric at trueblade.com (Eric V. Smith) Date: Mon, 30 Sep 2013 13:38:07 -0400 Subject: [Python-Dev] Getting Tulip (PEP 3156) into the 3.4 stdlib, marked provisional, named asyncio In-Reply-To: <CAP7+vJ+5ij-w0KAcF+fkFnwgrWcRTKUgkSMoamT-GWHt7o9b1w@mail.gmail.com> References: <CAP7+vJKSOHYeCqhsJVF_7epTDeOw4UP3crsK_mY7HaFYnwG1aA@mail.gmail.com> <CAP1=2W7A2Q3t7Bk7iDZ9ZPprFdi+yr152EgVV=ACaTZkaBbUmw@mail.gmail.com> <5248C873.6060701@trueblade.com> <CAP7+vJ+5ij-w0KAcF+fkFnwgrWcRTKUgkSMoamT-GWHt7o9b1w@mail.gmail.com> Message-ID: <5249B6FF.7030805@trueblade.com> On 09/29/2013 08:55 PM, Guido van Rossum wrote: > On Sun, Sep 29, 2013 at 5:40 PM, Eric V. Smith <eric at trueblade.com > <mailto:eric at trueblade.com>> wrote: > > On 9/27/2013 9:14 PM, Brett Cannon wrote: > > > I don't see any issue with redirecting the discussion. > python-tulip@ is > > acting like a SIG for the module, so no real precedent beyond it not > > being hosted as a mail.python.org <http://mail.python.org> > <http://mail.python.org> list. > > I'm sure I'm in the minority, but I'd like the discussion to take place > on a python.org <http://python.org> mailing list. I don't want to > log in to a Google > property, and I don't trust them with the mailing list archives. > > I know my voice counts less than active Tulip discussion participants, > but now at least I feel better for having said something. > > > I wish you'd said something a looong time ago when it would have been > easy to move the list. Even if we moved it now we'd have split archives. > Also, I'm not sure where the paranoia comes from. FWIW I'm less worried > about Google reading my personal email than about the python.org > <http://python.org> webmasters reading it. I wish I'd known the project would be such a success! I have a use for tulip, but not much time to offer development help currently, so I'll live with the outcome. I'd love to see it make 3.4. Eric. From solipsis at pitrou.net Mon Sep 30 20:11:49 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 30 Sep 2013 20:11:49 +0200 Subject: [Python-Dev] cpython (2.7): Add fake buildbottouch target. References: <3cpQhd3DJ5zPWd@mail.python.org> Message-ID: <20130930201149.4a034831@fsol> On Mon, 30 Sep 2013 16:18:57 +0200 (CEST) martin.v.loewis <python-checkins at python.org> wrote: > http://hg.python.org/cpython/rev/c1a294bbb4fa > changeset: 85882:c1a294bbb4fa > branch: 2.7 > parent: 85877:dd55d54b2a15 > user: Martin v. L?wis <martin at v.loewis.de> > date: Mon Sep 30 16:18:31 2013 +0200 > summary: > Add fake buildbottouch target. "make touch" does exist in 2.7, so was this new target needed? Regards Antoine. From solipsis at pitrou.net Mon Sep 30 20:16:44 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 30 Sep 2013 20:16:44 +0200 Subject: [Python-Dev] peps: document requirements References: <3cpRnD5xP2z7Lkm@mail.python.org> Message-ID: <20130930201644.4749fd69@fsol> On Mon, 30 Sep 2013 17:08:00 +0200 (CEST) christian.heimes <python-checkins at python.org> wrote: > + > +* It should return ``0`` for zero length input. (Note: This can be handled as > + special case, too.) What is this required for? The main requirement is that equal stuff hashes equal, but there's no reason for zero-length input to hash to zero, AFAIK. Regards Antoine. From christian at python.org Mon Sep 30 21:31:10 2013 From: christian at python.org (Christian Heimes) Date: Mon, 30 Sep 2013 21:31:10 +0200 Subject: [Python-Dev] peps: document requirements In-Reply-To: <20130930201644.4749fd69@fsol> References: <3cpRnD5xP2z7Lkm@mail.python.org> <20130930201644.4749fd69@fsol> Message-ID: <5249D17E.80304@python.org> Am 30.09.2013 20:16, schrieb Antoine Pitrou: > On Mon, 30 Sep 2013 17:08:00 +0200 (CEST) > christian.heimes <python-checkins at python.org> wrote: >> + >> +* It should return ``0`` for zero length input. (Note: This can be handled as >> + special case, too.) > > What is this required for? The main requirement is that equal stuff > hashes equal, but there's no reason for zero-length input to hash to > zero, AFAIK. http://hg.python.org/cpython/file/d7ba4ca59023/Objects/object.c#l852 We make the hash of the empty string be 0, rather than using (prefix ^ suffix), since this slightly obfuscates the hash secret It's not a hard requirement, hence 'should' instead of 'must' like in all other cases. Christian