Mailman 3 February 2012 - Python-ideas

Current status of PEP 403 (was Re: peps: Switch back to named functions, since the Ellipsis version degenerated badly)
by Nick Coghlan Feb. 24, 2012

Feb. 24, 2012

(switching lists to python-ideas, dropping python-dev and python-checkins) Context for anyone not following python-checkins: I recently moved PEP 3150 (statement local namespaces) to Withdrawn, rewrote PEP 403 with a different syntax proposal, retitled it to "Statement local class and function definitions" and moved it to Deferred. On Fri, Feb 24, 2012 at 2:37 AM, Jim Jewett <jimjjewett(a)gmail.com> wrote: > I understand that adding a colon and indent has its own problems, but > ... I'm not certain this is better, and I am certain that the desire > for indentation is strong enough to at least justify discussion in the > PEP. Fair point. The reason for the flat structure is that allowing a full suite screws with the scoping rules and gets us into the land of insane complexity that was PEP 3150. A decorator-inspired syntax makes it very clear (both to the reader and to the compiler) that there's only *one* name being forward referenced, rather than potentially hiding declarations of forward references an arbitrary distance from the statement that uses them. This doesn't actually lose any flexibility, since you can just make a forward reference to a class instead and use that as your local namespace (with ordinary attribute access semantics), rather than the brain-bender that was the proposed scoping rules for PEP 3150's given clause. The only other alternative syntax would be to use a custom suite definition that allowed only a single class or function definition statement, but I think having something that looks like a suite, but isn't one would be significantly worse than the current proposed syntax that merely allows a function (or class) definition's implied local name binding to be overridden with a custom statement. To (almost*) recreate the effect of an ordinary function definition with the in statement, you could write: in f = f def f(): pass And a decorated definition like: @deco1 @deco2 @deco3 def f(): pass Could (almost*) be expressed as: in f = deco1(deco2(deco3(f))) def f(): pass * The reason for the "almost" caveat is that, given the current PEP 403 semantics, recursive references to f() will resolve differently for the "in" statement cases - for the in statement, they will resolve directly to the innermost function definition, while for ordinary definitions they will be resolved according to the scoping rules for any name lookup. This could be an argument in favour of allowing *decorated* function and class definitions, rather than requiring that they be undecorated - if decorators are allowed, then recursive references would resolve directly to the post-decorated version. Alternatively, people could adopt a convention of prepending an underscore to the actual function name in cases where it mattered, meaning they would have easy access to *both* forms of the function (decorated and undecorated): in f = deco1(deco2(deco3(_f))) def _f(): return f, _f # decorated, undecorated In either case, whereas an ordinary recursive function definition can get confused by reassignments in the outer scope, an in-statement based definition would be truly recursive (via a cell reference) and hence ignore any subsequent changes in the outer namespace. Something else the PEP should mention explicitly is that, like __class__, a class object obviously won't be available while the class body is being executed. Only methods will be able to refer to the class by name, just as only methods can use __class__. Updates to PEP 403 are going to be pretty sporadic until some time after 3.3 release though - it's still very much in "this is a problem I am thinking about" territory rather than "this is a language addition I am proposing" (the latest round of updates were just to make sure I recorded my latest idea before I forgot about the details). Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

1 0

doctest (re-send to list)
by Mark Janssen Feb. 22, 2012

Feb. 22, 2012

On Mon, Feb 20, 2012 at 11:28 AM, Barry Warsaw <barry(a)python.org> wrote: > On Feb 17, 2012, at 02:57 PM, Mark Janssen wrote: > FWIW, I think doctests are fantastic and I use them all the time. There are > IMO a couple of things to keep in mind: > > - doctests are documentation first. Specifically, they are testable > documentation. What better way to ensure that your documentation is > accurate and up-to-date? (And no, I do not generally find skew between the > code and the separate-file documentation.) > > - I personally dislike docstring doctests, and much prefer separate reST > documents. These have several advantages, such as the ability to inject > names into doctests globals (use with care though), and the ability to set > up the execution context for doctests (see below). The fact that it's so > easy to turn these into documentation with Sphinx is a huge win. > > Since so many people point this out, let me say that I completely agree that > doctests are not a *replacement* for unittests, but they are a fantastic > *complement* to unittests. When I TDD, I always start writing the > (testable) documentation first, because if I cannot explain the component > under test in clearly intelligible English, then I probably don't really > understand what it is I'm trying to write. > > My doctests usually describe mostly the good path through the API. > Occasionally I'll describe error modes if I think those are important for > understanding how to use the code. However, for all those fuzzy corner cases, > weird behaviors, bug fixes, etc., unittests are much better suited because > ensuring you've fixed these problems and don't regress in the future doesn't > help the narrative very much. I think is an example of (mal)adapting to an incomplete module, rather than fixing it. I think doctest can handle all the points you're making. See clarification pointers below... >>1. Execution context determined by outer-scope doctest defintions. > > Can you explain this one? I gave an example in a prior message on this thread, dated Feb 17. I think it's clear there but let me know. Basically, the idea is that since the class def can also have a docstring, where better would setup and teardown code go to provide the execution context of the inner method docstrings? Now the question: is it useful or appropriate to put setup and teardown code in a classdef docstring? Well, I think this requires a committment on the behalf of the coder/documentor to concoct useful (didactic) example that could go there. For example, (as in the prior-referenced message) I imagine putting example of defining a variable of the classes type (">>> g = Graph({some complex, interesting initialization})"), which might return a (testable) value upon creation. Now this could, logically, be put in the classes __init__ method, but that doesn't make sense for defining an execution context, and *in addition*, that can be saved for those complex corner cases you mentioned earlier. > I usually put all this in an additional_tests() method, such as: Yes, I do the same for my modules with doctests. A dummy function which can catch all the non-interesting tests. This, still superior, in my opinion, than unittest. It is easier syntactically, as well as for casual users of your code (It has no leaning curve like understanding unittest). This superiority to unittest, by the way, is only realized if the second suggestion (smart comparisons) is implemented into doctest. >>2. Smart Comparisons that will detect output of a non-ordered type >>(dict/set), lift and recast it and do a real comparison. > > I'm of mixed mind with these. Yes, you must be careful with ordering, but I > find it less readable to just sort() some dictionary output for example. What > I've found much more useful is to iterate over the sorted keys of a dictionary > and print the key/values pairs. Yes, but you see you're destroying the very intent and spirit of doctest. The point is to make literate documentation. If you adapt to it's incompleteness, you reduce the power of it. >>Without #1, "literate testing" becomes awash with re-defining re-used >>variables which, generally, also detracts from exact purpose of the >>test -- this creates testdoc noise and the docs become less useful. >>Without #2, "readable docs" nicely co-aligning with "testable docs" >>tends towards divergence. > > I've no doubt that doctests could be improved, but I actually find them quite > usable as is, with just a little bit of glue code to get it all hooked up. As > I say though, I'm biased against docstring doctests. Well, hopefully, I've convinced you a little that the limitations in doctests over unittests are almost, if not entirely due, to the incompleteness of the module. If the two items I mentioned were implemented I think it would be far superior to unittest. (Corner cases, etc can all find a place, because every corner case should be documented somewhere anyway!!) Cheers!! mark santa fe, nm

1 0

sys.path is a hack - bringing it back under control
by anatoly techtonik Feb. 21, 2012

Feb. 21, 2012

Hi, I often find this in my scripts/projects, that I run directly from checkout: DEVPATH = os.path.dirname(os.path.abspath(__file__)) sys.path.insert(0, DEVPATH) This seems like a hack to me, because the process of sys.path modification is completely out of control for Python application developer, which means it is easy to break an application and get lost. I don't remember the exact user story for that bad association with sys.path (perhaps Django issue #1908), but something makes me feel that I am not alone: http://stackoverflow.com/questions/5500736/troubleshooting-python-sys-path What I'd like to propose is some control/info over what modified sys.path. The simplest case: 1. make sys.path a list of pairs (path, file-that-added-the-path) 2. make sys.path read-only 3. add sys.path.add() method for modification 4. logger for sys.path.add() events (or recipe how to implement it in documentation) This will help a lot. Limiting sys.path may cause a loss of some functionality if you need to remove some or replace it completely, but I don't know where the ability to reset() sys.path can be useful. -- anatoly t.

7 12

Lack a API WCHAR* PyUnicode_WCHAR_DATA(PyObject *o), This is important for OS dependent feature port.
by 罗勇刚(Yonggang Luo) Feb. 20, 2012

Feb. 20, 2012

Py_UCS1 <http://docs.python.org/dev/c-api/unicode.html#Py_UCS1>* PyUnicode_1BYTE_DATA(PyObject<http://docs.python.org/dev/c-api/structures.html#PyObject> * *o*) Py_UCS2 <http://docs.python.org/dev/c-api/unicode.html#Py_UCS2>* PyUnicode_2BYTE_DATA(PyObject<http://docs.python.org/dev/c-api/structures.html#PyObject> * *o*) <http://docs.python.org/dev/c-api/unicode.html#PyUnicode_1BYTE_DATA> <http://docs.python.org/dev/c-api/unicode.html#PyUnicode_2BYTE_DATA> Py_UCS4<http://docs.python.org/dev/c-api/unicode.html#Py_UCS4> * PyUnicode_4BYTE_DATA(PyObject<http://docs.python.org/dev/c-api/structures.html#PyObject> * *o*) <http://docs.python.org/dev/c-api/unicode.html#PyUnicode_4BYTE_DATA> Return a pointer to the canonical representation cast to UCS1, UCS2 or UCS4 integer types for direct character access. No checks are performed if the canonical representation has the correct character size; use PyUnicode_KIND()<http://docs.python.org/dev/c-api/unicode.html#PyUnicode_KIND> to select the right macro. Make sure PyUnicode_READY()<http://docs.python.org/dev/c-api/unicode.html#PyUnicode_READY> has been called before accessing this. New in version 3.3. PyUnicode_WCHAR_KIND<http://docs.python.org/dev/c-api/unicode.html#PyUnicode_WCHAR_KIND> PyUnicode_1BYTE_KIND<http://docs.python.org/dev/c-api/unicode.html#PyUnicode_1BYTE_KIND> PyUnicode_2BYTE_KIND<http://docs.python.org/dev/c-api/unicode.html#PyUnicode_2BYTE_KIND> PyUnicode_4BYTE_KIND<http://docs.python.org/dev/c-api/unicode.html#PyUnicode_4BYTE_KIND> Return values of the PyUnicode_KIND()<http://docs.python.org/dev/c-api/unicode.html#PyUnicode_KIND> macro. New in version 3.3. int PyUnicode_KIND(PyObject<http://docs.python.org/dev/c-api/structures.html#PyObject> * *o*) <http://docs.python.org/dev/c-api/unicode.html#PyUnicode_KIND> Return one of the PyUnicode kind constants (see above) that indicate how many bytes per character this Unicode object uses to store its data. *o* has to be a Unicode object in the “canonical” representation (not checked). New in version 3.3. -- 此致礼罗勇刚 Yours sincerely, Yonggang Luo

3 3

Personal Project Roadmap (Was: sys.path is a hack - bringing it back under control)
by anatoly techtonik Feb. 20, 2012

Feb. 20, 2012

On Mon, Feb 20, 2012 at 1:38 PM, Nick Coghlan <ncoghlan(a)gmail.com> wrote: > (however, I can't yet promise that it will make it into 3.3, since it > doesn't even have a reference implementation yet, and I have several > other things I want to get done first). > I think that the idea of personal personal project roadmap would rock. If I'd like something to be done faster, I could look at these "other things" to see if I can help with some of them. In addition I could copy some stuff to my own list to say that I am also interested. Once the item reaches the top in somebody's list (or there a critical mass is reached), he opens a hangout with other people or schedules a time for discussion. The login method is Python account. Items are either bugs from trackers or short inline notes in a tree-like structure. Will it improve the Python development process? -- anatoly t.

3 3

channel (synchronous queue)
by Matt Joiner Feb. 20, 2012

Feb. 20, 2012

Recently (for some) the CSP style of channel has become quite popular in concurrency implementations. This kind of channel allows sends that do not complete until a receiver has actually taken the item. The existing queue.Queue would act like this if it didn't treat a queue size of 0 as infinite capacity. In particular, I find channels to have value when sending data between threads, where it doesn't make sense to proceed until some current item has been accepted. This is useful when items are not purely CPU bound, and so generators are not appropriate. I believe this rendezvous behaviour can be added to queue.Queue for the maxsize=0 case, with maxsize=None being the existing "infinite queue" behaviour. Additionally a close method, Closed exception and other usability features like an __iter__ for receiving until closed can be added. The stackless class linked below also has some other possible ideas for performance reasons that make a lot of sense. Existing code using queue.Queue would remain completely unaffected by such additions if the default maxsize value is changed to maxsize=None, and maxsize=0 is not being explicitly passed (it's currently the default). Here are a few links for some background and ideas: http://gevent.org/gevent.queue.html#gevent.queue.Queue http://www.disinterest.org/resource/stackless/2.6-docs-html/library/stackle… http://en.wikipedia.org/wiki/Communicating_sequential_processes#Comparison_… http://golang.org/doc/go_spec.html#Channel_types

9 37

ScopeGuardStatement/Defer Proposal
by Manuel Barkhau Feb. 19, 2012

Feb. 19, 2012

Hi everybody, I'd like to suggest adopting something similar to the ScopeGuardStatement from the D programming language. A description of the D version can be found here: http://d.digitalmars.com/2.0/statement.html#ScopeGuardStatement It is also similar to golangs "defer" statement: http://golang.org/doc/go_spec.html#Defer_statements So these are roughly equivelent: defer {block} // in golang scope(exit) {block} // in D I have written a context manager that approximates the behavior of the scope statement in D: http://ideone.com/vNmq8 The use of lambdas or nested functions doesn't look very nice however. So, on to the proposal. I think the "defer" keyword is more appropriate than "scope" and the function like syntax of "scope(exit)" doesn't fit with the overall python syntax. There are three ways to define a defer block. - "defer: BLOCK", which is the same as "defer {BLOCK} in golang or "scope(exit) {BLOCK}" in D. - "defer EXPR as VAR: BLOCK", which is similar to "scope(failure)". It differs in that it specifies the exception that caused the failure and is only called for matching exceptions. - "defer EXPR: BLOCK else: BLOCK", where the else BLOCK is executed when no exception occurs. This is similar to "scope(success)" and the existing except: else: construct in python. As "defer:" is currently invalid syntax, there shouldn't be any code breakage from adding the new keyword. Some rules: - Deferred blocks are executed in the reverse lexical order in which they appear. - If a function returns before reaching a defer statement, it will not be executed. - If a defer block raises an error, a lexically earlier defer block may catch it. - If multiple defer blocks raise errors or return results, the raise or return of the lexically earlier defer will mask the previous result or error. Some example code: >>> def ordering_example(): ... print(1) ... defer: print(2) ... defer: print(3) ... print(4) ... >>> ordering_example() 1 4 3 2 Handling exceptions: >>> def defer_example(): ... # setup ... defer: # always executed ... # cleanup ... defer Exception as e: # executed if exception is raised ... # handle exception ... else: # executed if no exception is raised ... # success code ... ... # your usual code ... # possibly raise exception Equivalent using try/except/finally: >>> def try_example(): ... # setup ... try: ... # your usual code ... # possibly raise exception ... except Exception as e: ... # handle exception ... else: ... # success code ... finally: ... # cleanup The nesting advantage becomes more apparent when more are required. Here is an example from http://www.doughellmann.com/articles/how-tos/python-exception-handling/inde… #!/usr/bin/env python import sys import traceback def throws(): raise RuntimeError('error from throws') def cleanup(): raise RuntimeError('error from cleanup') def nested(): try: throws() except Exception as original_error: try: raise finally: try: cleanup() except: pass # ignore errors in cleanup def main(): try: nested() return 0 except Exception as err: traceback.print_exc() return 1 if __name__ == '__main__': sys.exit(main()) Here are the equivalent of main and nested functions using defer: def nested(): defer RuntimeError: pass # ignore errors in cleanup defer: cleanup() throws() def main(): defer Exception as err: traceback.print_exc() return 1 else: return 0 nested() Notice that we don't even need "defer Exception as original_error: raise" after "defer: cleanup()" in order to preserve the stack trace. It will go up the call stack, so long as no defer handles it or masks it with another exception. This proposal would probably have had a better chance before the introduction of the "with" statement, but I still think it may be useful in cases where you don't want to write a context manager. Context managers may also not have access to the scope they are used in, which may be inconvenient in some cases. For code where try/except/finally would otherwise be required, I think the advantages make this proposal at least worth considering. You don't need to nest your normal code in a try block and you can place error handling code together with relevant sections, rather than further down in an except block. I'm sure there is much I have overlooked, possibly this is technically difficult and of course there is the minor task of implementation. But other than that what do you think? Manuel

10 19

Adding shm_open to mmap?
by Mike Meyer Feb. 19, 2012

Feb. 19, 2012

One of the issues that showed up during the overlong TIOBE- thread and spinoffs is that there's no portable way to get a named shared memory segment (as distinguished from a disk-backed file) using the mmap module. Most unix variants provide a memory-backed file system that works for this, but it's name changes from distro to distro and even installation to installation. It's not clear to me that non-Unix platforms provide such a file system. The Posix solution is shm_open, which accepts a name for rendezvous and returns a file descriptor suitable for passing to mmap. Passing the file descriptor to anything but fstat, ftruncate, close and mmap is undefined. We'd also need to add shm_unlink to remove the shared segment, as the object created by shm_open isn't necessarily visible in the file system name space. shm_open has five values that can be used in it's flags argument, but those are shared with open and already available in the os module. This seems like a slam-dunk to me, but... 1) Is there some reason not to just add these two functions? 2) Are there any supported platforms with mmap and without shm_open/unlink? 3) Is this simple enough that a PEP isn't needed, just a patch in an issue? Thanks, <mike -- Mike Meyer <mwm(a)mired.org> http://www.mired.org/ Independent Software developer/SCM consultant, email for more information. O< ascii ribbon campaign - stop html mail - www.asciiribbon.org

7 23

Python 3000 TIOBE -3%
by anatoly techtonik Feb. 18, 2012

Feb. 18, 2012

Hi, I didn't want to grow FUD on python-dev, but a FUD there seems to be a good topic for discussion here. http://www.tiobe.com/index.php/content/paperinfo/tpci/index.html As you may see, Python is losing its positions. I blame Python 3 and that Python development is not concentrating on users enough [1], and that there is a big resistance in getting the things done (/moin/ prefix story) and the whole communication process is a bit discouraging. If it is not the cause, then the cause is the lack of visibility into the real problem, but what the real problem is? I guess the topic is for upcoming language summit at PyCon, but it will be hard for me to get there this year from Belarus, so it would be nice to read some opinions here. 1. http://python-for-humans.heroku.com/ -- anatoly t.

50 237

Fwd: Re: doctest
by Ian Bicking Feb. 18, 2012

Feb. 18, 2012

On Feb 17, 2012 3:58 PM, "Mark Janssen" <dreamingforward(a)gmail.com> wrote: > I find myself wanting to use doctest for some test-driven development, > and find myself slightly frustrated and wonder if others would be > interested in seeing the following additional functionality in > doctest: > > 1. Execution context determined by outer-scope doctest defintions > 2. Smart Comparisons that will detect output of a non-ordered type > (dict/set), lift and recast it and do a real comparison. > > Without #1, "literate testing" becomes awash with re-defining re-used > variables which, generally, also detracts from exact purpose of the > test -- this creates testdoc noise and the docs become less useful. I dunno... I find the discipline of defining your prerequesites to be a helpful feature of doctest (I find TestCase.setUp to be smelly). You can include a namespace in doctest invocations, but I'm guessing the problem is that you aren't able to give these settings when using some kind of test collector/runner? More flexible ways of defining doctest options (e.g., ELLIPSIS) would be helpful. > Without #2, "readable docs" nicely co-aligning with "testable docs" > tends towards divergence. IMHO this could be more easily solved by replacing the standard repr with one that is more predictable. At least that would handle dictionaries, it becomes a bit more difficult for custom types. Also it diverges from being exactly like the console, but eh, I don't think that's a big advantage. Unfortunately plugging in a custom repr is kind of hard; Python has a way to specifically compile expressions into "print repr(expr)" (more or less) but no general way to get the value of expressions (while also handling statements). But if you wanted to try it, I did figure out a terrible hack for it. Ian

2 1