Mailman 3 April 2005 - Python-Dev

defmacro (was: Anonymous blocks)
by Jim Jewett 25 Apr '05

25 Apr '05

Michael Chermside: > There are lots of things one might want to replace within macros, from > identifiers to punctuation, but I'd be willing to live with just two of > them: expressions, and "series-of-statements" (that's almost the same as > a block). There are only two places I'd want to be able to USE a macro: > where an expression is called for, and where a series-of-statements is > called for. In both cases, I'd be happy with a function-call like syntax > for including the macro. I have often wanted to replace (parts of) strings, either because I'm writing a wrapper or because I want a non-English version to be loadable without having to wrap strings in my own source code. This is best done as an import hook, but if I had read-write access to (a copy of) the source code, I would use it. I'm not sure I want that door opened, because if I start needing to parse regex substitions just to get a source code listing ... I won't be happy. I do think macros should be prevented from "changing the level" of the code it replaces. Any suites/statements/expressions (including parentheses and strings) that are open before the macro must still be open afterwards, and any opened inside the macro must be closed inside the macro. For example def foo(x): print x macro1(x) print x might print different values for x on the two lines, but I would be less comfortable if it could result in any of the following: def foo(x): print x while True: # An invisible loop, because of print x # Changing the indent level def foo(x): print x return # and you thought it would print twice! (This one is iffy) print x def foo(x) print x [(""" (unclosed string or paren eats up the rest of the file...) print x def foo(x) "Hah! my backspaces and rubouts eliminated the print statements!" def foo(x) print x def anotherfunc(x, y, z): print x # Hey, I didn't even mess with the indent! And to be honest, even def foo(x): macro1(x) stmt1() # syntax error, except for the macro, so not ambiguous expanding to def foo(x) while x: print x # macro does not end on same indent level stmt1() is ... not something I want to worry about when I'm reading. Michael Chermside: > (hint: the answer is that it ALL happens at runtime). I have mixed feelings on this. It is more powerful that way, but it also limits future implementations -- and I'm not sure the extra power is entirely a good thing. defmacro(): print x # Hey, x was in global scope at runtime when *I* tested On The Other Hand, this certainly isn't the only piece of python that could *usually* be moved to compile-time, and I suppose it could piggyback on whatever extension is used for speeding up attribute lookup. -jJ

1 0

RE: [Python-Dev] defmacro (was: Anonymous blocks)
by Robert Brewer 25 Apr '05

25 Apr '05

Michael Chermside wrote: > Jim: > > Why not just introduce macros? If the answer is "We > > should, it is just hard to code", then use a good > > syntax for macros. If the answer is "We don't want > > xx sss (S\<! 2k3 ] > > to ever be meaningful", then we need to figure out exactly what to > > prohibit. > [...] > > Do we want to limit the changing part (the "anonymous block") to > > only a single suite? That does work well with the "yield" > syntax, but it > > seems like an arbitrary restriction unless *all* we want > are resource > > wrappers. > > > > Or do we really just want a way to say that a function > should share its > > local namespace with it's caller or callee? In that case, > maybe the answer > > is a "lexical" or "same_namespace" keyword. > > My own opinion is that we DO want macros. I prefer a language > have a few, > powerful constructs rather than lots of specialized ones. (Yet I still > believe that "doing different things should look > different"... which is > why I prefer Python to Lisp.) I think that macros could solve a LOT of > problems. > > There are lots of things one might want to replace within macros, from > identifiers to punctuation, but I'd be willing to live with > just two of them: expressions, and "series-of-statements" > (that's almost the same as a block). There are only two places > I'd want to be able to USE a macro: where an expression is > called for, and where a series-of-statements is called for. > In both cases, I'd be happy with a function-call > like syntax for including the macro. By "function-call like syntax" you mean something like this? def safe_file(filename, body, cleanup): f = open(filename) try: body() finally: f.close() cleanup() ... defmacro body: for line in f: print line[:line.find(":")] defmacro cleanup: print "file closed successfully" safe_file(filename, body, cleanup) If macros were to be evaluated at runtime, I'd certainly want to see them be first-class (meaning able to be referenced and passed around); I don't have much of a need for anonymous macros. Robert Brewer MIS Amor Ministries fumanchu(a)amor.org

1 0

RE: [Python-Dev] defmacro (was: Anonymous blocks)
by Michael Chermside 25 Apr '05

25 Apr '05

Guido writes: > My problem with macros is actually more practical: Python's compiler > is too dumb. I am assuming that we want to be able to import macros > from other modules, and I am assuming that macros are expanded by the > compiler, not at run time; but the compiler doesn't follow imports > (that happens at run time) so there's no mechanism to tell the > compiler about the new syntax. And macros that don't introduce new > syntax don't seem very interesting (compared to what we can do > already). That's good to hear. It expresses fairly clearly what the challenges are in implementing macros for Python, and expressing the challenges makes it easier to attack the problem. My interest comes because some recent syntax changes (generators, generator expressions) have seemed to me like true language changes, but others (decorators, anonymous-blocks) to me just cry out "this would be easy as a macro!". I wrote: > I think it would be useful if we approached it like this: either what > we want is the full power of macros (in which case the syntax we choose > should be guided by that choice), or we want LESS than the full power > of macros. If we want less, then HOW less? > > In other words, rather than hearing what we'd like to be able to DO > with blocks, I'd like to hear what we want to PROHIBIT DOING with > blocks. I think this might be a fruitful way of thinking about the > problem which might make it easier to evaluate syntax suggestions. And > if the answer is that we want to prohibit nothing, then the right > solution is macros. Guido replied: > I'm personally at a loss understanding your question here. Perhaps you > could try answering it for yourself? You guys just think too fast for me. When I started this email, I replied "Fair enough. One possibility is...". But while I was trying to condense my thoughts down from 1.5 pages to something short and coherent (it takes time to write it short) everything I was thinking became obscelete as both Paul Moore and Jim Jewett did exactly the kind of thinking I was hoping to inspire: Paul: > What I feel is the key concept here is that of "injecting" code into a > template form (try...finally, or try..except..else, or whatever) [...] > Specifically, cases where functions aren't enough. If I try to > characterise precisely what those cases are, all I can come up with is > "when the code being injected needs to run in the current scope, not > in the scope of a template function". Is that right? Jim: > Why not just introduce macros? If the answer is "We should, it is just > hard to code", then use a good syntax for macros. If the answer is > "We don't want > xx sss (S\<! 2k3 ] > to ever be meaningful", then we need to figure out exactly what to > prohibit. [...] > Do we want to limit the changing part (the "anonymous block") to > only a single suite? That does work well with the "yield" syntax, but it > seems like an arbitrary restriction unless *all* we want are resource > wrappers. > > Or do we really just want a way to say that a function should share its > local namespace with it's caller or callee? In that case, maybe the answer > is a "lexical" or "same_namespace" keyword. My own opinion is that we DO want macros. I prefer a language have a few, powerful constructs rather than lots of specialized ones. (Yet I still believe that "doing different things should look different"... which is why I prefer Python to Lisp.) I think that macros could solve a LOT of problems. There are lots of things one might want to replace within macros, from identifiers to punctuation, but I'd be willing to live with just two of them: expressions, and "series-of-statements" (that's almost the same as a block). There are only two places I'd want to be able to USE a macro: where an expression is called for, and where a series-of-statements is called for. In both cases, I'd be happy with a function-call like syntax for including the macro. Well, that's a lot of "wanting"... now I all I need to do is invent a clever syntax that allows these in an elegant fashion while also solving Guido's point about imports (hint: the answer is that it ALL happens at runtime). I'll go think some while you guys zoom past me again. <wink> -- Michael Chermside

1 0

Proper place to put extra args for building
by Brett C. 25 Apr '05

25 Apr '05

I am currently adding some code for a Py_COMPILER_DEBUG build for use on the AST branch. I thought that OPT was the proper variable to put stuff like this into for building (``-DPy_COMPILER_DEBUG``), but that erases ``-g -Wall -Wstrict-prototypes``. Obviously I could just tack all of that into my own thing, but that seems like an unneeded step. >From looking at Makefile.pre.in it seems like CFLAGSFORSHARED is meant for extra arguments to the compiler. Is that right? And I will document this in Misc/Specialbuilds.txt to fix my initial blunderous checkin of specifying OPT (or at least clarifying it). -Brett

2 13

PEP 310 and exceptions
by hpk＠trillke.net 25 Apr '05

25 Apr '05

Hi all, probably unsuprisingly i am still pondering the idea of having an optional __except__ hook on block handlers. The PEP says this about this: An extension to the protocol to include an optional __except__ handler, which is called when an exception is raised, and which can handle or re-raise the exception, has been suggested. It is not at all clear that the semantics of this extension can be made precise and understandable. For example, should the equivalent code be try ... except ... else if an exception handler is defined, and try ... finally if not? How can this be determined at compile time, in general? In fact, i think the translation even to python code is not that tricky: x = X(): ... basically translates to: if hasattr(x, '__enter__'): x.__enter__() try: ... except: if hasattr(x, '__except__'): x.__except__(...) else: x.__exit__() else: x.__exit__() this is the original definition from the PEP with the added except clause. Handlers are free to call 'self.__exit__()' from the except clause. I don't think that anything needs to be determined at compile time. (the above can probably be optimized at the bytecode level but that is a side issue). Moreover, i think that there are more than the "transactional" use cases mentioned in the PEP. For example, a handler may want to log exceptions to some tracing utility or it may want to swallow certain exceptions when its block does IO operations that are ok to fail. cheers, holger

11 19

Error checking in init<module> functions
by Thomas Heller 24 Apr '05

24 Apr '05

I always wondered why there usually is very sloppy error checking in init<module> functions. Usually it goes like this (I removed declarations and some other lines for clarity): PyMODINIT_FUNC PyInit_zlib(void) { m = Py_InitModule4("zlib", zlib_methods, zlib_module_documentation, (PyObject*)NULL,PYTHON_API_VERSION); ZlibError = PyErr_NewException("zlib.error", NULL, NULL); if (ZlibError != NULL) { Py_INCREF(ZlibError); PyModule_AddObject(m, "error", ZlibError); } PyModule_AddIntConstant(m, "MAX_WBITS", MAX_WBITS); PyModule_AddIntConstant(m, "DEFLATED", DEFLATED); ver = PyString_FromString(ZLIB_VERSION); if (ver != NULL) PyModule_AddObject(m, "ZLIB_VERSION", ver); PyModule_AddStringConstant(m, "__version__", "1.0"); } Why isn't the result checked in the PyModule_... functions? Why is the failure of PyErr_NewException silently ignored? The problem is that when one of these things fail (although they are probably supposed to NOT fail) you end up with a module missing something, without any error message. What would be the correct thing to do - I assume something like if (PyModule_AddIntConstant(m, "MAX_WBITS", MAX_WBITS)) { PyErr_Print(); return; } Thomas

3 2

Re: switch statement
by Jim Jewett 22 Apr '05

22 Apr '05

Michael Chermside wrote: > Now the pattern matching is more interesting, but again, I'd need to > see a proposed syntax for Python before I could begin to consider it. > If I understand it properly, pattern matching in Haskell relies > primarily on Haskell's excellent typing system, which is absent in > Python. Why not just use classes? With either mixins or new-style classes, it is quite reasonable to use many small classes for fine distinctions. Change if predicate1(obj): action1(obj) elif predicate2(obj): action2(obj) ... else: default(obj) into either try: obj.action(locals()) except AttributeError: default(obj, locals()) or if hasattr(obj, "action"): obj.action(locals()) else: <default> And then define an action method (perhaps through inheritance from a mixin) for any object that should not take the default path. The object's own methods will have access to any variables used in the match and locals will have access to the current scope. If you have at least one class per "switch", you have a switch statement. The down sides are that (1) Your domain objects will have to conform to a least a weak OO model (or take the default path) (2) Logic that should be together will be split up. Either classes will be modified externally, or the "switch statement" logic will be broken up between different classes. If single-method mixins are used to keep the logic close, then real objects will have to pick an ancestor for what may seem like arbitrary reasons. These objections apply to any matching system based on types; the difference is that other languages have often already paid the price. For Python it is an incremental cost incurred by the match system. -jJ

1 0

Re: [Python-Dev] anonymous blocks
by Greg Ewing 22 Apr '05

22 Apr '05

Ka-Ping Yee wrote: > Can you explain what you meant by currying here? I know what > the word "curry" means, but i am having a hard time seeing how > it applies to your example. It's currying in the sense that instead of one function which takes all the args at once, you have a function that takes some of them (all except the thunk) and returns another one that takes the rest (the thunk). > Could you make up an example that uses more arguments? def with_file(filename, mode): def func(block): f = open(filename, mode) try: block(f) finally: f.close() return func Usage example: with_file("foo.txt", "w") as f: f.write("My hovercraft is full of parrots.") Does that help? -- Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg.ewing(a)canterbury.ac.nz +--------------------------------------+

1 0

Reference counting when entering and exiting scopes
by Matthew F. Barnes 22 Apr '05

22 Apr '05

Someone on python-help suggested that I forward this question to python-dev. I've been studying Python's core compiler and bytecode interpreter as a model for my own interpreted language, and I've come across what appears to be a reference counting problem in the `symtable_exit_scope' function in <Python/compile.c>. At this point I assume that I'm just misunderstanding what's going on. So I was hoping to contact one of the core developers before I go filing what could very well be a spurious bug report against Python's core. Here's the function copied from CVS HEAD: static int symtable_exit_scope(struct symtable *st) { int end; if (st->st_pass == 1) symtable_update_free_vars(st); Py_DECREF(st->st_cur); end = PyList_GET_SIZE(st->st_stack) - 1; st->st_cur = (PySymtableEntryObject *)PyList_GET_ITEM(st->st_stack, end); if (PySequence_DelItem(st->st_stack, end) < 0) return -1; return 0; } My issue is with the use of PyList_GET_ITEM to fetch a new value for the current scope. As I understand it, PyList_GET_ITEM does not increment the reference count for the returned value. So in effect we're borrowing the reference to the symtable entry object from the tail of the scope stack. But then we turn around and delete the object from the tail of the scope stack, which DOES decrement the reference count. So `symtable_exit_scope' has a net effect of decrementing the reference count of the new current symtable entry object, when it seems to me like it should stay the same. Shouldn't the reference count be incremented when we assign to "st->st_cur" (either explicitly or by fetching the object using the PySequence API instead of PyList)? Can someone explain the rationale here? --------------------------------------------------------------------------- As an addendum to the previous question, further study of the <Python/compile.c> code has made me believe that there's a reference counting problem in the `symtable_enter_scope' function as well (pasted below from CVS HEAD). Namely, that `prev' should be Py_XDECREF'd at some point in the function (at the end of the first IF block, perhaps?). static void symtable_enter_scope(struct symtable *st, char *name, int type, int lineno) { PySymtableEntryObject *prev = NULL; if (st->st_cur) { prev = st->st_cur; if (PyList_Append(st->st_stack, (PyObject *)st->st_cur) < 0) { st->st_errors++; return; } } st->st_cur = (PySymtableEntryObject *) PySymtableEntry_New(st, name, type, lineno); if (st->st_cur == NULL) { st->st_errors++; return; } if (strcmp(name, TOP) == 0) st->st_global = st->st_cur->ste_symbols; if (prev && st->st_pass == 1) { if (PyList_Append(prev->ste_children, (PyObject *)st->st_cur) < 0) st->st_errors++; } } Thanks, Matthew Barnes

3 4

marshal / unmarshal
by Scott David Daniels 21 Apr '05

21 Apr '05

What should marshal / unmarshal do with floating point NaNs (the case we are worrying about is Infinity) ? The current behavior is not perfect. Michael Spencer chased down a supposed "Idle" problem to (on Win2k): marshal.dumps(1e10000) == 'f\x061.#INF' marshal.loads('f\x061.#INF') == 1.0 Should loads raise an exception? Somehow, I thing 1.0 is not the best possible representation for +Inf. -- Scott David Daniels Scott.Daniels(a)Acm.Org

13 36