On 5/29/07, Blake Winton wrote:

Steve Howell wrote:

...

Let the Python interpreter build expression objects for you. You write: rowexpr: convert_to_euros(salary) > 50000 and deptno = 10 Python translates that into the same bytecode as: lambda row: convert_to_euros(row['salary'] > 50000 and row['deptno'] == 10

I'm sorry, why would that not be translated into:

lambda row: row['convert_to_euros'](row['salary'] > row['50000'] row['and'] row['deptno'] == row['10']

? Specifically, how would python know what to dereference and what not to? What if there were two things, named the same, one in the row and one in the namespace? (i.e. a variable named 'salary') How would you escape things which would have been dereferenced but you didn't want to be? (i.e. "rowexpr: convert_to_euros(salary) > salary")

I guess I'm kind of wasting my time here, since the introduction of a new keyword for this application really isn't going to happen, based on other decisions I've seen Guido make, but I do think that you need to think a little more about how the implementation of this feature would work. (Or perhaps you've done that thinking, and you just need to fill in the proposal with that information.)

Indeed, such half baked ideas have no chance of being taken seriously as language additions. Take a look at packages such as buzhug [1] and (especially) SqlAlchemy [2], not only because they might end up solving your problem but also to appreciate the complexity involved in supporting arbitrary SQL-like expressions at the language level, since you clearly underestimate it or haven't quite thought about it. George [1] http://buzhug.sourceforge.net/ [2] http://www.sqlalchemy.org/docs/ -- "If I have been able to see further, it was only because I stood on the shoulders of million monkeys."

"BJörn Lindqvist" wrote:

On 5/29/07, George Sakkis wrote:

...

Indeed, such half baked ideas have no chance of being taken seriously as language additions.

Remember that this list is called python-ideas for a reason. It is the place for half-baked ideas.

Indeed, but when they are called out as half-baked (explicitly or implicitly), and are offered no love by others in the list, perhaps it's time to let them die. On the other hand, there is the other argument that comp.lang.python is the right place to post half-baked ideas, which are then baked, brought to python-ideas for another round of "you didn't put in blueberries", before making it to python-dev/python-3000 . - Josiah

--- Blake Winton wrote:

Steve Howell wrote:

...

Let the Python interpreter build expression objects for you. You write: rowexpr: convert_to_euros(salary) > 50000 and deptno = 10 Python translates that into the same bytecode as: lambda row: convert_to_euros(row['salary'] > 50000 and row['deptno'] == 10

My language here was imprecise. I should have said something to the effect of this: The interpreter would translate the "rowexpr" into bytecode that would execute the same way as the "lambda" example given the scope of the method convert_to_euros in the example above. Also, the bytecode would not be the "same" as the lambda; it would just have similar complexity.

I'm sorry, why would that not be translated into:

lambda row: row['convert_to_euros'](row['salary'] > row['50000'] row['and'] row['deptno'] == row['10']

I am suggesting that in a rowexpr expression, instead of the typical local scope being the default scope, "row" would be the local scope for dereferencing. Since I am proposing an expression syntax, does not the local scope go away? If you look at where salary, row, and deptno sit in the expression, it seems pretty clear to me that a well-defined scoping scheme would try to dereference row by default for all those tokens, and probably give up there, throwing an exception. The convert_to_euros token in the expression obviously poses a thornier problem. Do you prefer row.convert_to_euros, if such a think exists, or do you go right to what would be the typical scope of a similar lambda expression, and prefer, say, the definition of "def convert_to_euros" that sits at the module level? I would think the latter. The interpreter surely can evaluate such an expression and determine that convert_to_euros is going to be used a callable. Now, whether convert_to_euros is actually callable at run time is another story, but if it isn't, an exception is thrown, and at least you're not debugging somebody else's library to figure out why. Python would just tell you. I don't mean to oversimplify this, which is why I'm batting it around on a forum with a lot of smart people. But clearly it could be done. Whether it *should* be done is, of course, a question for debate, and I accept this. ____________________________________________________________________________________Be a better Globetrotter. Get better travel answers from someone who knows. Yahoo! Answers - Check it out. http://answers.yahoo.com/dir/?link=list&sid=396545469

Josiah wrote: http://mail.python.org/pipermail/python-ideas/2007-May/000835.html Josiah, you are correct about all the following, and I apologize for any incorrect paraphrase: 1) I am proposing to add a mini-language within Python. 2) I want relational data processing tasks to be easier within Python. 3) I want rowexpr to acquire names from the passed first argument (with sensible scoping rules, of course). 4) Point #4 essentially amounts to free enclosing scope goodies. 5) My proposal does indeed smell of magic. 6) I do want the code to be easier to read for me (with the full caveat that YMMV). 7) I want the small clarity win. 8) I think the only way to achieve this particular clarity win is to add syntax. You may be incorrect about the following, or I may just be understanding your points, but I'm not wrong about any of these, since I'm just characterizing my own feelings: 1) I don't want rowexpr to be pretty much an alias for lambda; I actually want it to be more powerful. 2) I'm not sweating performance here; I'm already using an interpreted language. I don't want Python to be faster for my day-to-day tasks; I want it to be more expressive. I even--and I'll dare say--want the language to be BIGGER. 3) I'm not ignorant of the resistance to macro expansions in Python, but I do think a year-2010 Python interpreter could compile SQL syntax directly into bytecode. ____________________________________________________________________________________Be a better Globetrotter. Get better travel answers from someone who knows. Yahoo! Answers - Check it out. http://answers.yahoo.com/dir/?link=list&sid=396545469

Steve Howell wrote:

Josiah wrote: You may be incorrect about the following, or I may just be understanding your points, but I'm not wrong about any of these, since I'm just characterizing my own feelings:

1) I don't want rowexpr to be pretty much an alias for lambda; I actually want it to be more powerful.

Technically speaking, lambda is sufficient for turing completeness, so the only thing to be gained is a reduction in what you write. As you offer... predicate = rowexpr: manipulation(column1) > value1 and \ column2 < value2 predicate = lambda row: manipulation(row['column1']) > value1 and \ row['column2'] < value) Is that reduction worthwhile? I personally don't think so, but my list comprehensions tend to have fairly minimal predicates. One thing to take into consideration is that Guido has previously shot down 'order by' syntax in list comprehensions and generator expressions because he didn't think that they were Pythonic (I seem to remember 'ugly' and 'worthless', but maybe that was my response to them).

2) I'm not sweating performance here; I'm already using an interpreted language. I don't want Python to be faster for my day-to-day tasks; I want it to be more expressive. I even--and I'll dare say--want the language to be BIGGER.

That's fine, but realize that Python 3.0 is moving towards a smaller, clearer language; see the dictionary changes, range/xrange, exception handling, etc.

3) I'm not ignorant of the resistance to macro expansions in Python, but I do think a year-2010 Python interpreter could compile SQL syntax directly into bytecode.

It could, but I would happily bet you $100 that it won't. If you are really intent on getting this, despite the arguments against it (from everyone so far), you could add your own syntax with logix, which will handle arbitrary SQL statement and compile it into Python bytecode (given sufficient information on how to do so). Yes, that's a cop-out, but sometimes the only way people get the language that they want is if they can add syntax at their whim. I've personally found that while I disagree with Guido on very many things related to Python, I'm usually too lazy to bother to add syntax I think I may need when I'm able to (with very minimal effort) write a helper function or two to do basically everything I need in lieu of syntax. - Josiah

On Tue, May 29, 2007, Steve Howell wrote:

For all the real-world warts of SQL, I think SQL is a very aesthetically pleasing way to express transformations of relational data structures, and Python contains relational data structures, and therefore I think Python can benefit from using SQL as just one way of expressing relational transformations (and I'm still a little bit TIMTOWTDI from my Perl days, I fully admit).

Aside from the standard featuritis objection, my objection stems almost entirely from the difficulty of defining appropriate data structures on which to operate. SQL works partly because data in SQL tables is already *by definition* in a relational format -- which won't be true in Python, causing all kinds of runtime errors that IMO are inappropriate for SQL. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ "as long as we like the same operating system, things are cool." --piranha

On Tue, May 29, 2007, Steve Howell wrote:

Do you understand at least my motivation, if not necessarily agreeing with the wisdom of my overall proposal?

Sort of. I still think that your proposal deserves to be shot down for the same reasons that including regexes as part of the language should be shot down. I'm slightly less opposed to Talin's DSL idea, though. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ "as long as we like the same operating system, things are cool." --piranha

--- Josiah Carlson wrote:

It could, but I would happily bet you $100 that it won't.

Maybe not by 2010, but I'll make a gentlemen's bet that native SQL is in Python by 2005.

If you are really intent on getting this, despite the arguments against it (from everyone so far), you could add your own syntax with logix, which will handle arbitrary SQL statement and compile it into Python bytecode (given sufficient information on how to do so).

Ok, the logix approach to introducing new syntax is good to know. I thought my pain would at least resonate with a few people, but it obviously has not so far.

Yes, that's a cop-out, but sometimes the only way people get the language that they want is if they can add syntax at their whim. I've personally found that while I disagree with Guido on very many things related to Python, I'm usually too lazy to bother to add syntax I think I may need when I'm able to (with very minimal effort) write a helper function or two to do basically everything I need in lieu of syntax.

As I said in another reply, I have working code, so I don't NEED the syntax, just want it, think it's a good idea, and I hope I've picked the correct forum in expressing a sort of wouldn't-it-be-great-if-Python-did-this suggestion. My progression in programming has always been to think that the current paradigm was brilliant (even in my C++ days!), and then only to discover there were even better paradigms. ____________________________________________________________________________________Yahoo! oneSearch: Finally, mobile search that gives answers, not web links. http://mobile.yahoo.com/mobileweb/onesearch?refer=1ONXIC

Steve Howell wrote:

I don't mean to oversimplify this, which is why I'm batting it around on a forum with a lot of smart people. But clearly it could be done. Whether it *should* be done is, of course, a question for debate, and I accept this.

My objection this idea is that it's going in the opposite direction to what I'd like to see. In my opinion, having to embed one programming language inside another leads to ugly code. I would rather have an elegant way to deal with relational databases by writing Python code *instead* of SQL, than yet another mechanism for embedding SQL in Python. -- Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | Carpe post meridiem! | Christchurch, New Zealand | (I'm not a morning person.) | greg.ewing@canterbury.ac.nz +--------------------------------------+

Steve Howell wrote:

You're wanting to solve a slightly different problem than I was, but can you elaborate on this a bit?

I know, it's more or less the complementary problem to what you're talking about. My point is that I don't like embedding SQL in my Python even when I'm dealing with a relational database, so I'm even less inclined to do so when dealing with Python data structures. I find that the existing features of the Python language do that well enough already.

Are you objecting to some ugliness of SQL itself, or do you want a more powerful abstraction?

Some of both. There's the general awkwardness involved whenever one language is embedded in another, plus the fact that I don't particularly like some aspects of SQL in particular. But often I also want a more powerful abstraction. The SQL that I need at a given point in the program isn't always fixed, and I need to generate it dynamically. As an example, in a recent project I needed to extract data from a number of tables to produce a report. The user has a variety of choices as to which fields are included in the report, and can optionally specify selection criteria for various fields, either a single value or a range of values. To accommodate this efficiently, I have to dynamically generate an SQL statement which includes or excludes 'where' clause terms of various sorts, and joins as necessary to pull in requested data from auxiliary tables. In this kind of application, there are few or no complete SQL statements written into the source, only fragments that get combined by an SQL-generation framework of some kind. So a facility like the one you suggest wouldn't help with this kind of problem. -- Greg

Blake Winton wrote:

Steve Howell wrote:

...

Let the Python interpreter build expression objects for you. You write: rowexpr: convert_to_euros(salary) > 50000 and deptno = 10 Python translates that into the same bytecode as: lambda row: convert_to_euros(row['salary'] > 50000 and row['deptno'] == 10

I'm sorry, why would that not be translated into:

lambda row: row['convert_to_euros'](row['salary'] > row['50000'] row['and'] row['deptno'] == row['10']

Sticking to your logic, why would it be > instead of row['>']? This is, I guess, particularly simple with Python's tokenizer because neither 50000 nor and can be valid identifiers.

? Specifically, how would python know what to dereference and what not to? What if there were two things, named the same, one in the row and one in the namespace? (i.e. a variable named 'salary')

That's a harder one, indeed. Perhaps, nothing should be dereferenced (at ``compile'' time, don't know if that's a valid term in CPython's interpreter chain tho), as it is in a lambda-expression.

How would you escape things which would have been dereferenced but you didn't want to be? (i.e. "rowexpr: convert_to_euros(salary) > salary")

ACK. If included (and I have major doubts it has a chance over 0.001%), it could just support "easy" expressions, such as lambdas do today.

I guess I'm kind of wasting my time here, since the introduction of a new keyword for this application really isn't going to happen, based on other decisions I've seen Guido make, but I do think that you need to think a little more about how the implementation of this feature would work. (Or perhaps you've done that thinking, and you just need to fill in the proposal with that information.)

Well, discussing in a mature manner can IMO never be a waste of time but I have to agree, I would not want this as a keyword either. A common idiom (aka stdlib module) *could* be handy, though. Regards, Stargaming

--- Stargaming wrote:

That's a harder one, indeed. Perhaps, nothing should be dereferenced (at ``compile'' time, don't know if that's a valid term in CPython's interpreter chain tho), as it is in a lambda-expression.

Just tried a few examples, almost everything interesting happens at runtime: ======== PYTHON 2.3 Valid Python: salary = 2 print (lambda: salary)() # prints 2 --- Valid, but useless, Python (no errors) lambda: salary --- Run-time error: x = lambda: salary x() NameError: global name 'salary' is not defined --- Run-time error: x = lambda: convert_to_euros(salary) x() NameError: global name 'convert_to_euros' is not defined --- Run-time error: def convert_to_euros(amt): return amt.impossible salary = None x = lambda: convert_to_euros(salary) x() AttributeError: 'NoneType' object has no attribute 'impossible' --- Run-time error: def convert_to_euros(amt): raise 'this does not get here' row = None x = lambda row: convert_to_euros(row['salary']) x(row) TypeError: 'NoneType' object is unsubscriptable ======= THEORETICAL Valid Python: row = {'salary': 50000} def convert_to_euros(amt): return amt / 2 x = rowexpr: convert_to_euros(salary) print x(row) # prints 25000 --- Run-time error: def convert_to_euros(amt): raise 'this does not get here' row = None x = rowexpr: convert_to_euros(salary) x(row) TypeError: 'NoneType' object is unsubscriptable --- ____________________________________________________________________________________ Park yourself in front of a world of choices in alternative vehicles. Visit the Yahoo! Auto Green Center. http://autos.yahoo.com/green_center/

--- Steven Bethard wrote:

...

...

...

def rowexpr(expr): ... def evaluate(row): ... d = dict(row) ... d.update(globals()) ... exec '__result__ = ' + expr in d ... return d['__result__'] ... return evaluate ... def convert_to_euros(amt): ... return amt / 2 ... x = rowexpr('convert_to_euros(salary)') row = dict(salary=50000) print x(row) 25000

Thanks. I tried it out with a slightly more involved expression. x = rowexpr( 'convert_to_euros(salary) ' 'if dept == "foo" else 0') rows = [dict(dept='foo', salary=i) for i in range(10000)] transform = [x(row) for row in rows] Here were my findings: 1) Not horribly slow. 3.4 seconds on my box for 10000 calls 2) I introduced a syntax error, and it was more clear than I thought it would be. It happens at runtime, of course, which is less than ideal, and it doesn't directly point me out to the line of code with the syntax error, but it does suggest the error: File "sql.py", line 14, in <module> transform = [x(row) for row in rows] File "sql.py", line 5, in evaluate exec '__result__ = ' + expr in d File "<string>", line 1 __result__ = convert_to_euros(salary) if dept = "foo" else 0 ^ SyntaxError: invalid syntax ____________________________________________________________________________________ Don't get soaked. Take a quick peak at the forecast with the Yahoo! Search weather shortcut. http://tools.search.yahoo.com/shortcuts/#loc_weather