--- Arnaud Delobelle wrote:

On 26 May 2007, at 17:06, Steve Howell wrote:

...

I find myself often writing somewhat tedious code in Python that is just slicing/aggregating fairly simple data structures like a list of dictionaries.

Here is something that I'm trying to express, roughly:

charges = sum float(charge) from events on setup, install group by name

I don't really understand that :)

Let me try again: select name, bucket, sum(float(charge)) from events where bucket in ('install', 'setup') group by name, bucket The idea here is that events is a list of Python dictionaries keyed by fields "name", "bucket", and "charge", and I only care about install/setup charges. Here is a rough Python statement of what I'm trying to do: charges = {} for name, n_evts in groupby(events, itemgetter('name')): print name print dict([(bucket, sum([float(e['charge']) for e in b_evts])) for bucket, b_evts in groupby(n_evts, itemgetter('bucket')) if bucket in ('setup', 'install')])

...

Here is the Python code that I wrote:

def groupDictBy(lst, keyField): dct = {} for item in lst: keyValue = item[keyField] if keyValue not in dct: dct[keyValue] = [] dct[keyValue].append(item) return dct

isn't that itertools.groupby?

The problem with itertools.groupby is that it assumes sorted lists. So the code below does not behave how you might expect. It reports Joe's charges twice, and it loses Steve's $50 install charge. from itertools import groupby from operator import itemgetter events = [ {'name': 'Joe', 'bucket': 'install', 'charge': 100}, {'name': 'Joe', 'bucket': 'setup', 'charge': 20}, {'name': 'Steve', 'bucket': 'install', 'charge': 50}, {'name': 'Steve', 'bucket': 'setup', 'charge': 30}, {'name': 'Steve', 'bucket': 'install', 'charge': 1000}, {'name': 'Steve', 'bucket': 'upgrade', 'charge': 440}, {'name': 'Joe', 'bucket': 'setup', 'charge': 40}, ] charges = {} for name, n_evts in groupby(events, itemgetter('name')): print name print dict([(bucket, sum([float(e['charge']) for e in b_evts])) for bucket, b_evts in groupby(n_evts, itemgetter('bucket')) if bucket in ('setup', 'install')]) ____________________________________________________________________________________Pinpoint customers who are looking for what you sell. http://searchmarketing.yahoo.com/

--- Aahz wrote:

What's wrong with sqlite?

Nothing, but I want Python itself to give me the SQL syntax to manipulate lists-of-dictionaries like lists of dictionaries, without having to go through some a module. In the example that I gave earlier in the thread, what would the sqlite solution look like? To transform any particular list of dicitionaries into another list of dictionaries, would I have to do a create-table in sqlite, or does it already have some kind of mode where it can work directly on a Python dataset? ____________________________________________________________________________________ Be a PS3 game guru. Get your game face on with the latest PS3 news and previews at Yahoo! Games. http://videogames.yahoo.com/platform?platform=120121

--- Aahz wrote:

Think of it this way: if Python requires you to use a library for regex manipulation, what makes SQL-like data processing sufficiently special that it belongs in the language itself?

There is a difference. Regexes would never leverage python expressions *inside* the regex, even if they were more directly integrated into the language like Perl. (And, for the record, I prefer Python's philosophy to make regexes a module, but I do appreciate the short name, re, and the fact that it's battery included). In the case of SQL, I can imagine something like this: def convert_to_euros(salary): ... result_set = [[[ select deptno, total(convert_to_euros(salary)) from emp where country = 'France' group by deptno ]]] # where emp is a list of dictionaries Obviously, I understand that you can solve this problem in Python now, but you either have to do this: 1) Use more bulky way to describe the expression: sql(..., criteria=lambda row: row['country'] == 'France') 2) Rely on some library to parse the SQL for you, which is good enough for the year 2007, but I wonder why the Python interpreter itself couldn't interpret the SQL to bytecode (at startup time) just as easily as, say, SQLAlchemy or sqlite, does at runtime. ____________________________________________________________________________________Pinpoint customers who are looking for what you sell. http://searchmarketing.yahoo.com/

--- Stargaming wrote:

Are there any non-SQL use cases for SQL syntax?

Isn't that kind of a silly question? Of course not. SQL syntax only applies to relational data. Of course, such data doesn't have to exist in a "database" for the syntax to be useful. It just has to be relational. I would ask the opposite question--are there any cases where you manipulate basically relational data without the power of SQL? There are many answers there, and one them is Python. ____________________________________________________________________________________ Park yourself in front of a world of choices in alternative vehicles. Visit the Yahoo! Auto Green Center. http://autos.yahoo.com/green_center/

Steve Howell wrote:

Presumbably most people who use SQL databases already have to understand the syntax anyway. And those that don't can simply ignore it.

That simply will never be true. If there is new syntax added to Python, than the burden is on everyone to understand other people's code that uses such a feature. I'm very familiar with SQL and I am still not convinced that I want to see it in my Python. First of all, the concept of a relational table is ill-defined. I gather from your code that you believe this to be a list containing dictionaries, but why could this not be a tuple of tuples? For that matter, any iterable of iterables? Or how about using tuple'd indices, and so on.. It would be impossible to graft the SQL language onto all of the variations on the idea of a "table" in Python. Python does not have a relational table data structure, thus there cannot be a relational language used to address such a structure. It would not be wise to create a syntax that relies on a particular composition of data structures. -- Scott Dial scott@scottdial.com scodial@cs.indiana.edu