[Python-ideas] Query Language extension to Python

[Wes Turner]

On Tuesday, November 1, 2016, Pavel Velikhov <pavel.velikhov at gmail.com>
wrote:

> Hi Wes!
>
>   Right now we don’t push anything yet, we fetch everything into the
> Python’s runtime.
> But going forward the current idea is to push as much computation to the
> database as
> possible (most of the time the database will do a better job then our
> engine).
>
> If we run on top PySpark/Hadoop I think we should be able to completely
> translate 100% of
> PythonQL into these jobs.
>

That would be great; and fast.

A few more links that may be of use (in addition to ops._ in alchemy.py):

-
https://github.com/pythonql/pythonql/blob/master/Grammar.md#query-expressions
- https://github.com/cloudera/ibis/blob/master/ibis/expr/window.py


- http://www.ibis-project.org/faq.html#ibis-and-spark-pyspark
- https://github.com/cloudera/ibis/tree/master/ibis/spark
-
http://spark.apache.org/docs/latest/sql-programming-guide.html#datasets-and-dataframes
-
https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/DataFrameStatFunctions.scala
-
http://spark.apache.org/docs/latest/sql-programming-guide.html#datasets-and-dataframes


>
> On 1 Nov 2016, at 19:42, Wes Turner <wes.turner at gmail.com
> <javascript:_e(%7B%7D,'cvml','wes.turner at gmail.com');>> wrote:
>
> Cool!
>
> https://github.com/pythonql/pythonql/wiki/PythonQL-Intro-and-Tutorial
>
> How do I determine how much computation is pushed to the data? (Instead of
> pulling all the data and running the computation with one local node) ...
> https://en.wikipedia.org/wiki/Bulk_synchronous_parallel (MapReduce,)
>
> - http://pandas.pydata.org/pandas-docs/stable/io.html#sql-queries
> - http://pandas.pydata.org/pandas-docs/stable/generated/
> pandas.read_sql_query.html
> - https://github.com/yhat/pandasql/
> - http://docs.ibis-project.org/sql.html#common-column-expressions
> - https://github.com/cloudera/ibis/blob/master/ibis/sql/alchemy.py
>
> On Tuesday, November 1, 2016, Pavel Velikhov <pavel.velikhov at gmail.com
> <javascript:_e(%7B%7D,'cvml','pavel.velikhov at gmail.com');>> wrote:
>
>> Hi Folks,
>>
>>   We have released PythonQL, a query language extension to Python (we
>> have extended Python’s comprehensions with a full-fledged query language,
>> drawing from the useful features of SQL, XQuery and JSONiq). Take a look
>> at the project here: http://www.pythonql.org and lets us know what you
>> think!
>>
>>   The way PythonQL currently works is you mark PythonQL files with a
>> special encoding and the system runs a preprocessor for all such files. We
>> have
>> an interactive interpreter and Jupyter support planned.
>>
>> Best regards!
>> PythonQL team
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20161101/2089ccf2/attachment-0001.html>