<div dir="ltr"><div><a href="https://en.wikipedia.org/wiki/BigQuery">https://en.wikipedia.org/wiki/BigQuery</a><br></div><div><br></div><div>BigQuery Dashboards</div><div><br></div><div>- <a href="http://bigqueri.es/c/github-archive">http://bigqueri.es/c/github-archive</a></div><div>- <a href="https://redash.io/data-sources/google-bigquery">https://redash.io/data-sources/google-bigquery</a></div><div> - <a href="https://github.com/getredash/redash">https://github.com/getredash/redash</a></div><div> - <a href="https://github.com/getredash/redash/blob/master/requirements.txt">https://github.com/getredash/redash/blob/master/requirements.txt</a></div><div> - <a href="https://github.com/getredash/redash/blob/master/Dockerfile">https://github.com/getredash/redash/blob/master/Dockerfile</a></div><div> - <a href="https://github.com/docker/docker/blob/master/builder/dockerfile/parser/parser.go">https://github.com/docker/docker/blob/master/builder/dockerfile/parser/parser.go</a></div><div> - <a href="https://github.com/DBuildService/dockerfile-parse/issues">https://github.com/DBuildService/dockerfile-parse/issues</a></div><div> - <a href="https://github.com/getredash/redash/blob/master/docker-compose.yml">https://github.com/getredash/redash/blob/master/docker-compose.yml</a></div><div><br></div><div>Software Configuration Management / Dependency Management applications for BigQuery:</div><div><br></div><div><br></div><div>- <a href="https://opensource.googleblog.com/2017/03/operation-rosehub.html">https://opensource.googleblog.com/2017/03/operation-rosehub.html</a><br></div><div> - "Googlers used BigQuery and GitHub to patch thousands of vulnerable projects"</div><div> <a href="https://www.reddit.com/r/bigquery/comments/5x0x5z/googlers_used_bigquery_and_github_to_patch/">https://www.reddit.com/r/bigquery/comments/5x0x5z/googlers_used_bigquery_and_github_to_patch/</a></div><div><br></div><div><br></div><div>BigQuery Python Libraries</div><div><br></div><div>google-cloud-bigquery</div><div><br></div><div>- | Src: <a href="https://github.com/GoogleCloudPlatform/google-cloud-python">https://github.com/GoogleCloudPlatform/google-cloud-python</a></div><div>- | Pypi: <a href="https://pypi.python.org/pypi/google-cloud-bigquery">https://pypi.python.org/pypi/google-cloud-bigquery</a></div><div>- | Docs: <a href="https://cloud.google.com/bigquery/docs/reference/libraries#client-libraries-resources-python">https://cloud.google.com/bigquery/docs/reference/libraries#client-libraries-resources-python</a></div><div><br></div><div>google-api-python-client</div><div><br></div><div>- | Src: <a href="https://github.com/google/google-api-python-client">https://github.com/google/google-api-python-client</a><br></div><div>- | Pypi: <a href="https://pypi.python.org/pypi/google-api-python-client">https://pypi.python.org/pypi/google-api-python-client</a></div><div>- pandas.io.gbq uses google-api-python-client:</div><div> - Docs: <a href="http://pandas.pydata.org/pandas-docs/stable/io.html#google-bigquery-experimental">http://pandas.pydata.org/pandas-docs/stable/io.html#google-bigquery-experimental</a><div> - read_gbq()</div><div> <a href="http://pandas.pydata.org/pandas-docs/stable/generated/pandas.io.gbq.read_gbq.html#pandas.io.gbq.read_gbq">http://pandas.pydata.org/pandas-docs/stable/generated/pandas.io.gbq.read_gbq.html#pandas.io.gbq.read_gbq</a></div><div> - to_gbq()</div><div> <a href="http://pandas.pydata.org/pandas-docs/stable/generated/pandas.io.gbq.to_gbq.html#pandas-io-gbq-to-gbq">http://pandas.pydata.org/pandas-docs/stable/generated/pandas.io.gbq.to_gbq.html#pandas-io-gbq-to-gbq</a></div><div><br></div><div><br></div><div>Open Source Big Data Components for things like BigQuery:</div><div><br></div><div>Apache Drill</div><div><br></div><div>- | Wikipedia: <a href="https://en.wikipedia.org/wiki/Apache_Drill">https://en.wikipedia.org/wiki/Apache_Drill</a></div><div>- Apache Drill is similar to Google Dremel (which powers Google BigQuery)</div><div>- <a href="https://pypi.python.org/pypi/drillpy">https://pypi.python.org/pypi/drillpy</a></div><div><br></div><div>Apache Beam</div><div><br></div><div>- | Wikipedia: <a href="https://en.wikipedia.org/wiki/Apache_Beam">https://en.wikipedia.org/wiki/Apache_Beam</a></div><div>- | Src: <a href="https://github.com/apache/beam">https://github.com/apache/beam</a></div><div>- | Docs: <a href="https://beam.apache.org/documentation/sdks/python/">https://beam.apache.org/documentation/sdks/python/</a><br></div><div>- | Docs: <a href="https://beam.apache.org/get-started/quickstart-py/">https://beam.apache.org/get-started/quickstart-py/</a></div><div>- | Docs: <a href="https://github.com/apache/beam/tree/master/sdks/python/apache_beam/examples">https://github.com/apache/beam/tree/master/sdks/python/apache_beam/examples</a></div><div>- Google Cloud Dataflow is now of Apache Beam</div><div>- <a href="https://cloud.google.com/dataflow/model/bigquery-io">https://cloud.google.com/dataflow/model/bigquery-io</a></div><div><br></div><div><br></div><div>Parsing (and MAINTAINING) Pip Requirements.txt Files:</div><div><br></div><div>- | Src: <a href="https://github.com/pypa/pip/tree/master/pip/req">https://github.com/pypa/pip/tree/master/pip/req</a></div><div> - <a href="https://github.com/pypa/pip/issues/3884#issuecomment-236454008">https://github.com/pypa/pip/issues/3884#issuecomment-236454008</a></div><div> - <a href="https://github.com/pypa/pip/issues/1479">https://github.com/pypa/pip/issues/1479</a></div><div> - -> Pipfile, Pipfile.lock (``pipenv install pkgname --dev``)</div><div><br></div><div>- <a href="https://github.com/pyupio/safety-db#tools">https://github.com/pyupio/safety-db#tools</a></div><div> - <a href="https://pyup.io/">https://pyup.io/</a></div><div>- <a href="https://libraries.io/github/librariesio/pydeps">https://libraries.io/github/librariesio/pydeps</a></div><div> - <a href="https://github.com/librariesio/pydeps">https://github.com/librariesio/pydeps</a></div><div> - <a href="https://libraries.io/">https://libraries.io/</a></div><div><br></div><div><div>- Pipfile, Pipfile.lock</div><div> - | PyPI: <a href="https://pypi.python.org/pypi/pipenv">https://pypi.python.org/pypi/pipenv</a></div><div> - | PyPI: <a href="https://pypi.python.org/pypi/requirements-parser">https://pypi.python.org/pypi/requirements-parser</a></div><div> - | PyPI: <a href="https://pypi.python.org/pypi/pipfile">https://pypi.python.org/pypi/pipfile</a></div><div> - | Src: <a href="https://github.com/kennethreitz/pipenv">https://github.com/kennethreitz/pipenv</a></div><div> - These save to the Pipfile:</div><div> - ``pipenv install pkgname``</div><div> - ``pipenv install pkgname --dev``</div><div> - <a href="https://github.com/kennethreitz/pipenv/blob/master/pipenv/utils.py">https://github.com/kennethreitz/pipenv/blob/master/pipenv/utils.py</a></div><div> - pip reqs.txt <--> Pipfile</div></div><div><br></div><div>... Thought I'd get these together; hopefully they're useful.</div></div><div><br></div><div>Cool Jupyter notebook!</div><div>( <a href="https://github.com/lkraider/requirements-dataset/blob/master/index.ipynb" target="_blank" style="font-size:12.8px">https://github.com/lkraider/<wbr>requirements-dataset/blob/<wbr>master/index.ipynb</a> )</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Mar 7, 2017 at 5:06 AM, Jannis Gebauer <span dir="ltr"><<a href="mailto:ja.geb@me.com" target="_blank">ja.geb@me.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word">Hi,<div><br></div><div>I ran a couple of queries against GitHubs public big query dataset [0] last week. I’m interested in requirement files in particular, so I ran a query extracting all available requirement files.</div><div><br></div><div>Since queries against this dataset are rather expensive ($7 on all repos), I thought I’d share the raw data here [1]. The data contains the repo name, the requirements file path and the contents of the file. Every line represents a JSON blob, read it with:</div><div><br></div><div><div>with open('data.json') as f:</div><div> for line in f.readlines():</div><div> data = json.loads(line)</div></div><div><br></div><div>Maybe that’s of interest to some of you.</div><div><br></div><div>If you have any ideas on what to do with the data, please let me know.</div><div><br></div><div>—</div><div><br></div><div>Jannis Gebauer</div><div><br></div><div><br></div><div><br></div><div>[0]: <a href="https://cloud.google.com/bigquery/public-data/github" target="_blank">https://cloud.google.com/<wbr>bigquery/public-data/github</a></div><div>[1]: <a href="https://github.com/jayfk/requirements-dataset" target="_blank">https://github.com/jayfk/<wbr>requirements-dataset</a></div></div><br>______________________________<wbr>_________________<br>
Distutils-SIG maillist - <a href="mailto:Distutils-SIG@python.org">Distutils-SIG@python.org</a><br>
<a href="https://mail.python.org/mailman/listinfo/distutils-sig" rel="noreferrer" target="_blank">https://mail.python.org/<wbr>mailman/listinfo/distutils-sig</a><br>
<br></blockquote></div><br></div>