Hi, Making download stats available through BigQuery seems like a good idea, but as it is currently this seems a bit expensive on a user's end. For example, I just looked at stats for a package of mine and 840GB are reported to be processed (therefore billed) in one query. Is table clustering used when building these tables ? If not, could we have it at least on projects name ? (See the following about table clustering: https://medium.com/google-cloud/bigquery-optimized-cluster-your-tables-65e2f... ). Best, Laurent
Hi Laurent, and thanks for asking. I re-clustered the tables - find my work-log and notes here: - https://medium.com/@hoffa/python-pypi-stats-in-bigquery-reclustered-d80e583e... If you use my tables, a query that used to process 200.88GB is now only scanning 9.65GB - when filtering for a particular package. 95% reductions! For example: SELECT TIMESTAMP_TRUNC(timestamp, WEEK) week , REGEXP_EXTRACT(details.python, r'^\d*\.\d*') python , COUNT(*) downloads FROM `the-psf.pypi.downloads2017*` WHERE file.project='pyspark' GROUP BY week, python HAVING python != '3.6' AND week<'2017-12-30' ORDER BY week
Bump. Is this the right list to post this kind of question (and implicit RFC) ? Le dim. 21 oct. 2018 à 10:55, Laurent Gautier <lgautier@gmail.com> a écrit :
Hi,
Making download stats available through BigQuery seems like a good idea, but as it is currently this seems a bit expensive on a user's end. For example, I just looked at stats for a package of mine and 840GB are reported to be processed (therefore billed) in one query.
Is table clustering used when building these tables ? If not, could we have it at least on projects name ? (See the following about table clustering: https://medium.com/google-cloud/bigquery-optimized-cluster-your-tables-65e2f... ).
Best,
Laurent
participants (2)
-
Felipe Hoffa -
Laurent Gautier