Specify cluster image versions. Cloud Dataproc uses image versions to bundle operating system and big data components (including core and optional components) and GCP connectors into a single package that is deployed on a cluster. If you don’t specify an image version when creating a new cluster, Cloud Dataproc will default to…

Control projection — Query only the columns that you need. Projection refers to the number of columns that are read by your query. Projecting excess columns incurs additional (wasted) I/O and materialization (writing results). Avoid SELECT *. Prune partitioned queries When querying a partitioned table, use the _PARTITIONTIME pseudo column to…

BigQuery operations that are free of charge in any location: Batch loading data Automatic re-clustering Exporting data Deleting table, views, partitions, functions and datasets Metadata operations Cached queries Queries that result in an error Storage for the first 10 GB of data per month Query data processed for the first…

This post looks at calling BigQuery from devices through REST API. Devices often have a small footprint and have constraints and may not be possible to install Client Libraries. By using JWT directly as a bearer token, rather than an OAuth 2.0 …

BigQuery provides an excellent CLI — “bq extract” and an API as well as Client Library in many languages for achieving the same. In exporting table data documentation it says — You can use a service such as Dataflow to read data from BigQuery instead of manually exporting it. This…