Moving Data Between BigQuery Projects
-
Our company recently split its analytics setup into two BigQuery projects. The original project still receives website and sales data, while the new one is used by the finance team for reporting. They need updated totals every morning, but they shouldn’t have access to customer details or the rest of the raw tables. What’s the easiest way to move only the approved data into the second BigQuery project and keep it current?
-
You don’t need to duplicate every source table just to build a handful of finance reports. Create separate reporting tables that contain only the approved fields and totals, then update them after the daily imports finish. Give the finance project access only to those tables. This keeps customer data in the original project and prevents reporting queries from touching everything else.
-
A few reporting tables can be refreshed separately, but that becomes harder to manage as finance requests more data. At that point, you need one pipeline that selects the approved fields, transfers only new and changed records, and updates all destination tables after the source import is complete. That’s the type of BigQuery ETL process covered here: http://datrise.com/en/pipeline/bigquery-to-bigquery . It shows how an incremental sync can keep the second project current without repeatedly moving the full dataset. You can use the same pipeline for new finance tables as reporting requirements expand.
