Overview
The BigQuery source synchronizes data from a Google BigQuery table directly into Reelevant. This is ideal for teams that centralize their data in BigQuery and want to leverage it for personalization.
Configuration
Required Fields
| Field | Description |
|---|
projectId | The Google Cloud project ID that contains your BigQuery dataset. |
dataset | The BigQuery dataset name. |
table | The table name to synchronize. |
Authentication
BigQuery supports two authentication methods:
Authenticate using a Google Cloud service account key.| Field | Description |
|---|
clientEmail | The service account email address (e.g. my-sa@project.iam.gserviceaccount.com). |
privateKey | The service account private key (from the JSON key file). |
The minimum required role is BigQuery Data Viewer (roles/bigquery.dataViewer) on the target dataset. If you use a custom SQL query, the service account also needs BigQuery Job User (roles/bigquery.jobUser) on the project.
Authenticate using an OAuth connection configured in Reelevant. This is useful when you need to access BigQuery resources on behalf of a user rather than a service account.
Optional Fields
| Field | Description |
|---|
query | A custom SQL query to filter or transform data before import. When provided, the query is executed instead of reading the full table. |
tableProjectId | If the table belongs to a different GCP project than the service account, specify the table’s project ID here. |
customOAuthClientId | DatasourceAuth ID for OAuth-based authentication (alternative to service account). |
Custom SQL Queries
When you provide a custom SQL query, Reelevant executes it against BigQuery instead of reading the entire table. This is useful for:
- Filtering rows (e.g.
SELECT * FROM my_table WHERE active = true)
- Selecting specific columns
- Joining multiple tables
- Applying transformations
The query must be valid BigQuery Standard SQL. Make sure the service account has permission to run jobs in the project.
Incremental Sync
BigQuery supports incremental synchronization when used with the Keep Previous Data update mode. To enable this:
| Field | Description |
|---|
timestampField | The column used to track new or updated rows since the last sync. |
timestampFieldType | Whether the field is a datetime or timestamp type in BigQuery. |
If your datasource uses the Override Previous Data update mode (the default), the timestamp field is not needed — the entire table or query result is re-imported on each sync.
Supported Data Types
BigQuery data types are automatically converted:
| BigQuery Type | Behavior |
|---|
STRING, INT64, FLOAT64, BOOL | Passed through as-is. |
DATE, DATETIME, TIME, TIMESTAMP | Converted to their string representation. |
NUMERIC, BIGNUMERIC | Converted to JavaScript numbers. |
STRUCT (nested records) | Flattened into individual fields. |
ARRAY (repeated fields) | Preserved as arrays. |
How It Works
- Reelevant authenticates with BigQuery using the provided credentials.
- If a custom query is provided, it is executed. Otherwise, the full table is exported.
- For large tables, Reelevant uses an optimized export-to-GCS pipeline for better performance.
- Fields are extracted and made available for mapping.
- On subsequent syncs, the query or table is re-read (or incrementally fetched if configured).
Ensure the service account has read access to the specified dataset and table. The minimum required role is BigQuery Data Viewer. For custom queries, the service account also needs BigQuery Job User.
