Data Ingestion

Ingestion Modes

Reelevant supports multiple ingestion patterns depending on data volume, freshness requirements, and source system capabilities.

Mode	Description	Freshness	Use case
Worker (batch pull)	Scheduled job fetches and processes data from an external source	Minutes to hours	Full catalogue sync, daily CRM updates
Ingester (real-time push)	REST API receives events pushed by your systems	Seconds	Purchase events, stock changes, behavioural data
PubSub (streaming)	Continuous consumption from a message broker	Sub-second	High-volume event streams (e.g. Kafka topics)
Proxy (live query)	Data fetched on demand at Workflow execution time	Real-time	External APIs queried per-request

DataHub Source Types

The DataHub supports the following source connectors for batch (Worker mode) ingestion:

Source	Protocol	Formats supported	Typical use
URL	HTTP/HTTPS	CSV, JSON, XML, Avro	Public or authenticated API endpoints, hosted catalogue files
FTP / SFTP	FTP, SFTP	CSV, JSON, XML, Avro	Legacy systems, partner data feeds
Google Cloud Storage (GCS)	GCS API	CSV, JSON, XML, Avro	Data warehouse exports, scheduled dumps
Amazon S3	S3 API	CSV, JSON, XML, Avro	Data lake exports, cross-cloud feeds
BigQuery	BigQuery API	Structured query results	Analytics exports, CRM segments, computed tables
Snowflake	Snowflake API	Structured query results	Data warehouse tables and views
Kafka	Kafka protocol (with LZ4/Snappy compression)	JSON, Avro	Streaming event ingestion (purchases, navigation, stock updates)
Datasource Reference	Internal	Derived from existing Datasource	Computed fields, cross-Datasource joins
File Upload	Direct upload	CSV, JSON, XML, Avro	One-off imports, manual data loads

All batch sources support field mapping, type coercion, PGP decryption for encrypted files, and configurable CSV options (custom delimiters, header-less files with column definitions).

Datasource Categories

Datasources are typed to enable specialised behaviours (required fields, indexing strategies, query capabilities):

Category	Purpose	Required fields	Subtypes
Product	Product catalogues, offers, recommendations	Reference ID	User-defined, Localised, Best Product, Bought Together
CRM	Customer profiles, segments, attributes	Workflow User (identity key)	User-defined
Location	Stores, points of interest, geographic data	Latitude, Longitude, Reference ID	User-defined, Google My Business, Partoo, Weather
Analytics	Behavioural events, tracking data	Datetime	Reelevant events, Google Tag Manager, Workflow Events, Purchases
Generic	Flexible schema for any structured data	None	User-defined, Top Attributes

Data Update Strategies

When new data arrives, three strategies control how it merges with existing records:

Strategy	Behaviour	Use case
Override	Drop all existing records, replace with new dataset	Full catalogue refresh
Append	Keep existing records, add new rows	Time-series events, purchase history
Upsert	Merge by primary key — update existing, insert new	CRM profiles, stock levels, price changes

Record-Level Anonymisation

Upsert Datasources in Worker and Ingester modes support targeted anonymisation by primary key. Privacy systems can remove selected CRM records without replacing the complete dataset. Each operation accepts up to 10,000 identifiers. After processing, the selected records stop appearing in Datasource queries while unaffected records remain available. The source system and separately configured analytics destinations retain their own copies. Apply the corresponding deletion policy in each system to complete a right-to-erasure request. See Anonymising Customer Records for supported configurations, processing behaviour, and an operational checklist.

Integration Effort

Scenario	Your effort	Reelevant support
Connect a CSV URL feed	Configure source URL + field mapping in the UI	Auto-scheduled sync
BigQuery/Snowflake integration	Provide query + credentials in the UI	Managed scheduling
Kafka streaming	Provide broker details + topic + schema	Continuous ingestion
SFTP with PGP encryption	Upload PGP key, configure path pattern	Auto-decryption on fetch

All source configurations are managed through the platform UI — no code required. For programmatic management, see the API Reference.

Overview

For Technical Evaluators

Ingestion Modes

DataHub Source Types

Datasource Categories

Data Update Strategies

Record-Level Anonymisation

Integration Effort

Next Steps

Integration Channels

Field Mapping

​Ingestion Modes

​DataHub Source Types

​Datasource Categories

​Data Update Strategies

​Record-Level Anonymisation

​Integration Effort

​Next Steps

Integration Channels

Field Mapping

Ingestion Modes

DataHub Source Types

Datasource Categories

Data Update Strategies

Record-Level Anonymisation

Integration Effort

Next Steps