Custom Fetchers

Overview

A custom fetcher is an optional processing layer applied on top of a datasource’s standard retrieval. Instead of returning the stored rows as-is, the datasource runs the data through additional logic every time it is queried — filtering it to a single user, reshaping columns into rows, excluding products, or computing an aggregation. Custom fetchers run at query time, so they always reflect the latest stored data and the runtime context (such as the user being personalised for). This page documents the generic custom fetchers — the ones available to any account. Some custom fetchers implement customer-specific logic and are not covered here.

Custom fetchers do not have a self-serve interface yet. They are configured on a Datasource by Reelevant. Contact your Technical Account Manager if you want to enable one.

Retrieval modes

A datasource retrieves data in one of three modes, and each custom fetcher is built for one or more of them:

Mode	How data is retrieved	Typical sources
Pull-and-store	Reelevant fetches the source on a schedule, parses it, and stores it for querying.	URL, File, FTP/SFTP, BigQuery, Snowflake, GCS, S3
Event ingestion	Records arrive continuously as events and are appended to storage.	Kafka, Website Events
Real-time proxy	Reelevant calls the upstream source live on every request and does not store the result.	URL configured as a real-time API

The summary table below shows which mode each generic custom fetcher applies to.

Custom fetcher	Mode	Purpose
Per User	All modes	Restrict results to a single user’s records.
Unwind Rows	Pull-and-store	Pivot a wide row of columns into multiple long rows.
Product Exclusion	Pull-and-store	Include or exclude products listed in another datasource.
Expired Product Lifetime	Pull-and-store	Surface products a user is due to repurchase.
Proxy Lock	Real-time proxy	Deduplicate concurrent identical upstream calls.
Proxy Aggregation	Real-time proxy	Compute an aggregation over a real-time response.

Setting up a custom fetcher

There is no self-serve interface for custom fetchers yet, so they are configured through the Datasources API on the datasource’s draft version. Setup is two steps: declare the fetcher with the configure_fetcher step, then promote the version with the validate step. Both calls require an access token — see Authentication for how to obtain one.

# 1. Configure the custom fetcher on the datasource's draft version
curl -XPOST https://api.reelevant.com/v2/datasources/<datasource_id>/steps \
  -H "Authorization: Bearer <access_token>" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "configure_fetcher",
    "payload": {
      "name": "product-exclusion",
      "params": {
        "exclusionDatasourceId": "665f1c0b2a9d4e0008b3a1f2",
        "exclusionIdField": "sku",
        "productIdField": "reference",
        "mode": "exclusion"
      }
    }
  }'

# 2. Validate to promote the draft version to live
curl -XPOST https://api.reelevant.com/v2/datasources/<datasource_id>/steps \
  -H "Authorization: Bearer <access_token>" \
  -H "Content-Type: application/json" \
  -d '{ "name": "validate", "payload": {} }'

The payload.name is the fetcher identifier (e.g. per-user, generic-unwind-rows, product-exclusion), and payload.params holds the configuration fields documented for each fetcher below. For fetchers that take no configuration, send an empty object: "params": {}.

The validate step promotes the draft version to live. For pull-and-store datasources it also enqueues a verification job, so the new version only goes live once that job succeeds.

Per User

Restricts the datasource so a query returns only the records that belong to the user currently being personalised. It works on any retrieval mode and automatically targets the user identifier field defined in the Field Mapping (the CRM user field, or the purchase user field for a purchases datasource). When a datasource has a timestamp field, results are automatically sorted from most recent to oldest. The fetcher also exposes the resolved locale field so a Workflow can read the user’s locale alongside the data. This custom fetcher takes no configuration.

Use this on shared datasources (such as a CRM, purchases, or analytics feed) where a Workflow must only ever see the active user’s own rows.

Unwind Rows

Reshapes each stored row by turning its columns into several rows. One column is kept as an identifier, and every other column becomes its own row, with the original column name stored in one field and the cell value in another.

Configuration

Field	Required	Description
source	Yes	The column to keep as the identifier on every produced row.
destination	Yes	The name of the field that will hold the identifier value.
columnDestination	Yes	The name of the field that will hold the original column name.
valueType	Yes	The data type of the value field — `string` or `number`. When `number`, the value field is sortable.

Before and after

Consider a stored row that lists three category affinities for a user:

user	category1	category2	category3
user-42	shoes	jackets	hats

With source = user, destination = user, and columnDestination = category, the fetcher unwinds it into:

user	category	value
user-42	category1	shoes
user-42	category2	jackets
user-42	category3	hats

Sorting by the value field is applied after unwinding, so you can rank the produced rows by their numeric value when valueType is number.

Product Exclusion

Filters a product datasource against a second datasource that lists items to keep or remove. A typical use is hiding products a user has already purchased, or restricting recommendations to an allow-list.

Configuration

Field	Required	Description
exclusionDatasourceId	Yes	The datasource that holds the list of reference items.
exclusionIdField	Yes	The field in the exclusion datasource that contains the item IDs.
productIdField	Yes	The field in the product datasource to match those IDs against.
mode	Yes	`exclusion` removes matching products; `inclusion` keeps only matching products.

How it works

The fetcher reads the matching IDs from the exclusion datasource, respecting any query filters that apply to both datasources.
In exclusion mode, those IDs are removed from the product results; in inclusion mode, only those IDs are kept.
In inclusion mode, if the exclusion datasource returns no IDs, the product query returns no rows.

The exclusion list is read up to a configured limit. For very large lists, talk to your Technical Account Manager about the applicable threshold.

Expired Product Lifetime

Surfaces products that a user is due to repurchase, based on their purchase history and an expected product lifetime. It joins the current product datasource with a purchases datasource, computes when each previously bought product should run out, and returns only the products whose next purchase date falls inside a configured window.

Configuration

Field	Required	Description
purchaseDatasourceId	Yes	The datasource containing the user’s purchase events.
purchaseQuery	Yes	A base query applied to the purchases datasource.
purchaseReferenceIdField	Yes	The purchases field holding the product reference ID.
purchaseTimestampField	Yes	The purchases field holding the purchase date.
purchaseClientField	Yes	The purchases field holding the client (user) identifier.
temporalities	No	A list of rules describing the repurchase windows (see below).

Each entry in temporalities describes one repurchase window:

Field	Required	Description
maxLifetime	Yes	The maximum product lifetime, in days, this rule applies to.
windowStart	Yes	How many days before the expected next purchase the product becomes eligible.
windowEnd	Yes	How many days after the expected next purchase the product stays eligible.
minPurchaseCount	No	The minimum number of past purchases required for this rule to apply.

How it works

The fetcher looks at each product the user previously bought and predicts when they will need it again. The way the lifetime is derived depends on how many times the user bought that product.

It reads the user’s purchases (most recent first) and groups the purchase dates per product reference.
For each product it derives an expected lifetime and an expected next purchase date (see the two behaviours below).
It keeps the product only if today falls inside the matching temporality window — from windowStart days before the next purchase date to windowEnd days after it.
Kept products are enriched with purchaseCount, lastPurchaseDate, and nextPurchaseDate, and can be sorted by any of these.

Behaviour 1 — single purchase (lifetime lookup)

When the product was bought once, the fetcher cannot measure a real repurchase interval, so it reads the expected lifetime from the product’s own lifetime field in the Field Mapping. The next purchase date is lastPurchaseDate + lifetime. Example — a coffee capsule pack with a lifetime field of 30 days, bought once on 1 March, using the default window (windowStart 0, windowEnd 90):

Signal	Value
Purchases found	1 March
`purchaseCount`	1
Product lifetime field	30 days
`lastPurchaseDate`	1 March
`nextPurchaseDate`	31 March (1 March + 30 days)
Eligibility window	31 March → 29 June (next purchase date → +90 days)

The capsule pack is returned for any user query run between 31 March and 29 June. If the product has no lifetime field, a single-purchase product cannot be evaluated and is skipped.

Behaviour 2 — multiple purchases (computed average)

When the product was bought several times, the fetcher ignores the static lifetime field and instead computes the average interval between consecutive purchases for this specific user. The next purchase date is lastPurchaseDate + averageInterval. Example — the same capsule pack bought on 1 January, 1 February, and 1 March, using the default window:

Signal	Value
Purchases found	1 Jan, 1 Feb, 1 Mar
`purchaseCount`	3
Intervals	31 days (Jan→Feb), 28 days (Feb→Mar)
Average interval	≈ 29.5 days
`lastPurchaseDate`	1 March
`nextPurchaseDate`	≈ 30 March (1 March + 29.5 days)
Eligibility window	30 March → 28 June (next purchase date → +90 days)

Using the user’s own cadence makes the prediction far more accurate than a single static lifetime. If the computed average is less than one day, the product is skipped.

When no temporalities are configured, a default rule is applied: products with a lifetime up to 365 days that are due within the next 90 days (maxLifetime 365, windowStart 0, windowEnd 90).

Proxy Lock

Applies only to real-time proxy datasources. When Reelevant receives many concurrent requests that would trigger the same upstream call, this fetcher lets the first call run and makes the others wait briefly, so the result can be served from cache instead of repeating the call.

Configuration

Field	Required	Description
ttl	Yes	How long, in milliseconds, concurrent identical requests wait for the first call to complete.

Use this to protect a rate-limited or expensive upstream API when a single delivery fans out into many identical real-time calls.

Proxy Aggregation

Applies only to real-time proxy datasources. After fetching the live response, it computes a single aggregation over a chosen field and adds the result to every returned row.

Configuration

Field	Required	Description
targetField	Yes	The field to aggregate. Supports nested fields using dot notation (e.g. `price.amount`).
aggregationType	Yes	The aggregation to compute — `sum`, `avg`, `min`, or `max`.
outputField	Yes	The field name under which the computed result is added to each row.

Non-numeric and missing values are ignored. If no numeric values are found, the output field is empty.

Datasource Reference — building derived datasources from existing ones.
Special Configurations — other computed datasource types (Best Products, Merge, Cross-sell).
Field Mapping — defining the fields a custom fetcher reads and writes.

Getting Started

DataHub Deep Dive

Advanced Workflows

Email Rendering

Transformations

Advanced Analytics

Overview

Retrieval modes

Setting up a custom fetcher

Per User

Unwind Rows

Configuration

Before and after

Product Exclusion

Configuration

How it works

Expired Product Lifetime

Configuration

How it works

Behaviour 1 — single purchase (lifetime lookup)

Behaviour 2 — multiple purchases (computed average)

Proxy Lock

Configuration

Proxy Aggregation

Configuration

​Overview

​Retrieval modes

​Setting up a custom fetcher

​Per User

​Unwind Rows

​Configuration

​Before and after

​Product Exclusion

​Configuration

​How it works

​Expired Product Lifetime

​Configuration

​How it works

​Behaviour 1 — single purchase (lifetime lookup)

​Behaviour 2 — multiple purchases (computed average)

​Proxy Lock

​Configuration

​Proxy Aggregation

​Configuration

​Related pages

Overview

Retrieval modes

Setting up a custom fetcher

Per User

Unwind Rows

Configuration

Before and after

Product Exclusion

Configuration

How it works

Expired Product Lifetime

Configuration

How it works

Behaviour 1 — single purchase (lifetime lookup)

Behaviour 2 — multiple purchases (computed average)

Proxy Lock

Configuration

Proxy Aggregation

Configuration

Related pages