Overview
The URL source type fetches data from a web address. Use it for API endpoints, hosted CSV/JSON files, or any data feed accessible via HTTP/HTTPS.
Configuration
Required Fields
| Field | Description |
|---|
url | The full URL to the data source (API endpoint or hosted file URL). |
HTTP Options
These fields are nested under the options object.
| Field | Description |
|---|
method | The HTTP method to use: GET (default), POST, PUT, or PATCH. |
headers | Custom HTTP headers to include in the request (key-value pairs). Useful for API keys or custom tokens. |
query | URL query parameters to append to the request (key-value pairs). |
body | The request body (for POST, PUT, PATCH methods). |
timeout | Request timeout in milliseconds. |
oauth | DatasourceAuth ID for OAuth-protected APIs. |
Authentication
For publicly accessible URLs, no additional configuration is needed.
For OAuth-protected APIs, connect an OAuth integration configured in Reelevant. The platform handles token refresh automatically.
| Format | Description |
|---|
| CSV | Comma-separated values. Delimiter is auto-detected (comma, semicolon, tab, pipe). |
| JSON | Standard JSON files with a root array or object. |
| NDJSON | Newline-delimited JSON (one JSON object per line). |
| XML | XML files — the root element path is auto-detected. |
| Parquet | Apache Parquet columnar format. |
| Avro | Apache Avro serialization format. |
| XLSX | Microsoft Excel files. |
Compressed responses (.gz, .zip) are automatically decompressed before parsing.
PGP Decryption
The URL source supports fetching PGP-encrypted files. When pgpPrivateKey is configured, the response is decrypted before decompression and parsing.
| Field | Required | Description |
|---|
pgpPrivateKey | Yes | The PGP/GPG private key in armored (ASCII) format. |
pgpPassphrase | No | The passphrase for the private key, if encrypted. |
.asc) and binary (.pgp, .gpg) encrypted files are supported.
For APIs that paginate their responses, the URL source supports several pagination strategies:
Query Parameter
Body Parameter
Hypermedia Attribute
Page Token
Entry Token
Increment a query parameter (page number or offset) on each request.| Field | Description |
|---|
value | The query parameter name (e.g. page, offset). |
startAt | The initial value (default: 1). Some APIs start at 0. |
subtype | page (page number) or offset (record offset). |
Same as query parameter pagination, but the page/offset value is included in the request body.
Follow a URL found in the response body (e.g. a next attribute in the JSON response).| Field | Description |
|---|
value | The JSON path to the next page URL in the response (e.g. next_page_url). |
Use a token from the response body to request the next page.| Field | Description |
|---|
value | The JSON path to the token in the response body. |
param | The query parameter to use for the token in subsequent requests. |
removeParams | Remove other URL parameters when paginating (optional). |
Use a value from the last entry of the current page to request the next page.| Field | Description |
|---|
value | The field name in each entry that serves as a cursor. |
param | The query parameter to use for the cursor in subsequent requests. |
removeParams | Remove other URL parameters when paginating (optional). |
You can optionally set maximumPage to cap the number of pages fetched during a sync.
Runtime Variables
For API endpoints that require dynamic parameters (e.g. current date, user-specific IDs), you can define variables that are resolved at fetch time.
Each variable has:
| Field | Description |
|---|
name | The variable name used in the URL, headers, body, or query parameters. |
default | A static value or a dynamic expression evaluated at runtime. |
primitive | The data type (string, number, etc.). |
https://api.example.com/products?since={{last_sync_date}}.
How It Works
- Reelevant sends an HTTP request to the configured URL with the specified method, headers, and body.
- If pagination is configured, subsequent pages are fetched automatically.
- The response is parsed based on the detected format.
- Fields are extracted and made available for mapping.
- On subsequent syncs, the URL is re-fetched and the data is updated.
Make sure your URL returns data in a supported format. The platform auto-detects the format from the response content type and content.
