File Upload

File upload source configuration with drag-and-drop area

Overview

The File Upload source lets you import data by uploading a file directly from your computer. The platform parses the file and extracts fields for mapping.

Supported File Formats

The platform automatically detects the file format. The following formats are supported:

Format	Extension(s)	Description
CSV	`.csv`, `.tsv`, `.txt`	Comma-separated values. Delimiter is auto-detected (comma, semicolon, tab, pipe).
JSON	`.json`	Standard JSON files with a root array or object.
NDJSON	`.ndjson`, `.jsonl`	Newline-delimited JSON (one JSON object per line).
XML	`.xml`	XML files — the root element path is auto-detected.
Parquet	`.parquet`	Apache Parquet columnar format.
Avro	`.avro`	Apache Avro serialization format.
XLSX	`.xlsx`	Microsoft Excel files.

Compressed files (.gz, .zip) are automatically decompressed before parsing.

PGP-encrypted files cannot be uploaded directly via the File Upload source because format detection runs at upload time, before a decryption key can be provided. To import PGP-encrypted files, use URL, FTP/SFTP, S3, or GCS instead. See the PGP Decryption guide for details.

Upload via the UI

The easiest way to create a file-upload datasource is through the platform interface.

Create a new datasource

Navigate to DataHub and click Create datasource.

Select File Upload as the source type

In the source configuration step, choose File Upload.

Upload your file

Drag and drop your file into the upload area, or click to open the file browser and select a file from your computer. The platform automatically detects the file format and text encoding.

Configure field mapping

Once the file is parsed, fields are extracted and displayed. Configure the field mapping to define how data should be imported.

Validate and publish

Review your configuration and click Validate to publish the datasource. Data ingestion starts immediately.

To update data later, edit the datasource and re-upload a new file using the same drag-and-drop area.

Upload via the API

You can also create and update file-upload datasources programmatically using the Reelevant API. This is a multi-step process:

Authenticate

Obtain an access token via the authentication endpoint:

curl -XPOST https://api.reelevant.com/v2/auth/token \
  -H "Content-Type: application/json" \
  -d '{
    "username": "your-email",
    "password": "your-password",
    "grant_type": "password",
    "client_id": "<client_id>"
  }'

Use the returned access_token in subsequent requests.

Create a datasource

Create a new datasource of type worker:

curl -XPOST https://api.reelevant.com/v2/datasources/ \
  -H "Authorization: Bearer ${access_token}" \
  -H "Content-Type: application/json" \
  -d '{ "mode": "worker" }'

The response contains the datasource id (a 24-character hex string) which is needed for subsequent steps.

Upload the file

Upload your data file using the upload endpoint. The file must be sent as multipart/form-data:

curl -XPOST https://api.reelevant.com/v2/datasources/worker/${datasource_id}/upload \
  -H "Authorization: Bearer ${access_token}" \
  -F "File=@/path/to/your/data.csv"

The response returns the file source options (including bucket, path, format, and textEncoding) that you need for the next step.

Configure sources (configure_sources step)

Submit the configure_sources step with the file options returned by the upload:

curl -XPOST https://api.reelevant.com/v2/datasources/${datasource_id}/steps \
  -H "Authorization: Bearer ${access_token}" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "configure_sources",
    "payload": {
      "pipeline": [
        {
          "input": {
            "source": {
              "type": "file",
              "options": {
                "bucket": "<bucket from upload response>",
                "path": "<path from upload response>",
                "format": <format object from upload response>,
                "textEncoding": "<textEncoding from upload response>"
              }
            }
          }
        }
      ]
    }
  }'

Configure fields (configure_fields step)

After configuring sources, configure the field mapping. Use the fields extracted from the sample to define which fields to import:

curl -XPOST https://api.reelevant.com/v2/datasources/${datasource_id}/steps \
  -H "Authorization: Bearer ${access_token}" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "configure_fields",
    "payload": [<fields_map array>]
  }'

Use GET https://api.reelevant.com/v2/datasources/${datasource_id}/steps to retrieve the current step state including the extracted fields map, which you can then modify and submit.

Validate and publish

Submit the validate step to publish the datasource and trigger data ingestion:

curl -XPOST https://api.reelevant.com/v2/datasources/${datasource_id}/steps \
  -H "Authorization: Bearer ${access_token}" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "validate",
    "payload": {}
  }'

To update data on an existing datasource, repeat from step 3 (upload a new file) and then call the promote endpoint to trigger re-ingestion:

curl -XPOST https://api.reelevant.com/v2/datasources/worker/${datasource_id}/promote \
  -H "Authorization: Bearer ${access_token}"

Update via FTP / SFTP

When a file-upload datasource is created, Reelevant automatically generates FTP and SFTP credentials. Your systems can push updated files to the hosted endpoint, and the datasource will automatically re-process the latest file.

To re-initialise a Datasource from an FTP/SFTP folder without removing older files, see Ignore Files Before a Date.

Retrieving Credentials

Get the upload password

Retrieve the SFTP/FTP password for your datasource via the API:

curl -XGET https://api.reelevant.com/v2/datasources/worker/${datasource_id}/upload-password \
  -H "Authorization: Bearer ${access_token}"

The response contains the password field.

Note the connection details

Use the following connection parameters:

Field	Value
Host	`ftp.reelevant.com`
FTP/FTPS Port	`21`
SFTP Port	`22`
Username	Your datasource ID (the 24-character hex string)
Password	The password returned in step 1

Port 22 is the standard SFTP port. Port 8022 is still accepted for existing integrations. Use port 22 for new setups: some systems, such as Salesforce Commerce Cloud, only allow outbound connections on ports 22, 80, 443 and 587.

SFTP with SSH Key-Pair Authentication

Instead of using a password, you can configure your datasource to authenticate via an SSH key pair. Once a public key is configured, password authentication is disabled for that datasource — only key-based auth will work.

Generate an SSH key pair

If you don’t already have one, generate a key pair:

ssh-keygen -t ed25519 -f sftp-key -N ""

This creates sftp-key (private key) and sftp-key.pub (public key).

Use the patch step to set the sftpPublicKey field on your datasource:

curl -XPOST https://api.reelevant.com/v2/datasources/${datasource_id}/steps \
  -H "Authorization: Bearer ${access_token}" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "patch",
    "payload": {
      "sftpPublicKey": "<contents of sftp-key.pub>"
    }
  }'

Connect using your private key

sftp -i sftp-key ${datasource_id}@ftp.reelevant.com <<EOF
put /path/to/your/data.csv data.csv
quit
EOF

Once a public key is registered, password authentication is disabled for that datasource. Remove the sftpPublicKey (set it to an empty string via the patch step) to re-enable password auth.

Pushing a File

Once you have credentials (password or SSH key), upload a file using any standard FTP or SFTP client:

SFTP (password)
SFTP (SSH key)
FTP

sftp ${datasource_id}@ftp.reelevant.com <<EOF
put /path/to/your/data.csv data.csv
quit
EOF

Or using an SFTP library/client, connect with:

Host: ftp.reelevant.com
Port: 22
Username: <datasource_id>
Password: <password from API>

sftp -i /path/to/private-key ${datasource_id}@ftp.reelevant.com <<EOF
put /path/to/your/data.csv data.csv
quit
EOF

Or using an SFTP library/client, connect with:

Host: ftp.reelevant.com
Port: 22
Username: <datasource_id>
Private key: <path to your private key file>

curl -T /path/to/your/data.csv \
  ftp://ftp.reelevant.com/ \
  --user "${datasource_id}:${password}"

Or using any FTP client with:

Host: ftp.reelevant.com
Port: 21
Username: <datasource_id>
Password: <password from API>

After the file is uploaded, the Datasource automatically re-processes the new file and updates the data.

Pushing Several Files in One Session

Over both SFTP and FTP, files pushed back-to-back within the same session (connection) are grouped into a single update rather than one update per file. A group is processed when the session is closed, or after about one minute without a new upload if the session stays open. Files pushed later in a still-open session form the next group.

SFTP
FTP

sftp ${datasource_id}@ftp.reelevant.com <<EOF
put /path/to/your/products.csv products.csv
put /path/to/your/stores.csv stores.csv
quit
EOF

curl -T "{/path/to/your/products.csv,/path/to/your/stores.csv}" \
  ftp://ftp.reelevant.com/ \
  --user "${datasource_id}:${password}"

With a graphical FTP client, keep a single connection open and upload the files one after the other. Clients that open several parallel connections to transfer files create one group per connection.

If the connection drops while a file is still being transferred, that file is discarded and never processed. Files fully transferred earlier in the same session are still processed as usual.

The FTP/SFTP endpoint only accepts file uploads (STOR / write operations). You cannot delete or rename files on the server.

Use SFTP when security is a concern — it encrypts the connection and all data transfer. FTP with TLS (FTPS) is also supported.

File upload creates an initial import. To keep data updated automatically on a schedule, consider using a URL, FTP/SFTP, or cloud storage source instead.

Getting Started

DataHub Deep Dive

Advanced Workflows

Email Rendering

Transformations

Advanced Analytics

Overview

Supported File Formats

Upload via the UI

Upload via the API

Update via FTP / SFTP

Retrieving Credentials

SFTP with SSH Key-Pair Authentication

Pushing a File

Pushing Several Files in One Session

​Overview

​Supported File Formats

​Upload via the UI

​Upload via the API

​Update via FTP / SFTP

​Retrieving Credentials

​SFTP with SSH Key-Pair Authentication

​Pushing a File

​Pushing Several Files in One Session

Overview

Supported File Formats

Upload via the UI

Upload via the API

Update via FTP / SFTP

Retrieving Credentials

SFTP with SSH Key-Pair Authentication

Pushing a File

Pushing Several Files in One Session