Overview
The FTP and SFTP sources let you retrieve data files from remote file servers. This is useful for scheduled feeds from partners, legacy systems, or any infrastructure that publishes data via FTP/SFTP.
FTP Configuration
Required Fields
| Field | Description |
|---|
host | The FTP server hostname or IP address. |
port | The FTP port (default: 21). |
username | FTP login username. |
password | FTP login password. |
path | The file path on the server (see Wildcard Paths below). |
Optional Fields
| Field | Description |
|---|
secure | Enable FTPS (FTP over TLS) for encrypted connections. |
processType | File processing mode when using wildcard paths: last_unprocess (default), all_unprocess, or all. |
disableProxy | Disable the SOCKS5 proxy for the FTP connection (optional). |
SFTP Configuration
Required Fields
| Field | Description |
|---|
host | The SFTP server hostname or IP address. |
port | The SFTP port (default: 22). |
username | SFTP login username. |
path | The file path on the server (see Wildcard Paths below). |
Authentication
SFTP supports two authentication methods:
Authenticate using a username and password.| Field | Description |
|---|
password | SFTP login password. |
Authenticate using an SSH private key.| Field | Description |
|---|
privateKey | The SSH private key in PEM format. |
passphrase (optional) | The passphrase for the private key, if it is encrypted. |
Use SFTP when security is a concern — it encrypts the connection and all data transfer. If your SFTP server supports key-based authentication, prefer it over passwords.
| Format | Description |
|---|
| CSV | Comma-separated values. Delimiter is auto-detected (comma, semicolon, tab, pipe). |
| JSON | Standard JSON files with a root array or object. |
| NDJSON | Newline-delimited JSON (one JSON object per line). |
| XML | XML files — the root element path is auto-detected. |
| Parquet | Apache Parquet columnar format. |
| Avro | Apache Avro serialization format. |
| XLSX | Microsoft Excel files. |
Compressed files (.gz, .zip) are automatically decompressed before parsing.
PGP Decryption
FTP and SFTP sources support fetching PGP-encrypted files. When pgpPrivateKey is configured, files are decrypted transparently before decompression and parsing.
| Field | Required | Description |
|---|
pgpPrivateKey | Yes | The PGP/GPG private key in armored (ASCII) format. |
pgpPassphrase | No | The passphrase for the private key, if encrypted. |
.asc) and binary (.pgp, .gpg) encrypted files are supported.
Wildcard Paths
You can use a wildcard (*) in the file name to match multiple files. This is useful when:
- A new file is exported periodically with a different name (e.g.
/exports/products_20240101.csv, /exports/products_20240102.csv)
- Data is split across multiple files in the same directory
Example patterns:
/exports/products_*.csv — matches all CSV files starting with products_ in the /exports/ directory/data/*.json — matches all JSON files in the /data/ directory
File Processing Modes
When using wildcard paths, you can configure how files are selected via the processType field:
processType value | Description |
|---|
last_unprocess (default) | Process only the most recent file that hasn’t been processed yet. Ideal for full-dataset exports where only the latest file matters. |
all_unprocess | Process all files that haven’t been processed yet. Useful for incremental/delta exports. |
all | Process all matching files on every sync, regardless of whether they were processed before. Useful for full datasets split across multiple files with the same names. |
Reelevant-Hosted SFTP
For datasources configured via the platform, Reelevant can also provide an SFTP endpoint where your systems can push files directly. This is useful when:
- Your infrastructure prefers to push data rather than have it pulled
- You want to update data files without accessing the Reelevant platform
The SFTP credentials (host, username, password) are generated automatically when the datasource is created.
How It Works
- Reelevant connects to the FTP/SFTP server using the provided credentials.
- The file at the configured path is located. If a wildcard is used, matching files are listed.
- Files are downloaded, decompressed if needed, and parsed based on the detected format.
- Fields are extracted and made available for mapping.
- On subsequent syncs, files are re-fetched according to the configured processing mode.
