Skip to main content

​
Overview

The S3 source retrieves data files from an Amazon S3 bucket or any S3-compatible storage service (e.g. MinIO, DigitalOcean Spaces, OVH Object Storage). Use it when your data is stored as files in S3 and you want to import them into Reelevant for personalization.

​
Configuration

​
Required Fields

​
Optional Fields

For standard AWS S3, the accessKey must belong to an IAM user or role with at least s3:GetObject and s3:ListBucket permissions on the target bucket. With assumeRole, the role needs them instead, and the keys can be omitted.

​
Cross-account access with AssumeRole

When the bucket belongs to your own AWS account, you may prefer not to hand Reelevant a key that reads it directly. Instead, create an IAM role in your account that Reelevant is allowed to assume. Reelevant obtains short-lived credentials for that role, then uses them to list and download the files. Set the assumeRole field with the following options: bucket, path, region and the other options keep the same meaning. Leave accessKey and secretKey empty and only set assumeRole.roleArn. Reelevant assumes its own reelevant-customers-access role first, then assumes your role from it. Nothing has to be created on Reelevant’s side for a new customer. On your AWS account, configure the role as follows:
  1. Trust policy: allow the sts:AssumeRole action for the principal and condition below.
  2. Permissions policy: grant s3:ListBucket on the bucket and s3:GetObject on the objects under the configured path. If the bucket uses SSE-KMS encryption, also grant kms:Decrypt on the key.
  3. Maximum session duration: keep it greater than or equal to durationSeconds.
The external ID is your Reelevant company ID, which you can retrieve from your Technical Account Manager. Reelevant always sends it, so another Reelevant company cannot assume your role even if it knows its ARN.
Reelevant assumes your role from another role (role chaining), so AWS limits each session to one hour. Reelevant renews the credentials automatically before they expire, so long imports are not affected.

​
With your own credentials

Set accessKey and secretKey to the key of an IAM user allowed to assume the role, then configure the role’s trust policy for that user. If you set an externalId, add a condition requiring sts:ExternalId to equal that value.
The Datasource configuration is rejected if only one of accessKey and secretKey is set, or if roleArn is not a valid IAM role ARN. It is also rejected if durationSeconds is out of range, or if externalId is empty or set with Reelevant’s credentials. accessKey, secretKey and externalId are stored as sensitive fields and never displayed once saved. Temporary credentials are kept in memory only.

​
S3-Compatible Services

The S3 source works with any service that implements the S3 API. Set endpoint to your provider’s URL:

​
Supported File Formats

S3 automatically detects the file format. The following formats are supported:
Compressed files (.gz, .zip) are automatically decompressed before parsing.

​
PGP Decryption

The S3 source supports PGP-encrypted files. When pgpPrivateKey is configured, files are decrypted transparently before decompression and parsing. Both armored (.asc) and binary (.pgp, .gpg) encrypted files are supported.
See the PGP Decryption guide for details on key generation, supported formats, and error handling.

​
Wildcard Paths

You can use a wildcard (*) in the file path to match multiple files. This is useful when:
  • A new file is exported periodically with a different name (e.g. exports/products_20240101.csv, exports/products_20240102.csv)
  • Data is split across multiple files in the same directory
Example patterns:
  • exports/products_*.csv — matches all CSV files starting with products_ in the exports/ prefix
  • data/*.json — matches all JSON files under the data/ prefix

​
File Processing Modes

When using wildcard paths, you can configure how files are selected via the processType field: To exclude matching files modified before a date, set ignoreFilesBefore; see Ignore Files Before a Date.

​
How It Works

  1. Reelevant authenticates with S3 using the provided access key and secret key. With assumeRole, it first obtains temporary credentials for the configured role, using Reelevant’s credentials when no key is set.
  2. The specified bucket and path are accessed. If a wildcard is used, matching objects are listed.
  3. Files are downloaded, decompressed if needed, and parsed based on the detected format.
  4. Fields are extracted and made available for mapping.
  5. On subsequent syncs, files are re-fetched according to the configured processing mode.
Ensure the IAM user or role has s3:GetObject and s3:ListBucket permissions on the target bucket. For S3-compatible services, ensure the equivalent read permissions are granted.