Skip to main content

​
Overview

The ignoreFilesBefore source option skips files modified before a specified date. It is useful when re-initialising a Datasource from a folder that also contains older files. For example, a new full export can arrive beside earlier exports or deltas without moving or deleting any files. A new Datasource version has no import history, so its first import after validation reads every file matching the path.

​
How it works

The option applies only to wildcard paths, where * appears in the file name. Every matching file dated before the cutoff is skipped during each read: the sample at configuration, the validation import, the first import of the new version, and every scheduled import. Files dated on or after the cutoff follow the usual processType rules. Use an ISO 8601 date-time. UTC with Z is recommended, for example 2026-10-08T06:00:00Z. An invalid date rejects the configure_sources step with status 400 and the message Invalid ignoreFilesBefore date: <value>. This validation also applies to non-wildcard paths, where a valid date has no effect.

​
Date used per source type

The date compared with the cutoff depends on the source type: File Upload is useful when files reach the Datasource through Reelevant-hosted FTP/SFTP. In this case, the compared date is the upload time; see Update via FTP / SFTP.

​
Set the date

There is no UI field for this option. Add ignoreFilesBefore to the existing source options in the draft’s configure_sources submission, keeping all other options unchanged. Use GET /v2/datasources/{DATASOURCE_ID}/steps to retrieve the current source options. Before making the request, set: This example configures an FTP-fed File Upload source. Replace the placeholders with the bucket, company ID, folder, format object, and text encoding from the current source options:
After configuring the source, submit the draft’s validate step. Review the import in the Datasource logs.

​
Choose the date

Set the cutoff just before the new full file is uploaded, for example a few minutes earlier. For File Upload, processType is not used. To read the full file and every later delta, the Datasource must use the Keep Previous Data update mode. With any other update mode, each import reads only the most recent file. For FTP, SFTP, Amazon S3, and Google Cloud Storage, use "processType": "all_unprocess" to read the full file and every later delta. With the default last_unprocess, each import reads only the most recent unprocessed file. See the FTP/SFTP, Amazon S3, and Google Cloud Storage pages for their processing modes. Keep ignoreFilesBefore in place after re-initialisation. It continues to exclude older files from later imports.