Define File Path Range

You can limit an import so that it only uses certain files according to a specified date pattern.

Choose one of the following options. Each option is described in detail in the sections that follow.

Defining a Pattern as a Placeholder for the Current Date in File Paths

Under the Basic section you must define a path for the import. Instead of using a static string, a file path can contain patterns which are later replaced by values corresponding to the current date.

Allowed patterns are: %year%, %month%, %day%, %hour%, or %min%

The path containing the current date is displayed below the path containing variables.

Example:

/myfolder/%year%/%month%/*.log

This example imports all .log files from /myfolder/2016/08/ (and changes according to the current date).

You can also use a wild card to indicate that all files of one type within a specific folder should be used, as in the following example: /Users/testUser/data/*/.xml

Defining a Time Range to Apply to File Names Using Either Fixed Dates or Expressions

You can define a time range by using specific dates. In addition, you can define ranges by clicking on the calendar icon.

Using fixed dates

The start and end dates are date + time strings chosen with a drop-down calendar.

Example file pattern:

 /myfolder/%year%/%month%/*.log

Start Date: 2011-05-01 00:00:00
End Date: 2011-07-31 00:00:00
This pattern imports files with names that match:

/myfolder/2011/05/*.log
/myfolder/2011/06/*.log
/myfolder/2011/07/*.log

Using expressions

Instead of entering a fixed date string, you can also define a dynamic range using formulas which return a date. Formulas can use constants in their calculations, such as d (=days), h (=hours), m (=min). This provides more flexibility when using dates that slide with respect to the current date.

Example file pattern:

/myfolder/%month%/%day%.log

Start Date: NOW()-2d
End Date: NOW()

This example imports the following files today (04 Aug 2016):

 /myfolder/08/02.log, /myfolder/08/03.log and /myfolder/08/04.log

And tomorrow (05 Aug 2016), the example imports:

/myfolder/08/03.log, /myfolder/08/04.log, /myfolder/08/05.log

Defining a Time Range Using the Date the File was Last Modified

You can exclude files newer or older than x where x is a number of days relative to today.
This filter is based on the file modification date, usually the time the file was last modified (or created).

Datameer v7