tweakstreet-logo
  • Enter search terms

Get Files

looks for files and places them in result rows one by one

Processing

For each input row, the step recursively searches for files and folders inside a given base path. Results are generated for each individual file.

Settings

Name Type Description
Filesystem
 dict

The filesystem to use.

If the filesystem is nil, the local filesystem is used.

Evaluated for each input row
Base Path
 string

The base path to start looking for files.

Relative paths are interpreted as relative to the flow file.

Evaluated for each input row
Match Files
 boolean

If true, files are matched, if false files are not included in the result.

Evaluated for each input row
Match Folders
 boolean

If true, folders are matched, if false folders are not included in the result.

Evaluated for each input row
Pattern
 string

The pattern to match. Only matching files are included in the result.

The full path of each file system entry is tested.

A pattern can take the following forms:

Type Interpretation
string Regular expression. Matching files are included in the result.
When using the local filesystem, PathMatcher syntax can be used.
  • .*\.xml$ matches file names ending in .xml
  • glob:**/*.csv matches file names ending in csv (using path matcher glob syntax)
nil All found files are included in the result.

Evaluated for each input row
Filter
 function

Function of the form (string path, dict attributes) -> boolean returning whether to include the given path in the result.

The function is invoked with each found file after a successful pattern match, and is intended to filter out files based on additional criteria like file size etc.

If nil, all matching files are included in the result.

Evaluated for each input row
Max Depth
 long

The maximum depth when descending into folders.

If nil, no limit is applied. If the limit is 0, only the base path is considered.

Evaluated for each input row
On Error
 string

Determines the course of action when an I/O error happens during directory traversal:

  • error: throw an error
  • warn: write a warning message to the log and continue
  • ignore: ignore any errors and continue

Evaluated for each input row

Results

Name Type Description
path
 string A string containing the absolute path of the current file.
name
 string A string containing the name the current file.
size
 long The size of the current file in bytes.
is_file
 boolean True if the current file is a regular file, false otherwise.
is_dir
 boolean True if the current file is a directory, false otherwise.
mtime
 datetime The last modified time of the current file.
info
 dict A dict containing attributes the current file.

The following attributes are always provided:

Key Type Value
name string file name
path string absolute path
parent string absolute path to parent directory, can be nil for fs root
size long size of file in bytes, can be nil if path is not a file
is_file boolean true if path points to a regular file, false otherwise
is_dir boolean true if path points to a directory, false otherwise
is_symlink boolean true if path is a symbolic link
is_other boolean true if path is not a file or directory (could be a pipe, socket, etc.)
mtime datetime time of last modification, can be nil if file system does not track mtime
ctime datetime time of creation, can be nil if file system does not track ctime

Additional attributes might be present if provided by the underlying filesystem.
On this page