Steps overview

Steps are the processing units of every flow type. Steps accept input rows, perform processing, provide processing results, and generate output rows. The exact input, processing, and output semantics are specific for each step type.

Step settings

Each step performs its processing based on configuration settings. All settings are expression values. They can either be given litreally, or be the result of an expression, much like a cell value in a spreadsheet application.

Specifying settings as expressions allows for a meta-data-driven approach to processing. It is possible to dynamically construct the configuration of each setting based on parameters, meta-data, and the data currently processed.

Step results

Steps often perform activities that produce values as a result. For example, the Clock step looks up the current time, and the Http Request step performs a network request and receives a response.

Steps make such outcomes available as step results. Step results are available after the step has performed a unit of work. They can be placed in output fields.

Output fields

Output fields are named expression values which are added to output rows. In a typical scenario some subset of step results is placed in output fields, so the result values are available to subsequent steps.

User-defined variables

Steps often offer a way to specify user-defined variables. User-defined variables are useful if the step is configured using complex expressions, and breaking down these expressions into individual named values helps to keep the intent of the configuration clear.

Error handling

Steps can encounter errors during execution. Files can be missing, databases may be unavailable, data may be in bad format, etc. Some errors are recoverable in the sense that even though they occur, the flow can maintain integrity and continue processing. Some errors are not recoverable in the sense that further processing is not possible, or likely to generate heavily corrupted outcomes.

Steps define error gates to handle errors. If a step encounters a recoverable error, it tries to continue through the error gate. If another step is connected through the error gate, execution continues there and the field error is set on the control record. The error field contains details about what went wrong. If there is no step connected at the error gate, the flow aborts as a whole. If a step encounters an error that is not recoverable, no attempt is made to continue through the error gate and the flow aborts immediately.