tweakstreet-logo
  • Enter search terms

Distinct

Filters out duplicate rows, retains only unique rows

Processing

The step calculates group fields for each row, and only passes distinct combinations of group fields through the out gate. It discards all rows with previously seen group fields.

Group by

Defines the grouping of rows.

Evaluated for each input row

Type
Data type of the grouping field. Values are implicitly cast to this type.
Name
Name of the grouping field.
Value
Value of the grouping field.

Settings

Name Type Description
Collated groups
 boolean

This step can optimize row output and memory consumption if it is guaranteed that incoming rows are sorted by group fields.

  • If checked, the step compares group fields only with the previous row’s group fields. If they are different, it considers them distinct. In this mode, the step generates distinct rows as they arrive.

  • If not checked, the step compares group fields with all previously seen group fields. This mode requires more memory. In this mode, the step generates output rows after processing the complete set of rows.

Evaluated once when step initializes

Results

Name Type Description
group
 dict Current grouping variables.
On this page