tweakstreet-logo
  • Enter search terms

Junk Dimension

Inserts/Looks up records in a junk dimension table

Processing

For every input row, the step looks for a record with the given write field values in the database table.

If no such record exists, it is inserted, adding the configured meta field values to the record.

The primary key of the found or inserted record is made available as a result.

Settings

Name Type Description
Connection
 dict

The database connection to use.

Evaluated for each input row
Schema
 string

The database schema of the table.

Evaluated for each input row
Table
 string

The database table to insert into.

Evaluated for each input row
Commit Size
 long

Number of records after which data is committed.

  • When set to a number greater than 0, commits updates after having inserted the given amount of records.
  • When set to 0 a single commit is issued after all records are processed.
  • When set to nil this step does not issue any commits. This is useful if you are controlling transactional behaviour manually, or the database does not support transactions.
  • Any other value produces an error.

Evaluated once when step initializes
Cache
Use Cache
 boolean

Use a in-memory cache to hold table rows.

Evaluated once when step initializes
Cache Size
 long

Number of records cached

  • When set to a number greater than 0, creates a lookup cache of the given size.
  • When set to 0 all records are cached.
  • Any other value produces an error.

Evaluated once when step initializes
Pre-load Cache
 boolean

If true, the step pre-fills the cache with the contents of the table.

Evaluated once when step initializes
Pre-load SQL Filter
 string

If non-empty, this query text is appended as an AND condition to the cache pre-loading query.

It should contain a predicate to append to the where clause of the query, for example: country = 'FR'

This is useful to selectively pre-warm the cache with a specific subset of records.

Evaluated once when step initializes
Pre-load SQL Parameters
 list

If the pre-load cache filter contains ? parameter placeholders, they are bound to this list of values in order of appearance.

Evaluated once when step initializes

Write fields

Defines fields to look up and write to the database table.

Evaluated for each input row

Type
Data type of the field. Field values are implicitly cast to this type.
Name
Name of the table field to write to.
Value
The value to write.

Meta fields

Additional fields that are inserted when a new record is created.

Evaluated for each input row

Type
Data type of the field. Field values are implicitly cast to this type.
Name
Name of the table field to write to.
Value
The value to write.

Results

Name Type Description
key
 long Primary key for current record
On this page