/
12.1 New and Noteworthy

12.1 New and Noteworthy

Incremental Processing Improvements

Incremental Processing was introduced in 12.0 to allow users to configure incremental pipelines via Workbooks and Export Jobs that only process the latest data. Now users can:

  1. Include Import Jobs in incremental pipelines

  2. Chain Workbooks in incremental pipelines

  3. Select partitions as well when using advanced selection criteria

Folder-Based Custom Properties

Admins can now configure custom properties at the folder-level. This simplifies the configuration of shared job properties by allowing them to be configured for a folder instead of each job contained in that folder (or sub-folders). Child folders inherit the properties configured at parent, grand-parent, etc… to support sharing common settings for projects, deployment environments or teams.

Folder-Based Variables

Variables were originally introduced at the deployment-level (Datameer X server instance). Now they can be configured and/or overridden at the folder-level as well. This allows for teams, projects or deployment environments to have and manage their own variables without exposing them to all users of Datameer.

Snowflake Plugin Improvements

Private Key authentication support was added for Snowflake Connections, so it can be used for import and export jobs. Previously it was just implemented for the Snowflake DataWarehouse Connector.

Named Stage support has been added to the Snowflake DataWarehouse Connector to allow for Export Sheets that write to a named stage.

Kept Sheet Optimization

For newly created Workbooks, Datameer X will default to only marking “leaf” sheets as kept instead of all. Users can still manually control what sheets need to be kept if there are some unique requirements, but by default less data will need to be managed/stored. Existing Workbooks still use the previous “keep everything by default” strategy, but this can be updated through the UI or REST API to switch to the new behavior.

Quality of Life Improvements

  1. User: Combo-Boxes have been added to more dialogs and inspectors to make it easier to find/select columns when dealing with wide sheets and large workbooks.

  2. Admin: Scheduling and Triggering configuration is no longer included in the new job when copying Import Jobs, Workbook and Export jobs.

  3. Admin: JDBC-Based (Database) Exports now support post-execution hooks for running SQL after the export executes.

  4. Admin: Datameer X Database upgrade scripts now run in a single transaction, so no extra permissions are necessary.