11.1 New & Noteworthy

Amazon EMR

Datameer X now supports Amazon

• EMR 5.30.0
• EMR 5.31.0
• EMR 6.0
• EMR 6.1.0

Cloudera

Datameer X now supports Cloudera CDP-DC/ Cloudera Runtime 7.1.4.1.

New Grid Mode for Separating Computing and Storaging

As a Admin, you can now separate the computing engine from the storage location in the 'Cluster Configuration'. With that new grid mode you can choose both the engine as well as the storage from a drop-down and combine it in the needed combination,

You can also implement necessary configuration properties for both of the corresponding selections.

New Plug-in: Event Logging

With the new plug-in 'Event Logging Plug-in' it is possible to log every event for a Datameer X artifact. The plug-in is configurable in the 'Admin' section and comes with two logging modes:

• logging required, but basic information of selected events and
• logging all available information of the published event as json'

The log is downloadable with the 'System Usage Report'.

Renaming of "Hadoop Cluster"

Since Datameer X is not only executable with Hadoop exclusively, you can find the settings for the cluster configuration under the 'Cluster Configuration' tab within the 'Admin' tab.

SQL Dialects Renaming

The following SQL dialect names have been optimized:

• 'MsSQL2014' is now 'Microsoft SQL Server 2014'
• 'HSQL_file' is now 'HSQL file'
• 'Neebo V1.2.1' is now 'Datameer Spotlight'

Using Jetty 9.4 for Importing Custom SSL Certificates

For importing a custom SSL certificate, Datameer X now uses Jetty 9.4.

Task Order in Housekeeping Service

A new default property 'housekeeping.run.task-attempts-per-run' is set to a value of '50' per default to decrease the performance of the Housekeeping Service significantly when it is to low.

Workbook Health Check Improvements

You can now download the Workbook Health Check as a JSON from the Workbook Health Check page. Furthermore the configuration IDs of all broken Workbooks are displayed as a comma-separated list unter the Workbook Health section.

Token Authentication for Tableau Connections

You can now select the authentication mode 'SSO Token' when creating a connection to a Tableau Server.

Amazon S3 - IAM Role

There is a new authentication option for the Amazon S3 connection: You can now authenticate via the IAM role with an assumed role.

Importing from ClickHouse

Importing data via a JDBC connection from a ClickHouse database is now possible. You need to have the ClickHouse database driver installed in the 'Admin' tab and then configure the JDBC connection to execute imports and exports.

Variable Error

When exporting data within an JDBC Export Job and selecting a 'Schema' or 'Table' value that do not exist, an error message is now shown when clicking on "Next" in the Export Wizard.

Mail Notifications in the File Browser Inspector

You can now adjust and view the mail notifications for successful or failed jobs within the Workbook Inspector. Find the configuration in the File Browser Inspector section 'Config'. The already set mail notifications when creating a job are viewable there and adjustable by clicking in the relevant section.

Modify Scheduling Configuration from the File Browser

You can now schedule the job triggering from the File Browser Inspector by selecting the required time specification:

You can select from:

• Manually
• Data Driven
• Hourly
• Daily
• Weekly
• Monthly and 
• Custom

File Browser Performance Improvements

Due to a complete File Browser refactoring, you will now benefit from a better performance and the following improved functionalities:

• Datameer X comes now with a react-based Folder Tree view, that works more fluently when performing an action - drag and drop works as well as the context menu is available
• a job's progress bar which is shown in the File Browser is now showing the current progress updated immediately
• scrolling through the File Browser works more smooth than before

Creating Root Folders

New root folders can be created from the Folder Tree or the artifacts bar. Selecting "Create new → Root Folder" from the context menu creates the new root folder which can be renamed.

Creating new simple folders can be executed by selecting "Create new → Folder" from the context menu as well.

Sheet Duplication Warnings

When duplicating a Workbook sheet, you will now see a warning message if the following columns get excluded:

• columns that are referenced in formulas
• columns that are referenced in filters
• columns that are referenced in sortings.

You will also receive an information where the columns are used in. Columns that are used in multiple places are noted as 'is used in multiple places'.

Workbook Sheet Inspector Resize Functionality

A Workbook's Sheet Inspector is now resizable in length of the single sections.

Workbook Function 'ERROR'

The new function 'ERROR' allows to use 'IF' or 'CASE' function to result in an error when a condition does or does not pass. It sets a cell error when the error handling mode is 'Ignore - leaves the cell(s) blank, logs an error' and the whole column won't be in an error. This results in a field error with a passed-in string message.

This function is useful for use-cases around around validation and cleansing data.  Using 'ERROR' will also allow for dropping records or aborting a workbook run when the error handling mode is set to one of those respectively.

Functions for Dictionary Matching

Datameer X has now two new functions for dictionary matching: "MATCHING_DICTIONARY" and "MATCHING_DICTIONARY_FUZZY".

JSON Parsing - Adding Custom Nodes

You can now add a custom node into the 'JSON Tree View' on a node of type 'OBJECT' or 'ARRAY' and also remove the new node. The 'Raw JSON' tab provides an edit button to edit/provide new JSON string. When in edit mode, all JSON parsing functionality is disabled. 

Workbook Sheet Usage

A new workbook query includes the usage of a sheet. The new call includes the information and displays it in the workbook configuration and workbook details page. Therefore a user knows if a sheet must be kept or not.

Cascading Permissions

A user can now add permissions within 'Cascade Permissions' from a folder in the File Browser instead of fully replace the existing permissions.

Workbook Sheet Inspector Column Icons

Within a Workbook Sheet Inspector, for each column the corresponding type icon is shown. Furthermore the 'Name' column is resizable. If a column is in an error state the column type icon will show an error icon instead of the column type icon.

Reduced API Calls for Opened Workbooks

Due to performance optimization, there will be only one API call to the backend for all opened Workbooks in tabs on the Datameer X User Interface.