Page Comparison

Info

title	INFO

Within these release notes Datameer X provides information and resolutions on improvements, bug fixes, and visual changes for 11.0.x maintenance release versions.

Visit Datameer's Help Center for additional information or contact your customer service representative.

Section

Datameer X 11.0.5

New Features

1	Import/Export: Google BigQuery - Add partition filtering functionality	DAP-41202
	It is now possible to import from a partitioned table.

2	Workbook: Custom functions for dictionary matching	DAP-41237
	Datameer X has now two new functions for dictionary matching: "MATCHING_DICTIONARY" and "MATCHING_DICTIONARY_FUZZY".

3	Import/ Export: S3 connection - Add the IAM role ARN option	DAP-41298
	There is a new authentication option for the S3 connection: You can now authenticate via the IAM role with an assumed role.

Improvements

1	Supported Hadoop Distributions: Add support for Cloudera CDP-DC/ Cloudera Runtime 7.2.0.2	DAP-40999
	Datameer X now supports Cloudera CDP-DC/ Cloudera Runtime 7.2.0.2.

2	Plug-Ins: plugin-hive - do not overwrite the fetch size and use the value defined in the jdbc connection string	DAP-41130
	The fetch size is new increased up to 5,000 to boost the jdbc-based import jobs.

3	Authentication: Terminate all user sessions on logout	DAP-41159
	Datameer X presents a new property 'devicesconductor.session.management.enabled'. If it is set to true, a second login with the same user will be prohibited. The property is disabled by default.

4	Admin UI: Database driver tab - wide grid or horizontal scrollbar	DAP-41197
	The UI has now more screen space for the grid, showing the installed JDBC drivers. Resizing the File Browser window, the width of the table is being adjusted.

5	Export: Hive - Improve the help text for the replacement strategy during the Hive export	DAP-41223
	The user now sees a more specific help text for the replacement strategy for a Hive export.

Bug fixes

1	File Browser: File Browser Schedule Inspector - Custom schedule is not reflected properly	DAP-39595
	The custom scheduling is now reflected in the Schedule Inspector.	Bug fixed.

2	File Browser: Folder permissions of subfolder - 'Read more' link is not working, no warning message is displayed	DAP-40658
	If a parent folder is not shared with others but the subfolder is shared and the 'Read More' link is not working properly a warning message will be displayed.	Bug fixed.

3	File Browser: Disable the folder context menu 'Rename' in the Folder Tree	DAP-40713
	If no folder is selected within the Folder Tree the 'Rename' option from the context menu will be disabled.	Bug fixed.

4	File Browser: Wrong documentation link for the file search	DAP-40889
	When searching for files in the File Browser the help link now points to the correct corresponding documentation page.	Bug fixed.

5	Workbook: Wrong color highlighting of formula parameter in the Formula Editor	DAP-40933
	Formula highlighting works as expected again.	Bug fixed.

6	Workbook: Renaming columns in a row by clicking the 'tab' key is not possible anymore	DAP-41042
	Renaming columns in a row by clicking the 'tab' key is working properly again.	Bug fixed.

7	Workbook: SQL sheet error appears three times for the wrong query expression	DAP-41096
	The SQL sheet error now only appears once when using a wrong query expression.	Bug fixed.

8	Workbook: Scheduling information is shown on a Workbook Details page even when the scheduling is disabled	DAP-41097
	The scheduling information from the Workbook Details page is now up-to-date after changing the scheduling manually.	Bug fixed.

9	Datameer Distribution: Jars missing in installation package for CDH-7.x.x	DAP-41180
	The installation package for Cloudera CDH-7.x.x distributions is now complete.	Bug fixed.

10	Connection: 'Save Connection' dialog - 'Close' is visible under the 'close' icon	DAP-41205
	The string 'close' within the 'Save Connection' dialog does not appear anymore.	Bug fixed.

11	Workbook: Post Upgrade Issue - NullPointerException in partition filter with $lastestpartition	DAP-41222
	Datameer X will not throw a NullPointerException anymore if a user tries to update the partition filter in a Workbook.	Bug fixed.

12	REST API: Concurrent session management blocks REST API to work	DAP-41242
	The concurrent session feature still works when enabled. The REST API works independently from enabling or disabling the concurrent session feature.	Bug fixed.

13	Workbook: Sheet synchronization - partition dialog closes without an error if an invalid value is entered	DAP-41251
	An appropriate error message will be displayed if invalid values are entered in the 'Kept Sheets & Synchronization' dialog.	Bug fixed.

14	Workbook: Upload a media file is not working	DAP-41289
	Media files can be uploaded again.	Bug fixed.

15	File Browser: Dependencies View only opens after the second attempt	DAP-41291
	Clicking the 'Dependencies View' icon now opens the Dependencies View at the first try.	Bug fixed.

16	Workbook: Attempt to duplicate the Workbook leads to broken references in the new artifact	DAP-41297
	Duplicating or copying an artifact does not lead to broken references in the new artifact anymore.	Bug fixed.

Section

Datameer X 11.0.4

Improvements

1	Workbook Health Check: Log status and progress into separate logs file 'logs/workbook_health.log'	DAP-41120
	The Workbook Health Check logs are now being written into the file 'logs/workbook_health_check.log'.

2	Supported Hadoop Distributions: Add support for Cloudera CDH 6.3.4	DAP-41138
	Datameer X now supports Cloudera CDH 6.3.4.

3	Plug-ins: 'Plugin-dwh' - Make Data Warehouse plug-in default for Datameer distribution	DAP-41150
	The plugin 'plugin-dwh' is now available as a default plugin and comes automatically with all Datameer X distributions.

Bug Fixes

1	File Browser: After canceling 'Create new folder', the File Browser does not update when selecting another folder	DAP-40673
	After canceling 'Create new folder', it is possible to change the folder in the Folder Tree panel.	Bug fixed.

2	Pivot Sheet: Creating a Pivot sheet with a broken Workbook source ends up in an error dialog	DAP-40729
	Creating a Pivot sheet out of a broken Workbook is not possible since the error state of a broken Workbook is shown first before one can create a sheet out of a broken Workbook.	Bug fixed.

3	File Browser: Prevent the option to open the 'Job run details' page if the necessary capability is missing	DAP-41129
	If a user does not have the capability “Common job details“ he will not enter the job-execution details page and an error message is shown instead.	Bug fixed.

4	Import: Clickhouse - Ingest of empty tables is not working	DAP-41140
	Ingesting empty tables now works properly when pointing to an empty table in a Clickhouse database.	Bug fixed.

5	Workbook: Error message will not be shown if a 'FULL OUT JOIN' is done in an SQL sheet	DAP-41015
	Errors are now shown in SQL sheets as well, even transitive errors.	Bug fixed.

Section

Datameer X 11.0.3

New Features

1	Import/ Export: Enable Import to ClickHouse via a JDBC connection	DAP-40951
	Importing data via a JDBC connection from a ClickHouse database is now possible. You need to have the ClickHouse database driver installed and then configure the JDBC connection to execute imports and exports.

2	Supported Hadoop Distributions: Add support for Amazon EMR 5.31.0	DAP-41025
	Datameer now supports Amazon EMR version 5.31.0.

3	File Browser: Add new root folder in the Folder Tree panel	DAP-41033
	A user can now add a new top-level folder/ root folder in the Folder Tree panel in the File Browser. The newly created folder is named 'Untitled Folder' and can be renamed immediately after creating it. The capability 'CREATE_FOLDER' is needed.

Improvements

1	Workbook Health Check: Enable the download of the the report and print the configuration IDs of all broken Workbooks	DAP-40628
	You can now download the Workbook Health Check as a JSON from the Workbook Health Check page. Furthermore the configuration IDs of all broken Workbooks are displayed as a comma-separated list unter the Workbook Health section.

2	Housekeeping Service: Housekeeping Service should attempt to delete HDFS data artifacts for non-existent users	DAP-40759
	Deleting artifacts that are owned by a removed Datameer user is now possible. Deleting can be executed a Datameer Service user that has direct access to the HDFS cluster.

3	Rebranding: Datameer Classic is now DatameerX	DAP-40913
	In course of the rebranding, The former 'Datameer Classic' is now called 'DatameerX'. The new Datameer products are called 'Spectrum' and 'Spotlight'. Find more information about all of our products on our website.

4	SFTP/ SCP import: Reduce the amount of the opened connections while multiple small file import	DAP-41011
	The new configuration property 'das.cluster.hadoop.fs.cache.scheme.whitelist' which is configurable globally and on job level, reduces the amount of SFTP/ SCP sessions. With the FileServer cache enabled for scp/sftp scheme, there will be a single SSH connection for each cluster task launched, limiting the "connection refused" occurances.

5	Connections: Configurable root path prefix for Google Cloud Storage connector	DAP-41018
	The Google Cloud Storage connector now has a 'root path prefix' property.

6	SFTP/SCP Connection: Private key format requirements help text	DAP-41038
	Datameer provides a help text next to the field 'SSH key' in the connection details to make sure the key is in PEM format and other requirements like password-protected keys are not supported.

7	Plug-ins: Google Cloud Storage plug-in should resolve variables	DAP-41050
	Variable resolution for bucket and the root path is now implemented in the plug-in 'plugin-gsc'.

8	Authentication: SAML authentication - Disable the local account management	DAP-41058
	The mixed-mode parameter that defines whether or not local authentication is available while using SAML (or LDAP), can now be configured within the SAML plug-in settings. Having it set to false, only the external authentication mode is available. Having it set to true, both external and local authentication mode are available.

9	Connection: Amazon S3 Native connector should validate the region and bucket combination properly	DAP-41074
	When setting up an Amazon S3 Native connection the region is now validated against the bucket properly.

Bug Fixes

1	User Interface: Scrolling problems when the browser is vertically maximized	DAP-40723
	When the browser window is vertically maximized, the scroll bar works as expected and you can scroll up and down within a Workbook.	Bug fixed.

2	Workbook: Workbook preview missing when using 'Browse Column' in the Workbook Inspector	DAP-40987
	The Workbook preview is displayed while using 'Browse Columns' from another sheet.	Bug fixed.

3	Workbook: Sheet error appears three times	DAP-41005
	A sheet error is displayed once it happens.	Bug fixed.

4	Sequential triggering: Do not display a complete list of all variables on the import/ export job run details page	DAP-41007
	The list 'Variables' from the import/ export job run details page contains only the used variables.	Bug fixed.

5	Cluster Configuration: NullPointerException when saving Google Cloud Platform (Dataproc) on the 'Cluster Configuration' page	DAP-41012
	Editing and saving the cluster configuration does not throw an NullPointerException again and takes over the service account key for storage in a JSON format.	Bug fixed.

6	File Browser: Unable to 'Duplicate Workbook', the dialog remembers the previous filename value	DAP-41020
	Duplicating a Workbook works as expected. Executing the duplication creates a copy of the file and can be renamed as required. The default name of a duplicated Workbook is 'Copy_of_<initial Workbook name>'.	Bug fixed.

7	Connection: Google BigQuery export fails with staging bucket name that contains underscores	DAP-41045
	It is now possible to have the Google Cloud Storage bucket containing underscores as the Google BigQuery connector staging bucket/ directory.	Bug fixed.

8	Workbook: The penultimate column is not selected automatically after deleting the last column	DAP-41048
	The penultimate column is now selected automatically after deleting the last column.	Bug fixed.

9	Login: After session timeout - error when first re-login attempt	DAP-41055
	A user is able to log in to the Datameer User Interface after a session timeout from the first attempt on.	Bug fixed.

10	Security: Datameer Support Engineer Report extension allows reading random files from the Datameer application server OS	DAP-41059
	The Datameer extension Support Engineer Report reads files from the Datameer installation folder only (configuration as well as log files).	Bug fixed.

11	REST API: Path information disclosure when calling for a non-existent file	DAP-41060
	When calling for a non-existent file a '404' error is shown containing the requested file name.	Bug fixed.

12	Security: 'productDASData' text/ javascript block is available at the page's source even when the 'About' menu is disabled	DAP-41061
	When the 'About Datameer' menu is disabled, the 'productDASData' text/ javascript block is deactivated as well.	Bug fixed.

13	Workbook: Sheet errors are missing	DAP-41069
	All error messages for Workbook error are available, e.g. formula syntax error.	Bug fixed.

14	Plug-ins: plugin-s3 (S3 Native connector) on EMR - Export to a bucket in different S3 region fails	DAP-41073
	The plug-in is fixed. An export via the S3 Native connection works regardless of a bucket region.	Bug fixed.

15	Export: An export job should not inject 'DataStoreModel#setupConf' into the global 'JobConf'	DAP-41088
	An export job creates a dedicated 'JobConf' for export-only purposes.	Bug fixed.

16	Authentication: EMR- S3 connection uses IAM mode always when the cluster mode S3 is set to IAM	DAP-41100
	The Amazon S3 connection uses the configured authentication mode from the connection configuration: Access Secret/ Key.	Bug fixed.

Section

Datameer X 11.0.2

New Features

1	Supported Hadoop Distributions: Add support for Amazon EMR 6.1.0	DAP-40876
	Datameer X now supports Amazon EMR version 6.1.0.

2	JSON Parsing: Add ability to insert a custom node into the 'JSON Tree View'	DAP-39833
	A user can now add a custom node into the 'JSON Tree View' on a node of type 'OBJECT' or 'ARRAY' and also remove the new node. The 'Raw JSON' tab provides an edit button to edit/provide new JSON string. When in edit mode, all JSON parsing functionality is disabled.

Improvements

1	Import/ Export: Support for Azure Synapse (formerly Microsoft SQL Data Warehouse)	DAP-27739
	You can now import from or export to an Azure Synapse Database. The according dialect is being supported as well.

2	Supported Hadoop Distributions: Add support for Amazon EMR 5.30.1	DAP-40875
	Datameer X now supports Amazon EMR version 5.30.1.

3	Supported Hadoop Distributions: Add support for Cloudera CDP-DC/ Cloudera Runtime 7.1.3	DAP-40877
	Datameer X now supports Cloudera CDP-DC/ Cloudera Runtime version 7.1.3.

4	Connection: Rewrite Google BigQuery import connector to use Google BigQuery storage API	DAP-40883
	When importing from or exporting to Google BigQuery, the new Google BigQuery storage API is used for more efficient imports and exports..

5	Authentication: Remove 'gssapi-with-mic' from Datameer's 'default.properties'	DAP-40799
	The value 'gssapi-with-mic' is removed from the 'default.properties' key 'system.property.jsch.preferred.authentication'. The new added value is 'password,keyboard-interactive,publickey'.

6	Supported Hadoop Distributions: Add support for Apache Hadoop 2.7.7 and Hive 2.3.7	DAP-40895
	Datameer X now supports Hadoop version 2.7.7 and Hive version 2.3.7.

7	Workbook: Add ability to 'update' instead of 'replace' for 'Cascade Permissions'	DAP-40916
	A user can now add permissions within 'Cascade Permissions' from a folder in the File Browser instead of fully replace the existing permissions.

8	Google BigQuery: Do not use temporary table when streaming from tables	DAP-40939
	Imports from Google BigQuery now work without changes for tables, views and queries.

9	Plug-ins: JDBC/Snowflake - Unimplemented date/ time/ timestamp column splitting	DAP-40952
	The plug-in 'plugin-jdbc-v2' is now adapted to the column splitting needs.

10	Supported Hadoop Distributions: Add support for Cloudera CDP-DC/ Cloudera Runtime 7.1.2	DAP-40998
	Datameer X now supports Cloudera CDP-DC/ Cloudera Runtime version 7.1.2.

11	Supported Hadoop Distributions: Add support for Cloudera CDP-DC/ Cloudera Runtime 7.2.0.2	DAP-40999
	Datameer X now supports Cloudera CDP-DC/ Cloudera Runtime version 7.2.0.2.

Bug Fixes

1	Workbook: SQL - 'EXCEPT' and 'INTERSECT' operator null value handling	DAP-40739
	The Datameer X functions 'EXCEPT' and 'INTERSECT' work properly again. The null value handling returns an empty result as expected.	Bug fixed.

2	Workbook: Validate for sheet error when changing column type which is used for an encoding sheet	DAP-40892
	The sheet error is being registered and a proper error message is shown when changing the column type which is used for an encoding sheet.	Bug fixed.

3	Connection: ZooKeeper - Datameer X does not close ZooKeeper connections (HBase)	DAP-40970
	The HBase plug-in was fixed so Datameer X now closes ZooKeeper connections to HBase again when the connection is saved.	Bug fixed.

4	Connection: MariaDB dialect is missing export transformation 'SqlDateTransformation'	DAP-40989
	The needed export transformation is now added. MariaDB is changed to apply the export transformation 'SqLDateTransformation'.	Bug fixed.

5	Plug-ins: JDBC 2/ Snowflake - Integer column splitting skips table rows for import	DAP-40990
	All splitted columns from type integer can be imported completely.	Bug fixed.

6	Workbook: Sequential triggering does not respect the execution order	DAP-41000
	Sequential triggering executes the artifacts based on the execution order.	Bug fixed.

Section

Datameer X 11.0.1

New Features

1	Workbook: New call to insert usage of sheets into Workbook config and details page	DAP-40722
	A new workbook query includes the usage of a sheet. The new call includes the information and displays it in the workbook configuration and workbook details page. Therefore a user knows if a sheet must be kept or not.

Improvements

1	Workbook: Show each column in error state separately/ more prominent	DAP-40370
	Columns in an error state are highlighted in an error color in the columns list inside the Workbook Sheet Inspector. Clicking the column in error state highlights the column in the sheet. Furthermore the column list height is resizable now.

2	Workbook: Improve filter error validation by showing why it does not validate	DAP-40594
	When entering an invalid advanced filter definition, the error message states the problem of the filter definition.

3	Workbook Health Check: Provide the possibility to hide the loading screen and cancel the check	DAP-40607
	You can now hide the Workbook Health Check loading screen and continue working while the check is executed in the background. Furthermore you can cancel the check completely.

4	Security: SAML authentication - role mapping proceeding	DAP-40742
	In the 'Admin' tab in the SAML authentication you can now assign a role and/ or a group to a user directly, e.g. 'User1: ANALYST:sales'. Note that the group must exist when assigning the group to the role.

5

Logging: New log file to store Datameer X PID (Process ID)

DAP-40761

A new log file named 'start-pid.log' is introduced to log the Datameer X start time and the corresponding process ID. Whenever Datameer X starts, a new line is being added to the log file. Find the log file under '/logs/dir'.

6	Workbook: Add join sheet creation validation query	DAP-40784
	If an invalid join definition is trying to be created, a join validation provides a hint to the user why the join can not be executed.

7	Workbook: No consumer indication in the workbook configuration dialog if sheet is configured to be synced via Open Document Format	DAP-40818
	The workbook configuration dialog now shows an indication that also synced sheets have a dependency consumed by. A workbook detail page contains truncated entries and a tooltip if the tables are longer than the rendered column width.

8	Security: Make a SAML plug-in aware of a load balancer that offloads SSL	DAP-40868
	The load balancer offloads SSL to the Datameer X SAML plug-in. A new field is provided that enables - in addition to the existing Jetty-based SSL termination - the load balancer-based SSL offloading.

9	Add Support for Amazon EMR 5.30.1	DAP-40875
	Datameer X now supports Amazon EMR 5.30.1

10	Security: SAML - Add option to read roles only from repository	DAP-40886
	User roles can now be read from the repository. All SAML assertion properties (username, email, groups and roles) are configurable as well.

Bug Fixes

1	Export: Can't export string values into MariaDB	DAP-38661
	New Hadoop configuration parameters, that may be set in the 'Admin' tab → 'Cluster Configuration' section or during the export job configuration, override the dialect's default varchar field length. The export to MariaDB is now possible.	Bug fixed.

2	Import: Google BigQuery - Table list is empty after selecting a dataset	DAP-40542
	The table list appears not empty any more in the import job configuration.	Bug fixed.

3	Datameer X Distribution: 'Snappy-java-*.jar' is missed in installation packages for Amazon EMR 6.0.0	DAP-40721
	'Snappy-java-1.1.4.jar' is now bundled with the distribution.	Bug fixed.

4	Housekeeping: Set flag to DATA_DELETION_TURNED_OFF instead of DELETION_FAILED when Housekeeping data deletion is disabled	DAP-40744
	The status column in 'filesystem_artifact_to_delete table can now have the value '2=DATA_DELETION_DISABLED'. That means that artifacts are scheduled for deletion but the distributed storage artifact deletion has been disabled via the 'housekeeping.data.deletion.disabled=true' flag.	Bug fixed.

5	Workbook: Columns.unequalCount error in join sheet	DAP-40753
	An error is shown when submitting a join definition with duplicate keys also stating how to recover. Find more information here.	Bug fixed.

6	Workbook: IllegalArgumentException for join operation with multiple conditions	DAP-40754
	Join operations with multiple conditions work again.	Bug fixed.

7	Connections: Databricks import connector throws an 'org.apache.spark.sql.AnalysisException'	DAP-40764
	The exception is fixed and the import works as expected.	Bug fixed.

8	Datameer X distribution: Show port after starting a Datameer X distribution with 'bin/conductor.sh start	DAP-40765
	The port is now displayed after starting Datameer X.	Bug fixed.

9	Admin: Non-admin user can change folder ownership and select other users with the same role	DAP-40767
	If a user has no capability 'Change Folder Owner' but the capability 'User can modify every file and folder' he is not able to change the owner of the folder.	Bug fixed.

10	Import: Columns of UUID type are being dropped from a PostgreSQL import	DAP-40794
	The UUID type is now supported for the import and therefore the import job is executed successfully.	Bug fixed.

11	Workbook: Can't duplicate or copy a workbook if the sheet execution order does not match the view order	DAP-40805
	Duplicating or copying a workbook works as expected and Datameer X follows the sheet logical order regardless of their current sequential position.	Bug fixed.

12	Database drivers: The MariaDB database driver has a wrong name in the JDBC connection string template	DAP-40855
	The MariaDB database JDBC connection string now has the correct string template and you can establish a connection as expected.	Bug fixed.

13	Authentication: The default LDAP user filter does not filter out objects with username or email attribute set as an empty string	DAP-40884
	User objects with a username or email attribute set to an empty string are now filtered out and do not break the LDAP cache creation.	Bug fixed.

14	Datameer X distribution: Amazon EMR should use a conform Hive version	DAP-40890
	A Datameer X distribution with Amazon EMR now uses Hive 2.3.2.	Bug fixed.

15	Datameer X distribution: Amazon EMR versions should use patched Hive/ Hadoop jars if available	DAP-40891
	Datameer X uses patched Hive and Hadoop jars for compatibly reasons.	Bug fixed.

Versions Compared

Old Version 6

New Version 7

Key

Datameer X 11.0.5

New Features

Improvements

Bug fixes

Datameer X 11.0.4

Improvements

Bug Fixes

Datameer X 11.0.3

New Features

Improvements

Bug Fixes

Datameer X 11.0.2

New Features

Improvements

Bug Fixes

Datameer X 11.0.1

New Features

Improvements

Bug Fixes