INFO
Within these release notes Datameer X provides information and resolutions on improvements, bug fixes, and visual changes for 11.0.x maintenance release versions.
Visit Datameer's Help Center for additional information or contact your customer service representative.
Datameer X 11.0.3
New Features
1 | Import/ Export: Enable Import to ClickHouse via a JDBC connection | DAP-40951 |
|---|---|---|
Importing data via a JDBC connection from a ClickHouse database is now possible. You need to have the ClickHouse database driver installed and then configure the JDBC connection to execute imports and exports. |
| 2 | Supported Hadoop Distributions: Add support for Amazon EMR 5.31.0 | DAP-41025 |
|---|---|---|
| Datameer now supports Amazon EMR version 5.31.0. |
| 3 | File Browser: Add new root folder in the Folder Tree panel | DAP-41033 |
|---|---|---|
| A user can now add a new top-level folder/ root folder in the Folder Tree panel in the File Browser. The newly created folder is named 'Untitled Folder' and can be renamed immediately after creating it. The capability 'CREATE_FOLDER' is needed. |
Improvements
1 | Workbook Health Check: Enable the download of the the report and print the configuration IDs of all broken Workbooks | DAP-40628 |
|---|---|---|
You can now download the Workbook Health Check as a JSON from the Workbook Health Check page. Furthermore the configuration IDs of all broken Workbooks are displayed as a comma-separated list unter the Workbook Health section. |
| 2 | Housekeeping Service: Housekeeping Service should attempt to delete HDFS data artifacts for non-existent users | DAP-40759 |
|---|---|---|
| Deleting artifacts that are owned by a removed Datameer user is now possible. Deleting can be executed a Datameer Service user that has direct access to the HDFS cluster. |
3 | Rebranding: Datameer Classic is now DatameerX | DAP-40913 |
|---|---|---|
In course of the rebranding, The former 'Datameer Classic' is now called 'DatameerX'. The new Datameer products are called 'Spectrum' and 'Spotlight'. Find more information about all of our products on our website. |
4 | SFTP/ SCP import: Reduce the amount of the opened connections while multiple small file import | DAP-41011 |
|---|---|---|
The new configuration property 'das.cluster.hadoop.fs.cache.scheme.whitelist' which is configurable globally and on job level, reduces the amount of SFTP/ SCP sessions. With the FileServer cache enabled for scp/sftp scheme, there will be a single SSH connection for each cluster task launched, limiting the "connection refused" occurances. |
| 5 | Connections: Configurable root path prefix for Google Cloud Storage connector | DAP-41018 |
|---|---|---|
| The Google Cloud Storage connector now has a 'root path prefix' property. |
| 6 | SFTP/SCP Connection: Private key format requirements help text | DAP-41038 |
|---|---|---|
| Datameer provides a help text next to the field 'SSH key' in the connection details to make sure the key is in PEM format and other requirements like password-protected keys are not supported. |
| 7 | Plug-ins: Google Cloud Storage plug-in should resolve variables | DAP-41050 |
|---|---|---|
| Variable resolution for bucket and the root path is now implemented in the plug-in 'plugin-gsc'. |
8 | Authentication: SAML authentication - Disable the local account management | DAP-41058 |
|---|---|---|
The mixed-mode parameter that defines whether or not local authentication is available while using SAML (or LDAP), can now be configured within the SAML plug-in settings. Having it set to false, only the external authentication mode is available. Having it set to true, both external and local authentication mode are available. |
9 | Connection: Amazon S3 Native connector should validate the region and bucket combination properly | DAP-41074 |
|---|---|---|
When setting up an Amazon S3 Native connection the region is now validated against the bucket properly. |
Bug fixes
1 | User Interface: Scrolling problems when the browser is vertically maximized | DAP-40723 |
|---|---|---|
When the browser window is vertically maximized, the scroll bar works as expected and you can scroll up and down within a Workbook. |
|
| 2 | Workbook: Workbook preview missing when using 'Browse Column' in the Workbook Inspector | DAP-40987 |
|---|---|---|
| The Workbook preview is displayed while using 'Browse Columns' from another sheet. |  Bug fixed. |
| 3 | Workbook: Sheet error appears three times | DAP-41005 |
|---|---|---|
| A sheet error is displayed once it happens. | Bug fixed. |
4 | Sequential triggering: Do not display a complete list of all variables on the import/ export job run details page | DAP-41007 |
|---|---|---|
The list 'Variables' from the import/ export job run details page contains only the used variables. |
|
| 5 | Cluster Configuration: NullPointerException when saving Google Cloud Platform (Dataproc) on the 'Cluster Configuration' page | DAP-41012 |
|---|---|---|
| Editing and saving the cluster configuration does not throw an NullPointerException again and takes over the service account key for storage in a JSON format. | Bug fixed. |
6 | File Browser: Unable to 'Duplicate Workbook', the dialog remembers the previous filename value | DAP-41020 |
|---|---|---|
Duplicating a Workbook works as expected. Executing the duplication creates a copy of the file and can be renamed as required. The default name of a duplicated Workbook is 'Copy_of_<initial Workbook name>'. |
|
7 | Connection: Google BigQuery export fails with staging bucket name that contains underscores | DAP-41045 |
|---|---|---|
| It is now possible to have the Google Cloud Storage bucket containing underscores as the Google BigQuery connector staging bucket/ directory. |
|
| 8 | Workbook: The penultimate column is not selected automatically after deleting the last column | DAP-41048 |
|---|---|---|
| The penultimate column is now selected automatically after deleting the last column. | Bug fixed. |
| 9 | Login: After session timeout - error when first re-login attempt | DAP-41055 |
|---|---|---|
| A user is able to log in to the Datameer User Interface after a session timeout from the first attempt on. | Bug fixed. |
| 10 | Security: Datameer Support Engineer Report extension allows reading random files from the Datameer application server OS | DAP-41059 |
|---|---|---|
The Datameer extension Support Engineer Report reads files from the Datameer installation folder only (configuration as well as log files). |
|
11 | REST API: Path information disclosure when calling for a non-existent file | DAP-41060 |
|---|---|---|
When calling for a non-existent file a '404' error is shown containing the requested file name. |
|
12 | Security: 'productDASData' text/ javascript block is available at the page's source even when the 'About' menu is disabled | DAP-41061 |
|---|---|---|
When the 'About Datameer' menu is disabled, the 'productDASData' text/ javascript block is deactivated as well. |
|
| 13 | Workbook: Sheet errors are missing | DAP-41069 |
|---|---|---|
| All error messages for Workbook error are available, e.g. formula syntax error. | Bug fixed. |
14 | Plug-ins: plugin-s3 (S3 Native connector) on EMR - Export to a bucket in different S3 region fails | DAP-41073 |
|---|---|---|
The plug-in is fixed. An export via the S3 Native connection works regardless of a bucket region. |
|
| 15 | Export: An export job should not inject 'DataStoreModel#setupConf' into the global 'JobConf' | DAP-41088 |
|---|---|---|
| An export job creates a dedicated 'JobConf' for export-only purposes. | Bug fixed. |
| 16 | Authentication: EMR- S3 connection uses IAM mode always when the cluster mode S3 is set to IAM | DAP-41100 |
|---|---|---|
| The Amazon S3 connection uses the configured authentication mode from the connection configuration: Access Secret/ Key. | Bug fixed. |
Datameer X 11.0.2
New Features
1 | Supported Hadoop Distributions: Add support for Amazon EMR 6.1.0 | DAP-40876 |
|---|---|---|
Datameer X now supports Amazon EMR version 6.1.0. |
2 | JSON Parsing: Add ability to insert a custom node into the 'JSON Tree View' | DAP-39833 |
|---|---|---|
A user can now add a custom node into the 'JSON Tree View' on a node of type 'OBJECT' or 'ARRAY' and also remove the new node. The 'Raw JSON' tab provides an edit button to edit/provide new JSON string. When in edit mode, all JSON parsing functionality is disabled. |
Improvements
1 | Import/ Export: Support for Azure Synapse (formerly Microsoft SQL Data Warehouse) | DAP-27739 |
|---|---|---|
You can now import from or export to an Azure Synapse Database. The according dialect is being supported as well. |
2 | Supported Hadoop Distributions: Add support for Amazon EMR 5.30.1 | DAP-40875 |
|---|---|---|
Datameer X now supports Amazon EMR version 5.30.1. |
3 | Supported Hadoop Distributions: Add support for Cloudera CDP-DC/ Cloudera Runtime 7.1.3 | DAP-40877 |
|---|---|---|
Datameer X now supports Cloudera CDP-DC/ Cloudera Runtime version 7.1.3. |
4 | Connection: Rewrite Google BigQuery import connector to use Google BigQuery storage API | DAP-40883 |
|---|---|---|
When importing from or exporting to Google BigQuery, the new Google BigQuery storage API is used for more efficient imports and exports.. |
5 | Authentication: Remove 'gssapi-with-mic' from Datameer's 'default.properties' | DAP-40799 |
|---|---|---|
The value 'gssapi-with-mic' is removed from the 'default.properties' key 'system.property.jsch.preferred.authentication'. The new added value is 'password,keyboard-interactive,publickey'. |
6 | Supported Hadoop Distributions: Add support for Apache Hadoop 2.7.7 and Hive 2.3.7 | DAP-40895 |
|---|---|---|
Datameer X now supports Hadoop version 2.7.7 and Hive version 2.3.7. |
7 | Workbook: Add ability to 'update' instead of 'replace' for 'Cascade Permissions' | DAP-40916 |
|---|---|---|
A user can now add permissions within 'Cascade Permissions' from a folder in the File Browser instead of fully replace the existing permissions. |
8 | Google BigQuery: Do not use temporary table when streaming from tables | DAP-40939 |
|---|---|---|
Imports from Google BigQuery now work without changes for tables, views and queries. |
9 | Plug-ins: JDBC/Snowflake - Unimplemented date/ time/ timestamp column splitting | DAP-40952 |
|---|---|---|
The plug-in 'plugin-jdbc-v2' is now adapted to the column splitting needs. |
10 | Supported Hadoop Distributions: Add support for Cloudera CDP-DC/ Cloudera Runtime 7.1.2 | DAP-40998 |
|---|---|---|
Datameer X now supports Cloudera CDP-DC/ Cloudera Runtime version 7.1.2. |
11 | Supported Hadoop Distributions: Add support for Cloudera CDP-DC/ Cloudera Runtime 7.2.0.2 | DAP-40999 |
|---|---|---|
Datameer X now supports Cloudera CDP-DC/ Cloudera Runtime version 7.2.0.2. |
Bug fixes
1 | Workbook: SQL - 'EXCEPT' and 'INTERSECT' operator null value handling | DAP-40739 |
|---|---|---|
The Datameer X functions 'EXCEPT' and 'INTERSECT' work properly again. The null value handling returns an empty result as expected. |
|
2 | Workbook: Validate for sheet error when changing column type which is used for an encoding sheet | DAP-40892 |
|---|---|---|
The sheet error is being registered and a proper error message is shown when changing the column type which is used for an encoding sheet. |
|
| 3 | Connection: ZooKeeper - Datameer X does not close ZooKeeper connections (HBase) | DAP-40970 |
|---|---|---|
The HBase plug-in was fixed so Datameer X now closes ZooKeeper connections to HBase again when the connection is saved. |
|
4 | Connection: MariaDB dialect is missing export transformation 'SqlDateTransformation' | DAP-40989 |
|---|---|---|
The needed export transformation is now added. MariaDB is changed to apply the export transformation 'SqLDateTransformation'. |
|
5 | Plug-ins: JDBC 2/ Snowflake - Integer column splitting skips table rows for import | DAP-40990 |
|---|---|---|
All splitted columns from type integer can be imported completely. |
|
6 | Workbook: Sequential triggering does not respect the execution order | DAP-41000 |
|---|---|---|
Sequential triggering executes the artifacts based on the execution order. |
|
Datameer X 11.0.1
New Features
1 | Workbook: New call to insert usage of sheets into Workbook config and details page | DAP-40722 |
|---|---|---|
A new workbook query includes the usage of a sheet. The new call includes the information and displays it in the workbook configuration and workbook details page. Therefore a user knows if a sheet must be kept or not. |
Improvements
1 | Workbook: Show each column in error state separately/ more prominent | DAP-40370 |
|---|---|---|
Columns in an error state are highlighted in an error color in the columns list inside the Workbook Sheet Inspector. Clicking the column in error state highlights the column in the sheet. Furthermore the column list height is resizable now. |
2 | Workbook: Improve filter error validation by showing why it does not validate | DAP-40594 |
|---|---|---|
When entering an invalid advanced filter definition, the error message states the problem of the filter definition. |
3 | Workbook Health Check: Provide the possibility to hide the loading screen and cancel the check | DAP-40607 |
|---|---|---|
You can now hide the Workbook Health Check loading screen and continue working while the check is executed in the background. Furthermore you can cancel the check completely. |
4 | Security: SAML authentication - role mapping proceeding | DAP-40742 |
|---|---|---|
In the 'Admin' tab in the SAML authentication you can now assign a role and/ or a group to a user directly, e.g. 'User1: ANALYST:sales'. Note that the group must exist when assigning the group to the role. |
5 | Logging: New log file to store Datameer X PID (Process ID) | DAP-40761 |
|---|---|---|
A new log file named 'start-pid.log' is introduced to log the Datameer X start time and the corresponding process ID. Whenever Datameer X starts, a new line is being added to the log file. Find the log file under '/logs/dir'. |
6 | Workbook: Add join sheet creation validation query | DAP-40784 |
|---|---|---|
If an invalid join definition is trying to be created, a join validation provides a hint to the user why the join can not be executed. |
7 | Workbook: No consumer indication in the workbook configuration dialog if sheet is configured to be synced via Open Document Format | DAP-40818 |
|---|---|---|
The workbook configuration dialog now shows an indication that also synced sheets have a dependency consumed by. A workbook detail page contains truncated entries and a tooltip if the tables are longer than the rendered column width. |
8 | Security: Make a SAML plug-in aware of a load balancer that offloads SSL | DAP-40868 |
|---|---|---|
The load balancer offloads SSL to the Datameer X SAML plug-in. A new field is provided that enables - in addition to the existing Jetty-based SSL termination - the load balancer-based SSL offloading. |
9 | Add Support for Amazon EMR 5.30.1 | DAP-40875 |
|---|---|---|
Datameer X now supports Amazon EMR 5.30.1 |
10 | Security: SAML - Add option to read roles only from repository | DAP-40886 |
|---|---|---|
User roles can now be read from the repository. All SAML assertion properties (username, email, groups and roles) are configurable as well. |
Bug fixes
1 | Export: Can't export string values into MariaDB | DAP-38661 |
|---|---|---|
New Hadoop configuration parameters, that may be set in the 'Admin' tab → 'Cluster Configuration' section or during the export job configuration, override the dialect's default varchar field length. The export to MariaDB is now possible. |
|
2 | Import: Google BigQuery - Table list is empty after selecting a dataset | DAP-40542 |
|---|---|---|
The table list appears not empty any more in the import job configuration. |
|
3 | Datameer X Distribution: 'Snappy-java-*.jar' is missed in installation packages for Amazon EMR 6.0.0 | DAP-40721 |
|---|---|---|
'Snappy-java-1.1.4.jar' is now bundled with the distribution. |
|
4 | Housekeeping: Set flag to DATA_DELETION_TURNED_OFF instead of DELETION_FAILED when Housekeeping data deletion is disabled | DAP-40744 |
|---|---|---|
The status column in 'filesystem_artifact_to_delete table can now have the value '2=DATA_DELETION_DISABLED'. That means that artifacts are scheduled for deletion but the distributed storage artifact deletion has been disabled via the 'housekeeping.data.deletion.disabled=true' flag. |
|
5 | Workbook: Columns.unequalCount error in join sheet | DAP-40753 |
|---|---|---|
An error is shown when submitting a join definition with duplicate keys also stating how to recover. Find more information here. |
|
6 | Workbook: IllegalArgumentException for join operation with multiple conditions | DAP-40754 |
|---|---|---|
Join operations with multiple conditions work again. |
|
| 7 | Connections: Databricks import connector throws an 'org.apache.spark.sql.AnalysisException' | DAP-40764 |
|---|---|---|
The exception is fixed and the import works as expected. |
|
8 | Datameer X distribution: Show port after starting a Datameer X distribution with 'bin/conductor.sh start | DAP-40765 |
|---|---|---|
The port is now displayed after starting Datameer X. |
|
9 | Admin: Non-admin user can change folder ownership and select other users with the same role | DAP-40767 |
|---|---|---|
If a user has no capability 'Change Folder Owner' but the capability 'User can modify every file and folder' he is not able to change the owner of the folder. |
|
10 | Import: Columns of UUID type are being dropped from a PostgreSQL import | DAP-40794 |
|---|---|---|
The UUID type is now supported for the import and therefore the import job is executed successfully. |
|
11 | Workbook: Can't duplicate or copy a workbook if the sheet execution order does not match the view order | DAP-40805 |
|---|---|---|
Duplicating or copying a workbook works as expected and Datameer X follows the sheet logical order regardless of their current sequential position. |
|
12 | Database drivers: The MariaDB database driver has a wrong name in the JDBC connection string template | DAP-40855 |
|---|---|---|
The MariaDB database JDBC connection string now has the correct string template and you can establish a connection as expected. |
|
13 | Authentication: The default LDAP user filter does not filter out objects with username or email attribute set as an empty string | DAP-40884 |
|---|---|---|
User objects with a username or email attribute set to an empty string are now filtered out and do not break the LDAP cache creation. |
|
14 | Datameer X distribution: Amazon EMR should use a conform Hive version | DAP-40890 |
|---|---|---|
A Datameer X distribution with Amazon EMR now uses Hive 2.3.2. |
|
15 | Datameer X distribution: Amazon EMR versions should use patched Hive/ Hadoop jars if available | DAP-40891 |
|---|---|---|
Datameer X uses patched Hive and Hadoop jars for compatibly reasons. |
|