Versions Compared

Version Old Version 24 New Version 25
Changes made by

Andreas Küster

Juliane Wetzel

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Info
titleINFO

Within these release notes Datameer X provides information and resolutions on improvements, bug fixes, and visual changes for 10.1.x maintenance release versions.  

Visit Datameer's Help Center for additional information or contact your customer service representative. 

DatameerX workbook
Section

Datameer X 10.1.

12.2Bug Fixes

13

New Features

1

Workbook toolbar buttons not functioning in Internet Explorer
Export: Amazon Redshift incremental exportDAP-
41864
DatameerX workbook toolbar works with Internet Explorer.

(tick) Bug fixed.

2

Workbook toolbar buttons not functioning in latest Chrome versionDAP-41836
35849

Incremental export to Amazon Redshift is now available when configuring an export. For that, select "Incremental" as the Record Replacement Policy and select the required column from the dataset.


Improvements

1

Supported Hadoop Distributions: Add support for Cloudera Private Cloud/ Cloudera Runtime 7.1.5.14DAP-41578

Datameer X now supports Cloudera Private Cloud/ Cloudera Runtime 7.1.5.14.


Bug Fixes

1

Workbook: Workbook toolbar buttons are not working when using them in the Internet ExplorerDAP-41864

The Datameer X Workbook toolbar now works with Internet Explorer.

(tick) Bug fixed.

2

Export: Amazon Redshift (Fast Load) export is dropping recordsDAP-41838

The data export into Amazon Redshift (Fast Load) is working correctly again.

(tick) Bug fixed.

3

Workbook: Workbook toolbar buttons are not working when using them in the latest Chrome versionDAP-41836

The Datameer X Workbook toolbar now works with Chrome version 89.0.4389.90 (Official Build) and newer.

(tick) Bug fixed.

4

Execution Engine: Datameer X on Amazon EMR/ Google Cloud Platform - Failed to activate the plug-in 'plugin-dwh'DAP-41816

Since 'plugin-doh' is a conductor-only plug-in, it is not being copied to the cluster anymore and does not impact any job execution.

(tick) Bug fixed.

5

Import/ Export: Snowflake - Tables are not updating when selecting another schemaDAP-41721

When configuring a Snowflake export or import job, the tables are now updated when changing the schema from the drop-down.

(tick) Bug fixed.

6

User Interface Login: Intermittently login page is not refreshing after a successful loginDAP-41633

The correct login is working in all browsers for SAML, internal authentication as well as for external authentication (LDAP).

(tick) Bug fixed.

Section

Datameer X 10.1.12.2

Bug Fixes

1

Workbook: Workbook toolbar buttons are not working when using them in the Internet ExplorerDAP-41864

The Datameer X Workbook toolbar now works with Internet Explorer.

(tick) Bug fixed.

2

Workbook: Workbook toolbar buttons are not working when using them in the latest Chrome versionDAP-41836

The Datameer X Workbook toolbar now works with Chrome version 89.0.4389.90 (Official Build) and newer.

(tick) Bug fixed.

Section

Datameer X 10.1.12

New Features

1

Supported Hadoop Distributions: Add support for Amazon EMR 6.2.0DAP-41575

Datameer X now supports Amazon EMR 6.2.0.


2Supported Hadoop Distributions: Add support for Amazon EMR 5.32.0DAP-41574

Datameer X now supports Amazon EMR 5.32.0.


Improvements

1

Workbook: Auto-expand column header during "column edit" if the column name is partly invisibleDAP-41677

When renaming a column, the column width is now expanded fully visible without any truncation.


2

Supported Hadoop Distributions: Enable Apache Hadoop 2.8.5 supportDAP-41451

Datameer X now supports Apache Hadoop 2.8.5, also with Hive 2.3.8 as well as HBase 1.4.13.



Bug Fixes

1

Export: Snowflake - Export jobs failing with exceptionDAP-41557

Snowflake export jobs run successfully again after fixing the plug-in.

(tick) Bug fixed.

2

File Browser: Dependencies View artifact loses its color after selecting/ deselectingDAP-41322

In the Dependencies View a selected and then deselected artifact does not change the color to grey anymore.

(tick) Bug fixed.

...

Section

Datameer X 10.1.11

New Features

1

Datameer Service: New Event Logging Plug-inDAP-41323

The new Event Logging plug-in allows to log an artifact creation or deletion event. Certain types can be enabled and disabled for logging them. If an event type is not checked, no entry will be logged if the action happens in Datameer X. The log file is then downloadable with the System and Usage Report.



Improvements

1

Workbook: Auto-expand column header during "Column edit" if the column name is partly invisibleDAP-41677

When renaming a column, the column width is now expanded fully visible without any truncation.


2

Supported Hadoop Distributions: Add Support for Cloudera CDP-DC/ Cloudera Runtime 7.1.4.1DAP-41067

Datameer X now supports Cloudera CDP-DC/ Cloudera Runtime 7.1.4.1.



Bug fixes

1

Export: Snowflake - Invalid 'DATE' type exportDAP-41329

The Datameer 'DATE' values are displayed properly as Snowflake's 'TIMESTAMP' values or other data-like data types when exporting.

(tick) Bug fixed.

2

Performance: JDBC-based Data Link very large sample jobs do not limit the records, read and run too long on large tablesDAP-41334

Due to performance improvements, JDBC-based Data Link sample jobs finish faster and collect 5,000 sample records by default.

(tick) Bug fixed.

3

Service: Conductor.sh Check triggers tmp/ directory clean upDAP-41336

The tmp/ directory clean up is not triggered while Datameer X is running any more. The Conductor.sh Check command does not trigger the tmp/ directory clean up.

(tick) Bug fixed.

4

Workbook: Pivot Sheet throws and NullPointerException in case its source sheet has no recordsDAP-41620

When a source of a Pivot sheet has no records, an error is now displayed in the Pivot sheet that there is no data in the source sheet.

(tick) Bug fixed.

5Connections: Amazon S3 - IAM import/ export failureDAP-41667

After replacing the jar files the import and export are working as expected again.

(tick) Bug fixed.

6

File Browser: Dependencies View artifact loses its color after selecting/ deselectingDAP-41322

In the Dependencies View a selected and then deselected artifact does not change the color to grey anymore.

(tick) Bug fixed.

7

Workbook: SQL sheets - TRIM function in WHERE clauseDAP-41380

After applying the LIKE clause the TRIM function returns the required records accordingly.

(tick) Bug fixed.

Section

Datameer X 10.1.10

New Features

1

Import/Export: Google BigQuery - Add partition filtering functionalityDAP-41202

It is now possible to import from a partitioned table.


2Workbook: Custom functions for dictionary matchingDAP-41237

Datameer X has now two new functions for dictionary matching: "MATCHING_DICTIONARY" and "MATCHING_DICTIONARY_FUZZY".
3Import/ Export: S3 connection - Add the IAM role ARN optionDAP-41298

There is a new authentication option for the S3 connection: You can now authenticate via the IAM role with an assumed role.


Improvements

1

Plug-Ins: plugin-hive - do not overwrite the fetch size and use the value defined in the jdbc connection stringDAP-41130

The fetch size is new increased up to 5,000 to boost the jdbc-based import jobs.


2

Authentication: Terminate all user sessions on logoutDAP-41159

Datameer X presents a new property 'devicesconductor.session.management.enabled'. If it is set to true, a second login with the same user will be prohibited. The property is disabled by default.


3

Admin UI: Database driver tab - wide grid or horizontal scrollbarDAP-41197

The UI has now more screen space for the grid, showing the installed JDBC drivers. Resizing the File Browser window, the width of the table is being adjusted.


4

Export: Hive - Improve the help text for the replacement strategy during the Hive exportDAP-41223

The user now sees a more specific help text for the replacement strategy for a Hive export.



Bug fixes

1

File Browser: Disable the folder context menu 'Rename' in the Folder TreeDAP-40713

If no folder is selected within the Folder Tree the 'Rename' option from the context menu will be disabled.

(tick) Bug fixed.

2

File Browser: Wrong documentation link for the file searchDAP-40889

When searching for files in the File Browser the help link now points to the correct corresponding documentation page.

(tick) Bug fixed.

3

Workbook: Wrong color highlighting of formula parameter in the Formula EditorDAP-40933

Formula highlighting works as expected again.

(tick) Bug fixed.

4

Workbook: Scheduling information is shown on a Workbook Details page even when the scheduling is disabledDAP-41097

The scheduling information from the Workbook Details page is now up-to-date after changing the scheduling manually.

(tick) Bug fixed.

5Datameer Distribution: Jars missing in installation package for CDH-7.x.xDAP-41180

The installation package for Cloudera CDH-7.x.x distributions is now complete.

(tick) Bug fixed.

6

Connection: 'Save Connection' dialog - 'Close' is visible under the 'close' iconDAP-41205

The string 'close' within the 'Save Connection' dialog does not appear anymore.

(tick) Bug fixed.

7

Workbook: Post Upgrade Issue - NullPointerException in partition filter with $lastestpartitionDAP-41222

Datameer X will not throw a NullPointerException anymore if a user tries to update the partition filter in a Workbook.

(tick) Bug fixed.

8

REST API: Concurrent session management blocks REST API to workDAP-41242

The concurrent session feature still works when enabled. The REST API works independently from enabling or disabling the concurrent session feature.

(tick) Bug fixed.

9File Browser: Dependencies View only opens after the second attemptDAP-41291

Clicking the 'Dependencies View' icon now opens the Dependencies View at the first try.(tick) Bug fixed.
10Workbook: Attempt to duplicate the Workbook leads to broken references in the new artifactDAP-41297

Duplicating or copying an artifact does not lead to broken references in the new artifact anymore.(tick) Bug fixed.



Section

Datameer X 10.1.9

Improvements

1

Workbook Health Check: Log status and progress into separate logs file 'logs/workbook_health.log'DAP-41120

The Workbook Health Check logs are now being written into the file 'logs/workbook_health_check.log'.


2

Supported Hadoop Distributions: Add support for Cloudera CDH 6.3.4DAP-41138

Datameer X now supports Cloudera CDH 6.3.4.


3

Plug-ins: 'Plugin-dwh' - Make Data Warehouse plug-in default for Datameer distributionDAP-41150

The plugin 'plugin-dwh' is now available as a default plugin and comes automatically with all Datameer X distributions.



Bug Fixes

1

File Browser: After canceling 'Create new folder', the File Browser does not update when selecting another folderDAP-40673

After canceling 'Create new folder', it is possible to change the folder in the Folder Tree panel.

(tick) Bug fixed.

2

Pivot Sheet: Creating a Pivot sheet with a broken Workbook source ends up in an error dialogDAP-40729

Creating a Pivot sheet out of a broken Workbook is not possible since the error state of a broken Workbook is shown first before one can create a sheet out of a broken Workbook.


(tick) Bug fixed.

3

Workbook Details: The table for SheetDependent is missing/ not displayedDAP-40821

The table for 'SheetDependent' ("Consumed by x artifacts") is visible on the Workbook Details page.

(tick) Bug fixed.

4

File Browser: Prevent the option to open the 'Job run details' page if the necessary capability is missingDAP-41129

If a user does not have the capability “Common job details“ he will not enter the job-execution details page and an error message is shown instead.

(tick) Bug fixed.

5

Import: Clickhouse - Ingest of empty tables is not workingDAP-41140

Ingesting empty tables now works properly when pointing to an empty table in a Clickhouse database.

(tick) Bug fixed.



Section

Datameer X 10.1.8

New Features

1

Import/ Export: Enable Import to ClickHouse via a JDBC connectionDAP-40951

Importing data via a JDBC connection from a ClickHouse database is now possible. You need to have the ClickHouse database driver installed and then configure the JDBC connection to execute imports and exports.


2Supported Hadoop Distributions: Add support for Amazon EMR 5.31.0DAP-41025

Datameer now supports Amazon EMR version 5.31.0.
3File Browser: Add new root folder in the Folder Tree panelDAP-41033

A user can now add a new top-level folder/ root folder in the Folder Tree panel in the File Browser. The newly created folder is named 'Untitled Folder' and can be renamed immediately after creating it. The capability 'CREATE_FOLDER' is needed.


Improvements

1

Workbook Health Check: Enable the download of the the report and print the configuration IDs of all broken WorkbooksDAP-40628

You can now download the Workbook Health Check as a JSON from the Workbook Health Check page. Furthermore the configuration IDs of all broken Workbooks are displayed as a comma-separated list unter the Workbook Health section.


2

Rebranding: Datameer Classic is now DatameerXDAP-40913

In course of the rebranding, The former 'Datameer Classic' is now called 'DatameerX'. The new Datameer products are called 'Spectrum' and 'Spotlight'. Find more information about all of our products on our website.


3

SFTP/ SCP import: Reduce the amount of the opened connections while multiple small file importDAP-41011

The new configuration property 'das.cluster.hadoop.fs.cache.scheme.whitelist' which is configurable globally and on job level, reduces the amount of SFTP/ SCP sessions. With the FileServer cache enabled for scp/sftp scheme, there will be a single SSH connection for each cluster task launched, limiting the "connection refused" occurances.


4

Authentication: SAML authentication - Disable the local account managementDAP-41058

The mixed-mode parameter that defines whether or not local authentication is available while using SAML (or LDAP), can now be configured within the SAML plug-in settings. Having it set to false, only the external authentication mode is available. Having it set to true, both external and local authentication mode are available.


5

Connection: Amazon S3 Native connector should validate the region and bucket combination properlyDAP-41074

When setting up an Amazon S3 Native connection the region is now validated against the bucket properly.


6

Housekeeping Service: Housekeeping Service should attempt to delete HDFS data artifacts for non-existent usersDAP-41090

Deleting artifacts that are owned by a removed Datameer user is now possible. Deleting can be executed a Datameer Service user that has direct access to the HDFS cluster.



Bug Fixes

1

User Interface: Scrolling problems when the browser is vertically maximizedDAP-40723

When the browser window is vertically maximized, the scroll bar works as expected and you can scroll up and down within a Workbook.

(tick) Bug fixed.

2

Sequential triggering: Do not display a complete list of all variables on the import/ export job run details pageDAP-41007

The list 'Variables' from the import/ export job run details page contains only the used variables.

(tick) Bug fixed.

3

File Browser: Unable to 'Duplicate Workbook', the dialog remembers the previous filename valueDAP-41020

Duplicating a Workbook works as expected. Executing the duplication creates a copy of the file and can be renamed as required. The default name of a duplicated Workbook is 'Copy_of_<initial Workbook name>'.

(tick) Bug fixed.

4

Connection: Google BigQuery export fails with staging bucket name that contains underscoresDAP-41045

It is now possible to have the Google Cloud Storage bucket containing underscores as the Google BigQuery connector staging bucket/ directory.

(tick) Bug fixed.

5

Login: After session timeout - error when first re-login attemptDAP-41054

A user is able to log in to the Datameer User Interface after a session timeout from the first attempt on.

(tick) Bug fixed.

6Security: Datameer Support Engineer Report extension allows reading random files from the Datameer application server OSDAP-41059

The Datameer extension Support Engineer Report reads files from the Datameer installation folder only (configuration as well as log files).

(tick) Bug fixed.

7

REST API: Path information disclosure when calling for a non-existent fileDAP-41060

When calling for a non-existent file a '404' error is shown containing the requested file name.

(tick) Bug fixed.

8

Security: 'productDASData' text/ javascript block is available at the page's source even when the 'About' menu is disabledDAP-41061

When the 'About Datameer' menu is disabled, the 'productDASData' text/ javascript block is deactivated as well.

(tick) Bug fixed.

9

Plug-ins: plugin-s3 (S3 Native connector) on EMR - Export to a bucket in different S3 region failsDAP-41073

The plug-in is fixed. An export via the S3 Native connection works regardless of a bucket region.

(tick) Bug fixed.

10Export: An export job should not inject 'DataStoreModel#setupConf' into the global 'JobConf'DAP-41088

An export job creates a dedicated 'JobConf' for export-only purposes.(tick) Bug fixed.
11Authentication: EMR- S3 connection uses IAM mode always when the cluster mode S3 is set to IAMDAP-41100

The Amazon S3 connection uses the configured authentication mode from the connection configuration: Access Secret/ Key.(tick) Bug fixed.



Section

Datameer X 10.1.7

New Features

1

Supported Hadoop Distributions: Add support for Amazon EMR 6.1.0DAP-40876

Datameer X now supports Amazon EMR version 6.1.0.



Improvements

1

Import/ Export: Support for Azure Synapse (formerly Microsoft SQL Data Warehouse)DAP-27739

You can now import from or export to an Azure Synapse Database. The according dialect is being supported as well.


2

Supported Hadoop Distributions: Add support for Amazon EMR 5.30.1DAP-40875

Datameer X now supports Amazon EMR version 5.30.1.


3

Supported Hadoop Distributions: Add support for Cloudera CDP-DC/ Cloudera Runtime 7.1.3DAP-40877

Datameer X now supports Cloudera CDP-DC/ Cloudera Runtime version 7.1.3.


4

Connection: Rewrite Google BigQuery import connector to use Google BigQuery storage APIDAP-40883

When importing from or exporting to Google BigQuery, the new Google BigQuery storage API is used for more efficient imports and exports..


5

Authentication: Remove 'gssapi-with-mic' from Datameer's 'default.properties'DAP-40799

The value 'gssapi-with-mic' is removed from the 'default.properties' key 'system.property.jsch.preferred.authentication'. The new added value is 'password,keyboard-interactive,publickey'.


6

Authentication: SAML - Add option to read roles only from repositoryDAP-40886

You can now configure the SAML authentication to read all roles from the repository. Furthermore all four assertion properties are configurable (username, email, groups, roles).


7

Supported Hadoop Distributions: Add support for Apache Hadoop 2.7.7 and Hive 2.3.7DAP-40895

Datameer X now supports Hadoop version 2.7.7 and Hive version 2.3.7.


8

Workbook: Add ability to 'update' instead of 'replace' for 'Cascade Permissions'DAP-40916

A user can now add permissions within 'Cascade Permissions' from a folder in the File Browser instead of fully replace the existing permissions.


9

Google BigQuery: Do not use temporary table when streaming from tablesDAP-40939

Imports from Google BigQuery now work without changes for tables, views and queries.


10

Plug-ins: JDBC/Snowflake - Unimplemented date/ time/ timestamp column splittingDAP-40952

The plug-in 'plugin-jdbc-v2' is now adapted to the column splitting needs.


11

Supported Hadoop Distributions: Add support for Cloudera CDP-DC/ Cloudera Runtime 7.1.2DAP-40998

Datameer X now supports Cloudera CDP-DC/ Cloudera Runtime version 7.1.2.


12

Supported Hadoop Distributions: Add support for Cloudera CDP-DC/ Cloudera Runtime 7.2.0.2DAP-40999

Datameer X now supports Cloudera CDP-DC/ Cloudera Runtime version 7.2.0.2.



Bug Fixes

1

Workbook: SQL - 'EXCEPT' and 'INTERSECT' operator null value handlingDAP-40739

The Datameer X functions 'EXCEPT' and 'INTERSECT' work properly again. The null value handling returns an empty result as expected.

(tick) Bug fixed.

2

Authentication: The default LDAP user filter does not filter out objects with username or email attribute set as an empty stringDAP-40884

The corresponding plugin is patched. User objects with and username or email attribute set to an empty string are now filtered out and do not break the LDAP cache creation.

(tick) Bug fixed.

3

Workbook: Validate for sheet error when changing column type which is used for an encoding sheetDAP-40892

The sheet error is being registered and a proper error message is shown when changing the column type which is used for an encoding sheet.

(tick) Bug fixed.

4

Workbook: Inaccurate status and missing dialog if triggering the Workbook from configuration pageDAP-40915

If triggering the Workbook from the Workbook Configuration page by the 'Starting calculation process immediately after save' option the process dialog is visible again and the status bar shows the accurate status.

(tick) Bug fixed.

5

Workbook: The key shortcut 'ALT + O' does not workDAP-40923

Executing the key shortcut 'ALT + O' opens the Workbook from the File Browser as expected.

(tick) Bug fixed.

6Connection: ZooKeeper - Datameer X does not close ZooKeeper connections (HBase)DAP-40970

The HBase plug-in was fixed so Datameer X now closes ZooKeeper connections to HBase again when the connection is saved.

(tick) Bug fixed.

7

Connection: MariaDB dialect is missing export transformation 'SqlDateTransformation'DAP-40989

The needed export transformation is now added. MariaDB is changed to apply the export transformation 'SqLDateTransformation'.

(tick) Bug fixed.

8

Plug-ins: JDBC 2/ Snowflake - Integer column splitting skips table rows for importDAP-40990

All splitted columns from type integer can be imported completely.

(tick) Bug fixed.

9

Workbook: Sequential triggering does not respect the execution orderDAP-41000

Sequential triggering executes the artifacts based on the execution order.

(tick) Bug fixed.



Section

Datameer X 10.1.6

New Features

1

Admin: Support for token authentication within the Tableau plug-inDAP-40342

You can now select the authentication mode 'SSO Token' when creating a connection to a Tableau Server.


2

Workbook: New call to insert usage of sheets into Workbook config and details pageDAP-40722

A new workbook query includes the usage of a sheet. The new call includes the information and displays it in the workbook configuration and workbook details page. Therefore a user knows if a sheet must be kept or not.



Improvements

1

Workbook Health Check: Provide the possibility to hide the loading screen and cancel the checkDAP-40607

You can now hide the Workbook Health Check loading screen and continue working while the check is executed in the background. Furthermore you can cancel the check completely.


2

Security: SAML authentication - role mapping proceedingDAP-40742

In the 'Admin' tab in the SAML authentication you can now assign a role and/ or a group to a user directly, e.g. 'User1: ANALYST:sales'. Note that the group must exist when assigning the group to the role.


3

Logging: New log file to store Datameer X PID (Process ID)DAP-40761

A new log file named 'start-pid.log' is introduced to log the Datameer X start time and the corresponding process ID. Whenever Datameer X starts, a new line is being added to the log file. Find the log file under '/logs/dir'.


4

Workbook: Search field and 'go to column' feature in the Workbook Sheet InspectorDAP-40338

A user can now search for a column in the Workbook Sheet Inspector and also go directly to the column by clicking it.


5

Workbook: Show each column in error state separately/ more prominentDAP-40370

Columns in an error state are highlighted in an error color in the columns list inside the Workbook Sheet Inspector. Clicking the column in error state highlights the column in the sheet. Furthermore the column list height is resizable now.



6

Workbook: No consumer indication in the workbook configuration dialog if a sheet is configured to be synced via Open Document FormatDAP-40818

The workbook configuration dialog now shows an indication that also synced sheets have a dependency consumed by. A workbook detail page contains truncated entries and a tooltip if the tables are longer than the rendered column width.


7

Security: Make SAML plug-in aware of a load balancer that offloads SSLDAP-40868

The load balancer offloads SSL to the Datameer X SAML plug-in. A new field is provided that enables - in addition to the existing Jetty-based SSL termination - the load balancer-based SSL offloading.



Bug Fixes

1

Export: Can't export strings values into MariaDBDAP-38661

New Hadoop configuration parameters, that may be set in the 'Admin' tab → 'Cluster Configuration' section or during the export job configuration, override the dialect's default varchar field length. The export to MariaDB is now possible.

(tick) Bug fixed.

2

Workbook: Error message isn't visible in the File Browser if trying to run an export-only workbook/ sheet without dataDAP-40440

The error message that an export job of a workbook or sheet without data is failing, is now displayed again.

(tick) Bug fixed.

3

File Browser: 'Dependencies graph' performanceDAP-40727

The performance for using the 'All Dependencies' and 'Depends On' function in the File Browser is now improved.

(tick) Bug fixed.

4

Workbook: Columns.unequalCount error in join sheetDAP-40753

An error is shown when submitting a join definition with duplicate keys also stating how to recover. Find more information here.

(tick) Bug fixed.

5

Workbook: IllegalArgumentException for join operation with multiple conditionsDAP-40754

Join operations with multiple conditions work again.

(tick) Bug fixed.

6

Connections: Databricks import connector throws an 'org.apache.spark.sql.AnalysisException'DAP-40764

The exception is fixed and the import works as expected.

(tick) Bug fixed.

7Admin: Non-admin user can change folder ownership and select other users with the same roleDAP-40767

If a user has no capability 'Change Folder Owner' but the capability 'User can modify every file and folder' he is not able to change the owner of the folder.

(tick) Bug fixed.

8

Import: Columns of UUID type are being dropped from a PostgreSQL importDAP-40794

The UUID type is now supported for the import and therefore the import job is executed successfully.

(tick) Bug fixed.

9

Workbook: Can't duplicate or copy a workbook if the sheet execution order does not match the view orderDAP-40805

Duplicating or copying a workbook works as expected and Datameer X follows the sheet logical order regardless of their current sequential position.

(tick) Bug fixed.

10

Database drivers: The MariaDB database driver has a wrong name in the JDBC connection string templateDAP-40855

The MariaDB database JDBC connection string now has the correct string template and you can establish a connection as expected.

(tick) Bug fixed.

11

User Interface: Workbook icons are disabled for the own workbookDAP-40871

After the code was patched, the icons for a workbook are available and not greyed out anymore in the user interface.

(tick) Bug fixed.



Section

Datameer X 10.1.5

New Features

1

Database Support: Create Azure Cosmos database dialectDAP-40590

You can now connect Datameer X against Azure Cosmos. Therefore you need to configure Azure Cosmos DB as a connector and install the corresponding JDBC driver.


2

Supported Hadoop Distributions: Add support for MapR Extension Package 6.3 with MapR 6.1DAP-40652

Datameer X now supports MapR Extension Package 6.3 with MapR 6.1.


Improvements

1

Documentation: Help links shouldn't be shown in the UI when the property pointing to the documentation is invalid or emptyDAP-40386

No links that are pointing to the Datameer X documentation are shown in the UI when the 'doc.url' property is not set. The 'Help' link in the context menu does not appear anymore.


2

Workbook: Add indicator in workbook configuration and workbook details page if a sheet is used as dependencyDAP-40444

There are new indicators for sheets within a workbook whose sheets are either a source data or for an export. The indicator appears on both the workbook configuration page next to each sheet, as well as in the workbook's detail page.


3

Admin: Workbook Health Check improvementsDAP-40479

You can now zoom in and out within the health check graph and navigate back to the original size. Furthermore you can view all broken workbooks in a expandable list within the workbook health report. The workbook IDs are clickable and point to the workbook node in the health graph. If a node is then selected it can be opened in the info panel.


4

Datameer X White Labeling: White labeling Datameer X for OEMDAP-40550

Several default properties were created to change the Datameer X UI and URL to have the Datameer X application white labeled. Furthermore the Datameer X distribution allows to change the small and big logo as well as favicons.


5

Performance: Optimize Browser Service WB#isSync calls for better performanceDAP-40565

The workbook sync call is optimized to show the last sync and current sync status in the File Browser.


6

Workbook: Implement a workbook dependency cycle detectorDAP-40706

A new workbook dependency cycle detector to identify workbook dependency cycles is implemented on the '/dev' page. After all workbooks are checked, a HTML report is accessible. A workbook is checked for being part of a workbook cycle.



7

Performance: Change task order in Housekeeping ServiceDAP-40711

A new default property 'housekeeping.run.task-attempts-per-run' is set to a value of '50' per default to decrease the performance of the Housekeeping Service significantly when it is to low.



Bug Fixes

1

FileBrowser: Could not delete a file upload after saving a copyDAP-39776

Deleting a file upload after saving a copy now works again.

(tick) Bug fixed.

2

Workbook: If a user has both "view" and "edit" permissions they should be able to rename a folderDAP-40544

A user which has the new capability "Rename Folder" and edit permissions to the parent folder, he can rename a child folder within a parent's folder. Having only the "Rename Folder" permissions but not the permissions to edit the parent folder, the user can rename the folders that are located in the parent's folder. A user of a folder can always rename the folder's default settings.

(tick) Bug fixed.

3

ImportJob: Change links in 'Import Job Details' view to actual buttonsDAP-40382

The button 'Browse Data' works as expected again.

(tick) Bug fixed.

4

Import/ Export: Can't connect to the SFTP server hidden behind a proxyDAP-40401

Datameer X is now able to connect to a SFTP server that is hidden behind a proxy with the following properties:

No Format
#proxy settings to connect a SFTP server hidden by the proxy
das.network.proxy.sftp.type=[HTTP]
das.network.proxy.sftp.host=<host ip>
das.network.proxy.sftp.port=<port>
das.network.proxy.sftp.username=<username>
das.network.proxy.sftp.password=<password>

(tick) Bug fixed.

5

Workbook: Double-click to submit in credentials dialog can lead to an errorDAP-40487

Adding a datasource to a session-based import job doesn't lead to an error any more.

(tick) Bug fixed.

6

Admin: Health Check report sometimes does not update correctlyDAP-40509

When clicking the "Start" button from the Workbook Health Check the indicator now shows the correct status of the Workbook Health Check.

(tick) Bug fixed.

7

Datameer X Upgrade: When upgrading from Datameer X 6.4.13 to 7.5.10 several workbooks stuck at 'Registering operations for sheet' phaseDAP-40534

When upgrading the workbooks don't stuck anymore because of patches in *.jar libraries, e.g. big workbooks with sophisticated logic.

(tick) Bug fixed.

8

Import Job: The partition dialog in 'Browse all Data' does not close when clicking "Select"DAP-40536

The dialog now closes again when clicking "Select".

(tick) Bug fixed.

9

FileBrowser: Browser list-files slow performance when a folder contains workbooks with data-driven schedulingDAP-40544

The backend query was optimized to minimize to response time of folders within the FileBrowser that contain artifacts with a long chain of upstream dependencies, e.g. workbooks that have 10 to 15 layers of parent artifacts.

(tick) Bug fixed.

10

Import Job: Fix intermittent failing test for email format validationDAP-40551

The test for the email validation now works as expected and detects not accepted email addresses, e.g. within an import job configuration.

(tick) Bug fixed.

11

Import: Azure Databricks import failsDAP-40604

An Azure Databricks import now works as expected.

(tick) Bug fixed.

12

Admin: Workbook Health Check is brokenDAP-40606

The Workbook Health Check from the Admin tab now works properly again.

(tick) Bug fixed.

13

Export/ Hive: Data is being written as 'TEXT' format when duplicating an export job and setting a new table and data typeDAP-40655

When duplicating a Hive export job and changing its settings, the changes are respected without any changes of data type, output format etc. in the new Hive table

(tick) Bug fixed.

14

Setup: Users without submitted license agreement can not operate DatameerDAP-40681

The license agreement flag 'force.license-agreement' is set to 'TRUE' and the internal user management is configured to allow newly created users to login and accept the terms of licence to operate Datameer.

(tick) Bug fixed.

15

Setup: C3p0 connection pool exhaustionDAP-40687

After disabling the statement caching no c3p0 warnings about pool problems appear anymore.

(tick) Bug fixed.

16

Performance: Housekeeping Service MySQL CPU drainDAP-40604

A new default property 'housekeeping.run.task-attempts-per-run' is set to a value of '50' per default to decrease the performance of the Housekeeping Service significantly when it is to low.

(tick) Bug fixed.

17

Workbook: Prevent Out of Memory errors when handling workbook circular dependenciesDAP-40694

Out of Memory errors are now prevented by changing buggy code.

(tick) Bug fixed.

18

Workbook: Prevent Stack Overflow errors when handling workbook circular dependenciesDAP-40696

Stack Overflow errors are now prevented by changing buggy code.

(tick) Bug fixed.

19

Performance: Dependencies graph performance is lowDAP-40727

The performance for using the 'All Dependencies' and 'Depends On' function in the File Browser is now improved.

(tick) Bug fixed.

...

Section

Datameer X 10.1.4.1

Improvements

1

Datameer X White Labeling: White labeling Datameer X for OEMDAP-40550

Several default properties were created to change the Datameer X UI and URL to have the Datameer X application white labeled. Furthermore the Datameer X distribution allows to change the small and big logo as well as favicons.


2

Datameer X White Labeling: Help links shouldn't be shown if a property pointing to the public documentation is invalid or emptyDAP-40386

Links to the Datameer X public documentation are not shown within the white labeled Datameer X UI by default. The links are only shown when the property 'doc url' is set. This is also valid for the 'Help' menu.


3

FileBrowser: Optimize BrowserService calls for better performanceDAP-40565

The BrowserService is now optimized not to access "WB#getSheets" and "WB#isSync".


Bug Fixes

1

Upgrading Datameer X After an upgrade several workbooks stuck at the "Registering operations for sheet" phase.DAP-40534

Datameer X workbooks don't get stuck anymore.

(tick) Bug fixed.

2

Performance: Browser list-files show a slow performance when the folder contains workbooks with data-driven schedulingDAP-40544

Browser list-files work with regular performance when a folder contains workbooks with data-driven scheduling.

(tick) Bug fixed.



Section

Datameer X 10.1.4

Improvements

1

Supported Hadoop Distributions: Add Support for Cloudera CDP-DC/ Cloudera Runtime 7.1.1DAP-40110

Datameer X now supports Cloudera Runtime version 7.1.1.


2

Workbook: Change 'SHIFTTIMEZONE' function arguments' nameDAP-40199

The description of the function arguments has been changed in the formula builder to prevent ambiguities. There is no behavior change of the functionality of the function itself.


3

Setup: Enable the 'DAS_USER' property in the 'etc/das-env.sh' file to reflect the assigned service accountDAP-40247

Adjusting this parameter is critical in saving the Admin from themselves by preventing an accidental startup of Datameer X without first assuming the proper service account. The property 'DAS_USER' must be enabled in the 'etc/das-env.sh' file.


4Workbook: Extend variable value lengthDAP-40265

Within a fresh Datameer X distribution, it is now possible to create variable values with more than 255 characters.
5Performance: Validate Datameer X license status once a day onlyDAP-40272

A Datameer X license now gets checked with the first user authentication or scheduled job run after midnight (local time) and is valid 24 h afterwards.
6Workbook: Validate cross joins for SQL sheetsDAP-40276

Datameer X validates queries of SQL sheets and notifies an analyst if he accidentally created a cross join in a 'FROM' section. Furthermore Datameer X allows the explicit usage of a 'cross join' syntax. The validation can be enabled by setting the 'sql-cross-join.enabled property' in the 'default.properties' configuration file. 


7Setup: Document MariaDB as a supported database dialect for Datameer's metadata databaseDAP-40293

MariaDB is now listed as a valid setting for the 'system.property.db.mode'.
8Supported Hadoop Distributions: Add Support for Amazon EMR v5.30.0DAP-40355

Datameer X now supports Amazon EMR v5.30.0.
9Supported Hadoop Distributions: Add Support for Amazon EMR v6.0.0DAP-40354

Datameer X now supports Amazon EMR v6.0.0.

Bug Fixes

1

Import/ Export: Add tooltips for long column names in Full Data and Export Wizard step 'Choose Sheet'DAP-38627

Datameer X now renders tooltips for long column names.

(tick) Bug fixed.

2

Import Job: Browse all data, and job details are opened in separate tabsDAP-39613

Datameer X stays in the same browser tab during navigation after running an import job.

(tick) Bug fixed.

3

Import Job: 'DelegateImportFormat' and 'SequenceFileType' create an empty configuration instead of using the provided oneDAP-39884

Datameer X now validates for properly configured 'io.serializations' settings for custom sequence file formats.

(tick) Bug fixed.

4

Import/ Export: Error during an ImportJob creationDAP-40148

While creating a BigQuery-based ImportJob in a Google BigQuery connection, no 'IllegalArgumentException' error is shown in the 'Data Details' step anymore.

(tick) Bug fixed.

5

Workbook: Can not rename a source column from a SQL joined sheetDAP-40158

Renaming a source column of a SQL sheet doesn't lead to an error any more.

(tick) Bug fixed.

6

Import: Can't import an AVRO file via HDFS HA ConnectionDAP-40172

Importing AVRO files from HA HDFS connections now works.

(tick) Bug fixed.

7

Setup: Datameer X doesn't clean up temporary data DAP-40242

Datameer X does a clean-up of its temporary folder during a start-up as long it is configured under Datameer's installation directory, but keeps it untouched otherwise.

(tick) Bug fixed.

8

HiveServer2 Plug-in: Enable support of mid-column schema changes of an underlying Hive table (AVRO & Parquet)DAP-40267

Datameer's Hive plug-in is supporting mid-column schema changes for Parquet and AVRO storage formats now.

(tick) Bug fixed.

9

File Browser: Details of an error dialog box opens a new browser tabDAP-40292

Error details are now rendered in the current browser tab.

(tick) Bug fixed.

10

Import Job: An import job is still referenced in a workbook after deleted the datasource in the workbookDAP-40321

When deleting a datasource, the current workbook holds no further references to it anymore.

(tick) Bug fixed.

11

Workbook: Workbook job run fails with an SQL sheetDAP-40327

The 'deploy.mode' is passed in for SQL sheets correctly now and the job run doesn't fail any longer.

(tick) Bug fixed.

12

Administration: The "User can see every file and folder" permission grants 'View Full Results' without an explicit mention.DAP-40366

Using this permission does not grant access to view data, only to see the files and folders. The ability to grant view full results became be a separate option to modify underneath.

(tick) Bug fixed.

13

Connection: Issue connecting to Amazon AthenaDAP-40372

The connection to Athena now works when in the custom properties the 'das.jdbc.import.transaction-isolation=TRANSACTION_NONE' property is set.

(tick) Bug fixed.

14

File Browser: Abort in 'Duplicate Folder' or 'Paste Folder' does not workDAP-40374

Clicking the 'Abort' button while duplicating or pasting a folder now cancels the operation.

(tick) Bug fixed.

15

REST API: REST API v1 vs v2 - inconsistent handling of workbook UUIDsDAP-40400

The REST API v2 will generate a new random UUID during POST (while a valid UUID is required in the payload, e.g. '00000000-0000-0000-0000-000000000000' can be used as a template).

(tick) Bug fixed.

16

Plug-ins: Errors are swallowed, making it look like the plug-in was successfully initializedDAP-40406

A plug-in will not get registered if there has been an error during plug-in initialization.

(tick) Bug fixed.

17

HiveServer2: A workbook that contains a datalink drops the records after a schema change of the underlying Parquet tableDAP-40448

Datameer X now imports all records from before and after a schema change.

(tick) Bug fixed.

18

Job Status Notification: Validation of the e-mail address according to RFC2822 (in detail: e-mail can contain "&")DAP-40453

Job status notifications for import, export and workbooks jobs can be send via e-mail addresses that contain chars like '&' from now on.

(tick) Bug fixed.

19

Support Engineer Report: The property 'SupportEngineerReportController.filterFilesLastNDays()' must cut at midnight from the system default timezone, not UTCDAP-40467

The 'Support Engineer Report' now respects the system default timezone.

(tick) Bug fixed.

20

Plug-in Tez: 'CONSUMED_BYTES/RECORDS' is incorrect in some casesDAP-40471

The plug-in now reports the counters for all (aliased) inputs in the dataset.

(tick) Bug fixed.

21

Plug-in Tez: A 'job-conf.xml' debug artifact should be written only once per Tez job, not for each vertexDAP-40475

The 'job-conf.xml' debug artifact is only written once for a job now, therefore rate limits like in GCP/ GCS don't occur.

(tick) Bug fixed.

...

Section

Datameer X 10.1.3.1

New Features

1

Supported Hadoop Distributions: Add Support for Cloudera CDP / Cloudera Runtime 7.0(.3)DAP-39597

Cloudera Runtime 7.0(.3) is now supported.


Improvements

1

Connection: Step 'Save' in connection edit mode - show a list of impacted artifacts when the authentication mode is set to 'SESSION_BASED'DAP-39087

A list of the impacted artifacts is now displayed when the authentication mode is set to 'SESSION_BASED' in the section 'Save' during connections.


2

Storybook: Enable new authentication dialog to request credentialsDAP-39088

In the storybook a new React component is enabled which requests either user 'Name', 'Password' or 'SSH Key' depending on the used connection.


3

Supported Hadoop Distributions: Add support for Hortonworks HDP 3.1.5DAP-39819

Datameer X now supports Hortonworks HDP 3.1.5.


4Tableau Export Connection: Parallel TDE/ Hyper file generation for Tableau Server exportsDAP-39846

Setting the property 'das.splitting.disable-combining=true' while configuring a Tableau connection runs the jobs significantly faster. It is verified that an export of a sheet to the Tableau server launches multiple tasks.
5Supported Hadoop Distributions: Add support for Cloudera CDH 6.3.3DAP-39933

Datameer X now supports Cloudera CDH 6.3.3.
6Import job/ DataLink Wizard: Request credentials while edit/ createDAP-39935

The property 'enable.authenticationOnJobRun=true' activates the session base authentication feature.

If a connection is configured as 'SESSION_BASED' and the authentication is already entered, the wizard will not show the ask for the credentials group.

If a connection switches from 'PROVIDED/TEMPORARY' credentials to 'SESSION BASED' and edit/ save the import job, a check after saving in the database table 'job_configuration_property' the properties 'datastore.user_name' and 'datastore.password' must be deleted/ not exist for a session based connection import/datalink job. Furthermore a check after saving in the database table 'dap_job_configuration' column 'schedule_enabled' must be true (1) for 'PROVIDED' and 'TEMPORARY' and false (0) for 'SESSION BASED'.


7Workbook: Display a "Save" button in toolbarDAP-40047

You can now click on a 'Save' button in the UI toolbar again to safe changes within a workbook.
8Workbook: Workbook Saving - Default to working pathDAP-40055

When closing a workbook for the first time, the workbook is saved in the folder it was created from and not in home folder upon exit. A workbook message does not read "saved under home folder" but using the actual path where the workbook is stored. When closing the File Browser the workbook is saved in the folder it was created from.
9Supported Hadoop Distributions: Add HBase 2 support for Cloudera CDH 6 versionsDAP-40109

Datameer X now supports HBase 2 support for Cloudera CDH 6 versions.
10Tableau Plug-in: Tableau Plug-in allow TDE/ Hyper exports to be setup from non-Linux systemsDAP-40147

The plug-in for Tableau now allows TDE/ Hyper exports to be setup from non-Linux systems.
11Workbook Settings: Disable scheduling if "request credentials on job trigger" condition is fulfilledDAP-39939

The scheduling menu is disabled if "request credentials on job trigger" conditions are fulfilled. A warning dialog is displayed why the 'Trigger' section is disabled. The REST v2 artifacts for job configurations now contain a new state if scheduling is enabled/ disabled.
12REST API: Enable back authentication for REST API call via session cookiesDAP-40012

Datameer X has two new security models for for REST API calls via session cookies which can be switched with the property 'enable.rest.csrf.checking=true/false'. The standard mode requires a CSRF token while the loose mode allows the authentication without a CSRF token.

Bug Fixes

1

Column Inspector: 'Undefined' shown as a formula string in case of empty columnsDAP-39529

Empty columns do not show formulas in the column inspector any more.

(tick) Bug fixed.

2

Tableau Export: Tableau ExportJobs performance issueDAP-40034

The property 'das.splitting.disable-combining=true' while configuring a Tableau connection runs the jobs significantly faster and multiple tasks can be launched.

(tick) Bug fixed.

3

Workbook: Duplicating or copy/ pasting a workbook returns an errorDAP-40067

Workbooks can be copied and pasted without an error again.

(tick) Bug fixed.

4Workbook: 'EXPAND' function column position shows incorrect result for 'GROUPROWNUMBER'DAP-40087

Using the 'EXPAND(LIST)' function shows the correct sequence for 'GROUPROWNUMBER'.(tick) Bug fixed.
5Display key columns first flag causes 'IllegalStateException' in a 'Join Sheet'DAP-40121

Displaying the key columns first flag doesn't cause exceptions in a 'Join Sheet' any more.

(tick) Bug fixed.
6Workbook: Can't rename a column in the 'Join Sheet' because of a 'RegularRequirementNotMetFailure'DAP-40130

Column renaming in a 'Joint Sheet' is possible because the broken references check implemented for transformed sheets doesn't affect renaming at other sheet types.(tick) Bug fixed.
7Upgrading Datameer X 7.5.x to 10.1.2 workbook upgrade conversion breakageDAP-40140

Column changes are propagated to all depended workbooks and sheet, not only when upgrading.(tick) Bug fixed.
8Avro Import: 'NoSuchMethodError' with Cloudera CDH-5.14.4 (Hadoop distribution jackson jar binary incompatible)DAP-40142

Importing Avro files works on Cloudera CDH-5.14.4 distributions.(tick) Bug fixed.
9Workbook: Datameer X OOM crash when viewing sorted sheet dataDAP-40151

The new property 'das.data-download.merge-sort.max-total-file-size' is introduced to control (volume based) if the sorting will happen in full data view or not. The default value is 256MB.(tick) Bug fixed.
10Datameer X Mode: Prevent mixed content issues if Datameer X is configured in SSL modeDAP-40176

The button "New System" within 'External Systems' is available for entitled users such as an Admin in a SSL enabled installation, too.(tick) Bug fixed.
11Upgrading Process: After upgrade the search index could be wrong or missingDAP-39528

After the upgrading process, the old search index is deleted and a new index created.(tick) Bug fixed.
12Workbook: Pivot Sheet - While creating/ editing a Pivot sheet the buttons 'Toggle Column' and 'Go To Line' are not disabledDAP-39770

When a Pivot sheet is created or an existing Pivot sheet is edited in a workbook, the button 'Toggle Column' is disabled.

When a Pivot sheet is created or edited in a workbook, the button 'Go To Line' is disabled.

On Pivot sheet while being in Pivot preview mode clicking on 'Split', 'De-Duplicate' or 'Encode Column', the toolbar buttons dismiss the Pivot review mode.

(tick) Bug fixed.
13Workbook: 'Show Dependencies' is very slow and keeps loadingDAP-40023

The 'Dependencies' are now displayed in an appropriate amount of time when clicking the icon.(tick) Bug fixed.
14Workbook: It is not possible to use a column in a join if the column was in an error stateDAP-40052

You can use a column in a join if the column was in an error state before and switch sheet was involved.(tick) Bug fixed.
15Datameer X Grid Mode: EMR - Clustername nor ID are validated while saving the grid mode.DAP-40100

While saving an EMR connection configuration, the clustername and ID are now validated.(tick) Bug fixed.
16Neebo to Datameer X Accessibility: Data Warehouse plug-in shouldn't restrict the access to file descriptors by 'HADOOP_CLUSTER_ACCESS' capabilityDAP-40201

Datameer X data is accessible via Neebo since a user needs 'IMPORTJOB_DOWNLOAD' and 'WORKBOOK_DOWNLOAD' capabilities are guaranteed with at least the role 'ANALYST'.(tick) Bug fixed.
17Export Job: Options for 'On a schedule' are missingDAP-40227

When creating an export job, for the schedule method "on a schedule" several options are available, e.g. hourly, monthly.(tick) Bug fixed.
Section

Datameer X 10.1.2

New Features

1

Add REST API call to look up what a UUID's properties are.DAP-31229

For a UUID Datameer X provides the JSON containing the information about the artifact analog to a REST READ call for an artifact ID.


Improvements

1

Connection: In wizard section 'details' validate and store authentication on a session if the authentication session-based option is selectedDAP-39334

Authentication data is now validated and stored per session in the 'Connection Details' section within the connection wizard.


2

Connection: Show selected authentication mode on 'Connection Details'DAP-39335

Datameer X now shows the authentication mode on the 'Connection Details' page if the connection type provides such options (JDBC/MySQL, SSH, SFTP connections).


3

Connection: Implement a controller for session-based connection authenticationDAP-39936

A new service checks the session id and connection id for session-based authentication connections.


Bug Fixes

1

Workbook: Inconsistency in error messages for workbooks without a shared sourceDAP-38108

Error messages are adjusted and are now consistent for errors when working with a shared workbook without a shared source.

(tick) Bug fixed.

2

Mysql 8.x execute mysql-init.sql fails - no Grant options allowedDAP-39867

Datameer X has two new MySQL scripts for running MySQL 5.x and MySQL 8.x databases.

(tick) Bug fixed.

3

Security: After a failed login attempt, Datameer X exposes the LDAP query to fetch usersDAP-39953

The error message is shown again as expected and does not expose the LDAP query to fetch users anymore.

(tick) Bug fixed.

4

Google Cloud Storage is bundled with the wrong Hadoop libraryDAP-39979

The Google Cloud Storage connector is now bundled with all Hadoop 3 versions.

(tick) Bug fixed.

5Infographics: DataSetsController REST API copies full workbook or import job data to DFS (EhCache), therefore the conductor disk space is filling upDAP-39990

The DFSCache button on the developer's page is working again and data is now cleaned up as expected.(tick) Bug fixed.
6DevController: Cache Monitor 'Clear Cache' doesn't purge dataDAP-40005

The DFS cache is now cleared when clicking the "Clear Cache" button.(tick) Bug fixed.
7Workbook: Filtering on a column with the 'DENULLIFY' function causes an 'IllegalArgumentException'DAP-40017

The Datameer X function 'DENULLIFY' works as expected again.(tick) Bug fixed.
8FileBrowser: Increase polling interval for thr file browser's dependency viewDAP-40038

Due to performance, the polling interval while working with the file browser's dependencies view is now increased to 100 sec.(tick) Bug fixed.
9Workbook: Typo in backup and restore error messageDAP-40046

The error messages for duplicating and copying/pasting a shared workbook without sharing the data source is now correct.(tick) Bug fixed.