10.0.x Release Notes

INFO

Within these release notes Datameer X provides information and resolutions on improvements, bug fixes, and visual changes for 10.0.x maintenance release versions.

Visit Datameer's Help Center for additional information or contact your customer service representative.

Datameer X 10.0.7

New Features

1	Supported Hadoop Distributions: Add Support for Cloudera CDP / Cloudera Runtime 7.0(.3)	DAP-39597
	Cloudera Runtime 7.0(.3) is now supported.

Improvements

1	Connection: Step 'Save' in connection edit mode - show a list of impacted artifacts when the authentication mode is set to 'SESSION_BASED'	DAP-39087
	A list of the impacted artifacts is now displayed when the authentication mode is set to 'SESSION_BASED' in the section 'Save' during connections.

2	Storybook: Enable new authentication dialog to request credentials	DAP-39088
	In the storybook a new React component is enabled which requests either user 'Name', 'Password' or 'SSH Key' depending on the used connection.

3	Supported Hadoop Distributions: Add support for Hortonworks HDP 3.1.5	DAP-39819
	Datameer X now supports Hortonworks HDP 3.1.5.

4	Tableau Export Connection: Parallel TDE/ Hyper file generation for Tableau Server exports	DAP-39846
	Setting the property 'das.splitting.disable-combining=true' while configuring a Tableau connection runs the jobs significantly faster. It is verified that an export of a sheet to the Tableau server launches multiple tasks.

5	Supported Hadoop Distributions: Add support for Cloudera CDH 6.3.3	DAP-39933
	Datameer X now supports Cloudera CDH 6.3.3.

6

Import job/ DataLink Wizard: Request credentials while edit/ create

DAP-39935

The property 'enable.authenticationOnJobRun=true' activates the session base authentication feature.

If a connection is configured as 'SESSION_BASED' and the authentication is already entered, the wizard will not show the ask for the credentials group.

If a connection switches from 'PROVIDED/TEMPORARY' credentials to 'SESSION BASED' and edit/ save the import job, a check after saving in the database table 'job_configuration_property' the properties 'datastore.user_name' and 'datastore.password' must be deleted/ not exist for a session based connection import/datalink job. Furthermore a check after saving in the database table 'dap_job_configuration' column 'schedule_enabled' must be true (1) for 'PROVIDED' and 'TEMPORARY' and false (0) for 'SESSION BASED'.

7	Workbook: Display a "Save" button in toolbar	DAP-40047
	You can now click on a 'Save' button in the UI toolbar again to safe changes within a workbook.

8	Workbook: Workbook Saving - Default to working path	DAP-40055
	When closing a workbook for the first time, the workbook is saved in the folder it was created from and not in home folder upon exit. A workbook message does not read "saved under home folder" but using the actual path where the workbook is stored. When closing the File Browser the workbook is saved in the folder it was created from.

9	Supported Hadoop Distributions: Add HBase 2 support for Cloudera CDH 6 versions	DAP-40109
	Datameer X now supports HBase 2 support for Cloudera CDH 6 versions.

10	Tableau Plug-in: Tableau Plug-in allow TDE/ Hyper exports to be setup from non-Linux systems	DAP-40147
	The plug-in for Tableau now allows TDE/ Hyper exports to be setup from non-Linux systems.

Bug fixes

1	Column Inspector: 'Undefined' shown as a formula string in case of empty columns	DAP-39529
	Empty columns do not show formulas in the column inspector any more.	Bug fixed.

2	Tableau Export: Tableau ExportJobs performance issue	DAP-40034
	The property 'das.splitting.disable-combining=true' while configuring a Tableau connection runs the jobs significantly faster and multiple tasks can be launched.	Bug fixed.

3	Workbook: Duplicating or copy/ pasting a workbook returns an error	DAP-40067
	Workbooks can be copied and pasted without an error again.	Bug fixed.

4	Workbook: 'EXPAND' function column position shows incorrect result for 'GROUPROWNUMBER'	DAP-40087
	Using the 'EXPAND(LIST)' function shows the correct sequence for 'GROUPROWNUMBER'.	Bug fixed.

5	Display key columns first flag causes 'IllegalStateException' in a 'Join Sheet'	DAP-40121
	Displaying the key columns first flag doesn't cause exceptions in a 'Join Sheet' any more.	Bug fixed.

6	Workbook: Can't rename a column in the 'Join Sheet' because of a 'RegularRequirementNotMetFailure'	DAP-40130
	Column renaming in a 'Joint Sheet' is possible because the broken references check implemented for transformed sheets doesn't affect renaming at other sheet types.	Bug fixed.

7	Upgrading Datameer X 7.5.x to 10.1.2 workbook upgrade conversion breakage	DAP-40140
	Column changes are propagated to all depended workbooks and sheet, not only when upgrading.	Bug fixed.

8	Avro Import: 'NoSuchMethodError' with Cloudera CDH-5.14.4 (Hadoop distribution jackson jar binary incompatible)	DAP-40142
	Importing Avro files works on Cloudera CDH-5.14.4 distributions.	Bug fixed.

9	Workbook: Datameer X OOM crash when viewing sorted sheet data	DAP-40151
	The new property 'das.data-download.merge-sort.max-total-file-size' is introduced to control (volume based) if the sorting will happen in full data view or not. The default value is 256MB.	Bug fixed.

10	Datameer X Mode: Prevent mixed content issues if Datameer X is configured in SSL mode	DAP-40176
	The button "New System" within 'External Systems' is available for entitled users such as an Admin in a SSL enabled installation, too.	Bug fixed.

Datameer X 10.0.6

New Feature

1	Add REST API call to look up what a UUID's properties are	DAP-31229
	For a UUID Datameer X provides the JSON containing the information about the artifact analog to a REST READ call for an artifact ID.

Improvements

1	Connection: In wizard section 'details' validate and store authentication on a session if the authentication session-based option is selected	DAP-39334
	Authentication data is now validated and stored per session in the 'Connection Details' section within the connection wizard.

2	Connection: Show selected authentication mode on 'Connection Details'	DAP-39335
	Datameer X now shows the authentication mode on the 'Connection Details' page if the connection type provides such options (JDBC/MySQL, SSH, SFTP connections).

3	Connection: Implement a controller for session-based connection authentication	DAP-39936
	A new service checks the session id and connection id for session-based authentication connections.

4	Workbook: Enable flag for quick fix 'Workbook preview disappears in about 24 hours after execution'	DAP-39928
	The flag 'workbook-preview-service.has-new-data-check' is now added in the default properties and set 'true' per default.

Bug fixes

1	Workbook: Inconsistency in error messages for workbooks without a shared source	DAP-38108
	Error messages are adjusted and are now consistent for errors when working with a shared workbook without a shared source.	Bug fixed.

2	MySQL 8.x execute mysql-init.sql fails - no Grant options allowed	DAP-39867
	Datameer X has two new MySQL scripts for running MySQL 5.x and MySQL 8.x databases.	Bug fixed.

3	Security: After a failed login attempt, Datameer X exposes the LDAP query to fetch users	DAP-39953
	The error message is shown again as expected and does not expose the LDAP query to fetch users anymore.	Bug fixed.

4	Google Cloud Storage is bundled with the wrong Hadoop library	DAP-39979
	The Google Cloud Storage connector is now bundled with all Hadoop 3 versions.	Bug fixed.

5	Infographics: DataSetsController REST API copies full workbook or import job data to DFS (EhCache), therefore the conductor disk space is filling up	DAP-39990
	The DFSCache button on the developer's page is working again and data is now cleaned up as expected.	Bug fixed.

6	DevController: Cache Monitor 'Clear Cache' doesn't purge data	DAP-40005
	The DFS cache is now cleared when clicking the "Clear Cache" button.	Bug fixed.

7	Workbook: Filtering on a column with the 'DENULLIFY' function causes an 'IllegalArgumentException'	DAP-40017
	The Datameer X function 'DENULLIFY' works as expected again.	Bug fixed.

8	Workbook: Show dependencies is very slow and keeps loading	DAP-40023
	Due to performance, the polling interval while working with the file browser's dependencies view is now increased to 100 sec.	Bug fixed.

9	Workbook: Typo in backup and restore error message	DAP-40046
	The error messages for duplicating and copying/pasting a shared workbook without sharing the data source is now correct.	Bug fixed.

Datameer X 10.0.5

Improvements

1	Add HBase 2.0 support for HDP 3.1.0 and HDP 3.1.4	DAP-39823
	Datameer X now supports importing HBase data for HDP 3.1.0 and 3.1.4 based distributions Both are shipped with HBase 2.0 support.

2	Add Support for Amazon EMR v5.28.1	DAP-39877
	Datameer X now supports Amazon EMR 5.28.1.

3	Add Support for Amazon EMR v5.29.0	DAP-39878
	Datameer X now supports Amazon EMR 5.29.0.

4	Remove 'superuser' related properties from 'live.properties'	DAP-39910
	The 'das.superuser.*' properties are now listed in the default properties only (disabled per default).

Bug fixes

1	Property API: Key-up event for property info input fields of property API not working	DAP-39015
	A key-up event now gets recognized as a proper event and triggers the same actions as a general change event for input fields.	Bug fixed.

2	Hive Export: Drop & Create new tables doesn't work for partitioned tables	DAP-39390
	Datameer X now validates Hive export definitions for partitioned tables and rejects those for 'Drop and create new table' as an unsupported operation.	Bug fixed.

3	Browser Search: Search results don't show correct results when changing the folder	DAP-39556
	Datameer X now updates the search results for folders if another folder from the file browser is selected.	Bug fixed.

4

Partition data table should be indexed for efficient group by queries

DAP-39662

To improve application performance and decrease load from Datameer's metadata database, indexes for 'data_partition.relative_path' and 'data_partition.index_key' have been added to the metadata database.

Upgrade scripts persist the changes in Datameer's metadata database accordingly.

Bug fixed.

5	Hive 3.1: Can not create ZooKeeper Hive JDBC connection	DAP-39947
	Hive JDBC connection with ZooKeeper can now be created and used for imports.	Bug fixed.

Datameer X 10.0.4

Improvements

1	Workbook: SQL Editor should provide hint for all keyword types	DAP-39583
	The autocomplete list now provides a hint if the selected keyword is a SQL statement, SQL function or SQL data type.

Bug fixes

1	Hive datalink to partitioned table with defer sampling shows an empty preview in the workbook	DAP-38816
	Including partition columns when editing a datalink shows a preview in a workbook.	Bug fixed.

2	Cannot export Runtime Analytics sheet to Hive	DAP-39472
	Runtime Analytic sheets can now be exported to Hive.	Bug fixed.

3	File Browser: Subfolders inside user's home folder also have home folder icon	DAP-39524
	Only the user's home folder has the home icon.	Bug fixed.

4	LDAP users with more than 1,500 members in their group are not recognized	DAP-39680
	All users within a group of more than 1,500 members can now be authenticated.	Bug fixed.

5	Workbook: 'Sheet has no column' error on filtered, sorted, encoding or pivot sheets when renaming column in a source sheet	DAP-39726
	Renaming a column does not break the referenced sheet anymore. This affects all transform and data mining sheets.	Bug fixed.

6	'Server error' message in File Browser when selecting the user's home folder	DAP-39767
	The aforementioned error message isn't rendered anymore.	Bug fixed.

7	Import or export job run details page should not display 'Production Mode' information	DAP-39786
	Import and export job run details pages don't render the 'Production Mode' information, as this is not relevant for these types.	Bug fixed.

8	Changing a formula will add an additional equals sign every time to the formula string	DAP-39830
	Changing a formula doesn't add additional characters, like an equals (=) sign, anymore.	Bug fixed.

9	Plug-in Hive-JDBC: Include correct Hive JARs for EMR distributions	DAP-39843
	The correct httpclient and httpcore JAR dependencies are now packaged with the plugin hive-jdbc for EMR distributions.	Bug fixed.

Datameer X 10.0.3

Improvements

1	Add support for Hortonworks HDP 3.1.4	DAP-39350
	Datameer X now supports Hortonworks HDP 3.1.4.

Bug fixes

1	Hive 2 Plug-in: Show an error when list of databases or tables fails or is empty	DAP-38408
	Datameer X shows a corresponding error message in the user interface if tables are not available from the Hive Metastore database or cannot be derived.	Bug fixed.

2	Admin Page → Job Details Page: 'Show Logfile' shows log in the same window	DAP-39520
	Clicking the 'Show Logfile' button now shows the log file in a new tab.	Bug fixed.

3	Open Data Format: Can't delete sync table for one sheet and set table with the same name for another sheet	DAP-39525
	Removing and adding a synchronization for another sheet with the same name is now possible in one step.	Bug fixed.

4	Workbook: Adding more than 5,000 characters string in the formula is always trimmed to 5,000 which causes the formula editor to highlight as error	DAP-39542
	Datameer X now returns the non-trimmed value for the raw formula to be displayed in the formula editor.	Bug fixed.

5	Visual Explorer: Opening the Visual Explorer in a workbook which contains only a formula sheet leads to an error	DAP-39621
	Visual Explorer now opens without an error.	Bug fixed.

6	'Migrating from one Hadoop cluster to another' link on admin Hadoop cluster page points to wrong version space	DAP-39734
	The link now points to the correct version of the Datameer X product documentation.	Bug fixed.

7	Google Data Proc: (Un)secure hdfs tool doesn't work	DAP-39757
	The 'unsecure_hdfs_tool.sh' and 'secure_hdfs_tool.sh' scripts now work with Google Data Proc environments as well as with non-Google distributions.	Bug fixed.

8	WorkbookHealthCheck fails with java.lang.StackOverflowError exception	DAP-39805
	Datameer's WorkbookHealthCheck can now run for a large number of workbooks in an environment (e.g. 15,000 workbooks or more).	Bug fixed.

Datameer X 10.0.2

Improvements

1	Tableau Hyper Export: correct libstdc++ library version loading	DAP-39115
	Datameer X now injects the path to Tableau's native libraries into LD_LIBRARY_PATH for Tez environments when Tableau Server Datastores are used. This can be controlled via the custom property: das.tableau.hyper.native-library-path-injection={true \| false}

2	FileSplitter & InternalDatameerFileSplitter should use Hadoop FileSystem#listLocatedFileStatus for performance reasons	DAP-39516
	Datameer X now shows improved performance when large numbers of files get imported via an import job or Datalink.

3	Add support for Cloudera CDH 6.3.2	DAP-39657
	Datameer X now supports Cloudera CDH 6.3.2.

4	Add support for Amazon EMR 5.28.0	DAP-39695
	Datameer X now supports Amazon EMR 5.28.0.

Bug fixes

1	Workbook: Context menu doesn't popup on double click on a value sometimes	DAP-38301
	The context menu is opened when a workbook record entry gets double-clicked.	Bug fixed.

2	JDBC connection fails due to hardcoded unicode property	DAP-38958
	The definition of 'useUnicode', 'characterEncoding' and 'charSet' as optional parameters for JDBC based connections is now allowed. The checkbox 'Enable Character Encoding' is now visible in the UI.	Bug fixed

3	Admin page: Plug-ins can not be paused or disabled in Firefox version 69.0	DAP-39398
	A plug-in now can be paused, deleted and restarted again when using Firefox version 69.0 as web browser.	Bug fixed

4	Workbook: Buttons are enabled in File Browser Dialog when they shouldn't	DAP-39495
	Datameer X now ensures that the 'Add Data' button, 'Exchange Data' button and 'Open Workbook' are enabled in the correct context only.	Bug fixed

5	Workbook: When the 'Add Data' dialog is shown for a new workbook, it is not modal	DAP-39510
	The 'Add Data' dialog is now modal.	Bug fixed

6	Incorrect Daylight Saving Time (DST) related date value conversion during Hive import	DAP-39569
	Datameer X now properly handles DST related conversions while importing data values from Hive.	Bug fixed

7	HDFS BlockLocations topology paths lost when converting BlockLocation into FileBlockLocation	DAP-39581
	TopologyAwareAlgorithm now can optimize split combine operations depending on the data locality.	Bug fixed

8	Workbook: Renaming the source column of an Encoding sheet leads to an error if submitted by sheet switch	DAP-39622
	Renaming of a workbook column is handled when doing a sheet switch.	Bug fixed

9	Workbook: Jump to Column does not work correctly	DAP-39655
	The correct workbook column position is recognized when using 'Jump to Column' dialog.	Bug fixed

10	Hardcoded Docu links should point to docu space configured in default.properties ('docu.url' )	DAP-39693
	Documentation links within Datameer X now point to the correct page within the Datameer X user documentation.	Bug fixed

Datameer X 10.0.1

Improvements

1	Remember previous column position when switching sheets in a workbook	DAP-22392
	After switching sheets or opening other workbooks, Datameer X now switches back to the workbook sheet, row and column you were previously on.

2	File browser: show clickable links in dependency graph	DAP-38886
	When a node representing a Datameer X artifact is clicked in the dependency graph the file browser navigates to the folder in which the artifact is stored.

3	Improve record encoding for Datameer's Big Integer and Big Decimal to be Hive compatible	DAP-39367
	Newly written and synced Parquet files are now compatible with and readable by Hive.

4	Housekeeping Service: encapsulate each Housekeeping task in its own exception handling.	DAP-39506
	Each Housekeeping task now encapsulates its own exception handling.

5	Optimize multimap value removal code in CombineOnlyDataLocalAlgorithm	DAP-39565
	Combining splits performance has been improved especially for a large number of splits to combine.

Bug fixes

1	Scheduled jobs remain indefinitely in the processing queue if the Kerberos password is changed	DAP-39474
	A scheduled job occurring between a Kerberos password change no longer blocks the processing queue.	Bug fixed

2	Hot deployment of a plug-in fails if the plug-in contains a DataStoreType implementation	DAP-38885
	A plug-in containing a DataStoreType can now be installed and uninstalled in the 'Admin' tab section, (e.g. Hive plug-in.)	Bug fixed

3	Deploy via Cloud Formation Template in AWS: Injected Examples aren't working	DAP-39160
	Injected Examples are now backed by data sources in S3 and work for deployments on AWS EC2 instances as well.	Bug fixed

4	Housekeeping performance degradation	DAP-39283
	Improved robustness and traceability for housekeeping service allowing for the configuration of the number of HDFS artifacts to delete.	Bug fixed

5	REST API to download results doesn't return all records by setting 'rest.download-data.records-max=0'	DAP-39386
	Datameer X respects the setting 'rest.download-data.records-max=0' and provides all records for the download.	Bug fixed

6	Hive 2 plug-in: ImportJobs and Datalinks are failing on Hadoop Namenode and Resourcemanager high availability environments	DAP-39397
	Hive 2 plug-in based Importjobs and Datalinks now runs successfully on high availability environments.	Bug fixed

7	Workbook config: false errors on correct CRON pattern	DAP-39458
	Datameer X now recognizes a valid CRON pattern and does not throw an error on a valid pattern.	Bug fixed

8	Visual Explorer should dismiss dialogs in order on 'Escape'	DAP-39504
	Pressing 'Escape' dismisses the respective front dialog (e.g. x-y-axes definition dialog instead of closing the whole Visual Explorer dialog). (CT- "Escape" use double quotes ")	Bug fixed