Versions Compared

Version Old Version 1 New Version Current
Changes made by

Juliane Wetzel

Andreas Küster

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.


Info
titleINFO

Within these release notes Datameer X provides information and resolutions on improvements, bug fixes, and visual changes for 11.1.x maintenance release versions.  

Visit Datameer's Help Center for additional information or contact your customer service representative. 

Table of Contents

...

Datameer X 11.1.18

Improvements

...

Datameer X now supports Amazon EMR version 5.36.0.

...

Datameer X now supports Amazon EMR version 6.7.0.

...

Bug Fixes

...

The EMR cluster can now be set up in High Availability mode. 

...

(tick) Bug fixed.

...

When configuring the cluster mode in EMR, the default option for the region can now be saved.

...

(tick) Bug fixed.

...

Datameer X 11.1.17

Improvements

...

The correct user is now added in the session timeout logging in the 'user action.log' file.

...

After patching with an EMR plugin and setting the correct AWS region on the cluster settings page, Datameer can connect against the EMR cluster and run jobs successfully. 

Bug Fixes

...

The 'Task Attempt Logs' are also available if the folder name contains a '+'. 

...

(tick) Bug fixed.

...

After a hot fix, the export to the S3 bucket is now executable again.

...

(tick) Bug fixed.

...

Datameer X 11.1.16

Bug Fixes

...

The Data Link that points to S3 now succeeds when executing at the EMR cluster.

...

(tick) Bug fixed.

...

 It is now possible again to setup a HiveServer2 connection to Amazon EMR 5.34 and 5.35. 

...

(tick) Bug fixed.

...

Datameer X 11.1.15

Improvements

...

Datameer X now supports Amazon EMR version 5.35.0.

Bug Fixes

...

The JSON file can now be imported within the former estimated time duration after introducing the 'Nested array mode' during the import.

...

(tick) Bug fixed.

...

Datameer X 11.1.14

Improvements

...

Datameer X now supports Amazon EMR version 6.5.0.

...

 Changes for a user role configuration are now available in the event-logging.log file. 

...

Bug Fixes

...

The Inspector now displays the same information regardless on how an artifact is selected in both the File Browser or via the search bar.

...

(tick) Bug fixed.

...

When attempting to add a circular dependency column into a Workbook, the column is marked red and when hovering the column in error state the related error message is shown that this circular reference is not possible. 

...

(tick) Bug fixed.

...

Datameer X 11.1.13

Improvements

...

Datameer X now supports Amazon EMR version 5.34.0.

...

Cascade permissions can now be triggered via the REST API call. 

Bug Fixes

...

The Job graph now displays the details for the correct Workbook execution it belongs to.

...

(tick) Bug fixed.

...

Clicking the new "Update Preview" button now refreshes the preview, so the user can select/ deselect as many nodes as they want without having to wait for a UI refresh each time.

...

(tick) Bug fixed.

...

The bulk permissions change dialog only permits changing the owner if there is no associated import/ workbook job data/ history.

...

The documentation links for the following Workbook functions have been added to the function tab:

  • TIMESTAMPDIFF
  • CEILINGDATE
  • FLOORDATE

...

Datameer X 11.1.12

Improvements

...

The ADDTODATE function now takes a non-static value as a second argument with the new syntax: 

Code Block
ADDTODATE(<date>; <string/another column + date constant>

...

Bug Fixes

...

An active Datameer X user can always log in regardless of the permissions assigned to the role. 

...

(tick) Bug fixed.

...

After improving the HBase plugin, the connection as well as Import Jobs are working again.

...

(tick) Bug fixed.

...

Datameer X BIG_DECIMAL columns are not supported for Hive table mapping when a workbook sheet is synced via the External System feature to Hive. For that, the Hive table must not contain BIG_DECIMAL columns. 

...

Datameer X 11.1.11

Improvements

...

When creating a data link to a Google BigQuery connection, Datameer now only uses the amount of records that is needed to create the sample. For that, select "FIRST_N_RECORDS" in the 'Sampling Mode' section from the 'Scheduling' tab. This functionality is useful to reduce the cost of transferring data from the database to the cloud.

...

Datameer now provides the file path placeholder support for seconds (pattern: %sec%) as well as for milliseconds (%ms%) for file exports.

Bug Fixes

...

Variables are not highlighted in red in the Formula Builder unless there is a syntax error.

...

(tick) Bug fixed.

...

The event logging plug-in has been fixed regarding the log4shell functionality. Furthermore the RTA plug-in has been retired.

...

(tick) Bug fixed.

...

(tick) Bug fixed.

...

Datameer X 11.1.10

Improvements

...

Datameer X can now limit the number of selected records to the first n records of a table (n as the configured number of sample records).

...

Datameer X now supports Cloudera CDP/ Cloudera Runtime 7.1.7.

Bug Fixes

...

Datameer X now has added {nolookups} tag to log output patterns of log4j v2.

...

(tick) Bug fixed.

...

Datameer X does not drop space characters at the end of a string anymore while parsing it as a 'Fixed Width' object.

...

(tick) Bug fixed.

...

Plug-ins: BigQuery - unable to list tables if a table with "SNAPSHOT" type exists in the dataset

...

'Com.google.cloud' client libraries have been updated to version 1.32.2, therefore they can handle the "SNAPSHOT" type.

...

(tick) Bug fixed.

...

Datameer X does not use "Datameer" as substring during table creation anymore.

...

(tick) Bug fixed.

...

A PowerBI export now succeeds while executed as job executed on the cluster.

...

(tick) Bug fixed.

...

Datameer X runs HBase import jobs successfully in secured and unsecured cluster environments.

...

(tick) Bug fixed.

...

Export: Datameer database export jobs randomly fail because of HDFS permissions issue

...

Datameer export jobs now work reliable again in secured cluster environments.

...

(tick) Bug fixed.

...

Execution Engine: Job that processes broken a GZIP file never completes 

...

Datameer X now handles corrupted GZIP compressed files gracefully and terminates after importing all readable records.

...

(tick) Bug fixed.

...

Datameer X does not revert the FileBrowser's root folder permission during a restart.

...

(tick) Bug fixed.

...

Datameer X 11.1.9

Improvements

...

You can now implement properties for routing wasb/ wasbs traffic from the Datameer Application server and Data Nodes via a proxy server when connecting to the Azure Blob Storage.

...

Bug Fixes

...

The customer switched to Hadoop’s S3A protocol for the EMR grid mode & metadata reads/ writes, log streaming etc. The DatameerS3 protocol is still in use for imports/ exports due to backward compatibility and all various DatameerS3 & Amazon S3 connection settings (region, endpoint, IAM with assumed role mode, …).

...

(tick) Bug fixed.

...

Datameer X 11.1.8

Bug Fixes

...

(tick) Bug fixed.

...

The entire Folder Tree will not be reloaded anymore when an artifact has been deleted. Only the deleted artifacts and folders are not shown in the Folder Tree anymore.

...

(tick) Bug fixed.

...

The progress bar now reflects the current progress for the process. Errors from here are logged in the conductor.log file. 

...

(tick) Bug fixed.

...

Datameer X 11.1.7

New Features

...

1

...

We introduce a new capability for Datameer X users which can schedule jobs. This capability can be enabled/ disabled in the 'Admin' tab → 'Roles'. 

Improvements

...

Bug Fixes

...

(tick) Bug fixed.

...

Sorting artifacts in the File Browser works properly for all columns again and does not reset when scrolling. 

...

(tick) Bug fixed.

...

The documentation link for 'File Path and File Name Patterns' within the export job wizard now points to the correct documentation page. 

...

(tick) Bug fixed.

...

Datameer X 11.1.6

New Features

...

You can now use the new Workbook function 'NETWORKDAYS'. Using this function returns the number of the whole working days between a start date and an end date. Working days exclude weekend days and any dates that are identified as holidays. 

Bug Fixes

...

(tick) Bug fixed.

...

If a user tries to configure a Twitter Rest API and pressing the 'Authorize Datameer to retrieve data' button results in an error, now the actual error message is shown.

...

(tick) Bug fixed.

...

When closing an open Workbook, in which no changes were applied, the 'Close Workbook' dialog is not displayed at no time.

...

(tick) Bug fixed.

...

Datameer X 11.1.5

New Features

...

The Salesforce Connector now uses the Bulk API 2 to import data via a REST API. For running bigger datasets, custom properties for the import job are required, e.g. 'datameer.salexforce.maxrecord-per-result-request.  

...

Jobs can now be moved in the Job Scheduler queue. For that a new checkbox is included in the scheduling sections in Import/ Export/ Workbook Details wizard: 'Bypass Job Queue'. Jobs with that configuration will be picked always first by the Job Scheduler and ignore the maximal concurrent running jobs. A new role capability controls whether a user is allowed to use this option. 

...

Improvements

...

1

...

Datameer X now supports Amazon EMR version 6.3.0.

...

Bug Fixes

...

(tick) Bug fixed.

...

After fixing a swallowed SQL sheet and fixing some minor dependencies, the upgrade could be executed successfully.

...

(tick) Bug fixed.

...

When inserting a formula in the Workbook's 'Formula Builder' only the sheets and columns for the first reference and only the columns from the current sheet are offered for autocompletion. Furthermore, autocomplete filters out dummy columns, e.g. insert a formula in the second column, apply, and then go to the first column. 

...

(tick) Bug fixed.

...

Datameer X 11.1.4

Improvements

...

1

...

You can now cancel the 'Repoint Sheet' dialog after first applying a formula and then saving the sheet name. The applied formula is also being reset after cancelling the dialog.

...

Bug Fixes

...

After closing a Workbook, the associated Workbook tab is being closed  in the tab bar as well. 

...

(tick) Bug fixed.

...

All arrays of the JSON file are being imported. With the 'include root array' property, Datameer parses the whole array including the first object also. In general when importing JSON: If a JSON file contains a 'root array', means the first level if the data structure is a JSON array (file starts with [...]), the 'include array' checkbox has to be checked during the import wizard. If a JSON file contains an object in the first level and a nested data structure contains an array for the data ingest (file starts with {...[...]...}), the 'include root array' must not be checked. Then the JSON path to the 'target array' for the data ingest must be entered into the textbook during the import wizard. 

...

(tick) Bug fixed.

...

After updating the core libraries, the Azure Blob Storage connector is working as expected again. 

...

(tick) Bug fixed.

...

Datameer X 11.1.3

New Features

...

1

...


Info
titleINFO

Within these release notes Datameer X provides information and resolutions on improvements, bug fixes, and visual changes for 11.1.x maintenance release versions.  

Visit Datameer's Help Center for additional information or contact your customer service representative. 

Table of Contents


Section

Datameer X 11.2.14

Improvements

1Update 3rd party libraries known to have security vulnerabilities or performance issuesDAP-41554

Upgrade jasypt to 1.9.3 & delete unused hibernate-jasypt bridge code


2Remove Google Fusion tables from `plugin-webserviceDAP-42616

 Google Fusion Tables support has been removed from the code base as this is not available from Google anymore.

Bug Fixes

1Plug-in Tableau: failures with `ClassNotFoundException: com.fasterxml.jackson.jaxrs.base.ProviderBase` on Hadoop 3.3 based distributionsDAP-42615

Plug-in Tableau now can resolve com.fasterxml.jackson.jaxrs dependencies.

(tick) Bug fixed.


2REST endpoint `/rest/data/import-job/ ` throws NPE for datalinksDAP-42614

The REST endpoint now resolves datalinks.

(tick) Bug fixed.



Section

Datameer X 11.2.13

Improvements

1Update 3rd party libraries known to have security vulnerabilities or performance issuesDAP-42598

The following 3rd party libraries were updated to newer versions (or got removed):
  • DAP-42595: Update commons file-upload to version 1.5. Update commons-io to version 2.15.1.
  • DAP-42603: Replace log4j with reload4j
  • DAP-42604: Upgrade Jetty to version 9.4.53.v20231009
  • DAP-42608: Update aws-java-sdk to version 1.11.1034
  • DAP-42609: Update jersey-spring to version 2.41
  • DAP-42611: Remove commons-httpclient version 3.1


2Certify RedHat 8 with Datameer XDAP-42595

Successfully verified Datameer X can operate on RedHat 8 as operating system.


3Certify RedHat 9 with Datameer XDAP-42599

Successfully verified Datameer X can operate on RedHat 9 as operating system.


4Add support for Amazon AWS EMR 7.0.0DAP-42601

Datameer now supports Amazon AWS EMR version 7.0.0.


5Remove Datameer's ADLS Gen1 connectorDAP-42596

ADLS Gen1 will be discontinued by 2024.02.29. Datameer's connector got removed from the application therefore.

Bug Fixes

1Script "update_paths.sh" fails with HibernateException: "Unable to deserialize JSON due to PluginRegistry not being set"DAP-42594

The "update_paths.sh" script now updates the information in Datameer's metadata database as expected.

(tick) Bug fixed.




Section

Datameer X 11.2.12

Improvements

1Add Support for Cloudera Private Cloud / Cloudera Runtime 7.1.9.0DAP-42564

Datameer now supports Cloudera Private Cloud / Cloudera Runtime 7.1.9.0


2Add support for Amazon AWS EMR 6.10.0DAP-42587

Datameer now supports Amazon AWS EMR version 6.10.0.


3Add support for Amazon AWS EMR 6.11.0DAP-42586

Datameer now supports Amazon AWS EMR version 6.11.0.


4Add support for Amazon AWS EMR 6.12.0DAP-42585

Datameer now supports Amazon AWS EMR version 6.12.0.


5Add support for Amazon AWS EMR 6.13.0DAP-42584

Datameer now supports Amazon AWS EMR version 6.13.0.


6Add support for Amazon AWS EMR 6.15.0DAP-42581

Datameer now supports Amazon AWS EMR version 6.15.0.

Bug Fixes

1Joins on BigDecimal keys return different results when Map side vs Reduce side join is usedDAP-42574

BigDecimal values of different scales are considered equal for JOIN and GROUPBY operations.

(tick) Bug fixed.




Bug Fixes

Datameer X 11.1.2.3

Improvements

Datameer X 11.1.2.2

Bug Fixes

Aggregates with DISTINCT quantifier work again in SQL sheets. Furthermore multiple DISTINCT aggregates in a single SELECT command on the same column are supported while multiple DISTINCT aggregates with multiple different columns are not supported.
Section

Datameer X 11.2.11

Improvements

1Supported Hadoop Distributions: Add support for
Cloudera Private Cloud/ Cloudera Runtime 7.1.6
Amazon AWS EMR 6.14.0DAP-
41823
42564

Datameer
X
now supports
Cloudera Private Cloud/ Cloudera Runtime 7.1.6
Amazon AWS EMR version 6.14.0.


2
Admin Plug-ins: ‘plugin-tez’ - Upgrade to TEZ 0.10DAP-41279After a plug-in patch, the 'plugin-tez' plug-in is now available for Datameer X distributions again.
3Properties: AWS S3 http client max pool size should be configurableDAP-41912
The default maximum connection execution amount is now set to 100. With custom properties, you can change this amount to a higher or lower number. The custom count can also be set via Hadoop property in the cluster settings. 
4Workbook: Create Toolbar button to auto-expand all column width to make all column names visibleDAP-41603
Datameer has now a new toolbar button to expand all column width to make all column names fully visible.
5Workbook: Formula sheet repointingDAP-41549
When changing a formula sheet that points to another sheet, Datameer X now detects this changes and displays a dialog: The user can apply the changes to only the current column or to all columns. This dialog is only displayed, when when more than one column is affected. 
6Workbook: validate join sheet creation and provide supportive error messagesDAP-40809
If one is trying to create an invalid join definition, the join validation will give the user a hint why the join can not be created in a warning dialog.
1Workbook: Parquet format - Ingesting creates duplicate 'das' metadata columnsDAP-41874
When ingesting a Parquet file into Datameer by a File Upload, Import Job or Datalink, there will be no more duplicated columns displayed in the tab "Define Fields". 

(tick) Bug fixed.

2Snowflake Export: Tables are not updating when selecting another schemaDAP-41721

When configuring a Snowflake export or import job, the tables are now updated when changing the schema from the drop-down.

(tick) Bug fixed.

3Workbook synchronisation: Sheet synchronization - Validation error is still showing even after update DAP-41476

After a formerly incorrect formula has been changed into a valid formula, the validation error is no longer displayed. 

(tick) Bug fixed.

4File Browser: Remove the 'Import Job' tab from the tab bar after the job has been deletedDAP-41635
The 'Import Job' tab is now deleted from the tab bar and is no longer displayed, after the job has been deleted. (tick) Bug fixed.
5File Browser: Job status tooltip for queued jobs is missing position informationDAP-41244
The tooltip now shows the position information for the queue status of a job, e.g. position 2 of 5 queued jobs.(tick) Bug fixed.
6Workbook: Active state of the 'encode column' button is brokenDAP-41317
When the 'Encode Column' view in the Workbook Inspector is enabled, the button is in an active state.(tick) Bug fixed.
7FileBrowser: Right-click in an empty area opens the Browser context menu instead of the Folder Tree context menuDAP-41652
Right-clicking in an empty area within the File Browser under the Folder Tree now opens the Folder Tree context menu.(tick) Bug fixed.
8Workbook Inspector: Encoding inspector fly out always shows "Empty Value"DAP-41722
The fly out within the encoding section in the Workbook Inspector now shows the correct value for the different chars.(tick) Bug fixed.
9Upgrading Datameer X: Datameer X stops serving HTTP requests and returns 504 Gateway timeout errorsDAP-41889
After a new fixed release, no more freezes happen and Datameer X works as expected after the upgrade.(tick) Bug fixed.
10Upgrading Datameer X: Upgrade scripts breaks at the ‘Drop Sink Data Table’ step - ‘A foreign key constraint fails’DAP-41955
The upgrading script has been updated to let the Datameer X upgrade work as expected.(tick) Bug fixed.
11Workbook: Fx function in Workbook is not case sensitiveDAP-41992
Selecting a Fx function in a Workbook highlights the function in the Sheet Inspector with upper or lowercase or mixed letters(tick) Bug fixed.
12Workbook: FilesystemRestController#moveFolder causes an endless loopDAP-41989
No endless loop of error messages is caused anymore when moving folders. When trying to move a folder into its own subfolder, an error message is displayed. (tick) Bug fixed.
13Connections: Snowflake - Jobs using the Snowflake connection are failing & exports are failing as wellDAP-41951
After fixing the Snowflake plug-in, the connections are working as expected and exports can be executed as expected.(tick) Bug fixed.
14REST API: JobExecutionRestControllerV2 does not release the EntityManagerDAP-41922
The REST API for job execution is working as expected again. Jobs can be triggered and stopped with REST API commands.(tick) Bug fixed.
15Workbook: SQL sheet - 'COUNT'/ 'SUM'/ 'AVG'/ ... (DISTINCT) aggregates are not supportedDAP-41878
Aggregates with DISTINCT quantifier work again in SQL sheets. Furthermore multiple DISTINCT aggregates in a single SELECT command on the same column are supported while multiple DISTINCT aggregates with multiple different columns are not supported.(tick) Bug fixed.
Section
1Admin Plug-ins: ‘plugin-tez’ - Upgrade to TEZ 0.10DAP-41279
After a plug-in patch, the 'plugin-tez' plug-in is now available for Datameer X distributions again.
Section

1

Upgrading Datameer X: Upgrade scripts breaks at the ‘Drop Sink Data Table’ step - ‘A foreign key constraint fails’DAP-41955
The upgrading script has been updated to let the Datameer X upgrade work as expected.

(tick) Bug fixed.

Section

Datameer X 11.1.2.1

Bug Fixes

1

Upgrading Datameer X: Datameer X stops serving HTTP requests and returns 504 Gateway timeout errorsDAP-41889
After a new fixed release, no more freezes happen and Datameer X works as expected after the upgrade.

(tick) Bug fixed.

2

Workbook: SQL sheet - 'COUNT'/ 'SUM'/ 'AVG'/ ... (DISTINCT) aggregates are not supportedDAP-41878
Security: Upgrade JSch Library for SFPT/SCP connectionsDAP-42550

The Java SSH implementation has been updated to fulfill the latest security recommendations.

Bug Fixes

1Joins on BigDecimal keys return different results when Map side vs Reduce side join is usedDAP-42574

BigDecimal values of different scales are considered equal for JOIN and GROUPBY operations.

(tick) Bug fixed.



Section

Datameer X 11.2.10

Improvements

1Supported Hadoop Distributions: Add support for Amazon AWS EMR 6.9.0DAP-42563

Datameer now supports Amazon AWS EMR version 6.9.0.

Bug Fixes

1Filesystem: Identify filesystem not being closed leaks DAP-42559

The s3A filesystem closure leaks have been detected and closed. Datameer now runs again as expected. 

(tick) Bug fixed.


2Workbook: Workbook navigation tab sticks forever when the Workbook has been deletedDAP-42545

The Workbook tab is now being closed after the Workbook has been deleted. (tick) Bug fixed.


3Performance: UI - Context menu behavior causes high ressource usage in Workbooks with many sheetsDAP-42543

A sheet's context menu options can be used without any loss of performance.(tick) Bug fixed.



Section

Datameer X 11.2.9

Bug Fixes

1Google Data Proc: Version 2.1 - Can not create External System for HiveDAP-42529

Creating an External System for Hive works as expected.

(tick) Bug fixed.


2Workbook: 'WorkbookRestoreManager' lazy initialization failureDAP-42536

After restarting Datameer, opening a Workbook via the FileBrowser first initializes the 'WorkbookRestoreManager' again and one can open the Workbook directly by its URL again.(tick) Bug fixed.


3Authentication: 'Group Filters' section is not openingDAP-42537

The 'Group Filters' section on the Admin's configuration page when configuring external authentication can be opened and closed again.(tick) Bug fixed.



Section

Datameer X 11.2.8

Improvements

1Supported Hadoop Distributions: Add support for Google DataProc 2.0.67 and 2.1.14DAP-42396 and DAP-42521

Datameer now supports Google DataProc versions 2.0.67 and 2.1.14.


2Plug-ins: plugin-snowflake: Use an existing Snowflake stage for export file stagingDAP-42505

The plugin now provides the option to use a specific named staging location defined by a variable in the export as a specific storage option.

Bug Fixes

1Workbook: Custom Workbook undo configuration is reverted to default after a Datameer restartDAP-42520

The custom 'Workbook Undo' configuration is kept after restarting Datameer. 

(tick) Bug fixed.


2Performance: Datameer became unavailable because of high memory and CPU consumptionDAP-42522

After a patched Hadoop AWS file, Datameer works again without performance lost.(tick) Bug fixed.


3Export: Tableau - Export job setup using JSON Web Token authentication method causes Null Pointer ExceptionDAP-42533

The Tableau plugin has been patched. One can use the JWT authentication method again to execute export jobs or create new jobs using the JWT authentication method.(tick) Bug fixed.



Section

Datameer X 11.2.7

Improvements

1Plug-ins: plugin-snowflake - use Snowflake's internal stage for exportsDAP-42491

The plugin now provides the option to select Snowflake's internal stage as the default storage option. After writing the export data into the local file system, it will be imported to Snowflake via a 'PUT' command.


Bug Fixes

1Timezone not resolving correctly in Mexico DAP-42497

After fixing code, the correct timezone is now displayed again for date and time assets in Mexico.

(tick) Bug fixed.


2Workbook: Sorting by the 'Last Processed' time doesn't workDAP-42501

The sorting functionality now works again as expected in the File Browser's Artifacts bar.(tick) Bug fixed.




Section

Datameer X 11.2.6

Improvements

1REST API: Output a response when creating a Workbook vie the APIDAP-42481

REST API v2 calls for creating a new Workbook now return the following information: status, configuration ID, file ID and file UUID, and the. file path.


2Import/ Export: Use the local Datameer timezone instead of UTC while converting numeric values into a dateDAP-42475

Two new global properties ensure that Parquet int96/int64 codes timestamps are treated as being in Datameer's timezone now when set to 'false': 'das.import.parquet.int64.timestamp.adjusted-to-utc' and 'das.import.parquet.int96.timestamp.adjusted-to-utc'.


3Supported Hadoop Distributions: Add support for Cloudera Private Cloud/ Cloudera Runtime 7.1.8DAP-42394

Datameer now supports Cloudera Private Cloud/ Cloudera Runtime 7.1.8.

Bug Fixes

1Administration: Housekeeping doesn't work - jobs stuck in the job schedulerDAP-42485 & DAP-42489

Housekeeping can now be configured via the properties 'housekeeping.keep-deleted-data' and 'housekeeping.keep-deleted-data-max' to remove outdated files.

(tick) Bug fixed.


2REST API: Workbooks with schedule created via REST API v2 are not triggeredDAP-42484

Scheduled triggering for Workbooks that are created via the REST API v2 is now working without a need to manually resave the Workbook's configuration in the User Interface.(tick) Bug fixed.


3Import/ Export: Updating Snowflake drivers DAP-42479

The patched plugin 'snowflake-jdbc' is now working to enable the Datameer - Snowflake connection again. (tick) Bug fixed.



Bug Fixes

The Datameer X Workbook toolbar now works with Internet Explorer
Section

Datameer X 11.2.5

Improvements

1
.2

Improvements

1

Supported Hadoop Distributions: Add support for Cloudera Private Cloud/ Cloudera Runtime 7.1.5.14DAP-41578

Datameer X now supports Cloudera Private Cloud/ Cloudera Runtime 7.1.5.14.

2

Upgrading Datameer X: Upgrade process - 'database.sh' script should print the details of its executionDAP-41882

The 'database.sh' script now contains the status of the database upgrade, when executed. Furthermore, the location of the backup which is created by script has changed. Now the full path to the database file is displayed. Also there are additional entries in the 'database-tool.log' file.

3

Housekeeping: Housekeeping: Data entity should store owner to enable Housekeeping to remove HDFS items even if associated job has been deleted alreadyDAP-41288

Housekeeping has no longer issues in removing HDFS items even if an associated job has already been deleted. Data from where the job configuration has already been deleted is cleaned up successfully without falling back to the system user. The 'data.owner' field receives the value of the current owner of the job configuration.

1

Workbook: Workbook toolbar buttons are not working when using them in the Internet ExplorerDAP-41864
Security: Upgrade Jetty to version 9.4.50DAP-42458

Datameer is moved to the Jetty version 9.4.50 in order to ensure the services are not vulnerable to any known









vulnerabilities.

Bug Fixes

1Security: Penetration tests - GetFiles metadata, get the folder content via the folder IDDAP-42468

A user is not allowed to view any metadata for a directory without proper access (sees no folder path or file ID). 

(tick) Bug fixed.


2Security: Penetration tests - Any user may not see others schema logDAP-42469

A user is not allowed to see the parsing records from the artifacts created by other users.(tick) Bug fixed.


3Security: Penetration tests - The '/browser/list-file' endpoint returns metadata for any file in the systemDAP-42470

A user is not allowed to view artifacts metadata without having at least a 'View' permission for these items.(tick) Bug fixed.


4Security: Penetration tests - Dependency graph information doesn't check the permission authorizationDAP-42471

A user is not allowed to load an artifact’s dependency graph without having at least a 'View' permission for this item.

(tick) Bug fixed.


5Security: Penetration tests - detailed error messages displayedDAP-42472

Only generic error messages without error details are now returned to Datameer users. Stack traces are logged server-side and only accessible by developers or administrators. The associated property handles the error messages.

(tick) Bug fixed.
2


The Datameer X Workbook toolbar now works with Chrome version 89.0.4389.90 (Official Build) and newer.

(tick) Bug fixed.

Section

Datameer X 11.1.1

New Features

1

Supported Hadoop Distributions: Add support for Amazon EMR 6.2.0DAP-41575

Datameer X now supports Amazon EMR 6.2.0.

2
6
Export: Amazon Redshift (Fast Load) export is dropping recordsDAP-41838

The data export into Amazon Redshift (Fast Load) is working correctly again.

(tick) Bug fixed.

3

Workbook: Workbook toolbar buttons are not working when using them in the latest Chrome versionDAP-41836

The Datameer X Workbook toolbar now works with Chrome version 89.0.4389.90 (Official Build) and newer.

(tick) Bug fixed.

4

Execution Engine: Datameer X on Amazon EMR/ Google Cloud Platform - Failed to activate the plug-in 'plugin-dwh'DAP-41816

Since 'plugin-doh' is a conductor-only plug-in, it is not being copied to the cluster anymore and does not impact any job execution.

(tick) Bug fixed.

5

User Interface Login: Intermittently login page is not refreshing after a successful loginDAP-41633

The correct login is working in all browsers for SAML, internal authentication as well as for external authentication (LDAP).

(tick) Bug fixed.

Section

Datameer X 11.1.1.2

Bug Fixes

1

Workbook: Workbook toolbar buttons are not working when using them in the Internet ExplorerDAP-41864The Datameer X Workbook toolbar now works with Internet Explorer.

(tick) Bug fixed.

2

Workbook: Workbook toolbar buttons are not working when using them in the latest Chrome versionDAP-41836
Security: Penetration tests - '/rest/user-management/authenticable-users' should return list of usernames belonging to the same role only with CHANGE_FOLDER_OWNER capabilityDAP-42473

Only user who belong to the same role with the 'change folder owner' capability can view other user names from the REST API call. (tick) Bug fixed.



Section

Datameer X 11.2.4

Improvements

1Drivers: Update Redshift Native JDBC driver to use Amazon's JDBC42 driverDAP-42443

Since the Amazon Redshift Native JDCB driver is deprecated. Therefore the the new drive should be updated to  the Amazon Redshift JDBC42 driver.



Bug Fixes

1Plugins: plugin-hbase+cdh-7.1.7.0 - Import failsDAP-42440

Setting the property 'hbase.server.sasl.provider.extras=org.apache.hadoop.hbase.security.provider.GssSaslClientAuthenticationProvider' bypasses the service loader lookup code with providing authentication providers.

(tick) Bug fixed.



Section

Datameer X 11.2.3

Improvements

1Importing Data: Add an option to configure the default value for "Records for Schema Detection"DAP-42421

The new custom property 'das.conductor.default.record.sample.size' controls what 'Records for schema detection' value is set by default whenever a user creates a new Import Job, Data Link or File Upload. 

It should be adjusted, e.g. when working with large JSON objects (should be decreased to 250 - 500).

Once updated, it affects only newly created Import Jobs, the 'Records for schema detection' value for existing artifacts remains intact.



Bug Fixes

1Export: Snowflake - Exporting to a new table leads to duplicated columns error intermittentlyDAP-42012

After fixing the plug-in, the export sheet can be executed without any errors again.

(tick) Bug fixed.


2Export: Snowflake - Intermittent Export failure, insert value list does not match the column listDAP-42442

After a plug-in fix, the columns list now matches the insert value list as expected again.(tick) Bug fixed.



Bug Fixes

All entities are running in a row again, without any exceptions.
Section

Datameer X 11.2.2

Improvements

1Supported Hadoop Distributions: Add support for Amazon EMR
5
6.
32
8.0DAP-
41574
42400

Datameer X now supports Amazon EMR

5

version 6.

32.0.

Improvements

1

Upgrading Datameer X: Improved upgrading process

DAP-41750 / DAP-41749 /
DAP-41697 

Datameer X has now an improved upgrading process for new upgrades.

2

Workbook: Auto-expand column header during "column edit" if the column name is partly invisibleDAP-41677

When renaming a column, the column width is now expanded fully visible without any truncation.

3

Upgrading Datameer X: Schedule out of order upgradesDAP-41626

The upgrade tool is now upgrading out of order upgrades to e.g. schedule new schema versions which have a lower value than the current one.

4

Supported Hadoop Distributions: Enable Apache Hadoop 2.8.5 supportDAP-41451

Datameer X now supports Apache Hadoop 2.8.5, also with Hive 2.3.8 as well as HBase 1.4.13.

1

Export: Snowflake - Export jobs failing with exceptionDAP-41557
Snowflake export jobs run successfully again after fixing the plug-in.

(tick) Bug fixed.

2

File Browser: Dependencies View artifact loses its color after selecting/ deselectingDAP-41322

In the Dependencies View a selected and then deselected artifact does not change the color to grey anymore.

(tick) Bug fixed.

3

Import: Amazon EMR - Import partitioned data link from S3 does not workDAP-40340
The data link from S3 is now working again as expected.

(tick) Bug fixed.

4

Documentation: Documentation URL should point to public spaceDAP-41713
The public user docs are now available on 'https://datameer.atlassian.net/wiki/display/DAS110'.

(tick) Bug fixed.

5

REST API v2: Remove save dialog when closing a WorkbookDAP-41730
After restoring a Workbook with REST API v2, closing that Workbook in the File Browser does not trigger the 'Save' dialog anymore.

(tick) Bug fixed.

6

File Browser Inspector: Config tab - Missing information for File UploadsDAP-41623
In the File Browser 'Config' tab, the information that scheduling is not supported for Connections and File Uploads is displayed.

(tick) Bug fixed.

7

REST API v2: Rest call leads to an illegal state exceptionDAP-40157

7.0.



2Properties: Make HadoopUtil#HADOOP_ACCESS_TIMEOUT configurable DAP-42404

The timeout in a S3A filesystem can be changed via the properties 'fs.s3a.connection.establish.timout' and 'fs.s3a.connection.timeout' now in order to prevent job timeouts.


Bug Fixes

1Security: Spring framework CVE vulnerabilities DAP-42402

The user has been backported to Spring framework 4.3.

(tick) Bug fixed.



Section

Datameer X 11.2.1

Bug Fixes

1Import/ Export: Hive (with ADLS backed external table) - Failure on importDAP-42389

Hive import jobs now succeeds again when executed at the cluster after setting several properties and adapted Hadoop configuration.

(tick) Bug fixed.


2Plug-ins: Plugin Parquet - Datameer encodes DATE values in base64 while exporting into a Parquet fileDAP-42382

After the 'plugin-parquet' was patched, the DATE fields are exported correctly.

(tick) Bug fixed.


3Export: Tableau - Hyper jobs fail intermittently with "Hyper Server did not call back on the callback connection."DAP-42380

Users are now able to export hyper formatted files to Tableau without any failures again after the associated plug-in has been updated.(tick) Bug fixed.




Section

Datameer X 11.2.0

Improvements

1Supported Hadoop Distributions: Add support for Amazon EMR 5.36.0DAP-42369

Datameer X now supports Amazon EMR version 5.36.0.



2Supported Hadoop Distributions: Add support for Amazon EMR 6.7.0DAP-42375

Datameer X now supports Amazon EMR version 6.7.0.



3Backup & Restore: Performance and lack of logging problemDAP-42370

Performance issues during the backup and restore process are now executed at an expected speed for large number of artifacts.


Bug Fixes

1Plug-ins: EMR - 'plugin-emr' High Availability discovery mode picks local running Yarn Ressource Manager at 0.0.0.0:8088DAP-42378

The EMR cluster can now be set up in High Availability mode. 

(tick) Bug fixed.


2Plug-ins: EMR - 'plugin-emr' "default=obtain from ec2 environment" region option cannot be savedDAP-42372

When configuring the cluster mode in EMR, the default option for the region can now be saved.

(tick) Bug fixed.


3Upgrade: Java upgrade 'Upgrade Filesystem Artifact To Delete' doesn't trigger the database upgradeDAP-42

The Java upgrade script now updates the table 'Upgrade Filesystem Artifact To Delete' as expected or throws a clear error message. (tick) Bug fixed.