Versions Compared

Old Version 32

changes.mady.by.user Juliane Wetzel

Saved on

compared with

New Version 33

changes.mady.by.user Juliane Wetzel

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.


Info
titleINFO

Within these release notes Datameer X provides information and resolutions on improvements, bug fixes, and visual changes for 11.1.x maintenance release versions.  

Visit Datameer's Help Center for additional information or contact your customer service representative. 

Table of Contents

Section

Datameer X 11.1.

14

17

Improvements

1User Logs: Add the correct user for session timeoutDAP-42362

The correct user is now added in the session timeout logging in the 'user action.log' file.



2Execution Engine: Can't execute jobs at the EMR clusterDAP-42350

After patching with an EMR plugin and setting the correct AWS region on the cluster settings page, Datameer can connect against the EMR cluster and run jobs successfully. 



Bug Fixes

1Workbook: Can not open task attempt logs for a job run when the folder contains a '+'DAP-42359

The 'Task Attempt Logs' are also available if the folder name contains a '+'. 

(tick) Bug fixed.


2Export: Amazon S3 - Permissions for the export to S3DAP-42354

After a hot fix, the export to the S3 bucket is now executable again.

(tick) Bug fixed.


...

Section

Datameer X 11.1.15

Improvements

1Supported Hadoop Distributions: Add support for Amazon EMR 5.35.0DAP-42348

Datameer X now supports Amazon EMR version 5.35.0.



Bug Fixes

1JSON: JSON file import time significantly increased after an upgradeDAP-42349

The JSON file can now be imported within the former estimated time duration after introducing the 'Nested array mode' during the import.

(tick) Bug fixed.



Section

Datameer X 11.1.14

Improvements

1Supported Hadoop Distributions: Add support for Amazon EMR 6.5.0DAP-42334

Datameer X now supports Amazon EMR version 6.5.0.



2Security: Add tracking for 'Role' configuration changes DAP-42338

 Changes for a user role configuration are now available in the event-logging.log file. 



3Plugins: Google BigQuery - Implement support for external tables with data in Google DriveDAP-42341

Using the Google BigQuery plugin now allows importing from external tables with data stored in Google Drive. Further documentation can be consumed here: https://cloud.google.com/bigquery/external-data-drive#java.

Bug Fixes

1File Browser: Inspector shows different timestamps in the File Browser and as a search resultDAP-42328

The Inspector now displays the same information regardless on how an artifact is selected in both the File Browser or via the search bar.

(tick) Bug fixed.


2Workbook: Endless loop when attempting to add a second-level circular dependency column into a Workbook DAP-42344

When attempting to add a circular dependency column into a Workbook, the column is marked red and when hovering the column in error state the related error message is shown that this circular reference is not possible. 

(tick) Bug fixed.


...

Section

Datameer X 11.1.13

Improvements

1Supported Hadoop Distributions: Add support for Amazon EMR 5.34.0DAP-42336

Datameer X now supports Amazon EMR version 5.34.0.



2REST API: Cascading permissions can be triggered via the REST APIDAP-42337

Cascade permissions can now be triggered via the REST API call. 


Bug Fixes

1Workbook: Job Graph shows wrong Workbook DetailsDAP-42306

The Job graph now displays the details for the correct Workbook execution it belongs to.

(tick) Bug fixed.


2JSON: JSON Parser should not update the preview for every checkbox selected/ deselectedDAP-42313

Clicking the new "Update Preview" button now refreshes the preview, so the user can select/ deselect as many nodes as they want without having to wait for a UI refresh each time.

(tick) Bug fixed.


3Security: Bulk permissions dialog must prevent owner changes if there is job data or job historyDAP-42323

The bulk permissions change dialog only permits changing the owner if there is no associated import/ workbook job data/ history.

(tick) Bug fixed.


4Security: SAML plugin - Group sharing does not workDAP-42329

After fixing the plugin, members of a user group should have appropriate access to files and folders in the Datameer UI that have been shared within this group.(tick) Bug fixed.


5Import/Export: Compute 'bytes_processed' column from scratch from the data objectsDAP-42332

When creating the usage statistics report, the values from column ‘bytes_processed’ are now measured correctly.(tick) Bug fixed.


6Workbook: Can't start 'Create New Workbook' dialog in the folder that has a '+' in the name. DAP-42339

A user can start the 'Create New Workbook' dialog although the folder name contains a '+'. (tick) Bug fixed.


7Workbook: Missing documentation reference for several Workbook functionsDAP-42320

The documentation links for the following Workbook functions have been added to the function tab:

  • TIMESTAMPDIFF
  • CEILINGDATE
  • FLOORDATE
(tick) Bug fixed.



Section

Datameer X 11.1.12

Improvements

1Workbook: Allow ADDTODATE function to use a non-static value as a second argumentDAP-42322

The ADDTODATE function now takes a non-static value as a second argument with the new syntax: 

Code Block
ADDTODATE(<date>; <string/another column + date constant>




2Import/ Export: Implement %partition% in Export Job file mapping for Parquet and Avro filesDAP-42311

As of now, an Avro/ Parquet export of a time-partitioned sheet directly to the HDFS works with the '/myexport/%partition%/myfile' path when using %partition% as a file name placeholder.

Bug Fixes

1Authentication: SAML authentication - Invalid User Role Exception DAP-42318

An active Datameer X user can always log in regardless of the permissions assigned to the role. 

(tick) Bug fixed.


2Import: HBase Plug-in - Connection/ Import Job failureDAP-42315

After improving the HBase plugin, the connection as well as Import Jobs are working again.

(tick) Bug fixed.


3External Systems: Incorrect mapping for BIG_DECIMAL columnsDAP-42309

Datameer X BIG_DECIMAL columns are not supported for Hive table mapping when a workbook sheet is synced via the External System feature to Hive. For that, the Hive table must not contain BIG_DECIMAL columns. 

(tick) Bug fixed.


4Import/ Export: Can't create the Postgres database driver when Datameer X is using Java11DAP-42325

The Postgres database driver can be created when using Java11version.(tick) Bug fixed.


...

Section

Datameer X 11.1.11

Improvements

1Connections: Google BigQuery - DataLink queries all records from a target tableDAP-42283

When creating a data link to a Google BigQuery connection, Datameer now only uses the amount of records that is needed to create the sample. For that, select "FIRST_N_RECORDS" in the 'Sampling Mode' section from the 'Scheduling' tab. This functionality is useful to reduce the cost of transferring data from the database to the cloud.



2Export: Add seconds and milliseconds as file path placeholdersDAP-40874

Datameer now provides the file path placeholder support for seconds (pattern: %sec%) as well as for milliseconds (%ms%) for file exports.


Bug Fixes

1Workbook: Variables are highlighted red in the Formula Builder DAP-42290

Variables are not highlighted in red in the Formula Builder unless there is a syntax error.

(tick) Bug fixed.


2Security: Update event logging plug-in regarding log4shell/ log4j and retire RTA plug-inDAP-42307

The event logging plug-in has been fixed regarding the log4shell functionality. Furthermore the RTA plug-in has been retired.

(tick) Bug fixed.


3JSON Parsing: JSON parser tool should automatically deselect child nodes if the parent node is deselectedDAP-42299

When deselecting a parent node within the JSON array, all child nodes are automatically deselected for the user.

(tick) Bug fixed.



Section

Datameer X 11.1.10

Improvements

1Connections: JDBC connection - Data Link queries all or a subset of records from a target tableDAP-42276

Datameer X can now limit the number of selected records to the first n records of a table (n as the configured number of sample records).



2Supported Hadoop Distributions: Add Support for Cloudera Private Cloud / Cloudera Runtime 7.1.7DAP-42296

Datameer X now supports Cloudera CDP/ Cloudera Runtime 7.1.7.


Bug Fixes

1Security: Log4j v2 security breach (CVE-2021-44228) DAP-42300

Datameer X now has added {nolookups} tag to log output patterns of log4j v2.

(tick) Bug fixed.


2Import: Datameer drops trailing spaces from the string DAP-42291

Datameer X does not drop space characters at the end of a string anymore while parsing it as a 'Fixed Width' object.

(tick) Bug fixed.


3

Plug-ins: BigQuery - unable to list tables if a table with "SNAPSHOT" type exists in the dataset

DAP-42285

'Com.google.cloud' client libraries have been updated to version 1.32.2, therefore they can handle the "SNAPSHOT" type.

(tick) Bug fixed.


4Import/ Export: BigQuery view import & JDBC exports create temporary tables with "Datameer" in its name DAP-42284

Datameer X does not use "Datameer" as substring during table creation anymore.

(tick) Bug fixed.


5Export: Cluster execution of PowerBI ExportJob fails  DAP-42281

A PowerBI export now succeeds while executed as job executed on the cluster.

(tick) Bug fixed.


6Import: HBase - 'NoSuchColumnFamilyException' and Zookeeper connects to localhost with Tez jobDAP-42280

Datameer X runs HBase import jobs successfully in secured and unsecured cluster environments.

(tick) Bug fixed.


7

Export: Datameer database export jobs randomly fail because of HDFS permissions issue

 		DAP-42275

Datameer export jobs now work reliable again in secured cluster environments.

(tick) Bug fixed.


8

Execution Engine: Job that processes broken a GZIP file never completes 

DAP-42269

Datameer X now handles corrupted GZIP compressed files gracefully and terminates after importing all readable records.

(tick) Bug fixed.


9FileBrowser: Forbid users to create directories at the top-level folderDAP-42214

Datameer X does not revert the FileBrowser's root folder permission during a restart.

(tick) Bug fixed.


...

Section

Datameer X 11.1.8

Bug Fixes

1Security: 'SSLHandshakeException' - Protocol is not appropriateDAP-42262

Creating a SSL-secured IMAP connection can now be established after updating the libraries. 

(tick) Bug fixed.


2UI performance: Delays in displaying artifacts/ folders deletion resultsDAP-42247

The entire Folder Tree will not be reloaded anymore when an artifact has been deleted. Only the deleted artifacts and folders are not shown in the Folder Tree anymore.

(tick) Bug fixed.


3Cascading permissions: Progress bar doesn't reflect the current status - completion notification is not user-friendlyDAP-42245

The progress bar now reflects the current progress for the process. Errors from here are logged in the conductor.log file. 

(tick) Bug fixed.


...

Section

Datameer X 11.1.7

New Features

1

Admin: Job Scheduler - Introduce a control ability to schedule jobs DAP-42008

We introduce a new capability for Datameer X users which can schedule jobs. This capability can be enabled/ disabled in the 'Admin' tab → 'Roles'. 


Improvements

1Export: PostgreSQL/ MySQL/ ClickHouse - Show new textbook for 'Pre-Hook SQL Statement' if the capability is setDAP-41856

You can now enter one SQL statement to be executed before performing an export in the section 'Pre-Hook' when configuring an export job. If the pre-hook fails, the whole export job will be failing. This will also fail, when the export job has a defined pre-hook statement but the job owner does not have the capability to enter it. Note that the SQL statement is not restricted in any way and it has no unintended effects. 


2Export: New capablity 'SQL statement before export execution'DAP-41857

The new capability 'Pre-Hook SQL' allows to create, edit and delete a pre-hook SQL statement before an export job execution and can be set in the 'Admin' tab → 'Roles'. This capability is only set per default for the admin user. 

Bug Fixes

1Import: Google BigQuery - Datameer X can not convert Google BigQuery DATE to Datameer X DATE formatDAP-42220

When importing data from Google BigQuery to Datameer X, the correct pattern is used to convert from Google BigQuery DATE format to Datameer X DATE format. 

(tick) Bug fixed.


2File Browser: Artifacts details sorting issuesDAP-42191

Sorting artifacts in the File Browser works properly for all columns again and does not reset when scrolling. 

(tick) Bug fixed.


3Documentation: Broken documentation link for the export job configuration dialogDAP-42188

The documentation link for 'File Path and File Name Patterns' within the export job wizard now points to the correct documentation page. 

(tick) Bug fixed.


4REST API: Curl command for uploading a license should not create a user sessionDAP-41581

Using the curl command for uploading a license does not increase the session count for user sessions any more. (tick) Bug fixed.


5File Backup: Error when backing up folder in the GUIDAP-42241

A file backup works as expected again after patching Datameer X. (tick) Bug fixed.


6Setup: 'Enable-ssl-customization.sh' should actually create 'ssl.ini'DAP-42199

When executing the 'enable-ssl' script, the HTTP module ist automatically disabled and the HTTPS/ SSL module enabled. Furthermore the 'start.d' file and 'ssl.ini' are created to contain all default values and standard properties.(tick) Bug fixed.


...

Section

Datameer X 11.1.5

New Features

1Import/ Export: Salesforce Bulk API importsDAP-21069

The Salesforce Connector now uses the Bulk API 2 to import data via a REST API. For running bigger datasets, custom properties for the import job are required, e.g. 'datameer.salexforce.maxrecord-per-result-request.  



2Execution Engine: Allow jobs bypassing the Job Scheduler queueDAP-41875

Jobs can now be moved in the Job Scheduler queue. For that a new checkbox is included in the scheduling sections in Import/ Export/ Workbook Details wizard: 'Bypass Job Queue'. Jobs with that configuration will be picked always first by the Job Scheduler and ignore the maximal concurrent running jobs. A new role capability controls whether a user is allowed to use this option. 



3Workbook functions: New 'REGEXTRACT_LIST' functionDAP-42147

You can now use the 'REGEXTRACT_LIST' function. Using the function with data type 'string' extracts tokens that match a regular expression. It returns 'null' in case the string has no RegEx matches and returns a list instead of a string. If the string has a single RegEx match, the function returns a single element list. If the string has no RegEx matches, the function returns an empty list and if the string has multiple RegEx matches, the function returns all of them as the list elements.

Improvements

1

Supported Hadoop Distributions: Add support for Amazon EMR v6.3.0DAP-42063

Datameer X now supports Amazon EMR version 6.3.0.



2Supported Hadoop Distributions: Add support for Amazon EMR v5.33.0DAP-42064

Datameer X now supports Amazon EMR version 5.33.0. 


3Workbook/ Export : Enable sliding time window export of files by defining the partition column - partitioned exportDAP-39796

You can now use the new placeholder '%partition%' for the target path and the file names for export jobs. This placeholder can only be used, when the source Workbook sheet is partitioned. When set, the placeholder will be replaced by the partition value returned by a partition filter which is configured in the configuration of the export job. At least, if a filter returns more than one partition, the export job will be creating a separate file or folder for each position. 


4Import: Oracle Eloqua - Importing Data into Datameer XDAP-42077

You can now ingest import data from Oracle Eloqua into Datameer X to receive campaign responses. For that, a JDBC driver is needed. 


5REST API: Admins can pause and resume the job scheduler and auto compactionDAP-36390

You can now pause and resume the job scheduler and auto compaction with REST API commands. 


Bug Fixes

1Workbook: Repoiting - 'NullPointerException' if the column to be repointed is blankDAP-42141

When repointing columns from a source sheet to a duplicated sheet, there is no 'NullPointerException' error anymore.

(tick) Bug fixed.


2Upgrading Datameer X: Upgrade to Datameer X version 10.1.9 is blockedDAP-42131

After fixing a swallowed SQL sheet and fixing some minor dependencies, the upgrade could be executed successfully.

(tick) Bug fixed.


3Workbook: Autocomplete in the 'Formula Builder' offers columns from all sheetsDAP-42105

When inserting a formula in the Workbook's 'Formula Builder' only the sheets and columns for the first reference and only the columns from the current sheet are offered for autocompletion. Furthermore, autocomplete filters out dummy columns, e.g. insert a formula in the second column, apply, and then go to the first column. 

(tick) Bug fixed.


4Workbook: Renaming a column with a long name leads to a shift between the column header and the columnDAP-41921

After renaming a column name, no shift is visible between the column header and the column. (tick) Bug fixed.


5Workbook Health Check: Link to open a Workbook still uses the Workbook ID instead of the file IDDAP-42181

When opening a broken Workbook from the Workbook Health Check page, the file ID is used to open a Workbook now.(tick) Bug fixed.


6Workbook: Folder can't be dragged to the root in the Folder Tree panelDAP-42168

You can now drag a folder to the root in the Folder Tree panel. (tick) Bug fixed.


7Export: CSV partitioned export - improve help text and make the partition pattern option only visible on the CSV exportDAP-42138

The general help text for using a partition pattern export is now improved and visible in the CSV based export wizard: "The placeholder %partition% is only processed if the export sheet is partitioned!"(tick) Bug fixed.


8Login: After a successful login, the user must not be redirected to a Web Socket URLDAP-42062

When logging in after an automatical logout due to a user inactivity, a user is now redirected back to the previous location from before the logout. (tick) Bug fixed.


9Workbook: The Workbook view shows an error of a previous job run when triggering a new jobDAP-42099

The error from the former broken Workbook is not displayed anymore when triggering the fixed Workbook and the job graph is being displayed now instead.  (tick) Bug fixed.


10Workbook Settings: 'Data Driven' in the Workbook settings is not disabled when changing a JDBC connection to a session based DAP-40390

When a connection is configured with a session-based authentication and the Workbook is configured as 'Data Driven' by the source sheet, the Workbook is not triggered when the source is not triggered. (tick) Bug fixed.


11Export: Partition export job does not work in cluster modeDAP-42159

The partitioned export job with the pattern '%partition%' works in the cluster mode as expected. (tick) Bug fixed.


12Workbook: Display the autocomplete list for current sheet and column nameDAP-42180

When using a formula that contains the current sheet name and the column name, the autocomplete list is now displayed. (tick) Bug fixed.


...

Section

Datameer X 11.1.3

New Features

1

Supported Hadoop Distributions: Add support for Google Dataproc version 1.5DAP-40353

Datameer X now supports Google Dataproc version 1.5.


Improvements

1

Supported Hadoop Distributions: Add support for Cloudera Private Cloud/ Cloudera Runtime 7.1.6DAP-41823

Datameer X now supports Cloudera Private Cloud/ Cloudera Runtime 7.1.6.



2Admin Plug-ins: ‘plugin-tez’ - Upgrade to TEZ 0.10DAP-41279

After a plug-in patch, the 'plugin-tez' plug-in is now available for Datameer X distributions again.


3Properties: AWS S3 http client max pool size should be configurableDAP-41912

The default maximum connection execution amount is now set to 100. With custom properties, you can change this amount to a higher or lower number. The custom count can also be set via Hadoop property in the cluster settings. 


4Workbook: Create Toolbar button to auto-expand all column width to make all column names visibleDAP-41603

Datameer has now a new toolbar button to expand all column width to make all column names fully visible.


5Workbook: Formula sheet repointingDAP-41549

When changing a formula sheet that points to another sheet, Datameer X now detects this changes and displays a dialog: The user can apply the changes to only the current column or to all columns. This dialog is only displayed, when when more than one column is affected. 


6Workbook: validate join sheet creation and provide supportive error messagesDAP-40809

If one is trying to create an invalid join definition, the join validation will give the user a hint why the join can not be created in a warning dialog.

Bug Fixes

1Workbook: Parquet format - Ingesting creates duplicate 'das' metadata columnsDAP-41874

When ingesting a Parquet file into Datameer by a File Upload, Import Job or Datalink, there will be no more duplicated columns displayed in the tab "Define Fields". 

(tick) Bug fixed.


2Snowflake Export: Tables are not updating when selecting another schemaDAP-41721

When configuring a Snowflake export or import job, the tables are now updated when changing the schema from the drop-down.

(tick) Bug fixed.


3Workbook synchronisation: Sheet synchronization - Validation error is still showing even after update DAP-41476

After a formerly incorrect formula has been changed into a valid formula, the validation error is no longer displayed. 

(tick) Bug fixed.


4File Browser: Remove the 'Import Job' tab from the tab bar after the job has been deletedDAP-41635

The 'Import Job' tab is now deleted from the tab bar and is no longer displayed, after the job has been deleted. (tick) Bug fixed.


5File Browser: Job status tooltip for queued jobs is missing position informationDAP-41244

The tooltip now shows the position information for the queue status of a job, e.g. position 2 of 5 queued jobs.(tick) Bug fixed.


6Workbook: Active state of the 'encode column' button is brokenDAP-41317

When the 'Encode Column' view in the Workbook Inspector is enabled, the button is in an active state.(tick) Bug fixed.


7FileBrowser: Right-click in an empty area opens the Browser context menu instead of the Folder Tree context menuDAP-41652

Right-clicking in an empty area within the File Browser under the Folder Tree now opens the Folder Tree context menu.(tick) Bug fixed.


8Workbook Inspector: Encoding inspector fly out always shows "Empty Value"DAP-41722

The fly out within the encoding section in the Workbook Inspector now shows the correct value for the different chars.(tick) Bug fixed.


9Upgrading Datameer X: Datameer X stops serving HTTP requests and returns 504 Gateway timeout errorsDAP-41889

After a new fixed release, no more freezes happen and Datameer X works as expected after the upgrade.(tick) Bug fixed.


10Upgrading Datameer X: Upgrade scripts breaks at the ‘Drop Sink Data Table’ step - ‘A foreign key constraint fails’DAP-41955

The upgrading script has been updated to let the Datameer X upgrade work as expected.(tick) Bug fixed.


11Workbook: Fx function in Workbook is not case sensitiveDAP-41992

Selecting a Fx function in a Workbook highlights the function in the Sheet Inspector with upper or lowercase or mixed letters(tick) Bug fixed.


12Workbook: FilesystemRestController#moveFolder causes an endless loopDAP-41989

No endless loop of error messages is caused anymore when moving folders. When trying to move a folder into its own subfolder, an error message is displayed. (tick) Bug fixed.


13Connections: Snowflake - Jobs using the Snowflake connection are failing & exports are failing as wellDAP-41951

After fixing the Snowflake plug-in, the connections are working as expected and exports can be executed as expected.(tick) Bug fixed.


14REST API: JobExecutionRestControllerV2 does not release the EntityManagerDAP-41922

The REST API for job execution is working as expected again. Jobs can be triggered and stopped with REST API commands.(tick) Bug fixed.


15Workbook: SQL sheet - 'COUNT'/ 'SUM'/ 'AVG'/ ... (DISTINCT) aggregates are not supportedDAP-41878

Aggregates with DISTINCT quantifier work again in SQL sheets. Furthermore multiple DISTINCT aggregates in a single SELECT command on the same column are supported while multiple DISTINCT aggregates with multiple different columns are not supported.(tick) Bug fixed.


...