7.5.5 Release Notes
Improvements
1 | Optimize multimap value removal code in CombineOnlyDataLocalAlgorithm | DAP-39565 |
|---|---|---|
| Combining splits performance has been improved especially for large number of splits to combine. |
2 | Add Support for CDH 6.3.2 | DAP-39657 |
|---|---|---|
| Datameer now supports CDH 6.3.2. |
Bug Fixes
1 | Hot deployment of a plug-in fails if the plug-in contains an DataStoreType implementation | DAP-38885 | |
|---|---|---|---|
A plug-in containing a DataStoreType can now be installed and de-installed via Admin - Plug-in section, e.g. Hive plug-in. |
|
Bug fixed.2 | REST API to download results doesn't return all records by setting 'rest.download-data.records-max=0' | DAP-39386 | |
|---|---|---|---|
Datameer now respects the setting 'rest.download-data.records-max=0' and provides all records for download. |
|
3 | Hive 2 plug-in: ImportJobs and Datalinks are failing on Hadoop Namenode and Resourcemanager high available environments | DAP-39397 | |
|---|---|---|---|
Hive 2 plug-in based Importjobs and Datalinks now run successfully on high available environments. |
|
4 | Workbook tab view doesn't close after Close & Save | DAP-39417 | |
|---|---|---|---|
After closing and saving a workbook, the workbook tab gets closed and the user gets re-directed to the file browser. |
|
5 | Hive String queryTemplate for SHOW PARTITIONS is incorrect | DAP-39438 | |
|---|---|---|---|
The string for queryTemplate now shows the required syntax using back-ticks for every single part of the Hive query. |
|
6 | Visual Explorer: Explorer can't be closed via Escape key | DAP-39446 | |
|---|---|---|---|
Visual Explorer dialog now closes when the Escape key gets pressed. |
|
7 | Workbook config: false errors on correct CRON pattern | DAP-39458 | |
|---|---|---|---|
Datameer now recognizes a valid CRON pattern and does not throw an error on a valid pattern errornously. |
|
8 | Scheduled jobs remain indefinitely in the processing queue if the Kerberos password is changed | DAP-39474 | |
|---|---|---|---|
A scheduled job with in between changed Kerberos password does not block the processing queue anymore. |
|
9 | Visual Explorer should dismiss dialogs in order on Escape | DAP-39504 | |
|---|---|---|---|
Pressing escape dismisses the dialog which is on the top now (e.g. x-y-axes definition dialog instead of closing the whole Visual Explorer dialog) |
|
10 | Workbook: save dialog doesn't open when user selects to Close & Save | DAP-39509 | |
|---|---|---|---|
Closing a new workbook from the file browser now redirects trough the workbook save dialog. |
|
7.5.4 Release Notes
New Feature
1 | Support Engineer Plug-in - Customer Usage Analytics | DAP-36640 |
|---|---|---|
The SEP contains both high-level metadata and various logs from within Datameer environments. This allows administrators to easily pass necessary information to the Datameer support engineers. |
Improvements
1 | Plug-in Hive: Support Export into new table in format Parquet, ORC, Avro or Text File | DAP-38804 |
|---|---|---|
| When exporting data to Hive, the user is presented with a simple drop-down menu to select the file format. You can choose between Parquet, ORC, Avro and Text File. |
2 | Improve searching for folders within the file browser | DAP-38862 |
|---|---|---|
| Easily search for files by paths or by folder names. |
3 | Support for Parquet File Format with types Date and Timestamp | DAP-39000 |
|---|---|---|
If date and timestamp are data types, use monospace. The data types |
| 4 | Materialize all SQL sheet inputs, join optimizer | DAP-39011 |
|---|---|---|
Execution of SQL joins within workbooks is decreased when using the improved join optimizer. |
5 | Workbook validity report | DAP-39389 |
|---|---|---|
| Datameer provides report that identifies workbooks missing data sources or other similar issues. |
6 | Hive: partition location cache scheduling has been removed | DAP-39392 |
|---|---|---|
| Datameer does not cache partition locations on a schedule, only on demand. |
7 | Production Mode: In-Place Filter Prevents Projection Push-down for Data Links | DAP-39395 |
|---|---|---|
| In production mode, the column projection feature is available for data links. This is a performance improvement for workbooks using filter conditions and not processing all columns ingested from the datasource. |
8 | Add Support for Amazon EMR v5.27.0 | DAP-39457 |
|---|---|---|
| Datameer now supports Amazon EMR 5.27.0. |
9 | Add Support for CDH 6.2.1 | DAP-39461 |
|---|---|---|
| Datameer now supports CDH 6.2.1. |
10 | Add Support for CDH 6.3.1 | DAP-39462 |
|---|---|---|
| Datameer now supports CDH 6.3.1. |
Bug Fixes
1 | Artifact ( IJ, Wbk, FU, DL, EJ ) status not updated in dependency view | DAP-37822 |
|---|---|---|
The entity status in the dependency view reflects the current status (success/ error/ canceled) of this entity. |
|
2 | Workbook: Valid SQL queries are not highlighted as errors | DAP-37984 |
|---|---|---|
Datameer erroneously marked valid SQL statements as false. |
|
3 | Download workbook data runs in ArrayIndexOutOfBoundsException | DAP-38839 |
|---|---|---|
The download of workbook data is not influenced by workbook changes after the run. Furthermore partition selection is no longer possible after the sheet containing a date column was deleted. |
|
4 | Error when importing a worksheet from a sorted workbook in production mode | DAP-38874 |
|---|---|---|
Importing a worksheet from a sorted workbook in production modes has been fixed. |
|
5 | Hive2-plugin: Append-Export in s3 located table overwrites existing data ( unpartitioned tables ) | DAP-38918 |
|---|---|---|
Records in a Hive table with S3 location are appended when the job export configuration to this table is configured to "append" records. When the configuration is set to "overwrite", the records are overwritten. |
|
6 | AdminVariables: Hover state of buttons has been corrected | DAP-39113 |
|---|---|---|
While editing admin variables within IE 11, all buttons have the correct color now. The hover state disappears as expected. |
|
7 | Error when creating a variable with more than 255 chars | DAP-39114 |
|---|---|---|
Variable names can have a maximum of 255 characters. Variable values can have a maximum of 255 characters. |
|
8 | Workbook: variables not listed on workbook details / job run details page when job was scheduled/data driven | DAP-39154 |
|---|---|---|
When a workbook is data driven by an import job or scheduled, all variables are listed within the job run details/ details page. |
|
9 | SER: improve how the config files are displayed | DAP-39211 |
|---|---|---|
JSON config file are displayed as key-value pairs, making them both machine and human readable. |
|
10 | Workbook: formulas are applied on loss of focus | DAP-39258 |
|---|---|---|
When a column looses focus or when adding a new sheet, any changes to a formula are applied without user interaction. |
|
11 | Backup/Restore does not abort when encountering broken artifacts | DAP-39297 |
|---|---|---|
Even if there are broken artifacts, a backup continues and logs the broken files. |
|
12 | Hive-Jdbc: Rescan Schema leads to SaslException on NMU environment | DAP-39308 |
|---|---|---|
Rescan schema now works without generating an exception for hive JDBC running in DM NMU mode |
|
13 | Hive2-plugin: Export with mode Create New and policy RECREATE fails if table exists on NMU environment | DAP-39318 |
|---|---|---|
The export with mode "Create new if not exist" and policy "Drop and create new table" (RECREATE) works if the table exists on the NMU environment. This works for external and managed tables as well for HiveServer2 and Metastore based export jobs. |
|
14 | Workbook: inconsistent column names if sheet has empty columns | DAP-39322 |
|---|---|---|
Column naming for SQL sheets now follows the usual conventions (A, C, C1, ...). |
|
15 | Hive2-plugin: "Table Output Format" Select looks unformatted in Export Wizard | DAP-39324 |
|---|---|---|
The "Table Output Format" select field to the "Data Details" section within the Hive export wizard is now correctly formatted. |
|
16 | Admin page: inform user if SER plugin is disabled/unavailable | DAP-39326 |
|---|---|---|
When the Support Engineer Plugin is disabled or unavailable, the section "Support Engineer Plugin" is removed from the admin section. |
|
17 | SER: browser tab has wrong title | DAP-39328 |
|---|---|---|
Being on the "System & Usage Report" section now displays the correct heading ("System & Usage Report") in the browser. |
|
18 | SER: download icons disappears after clicking it | DAP-39329 |
|---|---|---|
After clicking the download icon for a log, the icon stays visible. |
|
19 | SER: environment information json is empty after download | DAP-39331 |
|---|---|---|
When downloading the whole "System & Usage" report, the environment jSON file contains content. |
|
20 | IE11: System & Usage Report shows blank page | DAP-39343 |
|---|---|---|
When opening the "System & Usage Report" section, all expected content appears. The download works again and single logs can be downloaded as expected. |
|
21 | Customer Upgrade: ExportJob forcing append | DAP-39365 |
|---|---|---|
Hive export jobs are no longer duplicated after upgrading from Datameer 7.1 to 7.5. In case of recreating or overwriting, old data from a Hive managed table is deleted. In case of recreating or overwriting from a Hive external table will not be deleted. |
|
22 | Error dialog has broken layout if variable name is too long | DAP-39393 |
|---|---|---|
Error dialogs with long variable names are displayed correctly. Long variable names are broken to the next line now. |
|
23 | Filebrowser: search function in folders with whitespace not working | DAP-39411 |
|---|---|---|
Searching on the file browser takes whitespace into account and doesn't cause errors. |
|
24 | Add to Formula doesn't add formula to formula editor from Browse Columns if working/browsing same sheet | DAP-39439 |
|---|---|---|
Clicking the "Add to formula" button adds a formula to the formulas editor from "Browse Columns" if working the same sheet. |
|
25 | v7.5: Customer Upgrade: Join Columns in Error | DAP-39456 |
|---|---|---|
Cascading errors in join sheets don't appear any more. After an upgrade to Datameer version 7.5, columns are migrated successfully. Circular dependencies are detected. Broken workbooks stay broken with the same error. |
|
7.5.3 Release Notes
Improvements
1 | Add support for Google Cloud Service (GCS) protocol to import and export files | DAP-20331 |
|---|---|---|
| Datameer supports import to and export from39146 Google Cloud Service (GCS) connections. |
2 | Join Operation Should Compute Inputs in Parallel instead of one at a time | DAP-39146 |
|---|---|---|
| For improved performance, join of two data links now computes inputs in parallel to prevent the creation of too many DAG(s) on the cluster, and the Map-Side join picks the smallest of the two inputs to cache in memory. |
3 | Plug-in Hive: improve implicit mapping for type BigInteger | DAP-39147 |
|---|---|---|
When exporting into existing Hive tables Datameer's "BIG_INTEGER" can also be exported as Hive "decimal." |
4 | Add Support for CDH 5.16.2 | DAP-39131 |
|---|---|---|
| Datameer now supports CDH 5.16.2. |
5 | Add Support for CDH 6.3.0 | DAP-39169 |
|---|---|---|
| Datameer now supports CDH 6.3.0. |
6 | Add Support for Amazon EMR 5.24.1 | DAP-39132 |
|---|---|---|
| Datameer now supports Amazon EMR 5.24.1. |
7 | Add Support for Amazon EMR 5.25.0 | DAP-39170 |
|---|---|---|
| Datameer now supports Amazon EMR 5.25.0 |
8 | Add Support for Amazon EMR 5.26.0 | DAP-39284 |
|---|---|---|
| Datameer now supports Amazon EMR 5.26.0 |
Bug Fixes
1 | Upgrade scripts are ignored when using Maria DB | DAP-39293 | |
|---|---|---|---|
For a fresh installation, the optional_upgrade table in Datameer's metadata database now gets filled, therefore all included optional upgrades for the current Datameer version are set as expected. |
|
2 | BigQuery import job Data Details wizard page UI reload doesnt adhere to selected table | DAP-39230 | |
|---|---|---|---|
When editing an existing BigQuery import, the selected table is the same as before the point of creation. |
|
3 | Google BigQuery import connector non-US region datasets failure | DAP-39217 | |
|---|---|---|---|
BigQuery connector works for datasets residing in regions other than USA. |
|
4 | After opening a workbook by path can't switch tabs back to that workbook | DAP-39197 | |
|---|---|---|---|
It is possible to switch tabs from file browser to the workbook when opening a workbook by path. |
|
5 | Can't create a Variable if Datameer serves via SSL | DAP-39167 | |
|---|---|---|---|
The mentioned source gets called via HTTPS as well when SSL/TLS is enabled in Datameer. |
|
6 | TDE/Hyper export with UNICODE symbols over U+D7FF fails job/crashes Datameer (in local ef) | DAP-39164 | |
|---|---|---|---|
Characters that caused a crash when exporting a sheet to Tableau are replaced with HTML entities. Such replacements are logged in the application log. |
|
7 | Workbook: variables in formula editor are highlighted in red | DAP-39155 | |
|---|---|---|---|
Defined variables are no longer highlighted as an error in workbook sheet formulas or SQL queries. |
|
8 | Workbook: error message is shown after duplication of a Recommendation or Column Dependencies sheet | DAP-39104 | |
|---|---|---|---|
No error message is shown after duplication of a Recommendation or Column Dependencies sheet. |
|
9 | Workbook: editor for SQL query not working correct in some cases | DAP-39062 | |
|---|---|---|---|
The SQL query editor now handles <space> and <return> key strokes as well as working with auto completion in the correct way. |
|
10 | Datameer is missing the "X-XSS-Protection" HTTP response header | DAP-39048 | |
|---|---|---|---|
`X-XSS-Protection: 1; mode=block` is set in the header of every response coming from the Datameer backend. |
|
11 | Workbook: show correct status bar and error/info dialog if workbook has status "timed out" | DAP-39044 | |
|---|---|---|---|
Datameer now shows the correct status bar and error or information dialog when a workbook has been timed out. |
|
12 | Can't read DECIMAL from an Avro file | DAP-39006 | |
|---|---|---|---|
Datameer now imports Avro data with primitive type bytes logicaltype decimal as BIG_DECIMAL in Datameer. |
|
13 | Workbook - Datasource Sheet: a schema change or an encoding change of the referenced import job breaks loading of the referencing workbook | DAP-38848 | |
|---|---|---|---|
Datameer now opens the referencing workbook after a schema or encoding change to the referenced import job. |
|
14 | Workbook: Data link with workbook pushdown filter reported total records are not always transparent | DAP-37857 | |
|---|---|---|---|
Datameer now distinguishes more clearly between availability of metrics for total and sample records. (Instead of displaying a count of total rows for a Datalink used by a workbook with a filter condition, we show the message "Showing <number> rows of an unknown number of total rows". |
|
7.5.2.1 Release Notes
Bug Fixes
1 | Can't create a Variable in the Admin section if Datameer serves via SSL | DAP-39167 |
|---|---|---|
The mentioned source gets called via HTTPS as well when SSL/TLS is enabled in Datameer. |
7.5.2 Release Notes
New Feature
1 | Certify Hive Plugin for Amazon's EMR Distribution 5.24 to the Standard Datameer Application | DAP-38274 |
|---|---|---|
| Improved connector to allow Hive access on EMR. |
Improvements
1 | Improve responsiveness of UI for customers with large number of job executions | DAP-39030 |
|---|---|---|
| Querying very large tables for job executions could take seconds, rendering the file browser UI unusable. Datameer now has an index for the job_status column in the metadata database to speed information queuing and display. |
2 | Integrate Plug-in Code into Plug-in NMU to notify users by email when jobs fail due to missing/stale keytab | DAP-38761 |
|---|---|---|
| Users get now notified by email when a Datameer job is failing because of a missing or stale keytab file. |
3 | Workbook Variables: improve icon and edit behaviour | DAP-38879 |
|---|---|---|
| Datameer now switches to edit mode for workbook variables and renders buttons active/inactive according to their expected states. |
4 | No plugin with id '<plugin_name>' found for deserializing class '<plugin_internal_class>' | DAP-38023 |
|---|---|---|
| Datameer now handles occurrences of YARN-359 or YARN-6641 gracefully so that errors indicating a specific plug-in isn't available can be mitigated. The workaround can be activated with the das.cluster.plugin.resources.fixer.enabled property. |
5 | Pivot Workbook Inspector: do not show red border and 'Select values to include" if column was removed manually | DAP-38910 |
|---|---|---|
| Datameer now renders an empty column selection field for column names that do not exist. |
Bug Fixes
1 | Hive Plug-in - Export: Data-driven Hive ExportJobs submitting with random impersonated user | DAP-39046 | |
|---|---|---|---|
| Datameer now runs the Hive export job with correct user impersonation. |
|
2 | Datameer can't recognise Int32 and Int64 Decimal values on a Parquet File Ingestion | DAP-39110 | |
|---|---|---|---|
Datameer now can handle Parquet files containing Binary, Fixed, INT32 and INT64, as well as DECIMAL column types |
|
3 | Workbook can not update SQL sheet when variable was deleted | DAP-39014 | |
|---|---|---|---|
| Datameer now allows to update a SQL Sheet with a valid query, even if former referenced workbook variable doesn't exist anymore. |
|
4 | Workbook: Pivot inspector - no hint when column is choosen but no value was selected | DAP-39128 | |
|---|---|---|---|
| Datameer renders a red border and indicates an error, if a column was choosen but no value was selected. |
|
5 | RESTv2 call to stop a job responds with error | DAP-38985 | |
|---|---|---|---|
| The REST call responds now with status 200 and stops the referenced job. |
|
6 | After re-run on a partitioned DataLink, partition information is missing in the workbook | DAP-38271 | |
|---|---|---|---|
| Datameer renders the partition information for a datalink in the Sunburst for selection, even if the datalink was triggered in between again. |
|
7 | Hive Plug-in: Transport Mode HTTP with Table Filter set to "Table and View" leads to an error on Kerberos secured environments | DAP-38979 | |
|---|---|---|---|
| This setup and configuration combination no longer generates an error. |
|
8 | Hive Plug-in: Hive Metastore - Create Delegation Token for Kerberos Secured Hive independent of configured Impersonation Mode | DAP-39034 | |
|---|---|---|---|
Datameer now creates the required delegation token for accessing a Kerberos Secured Hive. |
|
9 | Pivot Sheet: Closing the error dialog with Enter or Escape key doesn't work | DAP-39008 | |
|---|---|---|---|
| Dialogs for Pivot Sheet operations now handle Enter and Escape key submissions in the expected way. |
|
10 | NPE when configure plug-in where preferences extension are disabled | DAP-39091 | |
|---|---|---|---|
| Datameer now correctly handles disabling an extension point of plug-in and does not produce a "NullPointerError". |
|
11 | Upgrade: Many Workbooks failing after upgrading to 7.4.6 because of wrong auto-resource computation | DAP-39055 | |
|---|---|---|---|
Datameer auto-resource computation for required vCores no longer results in zero. |
|
12 | Undo/Redo throws exception after reorder SQL sheet with variable | DAP-38882 | |
|---|---|---|---|
Undo/Redo now works without errors when re-ordering SQL sheets with variables. |
|
13 | Workbook: invalid CSRF token shows error message over actual warning | DAP-39026 | |
|---|---|---|---|
| Datameer only shows the warning message for the invalid session. |
|
14 | Wrong icons on workbook | DAP-39105 | |
|---|---|---|---|
| Datameer now renders the correct icons in the workbook. |
|
15 | Add new "Timed out" icon for jobs | DAP-38769 | |
|---|---|---|---|
| Datameer renders the time-out icon on job run details and job history page. |
|
16 | Workbook: Sunburst does not open for second sheet when adding data of 2 workbooks | DAP-38284 | |
|---|---|---|---|
| Datameer renders the Sunburst for partition selection for every partitioned data source of a workbook. |
|
17 | Workbook exchange data source should respect "kept" attribute of the datasource being replaced | DAP-39001 | |
|---|---|---|---|
| When exchanging a datasource, the "kept" flag respects the original data sheet "kept" value. |
|
18 | Workbook: wrong syntax highlighting on formula sheets | DAP-39130 | |
|---|---|---|---|
Datameer now renders with the correct syntax highlighting in workbook formulas. |
|
19 | Cross-Site Scripting Vulnerability in Advanced Filter Dialog and SQL Sheet Query input fields | DAP-39149 | |
|---|---|---|---|
Advanced filter dialog and SQL sheet query input fields are not vulnerable to Cross Site Scripting attempts. |
|
20 | Memory-Backed Map-Side Join will never be chosen when input is two Data Links | DAP-39095 | |
|---|---|---|---|
The chosen join algorithm will be MEMORY_BACK_MAP_SIDE when two data links are being joined and at least one of the sides is small. |
|