Versions Compared

Old Version 1

changes.mady.by.user Juliane Wetzel

Saved on

compared with

New Version Current

changes.mady.by.user Konsta Danyliuk (Deactivated)

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: Removed my.datameer.com reference

Table of Contents

Upgrading From an Older Release

Datameer X customers can download updated software versions from http://my.datameer.comGet in touch with Datameer Support or your Datameer representative to receive the installation package download link.

To upgrade from an older release, you should first back up your data. Then you will need to upgrade the application and upgrade the database.

...

  • If you used Kerberos prior to 5.11, you need to install the plug-in during upgrade.
  • If you are running a Kerberos-secured cluster with impersonation enabled, you need to run the securethe secure_hdfs_tool.sh command line tool when tool when upgrading to versions 6 and above. For versions 6.1 and above, run the command after upgrading, starting, and stopping Datameer. Once you've run the command, restart Datameer.

...

  • Before upgrading Datameer X a plan should be made for application downtime scheduling, notifications, and creating a maintenance window for Datameer. Please take note of the system requirements and consider upgrading the MySQL service for Datameer's application database from 5.1 to 5.5 or 5.6, if applicable.
  • Backed up Datameer X files need to be included in the migration process. Previous files that aren't part of the migration process aren't supported in newer versions.
  • Jobs will need to be set to stop executing prior to the upgrade process. This could be done e.g. by Pause Job Scheduler.
  • Request possible assistance from the database administrator in case credentials for database are not available to the application owner.
  • Custom plug-ins which were created by Using the Plug-in SDK of a former Datameer X version need to be re-compiled!
  • Since it is recommended to not copy old configuration files or scripts to the new location, note the changes you have made in the previous setup. This information you will need to make necessary changes in the new configuration files.
    • As of Datameer X 7.4, a property file exists so Job Scheduler and Event Bus settings adjusted in the UI are read during Datameer X start up.
      • A properties file called "overrides.properties" isn't written on Datameer X but your systems HOME folder. (Path = <home>/.datameer/overrides.properties)
      • This property file is auto-created when adjusting the Event Bus or Job Scheduler settings under the Admin tab. This file can also be manually added and edited.
      • This is the last properties  file read on start up. (E.g., it overrides other property files like default.properties and deployMode.properties)
        This file can be modified to allow for storing and the restoring of custom properties..properties)
      • This file can be modified to allow for storing and the restoring of custom properties.
Tip
titleTIP

Symlinks that were created through the Datameer X installation must be updated through the updating process.

To update the symlink:

  1. Remove the existing symlink:

    No Format
    rm current


  2. Create a new symlink and change the working directory:

    No Format
    ln -s Datameer-<package> current
cd current


Disabling Housekeeping/Compaction services

In the event of a problem after a Datameer X upgrade, it is possible to roll back to the previous release. To allow for the possibility of a roll back and avoid any potential data loss, Datameer X strongly recommends disabling the Housekeeping and Compaction services before beginning the upgrade process. Once upgrade validation is complete and the environment is considered stable, the Housekeeping and Compaction services can be re-enabled.

  1. Open the /<Datameer X new version installation directory>/conf/default.properties file.
  2. Disable Housekeeping by setting the property housekeeping.enabled to false.
  3. Disable Compaction by setting the property auto-compaction.enabled to false.
  4. Start Datameer X and perform upgrade validation testing.
  5. Once the upgrade validated and is considered successful, turn Housekeeping and Compaction back on by setting the above mentioned properties back to true.

...

To validate whether any Workbook has been broken during an upgrade, Datameer X created the Workbook Health Check feature.  The tool reviews a Workbook's structure and reports any logical issues. This tool is available in versions 6.4.14, 7.1.13, 7.2.13, 7.4.11+, 7.5.4+, and 10.0.1+.  Datameer's best practice recommendations for upgrades are as follows:

    • Upgrade within the current branch to the version where the Workbook Health Check tool is available.  For Example, if the current version is 6.4.13, upgrade to 6.4.14.
    • Navigate http://<Datameer X host>:<port>/dev (admin rights are required), then click on Workbook Health Check and press Start button.
    • As soon as validation is complete (execution time varies based on the number and complexity of the existing Workbooks), you will receive a status summary and the broken Workbooks configuration IDs. The same information will also be written into Datameer conductor.log together with the error for every artifact. You might want to review broken Workbooks and fix the errors before the upgrade.  Note, a session timeout will interrupt the workbook validation check.
    • Upgrade to the desired Datameer X version.
    • Rerun the Workbook Health Check and ensure that there are no new broken items. In case there are Workbooks that have been broken by the upgrade, check the error messages contained in conductor.log and decide whether you could fix them manually or need to roll back and get Datameer X support assistance on the issue.

Backup Keyfiles and Keystore

If you have set up password encryption and/or enabled TLS  with custom certificates , backup your Keystore and Keyczar keyfiles.

...

No Format
#Detect databases
detect old database [/Users/marko/Desktop/Datameer-2.0.1-0.20.2/das-data/database/hsql-db]
detect new database [/Users/marko/Development/Java/datameer/dap/build/dist/Datameer-2.0.2-0.20.2/das-data/database/hsql-db]

Upgrade datameer database version [2.0.0.6] to version [2.0.2.0]

#Assert datameer is running
No

#Copy Das Data
copy das-data from [/Users/marko/Desktop/Datameer-2.0.1-0.20.2/das-data] to [/Users/marko/Development/Java/datameer/dap/build/dist/Datameer-2.0.2-0.20.2/das-data]

#Upgrade starts...
Process upgrade scripts on database jdbc:hsqldb:file:das-data/database/hsql-db
	upgrade-2.0.1.hsql

#Update schema-version
set system.schemaVersion on database [/Users/marko/Development/Java/datameer/dap/build/dist/Datameer-2.0.2-0.20.2/das-data/database/hsql-db] to [2.0.1]


#Update data uri's [/Users/marko/Desktop/Datameer-2.0.1-0.20.2/das-data] -> [/Users/marko/Development/Java/datameer/dap/build/dist/Datameer-2.0.2-0.20.2/das-data]

log4j:WARN No appenders could be found for logger (org.hibernate.type.BasicTypeRegistry).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Converting 12 data entries
file:/Users/marko/Desktop/Datameer-2.0.1-0.20.2/das-data/fileuploads/1/1 -> file:/Users/marko/Development/Java/datameer/dap/build/dist/Datameer-2.0.2-0.20.2/das-data/fileuploads/1/1
file:/Users/marko/Desktop/Datameer-2.0.1-0.20.2/das-data/fileuploads/2/2 -> file:/Users/marko/Development/Java/datameer/dap/build/dist/Datameer-2.0.2-0.20.2/das-data/fileuploads/2/2
file:/Users/marko/Desktop/Datameer-2.0.1-0.20.2/das-data/fileuploads/3/3 -> file:/Users/marko/Development/Java/datameer/dap/build/dist/Datameer-2.0.2-0.20.2/das-data/fileuploads/3/3
file:/Users/marko/Desktop/Datameer-2.0.1-0.20.2/das-data/fileuploads/4/4 -> file:/Users/marko/Development/Java/datameer/dap/build/dist/Datameer-2.0.2-0.20.2/das-data/fileuploads/4/4
file:/Users/marko/Desktop/Datameer-2.0.1-0.20.2/das-data/fileuploads/5/5 -> file:/Users/marko/Development/Java/datameer/dap/build/dist/Datameer-2.0.2-0.20.2/das-data/fileuploads/5/5
file:/Users/marko/Desktop/Datameer-2.0.1-0.20.2/das-data/workbooks/6/6 -> file:/Users/marko/Development/Java/datameer/dap/build/dist/Datameer-2.0.2-0.20.2/das-data/workbooks/6/6
file:/Users/marko/Desktop/Datameer-2.0.1-0.20.2/das-data/workbooks/7/7 -> file:/Users/marko/Development/Java/datameer/dap/build/dist/Datameer-2.0.2-0.20.2/das-data/workbooks/7/7
file:/Users/marko/Desktop/Datameer-2.0.1-0.20.2/das-data/workbooks/8/8 -> file:/Users/marko/Development/Java/datameer/dap/build/dist/Datameer-2.0.2-0.20.2/das-data/workbooks/8/8
file:/Users/marko/Desktop/Datameer-2.0.1-0.20.2/das-data/workbooks/9/9 -> file:/Users/marko/Development/Java/datameer/dap/build/dist/Datameer-2.0.2-0.20.2/das-data/workbooks/9/9
file:/Users/marko/Desktop/Datameer-2.0.1-0.20.2/das-data/fileuploads/10/10 -> file:/Users/marko/Development/Java/datameer/dap/build/dist/Datameer-2.0.2-0.20.2/das-data/fileuploads/10/10
file:/Users/marko/Desktop/Datameer-2.0.1-0.20.2/das-data/workbooks/11/11 -> file:/Users/marko/Development/Java/datameer/dap/build/dist/Datameer-2.0.2-0.20.2/das-data/workbooks/11/11
file:/Users/marko/Desktop/Datameer-2.0.1-0.20.2/das-data/fileuploads/10/12 -> file:/Users/marko/Development/Java/datameer/dap/build/dist/Datameer-2.0.2-0.20.2/das-data/fileuploads/10/12

Check or Update YARN Path Settings

 Check or update YARN application classpath and LD_LIBRARY path.  The  path settings are used to lookup and load libraries , therefor  incorrect path settings may lead into unexpected behavior .

...

Note
iconfalse

Example:

When upgrading from CDH 4.x to CDH 5.x the classpath was renamed.
 

Code Block
$HADOOP_CONF_DIR, $HADOOP_COMMON_HOME/*, $HADOOP_COMMON_HOME/lib/*, $HADOOP_HDFS_HOME/*, $HADOOP_HDFS_HOME/lib/*, $HADOOP_MAPRED_HOME/*, $HADOOP_MAPRED_HOME/lib/*, $YARN_HOME/*, $YARN_HOME/lib/*

to

Code Block
 $HADOOP_CONF_DIR, $HADOOP_COMMON_HOME/*, $HADOOP_COMMON_HOME/lib/*, $HADOOP_HDFS_HOME/*, $HADOOP_HDFS_HOME/lib/*, $HADOOP_MAPRED_HOME/*, $HADOOP_MAPRED_HOME/lib/*, $HADOOP_YARN_HOME/*, $HADOOP_YARN_HOME/lib/*


The correct YARN classpath can be found for the Datameer X client via command:

Code Block
yarn classpath | tr ':' ','


Retrigger the Search Index

Info
titleINFO

The search settings to enable file search within the 'File Browser' must be retriggered manually after the upgrade.

Recompile Plug-ins

Custom plug-Ins which were created by using the plug-in SDK need to be recompiled. Adjust the versioning and create the plug-in again under the new plug-in SDK.

...


Panel
titleAttention
  • Fields that were made unnecessary on the AD/LDAP configuration page have been removed.

  • There is no longer a radio button needed to switch between AD and LDAP.

  • The difference between AD and LDAP can be configured in field input.

    • Username Attribute: uid (LDAP), sAMAccountName (AD)

    • Group Definition: objectClass=groupOfNames (LDAP), objectClass=group (AD)
  • There is no longer a drop down box to switch between simple and custom group definition. (LDAP only)

Removing Extraneous Properties After Upgrade Adjustments

The support for MapReduce has been deprecated and was removed in Datameer in Datameer X 6.3. After it is verified that all jobs work without the MapReduce framework, remove MapReduce related custom properties from the Hadoop cluster page under the Administration tab and in the advanced settings of data links, import jobs, and/or workbooks.

Anchor
rollback
rollback
How to Roll Back to a Previous Database Version

When upgrading, a database dump is created by Datameer X the file name is similar to mysql_db_dump_<oldVer>.sql. If an upgrade failed (failing upgrade scripts / failing upgrade injectors during startup), the database paths might be conflicting, it is therefore necessary to restore the Datameer X MySQL database dump. Follow the steps to complete this task.

  1. Drop the new database, for example on the database server or the MySQL CLI.

    No Format
    DROP DATABASE dap;

     

  2. Using the mysql-init.sql script, create a new database.

    No Format
    mysql [-h <dbhost>] -u root -p < bin/mysql-init.sql

     

  3. Import and restore the previous database using the following shell command.

    No Format
    mysql [-h <dbhost>] -u root -p dap < /path/to/dump_file_name


...