If you have a completely new Datameer X installation, upgrading Datameer X is not necessary. |
The following is recommended before the upgrade is executed:
Make sure you have at least Datameer X version 10.1 before upgrading to Datameer X version 11.
Sometimes it is necessary to update in several release version steps, e.g. from Datameer. Updating from Datameer X versions lower than version 10.1 must first be upgraded to the latest Datameer X version 10.1.x. Upgrading directly to Datameer X version 11 is only from version 10.1 possible. |
Symlinks that were created through the Datameer X installation must be updated through the updating process. |
To update the symlink:
Remove the existing symlink:
rm current |
Create a new symlink and change the working directory:
ln -s Datameer-<package> current cd current |
Housekeeping and Compaction services must be disabled to allow for a possible roll back and to avoid any potential data loss. |
To disable the Housekeeping and Compaction services:
housekeeping.enabled
to 'housekeeping.enabled=false'.auto-compaction.enabled'
to 'auto--compaction=false'.Workbooks and other JSON files downloaded as a backup in older versions of Datameer X are not supported in newer versions of Datameer. When upgrading to a newer version of Datameer X ensure that all needed workbooks and files are part of the migration process so that they can be used in your new version of Datameer. |
Execute a Workbook Health Check to detect broken workbooks and review a workbook's structure and report any logical issues. The Workbook Health Check is available as of 11.0.0 for Datameer X 11. |
If you have set up password encryption and/or enabled SSL with custom certificates , backup your Keystore and Keyczar keyfiles.
The upgrading process consists of two parts:
|
Upgrading the Application
Updating Datameer X using user/ group 'root' is recommended to change the permissions to the user/ group 'datameer' for security reasons. |
View current running or queued jobs and pause the job scheduler so that jobs are not submitted to the cluster. |
INFO: Do not copy the entire old 'conf/' directory or 'conductor.sh' script to the new location. You only need to make changes in the new file for the changes you made previously. |
To upgrade the application:
Stop the Datameer X application gracefully.
<Datameer X Application Folder>/bin/conductor.sh stop |
Unzip the upgrade file and move the MySQL JDBC driver located in <Datameer X path>/das-data/jdbc-jars to the new installation's <New Datameer X path>/das-data/jdbc-jars> folder.
export DAS_DEPLOY_MODE=live |
When upgrading in Workgroup or Enterprise versions, consider allocating additional memory in 'etc/das-env.sh'. Make the necessary changes to the 'etc/das-env.sh' file.
INFO: Memory requirements and the main JAVA options have not changed in Datameer X version 10., so the values introduced in the previous version could be re-used.
# Adjust max available memory (-Xmx) according to your needs # WARNING: Path variables that may contain blanks should be added to jetty.sh start_das() method (see DAP-6342) export JAVA_OPTIONS="-Xmx2048m -XX:MaxPermSize=384m -Xms256m -XX:MaxNewSize=448m -XX:SurvivorRatio=6 -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=80 -XX:+HeapDumpOnOutOfMemoryError -XX:+CMSClassUnloadingEnabled -XX:+CMSPermGenSweepingEnabled" export JAVA_OPTIONS="$JAVA_OPTIONS -Dfile.encoding=utf-8" |
If needed, change the container sizing of the Map, Reduce and AM containers which respectively correspond to MapReduce and Tez jobs. Therefore change the settings of the properties:
das.job.map-task.memory=2048 das.job.reduce-task.memory=2048 das.job.application-manager.memory=2048 |
If existing, copy the files from the 'das-data' folder of the old distribution to the new location.
INFO: Keep the original 'das-data' information as a backup.
cp -r /<old-location>/das-data /<new-location>/ |
If needed, update the folder in which plug-in configurations are stored. Therefore open the 'default.properties' file from the file system of the previous Datameer X version, search for 'system.property.plugin.configs.dir=<folder name>' and open the 'default.properties' file from the file system of the Datameer X upgrade version. Now check or update the new property value to be the same folder name in the previous version and if needed, copy the contents of the previous plug-in configurations folder into the file system of the new upgraded version of Datameer.
INFO: Plug-in zip files from an older version are not compatible with newer versions. Request updated plugin versions from support or your Technical Account Manager.
# Defines the folder where to store plugin configurations system.property.plugin.configs.dir=<folder name> |
If needed, copy over the native libraries that you have added.
INFO: You don't have to copy over the native libraries that are already bundled with Datameer.
cp -r /<old-location>/lib/native/* /<new-location>/lib/native/ |
Migrate the SSL values from the previous 'start.ini' to the new one.
# Set up a demonstration keystore and truststore jetty.keystore=etc/keystore jetty.truststore=etc/keystore # Sets the demonstration passwords # OBF passwords aren't secure. They are only protected from casual observation. # See http://www.eclipse.org/jetty/documentation/current/configuring-security-secure-passwords.html jetty.keystore.password=OBF:1vny1zlo1x8e1vnw1vn61x8g1zlu1vn4 # storepwd jetty.keymanager.password=OBF:1u2u1wml1z7s1z7a1wnl1u2g # keypwd jetty.truststore.password=OBF:1vny1zlo1x8e1vnw1vn61x8g1zlu1vn4 # storepwd |
The following applies as prerequisites:
|
To upgrade the database, use the script provided in the Datameer X distribution installation folder.
./bin/database.sh upgrade
" from inside the distribution installation folder../bin/upgrade_db.sh
" from inside the distribution installation folder.The upgrade script will automatically:
./das-data/backups/
" while the "upgrade-db" script stores it in "/bin/upgrader
")./logs/database-tool.log
" while "upgrade-db" logs to "./bin/upgrader/logs
")The Datameer X upgrade is now completed and you can launch the application.
If you receive an error message that the thread stack overrun during an upgrade, raise the MySQL stack size. Example:
or higher if necessary. |
This applies only in case the Hadoop cluster has been upgraded. |
Path settings are used to lookup and load libraries. Therefore incorrect path settings may lead into unexpected behavior. |
The correct YARN application class path can be found via command:
|
To check or update the path settings:
Start the Datameer X application and log in. The Datameer X UI opens.
The search settings to enable file search within the 'File Browser' are retriggered automatically after the upgrade. |
If you need to enable SSL have a look at the SSL configuration page. |
After the application and database upgrade has completed, perform a test to validate a successful update. |
Ask your Datameer X users to review their most critical use-cases and ensure that they work as expected after the upgrade. Execute a Workbook Health Check again to detect broken workbooks and review a workbook's structure and report any logical issues. In case there are workbooks that have been broken by the upgrade, check the error messages contained in 'conductor.log' and decide whether you can fix them manually or need to roll back and get Datameer X support assistance on the issue. |
|
To perform the validation:
Custom plug-ins that were created by using the 'Plug-in SDK' need to be recompiled - therefore, adjust the versioning and create the plug-in again under the new 'Plug-in SDK' |
To recompile the plug-ins:
Housekeeping and Compaction services can be enabled after the upgrade. |
To enable Housekeeping/ Compaction services:
housekeeping.enabled'
to 'housekeeping.enabled=true'.auto-compaction.enabled'
to 'auto–compaction=true'.