Page Comparison

Upgrading From an Older Release

Datameer customers

Upgrading From an Older Release

Datameer customers can download updated software versions from from http://my.datameer.com

To upgrade from an older release, you should first back up your data. Then you will need to upgrade the application and upgrade the database.

...

When upgrading to a newer version of Datameer, ensure that all needed workbooks and files are part of the migration process so that they can be used in your new version of Datameer.

Anchor
planning planning

Upgrade Planning

• Before upgrading Datameer, a plan should be made for application downtime scheduling, notifications, and creating a maintenance window for Datameer. Please take note of the system requirements and consider upgrading the MySQL service for Datameer's application database from 5.1 to 5.5 or 5.6, if applicable.
• Backed up Datameer files need to be included in the migration process. Previous files that aren't part of the migration process aren't supported in newer versions.
• Jobs will need to be set to stop executing prior to the upgrade process. This could be done e.g. by Pause Job Scheduler.
• Request possible assistance from the database administrator in case credentials for database are not available to the application owner.
• Custom plug-ins which were created by Using the Plug-in SDK of a former Datameer version need to be re-compiled!
• Since it is recommended to not copy old configuration files or scripts to the new location, note the changes you have made in the previous setup. This information you will need to make necessary changes in the new configuration files.
  
  • As of Datameer 7.4, a property file exists so Job Scheduler and Event Bus settings adjusted Bus settings adjusted in the UI are read during Datameer start up.
    • A properties file called "overrides.properties" isn't written on Datameer but your systems HOME folder. (Path = <home>/.datameer/overrides.properties)
    • This property file is auto-created when adjusting the Event Bus or Job Scheduler settings under the Admin tab. This file can also be manually added and edited.
    • This is the last properties file properties  file read on start up. (E.g., it overrides other property files like default.properties and deployMode.properties) 
    • This file can be modified to allow for storing and the restoring of custom properties.

Disabling Housekeeping/Compaction services

...

Restarting Datameer is required to apply these changes.

Backup Keyfiles and Keystore

If you have set up password encryption and/or enabled TLS  with custom certificates

Anchor
validation validation

Workbook Validation

A Workbook is a complex structure that Datameer is constantly striving to improve.  Workbooks can be built in a near unlimited number of different combinations of Column Renames, Joins, Unions, Filters, and Functions - including user built functions.  While we do attempt to cover all the possibilities while implementing our upgrade code, some customer development decisions are unpredictable and as a result our Workbook upgrade code can not always successfully transform all workbooks.

To validate whether any Workbook has been broken during an upgrade, Datameer created the Workbook Health Check feature.  The tool reviews a Workbook's structure and reports any logical issues. This tool is available in versions 6.4.14, 7.1.13, 7.2.13, 7.4.11+, 7.5.4+, and 10.0.1+.  Datameer's best practice recommendations for upgrades are as follows:

• • Upgrade within the current branch to the version where the Workbook Health Check tool is available.  For Example, if the current version is 6.4.13, upgrade to 6.4.14.
  • Navigate http://<Datameer host>:<port>/dev (admin rights are required), then click on Workbook Health Check and press Start button.
  • As soon as validation is complete (execution time varies based on the number and complexity of the existing Workbooks), you will receive a status summary and the broken Workbooks configuration IDs. The same information will also be written into Datameer conductor.log together with the error for every artifact. You might want to review broken Workbooks and fix the errors before the upgrade.  Note, a session timeout will interrupt the workbook validation check.
  • Upgrade to the desired Datameer version.
  • Rerun the Workbook Health Check and ensure that there are no new broken items. In case there are Workbooks that have been broken by the upgrade, check the error messages contained in conductor.log and decide whether you could fix them manually or need to roll back and get Datameer support assistance on the issue.

Backup Keyfiles and Keystore

If you have set up password encryption and/or enabled TLS  with custom certificates , backup your Keystore and Keyczar keyfiles.

Upgrade the Application

1. Stop the application.

   Panel
   <Datameer Application Folder>/bin/conductor.sh stop

2. Unzip the upgrade file. 
   Move the MySQL JDBC driver located in JDBC driver located in <Datameer path>/das-data/jdbc-jars, as described into the  Install the JDBC Database Drivers section of the the Installation Guide.
   In case you are using Datameer with MySQL as an application database, please check that database mode is configured accordingly in your  etc/das-env.sh  file

   No Format
   export DAS_DEPLOY_MODE=live

3. If upgrading in Workgroup or Enterprise versions, users may want to consider allocating additional memory in etc/das-env.sh. 

   Panel

   # Adjust max available memory (-Xmx) according to your needs # WARNING: Path variables that may contain blanks should be added to jetty.sh start_das() method (see DAP-6342) export JAVA_OPTIONS="-Xmx2048m -XX:MaxPermSize=384m -Xms256m -XX:MaxNewSize=448m -XX:SurvivorRatio=6 -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=80 -XX:+HeapDumpOnOutOfMemoryError -XX:+CMSClassUnloadingEnabled -XX:+CMSPermGenSweepingEnabled" export JAVA_OPTIONS="$JAVA_OPTIONS -Dfile.encoding=utf-8"

4. Consider changing the container sizing of the Map, Reduce, and AM containers, which respectively correspond to MapReduce and Tez jobs. To do so, change the settings of the following properties:

   Panel
   das.job.map-task.memory=2048 das.job.reduce-task.memory=2048 das.job.application-manager.memory=2048

5. If existing, copy the files from from the das-data folder of the old distribution to the new location. Keep the original original das-data information  information as a backup.

   No Format
   cp -r /<old-location>/das-data /<new-location>/

6. Users need to check, and update if necessary, the folder where plug-in configurations are stored.

   Open the file system of the previous Datameer version you are upgrading from and navigate to the the default.properties file file.

   Search for the property file that defines the folder where plug-in configurations are stored.

   Panel
   # Defines the folder where to store plugin configurations system.property.plugin.configs.dir=<folder name>

   Open the file system of the upgrade version of Datameer and find the same property in the the default.properties file file.

   Check or update the new property value to be the same folder name in the previous version.

   Copy the contents of the previous plug-in configurations folder into the file system of the new upgraded version of Datameer.

7. If you made changes in your your conf/ directory previously (like changing changing live.properties and  and log4j-<xyz>.properties), you need to apply** the same changes to the new new conf/ directory       directory of your upgraded instance.

8. Copy over the native libraries that you have added to Datameer (if this applies). You don't have to copy over the native libraries that are already bundled with Datameer.

   No Format
   cp -r /<old-location>/lib/native/* /<new-location>/lib/native/

9. If you have made changes to the the conductor.sh script  script (for example, to enable SSH), apply** these changes to /<new-location>/bin/conductor.sh again again. Notice JAVA_OPTIONS has been moved to to etc/das-env.sh

10. Migrate the SSL the SSL values from the previous previous start.ini to  to the new one.

    Panel

    # Set up a demonstration keystore and truststore jetty.keystore=etc/keystore jetty.truststore=etc/keystore # Sets the demonstration passwords # OBF passwords aren't secure. They are only protected from casual observation. # See http://www.eclipse.org/jetty/documentation/current/configuring-security-secure-passwords.html jetty.keystore.password=OBF:1vny1zlo1x8e1vnw1vn61x8g1zlu1vn4 # storepwd jetty.keymanager.password=OBF:1u2u1wml1z7s1z7a1wnl1u2g # keypwd jetty.truststore.password=OBF:1vny1zlo1x8e1vnw1vn61x8g1zlu1vn4 # storepwd

11. Copy over files from from etc/custom-jars (these files could be database drivers or 3rd party libraries).
12. If you are using custom Datameer plugins (jar archives stored under under etc/custom-plugins), update those to be in sync (API compatible) with the new version of Datameer and be sure to remove jar files related to older versions from from etc/custom-plugins.

...

1. To upgrade the database, enter the following command:

   No Format
   bin/upgrade_db.sh

   If either the structure of the MySQL database or any of the tables in the database will be modified by the Datameer upgrade, running this command automatically creates a dump of the database.

   Note
   Ensure that the script is not executed under root, but by the same user account that starts and stops Datameer. Run the upgrade script directly from the <INSTALLDIR>. Preface the script with the bin directory.

2. If your database name is different than the default default dap or the database runs on a different server or customized port, you will need to provide the additional arguments:

   No Format
   bin/upgrade_db.sh -h [db hostname] -o [db port] -n [db name] -u [db username] -p [db password]

3. Check to ensure your database character encoding is set to UTF-8.
   In the Database, enter:

   No Format
   SELECT DEFAULT_CHARACTER_SET_NAME FROM INFORMATION_SCHEMA.SCHEMATA WHERE SCHEMA_NAME = '<The name of your Datameer database>';

   If UTF8 was returned, open the default properties and comment in the setting system.property.hibernate.connection.characterEncoding=utf8.
   If something other than UTF8 was returned, change the database character encoding setting to UTF8 or special characters may show as corrupt. 

4. To finish the upgrade, restart the application.
   

   Insert excerpt
   _startingDatameer _startingDatameer nopanel true

MySQL thread stack overrun during database upgrade

If you receive the above error message during an update, raise the MySQL stack size. For example,

mysqld -O thread_stack=192k

or higher if necessary.

Migrating from HSQL to MySQL

Datameer provides a tool to migrate data from an HSQL file database to a MySQL database:

1. Set-up MySQL as per the the Installation Guide

2. Migrate HSQL database to MySQL

   Code Block
   bin/migrate-db-tool.sh hsql-file:<Datameer user path>/Datameer/<version>/das-data/database mysql

   * Ensure this script is being run from root Datameer installation folder.

Upgrade the HSQL database

You have to use the script script bin/upgrade_hsql_db.sh to  to upgrade your HSQL database to a higher version. Ensure this script is run from the root Datameer installation folder.

usage: upgrade_hsql_db.sh
 -ndd,--new-das-data <folder>           new das-data folder
 -odd,--old-das-data <folder>           old das-data folder

With the the -pi option you define the previous datameer installation. We need this folder to find the HSQL database we want to upgrade. 

We backup your previous database into your current installation folder. Furthermore we use all upgrade scripts with name pattern  upgrade-<version>.hsql hsql in your current installation folder.
As an option you can use the parameters parameters -ndd and  and -odd . These parameters define the new/old location of your your das-data folder folder. When you running datameer in LOCAL mode it could be possible that you want to copy your your das-data folder  folder to a new location. A common use case is when the the das-data folder  folder is saved within the installation directory.

...

#Detect databases
detect old database [/Users/marko/Desktop/Datameer-2.0.1-0.20.2/das-data/database/hsql-db]
detect new database [/Users/marko/Development/Java/datameer/dap/build/dist/Datameer-2.0.2-0.20.2/das-data/database/hsql-db]

Upgrade datameer database version [2.0.0.6] to version [2.0.2.0]

#Assert datameer is running
No

#Copy Das Data
copy das-data from [/Users/marko/Desktop/Datameer-2.0.1-0.20.2/das-data] to [/Users/marko/Development/Java/datameer/dap/build/dist/Datameer-2.0.2-0.20.2/das-data]

#Upgrade starts...
Process upgrade scripts on database jdbc:hsqldb:file:das-data/database/hsql-db
	upgrade-2.0.1.hsql

#Update schema-version
set system.schemaVersion on database [/Users/marko/Development/Java/datameer/dap/build/dist/Datameer-2.0.2-0.20.2/das-data/database/hsql-db] to [2.0.1]


#Update data uri's [/Users/marko/Desktop/Datameer-2.0.1-0.20.2/das-data] -> [/Users/marko/Development/Java/datameer/dap/build/dist/Datameer-2.0.2-0.20.2/das-data]

log4j:WARN No appenders could be found for logger (org.hibernate.type.BasicTypeRegistry).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Converting 12 data entries
file:/Users/marko/Desktop/Datameer-2.0.1-0.20.2/das-data/fileuploads/1/1 -> file:/Users/marko/Development/Java/datameer/dap/build/dist/Datameer-2.0.2-0.20.2/das-data/fileuploads/1/1
file:/Users/marko/Desktop/Datameer-2.0.1-0.20.2/das-data/fileuploads/2/2 -> file:/Users/marko/Development/Java/datameer/dap/build/dist/Datameer-2.0.2-0.20.2/das-data/fileuploads/2/2
file:/Users/marko/Desktop/Datameer-2.0.1-0.20.2/das-data/fileuploads/3/3 -> file:/Users/marko/Development/Java/datameer/dap/build/dist/Datameer-2.0.2-0.20.2/das-data/fileuploads/3/3
file:/Users/marko/Desktop/Datameer-2.0.1-0.20.2/das-data/fileuploads/4/4 -> file:/Users/marko/Development/Java/datameer/dap/build/dist/Datameer-2.0.2-0.20.2/das-data/fileuploads/4/4
file:/Users/marko/Desktop/Datameer-2.0.1-0.20.2/das-data/fileuploads/5/5 -> file:/Users/marko/Development/Java/datameer/dap/build/dist/Datameer-2.0.2-0.20.2/das-data/fileuploads/5/5
file:/Users/marko/Desktop/Datameer-2.0.1-0.20.2/das-data/workbooks/6/6 -> file:/Users/marko/Development/Java/datameer/dap/build/dist/Datameer-2.0.2-0.20.2/das-data/workbooks/6/6
file:/Users/marko/Desktop/Datameer-2.0.1-0.20.2/das-data/workbooks/7/7 -> file:/Users/marko/Development/Java/datameer/dap/build/dist/Datameer-2.0.2-0.20.2/das-data/workbooks/7/7
file:/Users/marko/Desktop/Datameer-2.0.1-0.20.2/das-data/workbooks/8/8 -> file:/Users/marko/Development/Java/datameer/dap/build/dist/Datameer-2.0.2-0.20.2/das-data/workbooks/8/8
file:/Users/marko/Desktop/Datameer-2.0.1-0.20.2/das-data/workbooks/9/9 -> file:/Users/marko/Development/Java/datameer/dap/build/dist/Datameer-2.0.2-0.20.2/das-data/workbooks/9/9
file:/Users/marko/Desktop/Datameer-2.0.1-0.20.2/das-data/fileuploads/10/10 -> file:/Users/marko/Development/Java/datameer/dap/build/dist/Datameer-2.0.2-0.20.2/das-data/fileuploads/10/10
file:/Users/marko/Desktop/Datameer-2.0.1-0.20.2/das-data/workbooks/11/11 -> file:/Users/marko/Development/Java/datameer/dap/build/dist/Datameer-2.0.2-0.20.2/das-data/workbooks/11/11
file:/Users/marko/Desktop/Datameer-2.0.1-0.20.2/das-data/fileuploads/10/12 -> file:/Users/marko/Development/Java/datameer/dap/build/dist/Datameer-2.0.2-0.20.2/das-data/fileuploads/10/12

Check or Update YARN Path Settings

 Check  Check or update YARN application classpath and LD_LIBRARY path.  The    The  path settings are used to lookup and load libraries , therefor therefor  incorrect path settings may lead into unexpected behavior .

1. Start Datameer

2. Open the Administration tab

3. Open the Hadoop Cluster options

4. Check or update YARN application classpath and LD_LIBRARY path

    

$HADOOP_CONF_DIR, $HADOOP_COMMON_HOME/*, $HADOOP_COMMON_HOME/lib/*, $HADOOP_HDFS_HOME/*, $HADOOP_HDFS_HOME/lib/*, $HADOOP_MAPRED_HOME/*, $HADOOP_MAPRED_HOME/lib/*, $YARN_HOME/*, $YARN_HOME/lib/*

 $HADOOP_CONF_DIR, $HADOOP_COMMON_HOME/*, $HADOOP_COMMON_HOME/lib/*, $HADOOP_HDFS_HOME/*, $HADOOP_HDFS_HOME/lib/*, $HADOOP_MAPRED_HOME/*, $HADOOP_MAPRED_HOME/lib/*, $HADOOP_YARN_HOME/*, $HADOOP_YARN_HOME/lib/*

yarn classpath | tr ':' ','

Retrigger the Search Index

Recompile Plug-ins

Custom plug-Ins which were created by using the plug-in SDK need to be recompiled. Adjust the versioning and create the plug-in again under the new plug-in SDK.

...

If no unexpected errors occur the upgrade has been validated.

Removing Extraneous Properties After Upgrade Adjustments

The support for MapReduce has been deprecated and was removed in Datameer 6.3. After it is verified that all jobs work without the MapReduce framework, remove MapReduce related custom properties from the Hadoop cluster page under the Administration tab and in the advanced settings of data links, import jobs, and/or workbooks.

Anchor
rollback rollback

How to Roll Back to a Previous Database Version

When When upgrading, a database dump is created by Datameer, the file name is similar to to mysql_db_dump_<oldVer>.sql. If an upgrade failed (failing upgrade scripts / failing upgrade injectors during startup), the database paths might be conflicting, it is therefore necessary to restore the Datameer MySQL database dump. Follow the steps to complete this task.

1. Drop the new database, for example on the database server or the MySQL CLI.

   No Format
   DROP DATABASE dap;

    

2. Using the the mysql-init.sql script script, create a new database.
   

   No Format
   mysql [-h <dbhost>] -u root -p < bin/mysql-init.sql

    

3. Import and restore the previous database using the following shell command.
   

   No Format
   mysql [-h <dbhost>] -u root -p dap < /path/to/dump_file_name

...