Versions Compared

Version Old Version 11 New Version 12
Changes made by

Charles Bishop (Deactivated)

Charles Bishop (Deactivated)

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Table of Contents

...

If upgrading from a Datameer version before 6.1, we highly recommend first upgrading to Datameer 6.1 and verifying that all job functions work without the MapReduce framework before upgrading to Datameer 6.3.1  and /wiki/spaces/DAS60/pages/4631169606  

In 6.3, Datameer uses Parquet as its default storage format. While Datameer can still read data written in the sequence file format, all artifacts are now stored using the Parquet file format. Datameer still uses the sequence file format for writing intermediate files and preview files for performance reasons, but the final artifacts are saved using the Parquet format. Due to this change, it might not be possible to roll back after upgrading to Datameer 6.3, depending on the configuration of your artifacts. For example, if import jobs and workbooks are configured to purge historical data and only keep the last one result, the final artifacts are stored as Parquet files, and prior versions of Datameer cannot read them. 

...

  • If you used Kerberos prior to 5.11, you need to install the plug-in during upgrade.
  • If you are running a Kerberos-secured cluster with impersonation enabled, you need to run the secure_hdfs_tool.sh command line tool when upgrading to versions 6 and above. For versions 6.1 and above, run the command after upgrading, starting, and stopping Datameer. Once you've run the command, restart Datameer.

Anchor
planning
planning
Upgrade Planning

  • Before upgrading Datameer, a plan should be made for application downtime scheduling, notifications, and creating a maintenance window for Datameer. Please take note of the system requirements and consider upgrading the MySQL service for Datameer's application database from 5.1 to 5.5 or 5.6, if applicable.
  • Jobs will need to be set to stop executing prior to the upgrade process. This could be done e.g. by Pause Job Scheduler.
  • Request possible assistance from the database administrator in case credentials for database are not available to the application owner.
  • Custom Plug-Ins which were created by Using the Plug-in SDK of a former Datameer version needs need to be re-compiled!
  • Since it is recommended to not copy old configuration files or scripts to the new location, note the changes you have made in the previous setup. This information you will need to make necessary changes in the new configuration files.
    • As of Datameer 7.4, a property file exists so Job Scheduler and Event Bus settings adjusted in the UI are read during Datameer start up.
      • A properties file called "overrides.properties" isn't written on Datameer but your systems HOME folder. (Path = <home>/.datameer/overrides.properties)
      • This property file is auto-created when adjusting the Event Bus or Job Scheduler settings under the Admin tab. This file can also be manually added and edited.
      • This is the last properties  file read on start up. (E.g., it overrides other property files like default.properties and deployMode.properties)
      • This file can be modified to allow for storing and the restoring of custom properties.

Backup Keyfiles and Keystore

...

Follow the steps for upgrading your MySQL or HSQL database. As of Datameer 7.4: MariaDB is supported as an alternative to MySQL.

Upgrade the MySQL database

...

No Format
#Detect databases
detect old database [/Users/marko/Desktop/Datameer-2.0.1-0.20.2/das-data/database/hsql-db]
detect new database [/Users/marko/Development/Java/datameer/dap/build/dist/Datameer-2.0.2-0.20.2/das-data/database/hsql-db]

Upgrade datameer database version [2.0.0.6] to version [2.0.2.0]

#Assert datameer is running
No

#Copy Das Data
copy das-data from [/Users/marko/Desktop/Datameer-2.0.1-0.20.2/das-data] to [/Users/marko/Development/Java/datameer/dap/build/dist/Datameer-2.0.2-0.20.2/das-data]

#Upgrade starts...
Process upgrade scripts on database jdbc:hsqldb:file:das-data/database/hsql-db
	upgrade-2.0.1.hsql

#Update schema-version
set system.schemaVersion on database [/Users/marko/Development/Java/datameer/dap/build/dist/Datameer-2.0.2-0.20.2/das-data/database/hsql-db] to [2.0.1]


#Update data uri's [/Users/marko/Desktop/Datameer-2.0.1-0.20.2/das-data] -> [/Users/marko/Development/Java/datameer/dap/build/dist/Datameer-2.0.2-0.20.2/das-data]

log4j:WARN No appenders could be found for logger (org.hibernate.type.BasicTypeRegistry).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Converting 12 data entries
file:/Users/marko/Desktop/Datameer-2.0.1-0.20.2/das-data/fileuploads/1/1 -> file:/Users/marko/Development/Java/datameer/dap/build/dist/Datameer-2.0.2-0.20.2/das-data/fileuploads/1/1
file:/Users/marko/Desktop/Datameer-2.0.1-0.20.2/das-data/fileuploads/2/2 -> file:/Users/marko/Development/Java/datameer/dap/build/dist/Datameer-2.0.2-0.20.2/das-data/fileuploads/2/2
file:/Users/marko/Desktop/Datameer-2.0.1-0.20.2/das-data/fileuploads/3/3 -> file:/Users/marko/Development/Java/datameer/dap/build/dist/Datameer-2.0.2-0.20.2/das-data/fileuploads/3/3
file:/Users/marko/Desktop/Datameer-2.0.1-0.20.2/das-data/fileuploads/4/4 -> file:/Users/marko/Development/Java/datameer/dap/build/dist/Datameer-2.0.2-0.20.2/das-data/fileuploads/4/4
file:/Users/marko/Desktop/Datameer-2.0.1-0.20.2/das-data/fileuploads/5/5 -> file:/Users/marko/Development/Java/datameer/dap/build/dist/Datameer-2.0.2-0.20.2/das-data/fileuploads/5/5
file:/Users/marko/Desktop/Datameer-2.0.1-0.20.2/das-data/workbooks/6/6 -> file:/Users/marko/Development/Java/datameer/dap/build/dist/Datameer-2.0.2-0.20.2/das-data/workbooks/6/6
file:/Users/marko/Desktop/Datameer-2.0.1-0.20.2/das-data/workbooks/7/7 -> file:/Users/marko/Development/Java/datameer/dap/build/dist/Datameer-2.0.2-0.20.2/das-data/workbooks/7/7
file:/Users/marko/Desktop/Datameer-2.0.1-0.20.2/das-data/workbooks/8/8 -> file:/Users/marko/Development/Java/datameer/dap/build/dist/Datameer-2.0.2-0.20.2/das-data/workbooks/8/8
file:/Users/marko/Desktop/Datameer-2.0.1-0.20.2/das-data/workbooks/9/9 -> file:/Users/marko/Development/Java/datameer/dap/build/dist/Datameer-2.0.2-0.20.2/das-data/workbooks/9/9
file:/Users/marko/Desktop/Datameer-2.0.1-0.20.2/das-data/fileuploads/10/10 -> file:/Users/marko/Development/Java/datameer/dap/build/dist/Datameer-2.0.2-0.20.2/das-data/fileuploads/10/10
file:/Users/marko/Desktop/Datameer-2.0.1-0.20.2/das-data/workbooks/11/11 -> file:/Users/marko/Development/Java/datameer/dap/build/dist/Datameer-2.0.2-0.20.2/das-data/workbooks/11/11
file:/Users/marko/Desktop/Datameer-2.0.1-0.20.2/das-data/fileuploads/10/12 -> file:/Users/marko/Development/Java/datameer/dap/build/dist/Datameer-2.0.2-0.20.2/das-data/fileuploads/10/12

Check or Update YARN Path Settings

 Check or update YARN application classpath and LD_LIBRARY path.  The  path settings are used to lookup and load libraries , therefor  incorrect path settings may lead into unexpected behavior .

...

How to Roll Back to a Previous Database Version

When upgrading, a database dump is created by Datameer, the file name is similar to mysql_db_dump_<oldVer>.sql. If an upgrade failed (failing upgrade scripts / failing upgrade injectors during startup), the database paths might be conflicting, it is therefore necessary to restore the Datameer MySQL database dump. Follow the steps to complete this task.

  1. Drop the new database, for example on the database server or the MySQL CLI.

    No Format
    DROP DATABASE dap;

     

  2. Using the mysql-init.sql script, create a new database.

    No Format
    mysql [-h <dbhost>] -u root -p < bin/mysql-init.sql

     

  3. Import and restore the previous database using the following shell command.

    No Format
    mysql [-h <dbhost>] -u root -p dap < /path/to/dump_file_name

...