Versions Compared

Version Old Version 9 New Version 10
Changes made by

Charles Bishop (Deactivated)

Charles Bishop (Deactivated)

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Table of Contents

...

Note
titlePausing jobs being submitted to the cluster

Before performing a  graceful shutdown of Datameer ,  view the current running or queued jobs and  and double-check that there is nothing running or scheduled.

  1. Stop the application.

    Panel
    <Datameer Application Folder>/bin/conductor.sh stop

  2. Unzip the upgrade file.
    Move the MySQL JDBC driver located in <Datameer path>/das-data/jdbc-jars, as described into the the  Install the JDBC Database Drivers section of the Installation Guide.
    In case you are using Datameer with MySQL as an application database, please check that database mode is configured accordingly in your  etc/das-env.sh  file

    No Format
    export DAS_DEPLOY_MODE=live

  3. If upgrading in Workgroup or Enterprise versions, users may want to consider allocating additional memory in etc/das-env.sh. 

    Panel

    # Adjust max available memory (-Xmx) according to your needs
    # WARNING: Path variables that may contain blanks should be added to jetty.sh start_das() method (see DAP-6342)
    export JAVA_OPTIONS="-Xmx2048m -XX:MaxPermSize=384m -Xms256m -XX:MaxNewSize=448m -XX:SurvivorRatio=6 -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=80 -XX:+HeapDumpOnOutOfMemoryError -XX:+CMSClassUnloadingEnabled -XX:+CMSPermGenSweepingEnabled"
    export JAVA_OPTIONS="$JAVA_OPTIONS -Dfile.encoding=utf-8"

  4. Consider changing the container sizing of the Map, Reduce, and AM containers, which respectively correspond to MapReduce and Tez jobs. To do so, change the settings of the following properties:

    Panel
    das.job.map-task.memory=2048
das.job.reduce-task.memory=2048
das.job.application-manager.memory=2048

  5. If existing, copy the files from the das-data folder of the old distribution to the new location. Keep the original das-data information as a backup.

    No Format
    cp -r /<old-location>/das-data /<new-location>/

  6. Users need to check, and update if necessary, the folder where plug-in configurations are stored.

    Open the file system of the previous Datameer version you are upgrading from and navigate to the default.properties file.

    Search for the property file that defines the folder where plug-in configurations are stored.

    Panel

    # Defines the folder where to store plugin configurations
    system.property.plugin.configs.dir=<folder name>

    Open the file system of the upgrade version of Datameer and find the same property in the default.properties file.

    Check or update the new property value to be the same folder name in the previous version.

    Copy the contents of the previous plug-in configurations folder into the file system of the new upgraded version of Datameer.

  7. If you made changes in your conf/ directory previously (like changing live.properties and log4j-<xyz>.properties), you need to apply** the same changes to the new conf/ directory of your upgraded instance.

  8. Copy over the native libraries that you have added to Datameer (if this applies). You don't have to copy over the native libraries that are already bundled with Datameer.

    No Format
    cp -r /<old-location>/lib/native/* /<new-location>/lib/native/
  9. If you have made changes to the conductor.sh script (for example, to enable SSH), apply** these changes to /<new-location>/bin/conductor.sh again. Notice JAVA_OPTIONS has been moved to etc/das-env.sh .-location>/bin/conductor.sh again. Notice JAVA_OPTIONS has been moved to etc/das-env.sh

  10. Migrate the SSL values from the previous start.ini to the new one.

    Panel

    # Set up a demonstration keystore and truststore
    jetty.keystore=etc/keystore
    jetty.truststore=etc/keystore

    # Sets the demonstration passwords
    # OBF passwords aren't secure. They are only protected from casual observation.
    # See http://www.eclipse.org/jetty/documentation/current/configuring-security-secure-passwords.html
    jetty.keystore.password=OBF:1vny1zlo1x8e1vnw1vn61x8g1zlu1vn4 # storepwd
    jetty.keymanager.password=OBF:1u2u1wml1z7s1z7a1wnl1u2g # keypwd
    jetty.truststore.password=OBF:1vny1zlo1x8e1vnw1vn61x8g1zlu1vn4 # storepwd

  11. Copy over files from etc/custom-jars (these files could be database drivers or 3rd party libraries).
  12. If you are using custom Datameer plugins (jar archives stored under etc/custom-plugins), update those to be in sync (API compatible) with the new version of Datameer and be sure to remove jar files related to older versions from etc/custom-plugins.

...

No Format
#Detect databases
detect old database [/Users/marko/Desktop/Datameer-2.0.1-0.20.2/das-data/database/hsql-db]
detect new database [/Users/marko/Development/Java/datameer/dap/build/dist/Datameer-2.0.2-0.20.2/das-data/database/hsql-db]

Upgrade datameer database version [2.0.0.6] to version [2.0.2.0]

#Assert datameer is running
No

#Copy Das Data
copy das-data from [/Users/marko/Desktop/Datameer-2.0.1-0.20.2/das-data] to [/Users/marko/Development/Java/datameer/dap/build/dist/Datameer-2.0.2-0.20.2/das-data]

#Upgrade starts...
Process upgrade scripts on database jdbc:hsqldb:file:das-data/database/hsql-db
	upgrade-2.0.1.hsql

#Update schema-version
set system.schemaVersion on database [/Users/marko/Development/Java/datameer/dap/build/dist/Datameer-2.0.2-0.20.2/das-data/database/hsql-db] to [2.0.1]


#Update data uri's [/Users/marko/Desktop/Datameer-2.0.1-0.20.2/das-data] -> [/Users/marko/Development/Java/datameer/dap/build/dist/Datameer-2.0.2-0.20.2/das-data]

log4j:WARN No appenders could be found for logger (org.hibernate.type.BasicTypeRegistry).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Converting 12 data entries
file:/Users/marko/Desktop/Datameer-2.0.1-0.20.2/das-data/fileuploads/1/1 -> file:/Users/marko/Development/Java/datameer/dap/build/dist/Datameer-2.0.2-0.20.2/das-data/fileuploads/1/1
file:/Users/marko/Desktop/Datameer-2.0.1-0.20.2/das-data/fileuploads/2/2 -> file:/Users/marko/Development/Java/datameer/dap/build/dist/Datameer-2.0.2-0.20.2/das-data/fileuploads/2/2
file:/Users/marko/Desktop/Datameer-2.0.1-0.20.2/das-data/fileuploads/3/3 -> file:/Users/marko/Development/Java/datameer/dap/build/dist/Datameer-2.0.2-0.20.2/das-data/fileuploads/3/3
file:/Users/marko/Desktop/Datameer-2.0.1-0.20.2/das-data/fileuploads/4/4 -> file:/Users/marko/Development/Java/datameer/dap/build/dist/Datameer-2.0.2-0.20.2/das-data/fileuploads/4/4
file:/Users/marko/Desktop/Datameer-2.0.1-0.20.2/das-data/fileuploads/5/5 -> file:/Users/marko/Development/Java/datameer/dap/build/dist/Datameer-2.0.2-0.20.2/das-data/fileuploads/5/5
file:/Users/marko/Desktop/Datameer-2.0.1-0.20.2/das-data/workbooks/6/6 -> file:/Users/marko/Development/Java/datameer/dap/build/dist/Datameer-2.0.2-0.20.2/das-data/workbooks/6/6
file:/Users/marko/Desktop/Datameer-2.0.1-0.20.2/das-data/workbooks/7/7 -> file:/Users/marko/Development/Java/datameer/dap/build/dist/Datameer-2.0.2-0.20.2/das-data/workbooks/7/7
file:/Users/marko/Desktop/Datameer-2.0.1-0.20.2/das-data/workbooks/8/8 -> file:/Users/marko/Development/Java/datameer/dap/build/dist/Datameer-2.0.2-0.20.2/das-data/workbooks/8/8
file:/Users/marko/Desktop/Datameer-2.0.1-0.20.2/das-data/workbooks/9/9 -> file:/Users/marko/Development/Java/datameer/dap/build/dist/Datameer-2.0.2-0.20.2/das-data/workbooks/9/9
file:/Users/marko/Desktop/Datameer-2.0.1-0.20.2/das-data/fileuploads/10/10 -> file:/Users/marko/Development/Java/datameer/dap/build/dist/Datameer-2.0.2-0.20.2/das-data/fileuploads/10/10
file:/Users/marko/Desktop/Datameer-2.0.1-0.20.2/das-data/workbooks/11/11 -> file:/Users/marko/Development/Java/datameer/dap/build/dist/Datameer-2.0.2-0.20.2/das-data/workbooks/11/11
file:/Users/marko/Desktop/Datameer-2.0.1-0.20.2/das-data/fileuploads/10/12 -> file:/Users/marko/Development/Java/datameer/dap/build/dist/Datameer-2.0.2-0.20.2/das-data/fileuploads/10/12

Check or Update YARN Path Settings

 Check or update YARN application classpath and LD_LIBRARY path. The   The  path settings are used to lookup and load libraries , therefor therefor  incorrect path settings may lead into unexpected behavior .

...


Panel
titleAttention
  • Fields that were made unnecessary on the AD/LDAP configuration page have been removed.

  • There is no longer a radio button needed to switch between AD and LDAP.

  • The difference between AD and LDAP can be configured in field input.

    • Username Attribute: uid (LDAP), sAMAccountName (AD)

    • Group Definition: objectClass=groupOfNames (LDAP), objectClass=group (AD)
  • There is no longer a drop down box to switch between simple and custom group definition. (LDAP only)

Removing Extraneous Properties After Upgrade Adjustments

The support for MapReduce has been deprecated and was removed in /wiki/spaces/DAS60/pages/4631169606Datameer 6.3. After it is verified that all jobs work without the MapReduce framework, remove MapReduce related custom properties from the Hadoop cluster page under the Administration tab and in the advanced settings of data links, import jobs, and/or workbooks.

How to Roll Back to a Previous Database Version

When upgrading, a database dump is created by Datameer, the file name is similar to mysql_db_dump_<oldVer>.sql. If an upgrade failed (failing upgrade scripts / failing upgrade injectors during startup), the database paths might be conflicting, it is therefore necessary to restore the Datameer MySQL database dump. Follow the steps to complete this task.

  1. Drop the new database, for example on the database server or the MySQL CLI.

    No Format
    DROP DATABASE dap;

     

  2. Using the mysql-init.sql script, create a new database.

    No Format
    mysql [-h <dbhost>] -u root -p < bin/mysql-init.sql

     

  3. Import and restore the previous database using the following shell command.

    No Format
    mysql [-h <dbhost>] -u root -p dap < /path/to/dump_file_name

...