Versions Compared

Version Old Version 5 New Version 6
Changes made by

Charles Bishop (Deactivated)

Charles Bishop (Deactivated)

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Table of Contents

...

Warning

Datameer now uses Parquet as its default storage format as opposed to the previous sequence storage format. Sequence Files are still readable in Datameer but all new artifacts are written in Parquet. Depending on the configuration of your jobs, it might be difficult to roll back after upgrading to Datameer v66.3.

There is a property that forces Datameer to use Sequence Files instead of Parquet if a rollback is required. Under the JAVA_OPTIONS in etc/das-env.sh, add "-Duse-sequence-storage=true" before starting Datameer for the first time following the upgrade. This property has been removed as of Datameer v77.0.

If upgrading from a Datameer version before 6.1, we highly recommend first upgrading to Datameer v66.1  and /wiki/spaces/DAS60/pages/4631169606  

In 6.3, Datameer uses Parquet as its default storage format. While Datameer can still read data written in the sequence file format, all artifacts are now stored using the Parquet file format. Datameer still uses the sequence file format for writing intermediate files and preview files for performance reasons, but the final artifacts are saved using the Parquet format. Due to this change, it might not be possible to roll back after upgrading to Datameer v66.3, depending on the configuration of your artifacts. For example, if import jobs and workbooks are configured to purge historical data and only keep the last one result, the final artifacts are stored as Parquet files, and prior versions of Datameer cannot read them. 

External authentication changes

As of Datameer v66.3, its' required to use ports 389/636 to connect to LDAP or the ports 3268/3269 to connect to Active Directory.

Anchor

...

upgrade72

...

upgrade72
Upgrading to version

...

7.

...

2

To increase the efficiency in workbooks, filter (FilterSheetType) and sort (SortSheetType) sheet types are no longer supported. These sheet types have not been used as of Datameer v22.1 though users that have upgraded from v22.1 or earlier versions may still have these sheet types within their older workbooks. Users with these sheet types in their database first need to upgrade to an staging version Datameer v6.3 which migrates the filter/sort sheet type into a formula sheet type. After upgrading to Datameer v6 6.3, 6.4, 7.0 or 7.1, it is safe to upgrade v67.42.

For those upgrading to v67.4 2 that had the filter or sort sheet types converted into formula sheet types, workbooks do not display preview data in those sheets until the workbook has run again. 

...

Upgrade the MySQL database

Info
  • As of Datameer version 6.0, only versions MySQL versions 5.5 and above are supported.
  • It is highly recommend to backup the application database before proceeding further with the upgrade process.
  • If you are upgrading from an older release, you may need to migrate your database schema.

  1. To upgrade the database, enter the following command:

    No Format
    bin/upgrade_db.sh 

    If either the structure of the MySQL database or any of the tables in the database will be modified by the Datameer upgrade, running this command automatically creates a dump of the database.

    Note
    • Ensure that the script is not executed under root, but by the same user account that starts and stops Datameer.
    • Run the upgrade script directly from the <INSTALLDIR>. Preface the script with the bin directory.

  2. If your database name is different than the default dap or the database runs on a different server or customized port, you will need to provide the additional arguments:

    No Format
    bin/upgrade_db.sh -h [db hostname] -o [db port] -n [db name] -u [db username] -p [db password]

  3. Check to ensure your database character encoding is set to UTF-8.
    In the Database, enter:

    No Format
    SELECT DEFAULT_CHARACTER_SET_NAME FROM INFORMATION_SCHEMA.SCHEMATA WHERE SCHEMA_NAME = '<The name of your Datameer database>';

    If UTF8 was returned, open the default properties and comment in the setting system.property.hibernate.connection.characterEncoding=utf8.
    If something other than UTF8 was returned, change the database character encoding setting to UTF8 or special characters may show as corrupt. 

  4. To finish the upgrade, restart the application.

    Insert excerpt
    _startingDatameer
    _startingDatameer
    nopaneltrue

...

No Format
#Detect databases
detect old database [/Users/marko/Desktop/Datameer-2.0.1-0.20.2/das-data/database/hsql-db]
detect new database [/Users/marko/Development/Java/datameer/dap/build/dist/Datameer-2.0.2-0.20.2/das-data/database/hsql-db]

Upgrade datameer database version [2.0.0.6] to version [2.0.2.0]

#Assert datameer is running
No

#Copy Das Data
copy das-data from [/Users/marko/Desktop/Datameer-2.0.1-0.20.2/das-data] to [/Users/marko/Development/Java/datameer/dap/build/dist/Datameer-2.0.2-0.20.2/das-data]

#Upgrade starts...
Process upgrade scripts on database jdbc:hsqldb:file:das-data/database/hsql-db
	upgrade-2.0.1.hsql

#Update schema-version
set system.schemaVersion on database [/Users/marko/Development/Java/datameer/dap/build/dist/Datameer-2.0.2-0.20.2/das-data/database/hsql-db] to [2.0.1]


#Update data uri's [/Users/marko/Desktop/Datameer-2.0.1-0.20.2/das-data] -> [/Users/marko/Development/Java/datameer/dap/build/dist/Datameer-2.0.2-0.20.2/das-data]

log4j:WARN No appenders could be found for logger (org.hibernate.type.BasicTypeRegistry).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Converting 12 data entries
file:/Users/marko/Desktop/Datameer-2.0.1-0.20.2/das-data/fileuploads/1/1 -> file:/Users/marko/Development/Java/datameer/dap/build/dist/Datameer-2.0.2-0.20.2/das-data/fileuploads/1/1
file:/Users/marko/Desktop/Datameer-2.0.1-0.20.2/das-data/fileuploads/2/2 -> file:/Users/marko/Development/Java/datameer/dap/build/dist/Datameer-2.0.2-0.20.2/das-data/fileuploads/2/2
file:/Users/marko/Desktop/Datameer-2.0.1-0.20.2/das-data/fileuploads/3/3 -> file:/Users/marko/Development/Java/datameer/dap/build/dist/Datameer-2.0.2-0.20.2/das-data/fileuploads/3/3
file:/Users/marko/Desktop/Datameer-2.0.1-0.20.2/das-data/fileuploads/4/4 -> file:/Users/marko/Development/Java/datameer/dap/build/dist/Datameer-2.0.2-0.20.2/das-data/fileuploads/4/4
file:/Users/marko/Desktop/Datameer-2.0.1-0.20.2/das-data/fileuploads/5/5 -> file:/Users/marko/Development/Java/datameer/dap/build/dist/Datameer-2.0.2-0.20.2/das-data/fileuploads/5/5
file:/Users/marko/Desktop/Datameer-2.0.1-0.20.2/das-data/workbooks/6/6 -> file:/Users/marko/Development/Java/datameer/dap/build/dist/Datameer-2.0.2-0.20.2/das-data/workbooks/6/6
file:/Users/marko/Desktop/Datameer-2.0.1-0.20.2/das-data/workbooks/7/7 -> file:/Users/marko/Development/Java/datameer/dap/build/dist/Datameer-2.0.2-0.20.2/das-data/workbooks/7/7
file:/Users/marko/Desktop/Datameer-2.0.1-0.20.2/das-data/workbooks/8/8 -> file:/Users/marko/Development/Java/datameer/dap/build/dist/Datameer-2.0.2-0.20.2/das-data/workbooks/8/8
file:/Users/marko/Desktop/Datameer-2.0.1-0.20.2/das-data/workbooks/9/9 -> file:/Users/marko/Development/Java/datameer/dap/build/dist/Datameer-2.0.2-0.20.2/das-data/workbooks/9/9
file:/Users/marko/Desktop/Datameer-2.0.1-0.20.2/das-data/fileuploads/10/10 -> file:/Users/marko/Development/Java/datameer/dap/build/dist/Datameer-2.0.2-0.20.2/das-data/fileuploads/10/10
file:/Users/marko/Desktop/Datameer-2.0.1-0.20.2/das-data/workbooks/11/11 -> file:/Users/marko/Development/Java/datameer/dap/build/dist/Datameer-2.0.2-0.20.2/das-data/workbooks/11/11
file:/Users/marko/Desktop/Datameer-2.0.1-0.20.2/das-data/fileuploads/10/12 -> file:/Users/marko/Development/Java/datameer/dap/build/dist/Datameer-2.0.2-0.20.2/das-data/fileuploads/10/12

Check or Update YARN Path Settings

 Check or update YARN application classpath and LD_LIBRARY path.  The  path settings are used to lookup and load libraries , therefor  incorrect path settings may lead into unexpected behavior .

...


Panel
titleAttention
  • Fields that were made unnecessary on the AD/LDAP configuration page have been removed.

  • There is no longer a radio button needed to switch between AD and LDAP.

  • The difference between AD and LDAP can be configured in field input.

    • Username Attribute: uid (LDAP), sAMAccountName (AD)

    • Group Definition: objectClass=groupOfNames (LDAP), objectClass=group (AD)
  • There is no longer a drop down box to switch between simple and custom group definition. (LDAP only)

Removing Extraneous Properties After Upgrade Adjustments

The support for MapReduce has been deprecated and was removed in in Datameer v66.3. After it is verified that all jobs work without the MapReduce framework, remove MapReduce related custom properties from the Hadoop cluster page under the Administration tab and in the advanced settings of data links, import jobs, and/or workbooks.

How to Roll Back to a Previous Database Version

When upgrading, a database dump is created by Datameer, the file name is similar to mysql_db_dump_<oldVer>.sql. If an upgrade failed (failing upgrade scripts / failing upgrade injectors during startup), the database paths might be conflicting, it is therefore necessary to restore the Datameer MySQL database dump. Follow the steps to complete this task.

  1. Drop the new database, for example on the database server or the MySQL CLI.

    No Format
    DROP DATABASE dap;

     

  2. Using the mysql-init.sql script, create a new database.

    No Format
    mysql [-h <dbhost>] -u root -p < bin/mysql-init.sql

     

  3. Import and restore the previous database using the following shell command.

    No Format
    mysql [-h <dbhost>] -u root -p dap < /path/to/dump_file_name

...