Installation Guide

Before installing Datameer X, complete the following prerequisites:

• Install the Hadoop client
  • Datameer X application server, as well as all data nodes, are configured properly with host names, DNS, datetime, NTP, and other details 
    
    • Datameer X application server, as well as all data nodes, have Java 1.8 (Oracle recommended)
      
      • Check this installation using the following commands: java -version and echo $JAVA_HOME
    • Datameer X application server has Oracle Java Cryptography Extension (JCE) already installed. See Java SE Security for more information.
    • Commands such as hadoop, yarn, and mysql can be executed
• Install MySQL client
  • For Datameer's application database, the MySQL server must be prepared with the necessary access 
• Grant administrative rights or root access
• Ensure Internet access to download packages and plug-ins or have necessary ZIP files downloaded and available
• If using Kerberos, configure a Kerberos Secured cluster for secure impersonation

Creating the Datameer X User

Administrative rights are required to create the Datameer X user on the machine where Datameer X is being installed. This can be accomplished under the root account. Make sure the user ID is above 500 and that the account has enough resources and file descriptors available. 

• Create the user account under which the Datameer X service will be started and running later:

  Create user

  ./usr/sbin/groupadd --system datameer
  ./usr/sbin/useradd --system --create-home --gid datameer datameer

  These commands also create the directory /home/datameer. 

• Check the max number of open files - global level or per-user limits (or both) - and set it to 64K if it isn't set already. This configuration needs to be done on all nodes within the cluster and might require a reboot.

Creating Directories for Application, Cache, Logs, and Temporary Files 

For performance reasons and to have better control about where space on the file systems and on disks is used, create separate directories for application, cache, logs, and temporary files. Do this according the Linux Filesystem Hierarchy Standard (FHS). To create the directories and change the permissions you need administrative rights. Complete this task under the user account root.

• Create the directories for application, cache, logs, and temporary files:

  Create directories

  mkdir -p /opt/datameer
  chown -R datameer:datameer /opt/datameer
  mkdir -p /var/cache/datameer
  chown -R datameer:datameer /var/cache/datameer
  mkdir -p /var/log/datameer
  chown -R datameer:datameer /var/log/datameer
  mkdir -p /tmp/datameer
  chown -R datameer:datameer /tmp/datameer

Switching the User and Changing the Working Directory 

This should be the last task to which administrative rights are necessary. 

• Switch to the new Datameer X user and change to the working directory where Datameer X is being installed:

  Switch user and directory

  su - datameer
  cd /opt/datameer

  Proceed from within the Datameer X installation directory and under the user account datameer only. 

Downloading and Unzipping Datameer X

Get in touch with Datameer Support or your Datameer representative to receive the installation package download link.

Download and unzip the appropriate Datameer X package for your Hadoop cluster distribution:
INFO: If you already have a Datameer X installation, you can also start from here. 

curl -s -k -o Datameer-<package>.zip "https://download.datameer.com.s3.amazonaws.com/releases/Datameer-<version>/<dist>/Datameer-<package>.zip?<AWSproperties>" ; unzip Datameer* 

Creating Symlinks for Future Updates

Create the symlink and change the working directory:

ln -s Datameer-<package> current 
cd current

Selecting and Preparing the Database

To define which database to use, make an entry in the 'live.properties' file under 'conf/live.properties':

#Define which database to use: hsql-memory, hsql-file, mysql, mariadb
system.property.db.mode=mysql

or 

#Define which database to use: hsql-memory, hsql-file, mysql, mariadb
system.property.db.mode=mariadb 

Downloading and Installing the MySQL Database JDBC Connector 

By default, the Datameer X application runs with an HSQL file database that is created on the local filesystem under das-data/database/hsql-db. If you are setting up Datameer X for production use, Datameer strongly recommends using MySQL instead of the HSQL file database. 

• Download the official MySQL JDBC driver ZIP file, extract the driver from the archive file, and copy it into the correct destination:

  Download and install JDBC

  # Lookup latest JDBC driver version 
  JDBCDRV="$(curl -s -k 'https://dev.mysql.com/downloads/connector/j/' | grep -o -m 1 'mysql-connector-java.*zip')" 
  # Download latest JDBC driver version
  curl -s -L0 -k -O "https://dev.mysql.com/get/Downloads/Connector-J/${JDBCDRV}" 
  # Unzip driver package
  unzip mysql-connector* -d etc/custom-jars 
  # Move only the necessary JAR file 
  mv etc/custom-jars/mysql-connector*/*bin.jar etc/custom-jars
  # Clean up 
  rm -rf etc/custom-jars/mysql-connector-java-?.?.?? 

• Double-check if etc/custom-jars contain the latest mysql-connector-java-<version>-bin.jar:

  Check installation

  echo $JDBCDRV
  ll etc/custom-jars 

Configuring Datameer X for MySQL Database

Datameer X service depends on the MySQL database. The MySQL database is used for writing to workbooks, permission changes, job execution, scheduling, and more. To function properly, a response time should be between ten and twenty milliseconds. To run the application in MySQL mode, the following changes need to be implemented. 

1. Check the database connection:

   mysqladmin version
   mysqladmin ping
   mysqladmin status
   echo q | telnet -e q `hostname` 3306
   nc -z -w1 `hostname` 3306

   INFO: You can follow up later with using the Check if the Datameer X Application Database is Running and Accessible article.
   
   

2. Create a new database via the "./bin/mysql-init.sql" script:
   

   for MYSQL 5.x and lower:

   mysql -u <user> -p<password> -h <host/ip> -P <port> < bin/mysql-init.sql

   
   for MYSQL 8.x and higher:

   mysql -u <user> -p<password> -h <host/ip> -P <port> < bin/mysql8x-init.sql

3. Set the deploy mode to "live" in the "./etc/das-env.sh" file:
   

   # Change this to DAS_DEPLOY_MODE=live when you want to run in live mode against a mysql db
   export DAS_DEPLOY_MODE=live

   
   

4. Set the database name in the "./conf/default.properties" file:
   

   # Set the name of the MySql database DATAMEER uses.
   system.property.db.name=dap

   
   

5. Execute the command to initialize the database:
   

   ./bin/database.sh init

The database configuration is now completed.

Installing the License

If you have already received a Datameer X license:

1. Launch the Datameer X application and open the UI. The welcome page with all available licensing options is loaded. 
2. Press the button "Activate" and upload the key you received from Datameer. The license is being activated. You will be redirected to the login page.

Starting Datameer X

Start the Datameer X service.

• Working within the current installation directory, use the following commands:

  Start Datameer

  # Start the Datameer X service
  ./bin/conductor.sh start
  # Check the process ID (PID)
  ps -ef | grep -i "java.*jetty.*datameer" | grep -v grep | tr -s " " | cut -d " " -f2
  # Monitor the process booting and the log files
  cat logs/jvm-stdout.log; sleep 3; tail -F logs/`date +"%Y_%m_%d"`.stderrout.log logs/conductor.log

Stopping Datameer X

Stop the Datameer X service.

• Working within the current installation directory, use the following commands:

  Start Datameer

  # Stop the Datameer X service
  ./bin/conductor.sh stop 
  # Monitor the process shutting down 
  cat logs/jvm-stdout.log; sleep 3; tail -F logs/`date +"%Y_%m_%d"`.stderrout.log logs/conductor.log

Restarting Datameer X

Restart the Datameer X service.

• Working within the  current installation directory, use the following commands:

  Restart Datameer

  # Restart the Datameer X service
  ./bin/conductor.sh restart
  # Monitor the process booting and the log files
  cat logs/jvm-stdout.log; sleep 3; tail -F logs/`date +"%Y_%m_%d"`.stderrout.log logs/conductor.log

Datameer X Graceful Shutdown

Gracefully shut down the Datameer X service.

1. Pause the Job Scheduler located under the Admin tab in Datameer.
2. Wait for current jobs to be marked as completed.
3. When all jobs have been completed, use the "stop" command on conductor.sh
4. After the Datameer X application has been stopped, perform needed maintenance.
5. With all maintenance completed, resume Datameer X using the "start" command on conductor.sh.
6. Under Datameer's Admin tab, resume the Job Scheduler. 

Service Check 

Check if the Datameer X service is running and accessible. 

• Working within the current installation directory, use the following commands:

  Check service

  ./bin/conductor.sh check
  ps -ef | grep -i "java.*datameer" | grep -v grep
  lsof -i tcp@`hostname`:8080
  lsof -i tcp@`hostname`:8443
  lsof -i tcp | grep 'datameer'
  echo -e "GET /login \n\n" | openssl s_client -connect `hostname`:8443 -quiet | grep -i -m 1 'datameer'

• Monitor if the service is running and accessible later:

  Monitor service

  curl -k "https://`hostname`:8443/watchdog"
  lsof -i -p ps -ef | grep -i "java.*jetty.*datameer" | grep -v grep | tr -s " " | cut -d " " -f2

• You can also monitor the Datameer X core directory size in HDFS.

Configuring Datameer X for a Kerberos Secured Cluster

Before configuring Datameer X for a Kerberos Secured cluster, test Kerberos authentication and job execution on CLI.  

• Send a test job to the cluster:

  Test job execution

  hadoop jar /<distribution-specific-path>/hadoop-mapreduce-examples-*.jar pi -Dmapreduce.job.queuename=root.default 3 10

To configure Datameer X for a Kerberos-secured cluster, follow the Secure Mode Configuration instructions. 

Secure Hadoop Distributed Filesystem (HDFS)

You must have a properly configured connection to a Kerberos-secured cluster to use the tool to secure the Hadoop Distributed Filesystem (HDFS) .

• For the initial setup of secure impersonation, execute the following commands:

  Secure HDFS

  # Check current available access rights
  hadoop fs -ls /user/datameer
  # Configure Datameer X Core Directories (aka Private Folder) 
  ./bin/secure_hdfs_tool.sh -u -g <dasuser>
  hadoop fs -chown -R datameer:<dasuser> /user/datameer/.staging 
  hadoop fs -chmod -R 770 /user/datameer/.staging
  # Check if changes are made correctly 
  hadoop fs -ls /user/datameer

Starting Testing

Start the Datameer X service to do final testing. 

• Working within the current installation directory, use the following commands:

  Start Datameer

  # Start the Datameer X service
  ./bin/conductor.sh start
  # Check the process ID (PID)
  ps -ef | grep -i "java.*jetty.*datameer" | grep -v grep | tr -s " " | cut -d " " -f2
  # Monitor the process booting and the log files
  cat logs/jvm-stdout.log; sleep 3; tail -F logs/`date +"%Y_%m_%d"`.stderrout.log logs/conductor.log

Best Practices for Installing Datameer X

Implement frequent database backups

Datameer X service depends on the MySQL database, it is used for writing to workbooks, permission changes, job execution, scheduling, and more. It is highly recommended to backup the application database frequently.

0 * * * * mysqldump -u'dap' -p'dap' dap | gzip > /home/datameer/<company>_<system>_<datameer-version>_`date +\%Y\%m\%d_\%H\%M`.sql.gz

# Check from time to tome how long the database dump will take and if it fits into the timeslot
time mysqldump -u'dap' -p'dap' dap | gzip > /home/datameer/<company>_<system>_<datameer-version>_`date +\%Y\%m\%d_\%H\%M`.sql.gz
# Verify from time to time if the files are OK
gzip -d /home/datameer/company>_<system>_<datameer-version>_<date>_<time>.sql.gz 
head /home/datameer/<company>_<system>_<datameer-version>_<date>_<time>.sql

Enable and Configure Transport Layer Security (TLS)

Enable TLS for use with Datameer X in production environments. As Datameer X is packed with Jetty 9, you only need to enable modules.

• Enable TLS:

  Enable modules

  # Check default configuration
  java -jar start.jar --list-config | grep -i 'etc/jetty*'
  # Add SSL and HTTPS to the startup modules
  java -jar start.jar --add-to-start=ssl,https
  # Check final configuration 
  java -jar start.jar --list-config 

• Redirect all HTTP requests to HTTPS

• Configure TLS for Embedded Jetty (for more security)

• To change the HTTPS port follow the instructions under Configure TLS

  All port changes should be made in the start.ini file, which overrides jetty.port.

• Use your own custom certificate