Page Comparison

Complete the following prerequisites before installing Datameer:

• Install the Hadoop client
  • Datameer application server, as well all data nodes, are configured properly with host names, DNS, datetime, NTP, and other details 
    
    • Datameer application server, as well all data nodes, have Java 1.8 (Oracle recommended)
      
      • Check this installation using the following commands: java -version and echo $JAVA_HOME
    • Datameer application server has Oracle Java Cryptography Extension (JCE) already installed. See Java SE Security for more information.
    • Commands such as hadoop, yarn, and mysql can be executed
• Install MySQL client
  • For Datameer's application database, the MySQL server must be prepared with necessary access 
• Grant administrative rights or root access
• Ensure Internet access to download packages and plug-ins or have necessary ZIP files downloaded and available
• If using Kerberos, configure a Kerberos Secured cluster for secure impersonation

Anchor
create_user create_user

Create the Datameer User

Administrative rights are required to create the Datameer user on the machine where Datameer is being installed. This can be accomplished under the root account. Make sure the user ID is above 500 and that the account has enough resources and file descriptors available. 

• Create the user account under which the Datameer service will be started and running later:

  Code Block
  language bash title Create user
  ./usr/sbin/groupadd --system datameer ./usr/sbin/useradd --system --create-home --gid datameer datameer

  These commands also create the directory /home/datameer. 

• Check the max number of open files - global level or per-user limits (or both) - and set it to 64K if it isn't set already. This configuration needs to be done on all nodes within the cluster and might require a reboot.

Create Directories for Application, Cache, Logs, and Temporary Files 

For performance reasons and to have better control about where space on the file systems and on disks is used, create separate directories for application, cache, logs, and temporary files. Do this according the Linux Filesystem Hierarchy Standard (FHS). To create the directories and change the permissions you need administrative rights. Complete this task under the user account root.

• Create the directories for application, cache, logs, and temporary files:

  Code Block
  language bash title Create directories
  mkdir -p /opt/datameer chown -R datameer:datameer /opt/datameer mkdir -p /var/cache/datameer chown -R datameer:datameer /var/cache/datameer mkdir -p /var/log/datameer chown -R datameer:datameer /var/log/datameer mkdir -p /tmp/datameer chown -R datameer:datameer /tmp/datameer

Switch the User and Change the Working Directory 

This should be the last task to which administrative rights are necessary. 

• Switch to the new Datameer user and change to the working directory where Datameer is being installed:

  Code Block
  language bash title Switch user and directory
  su - datameer cd /opt/datameer

  Proceed from within the Datameer installation directory and under the user account datameer only. 

Anchor
download_unzip download_unzip

Download and Unzip Datameer

Download the appropriate Datameer package for your Hadoop cluster distribution. If you have already a Datameer installation you can also start from here. 

• Download and unzip. 

  Code Block
  language bash title Switch user and directory
  curl -s -k -o Datameer-<package>.zip "https://download.datameer.com.s3.amazonaws.com/releases/Datameer-<version>/<dist>/Datameer-<package>.zip?<AWSproperties>" ; unzip Datameer*

If you are an authorized Enterprise customer, you can get the download link for the latest public available package from https://my.datameer.com/workspace/downloads, or request one through your Customer Success Manager (CSM). 

Anchor
download_jdbc download_jdbc

Download and Install the MySQL Database JDBC Connector 

By default, the Datameer application runs with an HSQL file database that is created on the local filesystem under das-data/database/hsql-db. If you are setting up Datameer for production use, Datameer strongly recommends using MySQL instead of the HSQL file database. 
As of Datameer 7.4: MariaDB is supported as an alternative to MySQL.

• Download the official MySQL JDBC driver ZIP file, extract the driver from the archive file, and copy it into the correct destination:

  Code Block
  language bash title Download and install JDBC
  # Lookup latest JDBC driver version JDBCDRV="$(curl -s -k 'https://dev.mysql.com/downloads/connector/j/' | grep -o -m 1 'mysql-connector-java.*zip')"  # Download latest JDBC driver version curl -s -L0 -k -O "https://dev.mysql.com/get/Downloads/Connector-J/${JDBCDRV}"  # Unzip driver package unzip mysql-connector* -d etc/custom-jars  # Move only the necessary JAR file mv etc/custom-jars/mysql-connector*/*bin.jar etc/custom-jars # Clean up rm -rf etc/custom-jars/mysql-connector-java-?.?.??

• Double-check if etc/custom-jars contain the latest mysql-connector-java-<version>-bin.jar:

  Code Block
  language bash title Check installation
  echo $JDBCDRV ll etc/custom-jars

Anchor
configure_mysql configure_mysql

Configure Datameer for MySQL Database

Datameer service depends on the MySQL database. The MySQL database is used for writing to workbooks, permission changes, job execution, scheduling, and more. To function properly,  a response time should be between ten and twenty milliseconds. To run the application in MySQL mode, the following changes need to be implemented. As of Datameer 7.4: MariaDB is supported as an alternative to MySQL.

• Check database connection:

  Code Block
  language bash title Connection check
  mysqladmin version mysqladmin ping mysqladmin status echo q | telnet -e q `hostname` 3306 nc -z -w1 `hostname` 3306

  You can follow up later with using the Check if the Datameer Application Database is Running and Accessible article.

• Initialize application database:

  Code Block
  language bash title Initialize database
  mysql -uroot -p < bin/mysql-init.sql mysql -uroot -p dap < bin/create-tables.sql

• Configure for an enterprise environment:

  Code Block
  language bash title etc/das-env.sh
  # Create a backup of the original configuration file cp etc/das-env.sh etc/das-env.sh.original # Change the deploy mode sed -i "s/\(DAS_DEPLOY_MODE=\).*\$/\1live/" etc/das-env.sh # Uncomment the database name you will be using sed -i '/#.*DATAMEER_DB_NAME=/s/^#//' etc/das-env.sh # Uncomment the user that the application should be started at sed -i '/#.*DAS_USER=/s/^#//' etc/das-env.sh # Specify the maximum size of the memory allocation pool sed -i 's/Xmx2048m/Xmx4096m/' etc/das-env.sh # Create a log of changes made diff -e etc/das-env.sh.original etc/das-env.sh > changes.das-env.sh

  If you aren't using sed, changes can also be made using other editors.
  Example: 
  Defining the DAS_USER is commented out in etc/das-env.sh. To use this property, uncomment the line and add for the correct value. 

Installing the License

If you have already received a license from Datameer, copy the license file to: $INSTALL_LOCATION/das-<version>/etc/license or log into Datameer from the user interface and go to the Admin tab. Select License from the side menu and upload your license here.

The file must be named license.lic or it won't work.

If you don't have a license, email license@datameer.com.

If you attempt to launch the application without a license, you receive an message asking you to contact Datameer for a license file. Include the MAC address listed in this error message when contacting Datameer for a license. License information and usage can be viewed at any time by clicking the Admin tab and selecting License.

See License Information for information on how to update the license and for details about volume-based licensing.

At this time Datameer has been installed.

Start Datameer 

Start the Datameer service.

• Working within the current installation directory, use the following commands:

  Code Block
  language bash title Start Datameer
  # Start the Datameer service ./bin/conductor.sh start # Check the process ID (PID) ps -ef | grep -i "java.*jetty.*datameer" | grep -v grep | tr -s " " | cut -d " " -f2 # Monitor the process booting and the log files cat logs/jvm-stdout.log; sleep 3; tail -F logs/`date +"%Y_%m_%d"`.stderrout.log logs/conductor.log

Stop Datameer 

Stop the Datameer service.

• Working within the current installation directory, use the following commands:

  Code Block
  language bash title Start Datameer
  # Stop the Datameer service ./bin/conductor.sh stop # Monitor the process shutting down cat logs/jvm-stdout.log; sleep 3; tail -F logs/`date +"%Y_%m_%d"`.stderrout.log logs/conductor.log

Anchor
restart restart

Restart Datameer

Restart the Datameer service.

• Working within the  current installation directory, use the following commands:

  Code Block
  language bash title Restart Datameer
  # Restart the Datameer service ./bin/conductor.sh restart # Monitor the process booting and the log files cat logs/jvm-stdout.log; sleep 3; tail -F logs/`date +"%Y_%m_%d"`.stderrout.log logs/conductor.log

Anchor
graceful_shutdown graceful_shutdown

Datameer Graceful Shutdown

Gracefully shut down the Datameer service.

1. Pause the Job Scheduler located under the Admin tab in Datameer.
2. Wait for current jobs to be marked as completed.
3. When all jobs have been completed, use the "stop" command on conductor.sh
4. After the Datameer application has been stopped, perform needed maintenance.
5. With all maintenance completed, resume Datameer using the "start" command on conductor.sh.
6. Under Datameer's Admin tab, resume the Job Scheduler. 

Service Check 

Check if the Datameer service is running and accessible. 

• Working within the current installation directory, use the following commands:

  Code Block
  language bash title Check service
  ./bin/conductor.sh check ps -ef | grep -i "java.*datameer" | grep -v grep lsof -i tcp@`hostname`:8080 lsof -i tcp@`hostname`:8443 lsof -i tcp | grep 'datameer' echo -e "GET /login \n\n" | openssl s_client -connect `hostname`:8443 -quiet | grep -i -m 1 'datameer'

• Monitor if the service is running and accessible later:

  Code Block
  language bash title Monitor service
  curl -k "https://`hostname`:8443/watchdog" lsof -i -p ps -ef | grep -i "java.*jetty.*datameer" | grep -v grep | tr -s " " | cut -d " " -f2

• You can also monitor the Datameer core directory size in HDFS.

Configure Datameer for Kerberos Secured Cluster

Before configuring Datameer for a Kerberos Secured cluster, test Kerberos authentication and job execution on CLI.  

• Send a test job to the cluster:

  Code Block
  language bash title Test job execution
  hadoop jar /<distribution-specific-path>/hadoop-mapreduce-examples-*.jar pi -Dmapreduce.job.queuename=root.default 3 10

To configure Datameer for a Kerberos-secured cluster follow the Secure Mode Configuration instructions. 

Secure Hadoop Distributed Filesystem (HDFS)

You must have a properly configured connection to a Kerberos-secured cluster to use the tool to secure the Hadoop Distributed Filesystem (HDFS) .

• For initial setup of secure impersonation, execute the following commands:

  Code Block
  language bash title Secure HDFS
  # Check current available access rights hadoop fs -ls /user/datameer # Configure Datameer Core Directories (aka Private Folder) ./bin/secure_hdfs_tool.sh -u -g <dasuser> hadoop fs -chown -R datameer:<dasuser> /user/datameer/.staging hadoop fs -chmod -R 770 /user/datameer/.staging # Check if changes are made correctly hadoop fs -ls /user/datameer

Start Testing

Start the Datameer service to do final testing. 

• Working within the current installation directory, use the following commands:

  Code Block
  language bash title Start Datameer
  # Start the Datameer service ./bin/conductor.sh start # Check the process ID (PID) ps -ef | grep -i "java.*jetty.*datameer" | grep -v grep | tr -s " " | cut -d " " -f2 # Monitor the process booting and the log files cat logs/jvm-stdout.log; sleep 3; tail -F logs/`date +"%Y_%m_%d"`.stderrout.log logs/conductor.log

Best Practices for Installing Datameer

Anchor
database_backup database_backup

Implement frequent database backups

Datameer service depends on the MySQL database, it is used for writing to workbooks, permission changes, job execution, scheduling, and more. It is highly recommend to backup the application database frequently.

0 * * * * mysqldump -u'dap' -p'dap' dap | gzip > /home/datameer/<company>_<system>_<datameer-version>_`date +\%Y\%m\%d_\%H\%M`.sql.gz

# Check from time to tome how long the database dump will take and if it fits into the timeslot
time mysqldump -u'dap' -p'dap' dap | gzip > /home/datameer/<company>_<system>_<datameer-version>_`date +\%Y\%m\%d_\%H\%M`.sql.gz
# Verify from time to time if the files are OK
gzip -d /home/datameer/company>_<system>_<datameer-version>_<date>_<time>.sql.gz 
head /home/datameer/<company>_<system>_<datameer-version>_<date>_<time>.sql

Enable and configure transport layer security (TLS)

Enable TLS for use with Datameer in production environments. As Datameer is packed with Jetty 9, you only need to enable modules.

• Enable TLS:

  Code Block
  language bash title Enable modules
  # Check default configuration java -jar start.jar --list-config | grep -i 'etc/jetty*' # Add SSL and HTTPS to the startup modules java -jar start.jar --add-to-start=ssl,https # Check final configuration java -jar start.jar --list-config

• Redirect all HTTP requests to HTTPS

• Configure TLS for Embedded Jetty (for more security)

• To change the HTTPS port follow the instructions under Configure TLS

  Note
  All port changes should be made in the start.ini file, which overrides jetty.port.

• Use your own custom certificate