Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

Version 1 Current »

When configuring Datameer for Hortonworks HDP, the following properties need to be set in the Custom Properties section of the Hadoop Cluster page: 

hdp.version=<hdp version>
mapreduce.application.framework.path=/hdp/apps/${hdp.version}/mapreduce/mapreduce.tar.gz#yarn
mapreduce.application.classpath=/usr/hdp/${hdp.version}/hadoop/conf:/usr/hdp/${hdp.version}/hadoop/lib/*:/usr/hdp/${hdp.version}/hadoop/*:/usr/hdp/${hdp.version}/hadoop-hdfs/:/usr/hdp/${hdp.version}/hadoop-hdfs/lib/*:/usr/hdp/${hdp.version}/hadoop-hdfs/*:/usr/hdp/${hdp.version}/hadoop-yarn/lib/*:/usr/hdp/${hdp.version}/hadoop-yarn/*:/usr/hdp/${hdp.version}/hadoop-mapreduce/lib/*:/usr/hdp/${hdp.version}/hadoop-mapreduce/*:/usr/hdp/current/hadoop-mapreduce-client/*

The hdp.version property value should reflect the exact version of the HDP Hadoop cluster Datameer is connecting to. These properties are prefilled in the Hadoop Distribution Specific Properties section of the Edit Hadoop Cluster page.

Setting Up HDP with Spark

When configuring Datameer with Spark for Hortonworks HDP, the following properties need to be set in the Custom Properties section of the Hadoop Cluster page: 

hdp.version=<hdp version>
spark.driver.extraJavaOptions=-Dhdp.version=<hdp version>
spark.yarn.am.extraJavaOptions=-Dhdp.version=<hdp version>
spark.driver.extraLibraryPath=/usr/hdp/${hdp.version}/hadoop/lib/native
spark.executor.extraLibraryPath=/usr/hdp/${hdp.version}/hadoop/lib/native
spark.yarn.am.extraLibraryPath=/usr/hdp/${hdp.version}/hadoop/lib/native

The hdp.version property value should reflect the exact version of the HDP Hadoop cluster Datameer is connecting to. These properties are prefilled in the Hadoop Distribution Specific Properties section of the Edit Hadoop Cluster page.

  • No labels