Encrypted Shuffle and Encrypted Web UIs

Owned by Juliane Wetzel
Apr 30, 2020

Encrypted Shuffle

A security measure that allows for encrypting of the MapReduce shuffle using HTTPS as well as adding optional client authentication. This allows data being stored within the Mapper and Reducer to remain secure.

Encrypted Web UIs

Secure web server ports using HTTPS.

CDH versions 4.1 and above released support encryption of all HDFS data sent over the network.

Use the following steps if RPC encryption is configured after configuring Datameer X  

  1. Shut down the Datameer X service.
  2. Change the hadoop.rpc.protection setting on the cluster to privacy.
  3. Open the <datameer-install-path>/conf/das-common.properties file.
  4. Add the hadoop.rpc.protection=privacy setting in the HADOOP Properties Additions section.
  5. Start the Datameer X service.
  6. Check proper startup via tail -f <datameer-install-path>/logs/conductor.log
    If start up fails, gather the new conductor.log and revert back to its former state
  7. Set yarn.resourcemanager.webapp.https.address instead of webapp.address.
  8. Add the configuration property (hadoop.rpc.protection=privacy) in the Custom Properties field while configuring the Hadoop cluster in Datameer.


Note: Datameer X doesn't need any configuration changes to work with a SSL enabled cluster, but in MR2 architecture, Datameer X uses the ResourceManager web server to get node and cluster metrics. For this to work properly, the Datameer X application should be aware of the truststore and its password. To make Datameer X aware of this, go to <Datameer_Home>/etc/das-env.sh add the following system property to Java Options:


-Djavax.net.ssl.trustStorePassword=<password> -Djavax.net.ssl.trustStore=<path to truststore file>


Learn more about the Encrypted Shuffle and Encrypted Web UIs in Hadoop Security: Protecting your Big Data Platform.Â