When you schedule Datameer jobs which require additional resources that are unavailable at busy times, elasticity can help by scaling so those resource needs are always met. Amazon EMR is supported by Datameer so you have the ability to expand or shrink your processing hours on hardware based on your needs. This also applies to ad hoc workloads that sometimes require less resources and sometimes require more. And EMR's scaling configurations free administrators from managing individual applications for resources across your enterprise.
Setting up Datameer on EMR
Begin setting up EMR in Datameer by logging into Datameer with an account that has administrator permissions.
Under the Admin tab, select Hadoop Cluster from the menu on the left side of the screen. The current configurations for your Hadoop cluster are displayed. Click Edit at the bottom.
Select EMR Hadoop Cluster from the drop down menu under Cluster Mode.
Enter your Amazon S3 bucket address and the path to the storage folder.
Datameer uses S3 as storage for all files, both permanent and intermediate, for additional security.
Authenticate to your S3 bucket using your key/secret or select the box to authenticate via IAM role.
Under cluster settings, if the top box is left unchecked, the cluster name is used to pull the YARN host name.
Enter the name of the cluster running EMR.
Set the polling interval time in seconds for Datameer to check if there is a cluster with the name entered above.
Check the top box to enter a specific YARN host name for cluster.
Under Hadoop Properties, you can configure default property values or enter additional Hadoop distribution specific properties as well as custom properties.
Under logging options, select the severity of messages to be logged. The logging customization field allows to record exactly what is needed.
Click Save to complete the EMR Cluster setup.
Security
S3 authentication
When configuring the EMR Hadoop Cluster, you are presented with two options for authenticating to S3:
- Access Key/Secret
- IAM Role
Access key/secret
Datameer uses the Amazon S3 REST API with in turn uses a custom HTTP scheme based on a keyed-HMAC (Hash Message Authentication Code) for authentication. To authenticate a request, you first concatenate selected elements of the request to form a string. You then use your AWS secret access key to calculate the HMAC of that string. The output of the HMAC algorithm is the signature. It simulates the security properties of a real signature. This signature is added to the request in the standard HTTP Authorization header using the syntax "Authorization: AWS AWSAccessKeyId:Signature".
When the system receives an authenticated request, it fetches the AWS secret access key that you claim to have and uses it in the same way to compute a signature for the message it received. It then compares the signature it calculated against to the signature presented by the requester. If the two signatures match, the system concludes that the requester must have access to the AWS secret access key and therefore acts with the authority of the principal to whom the key was issued. If the two signatures do not match, the request is dropped and the system responds with an error message.
IAM role
IAM roles provide a convenient alternative to using access key/secret for authenticating to S3 from Amazon EC2 instances. When this option is selected, Datameer's S3 client uses the instance profile credentials to sign and authenticate the S3 requests. Instance profile credentials exist within the instance metadata associated with the IAM role for the EC2 instance. The EC2 instance on which Datameer runs is launched with the appropriate IAM role/instance profile. The same is used for launching the EMR Cluster. It is usually sufficient to use the default EC2 instance profile, EMR_EC2_DefaultRole, to launch both the EMR Cluster and the Datameer EC2 instance. The EMR instance, EC2 instance and S3 Bucket must be in the same AWS Region.
Encryption
Datameer uses S3 as storage for both permanent and intermediate files. Datameer does not write any intermediate or cached data locally on the cluster or to HDFS. The following diagram gives a high-level overview of supported encryption mechanisms.
Encrypting data at rest
Datameer supports encrypting data at rest on S3. The following server-side encryption mechanisms for S3 are supported within Datameer. Datameer doesn't support Amazon S3 Client-Side Encryption.
Datameer doesn't support explicit encryption of cached preview data, properties, configuration, deployment artifacts, or log files on the Datameer EC2 instance, although Amazon EC2 supports encryption of local disks.
LUKS encryption:
The Amazon EC2 instance store volumes and the attached Amazon EBS volumes of cluster instances are encrypted using LUKS. For more information about LUKS encryption, see the LUKS on-disk specification. At-rest encryption doesn't encrypt the EBS root device volume (boot volume).
Amazon EMR version 5.7.0 or later supports encryption of the EBS root device volume by specifying a custom AMI. For more information, see Customizing an AMI in the Amazon EMR Management Guide and How to protect data at rest with Amazon EC2 instance store encryption.
Server-side encryption with Amazon S3-Managed encryption keys (SSE-S3):
Server-side encryption with Amazon S3-managed encryption keys (SSE-S3) employs strong multi-factor encryption. Amazon S3 encrypts each object with a unique key. As an additional safeguard, it encrypts the key itself with a master key that it regularly rotates. Amazon S3 server-side encryption uses 256-bit Advanced Encryption Standard (AES-256), to encrypt your data.
To use SSE-S3, you need to create a bucket policy that enforces encryption as described here: https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingServerSideEncryption.html. Here is an example of a typical bucket policy:
{
"Version": "2012-10-17",
"Id": "PutObjPolicy",
"Statement": [
{
"Sid": "DenyIncorrectEncryptionHeader",
"Effect": "Deny",
"Principal": "*",
"Action": "s3:PutObject",
"Resource": "arn:aws:s3:::aes-encrypted-test-bucket/*",
"Condition": {
"StringNotEquals": {
"s3:x-amz-server-side-encryption": "AES256"
}
}
},
{
"Sid": "DenyUnEncryptedObjectUploads",
"Effect": "Deny",
"Principal": "*",
"Action": "s3:PutObject",
"Resource": "arn:aws:s3:::aes-encrypted-test-bucket/*",
"Condition": {
"Null": {
"s3:x-amz-server-side-encryption": "true"
}
}
}
]
On the Datameer side, you need to specify the custom property das.fs.s3-bucket.encryption.type=AES. When this property is present, Datameer uses the appropriate encryption header in all S3 requests to ensure encryption for all objects that are stored in your bucket. Both authentication mechanisms, instance profile and access key/secret based credentials, are supported.
Server-side encryption with AWS KMS–managed keys (SSE-KMS):
AWS Key Management Service (AWS KMS) is a service that combines secure highly available hardware and software to provide a key management system scaled for the cloud. AWS KMS uses customer master keys (CMKs) to encrypt your Amazon S3 objects. You use AWS KMS via the Encryption Keys section in the IAM console or via AWS KMS APIs to centrally create encryption keys, define the policies that control how keys can be used, and audit key usage to prove they are being used correctly. The first time you add an SSE-KMS–encrypted object to a bucket in a region, a default CMK is created for you automatically. This key is used for SSE-KMS encryption unless you select a CMK that you created separately using AWS Key Management Service. Creating your own CMK gives you more flexibility, including the ability to create, rotate, disable, and define access controls, and to audit the encryption keys used to protect your data.
Besides creating a KMS key and a key policy, you need to create a bucket policy that enforces encryption as described here: https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingKMSEncryption.html. Here is an example of a typical bucket policy:
{
"Version": "2012-10-17",
"Id": "PutObjPolicy",
"Statement": [
{
"Sid": "DenyIncorrectEncryptionHeader",
"Effect": "Deny",
"Principal": "*",
"Action": "s3:PutObject",
"Resource": "arn:aws:s3:::kms-encrypted-test-bucket/*",
"Condition": {
"StringNotEquals": {
"s3:x-amz-server-side-encryption": "aws:kms",
"s3:x-amz-server-side-encryption-aws-kms-key-id": "arn:aws:kms:us-east-1:123456789012:key/6789abcd-89ab-ef01-3456-456789abcdef"
}
}
},
{
"Sid": "DenyUnEncryptedObjectUploads",
"Effect": "Deny",
"Principal": "*",
"Action": "s3:PutObject",
"Resource": "arn:aws:s3:::kms-encrypted-test-bucket/*",
"Condition": {
"Null": {
"s3:x-amz-server-side-encryption": "true"
}
}
}
]
}
On the Datameer side, you need to specify the custom property das.fs.s3-bucket.encryption.type=KMS. When this property is present, Datameer uses the appropriate encryption header in all S3 requests to ensure encryption for all objects that are stored in your bucket. Both authentication mechanisms, instance profile and access key/secret based credentials, are supported.
Encrypting data In transit
Datameer implicitly supports encryption mechanisms for data in transit that are supported by Amazon EMR and S3. The encryption mechanisms are EMR release version and application (e.g., Hadoop, Tez, etc.) specific, and don't require any special handling in Datameer other than configuration.
- Hadoop
See Hadoop in Secure Mode in Apache Hadoop documentation. In this configuration, Hadoop RPC is set to "Privacy" and uses SASL. This is activated in Amazon EMR when in-transit encryption is enabled.
Rename and put your certificates in a zip file as documented in https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-encryption-certificates.html. Upload the zip to S3. Create a new Security Configuration from EMR Console. Disable at-rest encryption and enable in-transit encryption. In the TLS certificate provider section, choose "PEM" as Certificate provider type and enter fully qualified S3 path (e.g., "s3://test-bucket/my-certs.zip") for the certificates zip file in the S3 object field. Choose this Security Configuration while creating your EMR Cluster. If a wrong certificates zip path is specified, the Create Security Configuration and Create Cluster operations are still successful, but the cluster is terminated with an error "Access denied when trying to download from s3://test-bucket/my-certs.zip".
In Datameer, the following properties need to be specified in the Hadoop Cluster page custom properties section:
hadoop.rpc.protection=privacy
- Tez
See Tez Runtime Configuration in Tez documentation. In this configuration, Tez Shuffle Handler uses TLS (tez.runtime.ssl.enable).
In Datameer, the following properties need to be specified in the Hadoop Cluster page custom properties section:
tez.runtime.shuffle.ssl.enable=true
tez.runtime.shuffle.keep-alive.enabled=true
- Amazon S3
Calls are made to Amazon S3 using REST over HTTPS in Datameer. HTTPS is the default protocol for the S3 client SDK used by Datameer. No further configuration is necessary.
- MySQL
When MySQL is installed on Amazon RDS, SSL needs to be enabled on MySQL. Install the corresponding server certificate on the Datameer instance.
- Datameer Web/REST API
To enforce in-transit encryption for all calls from the client browser to the Datameer app server, SSL must be enabled in Jetty. If using a custom certificate, installed it on the Datameer ec2 instance. These instructions are provided in the Datameer's Installation Guide.
Using the REST API for EMR
Datameer's REST API is available to view and update EMR configrations.
View Datameer's EMR REST API.