Datameer Spotlight

Owned by Juliane Wetzel
Last updated: Jan 15, 2021

INFO

Datameer Spotlight gives organizations fast access and deep visibility into all of their enterprise data assets - whether in the cloud or on-premises - via a single unified self-service platform. With Datameer Spotlight business teams can discover, access, collaborate and analyze more data for faster, more trusted cloud analytics while eliminating complex data movement and maintaining strong governance.

Datameer X Prerequisites

To connect to Datameer Spotlight, the following prerequisites must be fulfilled:

  • your Datameer X instance must be from version 7.4 or higher
  • your Datameer X cluster must be run in grid mode 
  • your Hadoop cluster/ HDFS must be configured to allow access to the data within the HDFS
  • the plug-in 'Datameer DWH API' (including two extensions) must be installed
    INFO: The plug-in is installed as default Datameer X.
  • a job must be run at least once in Datameer X and also be kept
  • any data you wish to access through Spotlight must be in the 'Parquet' format (this has been the default format since Datameer 6.3)
  • your user must at least have the role 'ANALYST' to access the data:
    • read-only access to the Datameer folders and files

Datameer X Running on Hadoop/ HDFS

The following applies for running Datameer X on Hadoop/ HDFS:

  • the Datameer X instance must be reachable:
    • the REST API endpoint URL
    • user name
    • password
  • the Hadoop/ HDFS instance must be reachable:
    • the Hadoop NameNodes
    • the Hadoop DataNodes
  • this includes being able to resolve hostnames
  • all ports need to be accessible (Note that the ports might have been changed by your in-house administrators for security reasons. For specific Hadoop vendors other ports might be valid. Please view also the Hadoop vendor documentation): 
    • Hadoop NameNode client port: normally 8020, 9000 or 54310
    • Hadoop DataNodes: normally 50010 and 50020

Datameer X Running on Amazon EMR/ S3

The following applies for running Datameer X von EMR/ S3:

  • Spotlight must be able to read the data files at the S3 folder/ file locations, provided by the Datameer DWH API:
    • access key/secret must be known in the appropriate account that can read from Datameer X's internal storage bucket

    • these credentials need to be provided as additional "Connection Options" inside Spotlight when setting up the Datameer X connection using this format

      fs.s3a.access.key=your_access_key 
fs.s3a.secret.key=your_secret_key
    • these connection options only needed to be provided once for each user who needs to create connections to Datameer X. Future connections created by the same user will inherit these options.

These connection options are for creating connections only. Once a connection to Datameer X is created, other Spotlight users only need to provide their normal username and password to access that data (assuming they have permission in Datameer X to view that data).