Versions Compared

Old Version 3

changes.mady.by.user Azad Derakhshani

Saved on

compared with

New Version Current

changes.mady.by.user Juliane Wetzel

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Table of Contents

...

  1. Click the + (plus) button at the top left of the File Browser and select Connection, or right-click on a folder, select Create New then select Connection.
     
  2. From drop-down list, select Hive Server2 as the connection type. Click Next.

  3. Enter the HiveServer2 connection address and port number. 
    Select TCP/Binary or HTTP as the transport mode. (Transport mode option is available as of Datameer X v6.3)
         - If selecting HTTP as the transport mode, the property  hive.server2.thrift.http.path has the hardcoded value of cliservice. Learn more at Apache Hive.
         - HTTP mode using SSL is not supported.
         - HTTP mode can only be used if Hive supports token based authentication . Hive versions >= 1.2 supports this feature. 
    Optional: 
         - Enter a database filter to limit the amount listed databases when importing with this connector.
         - Enter an export path where exports from this connector are written. 
         - Enter any additional custom properties.

    Warning

    If the database filter for a connection is updated, all import and export jobs using a prohibited database fail upon running.

    Security

    Kerberos

    Select Kerberos Secured Hive and enter both the Hive and HDFS principals.

    LDAP

    Select LDAP/AD Authentication and select between having the HiveServer2 connector provide the authorization credentials or have the credentials provided by the individual import/export jobs.

    Sentry 

    Datameer X respects Sentry permissions from users running HiveServer2.

    How Datameer X and Sentry interact with users:

    When you add an artifact though Datameer X to Hive, it is added using the Datameer X service account. The Datameer X service account must have all permissions for Hive to authenticate with Sentry.

    However, data is accessed on the Hive cluster using the Datameer X user account. The Datameer X user account must have permissions for the requested data in Hive as well as Datameer X to authenticate with Sentry.

    At the transport level, HiverServer2 HiveServer2 has multiple connection methods available. Datameer X currently supports the following binary connection methods:

    • SASL

    • Non SSL
    • NOSASL
    • Kerberos

    • Plain

  4. Fill out a description and save the connector.

...

The dialog itself provides more detailed descriptions. If you are unsure of the best choices, contact your Hive or Hadoop Administrator.

Note
titleAs of Datameer X 7.5

Datameer X is able to compute the partition location while importing data from a partitioned Hive table without needing to retrieve it from the HiveServer 2. This improves performance if the default partition location pattern is used. The setting can be made globally from the plugin configuration page, and the default value can be overwritten in the import job or data link wizard.

As of Datameer X 7.5, Datameer X Classic data field type mapping is removed and only Hive specific mapping is performed, where Datameer X BigDecimal maps to Hive Decimal and Datameer X Date maps to Hive TimeStamp.

...

There are options for configuring data field type mapping from Datameer X to HiveServer2 upon export.

  • In Datameer X versions through 7.4, "Classic" is the default setting for the Hive plug-in. In Datameer X versions 7.5+, Hive implicit mapping is used. Details of this mapping can be found under Exporting: Datameer X Versions 7.5+ .
  • Hive specific is similar to the default setting with the change that Datameer's BigDecimal is mapped to Hive's Decimal and Datameer's Date is mapped to Hive's Timestamp data type.
    • BigDecimal
      • New Table - Datameer X exports the BigDecimal type to the Hive type Decimal with a precision (total number of digits) of 38 and a scale (number of digits to the right of the point) of 18.
      • Existing Table - Datameer X exports the BigDecimal type to the Hive type Decimal with the precision and scale defined on the Hive server. 
        • A maximum is set at a precision scale of (38,37).
      • If an exported value doesn't fit within the precision/scale of either a new or existing table, a failure occurs. 
    • Date
      • New Table - Datameer X exports the Date type to the Hive type Timestamp.
      • Existing Table - Datameer X exports the Date type to the Hive type Timestamp/Date/String depending on what is defined on the Hive server.

The mode has no influence when exporting into an existing partitioned Hive table.

Cache

Datameer X caches Java objects that represent partitions on HiveServer2. These partitions contain locations, columns names, and other information. Datameer X stores these objects on a local disk. When Datameer X needs to read Hive partitions it increases performance by using the stored cache instead of having to pull the same information each time it is needed.

...