/
Supported Data Sources

Supported Data Sources

Aug 02, 2019

Supported Data Types

Format / System

Importing and exporting

Product tested version

Datameer version availability

Notes
ImportExportImportExport
Amazon Redshift(tick) (tick) 8.2, 8.4

Native Amazon Redshift JDBC 4.1 driver or a PostgreSQL jdbc driver can be used.
Apache Avro(tick)(tick)


Supports default compression types.
Apache Web Server Logs(tick)(error)



Azure Blob Storage(tick)(tick)


(For HDP2.0, 2.2 and CDH+4 users)

Contact Datameer services for info.

Cassandra(tick)(error)0.6.5

Public available plugin, see https://github.com/zznate/datameer-cassandra-plugin
Cobol Copybooks(tick)(error)



CSV/TSV, etc.(tick)(tick)


Supports default compression types.
DB2(tick)(tick)

8.1 (since 1.3.9), 9.7 Express-C




Excel Workbooks(tick) (error)2007 and after


Fixed Width Text(tick)(error)



Google BigQuery(tick)

(error)


7.5

Google Spreadsheets(tick)

(error)





Greenplum(tick)(tick) HD 1.1, HD 2.1, 4.1.1.1


HBase(tick)(error)0.98.x, 0.96.x, 0.94.x, 0.92.x, 0.90.x

In order to satisfy the classloader requirements, hbase-protocol.jar must be included in Hadoop's classpath and the root Datameer classpath (/etc/custom-jars) for version 0.96.1 to 0.98.0

Hive JDBC(tick) (error)



Hive Metastore(tick) (tick) 0.13, 0.12, 0.11, 0.10, 0.9, 0.8, 0.7

Export file format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
HiveServer2(tick) (tick) 

0.13, 0.14, 1.1.0.x




HSQL-DB(tick)(tick)



HTML(tick) (error)



JSON(tick) (error)



Log4j Log File(tick) (tick) 



MBOX (email archive files)(tick)(error)



MS IIS Web Server Logs(tick)(error)



MSSQL(tick)(tick)SQL Express 2005 and 2008


MySQL(tick)(tick)5.5, 5.6, 5.7


Netezza(tick) (tick) 6.0.6


Oracle(tick)(tick)10g XE, 11g XE


OpenStack Swift(tick)(tick)



ORC (Optimized Row Columnar)(tick)(tick)

7.1
  • Only supported in conjunction with Hive.
  • Supports default compression types.
  • Export supported only for existing partitioned Hive tables.
Parquet(tick) (tick) Parquet 2.1

PostgreSQL(tick) (tick) 9.0.x


PowerBI(error)(tick) 



RCFile (Record Columnar File)(tick) (tick)

7.1

Export supported in conjunction with Hive only for existing partitioned Hive tables.

Sequence Files with Metadata(tick)(error)


Supports default compression types.
Snowflake(tick)(tick)
7.17.1
Spark(tick)(tick)



Sybase IQ(tick) (tick) 12.7


Tableau TDE(error)(tick)



Tableau TDSX(error)(tick)



Tableau Hyper(error)(tick)

7.4

Hadoop cluster native system library requirements:

  • GNU C Library (libc6) version >= 2.15
  • GNU Standard C++ Library v3 (libstdc++6) version >= 6.1.0
Teradata(tick) (tick)12 & 13

Teradata database needs to be configured to support the appropriate character set.
Teradata Aster(tick)(tick)5.0


Text Files(tick)(error)


Supports default compression types.

Vertica

(tick)

(tick)

5, 6.0, 6.1




XML

(tick) 

(error)




Supports default compression types.


Supported File Protocols

Protocol

Input

Output

File

(tick)

(tick)

HDFS

(tick)

(tick)

SSH (SCP and SFTP)

(tick)

(tick)

S3

(tick)

(tick)

Datameer supports Bitverse SSH Server/Client for the Windows platform. The root paths to be specified while creating the connection should look something like: /c:/mydata/folder1

Supported file compression codecs

Codec

Input

Output

Default compressionNotes

.gz

(tick)

(error)



.bz2

(tick)

(error)



.lzo

(tick)

(tick)

(tick)Additional native libraries are required.
Snappy(tick)

(error)

(tick)

.zip

(tick)

(tick)(tick)
  • Supported File Types: CSV, JSON, XML
  • Size: No size limitation from Datameer
  • Amount of Files: Support of zip files containing 1 file
  • Structure: File has to be direct in the root of the zip-file
.Z(tick)(error)

* LZO: tested with Fedora lzo.i386 v2.02-3.fc8 and http://github.com/kevinweil/hadoop-lzo (state from 2010-Jun-20)

Supported File Systems

Customer

File system

Special Hadoop configuration

appistry

storage:/

fs.storage.impl=org.apache.hadoop.fs.appistry.FabricStorageFileSystem
fs.abs.impl=org.apache.hadoop.fs.appistry.BlockedFabricStorageFileSystem
fs.appistry.storage.host=localhost
fs.appistry.chunked=false
fs.appistry.jetty.port=8085