Data Field Types in Datameer
When importing data into Datameer X it is important to know which type of data you are importing or need to import. When using your data within workbooks, certain Datameer X functions only work with certain types of data. Congruently, certain functions return only a specific type of data.
Also there are data requirements when using infographic widgets to visualize your data.
- 1 Data Field Types
- 1.1 Integer
- 1.2 Big integer
- 1.3 Float
- 1.4 Big decimal
- 1.5 Date
- 1.6 String
- 1.7 Boolean
- 1.8 List
- 1.9 Number
- 1.10 Any
- 2 Data Mapping in Avro
- 2.1 Import mapping
- 2.2 Export mapping
- 3 Data Mapping in Parquet
- 4 Data Mapping in External Systems
Data Field Types
Field type | Product icon | Description | Internal representation |
|---|---|---|---|
INTEGER |  | 64-Bit integer value | Java Long |
BIG_INTEGER | | Unlimited integer value | Java BigInteger |
FLOAT | | 64-Bit float value | Java Double |
BIG_DECIMAL | | High-precision float value | Java BigDecimal |
DATE | | Date object | Java Date |
STRING | | String object | Java String |
BOOLEAN | | Boolean object | Java Boolean |
LIST | | a collection of multiple values of one data type |
|
NUMBER | | float, big decimal, integer, or big integer |
|
ANY | | float, big decimal, integer, big integer, date, string, list, or Boolean |
|
In mathematics integers (aka whole numbers) are made up of the set of natural numbers including zero (0,1,2,3, ...) along with the negatives of natural numbers (-1,-2,-3, ...). When talking about Integers in computer programming, it is necessary to define a minimum and maximum value. Datameer X uses a 64-bit integer which allows the user to represent whole numbers between -9,223,372,036,854,775,808 to 9,223,372,036,854,775,807.
Big integers are like integers, but they are not limited to 64 bits. They are represented using arbitrary-precision arithmetic. Big integers represent only whole numbers. Big integers in Datameer X are treated differently than in Hive because Datameer X allows a larger range of values, so they are written as strings into a Hive table if you export. By default, the precision for big integers is set at 32 upon import. This can updated if needed in the default properties by changing the value of das.big-decimal.precision=.
In mathematics, there are real numbers that represent fractions (1/2, 12/68) or numbers with decimal places (12.75, -18.35). Datameer X uses double precision floating-point representation (aka float) to manipulate and represent real numbers. The complete range of numbers that can be represented this way is approximately 2 -1022 through (1+(1-2 -52))x2 1023. During import/upload, Datameer X automatically recognizes a number with either a single period (.) or single comma (,) as a decimal separator and defines this data as a float data type. After ingestion, Datameer X stores float and big decimal values using a period (.) character. The auto schema detection for the float data type works with CSV, JSON, XML, Key/value files.
Big decimals are similar to float values. The main advantage of this data field type is that they are exact to the number of decimal places for which they are configured, float values might be inaccurate in certain cases. If a number has more decimal places than big decimal was configured for, then the number is rounded. The number of decimal places can be configured in conf/default.properties:
# Maximum precision used for BIG_DECIMAL types. Precision is equal to the maximum number of digits a BigDecimal
# can have.
system.property.das.big-decimal.precision=32
32 digits is the default precision used by Datameer X for big decimal values upon import.
In Datameer X data in the DATE primitive data type is always represented in a Gregorian, month-day-year (MDY) format (e.g., "Sep 16, 2010 02:56:39 PM"). Datameer X detects if your data should be parsed into the DATE data type during ingest. This can also be done after ingest as other data types can be converted to the DATE primitive data type using workbook functions.
When using information other than numbers or dates in Datameer X it is represented as a string. This includes text, unparsed date patterns, URLs, JSON arrays, etc.
Boolean data in computing has two values, either true or false. It is used in many logical expressions and is derived from Boolean algebra created by George Boole in the 19th century.
In Datameer X multiple values can be combined into a list. Lists are a series of values of a single data type, which starts counting from zero (0).
In Datameer X integers, big integers, floats, and big decimals are considered to be numbers.
Some visualizations and functions are able to use data represented by any data field type. These can be either a number, a string, a date, or a Boolean.
Data Mapping in Avro
Import mapping
When importing data to Datameer X data types are mapped as follows:
Avro Schema Type | Datameer X Value Type |
|---|---|
Null | STRING |
Boolean | BOOLEAN |
Int | INTEGER |
Long | INTEGER |
Float | FLOAT |
Double | FLOAT |
Bytes | STRING |
Bytes with logical type Decimal | BIG_DECIMAL |
String | STRING |
Records | STRING |
Enums | STRING |
Arrays | STRING |
Maps | STRING |
Unions | STRING |
Fixed | STRING |
Fixed with logical type Decimal | BIG_DECIMAL |