Page Comparison

...

Section

Create Import/ Link/ Upload

Request Method POST

Description

Adds an import/ link/ upload file in Datameer.

Info

title	INFO

The maximum length for a table description is 255 characters.

URI Syntax

Code Blocknoformat
curl -u '<username>:<password>' -X POST -d @<job-payload>.json 'http://<Datameer-server-IP>:<port-number>/rest/import-job'

Example Response

Code Block
{ "configuration-id": 48, "status": "success" }

Additional Notes

Only use REST for file uploads when running in local mode. If the file isn't local, use import jobs or data links instead of a file upload.

<job-payload>.json

The payload is a JSON representation of all information entered during the creation process using the web interface.
An easy method of creating a payload is to use REST to read an import job, data link, or workbook and change the JSON that is returned

Section

Read Import/ Link/ Upload

Request Method

GET

Description

Returns information on a specified file imported/linked/uploaded within Datameer X including version number, class name, file information (uuid, path, description, name), run schedule, iterations set to keep, properties information, custom properties, and field data.

URI Syntax

Code Blocknoformat
curl -u '<username>:<password>' -X GET 'http://<Datameer-serverIP>:<port-number>/rest/import-job/<job-configuration-id>'

Example Response

Code Block

{
  "version": "2.1.1",
  "className": "datameer.dap.common.entity.DataSourceConfigurationImpl",
  "file": {
    "uuid": "2d5d3741-9f5d-42e1-9d54-e38c849f3a5a",
    "path": "/Data/ImportJobs/URLS.imp",
    "description": "",
    "name": "URLS"
  },
  "pullType": "MANUALLY",
  "minKeepCount": 1,
  "properties": {
    "GenericConfigurationImpl.temp-file-store": [
      "f4cf6037-042a-4c56-99e3-74b59753729c"
    ],
    "fileType": [
      "CSV"
    ],
    "filter.page.does.split.creation": [
      "false"
    ],
    "external.store": [
      "false"
    ],
    "delimiter": [
      ","
    ],
    "csv.max-lines-per-record": [
      "1"
    ],
    "file": [
      "/Users/charlesbishop/Desktop/URLTEST.csv"
    ],
    "detectColumnDefinition": [
      "SELECT_PARSE_AUTO"
    ],
    "quoteCharacter": [
      "\""
    ],
    "escapeCharacter": [
      ""
    ],
    "recordSampleSize": [
      "1000"
    ],
    "data.includes.header": [
      "true"
    ],
    "characterEncoding": [
      "UTF-8"
    ],
    "filter.maxAge": [
      ""
    ],
    "filter.minAge": [
      ""
    ],
    "fileNameTimeRange_startDate": [
      ""
    ],
    "fileNameTimeRange_mode": [
      "OFF"
    ],
    "collectAdditionalFields": [
      "false"
    ],
    "strictQuotes": [
      "false"
    ],
    "TextFileFormat": [
      "TEXT"
    ],
    "histogram.generation": [
      "false"
    ],
    "incrementalMode": [
      "false"
    ]
  },
  "hadoopProperties": "",
  "dataStore": {
    "path": "/Data/Connections/Datameer X server filesystem.dst",
    "uuid": "5875432e-da74-4947-8f81-7ba13347bdde"
  },
  "validationStrategy": "DROP_RECORD",
  "maxLogErrors": 1000,
  "maxPreviewRecords": 5000,
  "fields": [
    {
      "id": 31,
      "pattern": "",
      "acceptEmpty": true,
      "name": "URL",
      "origin": "0",
      "valueType": "{\"type\":\"STRING\"}",
      "include": true,
      "version": 3
    },
    {
      "id": 32,
      "pattern": "",
      "acceptEmpty": false,
      "name": "dasFileName",
      "origin": "fileInfo.fileName",
      "valueType": "{\"type\":\"STRING\"}",
      "include": false,
      "version": 3
    },
    {
      "id": 33,
      "pattern": "",
      "acceptEmpty": false,
      "name": "dasFilePath",
      "origin": "fileInfo.filePath",
      "valueType": "{\"type\":\"STRING\"}",
      "include": false,
      "version": 3
    },
    {
      "id": 34,
      "pattern": "",
      "acceptEmpty": false,
      "name": "dasLastModified",
      "origin": "fileInfo.lastModified",
      "valueType": "{\"type\":\"DATE\"}",
      "include": false,
      "version": 3
    }
  ]
}

Section

Upload Import/ Link/ Upload

Verification

Request Method PUT

Description

Edits an existing import/ link/ upload file in Datameer.

URI Syntax

Code Blocknoformat
curl -u '<username>:<password>' -X PUT -d @<job-payload>.json 'http://<Datameer-serverIP>:<port-number>/rest/import-job/<job-configuration-id>'

Example Response

Code Block
{"status": "success"}

Additional Notes

<job-payload>.json

The payload is a JSON representation of all information entered during the creation process using the web interface.
An easy method of creating a payload is to use REST to read an import job, data link, or workbook and change the JSON that is returned.

Section

Delete Import/ Link/ Upload

Request Method

DELETE

Description

Removes an import/ link/ upload file from Datameer.

URI Syntax

Code Blocknoformat
curl -u '<username>:<password>' -X DELETE 'http://<Datameer-serverIP>:<port-number>/rest/import-job/<job-configuration-id>'

Example Response

Code Block
{"status": "success"}

Section

View Metadata for Import/ Link/ Upload

Request Method

GET

Description

Returns data on the import/ link/ upload file including the path, number of records, time stamp, and ID.

URI Syntax

Code Blocknoformat
curl -u '<username>:<password>' -X GET 'http://<Datameer-serverIP>:<port-number>/rest/data/import-job/{configuration-id}'

Example Response

Code Block
{ "path": "/Data/ImportJobs/DatameerImport.imp", "record-count": 20, "datas": [ { "timestamp": "Mar 5, 2013 10:44:59 AM", "record-count": 20, "id": 19372 } ]

Section

Download Data of Import/ Link/ Upload

Request Method GET

Description

Returns the column names and records of the import/ link/ upload file

Info

title	INFO

The default limit for number of rows to download is 100,000, as this functionality is intended for small aggregated data sets. To adjust the record download limit, change the rest.download-data.records-max=100000 property in the <datameer_install>/conf/default.properties file. Setting the value to 0 unlocks the limit, but increasing the number could result in slower processing for the Datameer X conductor. Your local system might also have download restrictions. If you want to download more than 100,000 rows, we suggest creating an export job instead.

URI Syntax

Code Blocknoformat

curl -u '<username>:<password>' -X GET 'http://<Datameer-serverIP>:<port-number>/rest/data/import-job/{configuration-id}/download'
curl -u '<username>:<password>' -X GET 'http://<Datameer-serverIP>:<port-number>/rest/data/import-job/{configuration-id}/download?dataId={data-id}'

Example Response

Code Block

"ID","NAME","Email","Status","Check"
"1","""admin""","""admin@datameer.com""","0","true"
"2","""Ajay""","""Ajay@datameer.com""","2","true"
"3","""analyst""","""analyst@datameer.com""","1","true"
"4","""Bob""","""Bob@datameer.com""","2","false"
"5","""Alice""","""Alice@datameer.com""","2","false"
"6","""John""","""John@datameer.com""","2","false"
"7","""Andreas""","""Andreas@datameer.com""","2","true"
"8","""Arthur""","""Arthur@datameer.com""","2","true"
"9","""Dagmar""","""Dagmar@datameer.com""","2","false"
"10","""Frank""","""Frank@datameer.com""","2","true"
"11","""Johannes""","""Johannes@datameer.com""","2","true"
"12","""Marko""","""Marko@datameer.com""","2","true"
"13","""Marlon""","""Marlon@datameer.com""","2","true"
"14","""Mike""","""Mike@datameer.com""","2","true"
"15","""Orlando""","""Orlando@datameer.com""","2","true"
"16","""Peter""","""Peter@datameer.com""","2","true"
"17","""Ralf""","""Ralf@datameer.com""","2","false"
"18","""Shirish""","""Shirish@datameer.com""","2","true"
"19","""Stefan""","""Stefan@datameer.com""","2","true"
"20","""Teresa""","""Teresa@datameer.com""","2","true"

Additional Notes

The data ID can be found a job's details under Current Data.

The maximum number of records that can be returned by the API is configurable in conf/default.properties (and set to 100.000 by default).

Code Block

# The maximum number of records that should be returned by the download data REST API. 
# All additional records will be ignored and 206 HTTP response code (PARTIAL CONTENT) is returned.
rest.download-data.records-max=100000

Section

View Sheet Details of Import/ Link/ Upload

Request Method

GET

Description

eturns Returns the column names and data type of an import/ link/ upload file.

URI Syntax

Code Blocknoformat

curl -u '<username>:<password>' -X GET 'http://<Datameer-serverIP>:<port-number>/rest/sheet-details/{job-configuration-id}'
curl -u '<username>:<password>' -X GET 'http://<Datameer-serverIP>:<port-number>/rest/sheet-details?file=<path to import job>.<extension>'

Example Response

Code Block

{
    "columns": [
        {
            "name": "id",
            "type": "INTEGER"
        },
        {
            "name": "ident",
            "type": "STRING"
        },
        {
            "name": "type",
            "type": "STRING"
        },
        {
            "name": "name",
            "type": "STRING"
        },
        {
            "name": "latitude_deg",
            "type": "FLOAT"
        },
        {
            "name": "longitude_deg",
            "type": "FLOAT"
        },
        {
            "name": "elevation_ft",
            "type": "INTEGER"
        },
        {
            "name": "continent",
            "type": "STRING"
        },
        {
            "name": "iso_country",
            "type": "STRING"
        },
        {
            "name": "iso_region",
            "type": "STRING"
        },
        {
            "name": "municipality",
            "type": "STRING"
        },
        {
            "name": "scheduled_service",
            "type": "STRING"
        },
        {
            "name": "gps_code",
            "type": "STRING"
        },
        {
            "name": "iata_code",
            "type": "STRING"
        },
        {
            "name": "local_code",
            "type": "STRING"
        },
        {
            "name": "home_link",
            "type": "STRING"
        },
        {
            "name": "wikipedia_link",
            "type": "STRING"
        },
        {
            "name": "keywords",
            "type": "STRING"
        }
    ],
    "file": {
        "path": "/Examples/Public Data/Flight Delays/Airports.upl",
        "uuid": "39b5961a-4caa-4b9e-880e-d40f03b7543a"
    }
}

Section

View Import Job Dependencies

Request Method

GET

Description

Returns the hierarchy of up- and downstream dependencies for a given import job.

Prerequisites

This API requires enabling Datameer's Advanced Governance plug-in.

URI Syntax

Code Blocknoformat
curl -u '<username>:<password>' -X GET 'http://<server>:<port>/api/import-job/<configurationId>/dependencies[?direction=<upstream \| downstream>][&level=<number>]'

Example Response

Code Block

{
  "dependencies": [
    {
      "id": 23,
      "file": {
        "path": "/Users/admin/WorkbookTest",
        "name": "WorkbookTest",
        "type": "wbk",
        "description": "",
        "uuid": "db942d97-4785-436d-938b-45053e12bb75"
      }
    },
    {
      "id": 2,
      "file": {
        "path": "/Data/Connections/Datameer X server filesystem",
        "name": "Datameer X server filesystem",
        "type": "dst",
        "description": "A data store for the local filesystem where Datameer X is running.  Note, this works ONLY for Datameer X Personal, Workgroup, and Trial editions.\n",
        "uuid": "78a2fb59-375e-4eae-b338-956ef806f7b1"
      }
    }
  ],
  "fileCountWithNoReadPermission": 0,
  "_links": {
    "self": {
      "href": "http://localhost:8088/api/import-job/22/dependencies"
    }
  }

Additional Notes

Path parameters:

configurationId:
- Data type must be a number
- Use the configuration ID

Query parameters:

Direction:
- Data type must be a string
- Optional
- Shows either upstream or downstream
- If left blank, shows both
Level:
- Data type must be a number
- Optional
- If left blank, shows all levels

Section

List All Imports/ Links/ Uploads

Request Method

GET

Description

Returns a list of all the import/ link/ upload file in Datameer. This list displays the description, ID number, name, and the path of the file.

URI Syntax

Code Blocknoformat
curl -u '<username>:<password>' -X GET 'http://<Datameer-serverIP>:<port-number>/rest/import-job'

Example Response

Code Block

[ 
    {
        "description": "apache log file",
        "id": 5048,
        "name": "foo4965",
        "path": "/ImportJob2/foo4965.imp"
    },
    {
        "description": "apache log file",
        "id": 5049,
        "name": "foo4966",
        "path": "/ImportJob2/foo4966.imp"
    },
    {
        "description": "apache log file",
        "id": 5050,
        "name": "foo4967",
        "path": "/ImportJob2/foo4967.imp"
    },
    {
        "description": "apache log file",
        "id": 5051,
        "name": "foo4968",
        "path": "/ImportJob2/foo4968.imp"
    }
]

Section

Start Import/ Link/ Upload

Prerequisites

Request Method

POST

Description

Runs an import/ link/ upload file.

URI Syntax

Code Blocknoformat
curl -u '<username>:<password>' -X POST 'http://<Datameer-server-IP>:<port-number>/rest/job-execution?configuration=<import-job-configuration-id>'

Example Response

Code Block
{ "job-execution-id": 26, "job-execution-trigger": "RESTAPI", "job-execution-user": "admin", "status": "success" }

Additional Notes

Only use REST for file uploads when running in local mode. If the file isn't local, use import jobs or data links instead of a file upload.

Section

Cancel Import/ Link/ Upload

Request Method

POST

Description

Cancels the run order of a import /link/ upload file.

URI Syntax

Code Blocknoformat
curl -u '<username>:<password>' -X POST 'http://<Datameer-serverIP>:<port-number>/rest/job-execution/job-kill/<import-job-execution-id>'

Example Response

This command doesn't return a response.

Versions Compared

Old Version 1

New Version 2

Key

Create Import/ Link/ Upload

Read Import/ Link/ Upload

Upload Import/ Link/ Upload

Delete Import/ Link/ Upload

View Metadata for Import/ Link/ Upload

Download Data of Import/ Link/ Upload

View Sheet Details of Import/ Link/ Upload

View Import Job Dependencies

List All Imports/ Links/ Uploads

Start Import/ Link/ Upload

Cancel Import/ Link/ Upload