/
REST API Import Job, Data Link, and File Upload

REST API Import Job, Data Link, and File Upload

Jan 16, 2019

For each call, you can add the following command:

-w"\n%{http_code}\n"

After the username and password in order to receive more meaningful HTTP errors.

Read Import/Link/Upload

Request method - GET

Description

Returns information on a specified file imported/linked/uploaded within Datameer including version number, class name, file information (uuid, path, description, name), run schedule, iterations set to keep, properties information, custom properties, and field data.

URI syntax

curl -u '<username>:<password>' -X GET 'http://<Datameer-serverIP>:<port-number>/rest/import-job/<job-configuration-id>'

Example response

{ "version": "2.1.1", "className": "datameer.dap.common.entity.DataSourceConfigurationImpl", "file": { "uuid": "2d5d3741-9f5d-42e1-9d54-e38c849f3a5a", "path": "/Data/ImportJobs/URLS.imp", "description": "", "name": "URLS" }, "pullType": "MANUALLY", "minKeepCount": 1, "properties": { "GenericConfigurationImpl.temp-file-store": [ "f4cf6037-042a-4c56-99e3-74b59753729c" ], "fileType": [ "CSV" ], "filter.page.does.split.creation": [ "false" ], "external.store": [ "false" ], "delimiter": [ "," ], "csv.max-lines-per-record": [ "1" ], "file": [ "/Users/charlesbishop/Desktop/URLTEST.csv" ], "detectColumnDefinition": [ "SELECT_PARSE_AUTO" ], "quoteCharacter": [ "\"" ], "escapeCharacter": [ "" ], "recordSampleSize": [ "1000" ], "data.includes.header": [ "true" ], "characterEncoding": [ "UTF-8" ], "filter.maxAge": [ "" ], "filter.minAge": [ "" ], "fileNameTimeRange_startDate": [ "" ], "fileNameTimeRange_mode": [ "OFF" ], "collectAdditionalFields": [ "false" ], "strictQuotes": [ "false" ], "TextFileFormat": [ "TEXT" ], "histogram.generation": [ "false" ], "incrementalMode": [ "false" ] }, "hadoopProperties": "", "dataStore": { "path": "/Data/Connections/Datameer server filesystem.dst", "uuid": "5875432e-da74-4947-8f81-7ba13347bdde" }, "validationStrategy": "DROP_RECORD", "maxLogErrors": 1000, "maxPreviewRecords": 5000, "fields": [ { "id": 31, "pattern": "", "acceptEmpty": true, "name": "URL", "origin": "0", "valueType": "{\"type\":\"STRING\"}", "include": true, "version": 3 }, { "id": 32, "pattern": "", "acceptEmpty": false, "name": "dasFileName", "origin": "fileInfo.fileName", "valueType": "{\"type\":\"STRING\"}", "include": false, "version": 3 }, { "id": 33, "pattern": "", "acceptEmpty": false, "name": "dasFilePath", "origin": "fileInfo.filePath", "valueType": "{\"type\":\"STRING\"}", "include": false, "version": 3 }, { "id": 34, "pattern": "", "acceptEmpty": false, "name": "dasLastModified", "origin": "fileInfo.lastModified", "valueType": "{\"type\":\"DATE\"}", "include": false, "version": 3 } ] }

Create Import/Link/Upload

Request method - POST

Description

Adds an import/link/upload file in Datameer.

The maximum length for a table description is 255 characters.

URI syntax

curl -u '<username>:<password>' -X POST -d @<job-payload>.json 'http://<Datameer-server-IP>:<port-number>/rest/import-job'

Example response

{ "configuration-id": 48, "status": "success" }

Additional notes

Only use REST for file uploads when running in local mode. If the file isn't local, use import jobs or data links instead of a file upload.

<job-payload>.json

  • The payload is a JSON representation of all information entered during the creation process using the web interface.

  • An easy method of creating a payload is to use REST to read an import job, data link, or workbook and change the JSON that is returned

Update Import/Link/Upload

Request method - PUT

Description

Edits an existing import/link/upload file in Datameer.

URI syntax

curl -u '<username>:<password>' -X PUT -d @<job-payload>.json 'http://<Datameer-serverIP>:<port-number>/rest/import-job/<job-configuration-id>'

Example response

{"status": "success"}

Additional notes

<job-payload>.json

  • The payload is a JSON representation of all information entered during the creation process using the web interface.

  • An easy method of creating a payload is to use REST to read an import job, data link, or workbook and change the JSON that is returned.

Delete Import/Link/Upload

Request method - DELETE

Description

Removes an import/link/upload file from Datameer.

URI syntax

curl -u '<username>:<password>' -X DELETE 'http://<Datameer-serverIP>:<port-number>/rest/import-job/<job-configuration-id>'

Example response

{"status": "success"}

Metadata for Import/Link/Upload

Request method - GET

Description

Returns data on the import/link/upload file including the path, number of records, time stamp, and ID.

URI syntax

curl -u '<username>:<password>' -X GET 'http://<Datameer-serverIP>:<port-number>/rest/data/import-job/{configuration-id}'

Example response

{ "path": "/Data/ImportJobs/DatameerImport.imp", "record-count": 20, "datas": [ { "timestamp": "Mar 5, 2013 10:44:59 AM", "record-count": 20, "id": 19372 } ]

Download Data of Import/Link/Upload

Request method - GET

Description

Returns the column names and records of the import/link/upload file.

The default limit for number of rows to download is 100,000, as this functionality is intended for small aggregated data sets. To adjust the record download limit, change the rest.download-data.records-max=100000 property in the <datameer_install>/conf/default.properties file. Setting the value to 0 unlocks the limit, but increasing the number could result in slower processing for the Datameer conductor. Your local system might also have download restrictions. If you want to download more than 100,000 rows, we suggest creating an export job instead.

URI syntax

curl -u '<username>:<password>' -X GET 'http://<Datameer-serverIP>:<port-number>/rest/data/import-job/{configuration-id}/download curl -u '<username>:<password>' -X GET 'http://<Datameer-serverIP>:<port-number>/rest/data/import-job/{configuration-id}/download?dataId={data-id}

Example response

"ID","NAME","Email","Status","Check" "1","""admin""","""admin@datameer.com""","0","true" "2","""Ajay""","""Ajay@datameer.com""","2","true" "3","""analyst""","""analyst@datameer.com""","1","true" "4","""Bob""","""Bob@datameer.com""","2","false" "5","""Alice""","""Alice@datameer.com""","2","false" "6","""John""","""John@datameer.com""","2","false" "7","""Andreas""","""Andreas@datameer.com""","2","true" "8","""Arthur""","""Arthur@datameer.com""","2","true" "9","""Dagmar""","""Dagmar@datameer.com""","2","false" "10","""Frank""","""Frank@datameer.com""","2","true" "11","""Johannes""","""Johannes@datameer.com""","2","true" "12","""Marko""","""Marko@datameer.com""","2","true" "13","""Marlon""","""Marlon@datameer.com""","2","true" "14","""Mike""","""Mike@datameer.com""","2","true" "15","""Orlando""","""Orlando@datameer.com""","2","true" "16","""Peter""","""Peter@datameer.com""","2","true" "17","""Ralf""","""Ralf@datameer.com""","2","false" "18","""Shirish""","""Shirish@datameer.com""","2","true" "19","""Stefan""","""Stefan@datameer.com""","2","true" "20","""Teresa""","""Teresa@datameer.com""","2","true"