REST API Import Job, Data Link, and File Upload
INFO
Add the following command after the username and password to receive more meaningful HTTP errors:
-w"\n%{http_code}\n"
This command can be added to each call.
Create Import/ Link/ Upload
Request Method | POST |
Description | Adds an import/ link/ upload file in Datameer. INFO The maximum length for a table description is 255 characters. |
URI Syntax | curl -u '<username>:<password>' -X POST -d @<job-payload>.json 'http://<Datameer-server-IP>:<port-number>/rest/import-job' |
Example Response | { "configuration-id": 48, "status": "success" } |
Additional Notes | Only use REST for file uploads when running in local mode. If the file isn't local, use import jobs or data links instead of a file upload.
|
Read Import/ Link/ Upload
Request Method | GET |
Description | Returns information on a specified file imported/linked/uploaded within Datameer X including version number, class name, file information (uuid, path, description, name), run schedule, iterations set to keep, properties information, custom properties, and field data. |
URI Syntax | curl -u '<username>:<password>' -X GET 'http://<Datameer-serverIP>:<port-number>/rest/import-job/<job-configuration-id>' |
Example Response | { "version": "2.1.1", "className": "datameer.dap.common.entity.DataSourceConfigurationImpl", "file": { "uuid": "2d5d3741-9f5d-42e1-9d54-e38c849f3a5a", "path": "/Data/ImportJobs/URLS.imp", "description": "", "name": "URLS" }, "pullType": "MANUALLY", "minKeepCount": 1, "properties": { "GenericConfigurationImpl.temp-file-store": [ "f4cf6037-042a-4c56-99e3-74b59753729c" ], "fileType": [ "CSV" ], "filter.page.does.split.creation": [ "false" ], "external.store": [ "false" ], "delimiter": [ "," ], "csv.max-lines-per-record": [ "1" ], "file": [ "/Users/charlesbishop/Desktop/URLTEST.csv" ], "detectColumnDefinition": [ "SELECT_PARSE_AUTO" ], "quoteCharacter": [ "\"" ], "escapeCharacter": [ "" ], "recordSampleSize": [ "1000" ], "data.includes.header": [ "true" ], "characterEncoding": [ "UTF-8" ], "filter.maxAge": [ "" ], "filter.minAge": [ "" ], "fileNameTimeRange_startDate": [ "" ], "fileNameTimeRange_mode": [ "OFF" ], "collectAdditionalFields": [ "false" ], "strictQuotes": [ "false" ], "TextFileFormat": [ "TEXT" ], "histogram.generation": [ "false" ], "incrementalMode": [ "false" ] }, "hadoopProperties": "", "dataStore": { "path": "/Data/Connections/Datameer X server filesystem.dst", "uuid": "5875432e-da74-4947-8f81-7ba13347bdde" }, "validationStrategy": "DROP_RECORD", "maxLogErrors": 1000, "maxPreviewRecords": 5000, "fields": [ { "id": 31, "pattern": "", "acceptEmpty": true, "name": "URL", "origin": "0", "valueType": "{\"type\":\"STRING\"}", "include": true, "version": 3 }, { "id": 32, "pattern": "", "acceptEmpty": false, "name": "dasFileName", "origin": "fileInfo.fileName", "valueType": "{\"type\":\"STRING\"}", "include": false, "version": 3 }, { "id": 33, "pattern": "", "acceptEmpty": false, "name": "dasFilePath", "origin": "fileInfo.filePath", "valueType": "{\"type\":\"STRING\"}", "include": false, "version": 3 }, { "id": 34, "pattern": "", "acceptEmpty": false, "name": "dasLastModified", "origin": "fileInfo.lastModified", "valueType": "{\"type\":\"DATE\"}", "include": false, "version": 3 } ] } |
Update Import/ Link/ Upload
Request Method | PUT |
Description | Edits an existing import/ link/ upload file in Datameer. |
URI Syntax | curl -u '<username>:<password>' -X PUT -d @<job-payload>.json 'http://<Datameer-serverIP>:<port-number>/rest/import-job/<job-configuration-id>' |
Example Response | {"status": "success"} |
Additional Notes |
|
Verification |
Delete Import/ Link/ Upload
Request Method | DELETE |
Description | Removes an import/ link/ upload file from Datameer. |
URI Syntax | curl -u '<username>:<password>' -X DELETE 'http://<Datameer-serverIP>:<port-number>/rest/import-job/<job-configuration-id>' |
Example Response | {"status": "success"} |
View Metadata for Import/ Link/ Upload
Request Method | GET |
Description | Returns data on the import/ link/ upload file including the path, number of records, time stamp, and ID. |
URI Syntax | curl -u '<username>:<password>' -X GET 'http://<Datameer-serverIP>:<port-number>/rest/data/import-job/{configuration-id}' |
Example Response | { "path": "/Data/ImportJobs/DatameerImport.imp", "record-count": 20, "datas": [ { "timestamp": "Mar 5, 2013 10:44:59 AM", "record-count": 20, "id": 19372 } ] |
Download Data of Import/ Link/ Upload
Request Method | GET |
Description | Returns the column names and records of the import/ link/ upload file INFO The default limit for number of rows to download is 100,000, as this functionality is intended for small aggregated data sets. To adjust the record download limit, change the |
URI Syntax | curl -u '<username>:<password>' -X GET 'http://<Datameer-serverIP>:<port-number>/rest/data/import-job/{configuration-id}/download curl -u '<username>:<password>' -X GET 'http://<Datameer-serverIP>:<port-number>/rest/data/import-job/{configuration-id}/download?dataId={data-id} |
Example Response | "ID","NAME","Email","Status","Check" "1","""admin""","""admin@datameer.com""","0","true" "2","""Ajay""","""Ajay@datameer.com""","2","true" "3","""analyst""","""analyst@datameer.com""","1","true" "4","""Bob""","""Bob@datameer.com""","2","false" "5","""Alice""","""Alice@datameer.com""","2","false" "6","""John""","""John@datameer.com""","2","false" "7","""Andreas""","""Andreas@datameer.com""","2","true" "8","""Arthur""","""Arthur@datameer.com""","2","true" "9","""Dagmar""","""Dagmar@datameer.com""","2","false" "10","""Frank""","""Frank@datameer.com""","2","true" "11","""Johannes""","""Johannes@datameer.com""","2","true" "12","""Marko""","""Marko@datameer.com""","2","true" "13","""Marlon""","""Marlon@datameer.com""","2","true" "14","""Mike""","""Mike@datameer.com""","2","true" "15","""Orlando""","""Orlando@datameer.com""","2","true" "16","""Peter""","""Peter@datameer.com""","2","true" "17","""Ralf""","""Ralf@datameer.com""","2","false" "18","""Shirish""","""Shirish@datameer.com""","2","true" "19","""Stefan""","""Stefan@datameer.com""","2","true" "20","""Teresa""","""Teresa@datameer.com""","2","true" |
Additional Notes | The data ID can be found a job's details under Current Data. The maximum number of records that can be returned by the API is configurable in # The maximum number of records that should be returned by the download data REST API. # All additional records will be ignored and 206 HTTP response code (PARTIAL CONTENT) is returned. rest.download-data.records-max=100000 |
View Sheet Details of Import/ Link/ Upload
Request Method | GET |
Description | eturns the column names and data type of an import/ link/ upload file. |
URI Syntax | curl -u '<username>:<password>' -X GET 'http://<Datameer-serverIP>:<port-number>/rest/sheet-details/{job-configuration-id}' curl -u '<username>:<password>' -X GET 'http://<Datameer-serverIP>:<port-number>/rest/sheet-details?file=<path to import job>.<extension>' |
Example Response | { "columns": [ { "name": "id", "type": "INTEGER" }, { "name": "ident", "type": "STRING" }, { "name": "type", "type": "STRING" }, { "name": "name", "type": "STRING" }, { "name": "latitude_deg", "type": "FLOAT" }, { "name": "longitude_deg", "type": "FLOAT" }, { "name": "elevation_ft", "type": "INTEGER" }, { "name": "continent", "type": "STRING" }, { "name": "iso_country", "type": "STRING" }, { "name": "iso_region", "type": "STRING" }, { "name": "municipality", "type": "STRING" }, { "name": "scheduled_service", "type": "STRING" }, { "name": "gps_code", "type": "STRING" }, { "name": "iata_code", "type": "STRING" }, { "name": "local_code", "type": "STRING" }, { "name": "home_link", "type": "STRING" }, { "name": "wikipedia_link", "type": "STRING" }, { "name": "keywords", "type": "STRING" } ], "file": { "path": "/Examples/Public Data/Flight Delays/Airports.upl", "uuid": "39b5961a-4caa-4b9e-880e-d40f03b7543a" } } |
View Import Job Dependencies
Request Method | GET |
Description | Returns the hierarchy of up- and downstream dependencies for a given import job. |
Prerequisites | This API requires enabling /wiki/spaces/DASSB100/pages/32569527063. |
URI Syntax | curl -u '<username>:<password>' -X GET 'http://<server>:<port>/api/import-job/<configurationId>/dependencies[?direction=<upstream | downstream>][&level=<number>]' |
Example Response | { "dependencies": [ { "id": 23, "file": { "path": "/Users/admin/WorkbookTest", "name": "WorkbookTest", "type": "wbk", "description": "", "uuid": "db942d97-4785-436d-938b-45053e12bb75" } }, { "id": 2, "file": { "path": "/Data/Connections/Datameer X server filesystem", "name": "Datameer X server filesystem", "type": "dst", "description": "A data store for the local filesystem where Datameer X is running. Note, this works ONLY for Datameer X Personal, Workgroup, and Trial editions.\n", "uuid": "78a2fb59-375e-4eae-b338-956ef806f7b1" } } ], "fileCountWithNoReadPermission": 0, "_links": { "self": { "href": "http://localhost:8088/api/import-job/22/dependencies" } } |
Additional Notes | Path parameters:
Query parameters: |
List All Imports/ Links/ Uploads
Request Method | GET |
Description | Returns a list of all the import/ link/ upload file in Datameer. This list displays the description, ID number, name, and the path of the file. |
URI Syntax | curl -u '<username>:<password>' -X GET 'http://<Datameer-serverIP>:<port-number>/rest/import-job' |
Example Response | [ { "description": "apache log file", "id": 5048, "name": "foo4965", "path": "/ImportJob2/foo4965.imp" }, { "description": "apache log file", "id": 5049, "name": "foo4966", "path": "/ImportJob2/foo4966.imp" }, { "description": "apache log file", "id": 5050, "name": "foo4967", "path": "/ImportJob2/foo4967.imp" }, { "description": "apache log file", "id": 5051, "name": "foo4968", "path": "/ImportJob2/foo4968.imp" } ] |
Start Import/ Link/ Upload
Request Method | POST |
Description | Runs an import/ link/ upload file. |
Prerequisites | |
URI Syntax | curl -u '<username>:<password>' -X POST 'http://<Datameer-server-IP>:<port-number>/rest/job-execution?configuration=<import-job-configuration-id>' |
Example Response | { "job-execution-id": 26, "job-execution-trigger": "RESTAPI", "job-execution-user": "admin", "status": "success" } |
Additional Notes | Only use REST for file uploads when running in local mode. If the file isn't local, use import jobs or data links instead of a file upload. |
Cancel Import/ Link/ Upload
Request Method | POST |
Description | Cancels the run order of a import /link/ upload file. |
URI Syntax | curl -u '<username>:<password>' -X POST 'http://<Datameer-serverIP>:<port-number>/rest/job-execution/job-kill/<import-job-execution-id> |
Example Response | This command doesn't return a response. |