Datameer is customizable using the Datameer SDK and public API to create plug-ins. Types of plug-ins currently available include functions, file types, export formats, and connections. Additional plug-in types will be supported in the future.
Prerequisites
- Java 7 JDK
- Ant 1.7 or newer
- Cygwin DLL (for developers using Windows)
- If you are installing a plug-in that uses extension points, make sure to install the plug-in that supplies those extension points first.
Extension Points
Extension points are the sockets where plug-in developers can plug-in their extensions. Currently there are a number of extension points available:
Extension Point | Description |
---|---|
| Import file type (e.g., Apache log, CSV, etc.). This should be extended for adding support for custom file types. |
| Function. This should be extended to add functions. |
| Connection type. This should be extended when support for new non-file based imports should be added. (e.g., Hive, HBase, etc.) |
| Defines custom authenticators. |
| This should be extended to receive events provided by the Datameer Event Bus. |
| Export file type (e.g., CSV, Avro, Parquet, etc.). This should be extended to add support for custom export file types. |
Extensions
Extensions can be added for the available extension points. Every plug-in can contribute an arbitrary number of extensions to the Datameer system.
Development
Plug-ins can be developed using the plug-in SDK that is included in the Datameer release. When you have installed Datameer you will find the plug-in SDK in the following location where <version> is the version of Datameer you are using:
plugin-sdk-<version>.zip
Installing the Datameer SDK
To install the SDK, copy the zip file to your development machine and install it to the location of your choice using the following commands:
cd <destination> unzip /<path>/plugin-sdk-<version>.zip export DAS_PLUGIN_SDK_HOME=<path>
You are now ready to develop your first plug-in.
Creating a Plug-in
To create a plug-in
Run the provided script to create a new plug-in project:
$DAS_PLUGIN_SDK_HOME/bin/create-das-plugin.sh <project target dir>
For example:
$DAS_PLUGIN_SDK_HOME/bin/create-das-plugin.sh ~/development/das/my-first-plugin
This creates a plug-in project under
~/development/das/my-first-plugin
Change directories to the newly created project:
cd ~/development/das/my-first-plugin
Create Eclipse project files:
ant eclipse
- Import the plug-in project to Eclipse using File > Import ... > Existing Projects into Workspace > select the project directory.
- You are now ready to start developing Datameer extensions.
Define a New Extension Point Within a Plug-in
In the
plugin.xml
, add the name of the extension point you want to add.For example:
<?xml version="1.0" encoding="UTF-8"?> <plugin id="plugin-example" name="Example Plugin with Extension Point" version="${datameer.version}" provider-name="Datameer, Inc."> <requires> <import plugin="das.sdk"/> </requires> <extension-point id="datameer.plugin.example.ExampleExtensionPoint" name="Example Extension Point" /> </plugin>
Open your extension point and define your methods.
For example:package datameer.plugin.example; import datameer.com.google.common.annotations.Beta; import datameer.dap.sdk.plugin.SingletonExtension; public abstract class ExampleExtensionPoint implements SingletonExtension { public static final String EXTENSION_POINT_NAME = ExampleExtensionPoint.class.getName(); @Override public final String getExtensionPoint() { return EXTENSION_POINT_NAME; } /** Add your extension point specific methods here **/ public abstract Sting foo(); }
Documentation
The Javadoc of the Datameer SDK is available.
$DAS_PLUGIN_SDK_HOME/doc/index.html
Writing Extensions
See the following tutorials:
- Tutorial01 - Building an Input Adaptor for Custom File Types
- Tutorial02 - Building Custom Functions
- Tutorial04 - Building a Custom Authenticator
- Tutorial05 - Building an Import Adaptor for Custom Connections
- Tutorial06 - Building an Export Adaptor for Custom Connections
Plug-in File Format
Datameer plug-ins are zip files of the following structure:
File / Folder | Description |
---|---|
| All jar files in this directory will be added to the plug-in classpath. |
| Only used for developing the plug-in. Jar files under this directory will be added to the classpath of your IDE. Ignored by Datameer. |
| Only used for developing the plug-in. Jar files under this directory will be added to the classpath of your IDE. Ignored by Datameer. |
| Only used for developing the plug-in. Jar files under this directory will be added to the classpath during unit test execution. Ignored by Datameer. |
| This folder will be added to the plug-in classpath. Datameer will scan all classes for classes for annotations or implemented interfaces that are relevant to Datameer plug-in development. |
| Plug-in manifest containing plug-in meta data. |
Versioning
Edit the files plugin.xml
and build.properties
to assign the appropriate names and version numbers before the plug-in is built.
Datameer will show the name in the UI which is used in the name tag.
<plugin id="plugin-name" name="Plugin Name" version="<SDK-version>-<plugin-version>" provider-name="MyCompany"> ...
The plug-in file name is built from the name and version tag.
sdk.home=<path-to-SDK> version=<SDK-version>-<plugin-version> ...
Keep the following rules in mind when filling in the ID and name of a custom plug-in:
1. The values of the ID and the name must be different. If not, Datameer shows an empty plug-in list
2. The value of the ID can't contain space characters.
Installing and Testing a Plug-in
Learn more about installing a plug-in though the user interface (UI).
Bundle your plug-in as zip file:
ant zip
- In Datameer, go to the Admin Tab > Plug-ins page.
- At the bottom of the page under Upload Plug-ins heading, click Choose File.
- Select the file and click Upload.
Pausing jobs being submitted to the cluster
To avoid duplicate jobs being run while the Datameer server is down you can pause the incoming job submissions with the Job Scheduler.