Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

Version 1 Next »

Datameer is customizable using the Datameer SDK and public API to create plug-ins. Types of plug-ins currently available include functions, file types, export formats, and connections. Additional plug-in types will be supported in the future.

Prerequisites

  • Java 7 JDK
  • Ant 1.7 or newer
  • Cygwin DLL (for developers using Windows)
  • If you are installing a plug-in that uses extension points, make sure to install the plug-in that supplies those extension points first.

Extension Points

Extension points are the sockets where plug-in developers can plug-in their extensions. Currently there are a number of extension points available:

Extension Point

Description

datameer.dap.sdk.importjob.ImportFileType

Import file type (e.g., Apache log, CSV, etc.). This should be extended for adding support for custom file types.

datameer.dap.sdk.function.FunctionType

Function. This should be extended to add functions.

datameer.dap.sdk.datastore.DataStoreType

Connection type. This should be extended when support for new non-file based imports should be added. (e.g., Hive, HBase, etc.)

datameer.dap.sdk.authentication.AuthenticatorExtension

Defines custom authenticators.

datameer.dap.sdk.event.DatameerEventBusExtension

This should be extended to receive events provided by the Datameer Event Bus.

datameer.dap.sdk.exportjob.ExportFileType

Export file type (e.g., CSV, Avro, Parquet, etc.). This should be extended to add support for custom export file types.

Extensions

Extensions can be added for the available extension points. Every plug-in can contribute an arbitrary number of extensions to the Datameer system.

Development

Plug-ins can be developed using the plug-in SDK that is included in the Datameer release. When you have installed Datameer you will find the plug-in SDK in the following location where <version> is the version of Datameer you are using:

plugin-sdk-<version>.zip

Installing the Datameer SDK

To install the SDK, copy the zip file to your development machine and install it to the location of your choice using the following commands:

cd <destination>
unzip /<path>/plugin-sdk-<version>.zip
export DAS_PLUGIN_SDK_HOME=<path>

You are now ready to develop your first plug-in.

Creating a Plug-in

To create a plug-in

  1. Run the provided script to create a new plug-in project:

    $DAS_PLUGIN_SDK_HOME/bin/create-das-plugin.sh <project target dir>
    

    For example:

    $DAS_PLUGIN_SDK_HOME/bin/create-das-plugin.sh ~/development/das/my-first-plugin
    

    This creates a plug-in project under ~/development/das/my-first-plugin

  2. Change directories to the newly created project:

    cd ~/development/das/my-first-plugin
    
  3. Create Eclipse project files:

    ant eclipse
    
  4. Import the plug-in project to Eclipse using File > Import ... > Existing Projects into Workspace > select the project directory.
  5. You are now ready to start developing Datameer extensions.

Define a New Extension Point Within a Plug-in

  1. In the plugin.xml, add the name of the extension point you want to add. 

    For example:

    <?xml version="1.0" encoding="UTF-8"?>
    <plugin
       id="plugin-example"
       name="Example Plugin with Extension Point"
       version="${datameer.version}"
       provider-name="Datameer, Inc.">
    
       <requires>
          <import plugin="das.sdk"/>
       </requires>
    
       <extension-point id="datameer.plugin.example.ExampleExtensionPoint" name="Example Extension Point" />
    </plugin>

    Open your extension point and define your methods.
    For example:

    package datameer.plugin.example;
    
    import datameer.com.google.common.annotations.Beta;
    import datameer.dap.sdk.plugin.SingletonExtension;
    
    public abstract class ExampleExtensionPoint implements SingletonExtension {
    
        public static final String EXTENSION_POINT_NAME = ExampleExtensionPoint.class.getName();
    
        @Override
        public final String getExtensionPoint() {
            return EXTENSION_POINT_NAME;
        }
      
        /**
       Add your extension point specific methods here
       **/
        public abstract Sting foo();
    
    }

Documentation

The Javadoc of the Datameer SDK is available. 

$DAS_PLUGIN_SDK_HOME/doc/index.html 

Writing Extensions

See the following tutorials:

Plug-in File Format

Datameer plug-ins are zip files of the following structure:

File / Folder

Description

<root-dir>/lib/compile/

All jar files in this directory will be added to the plug-in classpath.

<root-dir>/lib/provided/

Only used for developing the plug-in. Jar files under this directory will be added to the classpath of your IDE. Ignored by Datameer.

<root-dir>/lib/runtime/

Only used for developing the plug-in. Jar files under this directory will be added to the classpath of your IDE. Ignored by Datameer.

<root-dir>/lib/test/

Only used for developing the plug-in. Jar files under this directory will be added to the classpath during unit test execution. Ignored by Datameer.

<root-dir>/classes-main/

This folder will be added to the plug-in classpath. Datameer will scan all classes for classes for annotations or implemented interfaces that are relevant to Datameer plug-in development.

<root-dir>/plugin.xml

Plug-in manifest containing plug-in meta data.

Versioning

Edit the files plugin.xml and build.properties to assign the appropriate names and version numbers before the plug-in is built.

Datameer will show the name in the UI which is used in the name tag. 

<path/workspace/plugin-name>/plugin.xml
<plugin
   id="plugin-name"
   name="Plugin Name"
   version="<SDK-version>-<plugin-version>"
   provider-name="MyCompany">
...

The plug-in file name is built from the name and version tag. 

<path/workspace/plugin-name>/build.properties
sdk.home=<path-to-SDK>
version=<SDK-version>-<plugin-version>
...

Keep the following rules in mind when filling in the ID and name of a custom plug-in:

1. The values of the ID and the name must be different. If not, Datameer shows an empty plug-in list
2. The value of the ID can't contain space characters.

Installing and Testing a Plug-in

  1. Bundle your plug-in as zip file:

    ant zip
    
  2. In Datameer, go to the Admin Tab > Plug-ins page.
  3. At the bottom of the page under Upload Plug-ins heading, click Choose File.
  4. Select the file and click Upload.

Pausing jobs being submitted to the cluster

To avoid duplicate jobs being run while the Datameer server is down you can pause the incoming job submissions with the Job Scheduler.

  • No labels