Sequence Files with Metadata

Overview

Datameer X provides a plug-in (plugin-sequencefile-v6) for importing data stored from a sequence file given the following conditions:

• The value writable of the sequence file is org.apache.hadoop.io.Text.
• The text value is a record with columns delimited by a character.
• The schema can be determined at run time given the sequence file metadata.
  
  

Setting Up the Import Job

If your SequenceFile data satisfies the condition listed above you can setup an Import Job to import the data into Datameer. From the Import Job Wizard, select Sequence File with Metadata from the File Type menu:

Next, configure the import job providing the paths to the sequence file data as well as specifying the delimiter character. 

Metadata Schema Strategy

In order to translate your delimited records, you must provide Datameer X with an implementation of a plug-in interface MetadataSchemaStrategy. This is a simple interface which allows developers to translate a record using the provided sequence file metadata into a Datameer X schema. Once the strategy is implemented and plugged into Datameer X it is for selection from the drop-down list and can be used to handle the mapping of all data contained in your sequence files 

public interface MetadataSchemaStrategy extends SingletonExtension {

    public static final String EXTENSION_POINT_ID = "datameer.das.plugin.sequencefile.importjob.MetadataSchemaStrategy";

    /**
     * Creates the Datameer X input schema based on the SequenceFile.Metadata of the file
     * being imported currently.

     * @param metadata metadata pulled from the SequenceFile currently being imported
     * @return the array of Field's that Datameer X should use a schema for this data
     */
    public Field[] createSchemaFromMetadata(SequenceFile.Metadata metadata);

    /**
     * Extracts the Field value for a given field based on the raw record which has already been split on the incoming
     * delimiter
     *
     * @param field the current Field
     * @param rawFields the array of String's resulting from splitting of the Text value
     * @return the Field's value
     */
    public Object parseFieldValue(Field field, String[] rawFields);

    /**
     * Provides the name of the strategy
     *
     * @return the strategy's name to be used in the drop down list
     */
    public String getName();
}

Take a look at an example implementation, Native Metadata Schema Strategy.