Datameer's Smart Execution™ technology dynamically chooses the best execution engine for your Datameer X job depending on the job configuration and data volume. This feature also uses a combination of multiple execution engines for separate tasks of a Datameer X job by logically allocating mixed workloads according to available resources for maximum processing efficiency.
The applications of Datameer's Smart Execution™ are valuable with the emergence of new computational frameworks joining the Hadoop ecosystem. Data sets come in many different sizes and some computing frameworks that work better for larger data sizes are not as efficient when working with smaller amounts of data. Smart Execution™ adapts the frameworks that perform best for the size of the data set.
Datameer X default execution framework
- Optimized MapReduce (Tez)
Datameer X execution frameworks with Smart Execution™
Data Flow Using Datameer's Smart Execution™
Datameer's goal is making data analysis simple whether you are working with 100 petabytes or just a few kilobytes. Smart Execution™ is a transparent process that runs behind the scenes to make the computation step faster and more efficient. After a job is started (import jobs, export jobs, file uploads, workbooks), Datameer X selects the best computational framework based on the size of your data. The data then runs though YARN which is the resource manager for all the computational engines. Finally, the data is stored using Hadoop Distributed File System (HDFS).
The Job Graph is a visualization to show details of the computational framework used for each task of a job.
To view the Job Graph:
- Select the job (Import/Upload/Export/ Workbook) and click on "Show Details".
- Scroll to the tab 'History' and click the required "Job ID".
- Open the "Job Graph" tab.
To view the Job Graph for a Workbook optionally:
- Open the Workbook.
- Click on the "Job Graph" icon from the tool bar.
- View the Job Graph.