IMSM: An Interval Migration Based Approach for Skew Mitigation in MapReduce

Background: Extreme growth of data necessitates the need for high-performance computing. MapReduce is among the most sought-after platform for processing large-scale data. Research work and analysis of the existing system has revealed its performance bottlenecks and areas of concern. MapReduce has the problem of skew on its processing nodes. This paper proposes an algorithm for MapReduce to balance the load and eliminate the skew on Map tasks. It reduces the execution time of job by lowering the completion time of the slowest task. Methods: The proposed method performs one-time settlement of load balancing among the Map tasks by analyzing the expected completion time of the Map tasks and redistributes the load. It uses intervals to migrate the overloaded or slows tasks and append them on the under loaded tasks. Results: Experiments revealed an improvement of up to 1.3x by implementing the proposed strategy. Comparison of the proposed technique with other relevant strategies exhibits a better distribution of load among Map tasks and lower level of the skew. Evaluation is done using different workloads. Conclusion: A significant improvement is observed in the performance and reduced completion time of job.