Parallel computing is a necessity given the computational demands of data analytics and machine learning applications today. Parallel computing involves breaking down large problems into smaller pieces that can be solved by multiple processors at the same time. Efficient and scalable parallel applications can often utilize thousands of processor cores simultaneously to solve massive problems quickly.
Historically, high-performance computing (HPC) experts have parallelized programs manually, starting with understanding the structure of the algorithm. Then, they modify the code so that large data structures and computations are divided into smaller ones that are distributed across processors. Furthermore, performance experts add code to coordinate communication across processors since there are usually data dependencies across compute pieces for non-trivial problems. This is a cumbersome and time consuming process that requires significant expertise, which makes it challenging for many typical use cases.
Automatic parallelization or auto-parallelization, refers to converting sequential code to a parallel version automatically. It essentially automates the manual parallelization process done by HPC experts, and is necessary for democratization of compute power across various organizations and applications. However, automatic-parallelization is known as an extremely challenging process and has been a holy grail of computer science for decades.
To provide some level of scalability without automatic - to client-server distributed systems, and is able to schedule and execute tasks across multiple processors. However, the efficiency and scalability of this approach is significantly lower than parallel computing as done by HPC experts1. Hence, this approach is becoming less viable over time since the efficiency and scalability requirements of data-intensive applications is growing.
Bodo’s engine is the first data analytics solution that incorporates automatic-parallelization. It uses a new inferential compiler technology that infers the structure of the program and transforms it to a parallel version as an HPC expert would do. By understanding the standard Python APIs such as Pandas, Bodo promises to democratize compute power for all developers across organizations.
1: Kamburugamuve S, Wickramasinghe P, Ekanayake S, Fox GC. Anatomy of machine learning algorithm implementations in MPI, Spark, and Flink. The International Journal of High Performance Computing Applications. 2018;32(1):61-73. doi:10.1177/1094342017712976